GlusterFS

GlusterFS Distribution

Below, we create a distributed volume using two bricks (XFS filesystems.) This spreads IO and files among two bricks.

# gluster peer status
No peers present
# gluster peer probe rhs-lab2
Probe successful
# gluster peer status
Number of Peers: 1

Hostname: rhs-lab2
Uuid: 6b6c9ffc-da79-4d24-8325-086d44869338
State: Peer in Cluster (Connected)
# gluster peer probe rhs-lab3
Probe successful
# gluster peer probe rhs-lab4
Probe successful
# gluster peer status
Number of Peers: 3

Hostname: rhs-lab2
Uuid: 6b6c9ffc-da79-4d24-8325-086d44869338
State: Peer in Cluster (Connected)

Hostname: rhs-lab3
Uuid: cbcd508e-5f80-4224-91df-fd5f8e12915d
State: Peer in Cluster (Connected)

Hostname: rhs-lab4
Uuid: a02f68d8-88af-4b79-92d8-1057dd85af45
State: Peer in Cluster (Connected)
# gluster volume create dist rhs-lab1:/data/dist rhs-lab2:/data/dist
Creation of volume dist has been successful. Please start the volume to access data.

# gluster volume info
 
Volume Name: dist
Type: Distribute
Volume ID: f9758871-20dc-4728-9576-a5bb5b24ca4f
Status: Created
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: rhs-lab1:/data/dist
Brick2: rhs-lab2:/data/dist

# gluster volume start dist
Starting volume dist has been successful

# gluster volume info
 
Volume Name: dist
Type: Distribute
Volume ID: f9758871-20dc-4728-9576-a5bb5b24ca4f
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: rhs-lab1:/data/dist
Brick2: rhs-lab2:/data/dist

# mount -t glusterfs rhs-lab1:/dist /mnt/dist

GlusterFS Mirroring

Below, we mirror data between two bricks (XFS volumes). This creates a redundant system and also allows for read performance to be improved.

# gluster volume create mirror replica 2 rhs-lab1:/data/mirror rhs-lab2:/data/mirror
Creation of volume mirror has been successful. Please start the volume to access data.
# gluster volume start mirror
Starting volume mirror has been successful
# gluster volume info mirror
 
Volume Name: mirror
Type: Replicate
Volume ID: 4edacef8-982c-46a9-be7e-29e34fa40f95
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: rhs-lab1:/data/mirror
Brick2: rhs-lab2:/data/mirror
# install -d /mnt/mirror
# mount -t glusterfs rhs-lab1:/mirror /mnt/mirror

Growing GlusterFS

Now we will add a new brick to our distributed filesystem. We will run a rebalance (optional) to get the files distributed ideally. This will involve distributing some existing files on to our new brick on rhs-lab3:

# gluster volume add-brick dist rhs-lab3:/data/dist
Add Brick successful
# gluster volume rebalance dist start
Starting rebalance on volume dist has been successful

After the rebalance, our distributed GlusterFS filesystem will have optimal performance and one third of the files will have moved to rhs-lab3.

# gluster volume rebalance dist status
                                    Node Rebalanced-files          size       scanned      failures         status
                               ---------      -----------   -----------   -----------   -----------   ------------
                               localhost                0            0            0            0      completed
                                rhs-lab4                0            0            0            0      completed
                                rhs-lab3                0            0            0            0      completed
                                rhs-lab2                0            0            0            0      completed

Growing a GlusterFS Replicated Volume

You can grow a replicated volume by adding pairs of bricks:

# gluster volume add-brick mirror rhs-lab3:/data/mirror rhs-lab4:/data/mirror
Add Brick successful
# gluster volume info mirror
 
Volume Name: mirror
Type: Distributed-Replicate
Volume ID: 4edacef8-982c-46a9-be7e-29e34fa40f95
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: rhs-lab1:/data/mirror
Brick2: rhs-lab2:/data/mirror
Brick3: rhs-lab3:/data/mirror
Brick4: rhs-lab4:/data/mirror

GlusterFS Brick Migration

Here is how you migrate data off of an existing brick and on to a new brick:

# gluster volume replace-brick dist rhs-lab3:/data/dist rhs-lab4:/data/dist start
replace-brick started successfully
# gluster volume replace-brick dist rhs-lab3:/data/dist rhs-lab4:/data/dist status
Number of files migrated = 0        Migration complete 
# gluster volume replace-brick dist rhs-lab3:/data/dist rhs-lab4:/data/dist commit
replace-brick commit successful
# gluster volume info
 
Volume Name: dist
Type: Distribute
Volume ID: f9758871-20dc-4728-9576-a5bb5b24ca4f
Status: Started
Number of Bricks: 3
Transport-type: tcp
Bricks:
Brick1: rhs-lab1:/data/dist
Brick2: rhs-lab2:/data/dist
Brick3: rhs-lab4:/data/dist
 
Volume Name: mirror
Type: Distributed-Replicate
Volume ID: 4edacef8-982c-46a9-be7e-29e34fa40f95
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: rhs-lab1:/data/mirror
Brick2: rhs-lab2:/data/mirror
Brick3: rhs-lab3:/data/mirror
Brick4: rhs-lab4:/data/mirror

Removing a Brick

# gluster volume remove-brick dist rhs-lab4:/data/dist start
Remove Brick start successful

GlusterFS Security

Currently, any GlusterFS peer can join your volume if it exists on your LAN. Securing GlusterFS can be accomplished with iptables by blocking TCP ports.