GlusterFS 1.2 Configuration Example for Four Bricks

From GlusterDocumentation

This information is out of date
and does not contain information related to the current version of Gluster

Documentation Home

This article was last updated for the version 1.2.3 of GlusterFS

[edit]

Introduction

In this article, I will show you how to configure GlusterFS to simulate a cluster with:

one single mount point
4 storage nodes serving the files

I said simulate since we will deploy such a cluster on only one computer, but you can as well set it up over physical different computers by changing the IP addresses.

[edit]

Configure the 4 storage server nodes

We create a directory for each storage server to be created:

toad@vlk:~/gfs$ sudo mkdir /home/cfs{0,1,2,3}
toad@vlk:~/gfs$ sudo chmod 1777 /home/cfs{0,1,2,3}
toad@vlk:~/gfs$ ls -ld /home/cfs[0-3]
drwxrwxrwt  2 root root 4096 2007-01-14 13:42 /home/cfs0
drwxrwxrwt  2 root root 4096 2007-01-14 13:42 /home/cfs1
drwxrwxrwt  2 root root 4096 2007-01-14 13:42 /home/cfs2
drwxrwxrwt  2 root root 4096 2007-01-14 13:42 /home/cfs3
toad@vlk:~/gfs$

Then we write 4 configuration files for server volume specification where we will merely change the storage destination and the listening port

[edit]

Server volume for the 1st node: server0.vol

volume brick
        type storage/posix
        option directory /home/cfs0
end-volume

volume server
        type protocol/server
        option transport-type tcp/server
        option listen-port 6996
        option bind-address 127.0.0.1
        subvolumes brick
        option auth.ip.brick.allow 127.0.0.1
end-volume

[edit]

Server volume for the 2nd node: server1.vol

volume brick
        type storage/posix
        option directory /home/cfs1
end-volume

volume server
        type protocol/server
        option transport-type tcp/server
        option listen-port 6997
        option bind-address 127.0.0.1
        subvolumes brick
        option auth.ip.brick.allow 127.0.0.1
end-volume

[edit]

Server volume for the 3rd node: server2.vol

volume brick
        type storage/posix
        option directory /home/cfs2
end-volume

volume server
        type protocol/server
        option transport-type tcp/server
        option listen-port 6998
        option bind-address 127.0.0.1
        subvolumes brick
        option auth.ip.brick.allow 127.0.0.1
end-volume

[edit]

Server volume for the 4th node: server3.vol

volume brick
        type storage/posix
        option directory /home/cfs3
end-volume

volume server
        type protocol/server
        option transport-type tcp/server
        option listen-port 6999
        option bind-address 127.0.0.1
        subvolumes brick
        option auth.ip.brick.allow 127.0.0.1
end-volume

[edit]

Configure the client node: client.vol

volume client0
 type protocol/client
 option transport-type tcp/client
 option remote-host 127.0.0.1
 option remote-port 6996
 option remote-subvolume brick
end-volume

volume client1
 type protocol/client
 option transport-type tcp/client
 option remote-host 127.0.0.1
 option remote-port 6997
 option remote-subvolume brick
end-volume

volume client2
 type protocol/client
 option transport-type tcp/client
 option remote-host 127.0.0.1
 option remote-port 6998
 option remote-subvolume brick
end-volume

volume client3
 type protocol/client
 option transport-type tcp/client
 option remote-host 127.0.0.1
 option remote-port 6999
 option remote-subvolume brick
end-volume

volume bricks
  type cluster/unify
  subvolumes client0 client1 client2 client3
  option rr.limits.min-free-disk 10GB # In later versions, its changed to % based value.
  option scheduler rr
end-volume

### Add writebehind feature
volume writebehind
  type performance/write-behind
  option aggregate-size 131072 # unit in bytes
  subvolumes bricks
end-volume

### Add readahead feature
volume readahead
  type performance/read-ahead
  option page-size 65536     # unit in bytes
  option page-count 16       # cache per file  = (page-count x page-size)
  subvolumes writebehind
end-volume

[edit]

The big picture

Here is how the previous configuration is looking like now:

[edit]

Start the storage cluster

We need to start the four instances of the glusterfs daemon to serve each directory:

toad@vlk:~/gfs$ sudo glusterfsd -f ./server0.vol
toad@vlk:~/gfs$ sudo glusterfsd -f ./server1.vol
toad@vlk:~/gfs$ sudo glusterfsd -f ./server2.vol
toad@vlk:~/gfs$ sudo glusterfsd -f ./server3.vol
toad@vlk:~/gfs$ ps w -C glusterfsd
  PID TTY      STAT   TIME COMMAND
13524 ?        Ss     0:00 [glusterfsd]
13526 ?        Ss     0:00 [glusterfsd]
13528 ?        Ss     0:00 [glusterfsd]
13532 ?        Ss     0:00 [glusterfsd]
toad@vlk:~/gfs$

Then we mount the cluster on one mount point:

toad@vlk:~/gfs$ mkdir ./glusterfs
toad@vlk:~/gfs$ sudo glusterfs -f ./client.vol ./glusterfs
toad@vlk:~/gfs$

We check if the partition is mounted:

toad@vlk:~/gfs$ mount | tail -1
glusterfs:13548 on /home/toad/gfs/glusterfs type fuse (rw,allow_other,default_permissions)

Hooray ! You have now a glusterFS cluster running on your local computer :)

[edit]

Use your cluster

Now you can create files...

toad@vlk:~/gfs$ touch ./glusterfs/file{0,1,2,3,4,5,6,7,8}
toad@vlk:~/gfs$

.. and check how the scheduler rr (round-robin) is behaving:

toad@vlk:~/gfs$ find /home/cfs[0-3] -type f
/home/cfs0/file3
/home/cfs0/file7
/home/cfs1/file2
/home/cfs1/file6
/home/cfs2/file1
/home/cfs2/file5
/home/cfs3/file0
/home/cfs3/file4
/home/cfs3/file8
toad@vlk:~/gfs$

[edit]

What's next ?

Now that you now how to set up a cluster of GlusterFS nodes, you can learn how to encrypt your communication channel using Stunnel on the next tutorial right here.

[edit]