GlusterFS 1.2 Multi-server Config Example

From GlusterDocumentation

1 Introduction
2 Version
3 The Servers
4 Client Configuration
5 Starting it up
6 The Author

Introduction

The configuration examples in this article will show how to configure four servers, each with a single storage device, into one larger GlusterFS device. Several performance options are shown in the examples below.

NOTE: These are minimal configuration files. Many other options and features exist. Please refer to the other pages in this wiki for further information.

[edit]

Version

The configuration files below were designed for use with glusterfs-1.3.0-pre2.2

[edit]

The Servers

Each server consists of a boot drive (/dev/sda1) and a large RAID array (/dev/sdb1). The RAID array is mounted at /mnt/raid in these examples. All servers and clients are in the 192.168.1.0/24 network. The facts that it is a 3.5TB array or formatted with xfs are not relevant, as you will see.

Server 1: IP address 192.168.1.33
/etc/glusterfs/ss1.vol

volume ss1
  type storage/posix           # POSIX FS translator
  option directory /mnt/raid   # Export this directory
end-volume

volume iothreads1    #iothreads can give performance a boost
   type performance/io-threads
   option thread-count 8
   subvolumes ss1
end-volume

### Add network serving capability to above brick.
volume server-ss1
  type protocol/server
  option transport-type tcp/server  # For TCP/IP transport
  subvolumes iothreads1
  option auth.ip.ss1.allow 192.168.1.* # Allow access to "ss1" volume
  option auth.ip.iothreads1.allow 192.168.1.* # Allow access to "ss1" volume
end-volume

Notice that each volume section is linked to the one above it by the 'subvolumes' statement.

Server 2: IP address 192.168.1.34
/etc/glusterfs/ss2.vol

Note that very little changes here. (only 5 characters - one of them is just a comment though)

volume ss2
  type storage/posix           # POSIX FS translator
  option directory /mnt/raid   # Export this directory
end-volume

volume iothreads2    #iothreads can give performance a boost
   type performance/io-threads
   option thread-count 8
   subvolumes ss2
end-volume

### Add network serving capability to above brick.
volume server-ss2
  type protocol/server
  option transport-type tcp/server  # For TCP/IP transport   
  subvolumes iothreads2
  option auth.ip.ss2.allow 192.168.1.* # Allow access to "ss2" volume
  option auth.ip.iothreads2.allow 192.168.1.* # Allow access to "ss2" volume
end-volume

/etc/glusterfs/ss3.vol (192.168.1.35) and /etc/glusterfs/ss4.vol (192.168.1.36) are not included here. From the above two examples, you can easily work out the pattern of how to create the other two files.

[edit]

Client Configuration

Here is where it all gets tied together...

Client Configuration: IP address 192.168.1.xxx (just not .33 through .36, of course)
/etc/glusterfs/client.vol

volume ss1c
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
  option remote-host 192.168.1.33      # IP address of the remote brick
#  option remote-subvolume ss1          # name of the remote volume (io-threads not included)
  option remote-subvolume iothreads1   # name of the remote volume
end-volume

volume ss2c
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
  option remote-host 192.168.1.34      # IP address of the remote brick
#  option remote-subvolume ss2          # name of the remote volume (io-threads not included)
  option remote-subvolume iothreads2   # name of the remote volume
end-volume

volume ss3c
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
  option remote-host 192.168.1.35      # IP address of the remote brick
#  option remote-subvolume ss3          # name of the remote volume (io-threads not included)
  option remote-subvolume iothreads3   # name of the remote volume
end-volume

volume ss4c
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
  option remote-host 192.168.1.36      # IP address of the remote brick
#  option remote-subvolume ss4          # name of the remote volume (io-threads not included)
  option remote-subvolume iothreads4   # name of the remote volume
end-volume


volume bricks
  type cluster/unify
  subvolumes ss1c ss2c ss3c ss4c
  option scheduler alu
  option alu.limits.min-free-disk  60GB              # Stop creating files when free-space lt 60GB
  option alu.limits.max-open-files 10000
  option alu.order disk-usage:read-usage:write-usage:open-files-usage:disk-speed-usage
  option alu.disk-usage.entry-threshold 2GB          # Units in KB, MB and GB are allowed
  option alu.disk-usage.exit-threshold  60MB         # Units in KB, MB and GB are allowed
  option alu.open-files-usage.entry-threshold 1024
  option alu.open-files-usage.exit-threshold 32
  option alu.stat-refresh.interval 10sec
end-volume

volume writebehind   #writebehind improves write performance a lot
  type performance/write-behind
  option aggregate-size 131072 # in bytes
  subvolumes bricks
end-volume

[edit]

Starting it up

Once all the configuration files have been created, do the following on each of the servers...

1. mount /dev/sdb1 /mnt/raid
2. glusterfsd -f /etc/glusterfs/ss1.vol -L NORMAL -l /var/log/glusterfs/glusterfsd.log

Repeat steps 1 and 2 for each of the servers while remembering to change the number for the ss1.vol file name for each server.

Once all four servers are running, on the client, execute the following...

glusterfs -f /etc/glusterfs/client.vol -l /var/log/glusterfs/glusterfs.log -L NORMAL /mnt/cluster

You should now have the cluster mounted on the client under /mnt/cluster. A 'df -h' should show the combined storage capacity of all four servers (14TB in this example)

[edit]