GlusterFS 1.3 High Availability Storage with GlusterFS
From GlusterDocumentation
Contents |
Introduction
The configuration examples in this article will show how to configure three servers, each with a single storage device, with the aim of creating a completely Redundant Server-Side GlusterFS Storage Cluster.
Basically we're trying to create a traditional SAN where all of the redundancy is handled on the server side. The clients are not concerned with maintaining data integrity and focus on their other primary roles.
Basic Overview
Basic Summary
Here you can see that Automatic File Replication and the Unify Translators happen on a separate server-side network. Each file is replicated three times which will give us 3 exact copies of the entire Namespace and Data volumes. Also in the event that a server is removed and added to the cluster the Self-Heal ability of the Unify Translator will ensure data remains intact.
This configuration aims to ensure the maximum amount of Redundancy and High Availability of Data. It is possible to lose 2/3 Storage Servers and yet the cluster will still operate. When the servers come back online all Self-Heal traffic will occur on a separate network and will not effect the Storage Clients.
Possible Problems
We are still investigating mount point redundancy
This is not entirely scalable, however for High Availability this is not really a problem.
There is a lot of Network and Disk IO load placed on the Server-Side.
Version
The configuration files below were designed for use with glusterfs-1.3-pre5.2
The Configuration Files
Each of the servers configurations are almost identical except for the IP Addresses change...
/etc/glusterfs/glusterfs-server.vol
Each server has a slightly different configuration to this as they will refer to a local posix volume and two remote client volumes. Essentially the only thing that changes are the names of the volumes and the IP Addresses.
volume mailspool-ds
type storage/posix
option directory /home/export/mailspool
end-volume
volume mailspool-ns
type storage/posix
option directory /home/export/mailspool-ns
end-volume
volume mailspool-santa2-ds
type protocol/client
option transport-type tcp/client
option remote-host 192.168.252.2
option remote-subvolume mailspool-ds
end-volume
volume mailspool-santa2-ns
type protocol/client
option transport-type tcp/client
option remote-host 192.168.252.2
option remote-subvolume mailspool-ns
end-volume
volume mailspool-santa3-ds
type protocol/client
option transport-type tcp/client
option remote-host 192.168.252.3
option remote-subvolume mailspool-ds
end-volume
volume mailspool-santa3-ns
type protocol/client
option transport-type tcp/client
option remote-host 192.168.252.3
option remote-subvolume mailspool-ns
end-volume
volume mailspool-ns-afr
type cluster/afr
subvolumes mailspool-ns mailspool-santa2-ns mailspool-santa3-ns
option replicate *:3
end-volume
volume mailspool-ds-afr
type cluster/afr
subvolumes mailspool-ds mailspool-santa2-ds mailspool-santa3-ds
option replicate *:3
end-volume
volume mailspool-unify
type cluster/unify
subvolumes mailspool-ds-afr
option namespace mailspool-ns-afr
option scheduler rr
end-volume
volume mailspool
type performance/io-threads
option thread-count 8
option cache-size 64MB
subvolumes mailspool-unify
end-volume
volume server
type protocol/server
option transport-type tcp/server
subvolumes mailspool
option auth.ip.mailspool-ds.allow 192.168.252.*,127.0.0.1
option auth.ip.mailspool-ns.allow 192.168.252.*,127.0.0.1
option auth.ip.mailspool.allow *
end-volume
/etc/glusterfs/glusterfs-client.vol Currently we are using DNS Round Robin to connect to the cluster until the High Availability Translator in version 1.4 is available.
volume santa
type protocol/client
option transport-type tcp/client # for TCP/IP transport
option remote-host roundrobin.gluster.local # DNS Round Robin pointing towards all 3 GlusterFS servers
option remote-subvolume mailspool # name of the remote volume
end-volume
volume writeback
type performance/write-behind
option aggregate-size 131072 # unit in bytes
subvolumes santa1
end-volume
volume readahead
type performance/read-ahead
option page-size 65536 # unit in bytes
option page-count 16 # cache per file = (page-count x page-size)
subvolumes writeback
end-volume
Starting it up
On each of the servers...
glusterfsd -L NORMAL -l /var/log/glusterfs/glusterfsd.log
And on the client server...
glusterfs -L DEBUG -l /var/log/glusterfs/glusterfsd.log -f /etc/glusterfs/glusterfs-client.vol /mnt/happy
The Author
- Author: Paul England -> <pengland (at) wxc.co.nz> -> kritical -> #gluster -> irc.freenode.net
