GlusterFS 1.3 P2P Cluster with Auto Healing
From GlusterDocumentation
Contents |
News Update
- 05-04-08 Now self hosting. New Link--> P2P-LAMP.
- 04-28-08 Site is down. Looks like I will be self hosting since I cannot contact hoster.
- 10-28-07 P2P-LAMP Project has gone live. This page will not be updated by me except for corrections. Check out the project--> P2P-LAMP
- 10-19-07 MYSQL integration coming soon. Testing...
- 10-17-07 gnanet found a bug in the init script. Fixed. Thank You! :o)
- 10-14-07 Scripts have been updated to remove Spread and Wackamole. Added UCARP. :o)
- 10-13-07 Well! Were have you been all my life? I found a perfect Virtual IP Distribution System.
- 10-13-07 A new Virtual IP Distribution System will need to be implemented. Since none exist, one will be coded. More news when available...
- 10-5-07 !!Warning!! Virtual IP distribution and failover is broke in Wackamole. I am researching a solution.
- 10-7-07 OK. Temporary solution found. Changes to config files and init script seemed to help. Still one small issue with detecting who has what VIP. Researching...
Introduction
This is my implementation of what I like to call P2P Clustering.
When I researched Clustering I was excited to find that GNU/Linux has a lot of Clustering projects. I soon found that none of the solutions implemented what I thought a cluster should be.(correct me if I am wrong) I also noticed pesky little things like...
1.) Some required kernel modules. I hate hacking a perfect kernel and hoping the System boots after a kernel upgrade.
2.) Some require you to recompile your apps to use special libraries to access the Cluster Resources. Ya Right!
3.) Some are not Posix compliant. Heck No!
I also noticed they all had the same apparent weakness. "Load Balancers". I know, this is were someone says, "Hey! You can have more than one Load Balancer!" You are correct, you can. But what if both, or all three, or all 4 load balancers go down. Then someone says, "Oh, that will never happen." Better knock on wood.
Before I decided to slap my own solution together I took a peek at Grid Computing. Grid computing is two or more computers acting as one Super Computer. This seemed promising, but I imediately found issues with this. Some applications will need to be re-coded and recompiled to run on a Grid. Configurations will need to be changed also. This seemed like a lot of work to maintain. Also none of the GNU/Linux Grid projects have been actively maintained lately. Scary!
I eventually gave up on these solutions and decided to implement my idea of a cluster. I did some research and found all the tools needed. We just need to put them together to creat a P2P Cluster.
This tutorial is partially based on a tutorial by Paul England (aka Kritical).GlusterFS High Availability Storage with GlusterFS. Extended to add P2P Virtual IP Management, Cluster Healing, and Apache.
Pre-Setup
In this setup a mount point of /raid5 will be distributed among the nodes. This is a 3 disk Raid5 @ /dev/md0 on each server. You can just create the directory /raid5 or whatever name You choose. Since we will be distributing the /etc/httpd and /var/www directories we will need to empty the directories. But first we need to backup the contents of the directories on at least one server, unless you wish to recreate the contents. A good place to backup would be a new directory /tmp/backup/etc/httpd and /tmp/backup/var/www. Now empty the new glusterfs mount points, /etc/httpd and /var/www. We will replace the files backed up. But we will use the distributed mount point. This will be done at the end of this tutorial.
Tools
2.) GlusterFS - Cluster Filesystem Yup Yup!
3.) Ntp - Network Time Synchronization (recommended)
4.) Apache - Web Server (optional)
These apps may be included in your distro. If not, you will need to download source and compile. I used CentOS 5 and every repository under the SUN. I still needed to compile some things. This tutorial will not go into details of compiling or installing the required tools.
UCARP
UCARP needs to be installed on all servers.
No special config files should be required as of v1.3. They are included with the source.
Just make sure after compiling and installing place the vip-up.sh and vip-down.sh from the source to /etc/.(make sure they are executable)
If you did a packaged install here are the scripts-->
Config File --> vip-up.sh
#! /bin/sh exec 2> /dev/null /sbin/ip addr add "$2"/24 dev "$1"
Config File --> vip-down.sh
#! /bin/sh exec 2> /dev/null /sbin/ip addr del "$2"/24 dev "$1"
Glusterfs Server(glusterfsd)
Glusterfs needs to be installed on all servers.
Server 1
Config File --> glusterfs-server.vol
volume raid5-httpd-conf-ds
type storage/posix
option directory /raid5/etc/httpd
end-volume
volume raid5-httpd-conf-ns
type storage/posix
option directory /raid5-ns/etc/httpd
end-volume
volume raid5-httpd-conf-centos-2-ds
type protocol/client
option transport-type tcp/client
option remote-host 192.168.43.129
option remote-subvolume raid5-httpd-conf-ds
option transport-timeout 10
end-volume
volume raid5-httpd-conf-centos-2-ns
type protocol/client
option transport-type tcp/client
option remote-host 192.168.43.129
option remote-subvolume raid5-httpd-conf-ns
option transport-timeout 10
end-volume
volume raid5-httpd-conf-centos-3-ds
type protocol/client
option transport-type tcp/client
option remote-host 192.168.43.130
option remote-subvolume raid5-httpd-conf-ds
option transport-timeout 10
end-volume
volume raid5-httpd-conf-centos-3-ns
type protocol/client
option transport-type tcp/client
option remote-host 192.168.43.130
option remote-subvolume raid5-httpd-conf-ns
option transport-timeout 10
end-volume
volume raid5-httpd-conf-ns-afr
type cluster/afr
subvolumes raid5-httpd-conf-ns raid5-httpd-conf-centos-2-ns raid5-httpd-conf-centos-3-ns
option replicate *:3
option scheduler random
end-volume
volume raid5-httpd-conf-ds-afr
type cluster/afr
subvolumes raid5-httpd-conf-ds raid5-httpd-conf-centos-2-ds raid5-httpd-conf-centos-3-ds
option replicate *:3
option scheduler random
end-volume
volume raid5-httpd-conf-unify
type cluster/unify
subvolumes raid5-httpd-conf-ds-afr
option namespace raid5-httpd-conf-ns-afr
option scheduler random
end-volume
volume raid5-httpd-conf
type performance/io-threads
option thread-count 8
option cache-size 64MB
subvolumes raid5-httpd-conf-unify
end-volume
volume raid5-var-www-ds
type storage/posix
option directory /raid5/var/www
end-volume
volume raid5-var-www-ns
type storage/posix
option directory /raid5-ns/var/www
end-volume
volume raid5-var-www-centos-2-ds
type protocol/client
option transport-type tcp/client
option remote-host 192.168.43.129
option remote-subvolume raid5-var-www-ds
option transport-timeout 10
end-volume
volume raid5-var-www-centos-2-ns
type protocol/client
option transport-type tcp/client
option remote-host 192.168.43.129
option remote-subvolume raid5-var-www-ns
option transport-timeout 10
end-volume
volume raid5-var-www-centos-3-ds
type protocol/client
option transport-type tcp/client
option remote-host 192.168.43.130
option remote-subvolume raid5-var-www-ds
option transport-timeout 10
end-volume
volume raid5-var-www-centos-3-ns
type protocol/client
option transport-type tcp/client
option remote-host 192.168.43.130
option remote-subvolume raid5-var-www-ns
option transport-timeout 10
end-volume
volume raid5-var-www-ns-afr
type cluster/afr
subvolumes raid5-var-www-ns raid5-var-www-centos-2-ns raid5-var-www-centos-3-ns
option replicate *:3
option scheduler random
end-volume
volume raid5-var-www-ds-afr
type cluster/afr
subvolumes raid5-var-www-ds raid5-var-www-centos-2-ds raid5-var-www-centos-3-ds
option replicate *:3
option scheduler random
end-volume
volume raid5-var-www-unify
type cluster/unify
subvolumes raid5-var-www-ds-afr
option namespace raid5-var-www-ns-afr
option scheduler random
end-volume
volume raid5-var-www
type performance/io-threads
option thread-count 8
option cache-size 64MB
subvolumes raid5-var-www-unify
end-volume
volume server
type protocol/server
option transport-type tcp/server
subvolumes raid5-httpd-conf raid5-var-www
option auth.ip.raid5-httpd-conf-ds.allow 192.168.43.*,127.0.0.1
option auth.ip.raid5-httpd-conf-ns.allow 192.168.43.*,127.0.0.1
option auth.ip.raid5-httpd-conf.allow 127.0.0.1
option auth.ip.raid5-var-www-ds.allow 192.168.43.*,127.0.0.1
option auth.ip.raid5-var-www-ns.allow 192.168.43.*,127.0.0.1
option auth.ip.raid5-var-www.allow 127.0.0.1
end-volume
Server 2
Config File --> glusterfs-server.vol
volume raid5-httpd-conf-ds
type storage/posix
option directory /raid5/etc/httpd
end-volume
volume raid5-httpd-conf-ns
type storage/posix
option directory /raid5-ns/etc/httpd
end-volume
volume raid5-httpd-conf-centos-1-ds
type protocol/client
option transport-type tcp/client
option remote-host 192.168.43.128
option remote-subvolume raid5-httpd-conf-ds
option transport-timeout 10
end-volume
volume raid5-httpd-conf-centos-1-ns
type protocol/client
option transport-type tcp/client
option remote-host 192.168.43.128
option remote-subvolume raid5-httpd-conf-ns
option transport-timeout 10
end-volume
volume raid5-httpd-conf-centos-3-ds
type protocol/client
option transport-type tcp/client
option remote-host 192.168.43.130
option remote-subvolume raid5-httpd-conf-ds
option transport-timeout 10
end-volume
volume raid5-httpd-conf-centos-3-ns
type protocol/client
option transport-type tcp/client
option remote-host 192.168.43.130
option remote-subvolume raid5-httpd-conf-ns
option transport-timeout 10
end-volume
volume raid5-httpd-conf-ns-afr
type cluster/afr
subvolumes raid5-httpd-conf-ns raid5-httpd-conf-centos-1-ns raid5-httpd-conf-centos-3-ns
option replicate *:3
option scheduler random
end-volume
volume raid5-httpd-conf-ds-afr
type cluster/afr
subvolumes raid5-httpd-conf-ds raid5-httpd-conf-centos-1-ds raid5-httpd-conf-centos-3-ds
option replicate *:3
option scheduler random
end-volume
volume raid5-httpd-conf-unify
type cluster/unify
subvolumes raid5-httpd-conf-ds-afr
option namespace raid5-httpd-conf-ns-afr
option scheduler random
end-volume
volume raid5-httpd-conf
type performance/io-threads
option thread-count 8
option cache-size 64MB
subvolumes raid5-httpd-conf-unify
end-volume
volume raid5-var-www-ds
type storage/posix
option directory /raid5/var/www
end-volume
volume raid5-var-www-ns
type storage/posix
option directory /raid5-ns/var/www
end-volume
volume raid5-var-www-centos-1-ds
type protocol/client
option transport-type tcp/client
option remote-host 192.168.43.128
option remote-subvolume raid5-var-www-ds
option transport-timeout 10
end-volume
volume raid5-var-www-centos-1-ns
type protocol/client
option transport-type tcp/client
option remote-host 192.168.43.128
option remote-subvolume raid5-var-www-ns
option transport-timeout 10
end-volume
volume raid5-var-www-centos-3-ds
type protocol/client
option transport-type tcp/client
option remote-host 192.168.43.130
option remote-subvolume raid5-var-www-ds
option transport-timeout 10
end-volume
volume raid5-var-www-centos-3-ns
type protocol/client
option transport-type tcp/client
option remote-host 192.168.43.130
option remote-subvolume raid5-var-www-ns
option transport-timeout 10
end-volume
volume raid5-var-www-ns-afr
type cluster/afr
subvolumes raid5-var-www-ns raid5-var-www-centos-1-ns raid5-var-www-centos-3-ns
option replicate *:3
option scheduler random
end-volume
volume raid5-var-www-ds-afr
type cluster/afr
subvolumes raid5-var-www-ds raid5-var-www-centos-1-ds raid5-var-www-centos-3-ds
option replicate *:3
option scheduler random
end-volume
volume raid5-var-www-unify
type cluster/unify
subvolumes raid5-var-www-ds-afr
option namespace raid5-var-www-ns-afr
option scheduler random
end-volume
volume raid5-var-www
type performance/io-threads
option thread-count 8
option cache-size 64MB
subvolumes raid5-var-www-unify
end-volume
volume server
type protocol/server
option transport-type tcp/server
subvolumes raid5-httpd-conf raid5-var-www
option auth.ip.raid5-httpd-conf-ds.allow *
option auth.ip.raid5-httpd-conf-ns.allow *
option auth.ip.raid5-httpd-conf.allow *
option auth.ip.raid5-var-www-ds.allow *
option auth.ip.raid5-var-www-ns.allow *
option auth.ip.raid5-var-www.allow *
end-volume
Server 3
Config File --> glusterfs-server.vol
volume raid5-httpd-conf-ds
type storage/posix
option directory /raid5/etc/httpd
end-volume
volume raid5-httpd-conf-ns
type storage/posix
option directory /raid5-ns/etc/httpd
end-volume
volume raid5-httpd-conf-centos-1-ds
type protocol/client
option transport-type tcp/client
option remote-host 192.168.43.128
option remote-subvolume raid5-httpd-conf-ds
option transport-timeout 10
end-volume
volume raid5-httpd-conf-centos-1-ns
type protocol/client
option transport-type tcp/client
option remote-host 192.168.43.128
option remote-subvolume raid5-httpd-conf-ns
option transport-timeout 10
end-volume
volume raid5-httpd-conf-centos-2-ds
type protocol/client
option transport-type tcp/client
option remote-host 192.168.43.129
option remote-subvolume raid5-httpd-conf-ds
option transport-timeout 10
end-volume
volume raid5-httpd-conf-centos-2-ns
type protocol/client
option transport-type tcp/client
option remote-host 192.168.43.129
option remote-subvolume raid5-httpd-conf-ns
option transport-timeout 10
end-volume
volume raid5-httpd-conf-ns-afr
type cluster/afr
subvolumes raid5-httpd-conf-ns raid5-httpd-conf-centos-1-ns raid5-httpd-conf-centos-2-ns
option replicate *:3
option scheduler random
end-volume
volume raid5-httpd-conf-ds-afr
type cluster/afr
subvolumes raid5-httpd-conf-ds raid5-httpd-conf-centos-1-ds raid5-httpd-conf-centos-2-ds
option replicate *:3
option scheduler random
end-volume
volume raid5-httpd-conf-unify
type cluster/unify
subvolumes raid5-httpd-conf-ds-afr
option namespace raid5-httpd-conf-ns-afr
option scheduler random
end-volume
volume raid5-httpd-conf
type performance/io-threads
option thread-count 8
option cache-size 64MB
subvolumes raid5-httpd-conf-unify
end-volume
volume raid5-var-www-ds
type storage/posix
option directory /raid5/var/www
end-volume
volume raid5-var-www-ns
type storage/posix
option directory /raid5-ns/var/www
end-volume
volume raid5-var-www-centos-1-ds
type protocol/client
option transport-type tcp/client
option remote-host 192.168.43.128
option remote-subvolume raid5-var-www-ds
option transport-timeout 10
end-volume
volume raid5-var-www-centos-1-ns
type protocol/client
option transport-type tcp/client
option remote-host 192.168.43.128
option remote-subvolume raid5-var-www-ns
option transport-timeout 10
end-volume
volume raid5-var-www-centos-2-ds
type protocol/client
option transport-type tcp/client
option remote-host 192.168.43.129
option remote-subvolume raid5-var-www-ds
option transport-timeout 10
end-volume
volume raid5-var-www-centos-2-ns
type protocol/client
option transport-type tcp/client
option remote-host 192.168.43.129
option remote-subvolume raid5-var-www-ns
option transport-timeout 10
end-volume
volume raid5-var-www-ns-afr
type cluster/afr
subvolumes raid5-var-www-ns raid5-var-www-centos-1-ns raid5-var-www-centos-2-ns
option replicate *:3
option scheduler random
end-volume
volume raid5-var-www-ds-afr
type cluster/afr
subvolumes raid5-var-www-ds raid5-var-www-centos-1-ds raid5-var-www-centos-2-ds
option replicate *:3
option scheduler random
end-volume
volume raid5-var-www-unify
type cluster/unify
subvolumes raid5-var-www-ds-afr
option namespace raid5-var-www-ns-afr
option scheduler random
end-volume
volume raid5-var-www
type performance/io-threads
option thread-count 8
option cache-size 64MB
subvolumes raid5-var-www-unify
end-volume
volume server
type protocol/server
option transport-type tcp/server
subvolumes raid5-httpd-conf raid5-var-www
option auth.ip.raid5-httpd-conf-ds.allow 192.168.43.*,127.0.0.1
option auth.ip.raid5-httpd-conf-ns.allow 192.168.43.*,127.0.0.1
option auth.ip.raid5-httpd-conf.allow 127.0.0.1
option auth.ip.raid5-var-www-ds.allow 192.168.43.*,127.0.0.1
option auth.ip.raid5-var-www-ns.allow 192.168.43.*,127.0.0.1
option auth.ip.raid5-var-www.allow 127.0.0.1
end-volume
Glusterfs Client(glusterfs)
Glusterfs Client is included in the Glusterfs install and will need to be setup on all servers.
Note: I disabled round robin on all clients because this did not work and did not suite the P2P design of this setup. :o)
Client 1
Config File --> glusterfs-client.vol
volume centos-etc-httpd
type protocol/client
option transport-type tcp/client
# option remote-host roundrobin.gluster.local
option remote-host 127.0.0.1
option remote-subvolume raid5-httpd-conf
end-volume
volume centos-var-www
type protocol/client
option transport-type tcp/client
# option remote-host roundrobin.gluster.local
option remote-host 127.0.0.1
option remote-subvolume raid5-var-www
end-volume
volume writeback
type performance/write-behind
option aggregate-size 131072
subvolumes centos-1
end-volume
volume readahead
type performance/read-ahead
option page-size 65536
option page-count 16
subvolumes writeback
end-volume
Client 2
Config File --> glusterfs-client.vol
volume centos-etc-httpd
type protocol/client
option transport-type tcp/client
# option remote-host roundrobin.gluster.local
option remote-host 127.0.0.1
option remote-subvolume raid5-httpd-conf
end-volume
volume centos-var-www
type protocol/client
option transport-type tcp/client
# option remote-host roundrobin.gluster.local
option remote-host 127.0.0.1
option remote-subvolume raid5-var-www
end-volume
volume writeback
type performance/write-behind
option aggregate-size 131072
subvolumes centos-1
end-volume
volume readahead
type performance/read-ahead
option page-size 65536
option page-count 16
subvolumes writeback
end-volume
Client 3
Config File --> glusterfs-client.vol
volume centos-etc-httpd
type protocol/client
option transport-type tcp/client
# option remote-host roundrobin.gluster.local
option remote-host 127.0.0.1
option remote-subvolume raid5-httpd-conf
end-volume
volume centos-var-www
type protocol/client
option transport-type tcp/client
# option remote-host roundrobin.gluster.local
option remote-host 127.0.0.1
option remote-subvolume raid5-var-www
end-volume
volume writeback
type performance/write-behind
option aggregate-size 131072
subvolumes centos-1
end-volume
volume readahead
type performance/read-ahead
option page-size 65536
option page-count 16
subvolumes writeback
end-volume
Init
You can start each service in order or use a Sysvinit script to start everything. :o)
Now this is my first Init script ever. So please be nice.(constructive input appreciated)
The Init script will need to be setup on all servers.
Server 1
Script --> /etc/init.d/p2p-cluster
#!/bin/bash
#
# /etc/rc.d/init.d/
#
#
#
#
#
# Source function library.
. /etc/init.d/functions
PIDFILE=/var/run/p2p-cluster.pid
start() {
echo -n "Syncing clock..."
ntpdate -u pool.ntp.org
echo
echo -n "Starting UCARP..."
nohup /usr/local/sbin/ucarp -i eth1 -s 192.168.233.128 -v 1 -p password -a 192.168.233.140 -u /etc/vip-up.sh -d /etc/vip-down.sh > /var/log/ucarp/ucarp.log &
RETVAL=$?
[ "$RETVAL" = 0 ] && touch /var/lock/subsys/ucarp
echo
echo -n "Starting glusterfsd..."
/usr/local/sbin/glusterfsd
RETVAL=$?
[ "$RETVAL" = 0 ] && touch /var/lock/subsys/glusterfsd
echo
echo -n "Mounting /etc/httpd..."
/usr/local/sbin/glusterfs -f /usr/local/etc/glusterfs/glusterfs-client.vol -n centos-etc-httpd /etc/httpd
RETVAL=$?
[ "$RETVAL" = 0 ] && touch /var/lock/subsys/etc-httpd
echo
echo -n "Mounting /var/www..."
/usr/local/sbin/glusterfs -f /usr/local/etc/glusterfs/glusterfs-client.vol -n centos-var-www /var/www
RETVAL=$?
[ "$RETVAL" = 0 ] && touch /var/lock/subsys/var-www
echo
echo -n "Healing /etc/httpd..."
sleep 2 && find /etc/httpd -type f -exec head -n 1 {} \;
echo
echo -n "Healing /var/www..."
sleep 2 && find /var/www -type f -exec head -n 1 {} \;
echo
echo -n "Starting Apache..."
/etc/init.d/httpd start
RETVAL=$?
[ "$RETVAL" = 0 ]
echo
}
stop() {
echo -n "Shutting down Apache..."
killall httpd
RETVAL=$?
[ "$RETVAL" = 0 ]
echo
echo -n "Shutting down UCARP... "
killall ucarp
RETVAL=$?
[ "$RETVAL" = 0 ] && rm -f /var/lock/subsys/ucarp
echo
echo -n "Shutting down glusterfsd..."
killall glusterfsd
RETVAL=$?
[ "$RETVAL" = 0 ] && rm -f /var/lock/subsys/glusterfsd
echo
echo -n "Dis-Mounting /etc/httpd..."
umount /etc/httpd
RETVAL=$?
[ "$RETVAL" = 0 ] && rm -f /var/lock/subsys/etc-httpd
echo
echo -n "Dis-Mounting /var/www..."
umount /var/www
RETVAL=$?
[ "$RETVAL" = 0 ] && rm -f /var/lock/subsys/var-www
echo
}
case "$1" in
start)
start
;;
stop)
stop
;;
status)
status p2p-cluster
;;
restart)
stop
start
;;
*)
echo "Usage: {start|stop|status|restart}"
exit 1
;;
esac
exit $?
Server 2
Script --> /etc/init.d/p2p-cluster
#!/bin/bash
#
# /etc/rc.d/init.d/
#
#
#
#
#
# Source function library.
. /etc/init.d/functions
PIDFILE=/var/run/p2p-cluster
start() {
echo -n "Syncing clock..."
ntpdate -u pool.ntp.org
echo
echo -n "Starting UCARP..."
nohup /usr/local/sbin/ucarp -i eth1 -s 192.168.233.129 -v 1 -p password -a 192.168.233.140 -u /etc/vip-up.sh -d /etc/vip-down.sh > /var/log/ucarp/ucarp.log &
RETVAL=$?
[ "$RETVAL" = 0 ] && touch /var/lock/subsys/ucarp
echo
echo -n "Starting glusterfsd..."
/usr/local/sbin/glusterfsd
RETVAL=$?
[ "$RETVAL" = 0 ] && touch /var/lock/subsys/glusterfsd
echo
echo -n "Mounting /etc/httpd..."
/usr/local/sbin/glusterfs -f /usr/local/etc/glusterfs/glusterfs-client.vol -n centos-etc-httpd /etc/httpd
RETVAL=$?
[ "$RETVAL" = 0 ] && touch /var/lock/subsys/etc-httpd
echo
echo -n "Mounting /var/www..."
/usr/local/sbin/glusterfs -f /usr/local/etc/glusterfs/glusterfs-client.vol -n centos-var-www /var/www
RETVAL=$?
[ "$RETVAL" = 0 ] && touch /var/lock/subsys/var-www
echo
echo -n "Healing /etc/httpd..."
sleep 2 && find /etc/httpd -type f -exec head -n 1 {} \;
echo
echo -n "Healing /var/www..."
sleep 2 && find /var/www -type f -exec head -n 1 {} \;
echo
echo -n "Starting Apache..."
/etc/init.d/httpd start
RETVAL=$?
[ "$RETVAL" = 0 ]
echo
}
stop() {
echo -n "Shutting down Apache..."
killall httpd
RETVAL=$?
[ "$RETVAL" = 0 ]
echo
echo -n "Shutting down UCARP... "
killall ucarp
RETVAL=$?
[ "$RETVAL" = 0 ] && rm -f /var/lock/subsys/ucarp
echo
echo -n "Shutting down glusterfsd..."
killall glusterfsd
RETVAL=$?
[ "$RETVAL" = 0 ] && rm -f /var/lock/subsys/glusterfsd
echo
echo -n "Dis-Mounting /etc/httpd..."
umount /etc/httpd
RETVAL=$?
[ "$RETVAL" = 0 ] && rm -f /var/lock/subsys/etc-httpd
echo
echo -n "Dis-Mounting /var/www..."
umount /var/www
RETVAL=$?
[ "$RETVAL" = 0 ] && rm -f /var/lock/subsys/var-www
echo
}
case "$1" in
start)
start
;;
stop)
stop
;;
status)
status p2p-cluster
;;
restart)
stop
start
;;
*)
echo "Usage: {start|stop|status|restart}"
exit 1
;;
esac
exit $?
Server 3
Script --> /etc/init.d/p2p-cluster
#!/bin/bash
#
# /etc/rc.d/init.d/
#
#
#
#
#
# Source function library.
. /etc/init.d/functions
PIDFILE=/var/run/p2p-cluster
start() {
echo -n "Syncing clock..."
ntpdate -u pool.ntp.org
echo
echo -n "Starting UCARP..."
nohup /usr/local/sbin/ucarp -i eth1 -s 192.168.233.130 -v 1 -p password -a 192.168.233.140 -u /etc/vip-up.sh -d /etc/vip-down.sh > /var/log/ucarp/ucarp.log &
RETVAL=$?
[ "$RETVAL" = 0 ] && touch /var/lock/subsys/ucarp
echo
echo -n "Starting glusterfsd..."
/usr/local/sbin/glusterfsd
RETVAL=$?
[ "$RETVAL" = 0 ] && touch /var/lock/subsys/glusterfsd
echo
echo -n "Mounting /etc/httpd..."
/usr/local/sbin/glusterfs -f /usr/local/etc/glusterfs/glusterfs-client.vol -n centos-etc-httpd /etc/httpd
RETVAL=$?
[ "$RETVAL" = 0 ] && touch /var/lock/subsys/etc-httpd
echo
echo -n "Mounting /var/www..."
/usr/local/sbin/glusterfs -f /usr/local/etc/glusterfs/glusterfs-client.vol -n centos-var-www /var/www
RETVAL=$?
[ "$RETVAL" = 0 ] && touch /var/lock/subsys/var-www
echo
echo -n "Healing /etc/httpd..."
sleep 2 && find /etc/httpd -type f -exec head -n 1 {} \;
echo
echo -n "Healing /var/www..."
sleep 2 && find /var/www -type f -exec head -n 1 {} \;
echo
echo -n "Starting Apache..."
/etc/init.d/httpd start
RETVAL=$?
[ "$RETVAL" = 0 ]
echo
}
stop() {
echo -n "Shutting down Apache..."
killall httpd
RETVAL=$?
[ "$RETVAL" = 0 ]
echo
echo -n "Shutting down UCARP... "
killall ucarp
RETVAL=$?
[ "$RETVAL" = 0 ] && rm -f /var/lock/subsys/ucarp
echo
echo -n "Shutting down glusterfsd..."
killall glusterfsd
RETVAL=$?
[ "$RETVAL" = 0 ] && rm -f /var/lock/subsys/glusterfsd
echo
echo -n "Dis-Mounting /etc/httpd..."
umount /etc/httpd
RETVAL=$?
[ "$RETVAL" = 0 ] && rm -f /var/lock/subsys/etc-httpd
echo
echo -n "Dis-Mounting /var/www..."
umount /var/www
RETVAL=$?
[ "$RETVAL" = 0 ] && rm -f /var/lock/subsys/var-www
echo
}
case "$1" in
start)
start
;;
stop)
stop
;;
status)
status p2p-cluster
;;
restart)
stop
start
;;
*)
echo "Usage: {start|stop|status|restart}"
exit 1
;;
esac
exit $?
Final Steps
Note: Make sure your log files exist! If not create them.
Now we start the services on each server.
$ /etc/init.d/p2p-cluster start
Just make sure that ucarp, glusterfsd, and 2 instances of glusterfs are running. Check with the command:
$ ps -A
If all is good you will see the mounts. Check with this command:
$ df
You should see the 2 new mounts.
glusterfs /etc/httpd
glusterfs /var/www
We should also ping the Virtual IPs from another host to make sure ucarp is working properly.
OK now we can replace the files from /tmp/backup to the distributed mounts /etc/httpd and /var/www.
You can now do this from the Server that has the backup. The files should distribute to each server.
If all is good, kill the p2p-cluster on each server, then restart the service:
$/etc/init.d/p2p-cluster stop
$/etc/init.d/p2p-cluster start
Obviously I could go on from here. But I believe if you have gotten this far you should be able to finish on your own. Remember if you need imediate help Gluster has a IRC channel--> #gluster @ irc.gnu.org. :o)
Important Notes
1.) I run My setup on CentOS 5 in VMWare Server GSX.
2.) You will need to make sure that the system clocks on all servers are synced. Some clock skew is ok, but not hours or days.(use a NTP client) You will have all kinds of issues with Replication and Self Healing if clocks are out of sync. (just a warning)
3.) The best way to bring a fresh setup online is to bring up one server at a time using the /etc/init.d/p2p-cluster sysvinit script. Bringing 2 servers online at the same time might cause Self Healing issues, but I have never seen this. (just another warning)
Author
Me! Kevin Evans (kevev) @ #gluster -> irc.gnu.org
Big Credits to Paul Englan kritical -> #gluster -> irc.freenode.net
