[Gluster-users] AFR write performance

[Michael McCallister]

Greetings,

I am having what I perceive to be AFR performance problems.  Before I 
get to that, I will briefly describe the setup...

/** Setup **/

I have glusterfs up and running using the gluster optimized fuse module 
on the latest Centos 5 kernel (2.6.18-164.6.1.el5 #1 SMP) running on two 
machines (server and client volume configurations are below).  Both the 
server and client run on both machines.  Both servers are connected by a 
single CAT6 cable running directly into the Gigabit NICs dedicated to 
this task (no switch is used).   My goal is simply to mirror files 
across both servers.  As far as the files themselves, it is mixed, but 
there are many and about 90% of them are under 50K.  Each server runs a 
Quad core Q6600 processor with 8GB or RAM.  The disks are quite speedy - 
running 15K RPM SAS drives hooked to a 3ware controller (RAID 5 with a 
512MB cache).  The filesystem is ext3 mounted with noatime.  Writing 
directly to the ext3 partition with dd if=/dev/zero of=/sites/disktest 
bs=1M count=2048 yields 2147483648 bytes (2.1 GB) copied, 4.68686 
seconds, 458 MB/s.  Kernel optimizations on both servers outside of a 
stock CentOS 5 setup include:

3ware controller specific to avoid iowait latency under load:

echo 64 > /sys/block/sda/queue/max_sectors_kb
/sbin/blockdev --setra 8192 /dev/sda
echo 128 > /sys/block/sda/queue/nr_requests
echo 64 > /sys/block/sda/device/queue_depth
echo 10 > /proc/sys/vm/swappiness
echo 16 > /proc/sys/vm/page-cluster
echo 2 > /proc/sys/vm/dirty_background_ratio
echo 40 > /proc/sys/vm/dirty_ratio

Tweaks for better network performance (sysctl.conf):

net/core/rmem_max = 8738000
net/core/wmem_max = 8738000

net/ipv4/tcp_rmem = 8192 873800 8738000
net/ipv4/tcp_wmem = 4096 873800 8738000


/** Gluster Results **/

It should be noted that for the below test results I did not see high 
CPU or IOwait times during the tests.  Also, there are no other active 
processes running on either server.  Doing a simple write test using "dd 
if=/dev/zero of=/sites/sites/glustertest bs=1M count=2048" I am seeing:

2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 27.8451 seconds, 77.1 MB/s

Which is acceptable for my purposes.  I expected around 80 MB/s with the 
gigabit NICs being the obvious bottleneck.  So for a more real-world 
test using the actual files to be clustered, I took a small subset of 
the files (22016 of them - 440M in total) and extracted them from a 
tarball onto the /sites/sites (mount point for the tests) replicated 
cluster.  It took 17m28.972s to extract all files.  By way of comparison 
it takes 0m5.102s when extracting just to the ext3 partition.  Here are 
some unlinking times (for the 22016 files) - gluster mount: 0m28.428s 
ext3: 0m0.456s.  And here are some read times: gluster mount: 
0m19.871s.  You can note from my config that these times are with 
"option flush-behind on" for write-behind.  During the write test, I 
monitored NIC stats on the receiving server to see how much the link was 
utilized - its peak was 4.80Mb - so the NIC was not the bottleneck 
either.  I just cannot find the hold up, the network, disks, and cpu are 
not loaded during the write test.

So the biggest issue seems to be AFR write performance.  Is this normal 
or is there something specific to my setup causing these problems?  
Obviously I am new to glusterfs so I do not know what to expect, but I 
think I must be doing something wrong.

Any help/advice/direction is greatly appreciated.  I have googled and 
googled and found no advice that has yielded real results.  Sorry if I 
missed something obvious that was documented.

Michael


Volume files (same on each server) were first created using the 
/usr/bin/glusterfs-volgen --raid 1 --cache-size 512MB --export-directory 
/sites_gfs --name sites1 172.16.0.1 172.16.0.2

/** Server - adapted from generated to add one other directory **/

volume posix_sites
  type storage/posix
  option directory /sites_gfs
end-volume

volume posix_phplib
  type storage/posix
  option directory /usr/local/lib/php_gfs
end-volume

volume locks_sites
    type features/locks
    subvolumes posix_sites
end-volume

volume locks_phplib
    type features/locks
    subvolumes posix_phplib
end-volume

volume brick_sites
    type performance/io-threads
    option thread-count 8
    subvolumes locks_sites
end-volume

volume brick_phplib
    type performance/io-threads
    option thread-count 8
    subvolumes locks_phplib
end-volume

volume server
    type protocol/server
    option transport-type tcp
    option auth.addr.brick_sites.allow *
    option auth.addr.brick_phplib.allow *
    option listen-port 6996
    subvolumes brick_sites brick_phplib
end-volume

/** Client - adapted from generated to try and fix write issues - to no 
avail**/

volume 172.16.0.1
    type protocol/client
    option transport-type tcp
    option remote-host 172.16.0.1
    option remote-port 6996
    option remote-subvolume brick_sites
end-volume

volume 172.16.0.2
    type protocol/client
    option transport-type tcp
    option remote-host 172.16.0.2
    option remote-port 6996
    option remote-subvolume brick_sites
end-volume

volume mirror-0
    type cluster/replicate
    subvolumes 172.16.0.1 172.16.0.2
end-volume

volume writebehind
    type performance/write-behind
    option cache-size 1MB
    option flush-behind on
    subvolumes mirror-0
end-volume

volume io-cache
    type performance/io-cache
    option cache-size 64MB
    subvolumes writebehind
end-volume