DownloadGlusterFS 3.0.4 cluster/replicate vs NFSv3 Benchmark
From GlusterDocumentation
This information is out of date
and does not contain information related to the current version of Gluster
Documentation Home
Benchmark specs
Following Benchmark_Submission_Guidelines...
Benchmark performed by : Antoine Beaupré <anarcat@koumbit.org> Benchmark performed on : may 2010 HARDWARE CONFIG, server "rhea" System Model : Tyan motherboard, otherwise generic PC Processor core speed : Athlon MP 2200+ Number of cores : 2 RAM size : 2GB RAM type : Kingston ECC RAID/Storage controller : AMD-768 [Opus] IDE controler Disk model/size : Seagate ST3500630A (sda), Western Digital WD5000AAKB-00H8A0 (sdb), mounted as a software RAID-1 (mdadm) Storage speed (hdparm -t) : sda: 136 MB in 3.07 seconds = 44.26 MB/sec, sdb: Timing buffered disk reads: 174 MB in 3.01 seconds = 57.90 MB/sec SOFTWARE CONFIG OS Distribution : Debian lenny Kernel version : 2.6.30-bpo.2-686 GlusterFS version : 3.0.4 Other FS Name / Version : ext3 Benchmark Application / Tool : iozone3 287-2, dd
The client and servers have gigabit connexions on a HP procurve gigabit switch.
Gluster setup
Gluster is setup in cluster/replicate mode. The above describes the least powerful machine of the cluster. Although the second machine has more powerful CPU (dual core E4540 2.5Ghz) and memory (4GB ram), the storage is an internal IDE drive. hdparm still yields 78MB/s so I will not consider this box to be the bottleneck for the tests.
Both servers are running with the following config:
volume posix1
type storage/posix
option directory /srv/home
end-volume
volume locks1
type features/locks
subvolumes posix1
end-volume
volume brick1
type performance/io-threads
option thread-count 8
subvolumes locks1
end-volume
volume server-tcp
type protocol/server
option transport-type tcp
option auth.addr.brick1.allow *
option transport.socket.listen-port 6996
option transport.socket.nodelay on
subvolumes brick1
end-volume
The client have the following config:
volume rhea-1
type protocol/client
option transport-type tcp
option remote-host rhea
option transport.socket.nodelay on
option transport.remote-port 6996
option remote-subvolume brick1
end-volume
volume desktop004-1
type protocol/client
option transport-type tcp
option remote-host desktop004
option transport.socket.nodelay on
option transport.remote-port 6996
option remote-subvolume brick1
end-volume
volume mirror-0
type cluster/replicate
subvolumes rhea-1 desktop004-1
end-volume
volume writebehind
type performance/write-behind
option cache-size 4MB
subvolumes mirror-0
end-volume
volume readahead
type performance/read-ahead
option page-count 4
subvolumes writebehind
end-volume
volume iocache
type performance/io-cache
option cache-size `grep 'MemTotal' /proc/meminfo | awk '{print $2 * 0.2 / 1024}' | cut -f1 -d.`MB
option cache-timeout 1
subvolumes readahead
end-volume
volume quickread
type performance/quick-read
option cache-timeout 1
option max-file-size 64kB
subvolumes iocache
end-volume
volume statprefetch
type performance/stat-prefetch
subvolumes quickread
end-volume
(rhea is the generic PC spec'd above, desktop002 is the more powerful machine.)
Testing methodology
We are interested in latency of read/writes here, so we'll use iozone for a compared benchmark with NFS. The magic incantation is:
iozone -Q -c -g 2G -a -R -b output.wks > output.txt
The main server (rhea) is running both glusterd and NFS. The other server is only running glusterd. The client (desktop004) has tested successively gluster and NFS.
Anomalies
The NFS benchmark failed near the end, which says a lot about the reliability of both NFS and gluster... The output was:
1048576 128 8954 8798 11285 12247 51158 8355 53767 183645 34390 8608 8745 11256 12305
1048576 256 8830 8770 12512 12252
Error writing block at 472645632
iozone: interrupted
The above was trying to write a 1GB file with a 50M record size.
We can also note that NFS is not redundant while gluster offers that reliability.
Results
Throughput
Reads are surprisingly slow in gluster, around 10 times slower in terms of absolute throughput. We get around 10MB/s for big files and 35 to 55MB/s for small (64 to 256 bytes) files, while NFS maxes out the disks, at a whopping 250MB/s.
Random reads are a bit better, comparable with NFS (2-3 times slower).
Writes are comparable, around 10MB/s on gluster, 12MB/s on NFS. Throughput is generally more stable according to the file size in gluster.
Gluster is also surprisingly bad at reading files backwards.
Re-writing files in gluster is also much slower (2-10MB/s vs 10-90MB/s for NFS).
So, in general, high volume of read()s are 2 to 10 tmies slower in gluster. Maybe that can be explained by the userland bridge which is optimized in the kernel for NFS, while gluster needs to go through fuse. Writes, however, are comparable, which is a very good (and surprising, given the distributed nature) thing.
Latency
In terms of latency, gluster fares much better, comparatively. It's still a tad slower, but at least it's comparable at all levels. The latency is also more consistent than NFS.
Conclusion
For our requirements, Gluster will fit the bill, as the focus is filesystem latency, and there are no critical failures there, while the added benefit (redundancy) is huge.
However, it seems there is either a configuration problem or a problem during the benchmark because according to the documentation, reads are supposed to be distributed amongst the nodes, which should make reads *faster*, not slower, than NFS. Unfortunately, this is not the case, which may be a problem for some users looking for high bandwidth reads.
This information is out of date
and does not contain information related to the current version of Gluster
Documentation Home
