[Gluster-users] Slow performance - 4 hosts, 10 gigabit ethernet, Gluster 3.2.3
thomas at thomax.com.au
Wed Sep 14 22:50:46 UTC 2011
Thanks for the reply - my comments inline below
From: Pavan T C [mailto:tcp at gluster.com]
Sent: Wednesday, 14 September 2011 9:19 PM
To: Thomas Jackson
Cc: gluster-users at gluster.org
Subject: Re: [Gluster-users] Slow performance - 4 hosts, 10 gigabit
ethernet, Gluster 3.2.3
> On Friday 09 September 2011 10:30 AM, Thomas Jackson wrote:
>> Hi everyone,
> Hello Thomas,
> Try the following:
> 1. In the fuse volume file, try:
> Under write-behind:
> "option cache-size 16MB"
> Under read-ahead:
> "option page-count 16"
> Under io-cache:
> "option cache-size=64MB"
TJ: Results here are not pretty!
root at my-host:~# dd if=/dev/zero of=/mnt/cluster-volume/test.file
10485760000 bytes (10 GB) copied, 107.888 s, 97.2 MB/s
> 2. Did you get 9Gbits/Sec with iperf with a single thread or multiple
TJ: Single thread
> 3. Can you give me the output of:
> sysctl -a | egrep 'rmem|wmem'
TJ: root at my-host:~# sysctl -a | egrep 'rmem|wmem'
error: permission denied on key 'vm.compact_memory'
vm.lowmem_reserve_ratio = 256 256 32
error: permission denied on key 'net.ipv4.route.flush'
net.core.wmem_max = 131071
net.core.rmem_max = 131071
net.core.wmem_default = 126976
net.core.rmem_default = 126976
net.ipv4.tcp_wmem = 4096 16384 4194304
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_wmem_min = 4096
error: permission denied on key 'net.ipv6.route.flush'
> 4. If it is not a problem for you, can you please create a pure distribute
setup (instead of distributed-replicate) and then report the numbers?
TJ: I've been able to do this with 2 hosts, while I was at it I also tested
a pure replica and pure stripe setup for comparison
Distribute = 313 MB/sec
Replica = 166 MB/sec
Stripe = 529 MB/sec
> 5. What is the inode size with which you formatted you XFS filesystem ?
> This last point might not be related to your throughput problem, but if
you are planning to use this setup for a large number of files,
> you might be better off using an inode size of 512 instead of the default
256 bytes. To do that, your mkfs command should be:
> mkfs -t xfs -i size=512 /dev/<disk device>
TJ: This is destined for use with VM images, probably a maximum of 200 files
total. That said, I have tried with a bigger inode size and also with ext4
with very similar results each time
In a totally bizarre turn of events, turning on port bonding (each host has
2x 10gig storage ports) in ACTIVE/BACKUP mode has increased the speed a fair
dd if=/dev/zero of=/mnt/cluster-volume/test.file bs=1M count=40000
41943040000 bytes (42 GB) copied, 176.459 s, 238 MB/s
I have noticed that the inodes are getting very frequently locked/unlocked
by the afs from some brief debugging, not sure if that is related
>> I am seeing slower-than-expected performance in Gluster 3.2.3 between
>> 4 hosts with 10 gigabit eth between them all. Each host has 4x 300GB
>> SAS 15K drives in RAID10, 6-core Xeon E5645 @ 2.40GHz and 24GB RAM
>> running Ubuntu
>> 10.04 64-bit (I have also tested with Scientific Linux 6.1 and Debian
>> Squeeze - same results on those as well). All of the hosts mount the
>> volume using the FUSE module. The base filesystem on all of the nodes
>> is XFS, however tests with ext4 have yielded similar results.
>> Command used to create the volume:
>> gluster volume create cluster-volume replica 2 transport tcp
>> node01:/mnt/local-store/ node02:/mnt/local-store/
>> node03:/mnt/local-store/ node04:/mnt/local-store/
>> Command used to mount the Gluster volume on each node:
>> mount -t glusterfs localhost:/cluster-volume /mnt/cluster-volume
>> Creating a 40GB file onto a node's local storage (ie no Gluster
>> dd if=/dev/zero of=/mnt/local-store/test.file bs=1M count=40000
>> 41943040000 bytes (42 GB) copied, 92.9264 s, 451 MB/s
>> Getting the same file off the node's local storage:
>> dd if=/mnt/local-store/test.file of=/dev/null
>> 41943040000 bytes (42 GB) copied, 81.858 s, 512 MB/s
>> 40GB file onto the Gluster storage:
>> dd if=/dev/zero of=/mnt/cluster-volume/test.file bs=1M count=40000
>> 41943040000 bytes (42 GB) copied, 226.934 s, 185 MB/s
>> Getting the same file off the Gluster storage
>> dd if=/mnt/cluster-volume/test.file of=/dev/null
>> 41943040000 bytes (42 GB) copied, 661.561 s, 63.4 MB/s
>> I have also tried using Gluster 3.1, with similar results.
>> According to the Gluster docs, I should be seeing roughly the lesser
>> of the drive speed and the network speed. The network is able to push
>> 0.9GB/sec according to iperf so that definitely isn't a limiting
>> factor here, and each array is able to do 400-500MB/sec as per above
>> benchmarks. I've tried with/without jumbo frames as well, which doesn't
make any major difference.
>> The glusterfs process is using 120% CPU according to top, and
>> glusterfsd is sitting at about 90%.
>> Any ideas / tips of where to start for speeding this config up?
> Gluster-users mailing list
> Gluster-users at gluster.org
Message protected by MailGuard: e-mail anti-virus, anti-spam and content
Click here to report this message as spam:
More information about the Gluster-users