[Gluster-users] speeding up getdents()/stat() and/or general perf tuning...

[mki-glusterfs at mozone.net]

Hi

I've been noticing some long delays on doing a simple `ls' in 
directories that haven't been recently accessed on a test glusterfs
system we've put together.  The system(s) consists of a 4 node 
DHT + AFR (x1) setup, running 2.0.6 all with 10GbE connectivity 
between the nodes (and no there is no network bottleneck here as
iperf proves that the throughput between the machines is ~9Gbps).

stat("usr/bin", {st_mode=S_IFDIR|0755, st_size=28672, ...}) = 0 <1.659084>
open("usr/bin", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3 <0.001231>
fstat(3, {st_mode=S_IFDIR|0755, st_size=28672, ...}) = 0 <0.000007>
fcntl(3, F_GETFD)                       = 0x1 (flags FD_CLOEXEC) <0.000005>
getdents(3, /* 75 entries */, 4096)     = 2256 <0.033347>
getdents(3, /* 75 entries */, 4096)     = 2280 <0.001858>
getdents(3, /* 41 entries */, 4096)     = 1256 <0.000961>
getdents(3, /* 73 entries */, 4096)     = 2216 <0.034120>
getdents(3, /* 73 entries */, 4096)     = 2296 <0.001393>
getdents(3, /* 11 entries */, 4096)     = 344 <0.000481>
getdents(3, /* 72 entries */, 4096)     = 2264 <0.028436>
mmap(NULL, 147456, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f061c9b5000 <0.000005>
getdents(3, /* 74 entries */, 4096)     = 2304 <0.001711>
getdents(3, /* 42 entries */, 4096)     = 1344 <0.001304>
getdents(3, /* 74 entries */, 4096)     = 2200 <3.176572> <- SLOW OPERATION
getdents(3, /* 74 entries */, 4096)     = 2288 <0.012123>
getdents(3, /* 10 entries */, 4096)     = 280 <0.000376>
getdents(3, /* 0 entries */, 4096)      = 0 <0.000243>
close(3)                                = 0 <0.000011>

The client configs are simple (stripped down for simplicity sakes):

brick 1 tcp 10.10.10.11
brick 2 tcp 10.10.10.12
...
brick 8 tcp 10.10.10.18
replicate1 brick1 brick2
replicate2 brick3 brick5
replicate3 brick4 brick7
replicate4 brick6 brick8
distribute replicate1 replicate2 replicate3 replicate4
io-threads thread-count 16
readahead page-count 16
io-cache cache-size 512MB
write-behind cache-size 1MB

The storage nodes are even simpler yet (trimmed):

posix background-unlink yes
posix-locks
io-threads thread-count 16
transport-type tcp

The client nodes are running an unpatched fuse 2.7.4 userland (debian 
lenny) and the default kernel 2.6.31.1 fuse module and I'm curious if 
the delay in the getdents/stat calls are because of fuse or something 
else config wise that I've managed to miss?

Anyone have any recommendations or thoughts as to how to improve the 
performance?

Thanks.

Mohan