[Gluster-users] Slow Replication over infiniband

[joel vennin]

Hi, I've a setup composed with 3 nodes(zodiac1, zodiac2, zodiac3)

Node config: 8 cores W5580  at 3.20GHz, 76 Go RAM, 40 To, infiniband mlx4_0
(20 Gb)
Gluster Version: 2.0.7
Kernel Linux 2.6.30 / Fuse 2.7.4


Config file:

#----------------------------------------------------------
# SERVER SIDE
volume posix-1
       type storage/posix
       option directory /data/yacht-data2
end-volume

volume locks-1
  type features/locks
  subvolumes posix-1
end-volume


volume brick-1
       type performance/io-threads
       option thread-count 4
       subvolumes locks-1
end-volume

volume server
       type protocol/server
       option transport-type                     ib-verbs
       option transport.ib-verbs.device-name     mlx4_0
       option auth.addr.brick-1.allow *
       subvolumes brick-1 #brick-2
end-volume


# CLIENT NODE DECLARATION

#
# ZODIAC 1 CLIENT
volume zbrick1
       type protocol/client
       option transport-type ib-verbs
       option remote-host 192.168.3.200
       option remote-subvolume brick-1
end-volume

# ZODIAC 2
volume zbrick2
       type protocol/client
       option transport-type   ib-verbs
       option remote-host      192.168.3.201
       option remote-subvolume brick-1
end-volume

# ZODIAC 3
volume zbrick3
       type protocol/client
       option transport-type   ib-verbs
       option remote-host      192.168.3.202
       option remote-subvolume brick-1
end-volume

# MEGA REPLICATE
volume mega-replicator
       type cluster/replicate
       subvolumes zbrick1 zbrick2 zbrick3
end-volume
#----------------------------------------------------------

Command on each server to start glusterfs:
   glusterfs -f myreplica.vol /mnt/data/



The scenario that I use:
    zodiac1 and zodiac2 are correctly synchronized. Once every thing is ok
for zodiac1 and zodiac2, I decide to start the zodiac3. I order to
synchronize it, I do on the
box ls -aLR /mnt/data. So the replication start but the transfer rate is
really slow: around 30 MB/s !

Have you an idea how can I increase this rate ?


Thank you.