[Gluster-users] some nodes mount unwritable /subvolumes/ of glusterfs (3.3b1)

[Harry Mangalam]

Just set up and mounted a glusterfs consisting of 6 nodes:

$ gluster volume info 

Volume Name: g6
Type: Distribute
Status: Started
Number of Bricks: 6
Transport-type: tcp
Bricks:
Brick1: pbs1:/data2
Brick2: pbs2:/data2
Brick3: pbs3:/data2
Brick4: pbs3:/data
Brick5: dabrick:/data2
Brick6: hef:/data2
Options Reconfigured:
auth.allow: 128.*


It's mounted on 9 similar (tho not identical) ubuntu 10.04.3 nodes 
(64b Kubuntu Dektop) in a cluster (claw[1-9]), which are running a 
self-compiled and installed version of 3.3b1. The gluster server 
bricks are also running 10.04.3 (64b server).

However, the glusterfs mounts differentially on the nodes:

root at bduc-login:~
576 $ for ITER in  $(seq 1 9); do    echo -n "claw${ITER}:";   ssh 
claw${ITER} 'df -h |egrep  "gluz|pbs"' ; done; 

claw1:pbs3:/g6               21T  114G   21T   1% /mnt/gluz
claw2:pbs3.calit2.uci.edu:/g6
                       21T  114G   21T   1% /mnt/gluz
claw3:pbs3.calit2.uci.edu:/g6
                       21T  114G   21T   1% /mnt/gluz
claw4:pbs3.calit2.uci.edu:/g6
                       21T  114G   21T   1% /mnt/gluz
claw5:pbs3.calit2.uci.edu:/g6
                      5.5T  2.7G  5.5T   1% /mnt/gluz <---
claw6:pbs3.calit2.uci.edu:/g6
                       21T  112G   21T   1% /mnt/gluz
claw7:pbs3.calit2.uci.edu:/g6
                       21T  114G   21T   1% /mnt/gluz
claw8:pbs3.calit2.uci.edu:/g6
                      5.5T  2.7G  5.5T   1% /mnt/gluz <---
claw9:pbs3.calit2.uci.edu:/g6
                       21T  112G   21T   1% /mnt/gluz

Obviously claw5 and claw8 are ... different.  It looks like they're 
mounting a component of the glusterfs (from the numbers, it's the 
contribution of "Brick5: dabrick:/data2"), but not the whole thing. 
The glusterfs was mounted on all of them with a script:

mount -t glusterfs pbs3.calit2.uci.edu:/g6 /mnt/gluz

Ive unmounted and remounted the glusterfs on claw8 with no difference.
I've also unmounted all instances of the glusterfs, then gluster-
stopped and restarted the volume on the gluster server node and then 
remounted the fs as well.  No diff.

When mounted, claw[58] can see some of the files not all of them (the 
same ones that are on the 'dabrick' volume, as you might guess) but 
cannot write to the mounted filesystem, even tho the user has 
permissions to write.

...?!

The nodes that are having the problem have never had a gluster fs 
mounted on them before.
 
-- 
Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
[ZOT 2225] / 92697  Google Voice Multiplexer: (949) 478-4487 
MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
--
This signature has been OCCUPIED!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20111028/d589fa81/attachment.html>