[Gluster-users] 3.0.5 client crash - afr_set_split_brain

[Jon Swanson]

Seeing a glusterfs client die oddly.

--Setup--
Client: 
Fedora 12 2.6.32.16-141.fc12.x86_64 
# rpm -qa |egrep 'fuse|glust'
fuse-2.8.4-1.fc12.x86_64
glusterfs-client-3.0.5-1.fc11.x86_64
fuse-libs-2.8.4-1.fc12.x86_64
glusterfs-common-3.0.5-1.fc11.x86_64


Servers - 6 nodes with a 3 x distribute:
Fedora 12 2.6.32.9-70.fc12.x86_64
[root at pdbsearch11.pm.prod ~]# rpm -qa | grep glust
glusterfs-common-3.0.5-1.fc11.x86_64
glusterfs-server-3.0.5-1.fc11.x86_64


Process:
1. Client copies a large amount of files to the gluster mount
2. Client tries to do a recursive list of all files copied (ls -R)
3. Recursive list comes across a file where the checksum does not match for some reason (see following log snipped)
4. Client dies horribly, the mount point will becoming invalid with the following error:
gluster-mount/file: Transport endpoint is not connected

I've tried to keep the snippets below as brief as possible.  If you think the volume definition files would help, let me know and i'll be happy to post those here as well.

Any help or suggestions are most welcome. 

Thanks!

---

This is the corresponding snipped from 'tail -f gluster-mount.log':

> [2010-07-21 16:34:48] N [client-protocol.c:6288:client_setvolume_cbk] pdbindex2-1: Connected to 192.168.201.88:6996, attached to remote volume 'brick'.

> [2010-07-21 16:35:33] E [afr.c:107:afr_set_split_brain] mirror-0: invalid argument: inode
> [2010-07-21 16:35:33] E [afr-self-heal-algorithm.c:768:sh_diff_checksum_cbk] mirror-0: checksum on /index.201007211105.deploy/file failed on subvolume indexcopy-0 (File descriptor in bad state)
> [2010-07-21 16:35:33] E [afr-self-heal-algorithm.c:768:sh_diff_checksum_cbk] mirror-0: checksum on /index.201007211105.deploy/file failed on subvolume indexcopy-1 (File descriptor in bad state)
> pending frames:
> frame : type(1) op(LOOKUP)
> frame : type(1) op(LOOKUP)
> frame : type(1) op(LOOKUP)
> 
> patchset: v3.0.5
> signal received: 11
> time of crash: 2010-07-21 16:35:33
> configuration details:
> argp 1
> backtrace 1
> dlfcn 1
> fdatasync 1
> libpthread 1
> llistxattr 1
> setfsid 1
> spinlock 1
> epoll.h 1
> xattr.h 1
> st_atim.tv_nsec 1
> package-string: glusterfs 3.0.5
> /lib64/libc.so.6(+0x32740)[0x7fa9c949b740]
> /usr/lib64/glusterfs/3.0.5/xlator/cluster/replicate.so(+0x4b2ea)[0x7fa9c85ff2ea]
> /usr/lib64/glusterfs/3.0.5/xlator/cluster/replicate.so(+0x4b557)[0x7fa9c85ff557]
> /usr/lib64/glusterfs/3.0.5/xlator/cluster/replicate.so(+0x4be10)[0x7fa9c85ffe10]
> /usr/lib64/glusterfs/3.0.5/xlator/cluster/replicate.so(afr_sh_algo_diff+0x196)[0x7fa9c85fffc2]
> /usr/lib64/glusterfs/3.0.5/xlator/cluster/replicate.so(afr_sh_data_sync_prepare+0x256)[0x7fa9c85e9a91]
> /usr/lib64/glusterfs/3.0.5/xlator/cluster/replicate.so(afr_sh_data_fix+0x5db)[0x7fa9c85ea078]
> /usr/lib64/glusterfs/3.0.5/xlator/cluster/replicate.so(afr_sh_data_fstat_cbk+0x167)[0x7fa9c85ea34e]
> /usr/lib64/glusterfs/3.0.5/xlator/cluster/distribute.so(dht_attr_cbk+0x238)[0x7fa9c8820e08]
> /usr/lib64/glusterfs/3.0.5/xlator/protocol/client.so(client_fstat_cbk+0x178)[0x7fa9c8a59868]
> /usr/lib64/glusterfs/3.0.5/xlator/protocol/client.so(protocol_client_interpret+0x1df)[0x7fa9c8a60274]
> /usr/lib64/glusterfs/3.0.5/xlator/protocol/client.so(protocol_client_pollin+0xc6)[0x7fa9c8a60ff5]
> /usr/lib64/glusterfs/3.0.5/xlator/protocol/client.so(notify+0x158)[0x7fa9c8a6154d]
> /usr/lib64/libglusterfs.so.0(xlator_notify+0xd8)[0x7fa9c9c1b639]
> /usr/lib64/glusterfs/3.0.5/transport/socket.so(socket_event_poll_in+0x46)[0x7fa9c6f59249]
> /usr/lib64/glusterfs/3.0.5/transport/socket.so(socket_event_handler+0xc4)[0x7fa9c6f5957c]
> /usr/lib64/libglusterfs.so.0(+0x3eefc)[0x7fa9c9c40efc]
> /usr/lib64/libglusterfs.so.0(+0x3f0ee)[0x7fa9c9c410ee]
> /usr/lib64/libglusterfs.so.0(event_dispatch+0x74)[0x7fa9c9c4140d]
> /usr/sbin/glusterfs(main+0xf53)[0x406187]
> /lib64/libc.so.6(__libc_start_main+0xfd)[0x7fa9c9487b1d]
> /usr/sbin/glusterfs[0x402679]
> ---------

If we look at the respective files, their checksums are fine:
> [16:40] ~> for i in `seq 10 15`; do echo -n "search$i: "; ssh search$i md5sum /data/export/index.201007211105.deploy/file; done
> search10: md5sum: /data/export/index.201007211105.deploy/file: No such file or directory
> search11: 8605b1467bece54ed7ccd13e086ee299  /data/export/index.201007211105.deploy/file
> search12: md5sum: /data/export/index.201007211105.deploy/file: No such file or directory
> search13: md5sum: /data/export/index.201007211105.deploy/file: No such file or directory
> search14: 8605b1467bece54ed7ccd13e086ee299  /data/export/index.201007211105.deploy/file
> search15: md5sum: /data/export/index.201007211105.deploy/file: No such file or directory

If we look at extended attributes however, we notice that 'trusted.posix.gen' is different:
> for i in `seq 10 15`; do echo -n "search$i: "; ssh pdbsearch$i getfattr -d -m - /data/export/index.201007211105.deploy/file; done
> search10: getfattr: /data/export/index.201007211105.deploy/file: No such file or directory
> search11: getfattr: Removing leading '/' from absolute path names
> # file: data/export/index.201007211105.deploy/file
> security.selinux="unconfined_u:object_r:default_t:s0
> trusted.afr.indexcopy-0=0sAAAAAQAAAAAAAAAA
> trusted.afr.indexcopy-1=0sAAAAAQAAAAAAAAAA
> trusted.posix.gen=0sTEFukQAAAEY=
> 
> search12: getfattr: /data/export/index.201007211105.deploy/file: No such file or directory
> search13: getfattr: /data/export/index.201007211105.deploy/file: No such file or directory
> search14: getfattr: Removing leading '/' from absolute path names
> # file: data/export/index.201007211105.deploy/file
> security.selinux="unconfined_u:object_r:default_t:s0
> trusted.afr.indexcopy-0=0sAAAAAQAAAAAAAAAA
> trusted.afr.indexcopy-1=0sAAAAAQAAAAAAAAAA
> trusted.posix.gen=0sTEaPaAAAAAI=
> 
> search15: getfattr: /data/export/index.201007211105.deploy/file: No such file or directory