[Gluster-users] Issue with files on glusterfs becoming unreadable.
Elbert Lai
elbert.lai at eng.admob.com
Thu Jun 11 18:03:47 UTC 2009
Also, in the logfiles on the clients, it looks like I get these types
of messages whenever I try to access a file that is no longer
accessible.
2009-06-11 07:58:24 E [fuse-bridge.c:675:fuse_fd_cbk] glusterfs-fuse:
22068570: /hourlogs/myDir0/1243432800.log => -1 (5)
2009-06-11 07:58:24 E [fuse-bridge.c:436:fuse_entry_cbk] glusterfs-
fuse: 22068579: /hourlogs/myDir1/1243400400.log => -1 (116)
2009-06-11 07:58:24 E [unify.c:850:unify_open] unify: /hourlogs/
myDir1/1243400400.log: entry_count is 3
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/
myDir1/1243400400.log: found on afr3
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/
myDir1/1243400400.log: found on afr2
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/
myDir1/1243400400.log: found on afr-ns
2009-06-11 07:58:24 E [fuse-bridge.c:675:fuse_fd_cbk] glusterfs-fuse:
22068580: /hourlogs/myDir1/1243400400.log => -1 (5)
2009-06-11 07:58:24 E [fuse-bridge.c:436:fuse_entry_cbk] glusterfs-
fuse: 22068583: /hourlogs/myDir2/1243411200.log => -1 (116)
2009-06-11 07:58:24 E [unify.c:850:unify_open] unify: /hourlogs/
myDir2/1243411200.log: entry_count is 3
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/
myDir2/1243411200.log: found on afr1
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/
myDir2/1243411200.log: found on afr3
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/
myDir2/1243411200.log: found on afr-ns
2009-06-11 07:58:24 E [fuse-bridge.c:675:fuse_fd_cbk] glusterfs-fuse:
22068584: /hourlogs/myDir2/1243411200.log => -1 (5)
2009-06-11 07:58:24 E [fuse-bridge.c:436:fuse_entry_cbk] glusterfs-
fuse: 22068599: /hourlogs/myDir3/1243472400.log => -1 (116)
2009-06-11 07:58:24 E [unify.c:850:unify_open] unify: /hourlogs/
myDir3/1243472400.log: entry_count is 3
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/
myDir3/1243472400.log: found on afr1
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/
myDir3/1243472400.log: found on afr3
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/
myDir3/1243472400.log: found on afr-ns
2009-06-11 07:58:24 E [fuse-bridge.c:675:fuse_fd_cbk] glusterfs-fuse:
22068600: /hourlogs/myDir3/1243472400.log => -1 (5)
2009-06-11 07:58:24 E [fuse-bridge.c:436:fuse_entry_cbk] glusterfs-
fuse: 22068603: /hourlogs/myDir4/1243404000.log => -1 (116)
2009-06-11 07:58:24 E [unify.c:850:unify_open] unify: /hourlogs/
myDir4/1243404000.log: entry_count is 3
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/
myDir4/1243404000.log: found on afr1
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/
myDir4/1243404000.log: found on afr-ns
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/
myDir4/1243404000.log: found on afr3
2009-06-11 07:58:24 E [fuse-bridge.c:675:fuse_fd_cbk] glusterfs-fuse:
22068604: /hourlogs/myDir5/1243404000.log => -1 (5)
2009-06-11 07:58:24 E [fuse-bridge.c:436:fuse_entry_cbk] glusterfs-
fuse: 22068619: /hourlogs/myDir5/1243447200.log => -1 (116)
2009-06-11 07:58:24 E [unify.c:850:unify_open] unify: /hourlogs/
myDir5/1243447200.log: entry_count is 4
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/
myDir5/1243447200.log: found on afr1
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/
myDir5/1243447200.log: found on afr3
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/
myDir5/1243447200.log: found on afr2
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/
myDir5/1243447200.log: found on afr-ns
2009-06-11 07:58:24 E [fuse-bridge.c:675:fuse_fd_cbk] glusterfs-fuse:
22068620: /hourlogs/myDir5/1243447200.log => -1 (5)
On Jun 11, 2009, at 10:33 AM, Elbert Lai wrote:
> elbert at host1:~$ dpkg -l|grep glusterfs
> ii glusterfs-client
> 1.3.8-0pre2 GlusterFS fuse client
> ii glusterfs-server
> 1.3.8-0pre2 GlusterFS fuse server
> ii libglusterfs0
> 1.3.8-0pre2 GlusterFS libraries and
> translator modules
>
> I have 2 hosts set up to use AFR with the package versions listed
> above. I have been experiencing an issue where a file that is copied
> to glusterfs is readable/writable for a while, then at some point it
> time, it ceases to be. Trying to access it only retrieves the error
> message, "cannot open `filename' for reading: Input/output error".
>
> Files enter glusterfs either via the "cp" command from a client or
> via "rsync". In the case of cp, the clients are all local and
> copying across a very fast connection. In the case of rsync, the 1
> client is itself a gluster client. We are testing out a later
> version of gluster, and it rsync's across a vpn.
>
> elbert at host2:~$ dpkg -l|grep glusterfs
> ii glusterfs-client 2.0.1-1 clustered file-
> system
> ii glusterfs-server 2.0.1-1 clustered file-
> system
> ii libglusterfs0 2.0.1-1 GlusterFS
> libraries and translator modules
> ii libglusterfsclient0 2.0.1-1 GlusterFS client
> library
>
> =========
> What causes files to become inaccessible? I read that fstat() had a
> bug in version 1.3.x whereas stat() did not, and that it was being
> worked on. Could this be related?
>
> When a file becomes inaccessible, I have been manually removing the
> file from the mount point, then copying it back in via scp. Then the
> file becomes accessible. Below I've pasted a sample of what I'm
> seeing.
>
>> elbert at tool3.:hourlogs$ cd myDir
>> ls 1244682000.log
>> elbert at tool3.:myDir$ ls 1244682000.log
>> 1244682000.log
>> elbert at tool3.:myDir$ stat 1244682000.log
>> File: `1244682000.log'
>> Size: 40265114 Blocks: 78744 IO Block: 4096 regular file
>> Device: 15h/21d Inode: 42205749 Links: 1
>> Access: (0755/-rwxr-xr-x) Uid: ( 1003/ elbert) Gid:
>> ( 6000/ ops)
>> Access: 2009-06-11 02:25:10.000000000 +0000
>> Modify: 2009-06-11 02:26:02.000000000 +0000
>> Change: 2009-06-11 02:26:02.000000000 +0000
>> elbert at tool3.:myDir$ tail 1244682000.log
>> tail: cannot open `1244682000.log' for reading: Input/output error
>
> At this point, I am able to rm the file. Then, if I scp it back in,
> I am able to successfully tail it.
>
> So,
>
> I have observed cases where the files had a Size of 0, and otherwise
> they were in the same state. I'm not totally certain, but it looks
> like if a file gets into this state from rsync, either it gets
> deposited in this state immediately (before I try to read it), or
> else it quickly enters this state. Speaking generally, file sizes
> tend to be several MB up to 150 MB.
>
> Here's my server config:
> # Gluster Server configuration /etc/glusterfs/glusterfs-server.vol
> # Configured for AFR & Unify features
>
> volume brick
> type storage/posix
> option directory /var/gluster/data/
> end-volume
>
> volume brick-ns
> type storage/posix
> option directory /var/gluster/ns/
> end-volume
>
> volume server
> type protocol/server
> option transport-type tcp/server
> subvolumes brick brick-ns
> option auth.ip.brick.allow 165.193.245.*,10.11.*
> option auth.ip.brick-ns.allow 165.193.245.*,10.11.*
> end-volume
>
> Here's my client config:
> # Gluster Client configuration /etc/glusterfs/glusterfs-client.vol
> # Configured for AFR & Unify features
>
> volume brick1
> type protocol/client
> option transport-type tcp/client # for TCP/IP transport
> option remote-host 10.11.16.68 # IP address of the remote brick
> option remote-subvolume brick # name of the remote volume
> end-volume
>
> volume brick2
> type protocol/client
> option transport-type tcp/client
> option remote-host 10.11.16.71
> option remote-subvolume brick
> end-volume
>
> volume brick3
> type protocol/client
> option transport-type tcp/client
> option remote-host 10.11.16.69
> option remote-subvolume brick
> end-volume
>
> volume brick4
> type protocol/client
> option transport-type tcp/client
> option remote-host 10.11.16.70
> option remote-subvolume brick
> end-volume
>
> volume brick5
> type protocol/client
> option transport-type tcp/client
> option remote-host 10.11.16.119
> option remote-subvolume brick
> end-volume
>
> volume brick6
> type protocol/client
> option transport-type tcp/client
> option remote-host 10.11.16.120
> option remote-subvolume brick
> end-volume
>
> volume brick-ns1
> type protocol/client
> option transport-type tcp/client
> option remote-host 10.11.16.68
> option remote-subvolume brick-ns # Note the different remote
> volume name.
> end-volume
>
> volume brick-ns2
> type protocol/client
> option transport-type tcp/client
> option remote-host 10.11.16.71
> option remote-subvolume brick-ns # Note the different remote
> volume name.
> end-volume
>
> volume afr1
> type cluster/afr
> subvolumes brick1 brick2
> end-volume
>
> volume afr2
> type cluster/afr
> subvolumes brick3 brick4
> end-volume
>
> volume afr3
> type cluster/afr
> subvolumes brick5 brick6
> end-volume
>
> volume afr-ns
> type cluster/afr
> subvolumes brick-ns1 brick-ns2
> end-volume
>
> volume unify
> type cluster/unify
> subvolumes afr1 afr2 afr3
> option namespace afr-ns
>
> # use the ALU scheduler
> option scheduler alu
>
> # This option makes brick5 to be readonly, where no new files are
> created.
> ##option alu.read-only-subvolumes brick5##
>
> # Don't create files one a volume with less than 5% free diskspace
> option alu.limits.min-free-disk 10%
>
> # Don't create files on a volume with more than 10000 files open
> option alu.limits.max-open-files 10000
>
> # When deciding where to place a file, first look at the disk-
> usage, then at
> # read-usage, write-usage, open files, and finally the disk-speed-
> usage.
> option alu.order disk-usage:read-usage:write-usage:open-files-
> usage:disk-speed-usage
>
> # Kick in if the discrepancy in disk-usage between volumes is more
> than 2GB
> option alu.disk-usage.entry-threshold 2GB
>
> # Don't stop writing to the least-used volume until the discrepancy
> is 1988MB
> option alu.disk-usage.exit-threshold 60MB
>
> # Kick in if the discrepancy in open files is 1024
> option alu.open-files-usage.entry-threshold 1024
>
> # Don't stop until 992 files have been written the least-used volume
> option alu.open-files-usage.exit-threshold 32
>
> # Kick in when the read-usage discrepancy is 20%
> option alu.read-usage.entry-threshold 20%
>
> # Don't stop until the discrepancy has been reduced to 16% (20% - 4%)
> option alu.read-usage.exit-threshold 4%
>
> # Kick in when the write-usage discrepancy is 20%
> option alu.write-usage.entry-threshold 20%
>
> ## Don't stop until the discrepancy has been reduced to 16%
> option alu.write-usage.exit-threshold 4%
>
> # Refresh the statistics used for decision-making every 10 seconds
> option alu.stat-refresh.interval 10sec
>
> # Refresh the statistics used for decision-making after creating 10
> files
> # option alu.stat-refresh.num-file-create 10
> end-volume
>
>
> #writebehind improves write performance a lot
> volume writebehind
> type performance/write-behind
> option aggregate-size 131072 # in bytes
> subvolumes unify
> end-volume
>
> Has anyone seen this issue before? Any suggestions?
>
> Thanks,
> -elb-
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20090611/5e41e87a/attachment.html>
More information about the Gluster-users
mailing list