[Gluster-users] open() issues on 3.0.0
Brian Smith
brs at usf.edu
Sun Jan 17 18:21:48 UTC 2010
Good find! That seems to be exactly the issue. My inode numbers for
the files in that directory are literally just 100k away from the 32-bit
cut-off. I've rm'ed and re-extraced to get lesser inode values and
voila! I guess intel must have thought that 2^32 inodes outta be enough
for everyone.
Many thanks,
-Brian
--
Brian Smith
Senior Systems Administrator
IT Research Computing, University of South Florida
4202 E. Fowler Ave. ENB308
Office Phone: +1 813 974-1467
Organization URL: http://rc.usf.edu
On Sun, 2010-01-17 at 13:12 -0500, Aaron Knister wrote:
> I ran into a similar problem when compiling code on a gluster mount
> with the intel compiler. The problem was intermittent and after
> digging into it further it appeared to only happy when compiling from
> source files that had very high inode numbers. There's a command run
> by icc that's a 32 bit executable and my guess is that somewhere
> internally the extremely large inode number wasn't able to be handled
> and caused the compilation to fail. The "fix" was to upgrade the
> compiler. I can verify that version 11.0.064 works without a hitch (at
> least in this regard).
>
>
> More Detail:
>
>
> I can reproduce this error with version 10.0.026 of the intel
> compiler.
>
>
> I have file test.c on a glusterfs with inode num 21479794854 (greater
> than 32 bits)
> 21479794854 -rw-r--r-- 1 aaron users 26 Jan 17 12:58 test.c
>
>
> Running icc fails:
> ...
> $ icc test.c
> Catastrophic error: could not open source file "test.c"
>
>
> compilation aborted for test.c (code 4)
> ...
>
>
> However if I copy the same file to a directory on the same fs that has
> a 32 bit inode number (le 4294967295) (hoping the new file will also
> have a 32 bit inode number):
> 11889296 -rw-r--r-- 1 aaron users 26 Jan 17 13:02 test.c
>
>
> Compilation is a success:
> ...
> $ !icc
> icc test.c
> $ echo $?
> 0
> ...
>
>
>
>
> If you run icc with the -v option you'll see that a program called
> mcpcom gets run and produces the error. At least with version 10.0.026
> it was a 32 bit program:
> $ file /opt/intel/cce/10.0.026/bin/mcpcom
> /opt/intel/cce/10.0.026/bin/mcpcom: ELF 32-bit LSB executable, Intel
> 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses
> shared libs), for GNU/Linux 2.2.5, not stripped
>
>
> Anyway I hope that helps.
>
>
> -Aaron
>
>
>
> On Jan 17, 2010, at 11:13 AM, Brian Smith wrote:
>
> > Hi,
> >
> > I'm new to Gluster, so please forgive any ignorance on my part.
> > I've
> > tried reading over everything I could find to resolve this myself.
> >
> > I'm having an issue building executables on our gluster volume
> > (there
> > are some other similar issues as well, but we'll deal with this one
> > first). I have the presta infiniband utilities extracted into a
> > directory in my glusterfs volume and I'm attempting to build them.
> > One
> > of the source files, util.c (coincidentally, the first one
> > referenced in
> > the make file) fails to build. I get an error message:
> >
> > Catastrophic error: could not open source file "util.c"
> >
> > Some further investigating reveals the command that causes the
> > issue:
> >
> > icc -I/opt/priv/openmpi-1.4.1/intel-10.1.018/include -pthread
> > -L/opt/priv/mx/lib -L/opt/priv/openmpi-1.4.1/intel-10.1.018/lib
> > -lmpi
> > -lopen-rte -lopen-pal -lmyriexpress -libverbs -lpsm_infinipath
> > -lnuma
> > -ldl -Wl,--export-dynamic -lnsl -lutil -c util.c
> > Catastrophic error: could not open source file "util.c"
> >
> > compilation aborted for util.c (code 4)
> >
> > And further, we see an strace of the command:
> > ...
> > 2717 6601 brk(0x957b000) = 0x957b000
> > 2718 6601 open("util.c", O_RDONLY) = 3
> > 2719 6601 stat64("util.c", {st_mode=S_IFREG|0600,
> > st_size=14370, ...})
> > = 0
> > 2720 6601 close(3) = 0
> > 2721 6601 fstat64(2, {st_mode=S_IFCHR|0620, st_rdev=makedev(136,
> > 1), ...}) = 0
> > 2722 6601 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|
> > MAP_ANONYMOUS, -1, 0x1000) = 0xfffffffff0bce000
> > 2723 6601 write(2, "Catastrophic error: could not open source file
> > \"util.c\"\n", 56) = 56
> > 2724 6601 write(2, "\n", 1) = 1
> > 2725 6601 exit_group(4)
> > ...
> >
> > We can also see that the file is indeed present:
> >
> > $ ls -l util.c
> > -rw------- 1 brs users 14370 Apr 8 2002 util.c
> > $ stat util.c
> > File: `util.c'
> > Size: 14370 Blocks: 32 IO Block: 4096 regular file
> > Device: 18h/24d Inode: 4295774314 Links: 1
> > Access: (0600/-rw-------) Uid: (1229806/ brs) Gid: (10001/
> > users)
> > Access: 2010-01-17 00:27:43.000000000 -0500
> > Modify: 2002-04-08 21:13:57.000000000 -0400
> > Change: 2010-01-17 00:27:43.000000000 -0500
> >
> > If I extract the tar-ball in /tmp, a local ext3 fs, the compile line
> > above works correctly.
> >
> > diff /work/b/brs/presta1.2/util.c /tmp/presta1.2/util.c
> >
> > also appears clean. Any ideas what is happening here? Is there an
> > issue with mmap2() and glusterfs?
> >
> > Many thanks in advance,
> >
> > -Brian
> >
> >
> >
> > --
> > Brian Smith
> > Senior Systems Administrator
> > IT Research Computing, University of South Florida
> > 4202 E. Fowler Ave. ENB308
> > Office Phone: +1 813 974-1467
> > Organization URL: http://rc.usf.edu
> >
> >
> >
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
> >
>
More information about the Gluster-users
mailing list