[Gluster-users] GlusterFS 3.1.1: "Permission denied" and "No such file or directory"

Wed Mar 9 12:39:34 UTC 2011

Dear all,

for a long time now, we've had the same problem:
A user wants to delete a directory, receives an error message 
("Permission denied"), but the directoy is removed anyway. This problem 
has not been solved yet.

Today, a new facet of the same problem arose:
-------------------------
[...]
I often can't copy files to certain directories /storage/cluster, the 
error is "permission denied". However, creating a new directory and 
using this as destination for copying works.

Currently, the problem is that I can't rename a certain directory:
[ekpcms4] /storage/cluster/ott $ mv NovemberGridding_presel-antiele 
NovemberGridding_presel-antiele.old
mv: cannot move `NovemberGridding_presel-antiele' to 
`NovemberGridding_presel-antiele.old': Permission denied

However, after this command, the original directory is not listed in an 
"ls" anymore and the new directory (with ".old") appears. So despite the 
error message, it seems to have worked.

But I can't create a directory named after the old name, 
"NovemberGridding_presel-antiele":
mkdir NovemberGridding_presel-antiele
mkdir: cannot create directory `NovemberGridding_presel-antiele': File 
exists
---------------------------

The setup is still the same as before. Any help is very much appreciated.

Best Regards,
Daniel

On 02/01/2011 10:50 AM, Daniel Zander wrote:
> Dear all,
>
> what I tried:
> On the client side client as normal user:
> mkdir /storage/cluster/zander/mailtest
>
> drwxr-xr-x 2 zander belle 66 Feb 1 10:28 mailtest
>
> Then I checked, where the directory is found:
> Brick1: 192.168.101.249:/storage/4/cluster (xfs) yes!
> Brick2: 192.168.101.248:/storage/5/cluster (xfs) yes!
> Brick3: 192.168.101.250:/storage/6/cluster (xfs) yes!
> Brick4: 192.168.101.247:/storage/7/cluster (xfs) no!
> Brick5: 192.168.101.246:/storage/8/cluster (ext4) no!
>
> This confused me, as I was under the impression that this directory
> should be found on all servers. I then created some files in that
> directory and checked, where they appeared. Again nothing found in
> bricks 4 and 5.
>
> I tried to check the logs.They directory /usr/local/var/log/glusterfs
> exists, but is empty. /var/log/glusterfs/ contains some logfiles,
> however. I zipped the directories from Bricks 3,4 and 5.
>
> You can find them here:
> http://www-ekp.physik.uni-karlsruhe.de/~zander/2011-02-01-fs6-logs.tar.gz
> http://www-ekp.physik.uni-karlsruhe.de/~zander/2011-02-01-fs7-logs.tar.gz
> http://www-ekp.physik.uni-karlsruhe.de/~zander/2011-02-01-fs8-logs.tar.gz
>
> I performed the test at 10:45 on February 1st (which was a few minutes
> ago).
>
> The error always happens, i.e. whenever someone wants to delete a
> directory. I am now sure it has to do with the fact that the directory
> is not written on Bricks 4 and 5. Data is writte onto these new bricks
> however, but it seems to me only in directories that already exist. The
> problem occured since we switched from glusterFS 3.0 to 3.1.1.
>
> Thanks for any any help,
> Daniel
>
>
>
> On 01/29/2011 04:20 AM, Pranith Kumar. Karampuri wrote:
>> If you have never edited vol files in 3.1.1 that means it is generated
>> by glusterd and it generates access-control xlator. So it is
>> definitely BUG 2296. Chown workaround will take care of it for now.
>> I need some data about "Delete problem", the circumstances it is
>> happening etc. I am not able to reproduce anything like what you are
>> stating. You said that the bricks you are using have both ext4 and
>> xfs. Do this test. Create one directory, do ls -l on that directory so
>> that we know the permissions/ownership etc. Check on which brick it is
>> created and find out whether that brick is xfs or ext4. Now delete the
>> directory. Do this until you get the error. Glusterfs stores the logs
>> under the directory /usr/local/var/log/glusterfs. Zip that directory
>> along with the "ls -l" outputs you collected and send them across. We
>> can take a look. It is helpful if you can confirm that the error is
>> happening only on the bricks with xfs or ext4 or both.
>>
>> Pranith.
>> ----- Original Message -----
>> From: "Daniel Zander"<zander at ekp.uni-karlsruhe.de>
>> To: "Pranith Kumar. Karampuri"<pranithk at gluster.com>
>> Cc: gluster-users at gluster.org
>> Sent: Friday, January 28, 2011 7:16:05 PM
>> Subject: Re: [Gluster-users] GlusterFS 3.1.1: "Permission denied" and
>> "No such file or directory"
>>
>> Hi!
>>
>>> To confirm the "Permission Denied" problem you are facing is indeed
>>> BUG 2296 following conditions should be met:
>>> 1) mount is fuse mount.
>> This I can confirm.
>>
>>> 2) access-control translator should be present on top of the posix
>>> translator in the brick volfile.
>> Unfortunately, neither me nor any colleague has any idea what this
>> means. We have no real sysadmins at our institute, sorry. A wild guess:
>> As we are using glusterFS 3.1.1, I have never edited or even seen any
>> volfiles. Or is that something different?
>>
>> As the workaround really seems to work, I think we'll stick with that
>> for the moment. Any more ideas about the "delete problem"?
>>
>> Best Regards,
>> Daniel
>>
>>
>>> you can see the reports of the following bugs for "Permission Denied
>>> on Fuse mount":
>>> http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=2304
>>> http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=2312
>>> http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=2296
>>>
>>> chown -R user:primary-group is a workaround that should fix the
>>> problem for now.
>>>
>>> Pranith.
>>> ----- Original Message -----
>>> From: "Daniel Zander"<zander at ekp.uni-karlsruhe.de>
>>> To: "Pranith Kumar. Karampuri"<pranithk at gluster.com>
>>> Cc: gluster-users at gluster.org
>>> Sent: Friday, January 28, 2011 3:44:46 PM
>>> Subject: Re: [Gluster-users] GlusterFS 3.1.1: "Permission denied" and
>>> "No such file or directory"
>>>
>>> Hi Pranith and all,
>>>
>>> I am not entirely sure what you mean by "how to trigger" the second
>>> problem.
>>>
>>> This problem always occurs. I simply create a folder called test and
>>> perform an rm -rf on this folder. I get the error message
>>>
>>> rm: cannot remove directory `test/': No such file or directory
>>>
>>> but the folder vanishes anyway.
>>>
>>> About the first problem again: When you say, non-primary group
>>> membership is treated as "others", this can't explain, as far as I
>>> understand, why sometimes writing into the very same directoy works and
>>> sometimes it fails. By the way: where can I see the bug reports?
>>>
>>> Issuing a chown -R<user>:<primary_group> should resolve these issues,
>>> shouldn't it?
>>>
>>> Thanks for your help,
>>> Daniel
>>>
>>>
>>> On 01/28/2011 09:32 AM, Pranith Kumar. Karampuri wrote:
>>>> hi Daniel,
>>>> The first problem seems like bug 2296. Non-primary group membership
>>>> was treated as "other". It is already resolved and will be available
>>>> in 3.1.3. Do you know how to trigger the second problem?.
>>>>
>>>> Pranith.
>>>>
>>>> ----- Original Message -----
>>>> From: "Daniel Zander"<zander at ekp.uni-karlsruhe.de>
>>>> To: gluster-users at gluster.org
>>>> Sent: Friday, January 28, 2011 1:53:14 PM
>>>> Subject: [Gluster-users] GlusterFS 3.1.1: "Permission denied" and
>>>> "No such file or directory"
>>>>
>>>> Dear all,
>>>>
>>>> since the upgrade from glusterFS 3.0.0 to 3.1.1, some users have
>>>> experienced strange problems. When they want to create a folder or a
>>>> file, this usually works, but sometimes, it doesn't. The error message
>>>> is something like "Permission Denied".
>>>>
>>>> The permissions and ownership of the affected folders seem to be set
>>>> correctly. I tried to chown -R user:group user/ anyway and it helped
>>>> once. It even occurs that when jobs try to write output back from our
>>>> batch system, most jobs can write files that are accessible, but a
>>>> small
>>>> number cannot write their output (to the same folder at the same time).
>>>>
>>>> Another problem, that might be related to the ones above is that
>>>> whenever a users tries to delete a directory, the following message
>>>> occurs:
>>>>
>>>> rm: cannot remove directory `test/': No such file or directory
>>>>
>>>> but the directory is deleted anyway.
>>>>
>>>> I've also seen files, that seem to have no permissions at all, e.g.
>>>>
>>>> ----------T 2 root root 4.1K Jan 17 10:26 mc_jpsi
>>>>
>>>> This can be fixed manually, but is still very strange.
>>>>
>>>> About our setup:
>>>>
>>>> Volume Name: lemmy
>>>> Type: Distribute
>>>> Status: Started
>>>> Number of Bricks: 5
>>>> Transport-type: tcp
>>>> Bricks:
>>>> Brick1: 192.168.101.249:/storage/4/cluster
>>>> Brick2: 192.168.101.248:/storage/5/cluster
>>>> Brick3: 192.168.101.250:/storage/6/cluster
>>>> Brick4: 192.168.101.247:/storage/7/cluster
>>>> Brick5: 192.168.101.246:/storage/8/cluster
>>>>
>>>>
>>>> There is sufficient space left on all bricks. The operating systems of
>>>> the servers are debian5 (once) and Ubuntu 10.04 server (4 times). The
>>>> bricks have different filesystems (ext4 and xfs).
>>>>
>>>> Thanks and Best Regards,
>>>> Daniel
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users