[Gluster-users] quota calculation gluster 3.2.3

Mohammed Junaid junaid at gluster.com
Mon Sep 5 04:51:45 PDT 2011


Coming back to the problem of quota sizes not converging, the converging may
not complete if glusterd is restarted before converging process is complete
because a thread is spawned by glusterd to do the converging. Hence, if
glusterd is restarted converging will fail or to put it precisely it will
not be completed. So running ls -lR on the mount point will solve the
problem.

On Mon, Sep 5, 2011 at 4:40 PM, Mohammed Junaid <junaid at gluster.com> wrote:

> Hi Anthony,
>
> This is quiet unusual, to debug this further can you get the output of "du
> -h" of mount-point and of directory soft (which is under mount point). This
> is will help a lot in debugging this issue. Also, I noticed that the volume
> info output provided is of a distributed-replicate setup which is different
> from the one provided previously (it was plain replicate). Also please
> provide the output of getfattr for the bricks and soft directory.
>
> Junaid
>
> On Mon, Sep 5, 2011 at 2:58 PM, anthony garnier <sokar6012 at hotmail.com>wrote:
>
>>
>> Volume configuration:
>>
>> yval1000:/ # gluster volume info
>>
>> Volume Name: venus
>> Type: Distributed-Replicate
>> Status: Started
>> Number of Bricks: 2 x 2 = 4
>>
>> Transport-type: tcp
>> Bricks:
>> Brick1: ylal3020:/soft/venus
>> Brick2: ylal3030:/soft/venus
>> Brick3: yval1000:/soft/venus
>> Brick4: yval1010:/soft/venus
>> Options Reconfigured:
>> features.quota-timeout: 120
>> features.limit-usage: /test:100MB,/psa:200MB,/:7GB,/soft:5GB
>> features.quota: on
>> geo-replication.indexing: on
>> network.ping-timeout: 10
>> performance.cache-size: 1GB
>> performance.cache-refresh-timeout: 60
>> nfs.port: 2049
>>
>>
>> Indeed values didn't converged automaticaly for big folder, for small
>> folder like /test and /psa it was fine (I waited 1 day before checking the
>> value). I was forced to run ls -lRa several times, but even with this
>> workaround, values for big folder like /soft are not rightfully calculated :
>>
>>
>> gluster volume quota venus list
>>     path          limit_set         size
>>
>> ----------------------------------------------------------------------------------
>> /test                     100MB                4.5MB <= OK
>> /psa                      200MB              167.0MB <=OK
>> /                           7GB                5.2GB <=Strange
>> /soft                       5GB                4.7GB <= Strange
>>
>> => the size of the entire filesystem is 1.4GB
>>
>>
>>
>> Anthony
>>
>>
>> ------------------------------
>> From: junaid at gluster.com
>> Date: Mon, 5 Sep 2011 14:38:06 +0530
>>
>> Subject: Re: [Gluster-users] quota calculation gluster 3.2.3
>> To: sokar6012 at hotmail.com
>> CC: gluster-users at gluster.org
>>
>> Hi Anthony,
>>
>> By volume configuration, I meant 'is the volume setup dht, afr, or
>> dht-afr', I would like to know the output of
>>
>>    volume info volume-name
>>
>> When data is already present before the activation of quota, it takes some
>> time for quota values to converge because it has to crawl through the entire
>> file-system to calculate the sizes. This is automatically done when you run
>>
>>    volume quota vol-name enable
>>
>> The output of
>>    volume quota vol-name list
>>
>> will reflect the aggregated value at that point in time. If the values are
>> not converging then its a issue. A simple work-around is to run ls -lR on
>> the mount point. Please confirm if the values were not converging
>> automatically, we will have a look into it.
>>
>> Junaid
>>
>>
>> On Mon, Sep 5, 2011 at 12:51 PM, anthony garnier <sokar6012 at hotmail.com>wrote:
>>
>>  Junaid,
>>
>> What do you mean by " the volume configuration on other two servers" ?
>> Configuration is the same on all the server..
>> I'm creating data with "dd" cmd, and yes, data was already present before
>> activation of quota.
>>
>>
>>
>> ------------------------------
>> From: junaid at gluster.com
>> Date: Fri, 2 Sep 2011 22:55:02 +0530
>>
>> Subject: Re: [Gluster-users] quota calculation gluster 3.2.3
>> To: sokar6012 at hotmail.com
>> CC: gluster-users at gluster.org
>>
>> Hi Anthony,
>>
>> Thanks for providing the input. What is the volume configuration on other
>> two servers (yval1000 and yval1010)? My question is same, what data where
>> you creating on these two machines? Was the data already present before
>> quota was enabled?
>>
>> On Fri, Sep 2, 2011 at 1:49 PM, anthony garnier <sokar6012 at hotmail.com>wrote:
>>
>>
>>  I've done a ls -lRa throught the mount point and now i get :
>> /soft/venus # gluster volume quota venus list
>>     path          limit_set         size
>>
>> ----------------------------------------------------------------------------------
>> /                           2GB                1.3GB
>> /test                     100MB                8.0KB
>> /psa                      200MB              151.0MB
>> /soft                    1500MB              826.3MB
>>
>> =>Seems to be consistent
>>
>> and now command  getfattr on yval1000 and yval1010 show :
>>
>> yval1000 :
>> getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/
>> trusted.glusterfs.quota.size=0x0000000028f9ca00
>>
>>
>> getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/soft
>> trusted.glusterfs.quota.size=0x000000001421d400
>>
>>
>> getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/psa
>>  trusted.glusterfs.quota.size=0x00000000053d6000
>>
>>
>> yval1010 :
>> getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/
>> trusted.glusterfs.quota.size=0x000000002904ea00
>>
>>
>> getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/soft
>> trusted.glusterfs.quota.size=0x000000002292ba00
>>
>>
>> getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/psa
>> trusted.glusterfs.quota.size=0x00000000053d6000
>>
>> Regards,
>>
>> Anthony
>>
>>
>> ------------------------------
>> From: sokar6012 at hotmail.com
>> To: junaid at gluster.com
>> CC: gluster-users at gluster.org
>> Subject: RE: [Gluster-users] quota calculation gluster 3.2.3
>> Date: Fri, 2 Sep 2011 07:57:19 +0000
>>
>>
>>  I forgot to check on the 2 other server.
>> So ylal3020 and ylal3030 show similar output send previously.
>>
>> But yval1000 and yval1010 show with all command :
>>  trusted.glusterfs.quota.size=0x0000000000000000
>>
>> Is it normal ?
>>
>> Regards,
>>
>> Anthony
>> ------------------------------
>> From: sokar6012 at hotmail.com
>> To: junaid at gluster.com
>> CC: gluster-users at gluster.org
>> Subject: RE: [Gluster-users] quota calculation gluster 3.2.3
>> Date: Fri, 2 Sep 2011 07:51:52 +0000
>>
>>  Hi Junaid,
>>
>> Here is the output of the diffrent command executed on both server.
>> Command show identical output on both server
>>
>> # getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus
>> getfattr: Removing leading '/' from absolute path names
>> # file: soft/venus
>> trusted.glusterfs.quota.size=0x0000000022ccac00
>>
>>  # getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/test
>> getfattr: Removing leading '/' from absolute path names
>> # file: soft/venus/test
>> trusted.glusterfs.quota.size=0x0000000000000000
>>
>>  # getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/psa
>> getfattr: Removing leading '/' from absolute path names
>> # file: soft/venus/psa
>> trusted.glusterfs.quota.size=0x000000000013d000
>>
>>  # getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/soft
>> getfattr: Removing leading '/' from absolute path names
>> # file: soft/venus/soft
>> trusted.glusterfs.quota.size=0x000000000d15ce00
>>
>>
>> # gluster volume quota venus list
>>     path          limit_set         size
>>
>> ----------------------------------------------------------------------------------
>> /                           2GB              556.8MB
>> /test                     100MB               0Bytes
>> /psa                      200MB                1.2MB
>> /soft                    1500MB              209.4MB
>>
>>
>> Indeed there is 90% of small file (~60000 files).
>>
>> Thx
>>
>> Anthony
>>
>>
>> ------------------------------
>> From: junaid at gluster.com
>> Date: Fri, 2 Sep 2011 06:18:30 +0530
>> Subject: Re: [Gluster-users] quota calculation gluster 3.2.3
>> To: sokar6012 at hotmail.com
>> CC: gluster-users at gluster.org
>>
>> Hi Anthony,
>>
>> To debug this further, can you send the output of
>>
>> getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus
>> getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/test
>> getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/psa
>> getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/soft
>>
>> from both the machine. Also check the output of
>>
>> gluster volume quota Replicate list
>>
>> Sometimes it takes a small amount of time to bring up the sizes. Also,
>> what kind of data where you creating (I mean large amount of small files or
>> large files because in case of small files, the directory sizes are not
>> accounted by quota to calculate the size unlike the du -h command which uses
>> the directory size as well).
>>
>> Junaid
>> On Thu, Sep 1, 2011 at 1:37 PM, anthony garnier <sokar6012 at hotmail.com>wrote:
>>
>>  Hi all,
>>
>> I've enable quota but I'm a bit confused by values displayed by GlusterFS
>>
>> Here is my volume :
>>
>> Volume Name: venus
>> Type: Replicate
>> Status: Started
>> Number of Bricks: 2
>> Transport-type: tcp
>> Bricks:
>> Brick1: ylal3020:/soft/venus
>> Brick2: ylal3030:/soft/venus
>> Options Reconfigured:
>> features.limit-usage: /test:100MB,/psa:200MB,/soft:1500MB,/:2GB
>> features.quota: on
>> diagnostics.client-log-level: ERROR
>> diagnostics.brick-log-level: ERROR
>> network.ping-timeout: 10
>> performance.cache-size: 2GB
>> nfs.port: 2049
>>
>> I've got 3 folders in the backend (/soft/venus) :
>> psa        160MB  (with du -sh)
>> soft         1.2GB
>> test         12KB
>> Total       1.4GB
>>
>> But when I list the quota with gluster I got :
>> # gluster volume quota venus list
>>     path          limit_set         size
>>
>> ----------------------------------------------------------------------------------
>> /test                     100MB               12.0KB                  <=
>> This one is OK
>> /psa                      200MB               64.4MB           <= not OK
>> /soft                    1500MB              281.8MB          <= not OK
>> /                           2GB              346.2MB              <= not
>> OK
>>
>> Any idea ?
>>
>> Regards,
>>
>> Anthony Garnier
>>
>>
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>
>>
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gluster.org/pipermail/gluster-users/attachments/20110905/3513075c/attachment-0001.htm>


More information about the Gluster-users mailing list