[Gluster-users] Gluster 9.3 not distributing data evenly across bricks
Diego Zuccato
diego.zuccato at unibo.it
Fri Dec 1 13:04:25 UTC 2023
IIUC you have to manually run a rebalance
gluster v rebalance datavol start
Diego
Il 01/12/2023 13:37, Shreyansh Shah ha scritto:
> Hi,
> We are running a gluster 9.3 volume with 47 bricks across 13 nodes. We
> are noticing uneven data distribution across the bricks. Some bricks are
> 100% utilised whereas some are < 85%. Due to this, data write randomly
> fails with error OSError(28, 'No space left on device') even when the
> volume has sufficient space on other bricks. We expect the glusterfs
> process to automatically sync and maintain equal data distribution
> across the bricks.
>
> This is the volume info:
>
> root at nas-5:~# gluster v info
>
> Volume Name: datavol
> Type: Distribute
> Volume ID: b4b52b4b-0ea0-4eeb-a359-5a0573d4f83a
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 47
> Transport-type: tcp
> Bricks:
> Brick1: 10.132.2.101:/data/data
> Brick2: 10.132.2.101:/data1/data
> Brick3: 10.132.2.101:/data2/data
> Brick4: 10.132.2.101:/data3/data
> Brick5: 10.132.2.102:/data/data
> Brick6: 10.132.2.102:/data1/data
> Brick7: 10.132.2.102:/data2/data
> Brick8: 10.132.2.102:/data3/data
> Brick9: 10.132.2.103:/data/data
> Brick10: 10.132.2.103:/data1/data
> Brick11: 10.132.2.103:/data2/data
> Brick12: 10.132.2.103:/data3/data
> Brick13: 10.132.2.104:/data/data
> Brick14: 10.132.2.104:/data3/data
> Brick15: 10.132.2.105:/data1/data
> Brick16: 10.132.2.105:/data2/data
> Brick17: 10.132.2.106:/data/data
> Brick18: 10.132.2.106:/data1/data
> Brick19: 10.132.2.106:/data2/data
> Brick20: 10.132.2.107:/data/data
> Brick21: 10.132.2.107:/data1/data
> Brick22: 10.132.2.107:/data2/data
> Brick23: 10.132.2.108:/data/data
> Brick24: 10.132.2.108:/data1/data
> Brick25: 10.132.2.108:/data2/data
> Brick26: 10.132.2.109:/data/data
> Brick27: 10.132.2.109:/data1/data
> Brick28: 10.132.2.109:/data2/data
> Brick29: 10.132.2.110:/data/data
> Brick30: 10.132.2.110:/data1/data
> Brick31: 10.132.2.111:/data/data
> Brick32: 10.132.2.111:/data1/data
> Brick33: 10.132.2.111:/data2/data
> Brick34: 10.132.2.112:/data/data
> Brick35: 10.132.2.112:/data1/data
> Brick36: 10.132.2.112:/data2/data
> Brick37: 10.132.2.113:/data/data
> Brick38: 10.132.2.113:/data1/data
> Brick39: 10.132.2.113:/data2/data
> Brick40: 10.132.2.108:/data3/data
> Brick41: 10.132.2.107:/data3/data
> Brick42: 10.132.2.106:/data3/data
> Brick43: 10.132.2.105:/data3/data
> Brick44: 10.132.2.110:/data2/data
> Brick45: 10.132.2.105:/data/data
> Brick46: 10.132.2.104:/data1/data
> Brick47: 10.132.2.104:/data2/data
> Options Reconfigured:
> performance.client-io-threads: on
> cluster.min-free-disk: 2%
> performance.cache-refresh-timeout: 60
> client.event-threads: 4
> server.event-threads: 4
> network.ping-timeout: 90
> storage.health-check-interval: 60
> storage.health-check-timeout: 60
> performance.io-cache-size: 8GB
> storage.fips-mode-rchecksum: on
> transport.address-family: inet
> nfs.disable: on
> root at nas-5:~# glusterfs --version
> glusterfs 9.3
> Repository revision: git://git.gluster.org/glusterfs.git
> <http://git.gluster.org/glusterfs.git>
> Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/
> <https://www.gluster.org/>>
> GlusterFS comes with ABSOLUTELY NO WARRANTY.
> It is licensed to you under your choice of the GNU Lesser
> General Public License, version 3 or any later version (LGPLv3
> or later), or the GNU General Public License, version 2 (GPLv2),
> in all cases as published by the Free Software Foundation.
> root at nas-5:~#
>
>
>
> This is the disk usage distribution when the write failed:
> *Node 1:*
> /dev/bcache1 9.7T 8.5T 1.2T 88% /data
> /dev/bcache3 3.9T 3.3T 563G 86% /data1
> /dev/bcache0 3.9T 3.3T 560G 86% /data2
> /dev/bcache2 3.9T 3.5T 395G 90% /data3
> *Node 2:*
> /dev/bcache0 9.7T 8.5T 1.2T 88% /data
> /dev/bcache1 3.9T 3.5T 421G 90% /data1
> /dev/bcache2 3.9T 3.6T 330G 92% /data2
> /dev/bcache3 3.9T 3.4T 513G 87% /data3
> *Node 3:*
> /dev/bcache0 9.7T 8.8T 882G 92% /data
> /dev/bcache1 3.9T 3.6T 335G 92% /data1
> /dev/bcache2 3.9T 3.4T 532G 87% /data2
> /dev/bcache3 3.9T 3.3T 564G 86% /data3
> *Node 4:*
> /dev/bcache0 9.7T 8.7T 982G 91% /data
> /dev/bcache1 3.9T 3.5T 424G 90% /data1
> /dev/bcache2 3.9T 3.4T 549G 87% /data2
> /dev/bcache3 3.9T 3.6T 344G 92% /data3
> *Node 5:*
> /dev/bcache0 9.7T 8.5T 1.3T 88% /data
> /dev/bcache1 3.9T 3.6T 288G 93% /data1
> /dev/bcache2 3.9T 3.4T 470G 89% /data2
> /dev/bcache3 9.9T 9.8T 101G 100% /data3
> *Node 6:*
> /dev/bcache0 9.7T 8.2T 1.5T 86% /data
> /dev/bcache1 3.9T 3.4T 526G 87% /data1
> /dev/bcache2 3.9T 3.5T 431G 90% /data2
> /dev/bcache3 9.9T 8.9T 1.1T 90% /data3
> *Node 7:*
> /dev/bcache0 9.7T 8.9T 783G 93% /data
> /dev/bcache1 3.9T 3.3T 561G 86% /data1
> /dev/bcache2 3.9T 3.5T 360G 91% /data2
> /dev/bcache3 9.9T 8.7T 1.2T 89% /data3
> *Node 8:*
> /dev/bcache0 9.7T 8.7T 994G 90% /data
> /dev/bcache1 3.9T 3.3T 645G 84% /data1
> /dev/bcache2 3.9T 3.4T 519G 87% /data2
> /dev/bcache3 9.9T 9.0T 868G 92% /data3
> *Node 9:*
> /dev/bcache0 10T 8.6T 1.4T 87% /data
> /dev/bcache1 8.0T 6.7T 1.4T 84% /data1
> /dev/bcache2 8.0T 6.8T 1.3T 85% /data2
> *Node 10:*
> /dev/bcache0 10T 8.8T 1.3T 88% /data
> /dev/bcache1 8.0T 6.6T 1.4T 83% /data1
> /dev/bcache2 8.0T 7.0T 990G 88% /data2
> *Node 11:*
> /dev/bcache0 10T 8.1T 1.9T 82% /data
> /dev/bcache1 10T 8.5T 1.5T 86% /data1
> /dev/bcache2 10T 8.4T 1.6T 85% /data2
> *Node 12:*
> /dev/bcache0 10T 8.4T 1.6T 85% /data
> /dev/bcache1 10T 8.4T 1.6T 85% /data1
> /dev/bcache2 10T 8.2T 1.8T 83% /data2
> *Node 13:*
> /dev/bcache1 10T 8.7T 1.3T 88% /data1
> /dev/bcache2 10T 8.8T 1.2T 88% /data2
> /dev/bcache0 10T 8.6T 1.5T 86% /data
>
>
>
>
> --
> Regards,
> Shreyansh Shah
> AlphaGrep* Securities Pvt. Ltd.*
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786
More information about the Gluster-users
mailing list