[Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC
Ramon Selga
ramon.selga at gmail.com
Tue Dec 12 18:58:48 UTC 2023
Dismiss my first question: you have SAS 12Gbps SSDs Sorry!
El 12/12/23 a les 19:52, Ramon Selga ha escrit:
> May ask you which kind of disks you have in this setup? rotational, ssd
> SAS/SATA, nvme?
>
> Is there a RAID controller with writeback caching?
>
> It seems to me your fio test on local brick has a unclear result due to some
> caching.
>
> Try something like (you can consider to increase test file size depending of
> your caching memory) :
>
> fio --size=16G --name=test --filename=/gluster/data/brick/wow --bs=1M
> --nrfiles=1 --direct=1 --sync=0 --randrepeat=0 --rw=write --refill_buffers
> --end_fsync=1 --iodepth=200 --ioengine=libaio
>
> Also remember a replica 3 arbiter 1 volume writes synchronously to two data
> bricks, halving throughput of your network backend.
>
> Try similar fio on gluster mount but I hardly see more than 300MB/s writing
> sequentially on only one fuse mount even with nvme backend. On the other side,
> with 4 to 6 clients, you can easily reach 1.5GB/s of aggregate throughput
>
> To start, I think is better to try with default parameters for your replica
> volume.
>
> Best regards!
>
> Ramon
>
>
> El 12/12/23 a les 19:10, Danny ha escrit:
>> Sorry, I noticed that too after I posted, so I instantly upgraded to 10.
>> Issue remains.
>>
>> On Tue, Dec 12, 2023 at 1:09 PM Gilberto Ferreira
>> <gilberto.nunes32 at gmail.com> wrote:
>>
>> I strongly suggest you update to version 10 or higher.
>> It's come with significant improvement regarding performance.
>> ---
>> Gilberto Nunes Ferreira
>> (47) 99676-7530 - Whatsapp / Telegram
>>
>>
>>
>>
>>
>>
>> Em ter., 12 de dez. de 2023 às 13:03, Danny <dbray925+gluster at gmail.com
>> <mailto:dbray925%2Bgluster at gmail.com>> escreveu:
>>
>> MTU is already 9000, and as you can see from the IPERF results, I've
>> got a nice, fast connection between the nodes.
>>
>> On Tue, Dec 12, 2023 at 9:49 AM Strahil Nikolov
>> <hunter86_bg at yahoo.com> wrote:
>>
>> Hi,
>>
>> Let’s try the simple things:
>>
>> Check if you can use MTU9000 and if it’s possible, set it on the
>> Bond Slaves and the bond devices:
>> ping GLUSTER_PEER -c 10 -M do -s 8972
>>
>> Then try to follow up the recommendations from
>> https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/chap-configuring_red_hat_storage_for_enhancing_performance
>>
>>
>>
>> Best Regards,
>> Strahil Nikolov
>>
>> On Monday, December 11, 2023, 3:32 PM, Danny
>> <dbray925+gluster at gmail.com
>> <mailto:dbray925%2Bgluster at gmail.com>> wrote:
>>
>> Hello list, I'm hoping someone can let me know what setting I
>> missed.
>>
>> Hardware:
>> Dell R650 servers, Dual 24 Core Xeon 2.8 GHz, 1 TB RAM
>> 8x SSD s Negotiated Speed 12 Gbps
>> PERC H755 Controller - RAID 6
>> Created virtual "data" disk from the above 8 SSD drives, for
>> a ~20 TB /dev/sdb
>>
>> OS:
>> CentOS Stream
>> kernel-4.18.0-526.el8.x86_64
>> glusterfs-7.9-1.el8.x86_64
>>
>> IPERF Test between nodes:
>> [ ID] Interval Transfer Bitrate Retr
>> [ 5] 0.00-10.00 sec 11.5 GBytes 9.90 Gbits/sec 0
>> sender
>> [ 5] 0.00-10.04 sec 11.5 GBytes 9.86 Gbits/sec receiver
>>
>> All good there. ~10 Gbps, as expected.
>>
>> LVM Install:
>> export DISK="/dev/sdb"
>> sudo parted --script $DISK "mklabel gpt"
>> sudo parted --script $DISK "mkpart primary 0% 100%"
>> sudo parted --script $DISK "set 1 lvm on"
>> sudo pvcreate --dataalignment 128K /dev/sdb1
>> sudo vgcreate --physicalextentsize 128K gfs_vg /dev/sdb1
>> sudo lvcreate -L 16G -n gfs_pool_meta gfs_vg
>> sudo lvcreate -l 95%FREE -n gfs_pool gfs_vg
>> sudo lvconvert --chunksize 1280K --thinpool gfs_vg/gfs_pool
>> --poolmetadata gfs_vg/gfs_pool_meta
>> sudo lvchange --zero n gfs_vg/gfs_pool
>> sudo lvcreate -V 19.5TiB --thinpool gfs_vg/gfs_pool -n gfs_lv
>> sudo mkfs.xfs -f -i size=512 -n size=8192 -d su=128k,sw=10
>> /dev/mapper/gfs_vg-gfs_lv
>> sudo vim /etc/fstab
>> /dev/mapper/gfs_vg-gfs_lv /gluster/data/brick xfs
>> rw,inode64,noatime,nouuid 0 0
>>
>> sudo systemctl daemon-reload && sudo mount -a
>> fio --name=test --filename=/gluster/data/brick/wow --size=1G
>> --readwrite=write
>>
>> Run status group 0 (all jobs):
>> WRITE: bw=2081MiB/s (2182MB/s), 2081MiB/s-2081MiB/s
>> (2182MB/s-2182MB/s), io=1024MiB (1074MB), run=492-492msec
>>
>> All good there. 2182MB/s =~ 17.5 Gbps. Nice!
>>
>>
>> Gluster install:
>> export NODE1='10.54.95.123'
>> export NODE2='10.54.95.124'
>> export NODE3='10.54.95.125'
>> sudo gluster peer probe $NODE2
>> sudo gluster peer probe $NODE3
>> sudo gluster volume create data replica 3 arbiter 1
>> $NODE1:/gluster/data/brick $NODE2:/gluster/data/brick
>> $NODE3:/gluster/data/brick force
>> sudo gluster volume set data network.ping-timeout 5
>> sudo gluster volume set data performance.client-io-threads on
>> sudo gluster volume set data group metadata-cache
>> sudo gluster volume start data
>> sudo gluster volume info all
>>
>> Volume Name: data
>> Type: Replicate
>> Volume ID: b52b5212-82c8-4b1a-8db3-52468bc0226e
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 1 x (2 + 1) = 3
>> Transport-type: tcp
>> Bricks:
>> Brick1: 10.54.95.123:/gluster/data/brick
>> Brick2: 10.54.95.124:/gluster/data/brick
>> Brick3: 10.54.95.125:/gluster/data/brick (arbiter)
>> Options Reconfigured:
>> network.inode-lru-limit: 200000
>> performance.md-cache-timeout: 600
>> performance.cache-invalidation: on
>> performance.stat-prefetch: on
>> features.cache-invalidation-timeout: 600
>> features.cache-invalidation: on
>> network.ping-timeout: 5
>> transport.address-family: inet
>> storage.fips-mode-rchecksum: on
>> nfs.disable: on
>> performance.client-io-threads: on
>>
>> sudo vim /etc/fstab
>> localhost:/data /data glusterfs
>> defaults,_netdev 0 0
>>
>> sudo systemctl daemon-reload && sudo mount -a
>> fio --name=test --filename=/data/wow --size=1G --readwrite=write
>>
>> Run status group 0 (all jobs):
>> WRITE: bw=109MiB/s (115MB/s), 109MiB/s-109MiB/s
>> (115MB/s-115MB/s), io=1024MiB (1074MB), run=9366-9366msec
>>
>> Oh no, what's wrong? From 2182MB/s down to only 115MB/s? What
>> am I missing? I'm not expecting the above ~17 Gbps, but I'm
>> thinking it should at least be close(r) to ~10 Gbps.
>>
>> Any suggestions?
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://meet.google.com/cpu-eiue-hvk
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://meet.google.com/cpu-eiue-hvk
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge:https://meet.google.com/cpu-eiue-hvk
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20231212/ea3433f2/attachment.html>
More information about the Gluster-users
mailing list