Re: striped LV and expected performance

Dne 18. 07. 25 v 9:13 Henry, Andrew napsal(a):
> I’m trying to improve the performance of a virtual machine guest running RHEL 9.6.  I’ve assigned 3 separate SAS controllers to the VM, and 3 vmdk disks each connected to their own controller, so there is no kernel bottleneck in the VM on the controller side when writing through to each vmdk under high load.
> 
> In the guest, I’m creating an LV as follows:
> 
> lvcreate -n swap -i 3 -I 64k --type=striped -L 16G vg00
> mkfs -t xfs -b size=4K -d su=64k,sw=3 -f /dev/vg00/swap
> 
> After mounting, I test write speeds using:
> 
> time dd if=/dev/zero of=/mnt/testfile1 bs=8192 count=65536 oflag=direct
> 
> 536870912 bytes (537 MB, 512 MiB) copied, 41.2043 s, 13.0 MB/s
> 536870912 bytes (537 MB, 512 MiB) copied, 27.3986 s, 19.6 MB/s
> 536870912 bytes (537 MB, 512 MiB) copied, 22.8359 s, 23.5 MB/s
> 
> time dd if=/dev/zero of=/mnt/testfile bs=64k count=163840 oflag=direct
> 
> 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 10.7506 s, 99.9 MB/s  ## only did one test here
> 
> time dd if=/dev/zero of=/mnt/testfile bs=1M count=10240 oflag=direct
> 
> 10737418240 bytes (11 GB, 10 GiB) copied, 19.158 s, 560 MB/s
> 10737418240 bytes (11 GB, 10 GiB) copied, 23.0384 s, 466 MB/s
> 10737418240 bytes (11 GB, 10 GiB) copied, 18.3788 s, 584 MB/s

Hi

This is kind of expected - the bigger the chunk is flushed at once to your 
drive - the better the performance gets.

Note - stripping is useful when you have relative 'slow' individual drive and 
you have very fast controllers capable of streaming 'parallel' IO operations.
This way you take the advantage.
If you have very fast storage - and you system can't gain any more by 
streaming in parallel - just stick with linearly joined storage....


2nd. part is - the faster the storage is - the bigger the individual strip may 
likely be needed to be better utilized - i.e. 64k can be too small - maybe use 
256K -giving you total strip size of  3 * 256K = 768K  - the there is 
alignment of writes - where optimal io size can be multiple of this number....

But you need to benchmark this - starting with 'raw' device.
Then add 'DM' layer. Benchmark again which stripe is best.
Then add filesystem on top - and see which stripe configuration again gives 
the best result with the stack.
Each layer may 'drop' some amount of speed - but it should be fairly minimal 
when the right parameters are selected....

>  
> I monitor speeds to the individual discs and the LV itself using:
> 
> iostat -xdmt 2
> 
> 
> My question(s):
> 
> When I create the LV using just one disk (linear lvm), I am seeing the same write speed results as above.  In iostat, I can see that eg. With 1MB block size in dd, it’s ~500MB/s to the dm-1 device and also ~500MB/s to the physical disk xvda.  When I create the LV as striped, I’m getting the same throughput on the dm-1 device (~500MB/s), but it is splitting that throughput evenly across all three PV’s, so about ~170MB/s to each xvd disk, totalling ~500MB/s at the LV side.  This was not my expectation.  I expected ~500MB/s to each device with striped LV, totalling ~1500MB/s on the dm-a device.  What am I missing?


Here we don't know your controller is even capable of handling parallel 
workload -  i.e. if you just you 'dd' on storage itself - no lvm/dm involved - 
are you able to  'stream'   3  'dd' commands on each drive in paralel give you 
the total bandwidth of 1500MB/s   -  you need to check this bottleneck first.

Also note - there are 2 ways how to do striping - one is  'old dm' striped 
target - and other is slightly more modern  --type raid0  striping - see 
whichever works better for you...

> 
> Second question is:
> 
> I’m testing database throughput.  I have the following setup:
> 
> DB  → 8KB block size
> XFS  → 4KB block size limited by pagesize in Linux Kernel on x86 architecture.
> LVM  →  64KB block size, and have also tested with 4KB, 8KB, 256KB, 512KB, 1MB, 4MB
> Xvd[abc]  →  4KB block size on these disks provided by the virtual host
> 
> The only part of that equation I can modify is the stripe size on the LVM.  Irrespective of which stripe size I choose, I’m getting similar results with dd (and fio for that matter) when testing different block sizes with my write tests.  In other words, it doesn’t seem to make any difference whatsoever which stripe size I choose in the LVM.  Is this a limitation cause by my virtualisation layer?  Would I have seen different (better) results if I’d used 3 physical disks connected to a physical host?
> 
> I suppose it’s normal to get low speeds with small block sizes and higher speeds with larger block sizes, but I’m still concerned that 8KB block sizes are writing to a fast disk system at only 13-20MB/s.  Is this common?
> 

Lot's of small write will never be fast - and will be likely even smaller with 
striped storage.  But you should start first figure where are the bottlenecks 
in your system.

Maybe you virtual system does have it self 'a single write pipe' - so any 
parallelization is stopped right there - so maybe you should start first with 
bare metal....


Regards

Zdenek