Re: striped LV and expected performance

I don't see where you talk about what real hardware there is outside of the=
 VM.

Are all 3 vmdk's on separate independent disks?

I have only seen adding extra controllers really matter when you have
a large number of disks per controller.  Ie 5 disks/controller works
better than 15 disks on one controller.   And this really only seemed
to matter with read io where the app had many threads (ie not direct
io).  Parallel writes with direct IO breaks down to write an io and
wait to each device...especially when the block size of the direct IO
is similar (or smaller) than the stripe size.

On Fri, Jul 18, 2025 at 2:14=E2=80=AFAM Henry, Andrew <[email protected]>=
 wrote:
>
> I=E2=80=99m trying to improve the performance of a virtual machine guest =
running RHEL 9.6.  I=E2=80=99ve assigned 3 separate SAS controllers to the =
VM, and 3 vmdk disks each connected to their own controller, so there is no=
 kernel bottleneck in the VM on the controller side when writing through to=
 each vmdk under high load.
>
> In the guest, I=E2=80=99m creating an LV as follows:
>
> lvcreate -n swap -i 3 -I 64k --type=3Dstriped -L 16G vg00
> mkfs -t xfs -b size=3D4K -d su=3D64k,sw=3D3 -f /dev/vg00/swap
>
> After mounting, I test write speeds using:
>
> time dd if=3D/dev/zero of=3D/mnt/testfile1 bs=3D8192 count=3D65536 oflag=
=3Ddirect
>
> 536870912 bytes (537 MB, 512 MiB) copied, 41.2043 s, 13.0 MB/s
> 536870912 bytes (537 MB, 512 MiB) copied, 27.3986 s, 19.6 MB/s
> 536870912 bytes (537 MB, 512 MiB) copied, 22.8359 s, 23.5 MB/s
>
> time dd if=3D/dev/zero of=3D/mnt/testfile bs=3D64k count=3D163840 oflag=
=3Ddirect
>
> 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 10.7506 s, 99.9 MB/s  ## only =
did one test here
>
> time dd if=3D/dev/zero of=3D/mnt/testfile bs=3D1M count=3D10240 oflag=3Dd=
irect
>
> 10737418240 bytes (11 GB, 10 GiB) copied, 19.158 s, 560 MB/s
> 10737418240 bytes (11 GB, 10 GiB) copied, 23.0384 s, 466 MB/s
> 10737418240 bytes (11 GB, 10 GiB) copied, 18.3788 s, 584 MB/s
>
> I monitor speeds to the individual discs and the LV itself using:
>
> iostat -xdmt 2
>
>
> My question(s):
>
> When I create the LV using just one disk (linear lvm), I am seeing the sa=
me write speed results as above.  In iostat, I can see that eg. With 1MB bl=
ock size in dd, it=E2=80=99s ~500MB/s to the dm-1 device and also ~500MB/s =
to the physical disk xvda.  When I create the LV as striped, I=E2=80=99m ge=
tting the same throughput on the dm-1 device (~500MB/s), but it is splittin=
g that throughput evenly across all three PV=E2=80=99s, so about ~170MB/s t=
o each xvd disk, totalling ~500MB/s at the LV side.  This was not my expect=
ation.  I expected ~500MB/s to each device with striped LV, totalling ~1500=
MB/s on the dm-a device.  What am I missing?
>
> Second question is:
>
> I=E2=80=99m testing database throughput.  I have the following setup:
>
> DB  =E2=86=92 8KB block size
> XFS  =E2=86=92 4KB block size limited by pagesize in Linux Kernel on x86 =
architecture.
> LVM  =E2=86=92  64KB block size, and have also tested with 4KB, 8KB, 256K=
B, 512KB, 1MB, 4MB
> Xvd[abc]  =E2=86=92  4KB block size on these disks provided by the virtua=
l host
>
> The only part of that equation I can modify is the stripe size on the LVM=
.  Irrespective of which stripe size I choose, I=E2=80=99m getting similar =
results with dd (and fio for that matter) when testing different block size=
s with my write tests.  In other words, it doesn=E2=80=99t seem to make any=
 difference whatsoever which stripe size I choose in the LVM.  Is this a li=
mitation cause by my virtualisation layer?  Would I have seen different (be=
tter) results if I=E2=80=99d used 3 physical disks connected to a physical =
host?
>
> I suppose it=E2=80=99s normal to get low speeds with small block sizes an=
d higher speeds with larger block sizes, but I=E2=80=99m still concerned th=
at 8KB block sizes are writing to a fast disk system at only 13-20MB/s.  Is=
 this common?
>
> /AH