Re: [linux-lvm] ThinPool performance problem with NVMe
Zdenek Kabelac <[email protected]>
| Newsgroups | dev.linux.lists.linux-lvm |
|---|---|
| Message-ID | <[email protected]> |
Dne 24. 06. 23 v 1:22 Anton Kulshenko napsal(a): > Hello. > > Please help me figure out what my problem is. No matter how I configure the > system, I can't get high performance, especially on writes. > > OS: Oracle Linux 8.6, 5.4.17-2136.311.6.el8uek.x86_64 > Platform: Gigabyte R282-Z94 with 2x 7702 64cores AMD EPYC and 2 TB of RAM > Disks: NVMe Samsung PM1733 7.68 TB > > What I do: > vgcreate vg1 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 > lvcreate -n thin_pool_1 -L 20T vg1 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 > /dev/nvme3n1 -i 4 -I 4 > > -i4 for striping between all disks, -I4 strip size. Also I tried 8, 16, 32... > In my setup I can't find a big difference. > stripe size needs to be aligned with some 'hw' properties. In the case of 'NVMe' where the write unit for optimal performance is usually 0.5M or more - using 4K block is basically massively destroying your performance since you generate for each large write huge amount of splits. > I only get 40k iops, while one drive at the same load easily gives 130k iops. > I have tried different block sizes, strip sizes, etc. with no result. When I > look in iostat I see the load on the disk where the metadata is: > 80 WMB/s, 12500 wrqm/s, 68 %wrqm > > I don't understand what I'm missing when configuring the system. > As mentioned by Mathew - you likely should start with some 'initial' thin-pool size - maybe sitting fully on single NVMe - possibly deploy metadata on 2nd. NVMe for better bus utilization. For striping - you would need to go with 512K units at least - then it's the question how it fits your workload... Anyway now you have way more things to experiment and benchmark and figure out what is the best on your particular hw. One more thing - increasing chunksize to 256K or 512K also may significantly raise the performance - but at the price of reduced sharing in case of taking a snapshot of a thin volume... Regards Zdenek _______________________________________________ linux-lvm mailing list [email protected] https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/