Poor Linux sequential read performance

Hello,

sorry if posting something that has been already said but I am new on 
this list.

My computer is ASUS NUC 15 Pro+ with 128 GB RAM, PCIe gen 5 bus running 
two attached NVMe drives - a 2TB PCIe 4 WD_BLACK SN770M 2TB (running the 
testing OS), the other PCIe 5 WD_BLACK SN8100 4000GB (the tested disk), 
each occupying four PCIe lanes.

I am running Ubuntu 24.04.3 with the 6.14 kernel. My main interest (and 
now concern) is the NVMe sequential read speed of large data.

I tested with the following command, taking the Seq1M-Q8T1-Read job into 
consideration for now:
fio --loops=5 --size=10Gi --filename="mnt/disktest.tmp" --stonewall 
--ioengine=libaio --direct=1 --group_reporting --time_based=1 --runtime=10 \
         --name=Seq1M-Q8T1-Read --bs=1Mi --iodepth=8 --numjobs=1 --rw=read \
         --name=Seq128K-Q32T1-Read --bs=128Ki --iodepth=32 --numjobs=1 
--rw=read \
         --name=Rnd4K-Q32T16-Read --bs=4Ki --iodepth=32 --numjobs=16 
--rw=randread \
         --name=Rnd4K-Q1T1-Read --bs=4Ki --iodepth=1 --numjobs=1 
--rw=randread \
         --name=Seq1M-Q8T1-Write --bs=1Mi --iodepth=8 --numjobs=1 
--rw=write \
         --name=Seq128K-Q32T1-Write --bs=128Ki --iodepth=32 --numjobs=1 
--rw=write \
         --name=Rnd4K-Q32T16-Write --bs=4Ki --iodepth=32 --numjobs=16 
--rw=randwrite \
         --name=Rnd4K-Q1T1-Write --bs=4Ki --iodepth=1 --numjobs=1 
--rw=randwrite

Although the 2TB PCIe4x4 NVMe disk scored 5 GB/s, the 4TB PCIe5x4 scored 
just 8.2 GB/s (below the PCIe 5 specification where current_link_width = 
4 and current_link_speed is 32 GT/s per udevadm report, and the disk 
should reach 15 GB/s).

Why I am worried about this result? I have made this exact test under 
Windows, exchanging systems on the same computer and gen 4 disk, first 
in WSL with the same Ubuntu distro with fio 3.36, then also with the 
native Windows fio 3.41 binary, performing the same benchmark on a 
mounted drive. Although the PCIe 4x4 disk scored similarly, the PCIe 5x4 
disk scored stunning 15.4 GB/s in both cases! The WSL kernel was the 6.6 
kernel supplied by Microsoft. I understand that the WSL disk access is 
virtualized and this should brake the benchmark, but it is greatly 
outperforming the native Linux distro. I am interested in reasons and 
ways to remedy that issue, or tell why it can't be remedied at worst. 
Thanks for your help.


Just what I've tried, with results unchanged or worse (however, I was 
unable to change interrupt coalescing):

PCIe Gen 5 Storage Array Optimization: Overcoming the Obstacles to Get 
the Best Performance | Blog | Xinnor 
<https://xinnor.io/blog/pcie-gen-5-storage-array-optimization-overcoming-the-obstacles-to-maximize-performance/>


-- 

Best regards

	*Petr Řehák*
/specialista sw Dolphin, programátor/
E-mail: [email protected] 	*Adaptech s.r.o.*
www.adaptech.cz <https://www.adaptech.cz>
Telefon: +420 605 321 321
E-mail: [email protected] 	*Sledujte nás na FB:*
Facebook <https://www.facebook.com/pages/Adaptech/1581295688751091>


_DŮLEŽITÉ SDĚLENÍ_: Tato zpráva elektronické komunikace včetně jakýchkoliv k ní připojených souborů je důvěrná a určená výhradně k použití osobě nebo subjektu, kterému byla adresována. V případě, že jste dostali tuto zprávu omylem, vymažte ji z vašeho systému. Žádným způsobem neužívejte a nesdílejte informace v této zprávě a informujte [email protected]. Jakékoliv názory nebo vyjádření v této zprávě jsou názory a vyjádření odesilatele a nemusí se shodovat s vyjádřeními Adaptech s.r.o.

_IMPORTANT INFORMATION_: This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please delete it from your system, do not use or disclose the information in any way and notify [email protected]. Any views or opinions expressed in this e-mail are those of the sender and do not necessarily coincide with those of Adaptech s.r.o.