Re: Bug report - Trim QD1 using io_uring is hitting unreal clat values

On Thu, Jul 2, 2026 at 5:35=E2=80=AFPM Alexa Mercado Sanchez
<[email protected]> wrote:
>
>
> Description
>
> When performing a pure trim QD1 RND/SEQ operation using io_uring, clat/cl=
at_hist/clat_ns entry in summary log seems faulty. I was able to reproduce =
this issue with different drives on different environments with different b=
lock sizes and in concurrency with another workload. If I switch the ioengi=
ne to libaio, I obtain the expected latencies in ms, with io_uring we are h=
itting latencies of 17 seconds across all the QoS buckets. I also tested on=
 older fio versions, 3.37 is not showing this behavior. The files I append =
here are for both 3.42 (filenames starting with 342) latest branch and 3.37=
 (filenames starting with 337). All logs appended here were tested on the s=
ame drive, same system. By opening the 342_clat_hist file appended, I reali=
zed the histogram is just parsing a single massive data point at the last c=
olumn of the histogram entries. I haven't seen this behavior in other opera=
tions other than trims.
>
>
> Environment
>
> Reproduced issue on two different environments
>
> -    OS: rocky 8.5
>
> -    Kernel: 5.15.24
>
>
> -    OS: rocky 10.1
>
> -    Kernel: 6.12.0-124.56.1.el10_1.x86_64
>
>
> fio version
>
> 3.42
>
>
> Reproduction steps
>
> 1. Drive sequentially filled to run trim operations
>
> 2. Run the appended trim_rnd_qd_1_128k_1w_itr6.fio
>
>
> Thank you for your attention to this matter,
>
> Alexa Mercado
> SSD Power and Performance Lead Software Engineer

I did some quick experiments on platforms with kernel 6.6 and 7.0 and
could not reproduce this issue. Please do the following:

1. Simplify your job to the minimum set of options required to trigger the =
issue
2. Since you have a build that works well and one that manifests this
issue, please do some more troubleshooting to narrow down the source
of the issue. Try versions between 3.37 and 3.42. Even better use git
bisect with multiple rounds of testing to pinpoint the exact commit
that triggers this issue.

Starting with kernel 6.12 the io_uring interface gained the ability to
send async trim operations. Previously it was only possible to use the
BLKDISCARD IOCTL for trim operations, so it is possible that something
is going wrong with how Fio is handling this, but more troubleshooting
is needed.

Vincent