Re: Bug report - Trim QD1 using io_uring is hitting unreal clat values

On Mon, Jul 6, 2026 at 1:44=E2=80=AFPM Alexa Mercado Sanchez
<[email protected]> wrote:
>
> Following your suggestions, I minimized the job and ran git bisect (good =
fio-3.37, bad fio-3.42) with an automated build+run harness. It landed clea=
nly and reproducibly on:
>
> commit 980fb7f2d9981bbfe4a3158d6293308c76a6d77a
> Author: Jens Axboe <[email protected]>
> Date:   Mon Sep 30 09:42:22 2024 -0600
>
>     engines/io_uring: add support for async TRIM
>
>     6.12 kernels and newer support async trim, which means the non-cmd pa=
th
>     no longer needs to quiesce the queue and issue a sync trim for a work=
load
>     that includes trim/discard requests.
>
>     The engine will assume the support is there, and if it gets -EINVAL w=
hen
>     trying an async trim, then it'll punt back to the using the sync trim
>     again.
>
>     Signed-off-by: Jens Axboe <[email protected]>
>
>  engines/io_uring.c  | 37 +++++++++++++++++++++++++++++++------
>  os/linux/io_uring.h |  5 ++++-
>  2 files changed, 35 insertions(+), 7 deletions(-)
> - Bisect found first bad commit
>
>
> Minimal reproducer (QD1):
>
> [trim_qd1]
> ioengine=3Dio_uring
> filename=3D/dev/nvmeXn1
> rw=3Drandtrim
> bs=3D128k
> iodepth=3D1
> direct=3D1
> time_based=3D1
> runtime=3D20
>
> Symptom: trim clat_ns.mean =E2=89=88 runtime/2 in seconds (min =E2=89=88 =
ramp_time, max =E2=89=88 ramp_time+runtime, all IOs in one histogram bin), =
while trim lat_ns.mean is correct (~195 =C2=B5s). So the completion latency=
 is being computed against an unset/zero issue_time.
>
> Note on kernels: you couldn't reproduce on 6.6, but I see it on both 5.15=
 and a RHEL10 6.12 build. I did some further debug and understood the diffe=
rences in code better. On both of mine the async discard returns -EINVAL, s=
o fio takes the punt-to-sync branch in fio_ioring_queue() (do_io_u_trim() =
=E2=86=92 FIO_Q_COMPLETED). That path never stamps io_u->issue_time, wherea=
s the normal ring path stamps it in fio_ioring_queued(). Since the engine s=
ets FIO_ASYNCIO_SETS_ISSUE_TIME, the sync-trim completion ends up with a ze=
ro issue time, which matches the observed clat. The real device latency (la=
t) is unaffected.
>

(Resending this with the HTML stripped out so that it makes it to the
mailing list)

With the help of the troubleshooting you did, I have committed a patch
that should fix the issue. Please try the tip of the master branch and
report back if there are any remaining problems.
Also, next time please include the mailing list in your reply and
avoid top posting.

Vincent