Re: [PATCH 2/2] io_uring: switch local task_work to a mpscq

On Mon, Jun 15, 2026 at 11:00 AM Jens Axboe <[email protected]> wrote:
>
> On 6/15/26 11:55 AM, Caleb Sander Mateos wrote:
> > On Fri, Jun 12, 2026 at 8:11?AM Jens Axboe <[email protected]> wrote:
> >>
> >> On 6/12/26 6:21 AM, Jens Axboe wrote:
> >>> On 6/11/26 11:24 PM, Caleb Sander Mateos wrote:
> >>>> On Thu, Jun 11, 2026 at 7:23?PM Jens Axboe <[email protected]> wrote:
> >>>>>
> >>>>> On 6/11/26 7:14 PM, Caleb Sander Mateos wrote:
> >>>>>> This is great stuff! I had also observed these hotspots on a ublk
> >>>>>> workload. Since incoming ublk requests post task work to the ublk
> >>>>>> server's io_urings and completed ublk requests post task work to the
> >>>>>> client's io_urings, there is significant cross-CPU contention on the
> >>>>>> task work queues.
> >>>>>
> >>>>> Glad you like it! Once I post v2 tomorrow, perhaps you can try and run
> >>>>> some tests with and without and see how it does for you?
> >>>>
> >>>> Haven't tested v2 yet, but v1 shows a 4% IOPS improvement on a ublk
> >>>> 4-KB read workload. The workload has 8 CPUs (unpaired hypertwins)
> >>>> running fio with io_uring submitting I/O to the ublk devices and 32
> >>>> ublk server CPUs (paired hypertwins) servicing the requests, achieving
> >>>> around 4M IOPS. Both the client and server CPUs look completely busy.
> >>>
> >>> That's a pretty nice improvement! Would be curious to hear what v2 looks
> >>> like.
> >
> > Looks the same as v1, which makes sense as both the client and server
> > are using IORING_SETUP_DEFER_TASKRUN.
>
> OK, sounds good.
>
> > I did observe fio seem to get stuck forever on one out of the 85 or so
> > runs, though. I'm a little concerned there might be a missing wakeup.
> > It was using the default iodepth_batch_complete_min=1 (waiting for
> > io_uring completions) and IORING_SETUP_DEFER_TASKRUN.
>
> There's a bug in v2 where it can get missed, the in-tree code should
> have that fixed. It was the atomic_dec_and_test() and
> atomic_try_cmpxchg() in io_req_local_work_add() racing.

Great, glad it's already fixed.

>
> >> And here's some more stuff on top you might find interesting. For a
> >> 6 NVMe drive test, it drops my task work usage from top-of-profiles
> >> to ~2%.
> >>
> >> https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux.git/log/?h=io_uring-tw-mpscq-batch
> >>
> >> The patches sit on top of the io_uring-tw-mpscq branch.
> >
> > Yeah there are some interesting ideas there.
> >
> > The ublk server isn't using UBLK_F_BATCH_IO, so it unfortunately
> > wouldn't benefit from the task work batching for
> > UBLK_U_IO_COMMIT_IO_CMDS. The batching would probably need to be
> > scoped to the whole io_submit_sqes() in order to allow batching across
> > the multiple UBLK_U_IO_COMMIT_AND_FETCH_REQ commands. I'm also not
> > sure about the claim that __ublk_walk_cmd_buf() won't sleep;
> > ublk_batch_commit_io() calls io_buffer_unregister_bvec(), which could
> > sleep depending on the io_uring issue_flags.
>
> It's very much just a POC series of things... I suspect to get the
> benefit of it, we'd need a bit of refactoring and reworking first. It
> was more to get the idea out/across, not going anywhere right now.
>
> > The NVMe passthrough task work batching could definitely reduce
> > contention on the task work queue. I'll run a perf test.
>
> Thanks!

I tried it out and the 4K read throughput looks a little lower
actually (about a 1.7% improvement over the baseline vs. 2.6% with
just v2). Since the workload is loading 24 NVMe devices, I suspect
there just isn't much to be gained from batching completions within a
single NVMe queue.

I do see the time in __ioreq_task_work_add() on the ublk server went
down from 1.04% to 0.43%, though there's now 0.67% in the newly added
io_local_work_flush_batch().

The time in update_io_ticks() (largely from blk_account_io_done() on
NVMe completions) increased from 0.18% to 1.11%, though I'm a little
surprised that would be caused by these patches.

Best,
Caleb