Re: [PATCH v5 4/7] fuse: {io-uring} Allow reduced number of ring queues

On 6/23/26 01:32, Joanne Koong wrote:
> zOn Fri, Jun 12, 2026 at 7:24 PM Joanne Koong <[email protected]> wrote:
>>
>> On Thu, May 28, 2026 at 3:55 PM Bernd Schubert via B4 Relay
>> <[email protected]> wrote:
>>>
>>> From: Bernd Schubert <[email protected]>
>>>
>>> Queues selection (fuse_uring_get_queue) can handle reduced number
>>> queues - using io-uring is possible now even with a single
>>> queue and entry.
>>>
>>> The FUSE_URING_REDUCED_Q flag is introduced tell fuse server that
>>> reduced queues are possible, i.e. if the flag is set, fuse server
>>> is free to reduce number queues.
>>>
>>> Notheworth is also that a fuse-io-uring is now marked as ready
>>> after the fist queue was created.
>>>
>>> Signed-off-by: Bernd Schubert <[email protected]>
>>> ---
>>>  fs/fuse/dev_uring.c       | 171 +++++++++++++++++++++++++++-------------------
>>>  fs/fuse/dev_uring_i.h     |   3 +
>>>  fs/fuse/inode.c           |   2 +-
>>>  include/uapi/linux/fuse.h |  10 ++-
>>>  4 files changed, 112 insertions(+), 74 deletions(-)
>>>
>>> diff --git a/fs/fuse/dev_uring.c b/fs/fuse/dev_uring.c
>>> index 497093384c31e729053d2f5046c9ec59461ac035..d02266b483c89d105bd6301133820697f7caba9c 100644
>>> --- a/fs/fuse/dev_uring.c
>>> +++ b/fs/fuse/dev_uring.c
>>> @@ -373,6 +397,30 @@ static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring,
>>>          * write_once and lock as the caller mostly doesn't take the lock at all
>>>          */
>>>         WRITE_ONCE(ring->queues[qid], queue);
>>> +
>>> +       /* Static mapping from cpu to per numa queues */
>>> +       node = cpu_to_node(qid);
>>> +       fuse_uring_cpu_qid_mapping(ring, qid, &ring->numa_q_map[node], node);
>>
>> Hi Bernd,
>>
>> I don't think we can assume node is within the bounds of numa_q_map.
>> numa_q_map gets allocated with num_online_nodes() # of entries but I
>> think the node returned in cpu_to_node() can exceed that if some nodes
>> were offline when we computed num_online_nodes() (eg num_online_nodes
>> = 2, thhe 2 online nodes are 0 and 3, while 1 and 2 are offline).
>>
>>> +
>>> +       /* global mapping */
>>> +       fuse_uring_cpu_qid_mapping(ring, qid, &ring->q_map, -1);
>>> +
>>> +       /*
>>> +        * Pairs with smp_load_acquire() in fuse_uring_select_queue().
>>> +        * Released before the per-numa bump below so that observing
>>> +        * numa_q_map[node].nr_queues > 0 implies q_map.nr_queues > 0.
>>> +        */
>>> +       smp_store_release(&ring->q_map.nr_queues,
>>> +                         ring->q_map.nr_queues + 1);
>>> +
>>> +       /*
>>> +        * smp_store_release, as the variable is read without fc->lock and
>>> +        * we need to avoid compiler re-ordering of updating the nr_queues
>>> +        * and setting ring->numa_queues[node].cpu_to_qid above
>>> +        */
>>> +       smp_store_release(&ring->numa_q_map[node].nr_queues,
>>> +                         ring->numa_q_map[node].nr_queues + 1);
>>> +
>>>         spin_unlock(&fch->lock);
>>>
>>>         return queue;
>>>
>>> @@ -1186,7 +1180,19 @@ static int fuse_uring_register(struct io_uring_cmd *cmd,
>>>         if (IS_ERR(ent))
>>>                 return PTR_ERR(ent);
>>>
>>> -       fuse_uring_do_register(ent, cmd, issue_flags);
>>> +       fuse_uring_prepare_cancel(cmd, issue_flags, ent);
>>> +       if (!READ_ONCE(ring->ready)) {
>>> +               WRITE_ONCE(fiq->ops, &fuse_io_uring_ops);
>>> +               WRITE_ONCE(ring->ready, true);
>>> +               wake_up_all(&fch->blocked_waitq);
>>> +       }
>>
>> I thought we had agreed at LSF that userspace would declare the number
>> of queues upfront and dispatch would be gated until all of them have
>> finished setup/registration rather than going ready when the first
>> entry in a queue gets registered? Did the plan change or am I
>> misremembering?
>>
>> I still have the same thoughts as previously [1] about it. I really
>> don't think we should allow requests to go through io-uring while
>> io-uring setup is still happening. If we want to add dynamic queue
>> addition in the future, we could always do that later through a new
> 
> Hi Bernd,
> 
> What do you think about dropping FUSE_URING_REDUCED_Q and reframing
> this series around dynamic queue addition instead? I don't mean to add
> more work to your plate, but my main reason is the uapi. REDUCED_Q is
> a narrow flag that's a subset of dynamic addition. Exposing general
> queue addition would line up with the decoupled queue-creation uapi
> the bufpool work will use, and it'd be more cohesive with the future
> feature to dynamically remove queues.
> 
> I think this series already has the bulk of the logic for dynamic
> addition anyways. The main missing piece looks like adding
> infrastructure to publish the mapping as an immutable RCU snapshot
> rather than mutating it in place, so a reader never sees a
> partially-built mapping.
> 
> What are your thoughts?

Except of libfuse not being ready to create queues on demand, the series
basically supports that in kernel. And I was also thinking to use CU to
update the mapping.

However, I'm lost about the relation of buf pools. I very much disagree
that adding buf pools has a relation to queues. Unless you want to make
it 2D, which gets complex to find the right queue. Reasons:

1) If I start a queue in libfuse with a 1MB buffer and later see that it
gets used, libfuse should add more buffers. However, I would not want it
to add queues, unless I see that that ring threads occupy all of the core.

2) Lowest latency is still achieved with one queue per cpu - especially
here it makes sense to have a very low buffer usage and to increase it
if needed.

3) At least one customer at DDN sets a very specific cpu configuration,
using more cores is strictly forbidden.


Thanks,
Bernd