[PATCH v6 0/6] fuse: add io-uring buffer pools and zero-copy

This series adds fuse io-uring buffer pools and zero-copy.

Prior to this series, ents and buffers are tightly coupled where each entry
has its own dedicated payload buffer, requiring N buffers for N entries where
each buffer must be large enough to accomodate the maximum payload size. This
is suboptimal as most request types require vastly less bytes than the maximum
payload size and some requests do not require payload buffers at all.

This series allows servers to pass in a buffer pool (a contiguous chunk of
memory) that the kernel will use as it wishes for servicing ents/requests.
This decoupling reduces the memory usage requirements needed to use
fuse-io-uring and lets the kernel do any optimizations for assigning payload 
memory to requests.

This series additionally adds zero copy to fuse io-uring. The server can
directly access client pages or page cache folios without copying data through
an intermediary buffer. This requires CAP_SYS_ADMIN privileges and using
buffer pools. The zero copy patch has a dependency on io-uring registered
bvec changes in [1].

This series is on top of commit 7d87a5a284bb and on top of the io-uring bvec
changes and the changes from the series in [2] applied.

The throughput improvements from registered buffers and zero-copy depends on
how much of the server's per-request latency is spent on data copying vs
backing I/O. When backing I/O dominates, the saved memcpy is a negligible
fraction of overall latency. Please also note that for the server to
read/write into the zero-copied pages, the read/write must go through io-uring
as an IORING_OP_READ_FIXED / IORING_OP_WRITE_FIXED operation.

The throughput improvement from zero-copy depends on how much of the
per-request latency is spent on data copying vs backing I/O. The gain
comes from eliminating the payload-buffer memcpy,  but accessing the
zero-copied pages requires the server to issue the read/write as an
IORING_OP_READ/WRITE_FIXED operation. The benefit is largest when the
mempcy is a meaningful fraction of per-request latency while backing i/o
is still noticable enough that the extra io-uring op's overhead doesn't
dominate.

Benchmarked with passthrough_hp (--nopassthrough, q_depth=8) on a
2-socket Intel Xeon Gold 6138 (40 cores / 80 threads), using fio (sync
engine, bs=1M, O_DIRECT, numjobs=2, 30s run + 10s ramp, 3 runs) where
direct-I/O throughput is against a RAM-backed (tmpfs) source (backing
I/O is not the bottleneck):

		baseline   registered-buf   zero-copy   (zc vs base)
direct read     ~5.1 GB/s  ~5.4 GB/s        ~8.9 GB/s   (+75%)
direct write    ~3.4 GB/s  ~4.8 GB/s        ~5.1 GB/s   (+50%)

On a device-bound NVMe (~2 GB/s reads) the read gain shrinks to ~10-16% (no
measurable gains for writes), as backing I/O rather than the eliminated copy
dominates latency. The benefit overall scales with how much of the per-request
latency is the data copy versus backing I/O.

The benchmark script and results can be found in [3]. The libfuse changes can
be found in [4]. To test the server, run:
sudo ~/libfuse/build/example/passthrough_hp ~/src ~/mounts/tmp --nopassthrough
-o io_uring_zero_copy -o io_uring_q_depth=8
Once this series is merged, the libfuse changes will be tidied up and
submitted upstream.

Thanks,
Joanne 

[1] https://lore.kernel.org/io-uring/[email protected]/T/#t
[2] https://lore.kernel.org/fuse-devel/[email protected]/
[3] https://github.com/joannekoong/linux/commits/fuse_zero_copy_benchmarks/
[4] https://github.com/joannekoong/libfuse/commits/zero_copy_v5

Changelog
---------
v5: https://lore.kernel.org/fuse-devel/[email protected]/
v5 -> v6:
* Remove WARN_ON tag, add missing barrier (Sashiko)
* Bring back some documentation from v4, rename uapi doc (Amir)

v4: https://lore.kernel.org/fuse-devel/[email protected]/
v4 -> v5:
* Address Miklos's feedback (separate uring cmd for adding bufpool, pass only
  bufpool addr and len, pass back bufpool offset instead of buf id, etc)
* Make zero-copy opt-in on file open
* Do any zeroing for short zero-copy reads instead of accidentally skipping that
* Drop Baokun and Jeff's reviewed-bys since the commits had modifications
* Run more rigorous benchmarks, on bare-metal machine

v3: https://lore.kernel.org/fuse-devel/[email protected]/
v3 -> v4:
* Add reviewed-bys
* Fix documentation typo, add paragraph about aborts to zero-copy commit
  message, undo unnecessary padding[6] change, add FUSE_HAS_URING_BUFPOOL
  advertisement

v2: https://lore.kernel.org/linux-fsdevel/[email protected]/
v2 -> v3:
* Rework the uapis to be more ergonomic. Use io-uring registered buffers
  infrastructure instead of doing pinning logic in fuse. Get rid of header
  pinning as that makes no real perf difference
* Rename from "buffer ring" to buffer pool. Logic is the same, just different
  naming

v1: https://lore.kernel.org/linux-fsdevel/[email protected]/
v1 -> v2:
* Drop kernel managed buffers from io-uring infrastructure and move it to fuse
* Add visual diagrams and more documentatoin to commit messages and
  documentation patch

Joanne Koong (6):
  fuse: decouple fuse_ring creation from ent registration
  fuse: add FUSE_IO_URING_CMD_ADD_QUEUE
  fuse: add io-uring buffer pools
  fuse: support registered buffer pools in io-uring
  fuse: add zero-copy over io-uring
  docs: fuse: document io-uring buffer pool and zero-copy uapi

 .../filesystems/fuse/fuse-io-uring.rst        |  36 +-
 Documentation/filesystems/fuse/index.rst      |   1 +
 .../fuse/uapi/fuse-uapi-io-uring.rst          | 129 ++++
 fs/fuse/args.h                                |   2 +
 fs/fuse/dev.c                                 |  27 +-
 fs/fuse/dev.h                                 |   2 +-
 fs/fuse/dev_uring.c                           | 577 +++++++++++++++---
 fs/fuse/dev_uring_i.h                         |  56 +-
 fs/fuse/file.c                                |   2 +
 fs/fuse/fuse_dev_i.h                          |   1 +
 fs/fuse/inode.c                               |   6 +-
 include/uapi/linux/fuse.h                     |  66 +-
 12 files changed, 821 insertions(+), 84 deletions(-)
 create mode 100644 Documentation/filesystems/fuse/uapi/fuse-uapi-io-uring.rst

-- 
2.52.0