[PATCH 00/16] NTB: Add direct TX/RX using PCI endpoint DMA

Hi,

This series lets ntb_transport transfer packets directly from client TX
buffers to peer RX buffers using PCI endpoint DMA, avoiding the
memory-window copy path. It raises throughput significantly.

My previous attempt at the same goal was:

  [RFC PATCH v4 00/38] NTB transport backed by PCI EP embedded DMA
  https://lore.kernel.org/r/[email protected]/

Since that RFC, prerequisite work has entered mainline in a different
form. Unlike that RFC, this series extends ntb_transport itself and can
fall back to the existing memory-window copy path at runtime.

Some work needed to run this series on my R-Car S4 testbed is still under
review; see Dependencies below.

The series contains:

  * Patches 1-5   : small fixes and prerequisites
  * Patches 6-14  : the main implementation for direct TX/RX
  * Patches 15-16 : optional features


Summary
=======

ntb_transport currently copies each packet through a fixed slot in a
memory window. This series adds an optional path where the receiver
publishes client-buffer DMA addresses and the sender's DMA writes
packets directly to them. The existing copy path remains available, and
direct RX and TX are negotiated independently.

Bidirectional direct TX/RX requires a multi-function PCI endpoint. vNTB
uses one function, while the PCI DMA EPF on another exposes the endpoint
DMA controller to the host. Each sender needs an ordered channel that can
transfer from mapped system memory to peer-published DMA addresses. The
tested setup used dw-edma with the PCI DMA EPF on PF0 and vNTB on PF1.


Design
======

Session
-------

ntb_transport automatically reconnects a logical QP while the physical
NTB link and its shared MW remain up. One peer may therefore start the
next direct-DMA "session" while the other is still finishing the
previous teardown. A fresh session ID distinguishes the two instances,
and direct DMA remains disabled until both peers have acknowledged it.
Otherwise, stale RX addresses from the previous instance could be reused
after their mappings have been released.

Resetting the physical NTB link on every QP close would avoid this
overlap, but would also tear down all QPs for ordinary operations such
as ntb_netdev close/open, MTU changes, or queue reconfiguration. This
series keeps the existing automatic QP reconnect behavior instead.

Quiesce
-------

A published RX address must stay mapped until the sender can no longer
write to it. Closing the local QP alone does not prove this. During
teardown, each QP publishes its final issued TX boundary, then writes a
quiesce marker to tell the peer that this boundary is final and no more
TX will be issued for the session. The CPU-written marker can arrive
before earlier DMA writes when the two paths use different PCIe ordering
domains. The peer therefore waits until it has consumed completion words
through that boundary before acknowledging quiesce and releasing the RX
mappings. Local TX mappings remain owned until DMA completes or the
channel has been stopped and synchronized.

Completion ordering
-------------------

The payload and its completion word are submitted to the same ordered
DMA channel. A separate CPU MMIO completion would not provide that
ordering: its posted write could pass an earlier DMA write. The MSI or
doorbell sent after the DMA callback can still reach the peer before the
completion word is visible. A read-back from the completion destination
would close that window, but would add a non-posted PCIe round trip to
every packet. It hurt throughput in testing. Notifications are therefore
hints, while the completion word decides when an RX buffer can be
reused. Optional polling lets the receiver find a completion that
arrives after its notification.

Per-direction negotiation
-------------------------

A side may be able to map direct RX buffers even when it cannot reserve
all TX DMA channels. RX and TX are therefore negotiated separately,
allowing direct DMA in one direction while the other keeps using the
copy path.

Protocol compatibility
----------------------

The extension keeps protocol version 4 and the original QP layout for
peers that do not advertise direct DMA. Version 4 has been in use for
about 11 years.


Dependencies
============

The multi-function endpoint setup described above requires:

  1. [PATCH v6 0/6] PCI: endpoint: Expose endpoint DMA resources (part 2/3)
     (https://lore.kernel.org/r/[email protected]/)

  2. [PATCH v5 0/3] PCI: endpoint: Add PCI DMA endpoint function (part 3/3)
     (https://lore.kernel.org/r/[email protected]/)

  3. [PATCH 0/3] PCI: endpoint: Support vNTB as a non-first EPF
     (https://lore.kernel.org/r/[email protected]/)

Series 1 and 2 will be combined in the next revision after the part 2
review settles, per Frank's feedback.


Performance improvement
=======================

The test used two R-Car S4 Spider boards connected with an OCuLink cable,
one as EP and the other as RC. The link was PCIe Gen4 x2 with DWC PCIe
controller IP v5.20 and eDMA (not HDMA). The workloads were:

- UDP: sudo iperf3 -ub0 -c $peer -l 65507 -w 512M -P 4 -t 10
- TCP: sudo iperf3 -Z -c $peer -l 65507 -P 8 -t 10

The results were:

  (unit: Gbps) (UL=EP->RC, DL=RC->EP)

          UL UDP  DL UDP   UL TCP  DL TCP
  ------- ------  -------  ------  ------
  Before  ~0.6     ~0.6     ~0.6     ~0.6
  After   ~19.5    ~17.3    ~12.3    ~10.8

Reaching the above throughput on R-Car S4 also requires:

- [PATCH 00/11] PCI/NTB: endpoint: packed vNTB memory windows
  (https://lore.kernel.org/r/[email protected]/)
- [PATCH v4 00/24] dmaengine: dw-edma: Support dynamic LL appends
  (https://lore.kernel.org/r/[email protected]/)

The test setup also used two unsubmitted platform changes: adding the PCIe
controller to the IPMMU allowlist and capping the EP DMA MRRS at 128 bytes.

The "After" result used:
- packed_mws=4 and `ethtool -L eth0 combined 4`
- `modprobe ntb_transport use_direct_dma=1 direct_dma_func=0 \
    direct_dma_ring_entries=256 direct_dma_poll=1`

Best regards,
Koichiro


Koichiro Den (16):
  NTB: ntb_transport: Abort link setup on QP MW allocation failure
  NTB: ntb_transport: Reject oversized TX buffers
  NTB: ntb_transport: Start TX offload thread after queue setup
  NTB: ntb_transport: Stop QP work before freeing a queue
  NTB: ntb_transport: Run RX processing on system workqueue
  NTB: ntb_transport: Define direct-DMA shared state
  NTB: ntb_transport: Negotiate direct-DMA queue layout
  NTB: ntb_transport: Add opt-in direct-DMA channel reservation
  NTB: ntb_transport: Allocate direct-DMA queue state
  NTB: ntb_transport: Implement direct-DMA QP session handshake
  NTB: ntb_transport: Implement direct-DMA RX buffer publication
  NTB: ntb_transport: Implement direct-DMA TX submission
  NTB: ntb_transport: Implement safe direct-DMA teardown
  NTB: ntb_transport: Enable direct-DMA queues
  NTB: ntb_transport: Report the direct-DMA payload limit
  NTB: ntb_transport: Add optional polling for direct-DMA RX

 drivers/net/ntb_netdev.c      |    4 +-
 drivers/ntb/ntb_transport.c   | 1523 +++++++++++++++++++++++++++++++--
 include/linux/ntb_transport.h |    1 +
 3 files changed, 1456 insertions(+), 72 deletions(-)

-- 
2.51.0