Re: [PATCH 00/15] PCI: endpoint: Remote DMA support via vNTB
Koichiro Den <[email protected]> Tue, 17 Mar 2026 11:44:20 +0900
| Newsgroups | dev.linux.lists.ntb,org.kernel.vger.dmaengine,org.kernel.vger.linux-doc,org.kernel.vger.linux-kernel,org.kernel.vger.linux-pci |
|---|---|
| Message-ID | <sn67hi7kljh7cgmgodatb3naz2astlaklqfobdbxyyzgoohxqb@4nnetbhqwba4> |
On Fri, Mar 13, 2026 at 01:49:50AM +0900, Koichiro Den wrote: > Hi, > > This series lets an endpoint-integrated DMA engine be consumed on the RC > side through vNTB. > > The initial target is DesignWare endpoint eDMA. pci-epf-vntb exports a > versioned DMA locator plus the minimum peer-visible resources, > ntb_hw_epf parses that locator and instantiates an auxiliary device after > LINK_UP, and dw-edma-aux binds to that child to expose a DMA engine > provider on the RC side. ntb_ep_dma is included both as the first > consumer and as a simple bring-up test. > > > Background > ========== > > I previously posted a broader RFC series: > > https://lore.kernel.org/all/[email protected]/ > > This series is not a direct continuation of that RFC. Its scope is > narrower, and the approach has changed substantially. > > That RFC had two architectural issues: > > 1. dw-edma-specific logic lived under drivers/ntb/hw/, even though > the exported DMA engine is not related to any NTB hardware. > 2. Remote-use channel delegation relied on vendor-specific peripheral > configuration. > > This series builds on the recently discussed pci_epc_aux_resource work > (see "Dependency" 3 below) and addresses both issues by: > - introducing vendor-neutral DMA-channel delegation in the PCI EPC > layer via pci_epc_delegate_dma_channels() and > - making vNTB and ntb_hw_epf aware of the remote DMA resource. > > On the EP side, pci-epf-vntb describes the exported DMA resources as > part of the vNTB BAR layout. On the RC side, ntb_hw_epf detects that > export and registers an auxiliary child device. A vendor-specific > frontend can then bind to that child device and reconstruct the remote > DMA provider. This series includes such a frontend for DesignWare eDMA > in dw-edma-aux. > > > Architecture > ============ > > EP kernel > pci-epf-vntb > - exports the usual vNTB control/db/MW resources > - optionally exports a versioned DMA slice > > RC kernel > ntb_hw_epf > - parses the control layout > - instantiates an auxiliary child for the exported DMA ABI > dw-edma-aux > - binds to that child and registers a DMAEngine provider > > > Series layout > ============= > > 01-05 prepare dw-edma and auxiliary-resource metadata > 06-10 export delegated controller-owned DMA resources through vNTB > 11-13 discover the exported DMA instance on the host and bind > dw-edma-aux > 14 adds ntb_ep_dma as the first consumer / smoke test > 15 documents the model and the configfs layout > > I did not split the infrastructure patches (01-13) away from its > consumer (14). The series is meant to be reviewed as one feature: > producer, discovery, consumer, and test coverage. Please let me add a small clarification to (1) help position this series relative to the existing pci-epf-test / pci_endpoint_test framework, and (2) clarify how this differs from the earlier RFCv4 series, as I may not have explained it clearly enough before, so I added a small ASCII diagram below. First, regarding the relation to pci-epf-test / pci_endpoint_test The existing pci-epf-test READ/WRITE tests already demonstrate DMA-assisted transfers between EP and RC buffers, and when private EPC-local DMA channels are used, this also exercises Endpoint-integrated DMA. What this series tries to demonstrate is a slightly different layer. In pci-epf-test, the DMA capability is hidden behind a test-specific command path where the RC asks pci-epf-test to trigger a predefined operation. In this series, the idea is instead to export EP-integrated DMA resources through pci-epf-vntb, let ntb_hw_epf discover them, and reconstruct a normal DMAEngine provider on the RC side. From the RC side, a consumer can obtain a local dma_chan (via the standard DMAEngine API, e.g. dma_request_channel()) and drive the transfer directly, while the actual data movement is performed by the remote EP-integrated DMA engine. This is why the test introduced here is ntb_ep_dma. The goal is not primarily to prove that the EP can move data, but to exercise the full NTB path: resource export in pci-epf-vntb, discovery in ntb_hw_epf, DMA provider instantiation on the RC side, and the first consumer use through a normal DMAEngine client. In that sense, this can be seen as a natural extension of what the READ test already demonstrated. The plumbing introduced here makes it possible for the RC side to: - use the standard dma_chan abstraction for remote DMA usage - keep the user-side implementation simple and clean - avoid wake-ups on the EP side to trigger transfers (better latency/overhead) Second, regarding the difference from the earlier RFCv4 series: https://lore.kernel.org/all/[email protected]/ Compared to that RFCv4 (which had a much broader scope), this series treats EPC-integrated DMA as a first-class resource in the vNTB <-> ntb_hw_epf path, similar to how doorbells are handled today, and makes the number of delegated channels configurable via vNTB configfs. The main challenge is how to define a vendor-neutral "DMA ABI v1" (see the ASCII diagram below) that can carry the necessary information for remote DMA operation over BAR mappings. See struct pci_ep_dma_hdr_v1 defined in Patch 9/15: https://lore.kernel.org/linux-pci/[email protected]/ The advantage of this approach is that it avoids pulling EPC-specific DMA details into the upper layer (e.g. NTB subsystem), unlike the earlier RFCv4 which pulled EPC-/vendor-specific DMA handling into the NTB layer (ref. the proposed drivers/ntb/hw/edma/ directory in: https://lore.kernel.org/all/[email protected]/) Below is a rough high-level sketch of this series design: EP with EPC-integrated DMA Any Host +-------------+ +--------------+ | vNTB | | ntb_hw_epf | | | | | | +-----+ | | +--------+ | | | DMA |----(DMA ABI)---->| decode | | | +-----+ | v1 | +--------+ | +--------^----+ +------:-------+ : : Delegated via : : Instantiated by EPC API : : vendor-specific (vNTB configfs : : aux driver knobs added) : : e.g. dw-edma-aux. : v +------------+ +-----------+ +-----------+ +-------------+ | dma_device | | Real | | Aux | | dma_device | | (EPC plat) |-----| DMA chans | | DMA chans |-----| (ntb_hw_epf)| +------------+ +-----------+ +-----------+ +-------------+ delegated ----. ,---- delegated (unusable) \ \ / / (usable) ++ ++ ++ ++ ++ ++ ++ ++ || || :: :: :: :: || || || || :: :: :: :: || || || || :: :: :: :: || || || || :: :: :: :: || || || || :: :: :: :: || || || || :: :: :: :: || || ++ ++ ++ ++ ++ ++ ++ ++ WR WR RD RD WR WR RD RD '---' '---' : : v v Upper layer NTB consumers can use these channels for efficient EP<->RC transfer via standard DMAEngine API (assuming each side uses a local dma_chan for TX submission) Any feedback would be greatly appreciated. Best regards, Koichiro > > > Test > ==== > > Tested on R-Car S4 Spider with the dependency below. > > 1. Configure and start pci_epf_vntb with DMA export enabled. > > The actual commands I used for testing: > > # modprobe pci_epf_vntb > # cd /sys/kernel/config/pci_ep/ > # mkdir functions/pci_epf_vntb/func1 > # echo 0x1912 > functions/pci_epf_vntb/func1/vendorid > # echo 0x0030 > functions/pci_epf_vntb/func1/deviceid > # echo 32 > functions/pci_epf_vntb/func1/msi_interrupts > # echo 16 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/db_count > # echo 128 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/spad_count > # echo 1 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/num_mws > # echo 0xF9000 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/mw1 > # echo 0xF9000 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/dma_offset > # echo 4 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/dma_num_chans > # echo 0x1912 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/vntb_vid > # echo 0x0030 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/vntb_pid > # echo 0x10 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/vbus_number > # echo 0 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/ctrl_bar > # echo 2 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/mw1_bar > # echo 2 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/dma_bar > # echo 4 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/db_bar > # ln -s controllers/e65d0000.pcie-ep functions/pci_epf_vntb/func1/primary/ > # echo 1 > controllers/e65d0000.pcie-ep/start > > 2. Boot or rescan the RC side and let ntb_hw_epf probe. > > 3. Load ntb_ep_dma on both EP and RC. > > 4. On the RC side, run the test as follows: > > # cat /sys/kernel/debug/ntb_ep_dma/0000:01:00.0/ready > # echo 1 > /sys/kernel/debug/ntb_ep_dma/0000:01:00.0/run > # cat /sys/kernel/debug/ntb_ep_dma/0000:01:00.0/result > > last_status: 0 > last_len: 4096 > local_buf_dma: 0xfffff000 > local_buf_size: 4096 > peer_ready: 1 > peer_state: pass # <----(*) > peer_dma: 0x4e11e000 > peer_size: 4096 > peer_seq: 1 > peer_xfer_len: 4096 > link_up: 1 > > (*) The peer reports "pass" after the transfer completes successfully, > > > Kernel base > =========== > > pci.git endpoint: > Commit 0b74f7d72399 ("PCI: endpoint: Propagate error from pci_epf_create()") > > > Dependency > ========== > > 1. [PATCH v4 00/10] PCI: endpoint: Differentiate between disabled and reserved BARs > https://lore.kernel.org/linux-pci/[email protected]/ > https://patchwork.kernel.org/project/linux-pci/list/?series=1065666 > > 2. [PATCH 0/2] dmaengine: dw-edma: Interrupt-emulation doorbell support > https://lore.kernel.org/dmaengine/[email protected]/ > https://patchwork.kernel.org/project/linux-dmaengine/list/?series=1054298 > Note: already landed in dmaengine/next. > > 3. [PATCH v10 0/7] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback > https://lore.kernel.org/all/[email protected]/ > https://patchwork.kernel.org/project/linux-pci/list/?series=1059820 > > 4. [PATCH v2 0/3] NTB: Allow drivers to provide DMA mapping device > https://lore.kernel.org/linux-pci/[email protected]/ > https://patchwork.kernel.org/project/linux-pci/list/?series=1062308 > Note: this series uses ntb_get_dma_dev() API. > > 5. [PATCH v2 00/10] PCI: endpoint: pci-epf-vntb: Document legacy MSI doorbell offset > https://lore.kernel.org/linux-pci/[email protected] > https://patchwork.kernel.org/project/linux-pci/list/?series=1058871 > Note: v2 title was incorrect. See my reply to the cover letter. > > Additionally, for ntb_ep_dma test to pass on R-Car S4 Spider: > > 6. [PATCH v2] PCI: dwc: rcar-gen4-ep: Mark BAR0 and BAR2 as Resizable BARs > https://lore.kernel.org/linux-pci/[email protected]/ > https://patchwork.kernel.org/project/linux-pci/list/?series=1052780 > Note: already landed in pci/next. > > 7. [PATCH v2] PCI: dwc: rcar-gen4: Use 4K EPC BAR alignment > https://lore.kernel.org/linux-pci/[email protected]/ > https://patchwork.kernel.org/project/linux-pci/list/?series=1062031 > > > Merge plan > ========== > > This series touches three areas: > > - PCI endpoint core and pci-epf-vntb > - DesignWare eDMA (dw-edma) > - an NTB test client > > The series intentionally keeps the infrastructure changes together with > their first consumer, the ntb_ep_dma test client. Splitting them further > would leave the infrastructure patches without a consumer, so the > patches are kept together as a single series. > > Mani is a maintainer for both the PCI EP and dw-edma. My initial thought > was therefore to collect acks from the relevant subsystems (PCI EP, > dw-edma, and NTB) and have the series applied through the PCI EP tree. > > However, I am of course open to any suggestions regarding the preferred > merge path or series split if maintainers think another approach would > be more appropriate. > > > Best regards, > Koichiro > > > Koichiro Den (15): > dmaengine: dw-edma: Cache DMA channel IDs in dw_edma_chip > PCI: endpoint: Add DMA channel metadata to pci_epc_aux_resource > PCI: dwc: ep: Report DMA channel metadata for aux resources > dmaengine: dw-edma: Add per-channel interrupt routing control > dmaengine: dw-edma: Compose MSI messages from allocated IRQs > PCI: endpoint: pci-epf-vntb: Fold MW runtime state into a struct > PCI: endpoint: Add EPC DMA channel delegation hooks > PCI: dwc: ep: Delegate exported eDMA channels through EPC ops > PCI: endpoint: Add pci-ep-dma helper for exported DMA ABI v1 > PCI: endpoint: pci-epf-vntb: Support DMA export and shared BAR layouts > NTB: hw: epf: Parse control-layout version and DMA locator > NTB: hw: epf: Enumerate auxiliary child for DMA ABI v1 > dmaengine: dw-edma: Add auxiliary-bus frontend for exported eDMA > NTB: Add ntb_ep_dma test client > Documentation: PCI: endpoint: Add vNTB DMA export HOWTO > > Documentation/PCI/endpoint/index.rst | 1 + > .../PCI/endpoint/pci-vntb-dma-howto.rst | 83 ++ > drivers/dma/dw-edma/Kconfig | 11 + > drivers/dma/dw-edma/Makefile | 1 + > drivers/dma/dw-edma/dw-edma-aux.c | 297 +++++++ > drivers/dma/dw-edma/dw-edma-core.c | 101 ++- > drivers/dma/dw-edma/dw-edma-core.h | 13 + > drivers/dma/dw-edma/dw-edma-v0-core.c | 26 +- > drivers/ntb/hw/epf/Kconfig | 1 + > drivers/ntb/hw/epf/ntb_hw_epf.c | 199 ++++- > drivers/ntb/test/Kconfig | 10 + > drivers/ntb/test/Makefile | 1 + > drivers/ntb/test/ntb_ep_dma.c | 695 +++++++++++++++ > .../pci/controller/dwc/pcie-designware-ep.c | 196 +++++ > drivers/pci/controller/dwc/pcie-designware.h | 11 + > drivers/pci/endpoint/Makefile | 2 +- > drivers/pci/endpoint/functions/pci-epf-vntb.c | 794 ++++++++++++++++-- > drivers/pci/endpoint/pci-ep-dma.c | 342 ++++++++ > drivers/pci/endpoint/pci-epc-core.c | 84 ++ > include/linux/dma/edma.h | 42 + > include/linux/pci-ep-dma.h | 130 +++ > include/linux/pci-epc.h | 31 + > 22 files changed, 2981 insertions(+), 90 deletions(-) > create mode 100644 Documentation/PCI/endpoint/pci-vntb-dma-howto.rst > create mode 100644 drivers/dma/dw-edma/dw-edma-aux.c > create mode 100644 drivers/ntb/test/ntb_ep_dma.c > create mode 100644 drivers/pci/endpoint/pci-ep-dma.c > create mode 100644 include/linux/pci-ep-dma.h > > -- > 2.51.0 > >