[PATCH 00/15] PCI: endpoint: Remote DMA support via vNTB

Hi,

This series lets an endpoint-integrated DMA engine be consumed on the RC
side through vNTB.

The initial target is DesignWare endpoint eDMA. pci-epf-vntb exports a
versioned DMA locator plus the minimum peer-visible resources,
ntb_hw_epf parses that locator and instantiates an auxiliary device after
LINK_UP, and dw-edma-aux binds to that child to expose a DMA engine
provider on the RC side. ntb_ep_dma is included both as the first
consumer and as a simple bring-up test.


Background
==========

I previously posted a broader RFC series:

  https://lore.kernel.org/all/[email protected]/

This series is not a direct continuation of that RFC. Its scope is
narrower, and the approach has changed substantially.

That RFC had two architectural issues:

  1. dw-edma-specific logic lived under drivers/ntb/hw/, even though
     the exported DMA engine is not related to any NTB hardware.
  2. Remote-use channel delegation relied on vendor-specific peripheral
     configuration.

This series builds on the recently discussed pci_epc_aux_resource work
(see "Dependency" 3 below) and addresses both issues by:
  - introducing vendor-neutral DMA-channel delegation in the PCI EPC
    layer via pci_epc_delegate_dma_channels() and
  - making vNTB and ntb_hw_epf aware of the remote DMA resource.

On the EP side, pci-epf-vntb describes the exported DMA resources as
part of the vNTB BAR layout. On the RC side, ntb_hw_epf detects that
export and registers an auxiliary child device. A vendor-specific
frontend can then bind to that child device and reconstruct the remote
DMA provider. This series includes such a frontend for DesignWare eDMA
in dw-edma-aux.


Architecture
============

  EP kernel
    pci-epf-vntb
      - exports the usual vNTB control/db/MW resources
      - optionally exports a versioned DMA slice

  RC kernel
    ntb_hw_epf
      - parses the control layout
      - instantiates an auxiliary child for the exported DMA ABI
    dw-edma-aux
      - binds to that child and registers a DMAEngine provider


Series layout
=============

  01-05 prepare dw-edma and auxiliary-resource metadata
  06-10 export delegated controller-owned DMA resources through vNTB
  11-13 discover the exported DMA instance on the host and bind
        dw-edma-aux
  14 adds ntb_ep_dma as the first consumer / smoke test
  15 documents the model and the configfs layout

I did not split the infrastructure patches (01-13) away from its
consumer (14). The series is meant to be reviewed as one feature:
producer, discovery, consumer, and test coverage.


Test
====

Tested on R-Car S4 Spider with the dependency below.

  1. Configure and start pci_epf_vntb with DMA export enabled.

     The actual commands I used for testing:

     # modprobe pci_epf_vntb
     # cd /sys/kernel/config/pci_ep/
     # mkdir functions/pci_epf_vntb/func1
     # echo 0x1912 >   functions/pci_epf_vntb/func1/vendorid
     # echo 0x0030 >   functions/pci_epf_vntb/func1/deviceid
     # echo 32 >       functions/pci_epf_vntb/func1/msi_interrupts
     # echo 16 >       functions/pci_epf_vntb/func1/pci_epf_vntb.0/db_count
     # echo 128 >      functions/pci_epf_vntb/func1/pci_epf_vntb.0/spad_count
     # echo 1 >        functions/pci_epf_vntb/func1/pci_epf_vntb.0/num_mws
     # echo 0xF9000 >  functions/pci_epf_vntb/func1/pci_epf_vntb.0/mw1
     # echo 0xF9000 >  functions/pci_epf_vntb/func1/pci_epf_vntb.0/dma_offset
     # echo 4 >        functions/pci_epf_vntb/func1/pci_epf_vntb.0/dma_num_chans
     # echo 0x1912 >   functions/pci_epf_vntb/func1/pci_epf_vntb.0/vntb_vid
     # echo 0x0030 >   functions/pci_epf_vntb/func1/pci_epf_vntb.0/vntb_pid
     # echo 0x10 >     functions/pci_epf_vntb/func1/pci_epf_vntb.0/vbus_number
     # echo 0 >        functions/pci_epf_vntb/func1/pci_epf_vntb.0/ctrl_bar
     # echo 2 >        functions/pci_epf_vntb/func1/pci_epf_vntb.0/mw1_bar
     # echo 2 >        functions/pci_epf_vntb/func1/pci_epf_vntb.0/dma_bar
     # echo 4 >        functions/pci_epf_vntb/func1/pci_epf_vntb.0/db_bar
     # ln -s controllers/e65d0000.pcie-ep functions/pci_epf_vntb/func1/primary/
     # echo 1 > controllers/e65d0000.pcie-ep/start

  2. Boot or rescan the RC side and let ntb_hw_epf probe.

  3. Load ntb_ep_dma on both EP and RC.

  4. On the RC side, run the test as follows:

     # cat /sys/kernel/debug/ntb_ep_dma/0000:01:00.0/ready
     # echo 1 > /sys/kernel/debug/ntb_ep_dma/0000:01:00.0/run
     # cat /sys/kernel/debug/ntb_ep_dma/0000:01:00.0/result

       last_status: 0
       last_len: 4096
       local_buf_dma: 0xfffff000
       local_buf_size: 4096
       peer_ready: 1
       peer_state: pass # <----(*)
       peer_dma: 0x4e11e000
       peer_size: 4096
       peer_seq: 1
       peer_xfer_len: 4096
       link_up: 1

       (*) The peer reports "pass" after the transfer completes successfully,


Kernel base
===========

pci.git endpoint:
Commit 0b74f7d72399 ("PCI: endpoint: Propagate error from pci_epf_create()")


Dependency
==========

1. [PATCH v4 00/10] PCI: endpoint: Differentiate between disabled and reserved BARs
   https://lore.kernel.org/linux-pci/[email protected]/
   https://patchwork.kernel.org/project/linux-pci/list/?series=1065666

2. [PATCH 0/2] dmaengine: dw-edma: Interrupt-emulation doorbell support
   https://lore.kernel.org/dmaengine/[email protected]/
   https://patchwork.kernel.org/project/linux-dmaengine/list/?series=1054298
   Note: already landed in dmaengine/next.

3. [PATCH v10 0/7] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback
   https://lore.kernel.org/all/[email protected]/
   https://patchwork.kernel.org/project/linux-pci/list/?series=1059820

4. [PATCH v2 0/3] NTB: Allow drivers to provide DMA mapping device
   https://lore.kernel.org/linux-pci/[email protected]/
   https://patchwork.kernel.org/project/linux-pci/list/?series=1062308
   Note: this series uses ntb_get_dma_dev() API.

5. [PATCH v2 00/10] PCI: endpoint: pci-epf-vntb: Document legacy MSI doorbell offset
   https://lore.kernel.org/linux-pci/[email protected]
   https://patchwork.kernel.org/project/linux-pci/list/?series=1058871
   Note: v2 title was incorrect. See my reply to the cover letter.

Additionally, for ntb_ep_dma test to pass on R-Car S4 Spider:

6. [PATCH v2] PCI: dwc: rcar-gen4-ep: Mark BAR0 and BAR2 as Resizable BARs
   https://lore.kernel.org/linux-pci/[email protected]/
   https://patchwork.kernel.org/project/linux-pci/list/?series=1052780
   Note: already landed in pci/next.

7. [PATCH v2] PCI: dwc: rcar-gen4: Use 4K EPC BAR alignment
   https://lore.kernel.org/linux-pci/[email protected]/
   https://patchwork.kernel.org/project/linux-pci/list/?series=1062031


Merge plan
==========

This series touches three areas:

  - PCI endpoint core and pci-epf-vntb
  - DesignWare eDMA (dw-edma)
  - an NTB test client

The series intentionally keeps the infrastructure changes together with
their first consumer, the ntb_ep_dma test client. Splitting them further
would leave the infrastructure patches without a consumer, so the
patches are kept together as a single series.

Mani is a maintainer for both the PCI EP and dw-edma. My initial thought
was therefore to collect acks from the relevant subsystems (PCI EP,
dw-edma, and NTB) and have the series applied through the PCI EP tree.

However, I am of course open to any suggestions regarding the preferred
merge path or series split if maintainers think another approach would
be more appropriate.


Best regards,
Koichiro


Koichiro Den (15):
  dmaengine: dw-edma: Cache DMA channel IDs in dw_edma_chip
  PCI: endpoint: Add DMA channel metadata to pci_epc_aux_resource
  PCI: dwc: ep: Report DMA channel metadata for aux resources
  dmaengine: dw-edma: Add per-channel interrupt routing control
  dmaengine: dw-edma: Compose MSI messages from allocated IRQs
  PCI: endpoint: pci-epf-vntb: Fold MW runtime state into a struct
  PCI: endpoint: Add EPC DMA channel delegation hooks
  PCI: dwc: ep: Delegate exported eDMA channels through EPC ops
  PCI: endpoint: Add pci-ep-dma helper for exported DMA ABI v1
  PCI: endpoint: pci-epf-vntb: Support DMA export and shared BAR layouts
  NTB: hw: epf: Parse control-layout version and DMA locator
  NTB: hw: epf: Enumerate auxiliary child for DMA ABI v1
  dmaengine: dw-edma: Add auxiliary-bus frontend for exported eDMA
  NTB: Add ntb_ep_dma test client
  Documentation: PCI: endpoint: Add vNTB DMA export HOWTO

 Documentation/PCI/endpoint/index.rst          |   1 +
 .../PCI/endpoint/pci-vntb-dma-howto.rst       |  83 ++
 drivers/dma/dw-edma/Kconfig                   |  11 +
 drivers/dma/dw-edma/Makefile                  |   1 +
 drivers/dma/dw-edma/dw-edma-aux.c             | 297 +++++++
 drivers/dma/dw-edma/dw-edma-core.c            | 101 ++-
 drivers/dma/dw-edma/dw-edma-core.h            |  13 +
 drivers/dma/dw-edma/dw-edma-v0-core.c         |  26 +-
 drivers/ntb/hw/epf/Kconfig                    |   1 +
 drivers/ntb/hw/epf/ntb_hw_epf.c               | 199 ++++-
 drivers/ntb/test/Kconfig                      |  10 +
 drivers/ntb/test/Makefile                     |   1 +
 drivers/ntb/test/ntb_ep_dma.c                 | 695 +++++++++++++++
 .../pci/controller/dwc/pcie-designware-ep.c   | 196 +++++
 drivers/pci/controller/dwc/pcie-designware.h  |  11 +
 drivers/pci/endpoint/Makefile                 |   2 +-
 drivers/pci/endpoint/functions/pci-epf-vntb.c | 794 ++++++++++++++++--
 drivers/pci/endpoint/pci-ep-dma.c             | 342 ++++++++
 drivers/pci/endpoint/pci-epc-core.c           |  84 ++
 include/linux/dma/edma.h                      |  42 +
 include/linux/pci-ep-dma.h                    | 130 +++
 include/linux/pci-epc.h                       |  31 +
 22 files changed, 2981 insertions(+), 90 deletions(-)
 create mode 100644 Documentation/PCI/endpoint/pci-vntb-dma-howto.rst
 create mode 100644 drivers/dma/dw-edma/dw-edma-aux.c
 create mode 100644 drivers/ntb/test/ntb_ep_dma.c
 create mode 100644 drivers/pci/endpoint/pci-ep-dma.c
 create mode 100644 include/linux/pci-ep-dma.h

-- 
2.51.0