Re: [PATCH v13 0/5] vfio/dma-buf: add TPH support for peer-to-peer access

 Hi Christian,

A gentle ping on this series, especially patch 3, which adds the
optional dma-buf get_pci_tph() callback. Could you please review this
when you have a chance?

https://lore.kernel.org/linux-pci/[email protected]/

Thanks,
Zhiping


On Fri, Jul 31, 2026 at 2:22 PM Zhiping Zhang <[email protected]> wrote:
>
> This series adds TLP Processing Hints (TPH) support to the VFIO dma-buf
> export path, allowing importing drivers (e.g. mlx5) to use the
> exporter's steering tag when performing peer-to-peer DMA into a
> VFIO-owned device.
>
> There is no separate in-tree vendor kernel driver for the target device:
> vfio-pci is the in-tree driver and the targeted device is managed
> from userspace via VFIO passthrough. That is why the ST has to flow
> through a uAPI: userspace owns the device and its ST table, so it is the
> entity that can configure a meaningful value for a given dma-buf. The
> kernel-visible participants are still in-tree: vfio-pci exports the
> dma-buf and mlx5 imports it.
>
> On the effect: the endpoint's PCIe ingress block uses the ST as
> an in-band instruction for the incoming P2P TLP -- selecting a target
> cache partition and, on writes, an in-flight operation on the data
> before it lands. The dma-buf callback keeps this opaque to the
> framework -- only the producer (userspace owner of the VFIO device)
> and the consumer (endpoint block) need to interpret the value. The
> dma-buf get_pci_tph callback itself is optional, but workloads that
> depend on the endpoint's in-flight operation need it because fallback
> does not produce the same result.
>
> The dma-buf hook is intentionally generic and discoverable rather than
> a private side channel. The exporter owns the completing address
> space for the dma-buf and decides whether it can provide a meaningful
> ST/PH tuple for that completer; the dma-buf core keeps the tuple opaque,
> and importers merely request the namespace they support and place the
> returned value on generated TLPs. Exporters that cannot derive a
> meaningful tuple simply return -EOPNOTSUPP.
>
> TPH is advisory: a steering tag that is not honored on the path (for
> example an intermediate routing element that does not forward the TPH
> prefix) is ignored and the request completes as an ordinary,
> non-TPH transaction (PCIe Base 6.4 sec 2.2.7.1). This series therefore
> targets the same-Root-Port / common-switch topology, where the ST
> reaches the completer; cross-Root-Port P2P is best-effort and is not
> gated in the uAPI, since supplying an unused ST is harmless and there
> is no discoverable "TPH routing" capability to test against.
>
> Patch 1 folds the reserved 0b10 "TPH Completer Supported" encoding into
> "not supported" in get_rp_completer_type(), so only architected values
> can reach the TPH Requester Enable field. It was previously posted
> standalone to linux-pci; per Alex Williamson's v12 review it now travels
> with the series, which removes the cross-tree ordering dependency and
> lets review tooling apply the series as posted.
> Patch 2 adds small PCI/TPH type helpers so drivers can query the enabled
> TPH requester mode and the device's TPH Completer Supported field
> without reaching into pci_dev internals (and so callers in
> CONFIG_PCIE_TPH=n builds get a clean fallback). pcie_tph_completer_type()
> applies the same reserved-encoding fold as get_rp_completer_type(),
> inlined locally so the helper is self-contained.
> Patch 3 adds the optional dma_buf_ops::get_pci_tph callback plus the
> dma_buf_get_pci_tph() importer wrapper so importers can fetch TPH
> metadata from an exporter under dmabuf->resv.
> Patch 4 implements get_pci_tph in vfio-pci and adds the new uAPI
> (VFIO_DEVICE_FEATURE_DMA_BUF_TPH) for userspace to attach the metadata.
> Patch 5 wires up the mlx5 RDMA driver as a consumer. It also enforces the
> dma_buf_get_pci_tph() steering-tag lifetime: the tag is only valid for the
> mapping it was queried against, and the mkey's TPH fields cannot be
> reprogrammed in place. mlx5 therefore records the registration-time tuple
> and re-queries after each dma-buf mapping is established under
> dmabuf->resv; unchanged tuples continue with the existing mkey, while
> changed or missing tuples fail the remap rather than continue with a
> stale hint. For vfio-pci BAR dma-bufs this is expected to be a no-op
> because invalidation is revoke/quiesce, not movement to a new backing
> placement, and the userspace-provided tuple is not changed by the
> revoke/un-revoke path.
>
> Build-tested with both CONFIG_PCIE_TPH=y and CONFIG_PCIE_TPH=n.
> Functional validation on the target topology: PCIe analyzer captures
> on the P2P TLPs confirm the ST emitted by mlx5 matches the value
> configured through VFIO_DEVICE_FEATURE_DMA_BUF_TPH, and the end-to-end
> P2P workload only produces results consistent with the endpoint's
> ST-selected in-flight operation. For example, with userspace
> configuring 8-bit ST=0xf0 and PH=2, an analyzer capture of a peer-to-
> peer MWr64 shows "STP MWr64 TC=0 OHC=2 ..." followed by "OHC-B
> ST=F0h PH=2 HV=1":
> (TLP Captures)
> 08000260 -> STP MWr64 TC=0 OHC=2 TS=0 Attr=0 L=8
> F0000004 -> RID=4h:0h.0h EP- Tag=F0h
> E0200000 -> AddrH=000020E0h
> 00080006 -> AddrL=06000800h
> 90F00000 -> OHC-B ST=F0h PH=2 HV=1 AMA=0 AV-
>
> The dma-buf get_pci_tph interface has also been exercised by a second,
> independent importer: a different vendor's NIC whose driver is not yet
> upstream, locally taught to call dma_buf_get_pci_tph(). A PCIe analyzer
> confirmed the ST it placed on outbound P2P TLPs matches the value
> configured through VFIO_DEVICE_FEATURE_DMA_BUF_TPH, the same result as
> with mlx5. Two unrelated importer drivers exercising the callback
> end-to-end shows the interface is not tied to a single consumer. That
> importer change is out-of-tree and not part of this series. For that
> second importer, with userspace configuring 8-bit ST=0xe0 and PH=0,
> an analyzer capture shows:
> (TLP Captures)
> 08200260 -> STP MWr64 TC=0 OHC=2 TS=1 Attr=0 L=8
> 4E00004C -> RID=4Ch:0h.0h EP- Tag=4Eh
> 00170000 -> AddrH=00001700h
> 00200006 -> AddrL=06002000h
> 10E00000 -> OHC-B ST=E0h PH=0 HV=1 AMA=0 AV-
>
> Changes since v12:
>   Patch 1 (PCI/TPH, new to the series): the reserved-encoding fold,
>   previously posted standalone to linux-pci [1], is now the first patch
>   here (Alex Williamson). Sashiko could not apply v12 because of that
>   external dependency; with the fold in-series and the mlx5 leak fix in
>   linux-next, v13 has none. The code is unchanged from the standalone
>   v3; the Fixes: tag is dropped, since no code path can reach the
>   reserved encoding today and the patch is hardening rather than a fix
>   for observed silicon.
>
>   Patch 2 (PCI/TPH): inline the reserved-encoding fold in
>   pcie_tph_completer_type() rather than calling the helper that earlier
>   folding revisions added; that helper was dropped in folding v3 per
>   Bjorn Helgaas and Wei Huang.
>
>   Patch 3 (dma-buf): no functional change.
>
>   Patch 4 (vfio/pci): also gate the DMA_BUF_TPH feature on
>   vdev->pci_ops->get_dmabuf_phys, matching
>   vfio_pci_core_feature_dma_buf(). Without it PROBE reported the feature
>   as supported on a device that advertises TPH Completer support but
>   cannot export a vfio dma-buf at all, so nothing could ever carry the
>   metadata (Alex Williamson, who raised this to uAPI-affecting
>   severity).
>
>   Patch 5 (mlx5): keep the !dev->st early-out in mlx5_st_alloc_index()
>   ahead of the pcie_tph_get_cpu_st() call, so splitting out
>   mlx5_st_alloc_index_by_tag() neither adds an ACPI _DSM invocation on
>   devices without ST support nor changes the errno userspace sees when
>   the _DSM lookup fails (Alex Williamson). The commit message now
>   describes this rather than presenting the split as a pure extraction.
>
> Previous link:
> v12: https://lore.kernel.org/linux-pci/[email protected]/
> v11: https://lore.kernel.org/linux-pci/[email protected]/
> v10: https://lore.kernel.org/linux-pci/[email protected]/
> v9: https://lore.kernel.org/dri-devel/[email protected]/
> v8: https://lore.kernel.org/dri-devel/[email protected]/
> v7: https://lore.kernel.org/dri-devel/[email protected]/
> v6: https://lore.kernel.org/dri-devel/[email protected]/
> v5: https://lore.kernel.org/dri-devel/[email protected]/
> v4: https://lore.kernel.org/linux-pci/[email protected]/
> v3: https://lore.kernel.org/linux-pci/[email protected]/
> v2: https://lore.kernel.org/linux-pci/[email protected]/
>
> Zhiping Zhang (5):
>   PCI/TPH: treat reserved 0b10 completer encoding as unsupported
>   PCI/TPH: Add requester/completer type helpers
>   dma-buf: add optional get_pci_tph() callback
>   vfio/pci: implement get_pci_tph and DMA_BUF_TPH feature
>   RDMA/mlx5: get tph for p2p access when registering dma-buf mr
>
>  drivers/dma-buf/dma-buf.c                     |  32 ++++
>  drivers/infiniband/hw/mlx5/main.c             |   1 +
>  drivers/infiniband/hw/mlx5/mlx5_ib.h          |  11 ++
>  drivers/infiniband/hw/mlx5/mr.c               | 151 +++++++++++++++++-
>  drivers/infiniband/hw/mlx5/odp.c              |   7 +
>  .../net/ethernet/mellanox/mlx5/core/lib/st.c  |  52 +++++-
>  drivers/pci/tph.c                             |  55 ++++++-
>  drivers/vfio/pci/vfio_pci_core.c              |   3 +
>  drivers/vfio/pci/vfio_pci_dmabuf.c            | 120 +++++++++++++-
>  drivers/vfio/pci/vfio_pci_priv.h              |  13 ++
>  include/linux/dma-buf.h                       |  25 +++
>  include/linux/mlx5/driver.h                   |  15 ++
>  include/linux/pci-tph.h                       |   8 +
>  include/uapi/linux/vfio.h                     |  43 +++++
>  14 files changed, 517 insertions(+), 19 deletions(-)
>
> --
> 2.53.0-Meta
>