RE: [PATCH v3 00/11] vfio/pci: Add CXL Type-2 device passthrough support

Thank you all for review suggestions. Closing the loop on v3. I will rework
patch series v4 as per the review feedback.

Current understanding of what changes in v4:

- Create v4 on top of cxl_reset v9 from Srirangan. Move the patch series to
  Linus's RC tree and apply Srirangan's cxl-reset support patch v9 on it.
- Patch 01 is dropped; vfio-cxl reads pdev->hdm directly.
- The memdev is created at bind via devm_cxl_probe_mem() and is removed on
  unbind. Probe takes no action that outlives it.
- Volatile state (DVSEC shadow, CM snapshot, ecap handlers) is acquired at
  open_device() and released at close_device(). Nothing volatile is read at bind.
- HDM mmap will make use of huge pfnmap as seen in vfio mmap routine.
- DVSEC is handled through ecap_perms readfn/writefn with a range check against
  the CXL DVSEC body; other DVSECs pass through unchanged.
- UAPI: the flag is VFIO_DEVICE_FLAGS_CXL; region index fields are removed from
  the device cap; component-register geometry moves to a region info cap; there
  is no firmware-committed flag.
- The CM capability snapshot uses readq() throughout; the lock-on-commit state
  machine is corrected so COMMIT is read-only while locked.
- The file moves to drivers/vfio/pci/cxl/ with a private header; duplicate
  register defines are dropped for the upstream ones; devres uses
  devm_add_action_or_reset().
- vfio-pci loads vfio-cxl through request_module() only when it finds a CXL
  device; vfio-cxl registers its ops at module_init; each bound device holds a
  module reference from probe to release.
- vfio-cxl owns the whole component BAR via pci_request_selected_regions(). A
  cxl_register_map flag (skip_sub_bar_request) lets cxl-core skip its HDM/RAS
  sub-block claim for an owning driver; cxl_pci is unchanged.
- The HPA is resolved before the guest sees the device; the guest only relocates
  CXL memory in GPA space. Range readiness happens at enumeration in cxl-core, so
  vfio-cxl waits on nothing and consumes a ready resource (No await_range_active
  exported)
--

Device lifecycle in v4

cxl-pci discovery / enumeration
- pdev->hdm is populated (decoder settings, count, base and size) via cxl_reset v9.
- No memdev here; a passthrough device binds to vfio-pci, not cxl_pci.

vfio-pci probe / vfio-cxl bind (reversible)
- Read hdm_hpa and hdm_size from pdev->hdm as u64 on vfio_pci_core_device.
- Walk the PCIe ecap for the CXL DVSEC body range; record offset and size.
- Record the component-register BAR index.
- request_module("vfio-cxl"); take one module reference, held until release.
- Create the memdev via devm_cxl_probe_mem(); devm removes it on unbind.
- Register pfn_address_space over the HDM range for error containment.
- Request the whole component BAR; cxl-core skips its sub-block claim.

vfio-pci open_device (volatile state, device-open lifetime)
- Read the DVSEC body into a shadow; read the CM array with readq().
- Validate pdev->hdm against the hardware.
- Install the ecap_perms DVSEC handler (range check, fall through for others).
- Allocate cxl_vfio_state and set vdev->cxl.

vfio-pci close_device
- Remove the DVSEC handler; free the shadow and snapshot; clear vdev->cxl.

vfio-pci remove / unbind
- Release the component BAR; unregister pfn_address_space; drop the module
  reference.
- The memdev is auto-removed by devm.

I will post v4 on this basis.

Thanks,
Manish

> -----Original Message-----
> From: Manish Honap <[email protected]>
> Sent: 13 July 2026 22:14
> To: Dave Jiang <[email protected]>; [email protected];
> [email protected]; [email protected]; [email protected]; Ankit Agrawal
> <[email protected]>; [email protected];
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected]
> Cc: Neo Jia <[email protected]>; Krishnakant Jaju <[email protected]>;
> Vikram Sethi <[email protected]>; Zhi Wang <[email protected]>;
> [email protected]; [email protected]; linux-
> [email protected]; [email protected]; linux-
> [email protected]; Manish Honap <[email protected]>
> Subject: RE: [PATCH v3 00/11] vfio/pci: Add CXL Type-2 device
> passthrough support
> 
> > -----Original Message-----
> > From: Dave Jiang <[email protected]>
> > Sent: 10 July 2026 21:57
> > To: Manish Honap <[email protected]>; [email protected];
> > [email protected]; [email protected]; [email protected]; Ankit Agrawal
> > <[email protected]>; [email protected];
> > [email protected]; [email protected]; [email protected];
> > [email protected]; [email protected]
> > Cc: Neo Jia <[email protected]>; Krishnakant Jaju <[email protected]>;
> > Vikram Sethi <[email protected]>; Zhi Wang <[email protected]>;
> > [email protected]; [email protected]; linux-
> > [email protected]; [email protected]; linux-
> > [email protected]
> > Subject: Re: [PATCH v3 00/11] vfio/pci: Add CXL Type-2 device
> > passthrough support
> >
> > External email: Use caution opening links or attachments
> >
> >
> > On 6/25/26 9:53 AM, [email protected] wrote:
> > > From: Manish Honap <[email protected]>
> > >
> > > CXL Type-2 accelerators (CXL.mem-capable GPUs and similar) cannot be
> > > passed through to virtual machines with stock vfio-pci because the
> > > driver has no concept of HDM decoder management, HDM region
> > > exposure, or component register virtualization.  This series adds
> > > those three pieces, sufficient for a guest to use the device's
> > > firmware-committed coherent memory under UVM / ATS.
> > >
> > > v3 is a rewrite of the v2 framework form, responding to Dan's
> > > request in the v2 review for "less emulation, narrower interfaces,
> > > and a closer mapping to the spec language."
> > > In this release, cxl-core exposes four EXPORT_SYMBOL_GPL helpers
> > > behind an opaque handle.  vfio-pci becomes a thin transport on top
> > > of
> > those.
> > > Please see "Changes since v2" and "Reviewer feedback addressed"
> > > below for the per-area summary.
> > >
> > > Motivation
> > > ==========
> > >
> > > A CXL Type-2 device exposes its HDM-mapped device memory through HDM
> > > decoders that BIOS programs and commits at boot.  To pass such a
> > > device to a guest, vfio-pci has to do three things at once:
> > >
> > >   1. Surface the firmware-committed HDM-mapped HPA range as a guest-
> > >      mmappable region.
> > >
> > >   2. Surface a CXL-spec-compliant view of the CXL Device DVSEC body,
> > >      the HDM Decoder Capability block, and the CXL.cache/mem cap-
> array
> > >      prefix, so the guest's CXL driver enumerates the same topology
> > >      the host saw.
> > >
> > >   3. Keep the host's committed decoder configuration intact (the
> > >      physical decoder is never reprogrammed) while letting the guest
> > >      observe and manage a shadow that follows the per-field write
> > >      semantics in the spec.
> > >
> > > The series builds on Alejandro Lucero-Palau's v28 work applied on
> > > for-7.3/cxl-type2-enabling [1] (sfc is the in-tree consumer today).
> > > vfio-pci becomes the second consumer.
> > >
> > > Architecture
> > > ============
> > >
> > > cxl-core owns the CXL semantics.  A new file
> > > drivers/cxl/core/passthrough.c (gated by hidden Kconfig
> > > CXL_VFIO_PASSTHROUGH) provides four exported symbols:
> > >
> > >     struct cxl_passthrough *
> > >     devm_cxl_passthrough_create(struct device *dev,
> > >                                 struct cxl_dev_state *cxlds);
> > >
> > >     int cxl_passthrough_dvsec_rw(p, off, val, sz, write);
> > >     int cxl_passthrough_hdm_rw  (p, off, val,      write);
> > >     int cxl_passthrough_cm_rw   (p, off, val,      write);
> > >
> > > cxl_passthrough is an opaque handle; vfio-pci sees no cxl-internal
> > > struct pointers.  The shadows are snapshotted at create time: the
> > > DVSEC body from PCI config space dword by dword, the CM cap-array
> > > and HDM block from the cxl-core MMIO mapping at cxlds->reg_map.base.
> > > Per-field write semantics follow below:
> > > CXL r4.0 8.1.3 DVSEC:
> > > - LOCK is RWO,
> > > - CONTROL/CONTROL2 are RWL gated on CONFIG_LOCK,
> > > - STATUS/STATUS2 are RW1C,
> > > - RANGE1 is HwInit, RANGE2 is RsvdZ
> > > CXL r4.0 8.2.4.20 HDM:
> > > - GLOBAL_CTRL RW,
> > > - decoder CTRL implements COMMIT/COMMITTED,
> > > - decoder BASE/SIZE RWL gated on COMMITTED or LOCK_ON_COMMIT,
> > > - cap header HwInit).
> > >
> > > vfio-pci becomes a thin transport.  The new module
> > > drivers/vfio/pci/cxl/ exposes two VFIO regions.
> > >
> > >   VFIO_REGION_SUBTYPE_CXL (HDM region): mmappable view of the
> > >   HDM-mapped HPA. The mmap fault handler calls vmf_insert_pfn() from
> > >   the physical HPA. pread/pwrite go through the memremap_wb() kva
> > >   captured at bind time.
> > >
> > >   VFIO_REGION_SUBTYPE_CXL_COMP_REGS (component register shadow):
> > >   pread/pwrite only, dword-aligned (-EINVAL on misalignment).
> > >   Each dword dispatches by offset to cxl_passthrough_cm_rw() or
> > >   cxl_passthrough_hdm_rw(). No shadow state on the vfio side; cxl-
> core
> > >   enforces the spec.
> > >
> > > CXL DVSEC config-space accesses use a clipping shim in
> > > vfio_pci_config_rw_single(). A config-space chunk that crosses the
> > > DVSEC body boundary is split: header bytes go through the generic
> > > perm-bits path, body bytes go through cxl_passthrough_dvsec_rw().
> > > The shim replaces v2's approach of repointing ecap_perms[]
> > >
> > > Sparse-mmap is exposed on the component BAR so userspace can mmap
> > > the non-component portions directly; only the CXL component register
> > > sub-range goes through pread/pwrite emulation. The CXL sub-range is
> > > also skipped from vfio_pci-core's request_selected_regions() set
> > > because cxl-core's devm_cxl_probe_mem() already holds a
> > > request_mem_region() on it; the asymmetric skip is matched by an
> > > asymmetric release on disable().
> > >
> > > Scope and out-of-scope
> > > ======================
> > >
> > > In scope (rejected at create time with -EOPNOTSUPP otherwise):
> > >
> > >   - Firmware-committed devices (HOST_FIRMWARE_COMMITTED set).
> > >   - Single HDM decoder (hdm_count == 1).
> > >   - No interleave (IW == 0).
> > >
> > > Out of scope, deferred for follow-on work:
> > >
> > >   - Multi-decoder devices and interleave.
> > >   - Guest-driven (non-firmware-committed) HDM commit.
> > >   - Hotplug, FLR, and sibling-function reset of CXL Type-2 devices.
> > >
> > > Changes since v2
> > > ================
> > >
> > > This is a rewrite, not an incremental update.  The structure of the
> > > series changed (20 patches in v2 to 11 in v3) because v3 collapses
> > > v2 patches 9-15 (detection, HDM emulation, media readiness, region
> > > management, HDM region, DVSEC emulation) into one cxl-core helper
> > > file and one vfio-pci consumer.
> > >
> > > Framework replaced by narrow opaque-handle helpers (patches 6, 8)
> > >
> > >   v2 carried a generic register-emulation framework split across
> four
> > >   state-machine files in cxl-core.
> > >   v3 collapses it into one file: drivers/cxl/core/passthrough.c
> > >   exposing the four EXPORT_SYMBOL_GPL helpers above behind a struct
> > >   cxl_passthrough opaque handle.
> > >
> > > Shadow ownership moved into cxl-core (patches 6, 8)
> > >
> > >   vfio-pci no longer keeps any per-field state. It forwards
> > >   (offset, value) into cxl-core, and cxl-core enforces the spec
> > >   (RWO, RWL, RW1C, HwInit, RsvdZ) with explicit CXL r4.0 section
> > >   references in the switch arms.
> > >
> > > DVSEC config-space clipping shim (patch 8)
> > >
> > >   v2 repointed ecap_perms[] to redirect CXL DVSEC reads and writes.
> > >   v3 keeps ecap_perms[] untouched and clips per-config-access chunks
> > >   at the DVSEC body boundary in vfio_pci_config_rw_single(); header
> > bytes
> > >   go through the generic perm-bits path, body bytes go through
> > >   cxl_passthrough_dvsec_rw(). The shim is local to the per-device
> > >   path.
> > >
> > > CONFIG_VFIO_PCI_CXL gates the new module (patch 7)
> > >
> > >   v2 had a CONFIG_VFIO_CXL_CORE Kconfig stub; v3 renames it to
> > >   CONFIG_VFIO_PCI_CXL to match the vfio-pci naming convention.
> > >   The hidden CXL_VFIO_PASSTHROUGH selects the cxl-core helper file
> > >   on demand. With both disabled, the cxl-core size is unchanged.
> > >
> > > UAPI rewritten with named fields (patch 5)
> > >
> > >   vfio_device_info_cap_cxl in v3 carries:
> > >     flags + HOST_FIRMWARE_COMMITTED bit
> > >     hdm_region_idx
> > >     comp_reg_region_idx
> > >     comp_reg_bar
> > >     comp_reg_offset
> > >     comp_reg_size
> > >   The DPA terminology is renamed to HDM region throughout.
> > >   CACHE_CAPABLE (HDM-DB indicator) is dropped;
> > >   it was informational only in v2 with no caller, and re-adding it
> > >   for an active CXL.cache plumbing series later.
> > >
> > > Selftests trimmed (patch 9)
> > >
> > >   v2 carried selftests for device detection, capability parsing,
> > >   region enumeration, HDM register emulation, HDM mmap with
> > >   page-fault insertion, FLR invalidation, and DVSEC register
> > >   emulation. v3 keeps a smoke-test set of six focused tests:
> > >
> > >     device_is_cxl                  GET_INFO advertises FLAGS_CXL
> > >                                    and a populated CAP_CXL.
> > >     hdm_region_mmap_rw             mmap one page, write+read back.
> > >     component_bar_sparse_mmap      SPARSE_MMAP cap excludes the
> > >                                    CXL component register sub-range.
> > >     comp_regs_cm_cap_array_read    pread of the CM cap-array
> > >                                    header at CXL_CM_OFFSET succeeds
> > >                                    (CAP_ID == 1).
> > >     dvsec_lock_byte_read           pread of the DVSEC CONFIG_LOCK
> > >                                    byte through the clipping shim
> > >                                    succeeds.
> > >     hdm_decoder_commit_fsm         COMMIT / COMMITTED state machine
> > >                                    and LOCK_ON_COMMIT behaviour.
> > >
> > >   FLR invalidation, page-fault insertion under load, and full
> > >   DVSEC field-by-field write coverage are deferred to a follow-on
> > >   selftest series. The current six are the minimal set that
> > >   exercises the kernel-side contract end-to-end.
> > >
> > > cxl-core prep patches split (patches 1-4)
> > >
> > >   v3 keeps the cxl-side enablers from v2 patches 1-4 but each as
> > >   a standalone change so the cxl maintainer can review the helper
> > >   API independently of the vfio consumer:
> > >
> > >     [1/11] cxl_get_hdm_info()
> > >     [2/11] cxl_await_range_active() split from media-ready wait
> > >     [3/11] cxl_register_map records BIR + BAR offset
> > >     [4/11] component/HDM register defines moved to
> > > uapi/cxl/cxl_regs.h
> > >
> > > Reviewer feedback addressed
> > > ===========================
> > >
> > > Dan
> > > ---
> > >
> > > - VFIO exposes HDM/host-visible region, not raw DPA; docs/UAPI say
> HDM
> > >   region, DPA only inside cxl-core where appropriate.
> > > - One vfio-pci device = one HDM region / one decoder, no interleave;
> > >   hdm_count != 1 → -EOPNOTSUPP.
> > > - Global HDM on DVSEC Range Base treated as legacy; RANGE1/RANGE2
> > >   read-only snapshot, guest writes dropped.
> > > - No guest/kernel lock games; DVSEC LOCK and HDM LOCK_ON_COMMIT RWO,
> > >   fixed at create from firmware snapshot.
> > > - Opaque cxl_passthrough handle only; vfio gets HPA via memdev probe
> +
> > >   layout via cxl_get_hdm_info(), rw via helpers.
> > > - No multi-region accelerator case in v3; single region enforced,
> > >   multi-region deferred.
> > > - cxl_await_range_active stays in cxl-core probe; not exported, vfio
> > does
> > >   not call it.
> > > - No guest LOCK→0 reprogram; guest cannot clear LOCK to remap host
> > HPA;
> > >   kernel uncommit tied to COMMIT, not LOCK alone.
> > >
> > > Jason / Gregory / Dan
> > > ---------------------
> > >
> > > - memremap(WB) + request_mem_region on HPA; conflicting
> > > direct-map/EFI
> > use
> > >   fails probe with -EBUSY.
> > >
> > > Jonathan
> > > --------
> > >
> > > - uapi/cxl/cxl_regs.h for register defines so VMMs need no private
> > >   kernel headers.
> > > - __free() locals on cxl-core/passthrough error paths instead of
> > >   struct-owned temporaries.
> > > - No "precommitted at probe" assumption; acquire checks COMMITTED in
> > >   HDM shadow and refuses if missing.
> > >
> > > Dave
> > > ----
> > >
> > > - memremap(MEMREMAP_WB) for HDM host mapping (not ioremap_cache).
> > > - Renamed cap flag to VFIO_CXL_CAP_HOST_FIRMWARE_COMMITTED for
> > clarity.
> > > - __free() / DEFINE_FREE() cleanup in new passthrough.c create path.
> > >
> > > Patch series
> > > ============
> > >
> > >  [1/11] cxl: Add cxl_get_hdm_info() helper for HDM decoder metadata
> > > [2/11] cxl: Split cxl_await_range_active() from media-ready wait
> > > [3/11] cxl: Record BIR and BAR offset in cxl_register_map  [4/11]
> cxl:
> > > Move component/HDM register defines to
> > >         uapi/cxl/cxl_regs.h
> > >  [5/11] vfio: UAPI for CXL Type-2 device passthrough  [6/11] cxl:
> > > Add register-virtualization helpers for vfio Type-2
> > >         passthrough
> > >  [7/11] vfio/pci: Add CONFIG_VFIO_PCI_CXL with bind-time CXL Type-2
> > >         acquisition
> > >  [8/11] vfio/pci/cxl: Add HDM + COMP_REGS regions and DVSEC clipping
> > >         shim
> > >  [9/11] selftests/vfio: Add CXL Type-2 device passthrough smoke test
> > > [10/11] docs: vfio-pci: Document CXL Type-2 device passthrough
> > > [11/11]
> > > vfio/pci: Provide opt-out for CXL Type-2 extensions
> > >
> > > Dependencies
> > > ============
> > >
> > > [1] [PATCH v28 0/5] Type2 device basic support
> > > https://lore.kernel.org/linux-cxl/20260618181806.118745-1-alejandro.
> > > lu
> > > [email protected]/
> > >
> > > [2] Previous version of this patch series [PATCH v2 00/20] vfio/pci:
> > > Add CXL Type-2 device passthrough support
> > > https://lore.kernel.org/linux-cxl/20260401143917.108413-1-mhonap@nvi
> > > di
> > > a.com/
> > >
> > > [3] Companion QEMU series
> > > [RFC 0/9] QEMU: CXL Type-2 device passthrough via vfio-pci
> > > https://lore.kernel.org/linux-cxl/20260427181235.3003865-1-mhonap@nv
> > > id
> > > ia.com/
> > >
> > > Manish Honap (11):
> > >   cxl: Add cxl_get_hdm_info() helper for HDM decoder metadata
> > >   cxl: Split cxl_await_range_active() from media-ready wait
> > >   cxl: Record BIR and BAR offset in cxl_register_map
> > >   cxl: Move component/HDM register defines to uapi/cxl/cxl_regs.h
> > >   vfio: UAPI for CXL Type-2 device passthrough
> > >   cxl: Add register-virtualization helpers for vfio Type-2
> passthrough
> > >   vfio/pci: Add CONFIG_VFIO_PCI_CXL with bind-time CXL Type-2
> > >     acquisition
> > >   vfio/pci/cxl: Add HDM + COMP_REGS regions and DVSEC clipping shim
> > >   selftests/vfio: Add CXL Type-2 device passthrough smoke test
> > >   docs: vfio-pci: Document CXL Type-2 device passthrough
> > >   vfio/pci: Provide opt-out for CXL Type-2 extensions
> > >
> > >  Documentation/driver-api/index.rst            |   1 +
> > >  Documentation/driver-api/vfio-pci-cxl.rst     | 282 ++++++
> > >  drivers/cxl/Kconfig                           |   7 +
> > >  drivers/cxl/core/Makefile                     |   1 +
> > >  drivers/cxl/core/passthrough.c                | 590 ++++++++++++
> > >  drivers/cxl/core/pci.c                        |  70 +-
> > >  drivers/cxl/core/regs.c                       |  35 +
> > >  drivers/cxl/cxl.h                             |  52 +-
> > >  drivers/vfio/pci/Kconfig                      |   2 +
> > >  drivers/vfio/pci/Makefile                     |   1 +
> > >  drivers/vfio/pci/cxl/Kconfig                  |  34 +
> > >  drivers/vfio/pci/cxl/Makefile                 |   2 +
> > >  drivers/vfio/pci/cxl/vfio_cxl_core.c          | 889
> > ++++++++++++++++++
> > >  drivers/vfio/pci/cxl/vfio_cxl_priv.h          |  71 ++
> > >  drivers/vfio/pci/vfio_pci.c                   |   9 +
> > >  drivers/vfio/pci/vfio_pci_config.c            |  31 +
> > >  drivers/vfio/pci/vfio_pci_core.c              |  68 +-
> > >  drivers/vfio/pci/vfio_pci_priv.h              |  93 ++
> > >  drivers/vfio/pci/vfio_pci_rdwr.c              |  17 +
> > >  include/cxl/cxl.h                             |  18 +
> > >  include/cxl/passthrough.h                     | 121 +++
> > >  include/linux/vfio_pci_core.h                 |   8 +
> > >  include/uapi/cxl/cxl_regs.h                   |  63 ++
> > >  include/uapi/linux/vfio.h                     |  46 +
> > >  tools/testing/selftests/vfio/Makefile         |   1 +
> > >  .../selftests/vfio/lib/vfio_pci_device.c      |  11 +-
> > >  .../selftests/vfio/vfio_cxl_type2_test.c      | 350 +++++++
> > >  27 files changed, 2821 insertions(+), 52 deletions(-)  create mode
> > > 100644 Documentation/driver-api/vfio-pci-cxl.rst
> > >  create mode 100644 drivers/cxl/core/passthrough.c  create mode
> > > 100644 drivers/vfio/pci/cxl/Kconfig  create mode 100644
> > > drivers/vfio/pci/cxl/Makefile  create mode 100644
> > > drivers/vfio/pci/cxl/vfio_cxl_core.c
> > >  create mode 100644 drivers/vfio/pci/cxl/vfio_cxl_priv.h
> > >  create mode 100644 include/cxl/passthrough.h  create mode 100644
> > > include/uapi/cxl/cxl_regs.h  create mode 100644
> > > tools/testing/selftests/vfio/vfio_cxl_type2_test.c
> > >
> > > base-commit: 90cf2e0d702c8a132ccbe72e7687f33c04c14658
> >
> > Hi Manish,
> > b4 seems to be failing with applying on top of this base-commit.
> >
> 
> Sorry about this; The series does not apply against a plain mainline
> tree because it depends on Alejandro's v28 "Type2 device basic support"
> 
> I created this patchseries on `origin/for-7.3/cxl-type2-enabling' with
> base commit as 90cf2e0d702c and applied Alejandro's patch-v28. on this
> tree, I see the patches can be applied correctly.
> 
> I see Srirangan has based his reset series on Linus's tree. I will
> follow similar guidelines for v4.
>