[PATCH v3 0/9] cxl: Support mixed-granularity region interleaves

v3 note for RobertR,
I appreciate the testing and tag offered in v2, but I did not keep it,
because as you see here in v3, the changes make that tag stale. I
appreciate if in addition to reviewing you can retest. Also, I'm not
certain on the tags I did give you. I used "Originally-by" but if 
Co-developed-by: with your Signed-off-by: is the more correct choice,
let me know.

Changes in v3:
- Reduce scope to coarse-to-fine ordering (RichardC, Sashiko)
- Reject out-of-order layouts (RichardC, Sashiko)
- P1: Factor the selector walk and per-port checks into helpers (RobertR)
  That pre-work is the new Patch 1/8 and the Series Structure section of
  this cover letter describe same below.
- P3: Derive granularity from the highest available selector span (RobertR)
- P3: Validate auto-programmed granularity against the derived value
- P4: Rename root_pos_stride() to root_positions_per_target() (RobertR)
- P4: Drop the "stride" terminology throughout (RobertR)
- The complex Patch 4 of v2 is split into patches 4-6 in v3.
  Also noted in the Series Structure section of cover letter below.
- P9: Move the peer-distance walk and per-function walkthroughs from the doc
  into in-code comments and kernel-doc (RobertR)
- Drop Reviewed-by tags
Link to v2:
https://lore.kernel.org/linux-cxl/[email protected]/

Changes in v2:
- Patch 1,2: Defer the unused selector var store to keep P1 bisectable (Sashiko)
- Patch 1: Make divide by 3 in get_selctor() work on 32-bit builds) (lkp)
- Patch 4: Use local vars in cxl_region_attach() for readability (DaveJ)
- Patch 5: Add NULL checks on unused mock arrays (Sashiko)
- Resolved errant err_rch unwind with rc7 merge (DaveJ)
- Rebase onto 7.1-rc7
- Update commit logs in 1,2,5 to align w changes in v2
Link to v1:
https://lore.kernel.org/all/[email protected]/

Begin Cover Letter:

A CXL region interleaves across decoder levels (root, optional
switches, endpoint). CXL Spec 4.0 Section 9.13.1 requires only
that each level use a different, consecutive range of HPA bits to
select its target. The driver has historically required equal
region and root decoder granularities. That blocks the legal
mixed-granularity arrangements permitted by Section 9.13.1, and
makes the 6-way and 12-way configurations defined in Section
9.13.1.1 (Tables 9-6, 9-7, and 9-8) impossible to create.

Two prior proposals addressed parts of this gap:

AlisonS added position arithmetic and sysfs gating to allow
auto and user-created regions for the 6-way and 12-way
configurations that have no same-granularity alternative:
https://lore.kernel.org/all/[email protected]/

RobertR introduced an HPA selector-bit model to allow multi-level
regions regardless of granularity ordering for auto regions:
https://lore.kernel.org/all/[email protected]/

This series combines those two approaches. It extends Robert's
selector-bit model from auto regions to user-created regions,
extends AlisonS's position-arithmetic and gating work to the mixed-
granularity layouts, and adds the remaining validation needed for
both paths.

When the region granularity is finer than the root decoder's (a
mixed-granularity region), each interleaving level must be finer than
its nearest interleaving ancestor -- coarse-to-fine from the root
down. Arrangements that reverse direction are rejected with -ENXIO on
both the auto and user-created paths. Same-granularity regions are
unaffected; their levels legitimately coarsen away from the root.

This is the full range the position arithmetic represents, and a
superset of the prior behavior, which accepted only
region_gran == root_gran. On user-created regions the driver derives
each level's granularity coarse-to-fine, so an out-of-order
arrangement cannot be expressed and is refused by the selector-fit
checks; the coarse-to-fine ordering gate is what guards
firmware-programmed (auto) layouts, where the granularities arrive
already set.

Series structure - changed in v3
----------------
Patch 1 factors the per-port target calculations into helpers with no
functional change. Patches 2 and 3 validate the interleave selector
bits and derive each port's granularity from them. Patch 4 generalizes
the position arithmetic for mixed-granularity layouts. Patches 5, 6,
and 7 allow mixed-granularity regions, enforce coarse-to-fine ordering,
and make the position check fatal. Patch 8 adds cxl_test coverage for
the new layouts. Patch 9 documents the region granularity model and
multi-level interleaving rules.


A companion NDCTL patchset that allows mixed-gran 'cxl create-region'
and adds the unit test is posted here:
https://lore.kernel.org/all/fa5c109f08824180f58341ebd9055545a2ff3142.1780099216.git.alison.schofield@intel.com/


Alison Schofield (9):
  cxl/region: Factor port target calculations
  cxl/region: Validate interleave selector bits
  cxl/region: Derive port granularity from selector bits
  cxl/region: Account for mixed-granularity in position calculations
  cxl/region: Allow mixed-granularity regions
  cxl/region: Enforce coarse-to-fine ordering for mixed-granularity
    regions
  cxl/region: Fail region creation on position check mismatch
  cxl/test: Add a topology to test mixed-granularity regions
  Documentation/cxl: Describe multi-level interleave validation

 Documentation/driver-api/cxl/index.rst        |   1 +
 .../cxl/linux/region-granularity.rst          |  66 ++
 drivers/cxl/core/region.c                     | 580 +++++++++++++-----
 tools/testing/cxl/test/cxl.c                  | 498 +++++++++++++--
 4 files changed, 968 insertions(+), 177 deletions(-)
 create mode 100644 Documentation/driver-api/cxl/linux/region-granularity.rst


base-commit: dc59e4fea9d83f03bad6bddf3fa2e52491777482
-- 
2.37.3