Re: [PATCH rdma-next 00/15] RDMA: Support HW requiring relaxed ordering and add Unordered access flag
Jason Gunthorpe <[email protected]>
| Newsgroups | org.kernel.vger.linux-rdma,org.kernel.vger.netdev |
|---|---|
| Message-ID | <178647669506.282103.2915831899844186795.b4-review@b4> |
> This series prepares the mlx5 stack for hardware that restricts strong- > ordered memory accesses, i.e. HW whose fabric doesn't guarantee memory > operation ordering. The HW may choose to reject creation of an mkey that > doesn't request relaxed ordering, so every mkey the driver creates - > internal and user-facing - has to be handled accordingly. > > The previously added 'ConnectX-10 NVLink-C2C' PCI device is such a device, > as will be any future PCIe Gen7 UIO devices. > > The series does four things: > > 1. Allow optional access flags on all memory registration > The uverbs ABI defines the optional access flags range as best-effort: > drivers must accept any flag in this range and treat support as > opportunistic, never failing registration. Several registration paths > (DM, DEVX UMEM, dmabuf ioctl, reg_mr ioctl) did not allow this range > and would reject such requests. These are fixed to allow newly added > optional access flag added later in the series and any future optional > access flag. > > 2. Groundwork and internal mkeys > Rename the relaxed-ordering IFC fields to the new naming, add the new > ordering capability and mkey-context bits, and consolidate RO-setting > to a single helper in mlx5_core so all consumers share one cap-gated > implementation. Then enable relaxed ordering on the internal mkeys that > don't require strong ordering (resource dump, vfio live-migration data, > ODP null mkey, memory window mkey), so they keep working on strict-RO > HW. > > 3. New Unordered access flag and enforcement > Add IB_ACCESS_UNORDERED, an optional access flag that additionally > allows read-after-write reordering, and wire up mlx5 support for it. > > 4. Enforce no-strong-ordering in user MR registration on HW that requires > it. You should mention the rdma-core series here which provides a full definition for what IB_ACCESS_UNORDERED actually means -- Jason