Re: [PATCH v4 02/21] iommufd: Add iommufd_sw_map_msi()

Hi Jason,

On Thu, Aug 20, 2026 at 07:09:08PM -0300, Jason Gunthorpe wrote:
> On Thu, Aug 20, 2026 at 11:41:31PM +0200, Andrew Jones wrote:
> > Add a descriptor-free counterpart to iommufd_sw_msi(). The existing
> > function is tied to a struct msi_desc and stores the result in the
> > descriptor. This variant returns the IOVA directly so callers can
> > pre-map MSI targets before any descriptor has been allocated.
> 
> I haven't grasped why it is like this?
> 
> The cover letter says:
> 
>   ARM can map a fixed doorbell PA per ITS and cache its IOVA on the
>   descriptor because affinity changes only hardware routing, not the
>   composed address. An IMSIC target PA changes with affinity, and MSI
>   composition may run in atomic context, so mapping the selected target on
>   demand is not an option. This series pre-maps the supervisor IMSIC page
>   for every possible CPU into a domain-local PA-to-IOVA table when remapped
>   IRQs are first allocated, allowing composition to select the target with
>   an O(1) lookup. If iommufd replaces a paging domain while IRQs remain
>   allocated, the incoming domain's table is rebuilt before it is attached.
> 
> There are a few confusiong things with this statement:
>  1) Okay the IMSIC PA changes dynamically but since it can be
>     premapped the PAs required is fixed and known. ARM doesn't change
>     the PA dynamically?

ARM's PA is fixed. On affinity change it uses MOVI to instruct the ITS to
change the routing.

>  2) Why do you say mapping on demand is not possible? ARM's
>     iommu_dma_prepare_msi() is not called in an atomic context and
>     does do the iommu mapping.

By "on demand" I mean after IRQ allocation, when selecting a new MSI
target may occur in atomic context. Mapping then is not possible. Like ARM,
this riscv implementation performs all mappings at alloc-irqs time.

> 
> What I rather expected was for riscv to have a PA window that is very
> big and not just one page, eg adjust iommu_dma_prepare_msi() so you
> can pass in the entire PA space that you need for the affinity
> changes. Maybe this is a list of phys_addr_t ?

That's more or less what this series does. There's no big PA window
because the IMSIC PAs aren't generally contiguous. iommu_dma_map_msi()
factors the mapping operation out of iommu_dma_prepare_msi() and returns
each IOVA directly. A batched list API would be possible, but it would
only move the loop.

> 
> Then keep with the ARM flow where everything happens at the same
> times as today. Instead of just mapping one page you map the entire
> list.

The goal of this approach was to match ARM's flow as much as possible
despite the quite different architecture. Here's a table showing how
they match up now.

.------------------------------------------------------------------------------.
| IRQ lifecycle phase | ARM IR / ITS              | RISC-V IR / IMSIC          |
|---------------------|---------------------------|----------------------------|
| IRQ allocation      | ITS allocates an event/   | IMSIC allocates a vector   |
|                     | LPI. The MSI target PA is | on a CPU. The MSI target   |
|                     | the fixed GITS_TRANSLATER | PA is that CPU's IMSIC     |
|                     | address.                  | page.                      |
|                     |                           |                            |
|                     | iommu_dma_prepare_msi()   | iommu_dma_map_msi()        |
|                     | is called from the IR     | is called from the IR      |
|                     | irqdomain .alloc          | irqdomain .alloc           |
|                     | callback. It calls        | callback. It's called once |
|                     | iommu_dma_map_msi() once  | for each possible IMSIC    |
|                     | for the fixed ITS PA.     | PA.                        |
|---------------------|---------------------------|----------------------------|
| MSI mapping setup   | iommu_dma_map_msi()       | iommu_dma_map_msi()        |
|                     | dispatches to             | dispatches to              |
|                     | iommu_dma_sw_map_msi()    | iommu_dma_sw_map_msi()     |
|                     | or iommufd_sw_map_msi().  | or iommufd_sw_map_msi().   |
|                     | One IOVA is cached in the | One IOVA per IMSIC PA is   |
|                     | MSI descriptor.           | cached in the IOMMU domain |
|                     |                           | table.                     |
|---------------------|---------------------------|----------------------------|
| Initial composition | Compose the fixed target  | Compose the selected CPU's |
|                     | IOVA plus the event ID.   | IMSIC PA and local ID,     |
|                     |                           | then substitute its IOVA.  |
|---------------------|---------------------------|----------------------------|
| Affinity change     | Send MOVI to change ITS-  | Allocate a vector on the   |
|                     | internal routing. The     | new CPU. The device's MSI  |
|                     | device MSI address does   | address changes to the new |
|                     | not change.               | CPU's IMSIC IOVA.          |
|                     |                           |                            |
|                     |                           | The parent IMSIC irqdomain |
|                     |                           | irq_set_affinity() selects |
|                     |                           | the new vector and         |
|                     |                           | recomposes from the top of |
|                     |                           | the hierarchy. The IR      |
|                     |                           | irqdomain's                |
|                     |                           | irq_compose_msi_msg()      |
|                     |                           | translates the new IMSIC   |
|                     |                           | PA to its IOVA.            |
|---------------------|---------------------------|----------------------------|
| Message update      | No new device MSI message | Recompose and rewrite the  |
|                     | is required.              | device MSI message through |
|                     |                           | the IRQ hierarchy.         |
|---------------------|---------------------------|----------------------------|
| Composition context | No IOMMU mapping or       | Composition may occur in   |
|                     | lookup is required during | atomic context, so all     |
|                     | an affinity change.       | IMSIC PAs must already be  |
|                     |                           | mapped and lookup cannot   |
|                     |                           | sleep.                     |
|---------------------|---------------------------|----------------------------|
| Domain replacement  | iommufd_group_setup_msi() | iommufd_group_setup_msi()  |
|                     | installs the one required | installs all required MSI  |
|                     | MSI mapping before attach.| mappings before attach.    |
|                     | The descriptor's cached   | riscv_iommu_ir_attach_     |
|                     | IOVA stays valid.         | paging_domain() builds the |
|                     |                           | new domain's PA-to-IOVA    |
|                     |                           | lookup table.              |
|---------------------|---------------------------|----------------------------|
| Domain destruction  | The ITS MSI mapping is    | The IMSIC mappings and     |
|                     | released with the domain. | PA-to-IOVA table are       |
|                     |                           | released with the domain.  |
.------------------------------------------------------------------------------.

Hopefully that helps describe the design better.

Thanks,
drew