Re: [PATCH v4 11/18] iommu: Restore and reattach preserved domains to devices

On Sat, Aug 08, 2026 at 02:27:16AM +0000, Samiullah Khawaja wrote:
> During default domain setup, restore the preserved domains by restoring
> the page tables using restore() iommupt op. Associated the restored
> domain with the iommu group of the preserved device, and reattach the
> domain to the device.
> 
> Signed-off-by: Samiullah Khawaja <[email protected]>
> ---
>  drivers/iommu/iommu.c            |  76 ++++++++++++++++++
>  drivers/iommu/liveupdate.c       | 130 +++++++++++++++++++++++++++++++
>  include/linux/iommu-liveupdate.h |  69 ++++++++++++++++
>  3 files changed, 275 insertions(+)
> 

../..

> diff --git a/drivers/iommu/liveupdate.c b/drivers/iommu/liveupdate.c
> index 20acf123b47a..04c0212cd81b 100644
> --- a/drivers/iommu/liveupdate.c
> +++ b/drivers/iommu/liveupdate.c
> @@ -708,3 +708,133 @@ void iommu_unpreserve_device(struct iommu_domain *domain, struct device *dev)
>  	liveupdate_flb_put_outgoing(&iommu_flb);
>  }
>  EXPORT_SYMBOL_GPL(iommu_unpreserve_device);
> +
> +static inline bool match_device_ser(struct iommu_device_ser *match,
> +				    struct pci_dev *pdev)
> +{
> +	return match->devid == pci_dev_id(pdev) && match->pci_domain_nr == pci_domain_nr(pdev->bus);
> +}
> +
> +/**
> + * iommu_init_device_preserved_data() - Initialize preserved state for device
> + * @dev: Target device
> + *
> + * Looks up incoming Live Update state for @dev and attaches it to the device if
> + * found.
> + */
> +void iommu_init_device_preserved_data(struct device *dev)
> +{
> +	struct iommu_device_ser *device_ser = NULL;
> +	struct iommu_device_array_ser *array;
> +	struct iommu_flb_obj *flb_obj;
> +	int ret, idx;
> +
> +	if (!dev_is_pci(dev))
> +		return;
> +
> +	ret = iommu_liveupdate_flb_get_incoming(&flb_obj);
> +	if (ret)
> +		return;
> +
> +	mutex_lock(&flb_obj->lock);
> +	array = phys_to_virt(flb_obj->ser->device_array_phys);
> +	iommu_liveupdate_for_each_arr(array) {
> +		iommu_liveupdate_for_each_obj(array, device_ser, idx) {
> +			if (match_device_ser(device_ser, to_pci_dev(dev))) {
> +				device_ser->hdr.flags |= IOMMU_SER_FLAG_INCOMING;
> +				goto out;
> +			}
> +		}
> +	}
> +
> +	device_ser = NULL;
> +out:
> +	WRITE_ONCE(dev->iommu->device_ser, device_ser);
> +	mutex_unlock(&flb_obj->lock);
> +	liveupdate_flb_put_incoming(&iommu_flb);
> +}
> +EXPORT_SYMBOL(iommu_init_device_preserved_data);
> +
> +/**
> + * iommu_release_restored_device() - Release a restored device
> + * @dev: Target device
> + */
> +void iommu_release_restored_device(struct device *dev)
> +{
> +	/*
> +	 * We do not support releasing the restored devices that are not
> +	 * reclaimed by the device drivers as they can fallback to the default
> +	 * domain.
> +	 */
> +	BUG_ON(dev_iommu_restored_state(dev));

Hi,
After a successful live update this is one sysfs write away, and nothing in
the series disarms it.

At PCI probe, iommu_init_device_preserved_data() matches the incoming FLB on
devid + pci_domain_nr and sets IOMMU_SER_FLAG_INCOMING.
Nothing clears the flag or device_ser afterwards. The group is meanwhile owned
on behalf of iommufd (iommu.c:3229-3231, "will be reclaimed later by the
entity (iommufd) that preserved them"), and iommufd_liveupdate_retrieve() is
-EOPNOTSUPP, so the reclaim that would end the restored state cannot happen
yet. The device is left with the state permanently set.

A remove then BUG_ONs, in pci_stop_and_remove_bus_device_locked(), with
pci_rescan_remove_lock held. Any device_del() gets there, so surprise unplug
too, not only sysfs;

Hit with my live-update work for AMD on top of this series, on a preserved
device with no driver bound:

  # echo 1 > /sys/bus/pci/devices/0000:43:00.0/remove

  kernel BUG at drivers/iommu/liveupdate.c:782!
  RIP: 0010:iommu_release_restored_device+0x24/0x30
  Call Trace:
   iommu_release_device+0x36/0x70
   iommu_bus_notifier+0x46/0x60
   ...
   device_del+0x267/0x3c0
   pci_stop_and_remove_bus_device_locked+0x22/0x40
   remove_store+0x8d/0xa0

Two suggestions:

  - Clear the flag and device_ser together when the restored state ends.
	The two helpers partition on that bit.
  - WARN_ON and fall back to the default domain rather than BUG_ON, and either
    way do not die holding pci_rescan_remove_lock.

The same missing clear should also fail a second preserve, since
iommu_preserve_device() returns -EBUSY when device_ser is set.
Reaching that needs a reclaim first, so it may not be live today; static reading, I have not run it.

Thanks,
Ankit

> +}
> +