RE: [PATCH] vfio/pci: Restore standard PCI config space in .slot_reset()

> -----Original Message-----
> From: Alex Williamson <[email protected]>
> Sent: 18 August 2026 15:38
> To: Keith Busch <[email protected]>
> Cc: Keith Busch <[email protected]>; [email protected]; linux-
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected]; Shameer Kolothum Thodi
> <[email protected]>
> Subject: Re: [PATCH] vfio/pci: Restore standard PCI config space in .slot_reset()
> 
> External email: Use caution opening links or attachments
> 
> 
> [Cc +Shameer]

Thanks Alex.

> 
> On Mon, 17 Aug 2026 17:21:23 -0600
> Keith Busch <[email protected]> wrote:
> 
> > On Mon, Aug 17, 2026 at 03:50:06PM -0600, Alex Williamson wrote:
> > > > In scenarios where userspace simply needs the underlying hardware
> > > > restored to its known-good baseline state post-reset, having vfio-pci
> > > > implement .slot_reset() and leverage the existing PCI core snapshot via
> > > > pci_restore_state() bridges this gap cleanly.
> > >
> > > Does it though?  Even for a simple reset to initial state we need to
> > > prevent the host and guest stepping on each other across the reset as
> > > well as tear down user modified state, like interrupts.
> >
> > Specifically considering host-guest interactions, I don't think this
> > scenario is handled at all. After 14 years, this is the state of QEMU:
> >
> >   static void vfio_err_notifier_handler(void *opaque)
> >   {
> >       VFIOPCIDevice *vdev = opaque;
> >
> >       if (!event_notifier_test_and_clear(&vdev->err_notifier)) {
> >           return;
> >       }
> >
> >       /*
> >        * TBD. Retrieve the error details and decide what action
> >        * needs to be taken. One of the actions could be to pass
> >        * the error to the guest and have the guest driver recover
> >        * from the error. This requires that PCIe capabilities be
> >        * exposed to the guest. For now, we just terminate the
> >        * guest to contain the error.
> >        */
> >
> >       error_report("%s(%s) Unrecoverable error detected. Please collect any
> data possible and then kill the guest", __func__, vdev->vbasedev.name);
> >
> >       vm_stop(RUN_STATE_INTERNAL_ERROR);
> >   }
> >
> > Is there another common VMM that actually does something useful to
> > continue from this event?
> 
> Not that I'm aware of, the kernel interface really isn't designed for
> recovery, it's designed only to notify.
> 
> > Outside virtualization, I'm more interested in enabling user space
> > DPDK-like drivers. I don't want to break anyone, so starting small here:
> > restoring the config space to the baseline before the device was handed
> > to a user space driver feels right.
> 
> But we have no hand-back-to-userspace mechanism currently.  Alone, it's
> certainly a step towards letting the device run again, but we really
> need more uAPI defined to provide coordination.
> 
> > > > A few questions I'd like to put to the list:
> > > > 1. Since the PCI core already takes responsibility for holding the early
> > > >    config space checkpoint, is calling pci_restore_state() during
> > > >    .slot_reset() the appropriate place for vfio-pci to apply it, or
> > > >    should this be explicitly driven/triggered via a VFIO ioctl?
> > > > 2. Does returning PCI_ERS_RESULT_RECOVERED here create subtle state
> > > >    issues if userspace directly modified config space registers that
> > > >    were not captured in pdev->saved_config_space?
> > > > 3. Should we pair this with an explicit "link restored / reset complete"
> > > >    eventfd notification so userspace knows exactly when it is safe to
> > > >    resume access?
> > > >
> > > > Appreciate any feedback or historical context on how VFIO and PCI error
> > > > recovery should interact here.
> > >
> > > Certainly the host saved state doesn't take into account user
> > > manipulation of the device since the last snapshot.
> >
> > Yeah, the kernel emits an eventfd that an error occured. The user side
> > should have some baseline from which to proceed. It feels outside the
> > scope of user space to save and restore such low level and early
> > initialization things like the PCI BAR config space.
> 
> There's a fair bit of config space the user cannot write, particularly
> BARs, so a VM has an obligation to restore the virtualized BARs to make
> a coherent view of the device, but a userspace driver can't effect a
> meaningful value change of the physical BAR register anyway.
> 
> > Also consider that the user space side may not have even been
> > initialized at the time a PCIe error occured. What happens then?
> 
> I'd tend to think a userspace driver would consider aborting if the
> device is triggering errors before they've even touched it.  Closing
> the device writes back the state saved on open.
> 
> > > But also, vfio-pci error handling is currently limited to generating
> > > an event when a non-recoverable error has occurred.  I don't think we
> > > can nudge it forward in any meaningful way by implementing a
> > > .slot_reset to restore a prior host snapshot.  That misses the user
> > > modified state, coordination across error handling, and may not even
> > > match the hand-off state of the device to the user.
> >
> > I totally agree. Lacking a notification, user space can at best poll
> > something specific to their device, but it'd be better to generically
> > coordinate this sequence with the kernel's error handling.
> >
> > Would it be acceptable to introduce additional eventfd's for each part
> > in the pcie error handling? Privately, I've proposed and tested the
> > user-space component to quiesce on .error_detected, then start from
> > scratch after the .slot_reset. But I still need something to restore the
> > config space, and I feel kernel is the right place to do it.
> 
> Absolutely the kernel needs to provide some restore of the device,
> especially where the user doesn't have access.  We need some mechanism
> for the user to observe the host recovery and know when it can access
> the device again.  I provided my high level vision of that in the link
> I previously shared.  Shameer has also been looking into this and can
> share his plans.

[+Michał, +Satyanarayana]
 
Right. I am working on a series trying to do much of what Alex described
in the other thread.  Currently in internal review and testing.

On the eventfd question, what I have is one recovery eventfd plus a
feature reporting the state and a sequence number, rather than one
eventfd per callback. The state says whether recovery is in progress,
whether the channel was frozen, whether the device was reset, and
whether it failed. A non-fatal recovery can complete before userspace
reacts to the event, so the state and the sequence are more use than
catching each phase live, I think.

In short:

- error_detected() stops being a single "something went wrong" signal.
  It votes on the actual channel state: CAN_RECOVER for a non-fatal
  error, NEED_RESET for a frozen channel, DISCONNECT for a permanent
  failure, and signals userspace that an event has started.
- slot_reset() restores config space with pci_restore_state(), tears
  down user modified interrupt state so it is rebuilt rather than
  resumed, and records that the device was reset for userspace to read.
- resume() puts PCI_COMMAND back, restores the DMA-BUF exports, unblocks
  access and signals the eventfd again. Userspace reads the state from
  the feature to see the event has finished.
- The device is blocked from user access for the whole sequence: config
  space, BARs, faults on mapped BARs, interrupts, DMA-BUF exports,
  runtime PM. That is most of the series, and getting the locking right
  is the hard part here.

It is opt-in. Until userspace installs the recovery eventfd, vfio-pci
behaves exactly as it does today.

With the state visible, QEMU can virtualise the AER capability to the
guest, through the AER interrupt or GHES, so a non-fatal error the host
recovered from need not end in vm_stop().

I plan to post it as an RFC once the merge window closes.
Would appreciate any comments in the meantime.

Thanks,
Shameer