Re: [RFC PATCH 00/11] igb: Add experimental VF live migration support

On 7/27/26 22:36, Alex Williamson wrote:
> On Mon, 27 Jul 2026 07:39:24 +0200
> Cédric Le Goater <[email protected]> wrote:
> 
>> Hello,
>>
>> Live migration of VFIO-passthrough devices - SR-IOV VFs, vGPUs - is a
>> growing requirement, but real hardware with migration support is
>> scarce and hard to debug. An emulated device provides a fully
>> controlled testbed for developing and validating the entire software
>> stack - vfio-pci variant drivers, VFIO core migration v2 framework,
>> QEMU, libvirt - and for tuning complex migration policies such as
>> downtime convergence. It also serves as an educational reference for
>> understanding VFIO migration end-to-end, from device state
>> serialization to dirty page tracking.
>>
>> This series adds an experimental VF live migration interface to the
>> emulated igb (82576) device. It enables a vfio-pci variant driver
>> (igb-vfio-pci) to migrate VFs using the standard VFIO migration v2
>> protocol with stop-copy and pre-copy support.
>>
>> The target scenario is nested virtualization:
>>
>>    L0 QEMU (these patches)
>>      igb PF with x-vf-migration=on
>>      └── VFs with migration BAR + vendor cap
>>
>>    L1 kernel
>>      igb-vfio-pci variant driver [1]
>>      translates VFIO migration v2 ioctls → BAR2 MMIO
>>
>>    L1 QEMU (stock, unmodified)
>>      vfio-pci device model, standard migration fd
>>
>>    L2 guest
>>      standard igbvf driver, unaware of migration
>>
>> The L1 QEMU is completely unmodified -- it sees a standard VFIO
>> migratable device and uses the normal migration fd path.
>>
>> * Design
>>
>> The migration interface is exposed through a hidden 64KB PCI BAR
>> (BAR2) on each VF, discovered via a vendor-specific PCI capability
>> ("MIGB", PCI_CAP_ID_VNDR). The BAR exposes a register-based state
>> machine that mirrors VFIO migration states (RUNNING, STOP, STOP_COPY,
>> RESUMING, PRE_COPY).
> 
> I think you're placing the migration BAR on the VF in order to
> implement this in a small footprint, QEMU + vfio-pci variant driver,
> without PF guest driver changes.  

yes.

> A model that better matches real
> world hardware might be to put the migration BAR on the PF, segmented
> per VF, and then have the PF driver vend those segments out to the VF
> drivers.  

True.

On the migration topic, I saw that the SR-IOV specs had "VF Migration
State Array" feature, which was deprecated.

> That would remove the BAR always mapped problem, 

That's the main problem today.

> but expands
> the footprint to include the PF driver.  However, we're not exactly
> clean with respect to the PF driver as implemented here when we're
> going around the PF driver's back to setup DMA mappings.

That's not uncommon today. The PDS vfio-pci variant driver setups DMA
mappings in the PF to migrate the VF. But yes, given that the IGB PF
has no idea that a VF could be migrated, it's a bit of a resource hijack.

> Can we take advantage of the fact that this is a virtual device to
> avoid all these warts?
> 
> For example, do we really need MMIO BAR space for the register set
> exposed or can we prune that down to some key registers and doorbells
> and move the rest to memory?  We can put the vendor capability in
> extended config space to give ourselves more room to work with if
> necessary.  

There is plenty of space in the extended config space. The required
register set is relatively small.

> We also don't really need to play by the physical rules for
> access, the variant driver in the L1 kernel can allocate contiguous
> ranges and write GPAs into config space registers.  

yes.

> L0 QEMU can just
> write migration data and dirty bitmaps directly to those GPAs,
> bypassing any pretense of DMA mapping.

yes that works.

A previous implementation of this proposal was allocating GPAs.
I then preferred a solution that was more PCI friendly. Anyhow,
it's not a problem to change it again, once we agree on the
HW interface.

> There might be some tricks we can steal from virtio as it seems to
> optionally honor things like vIOMMUs as well. 

Yes. That's where the PCI DMAs mapping were interesting. I will
look at it.

> Anyway, if we want to
> confine the implementation to the virtual VF, avoiding dependencies on
> the PF driver, both at the cross-driver API and device DMA state, I
> think we can probably lean harder on QEMU being able to push data into
> an arbitrary GPA regardless of the IO topology we're exposing.  

So, the extra PCI BAR is indeed a problem and we should switch to
extended config space to avoid it. I have been exploring several
ideas and this proposal is an hybrid MMIO/shared buffer solution.


Here are my plans for the next.

Reduce the register set to a minimum and use a shared buffer for
all commands. The driver would allocate a single large enough buffer
for the largest command payload (DIRTY_QUERY with bitmap) and write
its GPA into BUF_ADDR once at init. Each operation is: fill buffer,
kick CTRL, poll STATUS.


Registers in extended config space :

   0x00    Header        Cap ID
   0x04    CAPS          Features, max_ranges, pgsizes
   0x08    CTRL          Doorbell (state transitions, dirty ops)
   0x0C    STATUS        Completion + error code
   0x10    BUF_ADDR_LO   Shared buffer GPA low
   0x14    BUF_ADDR_HI   Shared buffer GPA high

Commands and payload :

   1. SET_STATE
   0x00    target_state    driver      Requested VFIO state

   2. SAVE
   0x00    data_size       device      Bytes written  (32-bit)
   0x04    reserved        -           May be more reserved bytes to align data
                                       on 32bits
   0x08    data[]          device      State blob

   3. LOAD
   0x00    data_size       driver      Bytes to load (32-bit)
   0x04    reserved        -           May be more reserved
   0x08    data[]          driver      State blob

   4. DIRTY_ENABLE
   0x00    iova            driver      Range start (64-bit)
   0x08    size            driver      Range size (64-bit)
   0x10    pgsize          driver      Page granularity
   0x14    reserved        -

   5. DIRTY_DISABLE

   6. DIRTY_QUERY
   Request (driver):
   0x00    iova            driver      Query range start (64-bit)
   0x08    size            driver      Query range size (64-bit)
   0x10    pgsize          driver      Page granularity (for queries)
   0x14    reserved        -

   Response (device):
   0x18    bitmap_size     device      Bytes in bitmap
   0x1C    dirty_pages     device      Set bits count
   0x20    dma_writes      device      DMA writes since enable (64-bit)
   0x28    reserved        -
   0x30    bitmap[]        device      Dirty page bitmap

   7. GET_STATS
   0x00    dma_writes      device      (32-bit)
   0x04    dma_bytes       device      (64-bit)
   0x0C    dirty_pages_set device
   0x10    dirty_pages_clr device
   0x14    dirty_page_count device
   0x18    dirty_query_cnt device


We could add a shared buffer header (not sure this is useful though)

   0x00    command         driver      Operation to perform
   0x04    status          device      Completion + error code
   0x08    payload_size    both        Size of payload following header
   0x0C    reserved        -           Alignment

Do we want 'flags' anywhere ?

Thanks,

C.