Re: [RFC PATCH 00/11] igb: Add experimental VF live migration support

On 7/28/26 07:50, Akihiko Odaki wrote:
> On 2026/07/28 2:35, Cédric Le Goater wrote:
>> Akihiko,
>>
>> On 7/27/26 07:39, Cédric Le Goater wrote:
>>> Hello,
>>>
>>> Live migration of VFIO-passthrough devices - SR-IOV VFs, vGPUs - is a
>>> growing requirement, but real hardware with migration support is
>>> scarce and hard to debug. An emulated device provides a fully
>>> controlled testbed for developing and validating the entire software
>>> stack - vfio-pci variant drivers, VFIO core migration v2 framework,
>>> QEMU, libvirt - and for tuning complex migration policies such as
>>> downtime convergence. It also serves as an educational reference for
>>> understanding VFIO migration end-to-end, from device state
>>> serialization to dirty page tracking.
> 
> This is another good example of using QEMU as a SR-IOV testbed.
> 
>>>
>>> This series adds an experimental VF live migration interface to the
>>> emulated igb (82576) device. It enables a vfio-pci variant driver
>>> (igb-vfio-pci) to migrate VFs using the standard VFIO migration v2
>>> protocol with stop-copy and pre-copy support.
> I suggest looking into virtio-net as an alternative to igb. It would allow avoiding lots of complexities of igb.
> 
>>>
>>> The target scenario is nested virtualization:
>>>
>>>    L0 QEMU (these patches)
>>>      igb PF with x-vf-migration=on
>>>      └── VFs with migration BAR + vendor cap
>>>
>>>    L1 kernel
>>>      igb-vfio-pci variant driver [1]
>>>      translates VFIO migration v2 ioctls → BAR2 MMIO
>>>
>>>    L1 QEMU (stock, unmodified)
>>>      vfio-pci device model, standard migration fd
>>>
>>>    L2 guest
>>>      standard igbvf driver, unaware of migration
>>>
>>> The L1 QEMU is completely unmodified -- it sees a standard VFIO
>>> migratable device and uses the normal migration fd path.
>>>
>>> * Design
>>>
>>> The migration interface is exposed through a hidden 64KB PCI BAR
>>> (BAR2) on each VF, discovered via a vendor-specific PCI capability
>>> ("MIGB", PCI_CAP_ID_VNDR). The BAR exposes a register-based state
>>> machine that mirrors VFIO migration states (RUNNING, STOP, STOP_COPY,
>>> RESUMING, PRE_COPY).
>>>
>>> Device state is serialized as a versioned blob of per-VF register
>>> (offset, value) pairs covering control, interrupt, RX/TX queue,
>>> receive address (RA/RA2), etc. plus TX context descriptors and
>>> VFRE/VFTE enable bits. The blob is transferred via DMA through the PF
>>> device, since VFIO owns the VF's IOMMU domain and the variant driver
>>> maps its DMA buffers through the PF.
>>>
>>> Dirty page tracking is implemented with per-range bitmaps maintained
>>> in IGBCore. All VF DMA paths in igb_core.c (TX data, RX data,
>>> descriptor writeback) are instrumented to record touched pages. The
>>> variant driver registers tracked IOVA ranges and queries dirty bitmaps
>>> through a DMA shared buffer, using a single MMIO doorbell
>>> (DIRTY_CTRL=QUERY) per iteration.
>>>
>>> * Caveats
>>>
>>> Patch 1 introduces PCI_BASE_ADDRESS_MEM_ALWAYS_ON, a QEMU-internal
>>> BAR flag that keeps the migration BAR mapped even after VFIO's
>>> Function Level Reset clears PCI_COMMAND_MEMORY.
>>>
>>> The x-vf-migration property is experimental (x- prefix, default off)
>>> and the migration BAR register interface may change.
>>>
>>> The dirty bitmaps are maintained inside the device, which is not
>>> realistic for discrete NICs without on-chip DRAM.
>>>
>>> * Testing
>>>
>>> The target scenario is nested virtualization: L0 runs QEMU with an
>>> igb PF (x-vf-migration=on), L1 runs the igb-vfio-pci variant driver
>>> and an unmodified QEMU, and L2 runs a standard igbvf driver.
>>>
>>> Migration under iperf3 load works correctly: dirty page tracking
>>> converges (from ~2000 pages per PRE_COPY iteration down to ~280 at
>>> STOP_COPY), and STOP_COPY stays under 250ms.
>>>
>>> * Todo/Ideas
>>>
>>>    1. Add migration blocker when x-vf-migration=on (no VMState yet) or
>>>       add VMState support for L0 migration (dirty bitmaps, tracking
>>>       engines, migration BAR registers, stats)
> 
> If you have AI assistance, why don't you just add migration blocker? It should be just one prompt away and not more troublesome than leaving this "Todo". ;)
> 
>>>    2. Add PRE_COPY match data validation (magic, version, caps)
>>>    3. Support driver-provided DMA bitmaps per dirty range
>>>       (DIRTY_RANGE_ADDR_LO/HI)
>>>    4. Add QMP/HMP knobs for error injection and capability tuning:
>>>      . Dirty rate throttling: artificial delay or rate limit on DMA
>>>        dirty tracking to simulate different convergence scenarios
>>>      . State blob size inflation: pad the migration blob to stress
>>>        large-state transfers and test DATA_SIZE limits
>>>      . Migration phase timing: expose per-VF counters for time spent
>>>        in each state (PRE_COPY duration, STOP_COPY latency)
>>>      . Hot page simulation: mark specific page ranges as always-dirty
>>>        to test worst-case convergence
>>>      . Error injection: force STATUS error codes, fail DMA transfers,
>>>        corrupt state blobs, mask CAPS bits, etc.
>>>    5. Add qtests for migration state machine transitions, dirty page
>>>       tracking ?
>>>
>>> * Credits
>>>
>>> Alex Williamson suggested the overall approach: a hidden migration BAR
>>> discovered via a vendor-specific PCI capability, the "vf-migration"
>>> device property to gate the feature. Thanks for the ever ongoing
>>> support and valuable discussions throughout these years.
>>>
>>> * AI disclaimer
>>>
>>> The lack of a migration-capable device has been a recurring pain point
>>> for VFIO development over the years, and we hope this proposal
>>> demonstrates the value of having one.
>>>
>>> Claude was used to analyze the IGB PF and VF internal state and
>>> identify the pain points of a working live migration of such devices.
>>> The generated code served as a starting point but *significant* time
>>> was then spent cleaning up, reworking, and shaping it into a clear,
>>> reviewable proposal. As QEMU does not yet accept AI-assisted
>>> contributions, this series is submitted as an RFC.
>>>
>>> Thanks,
>>>
>>> C.
>>>
>>> [1] https://github.com/legoater/vfio-pci-extras
>>>
>>> Cédric Le Goater (11):
>>>    pci: Add PCI_BASE_ADDRESS_MEM_ALWAYS_ON BAR flag
>>>    igb: Add x-vf-migration property and vendor-specific capability for
>>>      IGBVF
>>>    igb: Add migration BAR with state machine
>>>    igb: Add VF state serialization for live migration
>>>    igb: Add VF post-load fixups for live migration
>>>    igb: Add dirty page tracking for IGBVF migration
>>>    igb: Quiesce VFs on STOP and include PF enable state in migration blob
>>>    igb: Fix post-migration RX ring deadlock
>>>    igb: Send RARP after VF migration to update bridge FDB
>>>    docs: Add igb VF migration testing setup guide
>>>    igb: Add migration statistics registers to VF migration BAR
>>>
>>>   MAINTAINERS                           |    6 +
>>>   docs/system/device-emulation.rst      |    1 +
>>>   docs/system/devices/igb-migration.rst |  313 +++++++
>>>   docs/system/devices/igb.rst           |    6 +
>>>   hw/net/igb_common.h                   |   11 +
>>>   hw/net/igb_core.h                     |    8 +
>>>   hw/net/igb_migration.h                |  201 +++++
>>>   include/hw/pci/pci.h                  |    6 +
>>>   hw/net/igb.c                          |   18 +
>>>   hw/net/igb_core.c                     |  144 ++-
>>>   hw/net/igb_migration.c                | 1194 +++++++++++++++++++++++++
>>>   hw/net/igbvf.c                        |   29 +-
>>>   hw/pci/pci.c                          |    6 +-
>>>   hw/net/meson.build                    |    2 +-
>>>   hw/net/trace-events                   |   20 +
>>>   15 files changed, 1939 insertions(+), 26 deletions(-)
>>>   create mode 100644 docs/system/devices/igb-migration.rst
>>>   create mode 100644 hw/net/igb_migration.h
>>>   create mode 100644 hw/net/igb_migration.c
>>>
>>
>> First thanks for the quick review ! I will recap here :
>>
>> * bisectability, will improve the first patches.
>> * state serialization, the blob management is awful. I was expecting
>>    flames. Yes. it needs a rework. I lack a qbuf-style put/get API. The
>>    contents of the blob need a review. Seems feasible.
>> * relocation between VFs, on my TODO. The code already checks the vfn but
>>    needs tightening and better support.
>> * interrupts bits: will look into it. hopefully, the model "only" needs
>>    to be more precise. This was a difficult aspect of the igb to understand
>>    and I am still learning.
>> * dirty tracking, I worked on 2 different interfaces and some of the code
>>    fell through the cracks. DMA failures, bitmap clearing, size validation,
>>    all need improvements. Mostly addressed already, remaining items are code
>>    reorg. should be fine.
>> * quiesce: looks like a bug in the model. Will check.
>> * RARP : I got inspiration from the tests. you are right saying it belongs
>>    to the management layer. Keeping it for now as it simplifies testing, but
>>    will move it out.
>> * Stats: the addition is from yesterday. Needs fixes indeed.
> 
> Thanks for the summary. Well, the "AI disclaimer" states that you spent significant time on cleanup, but your recap highlights several critical issues.

The issues are mostly related to the IGB VF state. It was expected.
Dirty tracking issues reflect my various attempts of using MMIOs and
shared buffer. The rest are bugs :)

As for AI, yes, I find it useful to draft code, but really, it
requires a lot of directions and control. At the end, some parts
are not elegant or just wrong for some reasons. I don't trust the
output.

Where it is really good at, and where it saved me a lot of time,
is in analyzing QEMU and kernel logs of the L0/L1/L2 software stack.
Extracting the relevant info, correlating events, doing the timing
analysis, adding/removing logs. I did similar work in the past on
nested PPC emulation, and it was a painful and time-consuming process
without this kind of assistance.

Once the core debug is done, you can use it to produce python
scripts to automate the process and use that for the follow ups.
> 
> Ideally, AI assistance should minimize manual labor while keeping quality high. > Right now, the workflow seems inverted: the AI created the basic scaffolding, you spent significant time reworking it, 
> yet critical flaws not mentioned in the "Caveats" and "Todo/Ideas" sections still remain. I believe we can do better with AI assistance.

Your review uncovered several important issues in the interrupt
handling that I hadn't expected Claude (Opus 4.6) to miss.
  
> In fact, I am part of the Codex for Open Source program and use GPT-5.6 Sol Ultra extensively for reviews. It is quite effective at raising quality while cutting down manual effort. Most of the comments I provided came directly from Codex using a simple prompt: "review each commit." The model somehow possesses a latent understanding of QEMU requirements, allowing it to flag issues like bisectability breaks and lack of L0/L1/L2 boundary enforcement.

All were good findings.

> I expect that iterating with prompts like "review each commit" and "fix these issues" would automatically resolve most of these problems, reducing error-prone human intervention. I am not sure if Claude has the same depth of virtualization-specific knowledge, but even if it doesn't, you can feed your recap back into the model to let it handle the heavy lifting.

yes. I will try to fix most of the issues with the current model
before moving the the next version.

> Moving forward, I see two potential directions for this work:
> 
> - Focus on high-level design: Explore alternative architectures like hacking virtio-net SR-IOV or keeping RARP out of QEMU. You could send the results as a fresh RFC while openly noting the rough edges. This allows us to discuss the design architecture without spending massive effort polishing the implementation details.

Alex proposed good directions for a new HW implementation. I will
keep the igb vf as a target for now. Hopefully, I will have time
to fix the igb vf serialization and the interrupt handling for
better support. All issues seems feasible. Takes time.

virtio-net SR-IOV could be next.

As for RARP, yes. I will add a disclaimer saying that we need it
for tests. It's a extra patch any how.

  
> - Focus on code quality and advocacy: Burn more tokens to improve the code quality. Use this series as a concrete case study to demonstrate that AI assistance can successfully implement complex virtualization features. This could help shift project policy regarding AI usage and pave the way for upstreaming this and future AI-assisted patches.

Yes. that's one aspect.

Regarding the VFIO subsystem, improving testing is necessary and
it won't happen without a complex framework that people can use.

> 
> Whichever direction you choose, reframing the narrative in the cover letter will better align the series with the project's goals.
It's upstream first of course :)

Thanks,

C.