Re: Steps towards live migration

Jakub Růžička <[email protected]> Tue, 08 Jul 2025 15:49:48 +0200
Newsgroups	dev.linux.lists.coconut-svsm
Message-ID	<[email protected]>
On Mon Jun 30, 2025 at 2:09 PM CEST, Pankaj Gupta wrote:
> +CC [John & Joerg]
>
> Hi Jacob,
>
> I briefly looked at your code. I have some questions below to understand 
> your design more. Do you have a comprehensive design document somewhere?
>

There is no comprehensive design document at the moment.

>> Hi,
>> 
>> as mentioned during the last SVSM development talk, I am currently writing my
>> thesis on live migration of confidential guests. With this email I would like to
>> start a discussion so that the same problem is not addressed multiple times
>
> Ofcourse. That's the reason we shared our design in the kvm forum talk 
> and asked for collaboration on common components (open vendor agnostic 
> problems):
>
> https://kvm-forum.qemu.org/2024/SNP_Live_Migration_KVM_forum_2024_svDwxa3.pdf
>
>
>> independently. I also welcome any feedback to make the code written useful for
>> SVSM Coconut (not only for the thesis). Development is done on machine with
>> SEV-SNP support. I made a pull request with the SVSM patches[1] to ease the
>> discussion.
>> 
>> Here is a summary of progress to date:
>> 
>> All validated guest pages can be transferred from the source SVSM to the target
>> SVSM. Currently, no packaging (no confidentiality) is performed, but a hash of
>> all transferred/received blocks is computed that can be compared to verify that
>> the channel is working correctly. Also, the main migration function of the
>
> Any analysis on algorithm you plan to use for this? IIUC you are 
> currently sharing the hash from source to destination? Wouldn't the 
> 'aes-gcm' in [4] will do that in addition to encryption?

Yes, it would, but it was not used when I sent the email.

>> source and destination SVSM is busy waiting for a signal from QEMU to start
>> outbound or incoming migration.
>
>  > > A single shared page called MigrationPage is used for data transfer and
>> communication with QEMU. The migration page contains two registers and a buffer:
>> a status register, a data register, and a data buffer. The status register is
>> used to signal a change in status (e.g., migration starts, migration is
>> complete). The data register is used to signal that a new page has been prepared
>> in the data buffer by the provider or processed by the consumer. The roles of
>> provider and consumer are switched between SVSM and QEMU on the source and
>> destination machines.
>
> This looks like a communication channel between Qemu and SVSM. We are 
> re-using few bits in per CPU ghcb page for SVSM <-> host commands 
> communication.

The communication protocol description is not yet available for reading, or?

>
> I can understand you are not using additional vCPUs. All this can be too 
> much work for guest general purpose vCPUs.
>
>> 
>> The QEMU patch[2] implements communication with the SVSM migration handler and
>> block transfer from source to destination. The idea is that creating the
>> communication channel is all the hypervisor should do, the rest should be done
>> in SVSM.
>> 
>> Current plan for the future in order of realisation:
>> 
>> (1) A function that puts all hosted vCPUs (except the migration handler) into a
>
> [...]
>
>>      spinning state. The hypervisor is not trusted, so the SVSM must be able to
>>      ensure that it is not running any vCPUs. For this task, I consider two-phase
>>      checkpointing[3].
>
> Can you please elaborate more this.

At the beginning of the back-out phase, all guest vCPUs except the migration
handler should be stopped. To ensure that the hypervisor does not let any of the
vCPUs start, I want all vCPUs to enter a wait loop so that even if the
hypervisor allowed the vCPUs to start, no guest code would be run.

>> (2) Migrate the machine from the source to the destination with all vCPUs
>>      stopped by the function from (1).
>
> You mean black-out phase?

Yes.

>> (3) Secret key establishment - though about using [5].
>
> We need to tackle this problem. A probable collaboration point, maybe 
> coupled with attestation and migration key sharing.
>
>> (4) Package the pages - authenticated encryption using [4].
>> (5) Dirty page tracking.
>> (6) Move the SVSM migration handler on an extra vCPU.
>> (7) Start migration handler on signal instead of busy-waiting loop.
>
> There still are other open questions at more granular level. Can discuss 
> those as well, once I go through your complete design.
>
> But at higher level, I would like to reuse more of the existing 
> functionality in Qemu for live migration and use SVSM for memory 
> packaging, guests memory permission setting at VMPL0, and live migration 
> sanity related tasks.

By reuse more of the existing functionality. Do you mean using the current
migration in QEMU and registering different SaveVMHandlers?

What's the plan for tracking dirty sites? I understood from the slides of the
last kvm forum that this task should be done in SVSM. Have you already
implemented this? If so, are you using dirty page tracking in KVM or is it
implemented in SVSM.

Regards,
Jakub

> Best regards,
> Pankaj
>
>> 
>> Best regards,
>> Jakub
>> 
>> [1] https://github.com/coconut-svsm/svsm/pull/745
>> [2] https://github.com/coconut-svsm/qemu/pull/23
>> [3] https://ipads.se.sjtu.edu.cn/_media/publications/sgxmigration-dsn17.pdf
>> [4] https://github.com/RustCrypto/AEADs/blob/master/aes-gcm/src/lib.rs
>> [5] https://github.com/nihalpasham/static-dh-ecdh
signature.asc (application/pgp-signature, 228 B)