[RFC] RISC-V KVM: guest local sfence.vma may miss stale VS-stage TLB after vCPU migration

Hi all,

I would like to ask whether RISC-V KVM has a gap around guest sfence.vma
and VS-stage TLB invalidation when a vCPU migrates across host CPUs.

On bare metal, Linux uses mm_cpumask in __flush_tlb_range() to decide
whether to send a local or remote sfence.vma. That works because the cpumask
reflects physical CPUs. In a guest, however, the kernel only sees vCPUs, so
from the guest point of view a task may never "migrate" even though the
underlying vCPU moved to a different host CPU. In other words, guest
mm_cpumask is effectively a vCPU mask, not a host CPU mask.

There is also a related switch_mm()/set_mm_asid() aspect. With the ASID
allocator enabled, RISC-V Linux does not flush on every context switch. A
local_flush_tlb_all() is only done when a deferred ASID rollover flush is
pending for the current CPU. That is fine on bare metal, because previous
shootdowns are also based on physical CPUs. In a guest, however, the earlier
shootdown may have been only a guest-local sfence.vma and may have missed an
old host CPU where the same vCPU ran previously.

A simplified sequence is:

1. A guest task with mm A runs on vcpu0 while vcpu0 is scheduled on host
   CPU1. CPU1 fills a VS-stage TLB entry for that mm/ASID.
2. vcpu0 migrates to host CPU0. KVM's vCPU migration sanitization is local
   to the current host CPU, so it does not invalidate the VS-stage TLB entry
   that was left on host CPU1.
3. The guest later updates mm A's page table from vcpu0, for example due to
   COW. Since the guest still sees mm A as having run only on guest CPU0,
   __flush_tlb_range() can choose a local sfence.vma. That local sfence.vma
   only affects host CPU0, where vcpu0 is currently running, so the stale
   VS-stage entry on host CPU1 remains.
4. Later, the task is scheduled again while vcpu0 is running on host CPU1.
   If the ASID is still valid and no deferred ASID rollover flush is pending,
   set_mm_asid() may not issue any flush. The access on host CPU1 can then
   hit the old VS-stage TLB entry.

We hit this on a XiangShan RISC-V system. A bash task in the guest was
scheduled on vcpu0 while that vCPU was running on host CPU1. A read-only
page was prefetched and cached in CPU1's VS-stage TLB. Later the same vCPU
migrated to host CPU0, the guest triggered COW and updated the mapping, and
the guest still issued only a local sfence.vma. The stale VS-stage entry on
CPU1 was not invalidated. After running for some time, the task later
accessed that mapping while running on host CPU1 again and hit the stale
translation. The data in that page was a pointer from the GOT, so it
dereferenced to NULL and eventually caused a segmentation fault.

I checked current upstream RISC-V KVM (v7.0.2) and I do not see HSTATUS.VTVM
being enabled, so guest sfence.vma is not trapped. I found
kvm_riscv_local_tlb_sanitize() on host CPU migration, which handles G-stage
VMID entries on the current host CPU, but I do not think it generally solves
the stale VS-stage TLB case described above.

I also noticed the vendor-specific kvm_riscv_vsstage_tlb_no_gpa path in
current upstream. It looks like a workaround for a particular implementation
issue on Andes AX66, where the VS-stage TLB does not cache guest physical
address and VMID, so the normal VMID/GPA-based invalidation model is
insufficient. In that sense, it does not seem to be a generic solution for
guest sfence.vma coherence across vCPU migration.

I am wondering what the right direction should be. Enabling HSTATUS.VTVM and
trapping guest sfence.vma would allow KVM to translate guest local sfence.vma
into the required host-side hfence.vvma, but that may be too heavy for the
common case. Another possible direction may be to make the current
Andes-specific vCPU migration flush logic more generic, so implementations
that need VS-stage TLB sanitization on vCPU migration can opt in through a
common mechanism. A third possibility may be to define a lightweight SBI
interface for guest local sfence.vma, so Linux could use it in
__flush_tlb_range() for the local case when running under KVM, and KVM could
then decide whether a local sfence.vma is sufficient or whether host-side
hfence.vvma is also needed.

My questions are:
1. Is the behavior above expected on current RISC-V KVM?
2. If not, what would be the preferred fix direction?
3. Should this be handled by trapping guest sfence.vma with VTVM, by
   generalizing the existing vCPU-migration VS-stage TLB flush logic, or by
   adding a lighter SBI-mediated path for guest local sfence.vma?
4. Or is there another intended mechanism to keep VS-stage TLB state coherent
   across vCPU migration?

If needed, I can send a reproducer and the exact hardware/kernel details.

Thanks,
Yaxing Guo

-- 
kvm-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kvm-riscv