Re: [PATCH] KVM: SVM: make svm_flush_tlb_gva do a full asid flush if NPT enabled

On Thu, Jul 30, 2026 at 10:26:46AM -0600, Tycho Andersen wrote:

Hi Tycho, 

> Ok, makes sense. I found enough hardware yesterday to run ~35 VMs with
> a bit of memory pressure, so smaller is better for me.
I run it on a 2 socket single host with 128c/256t and 1.5TB RAM - not really pushing memory limits. So cannot comment if actually memory pressure is required.


> When you say "200 VM hours", I guess that's an average? Or do you find
> they need to run that long to see the fault?
Most of the times I would say I see the first BSOD on 42VM fleet within first 10-11 hours after experiment's start -> equovalent of 400-500 VM-hours. However if running over a course of several days, average time drops to more like 200-300 VM-hours per crash. Also sometimes they batch up. E.g. a few within 1-2 hours and then nothing for a day.


> Also, how are you detecting the BSOD? I'm just waiting for ssh to stop
> responding and screen capping the VNC, but maybe there's a better way.

Yeah that's a bit if pain. SSH is not always reliable, since I'm somewhat thrashing VMs (also storage under the sql pressure get's somewhat slow with latency spikes up to several seconds in VM) - timeout aren't that uncommon . So what appeared to work better is using SAC over serial.

Here is a script I use. As a bonus it also keeps a track of VMs' uptimes.


---------8<----------------
 
#!/usr/bin/env bash
LOG=/tmp/serial-stress.log
UPTIME_LOG=/tmp/serial-uptimes.log
CYCLE=0
while true; do
    CYCLE=$((CYCLE + 1))
    TS=$(date '+%Y-%m-%d %H:%M:%S')
    OK=0
    FAIL=""
    UPTIMES=""
    for N in $(seq 1 42); do
        UP=$( (sleep 0.2; printf "\r\nid\r\n"; sleep 1.2) | timeout 3 socat UNIX-CONNECT:/tmp/serial-${N}.sock STDIO 2>/dev/null | grep -o "Time since last reboot:.*" | sed 's/Time since last reboot: //' | tr -d '\r')
        if [ -z "$UP" ]; then
            FAIL="$FAIL vm-$N"
        else
            OK=$((OK+1))
            UPTIMES="$UPTIMES vm-$N=$UP"
        fi
    done
    if [ -n "$FAIL" ]; then
        echo "[$TS] cycle=$CYCLE ${OK}/42 FAIL:$FAIL" >> $LOG
    else
        echo "[$TS] cycle=$CYCLE 42/42 OK" >> $LOG
    fi
    if [ $((CYCLE % 10)) -eq 0 ]; then
        echo "[$TS] cycle=$CYCLE$UPTIMES" >> $UPTIME_LOG
    fi
    sleep 5
done
--------8<---------------

most of the time you if vm is listed in the output (I personally redirect it to stress-serial.log file and watch it) as FAILED for 2-3 consequative rounds  - it is a BSOD, then your agent can take a screenshot and confirm.

> Thanks for this as well, I only had hv-tlbflush=on, I'll go ahead and
> adapt my command line to yours.
Not 100% if others do matter, but that's how I run it. Another thing to notice, I'm somehow more successful reproducing BSODs when storage path is stressed. When I stress networking path I get BSOD waaaaay more seldome, and cannot ATM state that these BSODs have the same root cause even thought they also indicate signs of memory corruption/stale cache. 

Hope it helps,
Let me know if you were able to hit a BSOD. With 35 VMs I would expect to see it within first 15 hours or so.


Thanks,
Al