Re: [linux-next:master] [fs] 313c47f4fe: BUG:kernel_hang_in_test_stage

On Tue, Feb 03, 2026 at 09:30:43AM +0800, Oliver Sang wrote:
> hi, Christian Brauner,
> 
> sorry for late. it cost us some time to double confirm.
> 
> On Sat, Jan 31, 2026 at 12:41:12PM +0100, Christian Brauner wrote:
> > On Fri, Jan 30, 2026 at 05:59:00PM +0100, Christian Brauner wrote:
> > > On Tue, Jan 27, 2026 at 02:26:09PM +0800, kernel test robot wrote:
> > > > 
> > > > 
> > > > Hello,
> > > > 
> > > > kernel test robot noticed "BUG:kernel_hang_in_test_stage" on:
> > > > 
> > > > commit: 313c47f4fe4d07eb2969f429a66ad331fe2b3b6f ("fs: use nullfs unconditionally as the real rootfs")
> > > > https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master
> > > > 
> > > > [test failed on linux-next/master ca3a02fda4da8e2c1cb6baee5d72352e9e2cfaea]
> > > > 
> > > > in testcase: trinity
> > > > version: 
> > > > with following parameters:
> > > > 
> > > > 	runtime: 300s
> > > > 	group: group-00
> > > > 	nr_groups: 5
> > > > 
> > > > 
> > > > 
> > > > config: x86_64-kexec
> > > > compiler: clang-20
> > > > test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 32G
> > > > 
> > > > (please refer to attached dmesg/kmsg for entire log/backtrace)
> > > 
> > > The reproducer doesn't work:
> > > 
> > > ubuntu@pengar:~/data/kernel/linux/MODULES/lkp-tests$ sudo bin/lkp qemu -k ../../vmlinux -m ./modules.cgz job-script # job-script
> > > result_root: /home/ubuntu/.lkp//result/trinity/group-00-5-300s/vm-snb/yocto-x86_64-minimal-20190520.cgz/x86_64-kexec/clang-20/313c47f4fe4d07eb2969f429a66ad331fe2b3b6f/15
> > > downloading initrds ...
> > > skip downloading /home/ubuntu/.lkp/cache/osimage/yocto/yocto-x86_64-minimal-20190520.cgz
> > > 19270 blocks
> > > /usr/bin/wget -q --timeout=3600 --tries=1 --local-encoding=UTF-8 https://download.01.org/0day-ci/lkp-qemu/osimage/pkg/debian-x86_64-20180403.cgz/trinity-static-x86_64-x86_64-1c734c75-1_2020-01-06.cgz -N -P /home/ubuntu/.lkp/cache/osimage/pkg/debian-x86_64-20180403.cgz
> > > Failed to download osimage/pkg/debian-x86_64-20180403.cgz/trinity-static-x86_64-x86_64-1c734c75-1_2020-01-06.cgz
> > > cat: '': No such file or directory
> > > exec command: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -fsdev local,id=test_dev,path=/home/ubuntu/.lkp//result/trinity/group-00-5-300s/vm-snb/yocto-x86_64-minimal-20190520.cgz/x86_64-kexec/clang-20/313c47f4fe4d07eb2969f429a66ad331fe2b3b6f/15,security_model=none -device virtio-9p-pci,fsdev=test_dev,mount_tag=9p/virtfs_mount -kernel ../../vmlinux -append root=/dev/ram0 RESULT_ROOT=/result/trinity/group-00-5-300s/vm-snb/yocto-x86_64-minimal-20190520.cgz/x86_64-kexec/clang-20/313c47f4fe4d07eb2969f429a66ad331fe2b3b6f/0 BOOT_IMAGE=/pkg/linux/x86_64-kexec/clang-20/313c47f4fe4d07eb2969f429a66ad331fe2b3b6f/vmlinuz-6.19.0-rc1-00006-g313c47f4fe4d branch=internal-devel/devel-hourly-20260124-050739 job=/lkp/jobs/scheduled/vm-meta-17/trinity-group-00-5-300s-yocto-x86_64-minimal-20190520.cgz-313c47f4fe4d-20260126-53110-19zhjsh-2.yaml user=lkp ARCH=x86_64 kconfig=x86_64-kexec commit=313c47f4fe4d07eb2969f429a66ad331fe2b3b6f intremap=posted_msi watchdog_thresh=240 rcuperf.shutdown=0 rcuscale.shutdown=0 refscale.shutdown=0 audit=0 kunit.enable=0 ia32_emulation=on max_uptime=7200 LKP_LOCAL_RUN=1 selinux=0 debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 net.ifnames=0 printk.devkmsg=on panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 drbd.minor_count=8 systemd.log_level=err ignore_loglevel console=tty0 earlyprintk=ttyS0,115200 console=ttyS0,115200 vga=normal rw  ip=dhcp result_service=9p/virtfs_mount -initrd /home/ubuntu/.lkp/cache/final_initrd -smp 2 -m 12872M -no-reboot -device i6300esb -rtc base=localtime -device e1000,netdev=net0 -netdev user,id=net0 -display none -monitor null -serial stdio
> > > qemu-system-x86_64: Error loading uncompressed kernel without PVH ELF Note
> > > 
> > > The paths for the downloads in the job script are wrong or don't work.
> > > Even if I manually modify the above path I still get in the next step:
> > > 
> > > /usr/bin/wget -q --timeout=3600 --tries=1 --local-encoding=UTF-8 https://download.01.org/0day-ci/lkp-qemu/modules.cgz -N -P /home/ubuntu/.lkp/cache
> > > Failed to download modules.cgz
> > > cat: '': No such file or directory
> > > 
> > > I need a way to reproduce the issue to figure out exactly what is
> > > happening.
> > 
> > Ok, I got it all working and can run the reproducer.
> 
> not sure how you solve it? from above log, the problem is caused by
> trinity-static-x86_64-x86_64-1c734c75-1_2020-01-06.cgz, do you have any log?
> internally, it's a soft link pointing to a debian version, so maybe there are
> some code issue for uploading to https://download.01.org/0day-ci.
> 
> if you could share more information with us, we could check further to improve
> our process and reproducer. thanks a lot!
> 
> we will also check by ourselves, so no problem at all if you ignore this.

I manually edited the job-script file to point to something existing on
the server. If I delete everything that I did and restart the test using
just the parameters and the job-script provided with your link then I
run into the issues I mentioned in my first mail again...

        ubuntu@pengar:~/data/kernel/linux/MODULES/lkp-tests$ sudo bin/lkp qemu -k ~/data/repro.nullfs/lkp/vmlinuz-6.19.0-rc1-00006-g313c47f4fe4d -m ~/data/repro.nullfs/lkp/modules-313c47f4fe4d.cgz job-script
        The approx. disk space requirements are

        10M             simple boot test in rootfs openwrt
        50M             simple boot test in rootfs debian
        1G              plan to run a number of different tests
        100G or more    IO tests

        Please enter a dir with enough disk space, or simply press Enter to accept the default.
        You may still symlink /home/ubuntu/.lkp to a more suitable place in future.
        /home/ubuntu/.lkp => /home/ubuntu/data/lkp/
        ~/data/kernel/linux/MODULES/lkp-tests/programs/lkp-src/pkg ~/data/kernel/linux/MODULES/lkp-tests
        x86_64
        ==> Making package: lkp-src 0-1 (Tue Feb  3 10:27:03 UTC 2026)
        ==> Checking runtime dependencies...
        ==> Checking buildtime dependencies...
        ==> WARNING: Using existing $srcdir/ tree
        ==> Removing existing $pkgdir/ directory...
        ==> Starting build()...
        make: Entering directory '/home/ubuntu/data/kernel/linux/MODULES/lkp-tests/bin/event'
        gcc  -D_FORTIFY_SOURCE=2  -c -o wakeup.o wakeup.c
        gcc  -Wl,-O1,--sort-common,--as-needed,-z,relro -static -o wakeup wakeup.o
        rm -f wakeup.o
        strip wakeup
        make: Leaving directory '/home/ubuntu/data/kernel/linux/MODULES/lkp-tests/bin/event'
        ==> Entering fakeroot environment...
        x86_64
        ==> Starting package()...
        ==> Creating package "lkp-src"...
        10999 blocks
        renamed '/home/ubuntu/.lkp/cache/lkp-x86_64.cgz.tmp' -> '/home/ubuntu/.lkp/cache/lkp-x86_64.cgz'
        ==> Leaving fakeroot environment.
        ==> Finished making: lkp-src 0-1 (Tue Feb  3 10:27:05 UTC 2026)
        ~/data/kernel/linux/MODULES/lkp-tests
        result_root: /home/ubuntu/.lkp//result/trinity/group-00-5-300s/vm-snb/yocto-x86_64-minimal-20190520.cgz/x86_64-kexec/clang-20/313c47f4fe4d07eb2969f429a66ad331fe2b3b6f/0
        downloading initrds ...
        use local modules: /home/ubuntu/.lkp/cache/modules-313c47f4fe4d.cgz
        /usr/bin/wget -q --timeout=3600 --tries=1 --local-encoding=UTF-8 https://download.01.org/0day-ci/lkp-qemu/osimage/yocto/yocto-x86_64-minimal-20190520.cgz -N -P /home/ubuntu/.lkp/cache/osimage/yocto
        19270 blocks
        /usr/bin/wget -q --timeout=3600 --tries=1 --local-encoding=UTF-8 https://download.01.org/0day-ci/lkp-qemu/osimage/pkg/debian-x86_64-20180403.cgz/trinity-static-x86_64-x86_64-1c734c75-1_2020-01-06.cgz -N -P /home/ubuntu/.lkp/cache/osimage/pkg/debian-x86_64-20180403.cgz
        Failed to download osimage/pkg/debian-x86_64-20180403.cgz/trinity-static-x86_64-x86_64-1c734c75-1_2020-01-06.cgz
        cat: '': No such file or directory
        exec command: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -fsdev local,id=test_dev,path=/home/ubuntu/.lkp//result/trinity/group-00-5-300s/vm-snb/yocto-x86_64-minimal-20190520.cgz/x86_64-kexec/clang-20/313c47f4fe4d07eb2969f429a66ad331fe2b3b6f/0,security_model=none -device virtio-9p-pci,fsdev=test_dev,mount_tag=9p/virtfs_mount -kernel /home/ubuntu/data/repro.nullfs/lkp/vmlinuz-6.19.0-rc1-00006-g313c47f4fe4d -append root=/dev/ram0 RESULT_ROOT=/result/trinity/group-00-5-300s/vm-snb/yocto-x86_64-minimal-20190520.cgz/x86_64-kexec/clang-20/313c47f4fe4d07eb2969f429a66ad331fe2b3b6f/0 BOOT_IMAGE=/pkg/linux/x86_64-kexec/clang-20/313c47f4fe4d07eb2969f429a66ad331fe2b3b6f/vmlinuz-6.19.0-rc1-00006-g313c47f4fe4d branch=internal-devel/devel-hourly-20260124-050739 job=/lkp/jobs/scheduled/vm-meta-17/trinity-group-00-5-300s-yocto-x86_64-minimal-20190520.cgz-313c47f4fe4d-20260126-53110-19zhjsh-2.yaml user=lkp ARCH=x86_64 kconfig=x86_64-kexec commit=313c47f4fe4d07eb2969f429a66ad331fe2b3b6f intremap=posted_msi watchdog_thresh=240 rcuperf.shutdown=0 rcuscale.shutdown=0 refscale.shutdown=0 audit=0 kunit.enable=0 ia32_emulation=on max_uptime=7200 LKP_LOCAL_RUN=1 selinux=0 debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 net.ifnames=0 printk.devkmsg=on panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 drbd.minor_count=8 systemd.log_level=err ignore_loglevel console=tty0 earlyprintk=ttyS0,115200 console=ttyS0,115200 vga=normal rw  ip=dhcp result_service=9p/virtfs_mount -initrd /home/ubuntu/.lkp/cache/final_initrd -smp 2 -m 12169M -no-reboot -device i6300esb -rtc base=localtime -device e1000,netdev=net0 -netdev user,id=net0 -display none -monitor null -serial stdio
        early console in setup code
        No EFI environment detected.
        early console in extract_kernel
        input_data: 0x00000000031ee2c4
        input_len: 0x0000000000ce259d
        output: 0x0000000001000000
        output_len: 0x0000000002e81808
        kernel_total_size: 0x0000000002c28000
        needed_size: 0x0000000003000000
        trampoline_32bit: 0x0000000000000000

        Decompressing Linux... Parsing ELF... done.
        Booting the kernel (entry_offset: 0x0000000002718ff0).

So I edit job-script and do:

        # export bm_initrd='/osimage/pkg/debian-x86_64-20180403.cgz/trinity-static-x86_64-x86_64-1c734c75-1_2020-01-06.cgz'
        export bm_initrd='/osimage/pkg/debian-13-x86_64-20250902.cgz/trinity-x86_64-294c4652-1_20251011.cgz'

but that doesn't work because the glibc version is too old for the
trinity test thing. So I switch to another bm_initrd:

        # original: export bm_initrd='/osimage/pkg/debian-x86_64-20180403.cgz/trinity-static-x86_64-x86_64-1c734c75-1_2020-01-06.cgz'
        # export bm_initrd='/osimage/pkg/debian-13-x86_64-20250902.cgz/trinity-x86_64-294c4652-1_20251011.cgz'
        export bm_initrd='/osimage/pkg/yocto-x86_64-minimal-20190520.cgz/trinity-static-x86_64-x86_64-1c734c75-1_2020-01-06.cgz'

And that one got it working and I managed to reproduce the issue.

The tests executes random VFS system calls including pivot_root(). I
added debugging output into that system call:

And then it becomes clear:

        [   21.185641][ T5251] VFS: BEFORE PIVOT ROOT FROM /var/volatile/tmp to /var/volatile/tmp
        [   21.185645][ T5251] pivot_root: overmounts from nullfs BEFORE PIVOT ROOT (nullfs):
        [   21.185646][ T5251]   [0] ffff88816a4e3c80 (rootfs)
        [   21.185709][ T5251] VFS: AFTER PIVOT ROOT FROM / to /
        [   21.192478][ T5251] pivot_root: overmounts from nullfs AFTER PIVOT ROOT (nullfs):
        [   21.192480][ T5251]   [0] ffff88816a4e2d80 (tmpfs)
        [   21.201027][ T5251]   [1] ffff88816a4e3c80 (rootfs)

        <snip>

        [   29.328721][ T5250] VFS: BEFORE PIVOT ROOT FROM /var/volatile/tmp to /var/volatile/tmp
        [   29.331584][ T5250] pivot_root: overmounts from nullfs BEFORE PIVOT ROOT (nullfs):
        [   29.334168][ T5250]   [0] ffff88810ca52300 (rootfs)
        [   29.335742][ T5250] VFS: AFTER PIVOT ROOT FROM / to /
        [   29.337399][ T5250] pivot_root: overmounts from nullfs AFTER PIVOT ROOT (nullfs):
        [   29.339935][ T5250]   [0] ffff88811efba300 (tmpfs)
        [   29.341133][ T5250]   [1] ffff88810ca52300 (rootfs)

        <snip>

        [   30.507784][ T1768] /lkp/lkp/src/monitors/meminfo: line 45: date: not found
        [   30.507784][ T1768] /lkp/lkp/src/monitors/meminfo: line 46: cat: not found
        [   30.507784][ T1768] /lkp/lkp/src/monitors/meminfo: line 25: /lkp/lkp/src/bin/event/wait: not found

During random system call execution
pivot_root("/var/volatile/tmp", "/var/volatile/tmp")
is called. This makes the "/var/volatile/tmp" tmpfs mount the rootfs for
everyone and mounts the old rootfs on top of the new rootfs. That means
as soon as anything is called that relies on binaries that are located
in the old rootfs they won't find it anymore as the fs root of all
tasks has been set to /var/volatile/tmp

Before nullfs that pivot_root() call would have failed because the
initramfs mount had no parent mount. nullfs makes that finally work.

I would like to try and enable nullfs unconditional before we resort to
making it a boot option. pivot_root() is inherently destructive for a
test setup so I would just do:

diff --git a/syscalls/pivot_root.c b/syscalls/pivot_root.c
index 3a33fcc5..13c00b07 100644
--- a/syscalls/pivot_root.c
+++ b/syscalls/pivot_root.c
@@ -11,4 +11,5 @@ struct syscallentry syscall_pivot_root = {
        .arg2name = "put_old",
        .arg2type = ARG_ADDRESS,
        .group = GROUP_VFS,
+       .flags = AVOID_SYSCALL, /* May end up switching everyone's rootfs. */
 };

You would see the same problem if instead of running from the initramfs
mount you'd be running from a separate rootfs. In other words running
these tests with a rootfs would surface the same error.

I'd appreciate it if you would patch trinity with my diff above (no
attribution needed). I suspect people generally don't run system call
fuzzers in their workloads so I'd like to move forward and only revert
if we have to.