Re: qemu-user viability (was Re: [SECURITY] [DSA 5983-1] qemu security update)

On 8/25/25 04:05, John Paul Adrian Glaubitz wrote:
> Hello,
> 
> On Sun, 2025-08-24 at 13:07 -0500, Rob Landley wrote:
>> On 8/23/25 09:19, Thorsten Glaser wrote:
>>>> There are no alternatives - qemu is unique in this regard.  And
>>>> it has never been designed for this usage.  What we had for 15+
>>>> years, unnoticed, is like `chmod u+s /bin/sh`, which is never
>>>> supposed to be used like this.
>>>
>>> Perhaps, but there’s shades in between.
>>
>> I find qemu system emulation a LOT less problematic.
>>
>> For sh4 I boot qemu-system-sh4 and then use a network block device to
>> provide swap (so the 64mb limitation of the board isn't a limiting
>> factor). The sh4 build in toybox's mkroot works fine for this (binaries
>> at https://landley.net/bin/mkroot if you'd like to try). It also works
>> with -hda but I think can only provide _one_ of those so you have to
>> partition it, which I generally don't bother.
> 
> Building packages with just 64 MB is nearly impossible which is why qemu-system
> is not really suitable for my usecase.

Which is why I add swap space as described above.

> There are plans to add Goldfish support
> for sh4 similar to m68k but I don't know when this is going to happen.

That would be nice, but you can already give the r2d board a gig of swap 
if you want to. (I usually just added enough to bring it up to par with 
the other build nodes, which was 256 megs for 32 bit and 512 for 64 bit 
systems, but the page tables don't get that big if you add more. It 
thrashed a little in places but single processor builds completed, and 
if you set the swap up right all the thrashing was within the host's ram 
so not that bad.)

Plus, if you're doing the distcc trick the heavy lifting of compilation 
gets moved outside the emulator anyway, so it can actually use _less_ 
memory. (Well, for C programs, the horror that is C++ always finds new 
ways to suck and that often moves half of compiling into a single 
threaded linker callback plugin because of course it does, but making 
stupid C++ builds _fast_ was never really an option. And "benefits less" 
doesn't mean a distcc setup would break it, just wouldn't use distcc and 
fall back to the native compiler instead.)

Distcc 1) runs the preprocessor locally, 2) sends the preprocessed .c 
file through the network to a "remote compute node" which sends back the 
.o file (or error output), and then 3) links the result locally. This 
means the build THINKS it's fully native even when the remote node is 
actually (from its perspective) running a cross compiler: there's only 
one set of headers, one set of libraries, one type of binary being built 
(modulo -fpic but that's not new). In this case "remote" meant 10.0.2.2 
passing through to 127.0.0.1 on the host, the script that set it up was 
at 
https://github.com/landley/aboriginal/blob/master/sources/toys/dev-environment.sh#L46 
and yes that was last touched 11 years ago so my memory's a bit hazy.

I was mostly using that to get parallel builds out of UP boards, though. 
The sweet spot on "normal" boards was -j3, beyond that the preprocessing 
and data transmission through the virtual network card bottlenecked the 
process. (I don't remember if I tried it with sh4, it's been a while.)

What I _meant_ to do was set up a "compute cluster" of vms that could be 
told what package to build next, install any missing prerequisite 
packages to their local filesystem from the repository, fetch, build, 
and upload the new package to the repo, rinse repeat. Never got around 
to it (because I was trying to make it work with gentoo, and gentoo 
turned out to have some SERIOUS design flaws when it came to being 
architecture agnostic in ANY WAY. Every single ebuild in the tree is 
tagged with every architecture it supports, so adding a new architecture 
means touching every package in the tree! No really!)

>>> Yes. People expect to be able to run foreign-arch chroots.
>>> Entire buildd setups partly rely on this, too…
>>
>> A qemu-system vm doesn't require any weird translation. As far as the
>> kernel running in the emulator is concerned, everything is entirely
>> native. :)
> 
> Did you ever try building GCC or WebKit with qemu-system for sh4?

gcc yes, webkit no.

I natively built the whole of Linux From Scratch and bits of Beyond 
Linux From Scratch (an x11 demo system running xeyes and xchess and so 
on) with my aboriginal linux setup, which was designed to work the same 
way on all architectures (host agnostic, the scripts all just built 
natively for whatever the current system was).

https://landley.net/aboriginal/control-images/

https://github.com/landley/control-images/tree/master/images/lfs-bootstrap/mnt

But again, last touched 11 years ago and the beyond linux from scratch 
demo I did for qualcomm/hexagon was 2011 (all that x11 build stuff, for 
a board that didn't have a graphics card so all the windows were 
displaying remote through the network anyway). Happy to answer 
questions, but I may have to work bits of it out again.

My general intention to repeat this in my mkroot project is why 
https://landley.net/bin/toolchains/latest/ has *-native.sqf next to each 
*-cross.tar.gz (the VM can mount the squashfs to pull in a basic native 
toolchain without really having to integrate it into a given build vm 
environment) and also why I did 
https://github.com/landley/toybox/blob/master/mkroot/packages/lfs-sources 
(which hit the problem that some of the patches in LFS 12 sources are 
diff -c format not diff -u format and my patch didn't understand how to 
apply those, it's on the todo list).

And part of the reason to use network block devices for this is if two 
different qemu instances try to share the same -hdb file.img it does a 
stupid file locking thing and the second one refuses to launch, even 
when it's read only, and no there is no way to disable this without 
patching qemu. But the nbd-server I wrote for toybox doesn't care. Yes I 
tend to run multiple architectures in parallel, that's why 
https://github.com/landley/toybox/blob/master/mkroot/testroot.sh#L57 is 
doing a copy of the sqf file for each instance.

Seriously: this worked fine for me years ago. I just wasn't messing with 
debian much back then, and have always found the developer side of 
debian... "insufficiently documented".

> Adrian

Rob