Re: Cross compiling user applications for armv7

[Gack: I typed "16" in places that I intended to reference "15"
for the status of a bootable armv7 FreeBSD kernel.]

On Sep 21, 2025, at 08:04, Mark Millard <[email protected]> wrote:

> On Sep 21, 2025, at 03:08, Michał Kruszewski <[email protected]> wrote:
> 
>> I have finally managed to cross compile packages.
>> It looks like the only working configuration is as follows:
>> 1. The system must be installed as ROOT.
>> 2. BUILD_AS_NON_ROOT in poudriere config must be set to NO.
>> No other configuration works, and I tested a couple of them.
> 
> Hopefully Bryan D. and/or Baptiste D. will reply to my earlier
> question about use of pkg's INSTALL_AS_USER with poudriere(-devel).
> 
>> There is still one problem.
>> The poudriere process freezes on:
>> Packing files for repository:   0%
> 
> So your context replicated the qemu hangup that I reported to
> Warner from my attempt at doing a tiny build [poudriere(-devel)
> itself, with pkg also building]. What version of FreeBSD was
> involved in the replication. (Mine was a main 16 boot
> kernel and world and poudriere jail world.)
> 
> QUOTE from the exchange with Warner:
>>> I'll note that my historical (2017..2020-May) problem with
>>> poudriere-devel use involving qemu-user-static with
>>> poudriere buidlers has been incorrect qemu-user-static
>>> operation, especially processes that hang up (e.g, uwait)
>>> in the poudriere builders --that work fine on real hardare.
>>> It was not getting the combination set up in the first
>>> place that was the issue. In mid 2020-May I gave up trying
>>> to involve qemu-user-static, which also stopped my
>>> interactions with Kyle E. in the subject area.
>>> 
>>> On Sep 13, 2025, at 17:07, Warner Losh <[email protected]> wrote:
>>>> Kyle and I have fixed a lot of these... Rust has been hit or miss... The rest seem to be OK...
>> 
>> I tried to build poudriere-devel (and implicitly pkg) as armv7
>> on amd64 today. It hungup after the builders completed:
>> 
>> . . .
>> [00:12:08] Stopping 5 builders
>> [00:12:08] Creating pkg repository
>> pkg-static: Both ABI_FILE and OSVERSION are set, ABI_FILE overrides OSVERSION
>> pkg-static: Warning: Major OS version upgrade detected.  Running "pkg bootstrap -f" recommended
>> Creating repository in /tmp/packages: 100%
>> Packing files for repository:   0%
>> 
>> 
>> At this point it hung up: STATE uwait
>> for a bunch of:
>> 
>> /usr/local/bin/qemu-arm-static /.p/pkg-static repo -o /tmp/packages /packages{qemu-arm-static}
>> 
>> 
>> NOTE: main 16 can not yet "pkg bootstrap -f" as there
>>     is no upstream FreeBSD-ports distributed yet.
> END QUOTE
> 
> Such historical qemu problems majorly contributed
> to why I use hardware that is armv7 capable for
> user-mode code for builds targeting armv7.
> 
>> I do all of this to evaluate FreeBSD as a potential replacement for Linux for embedded projects.
>> I already gathered some experience and have some questions that maybe some of you will be able to answer.
>> 
>> Is it possible to cross compile FreeBSD system for embedded target without root privileges?
>> If you work on your personal PC having a root privileges is not a problem.
>> However, I would like to enable multiple users to cross compile on a server machine.
>> I don't want to grant root privileges to all of them.
>> I know that I can configure doas or sudo.
>> However, this is a lot of manual tweaking.
>> Buildroot is capable of generating embedded Linux system without root privileges.
> 
> I do not know if there is any plan for 
> poudriere(-devel) to support use of pkg's
> INSTALL_AS_USER vs. not. (I did not find
> evidence that it now does.)
> 
> NOTE:
> Warner found in his experiments that the
> INSTALL_AS_USER assignment has to be via
> the pkg command having a -o INSTALL_AS_USER=
> on the pkg command line.
> 
>> How to cross compile packages for a newer system version than the host machine?
>> In embedded projects it is very common that the target platform utilizes newer version of the kernel/system than the host machine.
> 
> Additional question: over what timescale might the need
> for armv7 support to last?
> 
> While lib32 support for armv7 may last indefinitely,
> including whatever in the kernel is required for that,
> I'm unclear on the long term status of either of:
> 
> ) chroot into an armv7 world
> ) jails based on an armv7 world
> 
> after freebsd 16.* (FreeBSD 17+).

I should have referenced: 15.* and 16+

> 
> poudiere(-devel) based port package builds depends on:
> jails.
> 
> If jails were not to be supported indefinitely, could you
> get by with targeting FreeBSD 16.*-RELEASE's indefinitely
> but not FreeBSD 17 or later? Something to think about.
> 
>> Cross compiling newer FreeBSD kernel and world is easy.
> 
> While FreeBSD 16.* will have armv7 kernels, after that

I should have referenced: 15.*

> has no such guarantee --and seems unlikely for the kernel
> to support booting atively, given that even for FreeBSD
> 16.*-RELEASE armv7 is the last and only 32-bit boot

I should have referenced: 15.*-RELEASE

> context supported by a FreeBSD 16.* kernel.

Again, the above should have referenced: 15.*

> If the support time frame is spanned by FreeBSD 16.* then

Again, the above should have referenced: 15.*

> this might not be a problem. But it you need the support
> after that, ???
> 
>> However, how to compile packages?
>> You can't create a jail utilizing a newer system version.
>> I tried building packages for 16-CURRENT and the process simply failed.
> 
> If  the jail is executing a newer world, then the
> jail is executing code that may attempt to use new
> kernel services or depend on new kernel behavior.
> Such is not supported.
> 
>> Poudriere even warned me that it is going to fail.
>> 
>> Cross-compiling packages using poudriere for armv7 is slow, it is extremely slow.
> 
> Such historical performance contributed some to
> why I use hardware that is armv7 capable for
> user-mode code --and possibly would have even
> without the historical hangup problems mentioned
> earlier.
> 
>> I utilize 11 cores for building.
>> Cross compiling Python, I don't count kernel and world compilation, takes approximately 1.5 times more than compiling the whole Linux based system plus Python using Buildroot.
>> Is there anything I can do about that?
> 
> If python bootstraps partially via the system C/C++
> toolchain, that part would likely not be a major
> contributor. But if much of its activity is based
> on executing some subset of phython itself, it is
> going to be qemu emulation executing the python
> interpreter: 2 layers of emulation/interpretation.
> 
> If you wanted a comparison/contrast to armv7
> native-capable aarch64 timing, I've access to
> the following that still seem to be available
> for purchase (as new, via official channels):
> 
> RPi3B
> RPi4B's
> RPi5B's (using an EDK2 draft for booting)
> SolidRun Honeycomb (16 cortex-A72 cores)
> 
> I could build a set of port packages from
> scratch, one one of those gathering and
> reporting time.
> 
> But, based on your prior notes, I'm guessing
> you have no interest in this route or type
> of information.




===
Mark Millard
marklmi at yahoo.com