Re: Cross compiling user applications for armv7

I actually build system from the current main branch.
How do I set up poudriere in such a case? 


Regards,
Michał Kruszewski


Sent with Proton Mail secure email.

On Sunday, September 14th, 2025 at 7:40 AM, Mark Millard <[email protected]> wrote:

> On Sep 13, 2025, at 22:00, Warner Losh [email protected] wrote:
> 
> > On Sat, Sep 13, 2025 at 9:46 PM Mark Millard [email protected] wrote:
> > 
> > On Sep 13, 2025, at 17:07, Warner Losh [email protected] wrote:
> > 
> > > On Sat, Sep 13, 2025 at 10:45 AM Michał Kruszewski [email protected] wrote:
> > > 
> > > > Ok, let's start from the beginning because I feel a little bit lost.
> > > > The buildenv concept is ok when you have no dependencies.
> > > > However, I have.
> > > > 
> > > > I see 3 potential options.
> > > > 1. Use arm64 as an intermediate platform.
> > > > This is unacceptable approach for me.
> > > > 2. Setup poudriere for armv7 manually.
> > > > I tried for 2 days and simply failed.
> > > > 3. Setup qemu, and install dependencies and build custom code inside VM.
> > > > 
> > > > Solutions 2 and 3 are acceptable for me.
> > > > I guess 2 is better than 3, as it should be more performant.
> > > > However, I have no idea how to implement idea 2 or 3 in practice.
> > > > 
> > > > I also don't understand why setting up poudriere for armv7 is so hard.
> > > > What is missing by default that I can't do it with a single command?
> > > 
> > > You can do it with a single command, just not from binaries because our armv7 packaging was always just for embedded, so we didn't do the right thing for normal installs.
> > 
> > If one is okay with use of 14.3 pkgbase materials as the source for
> > a armv7 poudriere-devel jail on amd64,
> > 
> > # poudriere jail -c -jarmv7 -aarmv7 -U https://pkg.freebsd.org -mpkgbase=base_release_3 -v 14
> > 
> > should produce a poudriere-devel jail targeting armv7 that is made from pre-built
> > binaries (pkgbase packages to be installed).
> > 
> > The above does presume that emulators/qemu-user-static (or
> > emulators/qemu-user-static-devel ?) has been installed
> > already and that binmiscctl has been run so that armv7 has
> > been set up.
> > 
> > > However, https://forums.freebsd.org/threads/poudriere-image-creation-for-arm-architecture.91870/ has a good recipe for the source build. Have you tried it.
> > > 
> > > poudriere jail -c -x -j armv7 -m git+https -a arm.armv7 -v releng/14.3 -K GENERIC
> > > poudriere ports -c -m git+https -B main -p HEAD
> > 
> > "man 8 poudriere-jail" no longer lists -x . Instead it lists:
> > 
> > -X Do not build and setup native-xtools cross compile tools
> > in jail when building for a different TARGET ARCH than
> > the host. Only applies if TARGET_ARCH and HOST_ARCH are
> > different.
> > 
> > Hmmm. I don't recall what it.
> > 
> > > are the commands to setup the root and to get ports so you can do a poudriere bulk to build the ports you are interested in.
> > > 
> > > > Why the things that are missing are so hard to recreate manually?
> > > 
> > > So to do #2 manually... The following blog, something I wrote long ago, but that should still work
> > > https://wiki.freebsd.org/WarnerLosh/A_Brief_Guide_To_Cross_Compiling_FreeBSD
> > > has the steps (though you'll need to s/armeb/armv7/ in it). There's three commands to creating an image. But the results are almost the same as the above the first poudriere command while being less flexible or performant.
> > > 
> > > As for #3, it's a bit less performant, but you can run FreeBSD inside of qemu using the virt* devices easily enough (though I think there's a bug with 14.3's kernel so you have to use em NIC not virtnet):
> > > 
> > > https://bsdimp.blogspot.com/2023/12/freebsdarmv7-in-qemu.html
> > > 
> > > has a writeup I did a while ago. It's still accurate and suggests the virtnet bug may be fixed in all 14.x releases. I run this setup from time to time still to test things in a armv7 system. It's OK if what you are building is small, but can get tedious if you have lots of complex dependencies.
> > > 
> > > Hope this helps.
> > 
> > I'll note that my historical (2017..2020-May) problem with
> > poudriere-devel use involving qemu-user-static with
> > poudriere buidlers has been incorrect qemu-user-static
> > operation, especially processes that hang up (e.g, uwait)
> > in the poudriere builders --that work fine on real hardare.
> > It was not getting the combination set up in the first
> > place that was the issue. In mid 2020-May I gave up trying
> > to involve qemu-user-static, which also stopped my
> > interactions with Kyle E. in the subject area.
> > 
> > Kyle and I have fixed a lot of these... Rust has been hit or miss... The rest seem to be OK...
> 
> 
> I tried to build poudriere-devel (and implicitly pkg) as armv7
> on amd64 today. It hungup after the builders completed:
> 
> . . .
> [00:12:08] Stopping 5 builders
> [00:12:08] Creating pkg repository
> pkg-static: Both ABI_FILE and OSVERSION are set, ABI_FILE overrides OSVERSION
> pkg-static: Warning: Major OS version upgrade detected. Running "pkg bootstrap -f" recommended
> Creating repository in /tmp/packages: 100%
> Packing files for repository: 0%
> 
> 
> At this point it hung up: STATE uwait
> for a bunch of:
> 
> /usr/local/bin/qemu-arm-static /.p/pkg-static repo -o /tmp/packages /packages{qemu-arm-static}
> 
> 
> NOTE: main 16 can not yet "pkg bootstrap -f" as there
> is no upstream FreeBSD-ports distributed yet.
> 
> > Warner
> > 
> > > Warner
> > > 
> > > > Regards,
> > > > Michał Kruszewski
> > > > 
> > > > Sent with Proton Mail secure email.
> > > > 
> > > > On Saturday, September 13th, 2025 at 6:09 PM, Warner Losh [email protected] wrote:
> > > > 
> > > > > On Sat, Sep 13, 2025 at 9:50 AM Michał Kruszewski [email protected] wrote:
> > > > > 
> > > > > > It is more about how hard it is to setup an environment for cross compiling your own kernel modules or applications for armv7.
> > > > > > I don't really care whether I use Buildroot way of doing things or FreeBSD way.
> > > > > > The difference is that with Buildroot it takes 10 minutes to setup such environment.
> > > > > > With FreeBSD I am struggling for two days.
> > > > > > And the only advice I get is to use arm64 machine.
> > > > > > Well, I don't have any.
> > > > > > And even if I had, I wouldn't use it, because I find this concept deeply "misengineered".
> > > > 
> > > > And it is bad engineering. Mark sometimes gives bad advice that's based on his experience where it was "easier" to do it this way. I just wish I'd replied more quickly.
> > > > 
> > > > > I know that you want to help, and I appreciate it.
> > > > > I just feel frustrated with the complexity of this conceptually trivial task.
> > > > 
> > > > No. I get it. I'm frustrated too since I've been working for years to make it better, and there's still these rough edges that turn people off.
> > > > 
> > > > Warner
> > > > 
> > > > > Regards,
> > > > > Michał Kruszewski
> > > > > 
> > > > > Sent with Proton Mail secure email.
> > > > > 
> > > > > On Saturday, September 13th, 2025 at 4:29 PM, Mark Millard [email protected] wrote:
> > > > > 
> > > > > > On Sep 13, 2025, at 00:20, Michał Kruszewski [email protected] wrote:
> > > > > > 
> > > > > > > Basically you are saying that I should have a separate physical arm64 machine for doing embedded armv7 development on FreeBSD.
> > > > > > > If I understand this correctly, I just can't understand how can people say that FreeBSD is embedded friendly.
> > > > > > > This is so much complex and convoluted compared to just using, for example, Buildroot on Linux.
> > > > > > > This also doesn't feel like a *nix way of solving things.
> > > > > > > I thought that after cross compiling world and kernel, there is some shell script that I can simply source to start cross-development for target platform.
> > > > > > 
> > > > > > I found this AI based note about Buildroot's way of
> > > > > > putting cross-build toolchains to use:
> > > > > > 
> > > > > > QUOTE
> > > > > > Buildroot pre-configures and compiles all necessarily
> > > > > > packages during the build process to create a custom
> > > > > > root filesystem.
> > > > > > . . .
> > > > > > Buildroot focuses on static builds, meaing all selected
> > > > > > packages are compiled and integrated into the final root
> > > > > > filesystem image during the build process. There is no
> > > > > > concept of installing or updating packages on the target
> > > > > > after the initial build.
> > > > > > END QUOTE
> > > > > > 
> > > > > > This is certainly not what the FreeBSD ports tree is
> > > > > > set up for. Buildroot or analogous does not exist for
> > > > > > FreeBSD --or most *BSD or Unix-- to my knowledge.
> > > > > > 
> > > > > > May be NetBSD's pkgsrc cross build support:
> > > > > > 
> > > > > > https://cdn.netbsd.org/pub/pkgsrc/current/pkgsrc/doc/HOWTO-use-crosscompile
> > > > > > 
> > > > > > would be closer to how you like to work? (I've never
> > > > > > used such.)
> > > > > > 
> > > > > > > Regards,
> > > > > > > Michał Kruszewski
> > > > > > > 
> > > > > > > Sent with Proton Mail secure email.
> > > > > > > 
> > > > > > > On Saturday, September 13th, 2025 at 12:23 AM, Mark Millard [email protected] wrote:
> > > > > > > 
> > > > > > > > On Sep 12, 2025, at 14:10, Michał Kruszewski [email protected] wrote:
> > > > > > > > 
> > > > > > > > > A simple way is to set up an armv7 chroot / jail on an arm64 host such as a Raspberry Pi (Apple devices don't work!).
> > > > > > > > > Then it's just like a native environment, but usually much faster.
> > > > > > > > 
> > > > > > > > > I don't have physical arm64 host.
> > > > > > > > > I also don't want to create a VM.
> > > > > > > > > The whole idea seems weird and unnatural.
> > > > > > > > 
> > > > > > > > Ignoring the RPi4B detail:
> > > > > > > > 
> > > > > > > > FreeBSD official armv7 port-packages are built this way
> > > > > > > > on arm64 hardware that natively supports user-space
> > > > > > > > armv7 code. (ampere* systems are used.)
> > > > > > > > 
> > > > > > > > FreeBSD official i386 port-packages are built this way
> > > > > > > > on amd64 hardware that natively supports user-space
> > > > > > > > i386 code.
> > > > > > > > 
> > > > > > > > No use of qemu variants of any kind: no attempt at
> > > > > > > > non-native-capable host environments.
> > > > > > > > 
> > > > > > > > (Back when amd64 and qemu was used for the likes
> > > > > > > > of armv7, lots of stuff failed to build mcuh of
> > > > > > > > the time over the years --stuff that builds
> > > > > > > > just fine now. This was abandoned after native
> > > > > > > > became available. armv6 was always qemu based
> > > > > > > > on amd64 and could not build much as of the last
> > > > > > > > time a build was run.)
> > > > > > > > 
> > > > > > > > > It sounds more like an exotic workaround.
> > > > > > > > 
> > > > > > > > It is the official technique used for what FreeBSD
> > > > > > > > distributes for armv7 port-packages.
> > > > > > > > 
> > > > > > > > > Regards,
> > > > > > > > > Michał Kruszewski
> > > > > > > > > 
> > > > > > > > > Sent with Proton Mail secure email.
> > > > > > > > > 
> > > > > > > > > On Friday, September 12th, 2025 at 10:46 PM, Robert Clausecker [email protected] wrote:
> > > > > > > > > 
> > > > > > > > > > Hi Michał,
> > > > > > > > > > 
> > > > > > > > > > A simple way is to set up an armv7 chroot / jail on an arm64 host such as
> > > > > > > > > > a Raspberry Pi (Apple devices don't work!). Then it's just like a native
> > > > > > > > > > environment, but usually much faster.
> > > > > > > > > > 
> > > > > > > > > > Yours,
> > > > > > > > > > Robert Clausecker
> > > > > > > > > > 
> > > > > > > > > > Am Fri, Sep 12, 2025 at 04:19:42PM +0000 schrieb Michał Kruszewski:
> > > > > > > > > > 
> > > > > > > > > > > Is there any tutorial on how to cross compile custom user application for armv7?
> > > > > > > > > > > I struggle t find any.
> > > > > > > > > > > Cross compiling the system is pretty easy.
> > > > > > > > > > > However, how can one cross compile custom user application or kernel drivers.
> > > > > > > > > > > 
> > > > > > > > > > > Regards,
> > > > > > > > > > > Michał Kruszewski
> > > > > > > > > > > 
> > > > > > > > > > > Sent with Proton Mail secure email.
> > > > > > > > > > 
> > > > > > > > > > --
> > > > > > > > > > () ascii ribbon campaign - for an encoding-agnostic world
> > > > > > > > > > /\ - against html email - against proprietary attachments
> > > > > > > > 
> > > > > > > > ===
> > > > > > > > Mark Millard
> > > > > > > > marklmi at yahoo.com
> > > > > > 
> > > > > > ===
> > > > > > Mark Millard
> > > > > > marklmi at yahoo.com
> 
> 
> 
> 
> ===
> Mark Millard
> marklmi at yahoo.com