Re: Cross compiling user applications for armv7

On Sep 13, 2025, at 17:07, Warner Losh <[email protected]> wrote:

> On Sat, Sep 13, 2025 at 10:45 AM Michał Kruszewski <[email protected]> wrote:
>> Ok, let's start from the beginning because I feel a little bit lost.
>> The buildenv concept is ok when you have no dependencies.
>> However, I have.
>> 
>> I see 3 potential options.
>>   1. Use arm64 as an intermediate platform.
>>      This is unacceptable approach for me.
>>   2. Setup poudriere for armv7 manually.
>>      I tried for 2 days and simply failed.
>>   3. Setup qemu, and install dependencies and build custom code inside VM.
>> 
>> Solutions 2 and 3 are acceptable for me.
>> I guess 2 is better than 3, as it should be more performant.
>> However, I have no idea how to implement idea 2 or 3 in practice.
>> 
>> I also don't understand why setting up poudriere for armv7 is so hard.
>> What is missing by default that I can't do it with a single command?
>> 
> You can do it with a single command, just not from binaries because our armv7 packaging was always just for embedded, so we didn't do the right thing for normal installs.

If one is okay with use of 14.3 pkgbase materials as the source for
a armv7 poudriere-devel jail on amd64,

# poudriere jail -c -jarmv7 -aarmv7 -U https://pkg.freebsd.org -mpkgbase=base_release_3 -v 14

should produce a poudriere-devel jail targeting armv7 that is made from pre-built
binaries (pkgbase packages to be installed).

The above does presume that emulators/qemu-user-static (or
emulators/qemu-user-static-devel ?) has been installed
already and that binmiscctl has been run so that armv7 has
been set up.

> However, https://forums.freebsd.org/threads/poudriere-image-creation-for-arm-architecture.91870/ has a good recipe for the source build. Have you tried it.
> 
> poudriere jail -c -x -j armv7 -m git+https -a arm.armv7 -v releng/14.3 -K GENERIC
> poudriere ports -c -m git+https -B main -p HEAD

"man 8 poudriere-jail" no longer lists -x . Instead it lists:

     -X               Do not build and setup native-xtools cross compile tools
                      in jail when building for a different TARGET ARCH than
                      the host.  Only applies if TARGET_ARCH and HOST_ARCH are
                      different.

> are the commands to setup the root and to get ports so you can do a poudriere bulk to build the ports you are interested in.
> 
>> Why the things that are missing are so hard to recreate manually?
> 
> 
> So to do #2 manually... The following blog, something I wrote long ago, but that should still work
> https://wiki.freebsd.org/WarnerLosh/A_Brief_Guide_To_Cross_Compiling_FreeBSD
> has the steps (though you'll need to s/armeb/armv7/ in it). There's three commands to creating an image. But the results are almost the same as the above the first poudriere command while being less flexible or performant.
> 
> As for #3, it's a bit less performant, but you can run FreeBSD inside of qemu using the virt* devices easily enough (though I think there's a bug with 14.3's kernel so you have to use em NIC not virtnet):
> 
> https://bsdimp.blogspot.com/2023/12/freebsdarmv7-in-qemu.html
> 
> has a writeup I did a while ago. It's still accurate and suggests the virtnet bug may be fixed in all 14.x releases. I run this setup from time to time still to test things in a armv7 system. It's OK if what you are building is small, but can get tedious if you have lots of complex dependencies.
> 
> Hope this helps.

I'll note that my historical (2017..2020-May) problem with
poudriere-devel use involving qemu-user-static with
poudriere buidlers has been incorrect qemu-user-static
operation, especially processes that hang up (e.g, uwait)
in the poudriere builders --that work fine on real hardare.
It was not getting the combination set up in the first
place that was the issue. In mid 2020-May I gave up trying
to involve qemu-user-static, which also stopped my
interactions with Kyle E. in the subject area.

> Warner
>  
>> 
>> Regards,
>> Michał Kruszewski
>> 
>> 
>> Sent with Proton Mail secure email. 
>> 
>> On Saturday, September 13th, 2025 at 6:09 PM, Warner Losh <[email protected]> wrote:
>>> 
>>> 
>>> On Sat, Sep 13, 2025 at 9:50 AM Michał Kruszewski <[email protected]> wrote:
>>>> It is more about how hard it is to setup an environment for cross compiling your own kernel modules or applications for armv7.
>>>> I don't really care whether I use Buildroot way of doing things or FreeBSD way.
>>>> The difference is that with Buildroot it takes 10 minutes to setup such environment.
>>>> With FreeBSD I am struggling for two days.
>>>> And the only advice I get is to use arm64 machine.
>>>> Well, I don't have any.
>>>> And even if I had, I wouldn't use it, because I find this concept deeply "misengineered".
>>>> 
>> And it is bad engineering. Mark sometimes gives bad advice that's based on his experience where it was "easier" to do it this way. I just wish I'd replied more quickly.
>>> I know that you want to help, and I appreciate it.
>>> I just feel frustrated with the complexity of this conceptually trivial task.
>> 
>> No. I get it. I'm frustrated too since I've been working for years to make it better, and there's still these rough edges that turn people off.
>> 
>> Warner
>>> Regards,
>>> Michał Kruszewski
>>> 
>>> 
>>> Sent with Proton Mail secure email.
>>> 
>>> On Saturday, September 13th, 2025 at 4:29 PM, Mark Millard <[email protected]> wrote:
>>> 
>>> > 
>>> > On Sep 13, 2025, at 00:20, Michał Kruszewski [email protected] wrote:
>>> > 
>>> > > Basically you are saying that I should have a separate physical arm64 machine for doing embedded armv7 development on FreeBSD.
>>> > > If I understand this correctly, I just can't understand how can people say that FreeBSD is embedded friendly.
>>> > > This is so much complex and convoluted compared to just using, for example, Buildroot on Linux.
>>> > > This also doesn't feel like a *nix way of solving things.
>>> > > I thought that after cross compiling world and kernel, there is some shell script that I can simply source to start cross-development for target platform.
>>> > 
>>> > 
>>> > I found this AI based note about Buildroot's way of
>>> > putting cross-build toolchains to use:
>>> > 
>>> > QUOTE
>>> > Buildroot pre-configures and compiles all necessarily
>>> > packages during the build process to create a custom
>>> > root filesystem.
>>> > . . .
>>> > Buildroot focuses on static builds, meaing all selected
>>> > packages are compiled and integrated into the final root
>>> > filesystem image during the build process. There is no
>>> > concept of installing or updating packages on the target
>>> > after the initial build.
>>> > END QUOTE
>>> > 
>>> > This is certainly not what the FreeBSD ports tree is
>>> > set up for. Buildroot or analogous does not exist for
>>> > FreeBSD --or most *BSD or Unix-- to my knowledge.
>>> > 
>>> > May be NetBSD's pkgsrc cross build support:
>>> > 
>>> > https://cdn.netbsd.org/pub/pkgsrc/current/pkgsrc/doc/HOWTO-use-crosscompile
>>> > 
>>> > would be closer to how you like to work? (I've never
>>> > used such.)
>>> > 
>>> > > Regards,
>>> > > Michał Kruszewski
>>> > > 
>>> > > Sent with Proton Mail secure email.
>>> > > 
>>> > > On Saturday, September 13th, 2025 at 12:23 AM, Mark Millard [email protected] wrote:
>>> > > 
>>> > > > On Sep 12, 2025, at 14:10, Michał Kruszewski [email protected] wrote:
>>> > > > 
>>> > > > > A simple way is to set up an armv7 chroot / jail on an arm64 host such as a Raspberry Pi (Apple devices don't work!).
>>> > > > > Then it's just like a native environment, but usually much faster.
>>> > > > 
>>> > > > > I don't have physical arm64 host.
>>> > > > > I also don't want to create a VM.
>>> > > > > The whole idea seems weird and unnatural.
>>> > > > 
>>> > > > Ignoring the RPi4B detail:
>>> > > > 
>>> > > > FreeBSD official armv7 port-packages are built this way
>>> > > > on arm64 hardware that natively supports user-space
>>> > > > armv7 code. (ampere* systems are used.)
>>> > > > 
>>> > > > FreeBSD official i386 port-packages are built this way
>>> > > > on amd64 hardware that natively supports user-space
>>> > > > i386 code.
>>> > > > 
>>> > > > No use of qemu variants of any kind: no attempt at
>>> > > > non-native-capable host environments.
>>> > > > 
>>> > > > (Back when amd64 and qemu was used for the likes
>>> > > > of armv7, lots of stuff failed to build mcuh of
>>> > > > the time over the years --stuff that builds
>>> > > > just fine now. This was abandoned after native
>>> > > > became available. armv6 was always qemu based
>>> > > > on amd64 and could not build much as of the last
>>> > > > time a build was run.)
>>> > > > 
>>> > > > > It sounds more like an exotic workaround.
>>> > > > 
>>> > > > It is the official technique used for what FreeBSD
>>> > > > distributes for armv7 port-packages.
>>> > > > 
>>> > > > > Regards,
>>> > > > > Michał Kruszewski
>>> > > > > 
>>> > > > > Sent with Proton Mail secure email.
>>> > > > > 
>>> > > > > On Friday, September 12th, 2025 at 10:46 PM, Robert Clausecker [email protected] wrote:
>>> > > > > 
>>> > > > > > Hi Michał,
>>> > > > > > 
>>> > > > > > A simple way is to set up an armv7 chroot / jail on an arm64 host such as
>>> > > > > > a Raspberry Pi (Apple devices don't work!). Then it's just like a native
>>> > > > > > environment, but usually much faster.
>>> > > > > > 
>>> > > > > > Yours,
>>> > > > > > Robert Clausecker
>>> > > > > > 
>>> > > > > > Am Fri, Sep 12, 2025 at 04:19:42PM +0000 schrieb Michał Kruszewski:
>>> > > > > > 
>>> > > > > > > Is there any tutorial on how to cross compile custom user application for armv7?
>>> > > > > > > I struggle t find any.
>>> > > > > > > Cross compiling the system is pretty easy.
>>> > > > > > > However, how can one cross compile custom user application or kernel drivers.
>>> > > > > > > 
>>> > > > > > > Regards,
>>> > > > > > > Michał Kruszewski
>>> > > > > > > 
>>> > > > > > > Sent with Proton Mail secure email.
>>> > > > > > 
>>> > > > > > --
>>> > > > > > () ascii ribbon campaign - for an encoding-agnostic world
>>> > > > > > /\ - against html email - against proprietary attachments
>>> > > > 
>>> > > > ===
>>> > > > Mark Millard
>>> > > > marklmi at yahoo.com
>>> > 
>>> > 
>>> > ===
>>> > Mark Millard
>>> > marklmi at yahoo.com
>>> 
>> 
> 
===
Mark Millard
marklmi at yahoo.com