RE: [yocto-patches] [auh][PATCH 2/3] modules/steps.py: refactor compile() to use parallel per-machine builds

Hi Alex,

I totally agree with you that "It's far more likely that an update builds fine by itself, but breaks something not covered by AUH recipe builds".

Let me first share my local practice and then share my thoughts on AUH improvements.

Here's my local practice for LTS releases:
Upgrade recipes (stable version upgrades only) for each layer using AUH (parallel build enabled) + manually checking the upgrade results + world build

I'm using world build instead of some extra build tests for each upgrade because I'm upgrading quite a few recipes in multiple layers, the upgrades are stable upgrades and the server is powerful enough to finish a world build in a few hours.

What I have in mind for AUH improvement for oe-core master is that, apart from this parallel change, we can also add some other changes to AUH so that it can be more efficient and save developers time.

1. A new step, compile_direct_dependents, could be added after the compile step.
What this step does it building recipes that depend directly on the recipe that  was just upgraded. The can be got by using 'bitbake -g world' + some filtering.
This step can be enabled by default, but could be also be skipped by settings. When enabled, it only builds those recipes for the first machine.

2. The machines settings could be extended to include fragements
e.g.,
machines = qemux86-64:core/yocto-autobuilder/multilib-x86-lib32 qemuarm64_musl:some-fragment-to-enable-debug-build ....
In this way, we're giving AUH more build coverage.

With the above improvements finished, a successful AUH upgrade in oe-core means:
1) it builds successfully for all specified machines and configurations
2) all recipes that directly depend on it build successfully
3) testimage passes (which means if it has ptest, its ptest passes)

When the emails are sent out, the maintainers really do not have much to do with a successful upgrade. The maintainer just reviews the changes, correct license if there's some license change, put some commit log and send out the patch which is a sign of commitment to fix any further problems. When the AUH upgrade fails, the maintainer can also easily  see what the problem is and how to reproduce it.

What do you think about this?

Regards,
Qi


-----Original Message-----
From: Alexander Kanavin <[email protected]> 
Sent: Monday, April 27, 2026 4:55 PM
To: Chen, Qi <[email protected]>
Cc: [email protected]; MacLeod, Randy <[email protected]>
Subject: Re: [yocto-patches] [auh][PATCH 2/3] modules/steps.py: refactor compile() to use parallel per-machine builds

On Mon, 27 Apr 2026 at 04:09, ChenQi <[email protected]> wrote:

> With this parallel build added, Yocto can configure oe-core's upgrade 
> to use machines that oe-core uses as the acceptance criteria. This 
> means a successful auh upgrade will very like to be merged without any change.
> If there is some failure, e.g., qemuarm64_musl, the 
> developers/maintainers can see it directly without doing more local builds.

For the purpose of updating oe-core, I feel a bit different about benefits of adding more machines actually. I think it's unlikely that adding more entries to the two existing ones (qemux86_64 and
qemuarm_musl) will help uncover more issues - if 32 bit and musl variants are already covered between these two, then it's unlikely there will a particular 32 bit x86 issue, or 64 bit arm issue, or a risc-v issue, e.g. something that those additional machines would catch.

It's far more likely that an update builds fine by itself, but breaks something not covered by AUH recipe builds, e.g. a dependent recipe, ptests, oe-selftest, or runtime behavior. Such things are unrealistic to test for each update, and so they're done after the fact, in a-full.

I agree that speeding up AUH would be beneficial, but that is better done in the outermoost loop, which takes recipe groups and updates them one by one. Parallelizing that is more complicated, as each update operation would need its own copy of oe-core repo in addition to its own build directory, but it would really run everything in parallel, including devtool upgrade (which can be very sluggish on large source trees or large commit histories), buildhistory, and test builds. And of course, lack of AUH tests hinders such large refactor efforts as well.

Alex