Re: [yocto-patches] [auh][PATCH 2/3] modules/steps.py: refactor compile() to use parallel per-machine builds

Hello Qi,

thanks for the detailed answer. I fully agree those would be useful
improvements, my concern is that AUH right now is a very shaky
foundation for doing them. Specifically:

1. No test suite. This is becoming a real problem, as I struggle to
have confidence in non-trivial patches, and especially when they add
new features instead of refactoring code. This code can and does break
easily, as the amount of corner cases is particularly large. If I take
this patch, you'll expect me to take more patches that add even more
code, while still not having any way to test them.

This could be addressed by having 'mock' versions of bitbake and
devtool in the test suite. The 'mock' versions would print things that
AUH expects, both in 'success' and 'fail' scenarios, and maybe write
out 'upgraded' recipes, but otherwise do nothing. That way, all or
most code paths in AUH would be exercised, without having to run a
'real' upgrade, which is heavy, takes forever, and not really suitable
for testing changes.

2. Which brings me to the other issue, and it probably should be
addressed first: the brittle interface with devtool and bitbake. AUH
makes decisions based on finding strings in human readable output of
these tools. This isn't great. One way to address that is to move at
least the upgrade/finish/build/test/commit loop into a new devtool
plugin, so that it would use devtool libraries directly, and interface
with bitbake via tinfoil. Then AUH's job would be simply to determine
what needs to be updated, call devtool with appropriate command line
arguments (or maybe json config with the 'upgrade job' specification)
and to make reports/emails from resulting patches and logs.

Once we have these things, adding more features becomes a lot more
sustainable. I understand such a refactor is non-trivial, but equally
AUH as it is can't take much more extensions.

I also have a side question: when you say "Upgrade recipes (stable
version upgrades only) ", how do you determine the stable versions?
There's an ongoing discussion about this with some patches; figuring
out what the stable versions are isn't trivial.

Alex


On Tue, 28 Apr 2026 at 05:34, Chen, Qi <[email protected]> wrote:
>
> Hi Alex,
>
> I totally agree with you that "It's far more likely that an update builds fine by itself, but breaks something not covered by AUH recipe builds".
>
> Let me first share my local practice and then share my thoughts on AUH improvements.
>
> Here's my local practice for LTS releases:
> Upgrade recipes (stable version upgrades only) for each layer using AUH (parallel build enabled) + manually checking the upgrade results + world build
>
> I'm using world build instead of some extra build tests for each upgrade because I'm upgrading quite a few recipes in multiple layers, the upgrades are stable upgrades and the server is powerful enough to finish a world build in a few hours.
>
> What I have in mind for AUH improvement for oe-core master is that, apart from this parallel change, we can also add some other changes to AUH so that it can be more efficient and save developers time.
>
> 1. A new step, compile_direct_dependents, could be added after the compile step.
> What this step does it building recipes that depend directly on the recipe that  was just upgraded. The can be got by using 'bitbake -g world' + some filtering.
> This step can be enabled by default, but could be also be skipped by settings. When enabled, it only builds those recipes for the first machine.
>
> 2. The machines settings could be extended to include fragements
> e.g.,
> machines = qemux86-64:core/yocto-autobuilder/multilib-x86-lib32 qemuarm64_musl:some-fragment-to-enable-debug-build ....
> In this way, we're giving AUH more build coverage.
>
> With the above improvements finished, a successful AUH upgrade in oe-core means:
> 1) it builds successfully for all specified machines and configurations
> 2) all recipes that directly depend on it build successfully
> 3) testimage passes (which means if it has ptest, its ptest passes)
>
> When the emails are sent out, the maintainers really do not have much to do with a successful upgrade. The maintainer just reviews the changes, correct license if there's some license change, put some commit log and send out the patch which is a sign of commitment to fix any further problems. When the AUH upgrade fails, the maintainer can also easily  see what the problem is and how to reproduce it.
>
> What do you think about this?
>
> Regards,
> Qi
>
>
> -----Original Message-----
> From: Alexander Kanavin <[email protected]>
> Sent: Monday, April 27, 2026 4:55 PM
> To: Chen, Qi <[email protected]>
> Cc: [email protected]; MacLeod, Randy <[email protected]>
> Subject: Re: [yocto-patches] [auh][PATCH 2/3] modules/steps.py: refactor compile() to use parallel per-machine builds
>
> On Mon, 27 Apr 2026 at 04:09, ChenQi <[email protected]> wrote:
>
> > With this parallel build added, Yocto can configure oe-core's upgrade
> > to use machines that oe-core uses as the acceptance criteria. This
> > means a successful auh upgrade will very like to be merged without any change.
> > If there is some failure, e.g., qemuarm64_musl, the
> > developers/maintainers can see it directly without doing more local builds.
>
> For the purpose of updating oe-core, I feel a bit different about benefits of adding more machines actually. I think it's unlikely that adding more entries to the two existing ones (qemux86_64 and
> qemuarm_musl) will help uncover more issues - if 32 bit and musl variants are already covered between these two, then it's unlikely there will a particular 32 bit x86 issue, or 64 bit arm issue, or a risc-v issue, e.g. something that those additional machines would catch.
>
> It's far more likely that an update builds fine by itself, but breaks something not covered by AUH recipe builds, e.g. a dependent recipe, ptests, oe-selftest, or runtime behavior. Such things are unrealistic to test for each update, and so they're done after the fact, in a-full.
>
> I agree that speeding up AUH would be beneficial, but that is better done in the outermoost loop, which takes recipe groups and updates them one by one. Parallelizing that is more complicated, as each update operation would need its own copy of oe-core repo in addition to its own build directory, but it would really run everything in parallel, including devtool upgrade (which can be very sluggish on large source trees or large commit histories), buildhistory, and test builds. And of course, lack of AUH tests hinders such large refactor efforts as well.
>
> Alex