Re: parallel builds

On Fri, 2026-07-31 at 14:48 +0200, Bruno Haible wrote:
> Richard Purdie wrote:
> > > The GCS only say that the developer should "try to" make parallel
> > > builds work.
> > the GCS could really do with making that a little stronger.
>=20
> I disagree with that opinion, because
>=20
> > Of the ~1000 open source projects in OpenEmbedded-Core (components to
> > make a basically functional system and/or are commonly used), there are
> > about 6 which have to disable it.
>=20
> 1) I doubt these numbers. Races due to parallel builds happen only with a=
 low
> =C2=A0=C2=A0 probability. Therefore there may well be 200 or 500 packages=
 that don't
> =C2=A0=C2=A0 support parallel builds, and you have noticed only about 6 o=
f them so far.
> =C2=A0=C2=A0 To find more, try a larger degree of parallelization: "make =
-j1000" or so.

I'm not sure if you know much about OpenEmbedded/Yocto Project but it
has been around for a couple of decades and in that time we've gone
from a few hundred of these recipes (software components) having
parallel make issues to just the ones I mentioned. That is a matter of
public record in the repository, you can look for PARALLEL_MAKE =3D ""
and PARALLEL_MAKEINST =3D "" markup.

I know one fellow developer runs all his builds with a make fuzzer,
specifically designed to randomise make call order to expose these
kinds of issues.

The project is built by many users including many large organisations
(see YP's members) with high parallelism on large servers. Our own
testing infrastructure runs at -j 24 for everything. No, it isn't 1000s
but it is high enough we do find issues and send patches to fix them.

We have also done work to analyse how effective parallel make is, and
above around 20-30, it doesn't really help our overall build times.

Put another way, I'm not claiming every parallel make bug is fixed. I
am claiming a majority of software and developers are using parallel
make such that it makes sense it should be the default supported thing
now, not the other way around and that most software we're building
doesn't show any issue day to day.

> 2) There is still the problem of "the log file is garbage". When the buil=
d
> =C2=A0=C2=A0 fails and you need to look at the log file, it will lead you=
r investigation
> =C2=A0=C2=A0 to wrong directions. So, a developer gains 5 minutes of wall=
 time waiting
> =C2=A0=C2=A0 for the build, and then loses 30 minutes debugging in the wr=
ong area.

It can be an issue but not one we've had a lot of trouble with to be honest=
.

Cheers,

Richard