Re: s390x builder outage

On Sat, Aug 15, 2026 at 5:09 PM Fabio Valentini <[email protected]> wrote:
>
> On Sat, Aug 15, 2026 at 4:38 PM Chris Adams <[email protected]> wrote:
> >
> > Once upon a time, Fabio Valentini <[email protected]> said:
> > > When there are idle builders available, then ppc64le builders appear
> > > to be slightly slower than s390x.
> > > But when all s390x builders are busy, they become the bottleneck, both
> > > in terms of wait time (time until a build is assigned to an s390x
> > > builder) *and* build time (more builds running on the same hardware?).
> >
> > That sounds like maybe too many parallel builds are being sent to the
> > same hardware, which I would think would be easy to adjust.
>
> Not really, no ... provisioning fewer builders to the same hardware
> just makes the bottleneck worse (fewer builders available). From what
> I can tell, the current setup (number of builders / available
> hardware) is pretty much the best that can be done - without actually
> getting more hardware.
>

To make it clear, we have a shortage of hardware for *both*
architectures. But IBM Z impairs us significantly more than POWER
does. There are a number of things that have gone wrong to lead us
here, the key one is that Red Hat does no anticipatory tracking of
utilization and latency to identify future capacity issues. Ideally,
this will become a top priority for CLE to resolve[1], and then there
will be investment to "catch up" to optimal levels of builder
capacity.

[1]: https://redhat.atlassian.net/browse/CLE-6650



-- 
真実はいつも一つ！/ Always, there's only one truth!
-- 
_______________________________________________
devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/[email protected]
Do not reply to spam, report it: https://forge.fedoraproject.org/infra/tickets/issues/new