Re: s390x builder outage

On Tue, 2026-08-18 at 12:16 -0700, Kevin Fenzi wrote:
> I love having stats (Thats Michel for gathering those up)! however, I
> don't think
> not having them is anything like the cause of this issue. The cause
> is that
> we simply don't have enough resources. I don't think these stats
> answer
> however: How much more resource do we need to make s390x keep up with
> the other arches? My gut says 2.5-3x we have now (allowing us to make
> 45
> regular builders, putting it on par with other arches) would go a
> long
> way. But perhaps others could come up with a more informed ask based
> on
> that data that has been collected?
> 
OK, for those who want to help dig into the data, now is probably a
good time

https://forge.fedoraproject.org/packaging/koji-lag-metrics
has been updated with daily/weekly/monthly reports stretching back to
January 2025

The README.md points to more resources, but for completeness I'll
include it here


https://github.com/slopfest/sandogasa/blob/v0.21.0/tools/koji-
lag/FINDINGS.md has the LLM-assisted findings (most of these are
reproducible by
running `koji-lag` so there is hopefully not much room for
hallucinations)


https://github.com/slopfest/sandogasa/tree/main/tools/koji-lag#getting-
a-published-store has instructions for getting the SQLite database
(lightly edited from
the one tagged v0.21.0). It's about 1 GB compressed and 3.4 GB
uncompressed so you'd need at least 5 GB before attempting this.

And if you're the data analyst type, there is an example Jupyter
notebook showing how to load the data and play with it in 
https://github.com/slopfest/sandogasa/blob/main/tools/koji-
lag/notebooks/arch-lag.ipynb - I would recommend cloning the sandogasa
repo in that case, and also
the Fedora PGM schedule repo, which ... uh, is still on Pagure.io: 
https://pagure.io/fedora-pgm/schedule/

I'll work on updating the Fedora packages for sandogasa over the
weekend so people won't have to compile koji-lag - but it's not that
big a job if you just compile that binary (or run it from a checkout
with `cargo run -p koji-lag`)

Some TL;DR

Trends

- s390x builds get slower over time, while other architectures'
builds get faster

- s390x being slower means that even if we have enough capacity to not
have builds queuing, it will still delay completion

- the average Fedora maintainer is not badly affected, but those that
do a lot of builds are - the p90 for the top 10 submitters is huge
(1.83 hours)

- keeping s390x for only ELN + EPEL would definitely relieve our
capacity issue for now, while if we want to stop s390x being such a big
blocker we need at least another mainframe (so double the capacity)

Best regards,

-- 
 _o) Michel Lind
_( ) https://keyoxide.org/5dce2e7e9c3b1cffd335c1d78b229d2f7ccc04f2
     README:    https://fedoraproject.org/wiki/User:Salimma#README

-- 
_______________________________________________
devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/[email protected]
Do not reply to spam, report it: https://forge.fedoraproject.org/infra/tickets/issues/new

-----BEGIN PGP SIGNATURE-----

iJAEABYKADgWIQRdzi5+nDsc/9M1wdeLIp0vfMwE8gUCaol5phocc2FsaW1tYUBm
ZWRvcmFwcm9qZWN0Lm9yZwAKCRCLIp0vfMwE8gO+AQD9DXxmVUYuZlzRJ88m2/Wc
SScgCiebCmcLHgO3oT9jkwD/WyqywnDLeYw//w92jAGoD9e9xTBT4j1qZNg9wfFc
NAA=
=2SLv
-----END PGP SIGNATURE-----