Re: s390x builder outage

On Sun, Aug 16, 2026 at 07:44:58PM +0200, Florian Weimer via devel wrote:
> 
> I think the mainframes are connected to a SAN.  The I/O stall issue
> seems to predominantly impact certain POWER machines that were ordered
> with 7,200 RPM SATA disks long after SSDs had become common pretty much
> everywhere else.
> 
> Anyway, I remember a time when builds tend to fail on s390x first
> because it was the fastest architecture to install the build
> dependencies and begin building.  So s390x wasn't always the slowest
> architecture, at least in one possible sense.

I'll just reply here (I was out yesterday). Happy to provide details on
any hardware/setup, and happy to have ideas to improve things...

Yes, there were power9 machines with 7200rpm sata drives. We switched
builders on those machines to use iscsi from netapp a long while back.
Then when we moved to rdu3 last year we replaced them entirely with
power10 machines. The old power9 machines are being used for copr builds
still, but no fedora koji builds in any way. The power10 buildvm's are
backed by netapp iscsi. No spinning anything there.

The s390x instance we are using is a single lpar in production and
another (much smaller) one in staging. I don't have visibility into the
storage that backs the lpar, but I am close to 100% sure it's all solid
state/SAN.

It has 573440MB of memory and 48 cpus and about 3tb of disk space.
It's running fedora 44 and runs buildvm's on it. Now we can carve that
up however we like. The two extremes are bad:

- 1 gigantic host. This would build things much faster! 48 cpus! 500GB
  memory! However, it would cause a very long line behind it and things
  would back up horribly.

- 48 hosts with 1 cpu each. This would allow less stalls for smaller
  builds, but bigger builds would take many many times as long. Also, we
  don't have enough disk space to do that.

So, its currently 19 buildvm's. 1 is used as a cache server (so common
buildroot packages install quickly). 2 are used as compose hosts (needed
to run runroot jobs to make install images). 1 is a 'heavybuilder'
which is needed for webkitgtk in particular that won't build on a normal
builder and the rest are regular builders.

I love having stats (Thats Michel for gathering those up)! however, I don't think
not having them is anything like the cause of this issue. The cause is that
we simply don't have enough resources. I don't think these stats answer
however: How much more resource do we need to make s390x keep up with
the other arches? My gut says 2.5-3x we have now (allowing us to make 45
regular builders, putting it on par with other arches) would go a long
way. But perhaps others could come up with a more informed ask based on
that data that has been collected?

kevin
-- 
_______________________________________________
devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/[email protected]
Do not reply to spam, report it: https://forge.fedoraproject.org/infra/tickets/issues/new