Re: s390x builder outage

Hi Kevin,

On Tue, 2026-08-18 at 12:16 -0700, Kevin Fenzi wrote:
> On Sun, Aug 16, 2026 at 07:44:58PM +0200, Florian Weimer via devel
> wrote:
> > 
> > I think the mainframes are connected to a SAN.  The I/O stall issue
> > seems to predominantly impact certain POWER machines that were
> > ordered
> > with 7,200 RPM SATA disks long after SSDs had become common pretty
> > much
> > everywhere else.
> > 
> > Anyway, I remember a time when builds tend to fail on s390x first
> > because it was the fastest architecture to install the build
> > dependencies and begin building.  So s390x wasn't always the
> > slowest
> > architecture, at least in one possible sense.
> 
> I'll just reply here (I was out yesterday). Happy to provide details
> on
> any hardware/setup, and happy to have ideas to improve things...
> 
> Yes, there were power9 machines with 7200rpm sata drives. We switched
> builders on those machines to use iscsi from netapp a long while
> back.
> Then when we moved to rdu3 last year we replaced them entirely with
> power10 machines. The old power9 machines are being used for copr
> builds
> still, but no fedora koji builds in any way. The power10 buildvm's
> are
> backed by netapp iscsi. No spinning anything there.
> 
> The s390x instance we are using is a single lpar in production and
> another (much smaller) one in staging. I don't have visibility into
> the
> storage that backs the lpar, but I am close to 100% sure it's all
> solid
> state/SAN.
> 
> It has 573440MB of memory and 48 cpus and about 3tb of disk space.
> It's running fedora 44 and runs buildvm's on it. Now we can carve
> that
> up however we like. The two extremes are bad:
> 
> - 1 gigantic host. This would build things much faster! 48 cpus!
> 500GB
>   memory! However, it would cause a very long line behind it and
> things
>   would back up horribly.
> 
> - 48 hosts with 1 cpu each. This would allow less stalls for smaller
>   builds, but bigger builds would take many many times as long. Also,
> we
>   don't have enough disk space to do that.
> 
> So, its currently 19 buildvm's. 1 is used as a cache server (so
> common
> buildroot packages install quickly). 2 are used as compose hosts
> (needed
> to run runroot jobs to make install images). 1 is a 'heavybuilder'
> which is needed for webkitgtk in particular that won't build on a
> normal
> builder and the rest are regular builders.
> 
> I love having stats (Thats Michel for gathering those up)! however, I
> don't think
> not having them is anything like the cause of this issue. The cause
> is that
> we simply don't have enough resources.
Oh, totally agreed with this. I'm hoping the data will give us a
stronger case for asking for more resources (or, alternatively, give
up)

>  I don't think these stats answer
> however: How much more resource do we need to make s390x keep up with
> the other arches? My gut says 2.5-3x we have now (allowing us to make
> 45
> regular builders, putting it on par with other arches) would go a
> long
> way. But perhaps others could come up with a more informed ask based
> on
> that data that has been collected?
> 
I'm still collecting more data, and plan to put up both the sqlite
database containing the raw data and an IPython notebook so it'll be
easier for people to poke around.

Hang on tight - my collection tool is still being tuned, and I'm still
slowly grabbing more data, ideally we'd have enough data to compare
mass rebuilds from F42 to F45.

But preliminarily - apart from mass rebuilds, and known outages like
the storage one from May, I'm also seeing delays happening roughly once
a month over the past year or so, and it's more heavily affecting more
active packagers compared to the average packager (probably because
they are more likely to build a set of packages with a large dependency
graph)

Best regards,


-- 
 _o) Michel Lind
_( ) https://keyoxide.org/5dce2e7e9c3b1cffd335c1d78b229d2f7ccc04f2
     README:    https://fedoraproject.org/wiki/User:Salimma#README

-- 
_______________________________________________
devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/[email protected]
Do not reply to spam, report it: https://forge.fedoraproject.org/infra/tickets/issues/new

-----BEGIN PGP SIGNATURE-----

iJAEABYKADgWIQRdzi5+nDsc/9M1wdeLIp0vfMwE8gUCaocpgRocc2FsaW1tYUBm
ZWRvcmFwcm9qZWN0Lm9yZwAKCRCLIp0vfMwE8mlkAQDNg9TU+P/VRG5vokBBKljf
pukd1eWNloRko0WEeSHJyAD/YErBcKFhaunl0zf2PK+HZS25/wwMASvVmAbnppqN
ZgI=
=F6K7
-----END PGP SIGNATURE-----