Re: s390x builder outage

On Sun, Aug 16, 2026, at 13:44, Florian Weimer via devel wrote:
> * Stephen J. Smoogen:
>
>> The issue has always been that the power and s390x have been spread
>> thin. For the longest time the servers were loaners from IBM and when
>> ones were purchased they were usually the bottom of the line because
>> the cost difference cut into how many x86 systems were available. (It
>> was something like 4:1) The s390x is spread between multiple Red Hat
>> teams and if testing or other IO intensive work is going on our usage
>> would be highly impacted.
>
> I think the mainframes are connected to a SAN.  The I/O stall issue
> seems to predominantly impact certain POWER machines that were ordered
> with 7,200 RPM SATA disks long after SSDs had become common pretty much
> everywhere else.
>
> Anyway, I remember a time when builds tend to fail on s390x first
> because it was the fastest architecture to install the build
> dependencies and begin building.  So s390x wasn't always the slowest
> architecture, at least in one possible sense.
>

Those were usually when whichever z-machine was completely unloaded. Fedora gets moved onto new systems as a test load as it is low risk to other deliverables. I don’t know what the current disk sharing is but for the longest time the s390x would require one builder to be a dedicated caching box for the others to pull https requests and 3 had to use NFS-over-ssh to share the disk space composes require to occur.

Those were the usual breaking points for daily work and would require restarts regularly.



> Thanks,
> Florian
>
> -- 
> _______________________________________________
> devel mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> Fedora Code of Conduct: 
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/[email protected]
> Do not reply to spam, report it: 
> https://forge.fedoraproject.org/infra/tickets/issues/new

-- 
Stephen J Smoogen.
Let us be kind to one another, for most of us are fighting a hard battle. -- Ian MacClaren
-- 
_______________________________________________
devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/[email protected]
Do not reply to spam, report it: https://forge.fedoraproject.org/infra/tickets/issues/new