Re: s390x builder outage

On Fri, 2026-08-14 at 13:47 -0400, Neal Gompa wrote:
> On Fri, May 8, 2026 at 5:59 AM Peter Robinson <[email protected]>
> wrote:
> > 
> > On Thu, 7 May 2026 at 18:20, Adam Williamson
> > <[email protected]> wrote:
> > > 
> > > On Thu, 2026-05-07 at 13:38 +0200, Peter Lemenkov wrote:
> > > > Hans, thanks for raising this question once again. Today I
> > > > again asked
> > > > myself why s390x (the past) is even a primary architecture
> > > > while
> > > > RISC-V (the future) is not?
> > > 
> > > s390x is not "the past". It may be unfashionable, but a lot of
> > > people
> > > still run stuff on mainframes and will continue to do so for the
> > > foreseeable future. It's a live, actively developed arch.
> > > 
> > > Yes, there are specific problems with the IBM/RH backing for
> > > s390x in
> > > Fedora which, if not resolved, may form a reasonable argument for
> > > dropping/downgrading it in Fedora. But calling it "the past" is
> > > just
> > > off.
> > 
> > I agree with Adam here, and frankly the issue here isn't anything
> > to
> > do with the s390x builders as an architecture, it seems whoever
> > runs
> > the RH infrastructure (outside of the Fedora team's control) hasn't
> > been monitoring the storage array or the support contract for the
> > 3rd
> > party who provides monitoring has failed,  so I don't think the
> > discussion of the architecture is the problem here but the complete
> > and utter failure of RH/IBM to run proper infrastructure. I have no
> > doubt it's affecting other builders internally as well. I look
> > forward
> > to the RCA.
> > 
> 
> So Michel did some excellent work to quantify exactly *how* poor the
> performance is for s390x relative to other architectures. As of last
> month, s390x is orders of magnitude worse than the next bad one
> (ppc64le):
> https://forge.fedoraproject.org/packaging/koji-lag-metrics/src/branch/main/reports/monthly/2026/07/report.txt
> 
> Myself, Fabio, Maxwell, and Michel are prepared to propose the total
> removal of s390x support in the Fedora Project (affecting all
> targets)
> for Fedora 46 if there isn't progress improving performance and
> capacity by the end of the year.
> 
> We have already conveyed this to Red Hat during the last leadership
> call, so they are aware of the clock ticking too.
> 
> 
Alternately we could keep it alive in at least ELN and EPEL - but
personally, apart from the inconvenience for normal builds and for
trying to do the right thing and using PRs to get some additional CI
when updating my own packages (sometimes I end up merging because the
build is stuck on s390x) - I worry about how slow builders on
architectures that we don't care about on desktops are slowing down the
release of security updates on the architectures we do care about on
desktop/server/cloud (i.e. mostly x86_64 and aarch64).

Best regards,

-- 
 _o) Michel Lind
_( ) https://keyoxide.org/5dce2e7e9c3b1cffd335c1d78b229d2f7ccc04f2
     README:     https://fedoraproject.org/wiki/User:Salimma#README

-- 
_______________________________________________
devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/[email protected]
Do not reply to spam, report it: https://forge.fedoraproject.org/infra/tickets/issues/new

-----BEGIN PGP SIGNATURE-----

iJAEABYKADgWIQRdzi5+nDsc/9M1wdeLIp0vfMwE8gUCan9coxocc2FsaW1tYUBm
ZWRvcmFwcm9qZWN0Lm9yZwAKCRCLIp0vfMwE8hkYAQD9Hy9qDAWs2av+YxffYypl
LAALgnJ/P9yqwo6e9y1w5gD/e5Viq+pzhslEZD0oVOU5CrG4ULE5pourxQxViakQ
EQw=
=Fwb7
-----END PGP SIGNATURE-----