RE: help needed to manage s390x host for ci.debian.net

Hello Paul, Philipp and team -

There is some issue with 5.10.0-21 kernel and we are working on it. This can cause performance impact on CI servers.
So I request to downgrade all our CI machines which are on 5.10.0-21 to 5.10.0-20.
There are pretty good changes that we get better output if we are currently running with 5.10.0-21.

Besides this I would like to understand more about the performance:
As Paul mentioned we have upgraded CI servers to better capacity in May last year. Is today’s performance worse than what we observed right after upgrade? Is the performance deteriorated consistently over period of time or suddenly observed? Is there any incidence – like change/upgrade in software or hardware component which is coinciding with it if it is sudden change?

Meanwhile we are working on the kernel fix. I will keep this mail thread updated.

Thanks,
-Dipak Zope
Debian s390 porting team

From: Philipp Kern <[email protected]>
Date: Monday, 13 February 2023 at 1:28 PM
To: Paul Gevers <[email protected]>, debian-s390 <[email protected]>, [email protected] <[email protected]>, Paul Flint <[email protected]>
Cc: Debian CI team <[email protected]>
Subject: [EXTERNAL] Re: help needed to manage s390x host for ci.debian.net
Hi,

On 12.02.23 22:38, Paul Gevers wrote:
> I have munin [1], but as said, I'm not a trained sysadmin. I don't know
> what I'm looking for if you ask "statistics on the network".

This is more of a software development / devops question than a sysadmin
question, but alas. What I am interested in is *application-level*
logging on reconnects. Presumably the connection to RabbitMQ is
outbound? Is it tunneled? Does your application log somewhere when a
reconnect happens? Does it say when it successfully connected?

I'd expect good software to log something like this:

[10:00:00] Connecting to broker "rabbitmq.debci.debian.net:12345"...
[10:00:05] Connected to broker "rabbitmq.debci.debian.net:12345".

And also:

[10:00:00] Connecting to broker "rabbitmq.debci.debian.net:12345"...
[10:00:01] Connection to broker "rabbitmq.debci.debian.net:12345"
failed: Connection refused

Kind regards
Philipp Kern