Re: help needed to manage s390x host for ci.debian.net

Hi,

On 13-02-2023 15:59, Dipak Zope1 wrote:
> There is some issue with 5.10.0-21 kernel and we are working on it. This 
> can cause performance impact on CI servers.

I have rebooted to the old kernel yesterday. That helps a bit indeed, 
although most of the issues I reported predate that kernel upgrade.

> As Paul mentioned we have upgraded CI servers to better capacity in May 
> last year. Is today’s performance worse than what we observed right 
> after upgrade?

As you can see e.g. here [1,2] it comes and goes (albeit sometimes the 
queue was empty). I don't think its very different, I just never got out 
of the s390x host what I was expecting. Long time I blamed it on the 
"stealing" that happens on a shared host, but I think there's more.

[1] 
https://ci.debian.net/munin/ci-worker-s390x-01/ci-worker-s390x-01/debci_packages_processed.html

[2] 
https://ci.debian.net/munin/ci-worker-s390x-01/ci-worker-s390x-01/cpu.html

> Is the performance deteriorated consistently over period 
> of time or suddenly observed? Is there any incidence – like 
> change/upgrade in software or hardware component which is coinciding 
> with it if it is sudden change?

James Addison suggested in [3] to increase a prefetch counter in amqp 
(although its the same on all hosts); I have done so on the s390x host 
and at least initially it seems to help keeping the host busier.

[3] https://salsa.debian.org/ci-team/debci/-/issues/92#note_381306

Paul