Re: CI congestion/starvation

On Sun, Mar 1, 2026 at 10:46 AM Johnny Jazeix <[email protected]> wrote:

> Le sam. 28 févr. 2026 à 22:24, Ingo Klöcker <[email protected]> a écrit :
> >
> > On Samstag, 28. Februar 2026 20:53:38 Mitteleuropäische Normalzeit Johnny
> > Jazeix wrote:
> > > Hi,
> > > today we also have a lot of congestion. After discussion with Ben,
> > > it's due to a new Gear update which uses the resources of the CI for
> > > multiple hours.
> > > Would it be possible to spread the changes done to each repo during a
> > > full day (with sleeps between each git push) instead of doing them at
> > > once to let other projects use the CI?
> >
> > You do realize that this would mean that the people who do our releases
> would
> > have to sit the full day in front of their computer?
> >
>
> I don't know the exact process, but I guess all the pushes are not
> done manually but via a script?
> How often is there an error requiring human intervention? If it is
> none, the script can run in background and the person can live its
> life?
>

Putting sleeps in between each push would make release preparation
activities quite difficult, as pushing the version bumps is just one part
of the process.


>
> > A Gear release happens once a month. I really don't think that's a big
> > problem. (Yes, there's also Plasma, but I think that's a lot less
> projects,
> > and Frameworks.) Just make sure that you don't plan a release of a
> non-Gear
> > project around the release date of Gear (or Plasma or Frameworks).
> Marketing-
> > wise it's anyway better to avoid such a collision.
> >
>
> You don't but other people are impacted. Maybe we can run these heavy
> process at a "better" time where less developers are active (I guess
> we can have stats from the CI usage)?
>

It took the CI nodes approximately 10 hours to work through all of the
builds for the record (they're just finishing up now, from when they were
triggered at 2pm UTC).
That includes all the other builds they also received during that time they
would normally service.

During this time the CI nodes completed a total of 5,211 builds, with the
vast majority of these jobs completing either in a matter of seconds (for
the JSON/XML/etc validation jobs) or in the space of a few minutes (for
conventional CI and CD jobs).
4,807 of those took less than 10 minutes (160 hours of CI time), 346 of
them took between 10-25 minutes (85 hours of CI time) and 77 of them took
more than 25 minutes (55 hours of CI time) for a total of 301 CI hours
(difference of 1 hour due to rounding).

During this we had conventional Linux CI jobs that completed in under a
minute (which includes VM provisioning, cloning sources, unpacking
dependencies, configure, build, install, publishing build artifacts, and
running tests) as well as jobs for other OSes completing in 2-3 minutes.

In terms of optimisation, the CI jobs enabled for pim/pim-sieve-editor need
to be reviewed, as it is running inappropriate jobs considering the nature
of that repository.
The results of those runs contributed to 2 hours of wasted CI time.

Data for all this is attached.

That means it is actually not possible to make it non-disruptive, as doing
it at a different time would just be a means of favouring one timezone (say
EU) over others - it simply takes a significant amount of time to rebuild
the world (which is essentially what a Gear release entails).


>
> > Regards,
> > Ingo
>

Thanks,
Ben