Re: CI congestion/starvation

Hi,
today we also have a lot of congestion. After discussion with Ben,
it's due to a new Gear update which uses the resources of the CI for
multiple hours.
Would it be possible to spread the changes done to each repo during a
full day (with sleeps between each git push) instead of doing them at
once to let other projects use the CI?

Cheers,

Johnny

Le dim. 15 févr. 2026 à 19:26, Ben Cooksley <[email protected]> a écrit :
>
> On Mon, Feb 16, 2026 at 3:52 AM Alexander Semke <[email protected]> wrote:
>>
>> On 13/02/26 12:37, Ben Cooksley wrote:
>> > [...]
>> > Resource utilisation wise, i've not looked into whether there has been
>> > a significant bump in the number of jobs, but over the past year some
>> > additional CD support has been added so that indicates some trouble there.
>> > System utilisation stats since the start of February are rather telling
>> >
>> >           full_path           |    time_used     | job_count
>> > ------------------------------+------------------+-----------
>> > graphics/krita               | 228:03:25.300251 |       451
>> > plasma/kwin                  | 164:34:16.119352 |      2032
>> > plasma/plasma-workspace      | 136:03:27.534157 |      1044
>> > multimedia/kdenlive          | 113:13:39.087112 |       566
>> > network/ruqola               | 89:02:24.704107  |       369
>> > pim/messagelib               | 83:58:38.936333  |       648
>> > graphics/drawy               | 77:38:55.987716  |      2101
>> > network/neochat              | 76:37:12.290371  |      1069
>> > plasma/plasma-desktop        | 62:27:04.468738  |       619
>> > utilities/kate               | 59:05:46.517227  |       438
>> > office/kmymoney              | 56:58:08.349153  |       241
>> > education/labplot            | 55:19:06.879103  |       423
>> > frameworks/kio               | 43:04:55.33379   |       744
>> > kde-linux/kde-linux-packages | 39:40:15.332536  |        63
>> > libraries/ktextaddons        | 31:46:16.168865  |       356
>> > office/calligra              | 31:33:35.74806   |        62
>> > kdevelop/kdevelop            | 30:57:38.655126  |       113
>> > education/kstars             | 30:04:42.328143  |        71
>> > sysadmin/craft-ci            | 28:36:13.927477  |        53
>> > bjordan/kdenlive             | 24:13:20.564568  |       191
>> >
>> >
>> > Contrast that with the full month of September 2025:
>> >
>> >            full_path           |    time_used     | job_count
>> > -------------------------------+------------------+-----------
>> > graphics/krita                | 382:38:14.237481 |       882
>> > plasma/kwin                   | 365:27:01.96681  |      3531
>> > multimedia/kdenlive           | 294:59:01.874773 |      1299
>> > network/neochat               | 233:26:31.601019 |      2298
>> > packaging/flatpak-kde-runtime | 225:01:38.542109 |       269
>> > network/ruqola                | 219:56:34.415103 |       946
>> > plasma/plasma-workspace       | 194:15:09.699541 |      2014
>> > sysadmin/ci-management        | 118:10:28.754041 |       233
>> > libraries/ktextaddons         | 105:25:58.724964 |      1084
>> > office/kmymoney               | 101:50:27.612055 |       572
>> > plasma/plasma-desktop         | 101:25:05.185953 |      1148
>> > kde-linux/kde-linux-packages  | 101:17:06.284095 |       144
>> > education/rkward              | 84:32:07.118316  |      1033
>> > kdevelop/kdevelop             | 84:04:58.804139  |       270
>> > utilities/kate                | 83:49:15.561603  |       663
>> > frameworks/kio                | 76:46:04.377597  |      1032
>> > graphics/okular               | 74:58:19.8886    |       628
>> > pim/akonadi                   | 62:23:56.252831  |       531
>> > pim/itinerary                 | 60:05:52.269941  |       578
>> > network/kaidan                | 56:37:44.446695  |      1773
>>
>>
>> Thanks Ben, this is an interesting information. Would it be feasible for
>> you to send such a report to the mailing list on the monthly base? Or
>> even add this as the output for every merge request on gitlab? I think
>> teams can probably easily improve in the various areas, at least in some
>> of them, and optimize the build and the tests to reduce the overall
>> runtime. Providing more transparency into the consumption of resources
>> on CI per project, per job and per test can support such optimizations
>> and will also help to raise the awareness for the impact of certain
>> changes on CI.
>
>
> It has to be manually put together based on queries from the database, so can't be put on every MR.
> It can be run periodically though.
>
>>
>>
>> For me, waiting for CI is really annoying and also disrupting personal
>> workflows sometimes (waiting today for 3h already in
>> https://invent.kde.org/education/labplot/-/merge_requests/845, gave up
>> and switched to a different branch already and will need to go back in
>> case of errors). For LabPlot we did already multiple optimizations in
>
>
> Looks like we dropped 2 runners again (node2 and node3) which i've now revived.
> This hasn't been helped by a number of large projects doing quite a bit of activity today (which is unusual for a Sunday)
>
>              full_path              |    time_used    | job_count
> ------------------------------------+-----------------+-----------
> plasma/plasma-desktop              | 07:25:58.851394 |        44
> graphics/krita                     | 07:16:57.78504  |        12
> multimedia/kdenlive                | 05:17:33.323612 |        30
> office/kmymoney                    | 03:54:42.976902 |        16
> pehg/kaidan                        | 03:16:03.91073  |        30
> snap-packaging/kf6-core-sdk        | 03:15:04.291499 |         3
> system/dolphin                     | 03:10:27.969557 |        48
> utilities/ark                      | 03:04:04.142    |        33
> network/neochat                    | 02:59:31.366648 |        85
> education/labplot                  | 02:50:22.827995 |        23
> libraries/ktextaddons              | 02:44:16.520082 |        35
> pim/messagelib                     | 02:17:45.340389 |        14
> pim/itinerary                      | 02:14:12.165157 |         8
> cwo/plasma-desktop                 | 02:09:26.453516 |         7
> deeptangshusaha/kdeconnect-android | 01:42:42.818736 |         9
> office/skrooge                     | 01:35:08.920711 |         9
> qazcetelic/kdeconnect-android      | 01:13:10.125977 |         6
> graphics/drawy                     | 01:11:45.894211 |        44
> lito/kdeconnect-kde                | 01:06:17.659945 |        24
> graphics/digikam                   | 01:05:11.594924 |        13
>
>
>>
>>
>> the past - not because of the long pending time but just because the
>> runtime itself was painful and we plan to do another iteration and to
>> invest more here soon. The problem with the pending time come on top now
>> and I think teams should check what can be improved here in general
>> without putting the focus on more hardware only.
>
>
> Optimisations, especially when they are things that should be done anyway - like removing tests that are broken (timeout, fail) or which don't actually test anything of meaning - are always useful.
>
> I do believe in this instance there is a hardware element to it - even if it is to simply improve reliability.
>
> i've noticed that crashes tend to happen most frequently during the launch stage as well, so i'm considering adding locking to vm-runner so only one VM can be starting at any given moment - there is a possibility we are triggering a bug in KVM itself (or elsewhere in the kernel as shipped by Debian). Given the reliability has been declining since the initial introduction of VM based CI i'm not 100% convinced this is the case though.
>
>>
>>
>>
>> --
>>
>> Alexander
>>
>
> Cheers,
> Ben