Re: Skyrocketing test run times

Thomas Baumgart <[email protected]> Wed, 04 Feb 2026 10:02:47 +0100
Newsgroups	gmane.comp.kde.devel.general,gmane.comp.kde.devel.plasma,gmane.comp.kde.devel.frameworks
Message-ID	<3688312.dWV9SEqChM@sy-346-nb>
Hi,

On Mittwoch, 4. Februar 2026 09:26:49 CET Ben Cooksley wrote:

> On Wed, Feb 4, 2026 at 9:07 PM Ben Cooksley <[email protected]> wrote:
> 
> > On Wed, Feb 4, 2026 at 8:25 PM Vlad Zahorodnii <[email protected]>
> > wrote:
> >
> >> Hello,
> >>
> >
> > HI Vlad,
> >
> >
> >>
> >> Over approximately the past weekend something happened in our CI and now
> >> it takes quite long time for tests to run. For example, in kwin, we have
> >> a test that used to run for about 20 seconds, and now it takes about 5
> >> or so minutes to finish running. Speaking for kwin, there were no
> >> changes that could increase test run times so dramatically.
> >>
> >> January 26th:
> >>
> >>          Start  61: kwin-testOutputChanges
> >>   61/158 Test  #61: kwin-testOutputChanges
> >> .............................   Passed   19.36 sec
> >>
> >> January 29th:
> >>
> >>          Start  61: kwin-testOutputChanges
> >>   61/158 Test  #61: kwin-testOutputChanges
> >> .............................   Passed   43.93 sec
> >>
> >> January 30th:
> >>
> >>          Start  61: kwin-testOutputChanges
> >>   61/158 Test  #61: kwin-testOutputChanges
> >> .............................   Passed   45.91 sec
> >>
> >> Februrary 3rd:
> >>
> >>          Start  61: kwin-testOutputChanges
> >>   61/158 Test  #61: kwin-testOutputChanges
> >> .............................   Passed  254.19 sec
> >>
> >> FreeBSD appears to be fine.
> >>
> >> We suspect that test run times blew up due to enabling LSAN in various
> >> libraries (kwin itself has no LSAN enabled yet). The issue doesn't
> >> appear to be specific to only kwin, people reported that they've seen
> >> similar issues in other projects too. Maybe something else happened to
> >> CI that sysadmins will be able to clarify.
> >>
> >
> > Nothing else happened to CI recently aside from the enablement of LSAN.
> >
> > The underlying SUSE images were for Qt 6.10 at least last rebuilt on
> > January 25th, which is well before your "last good" date.
> >
> > The only change to CI between January 30th and February 3rd
> > was fast_unwind_on_malloc=0 being added by default, even though it is
> > primarily for the benefit of LSAN.
> > I've now made changes to only set fast_unwind_on_malloc=0 if LSAN is
> > explicitly enabled for a repository - hard to tell if that will fix the
> > issue though as KWin takes a while to build.
> >
> 
> For the record, as per https://invent.kde.org/plasma/kwin/-/jobs/3958895
> which completed moments ago:
> 
>         Start  61: kwin-testOutputChanges
>  61/158 Test  #61: kwin-testOutputChanges .............................
> Passed   19.58 sec


I noticed that same increase while running KMyMoney tests and especially
one that has many ctor/dtor calls failed due to hitting the runtime limit
of 60 seconds. 

https://invent.kde.org/office/kmymoney/-/jobs/3924048
  search for "13/79" which finished in < 3 secs

https://invent.kde.org/office/kmymoney/-/pipelines/1153369
  This is the pipeline that failed first (only for opensuse though)
  https://invent.kde.org/office/kmymoney/-/jobs/3947531
  Search for "13/79" and see the timeout after 60 secs

The test case btw did not change at all.

Maybe, that provides some indications what causes these extended
runtimes.

-- 

Regards

Thomas Baumgart

-------------------------------------------------------------
Real programmers don’t write workarounds. They tell their users
to upgrade their software. -- Shlomi Fish
-------------------------------------------------------------
signature.asc (application/pgp-signature, 868 B)