Re: Skyrocketing test run times

On 2/4/26 10:26 AM, Ben Cooksley wrote:
> On Wed, Feb 4, 2026 at 9:07 PM Ben Cooksley <[email protected]> wrote:
>
>     On Wed, Feb 4, 2026 at 8:25 PM Vlad Zahorodnii
>     <[email protected]> wrote:
>
>         Hello,
>
>
>     HI Vlad,
>
>
>         Over approximately the past weekend something happened in our
>         CI and now
>         it takes quite long time for tests to run. For example, in
>         kwin, we have
>         a test that used to run for about 20 seconds, and now it takes
>         about 5
>         or so minutes to finish running. Speaking for kwin, there were no
>         changes that could increase test run times so dramatically.
>
>         January 26th:
>
>                  Start  61: kwin-testOutputChanges
>           61/158 Test  #61: kwin-testOutputChanges
>         .............................   Passed   19.36 sec
>
>         January 29th:
>
>                  Start  61: kwin-testOutputChanges
>           61/158 Test  #61: kwin-testOutputChanges
>         .............................   Passed   43.93 sec
>
>         January 30th:
>
>                  Start  61: kwin-testOutputChanges
>           61/158 Test  #61: kwin-testOutputChanges
>         .............................   Passed   45.91 sec
>
>         Februrary 3rd:
>
>                  Start  61: kwin-testOutputChanges
>           61/158 Test  #61: kwin-testOutputChanges
>         .............................   Passed  254.19 sec
>
>         FreeBSD appears to be fine.
>
>         We suspect that test run times blew up due to enabling LSAN in
>         various
>         libraries (kwin itself has no LSAN enabled yet). The issue
>         doesn't
>         appear to be specific to only kwin, people reported that
>         they've seen
>         similar issues in other projects too. Maybe something else
>         happened to
>         CI that sysadmins will be able to clarify.
>
>
>     Nothing else happened to CI recently aside from the enablement of
>     LSAN.
>
>     The underlying SUSE images were for Qt 6.10 at least last rebuilt
>     on January 25th, which is well before your "last good" date.
>
>     The only change to CI between January 30th and February 3rd
>     was fast_unwind_on_malloc=0 being added by default, even though it
>     is primarily for the benefit of LSAN.
>     I've now made changes to only set fast_unwind_on_malloc=0 if LSAN
>     is explicitly enabled for a repository - hard to tell if that will
>     fix the issue though as KWin takes a while to build.
>
>
> For the record, as per 
> https://invent.kde.org/plasma/kwin/-/jobs/3958895 which completed 
> moments ago:

That branch contained some other things that could interfere with test 
results. I started https://invent.kde.org/plasma/kwin/-/jobs/3959078 and 
yeah it looks like test run times are back to the Jan 29-30th level.

So, it seems like fast_unwind_on_malloc=0 is the culprit then?

Regards,
Vlad


>         Start  61: kwin-testOutputChanges
>  61/158 Test  #61: kwin-testOutputChanges 
> .............................   Passed   19.58 sec
>
>
>         Either way, the current state of CI is not great.
>         Hypothetically, test
>         timeouts can be increased but QSignalSpy's have hardcoded
>         timeouts that
>         can be too low for the current CI. And in case of kwin, 5
>         minutes for a
>         test is simply not a workable thing.
>
>         Regards,
>         Vlad
>
>
>     Thanks,
>     Ben
>
>
> Cheers,
> Ben