Re: Skyrocketing test run times

On 2/4/26 10:16 AM, Ben Cooksley wrote:
> On Wed, Feb 4, 2026 at 10:05 PM Vlad Zahorodnii 
> <[email protected]> wrote:
>
>     On 2/4/26 10:26 AM, Ben Cooksley wrote:
>>     On Wed, Feb 4, 2026 at 9:07 PM Ben Cooksley <[email protected]>
>>     wrote:
>>
>>         On Wed, Feb 4, 2026 at 8:25 PM Vlad Zahorodnii
>>         <[email protected]> wrote:
>>
>>             Hello,
>>
>>
>>         HI Vlad,
>>
>>
>>             Over approximately the past weekend something happened in
>>             our CI and now
>>             it takes quite long time for tests to run. For example,
>>             in kwin, we have
>>             a test that used to run for about 20 seconds, and now it
>>             takes about 5
>>             or so minutes to finish running. Speaking for kwin, there
>>             were no
>>             changes that could increase test run times so dramatically.
>>
>>             January 26th:
>>
>>                      Start  61: kwin-testOutputChanges
>>               61/158 Test  #61: kwin-testOutputChanges
>>             .............................   Passed  19.36 sec
>>
>>             January 29th:
>>
>>                      Start  61: kwin-testOutputChanges
>>               61/158 Test  #61: kwin-testOutputChanges
>>             .............................   Passed  43.93 sec
>>
>>             January 30th:
>>
>>                      Start  61: kwin-testOutputChanges
>>               61/158 Test  #61: kwin-testOutputChanges
>>             .............................   Passed  45.91 sec
>>
>>             Februrary 3rd:
>>
>>                      Start  61: kwin-testOutputChanges
>>               61/158 Test  #61: kwin-testOutputChanges
>>             .............................   Passed 254.19 sec
>>
>>             FreeBSD appears to be fine.
>>
>>             We suspect that test run times blew up due to enabling
>>             LSAN in various
>>             libraries (kwin itself has no LSAN enabled yet). The
>>             issue doesn't
>>             appear to be specific to only kwin, people reported that
>>             they've seen
>>             similar issues in other projects too. Maybe something
>>             else happened to
>>             CI that sysadmins will be able to clarify.
>>
>>
>>         Nothing else happened to CI recently aside from the
>>         enablement of LSAN.
>>
>>         The underlying SUSE images were for Qt 6.10 at least last
>>         rebuilt on January 25th, which is well before your "last
>>         good" date.
>>
>>         The only change to CI between January 30th and February 3rd
>>         was fast_unwind_on_malloc=0 being added by default, even
>>         though it is primarily for the benefit of LSAN.
>>         I've now made changes to only set fast_unwind_on_malloc=0 if
>>         LSAN is explicitly enabled for a repository - hard to tell if
>>         that will fix the issue though as KWin takes a while to build.
>>
>>
>>     For the record, as per
>>     https://invent.kde.org/plasma/kwin/-/jobs/3958895 which completed
>>     moments ago:
>
>     That branch contained some other things that could interfere with
>     test results. I started
>     https://invent.kde.org/plasma/kwin/-/jobs/3959078 and yeah it
>     looks like test run times are back to the Jan 29-30th level.
>
>     So, it seems like fast_unwind_on_malloc=0 is the culprit then?
>
>
> Yes, quoting from https://source.android.com/docs/security/test/asan
>
> [quote]
> ASan uses a fast, frame-pointer-based unwinder to record a stack trace 
> for every memory allocation and deallocation event in the program. 
> Most of Android is built without frame pointers. As a result, you 
> often get only one or two meaningful frames. To fix this, either 
> rebuild the library with ASan (recommended!), or with:
>
> LOCAL_CFLAGS:=-fno-omit-frame-pointer
> LOCAL_ARM_MODE:=arm
>
> Or set ASAN_OPTIONS=fast_unwind_on_malloc=0 in the process 
> environment. The latter can be very CPU-intensive, depending on the load.
> [/quote]
>
> That would align with what we're seeing here

Hi,

thanks for investigating this.

I'm a bit surprised that fast_unwind_on_malloc seems to have an effect 
when LSAN is disabled. That said, we still should have the problem when 
LSAN *is* enabled, no?

The reason I added fast_unwind_on_malloc=0 is because otherwise we would 
sometimes get unusable backtraces from LSAN, so I would not want to lose 
that.

A possible remedy could be to build Qt with ASAN support. That's one 
additional flag in the buildsystem invokation. fast_unwind_on_malloc 
still would make sense to cover system libraries, but maybe it would be 
okay to do without that.

Cheers

Nico