Re: accumulating projects in kate lead to excessive kate startup time due to git

On Sunday, 19 June 2022 12:37:21 PDT Milian Wolff wrote:
> OK, this is apparently totally unrelated to git and kate. Thiago, do you
> happen to have an insight here maybe? Is it known that using QProcess can
> really badly influence the runtime behavior of malloc in other threads?

It could... but it shouldn't be that meaningful. While the child process is 
between fork() and execve(), it is sharing pages with the parent process but 
they are COW. So attempting to allocate memory in the parent process *could* 
cause that very COW to happen.

If you waitForStarted() and the problem disappears, then this is why. But I 
doubt it will be.

> Here's a small example to trigger this behavior already:
> 
> https://invent.kde.org/-/snippets/2239
> 
> I have nproc == 24. Let's run this without any external processes:

The example does waitForFinished, so the child process is surely past the 
execve() stage.

The threading code won't linearly scale. It is entirely thread-safe, but 
you'll see contentions. The entire parent process' virtual address space is 
shared with all children (one per thread) and given that many threads, the 
chances are going to be very high that one thread will malloc() while at least 
one child is still between fork() and execve(), thus causing COW.

That said, this problem wouldn't be exclusive to memory allocations. The 
simple fact of *running* threads will cause COW on those threads' stacks, plus 
whatever other structures they touch.

-- 
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
   Software Architect - Intel DPG Cloud Engineering