Re: Speeding up subprocess spawns

Have you tried this?
https://github.com/SCons/scons/pull/3703

On Thu, Jun 25, 2020 at 12:09 AM Yonatan <[email protected]> wrote:

> Hello, SCons community,
>
> Recently, I debugged the build-process performance of a large project
> built with SCons.
> Trying to figure out exactly where time is spent (gcc vs. scons vs.
> linking), I ran a system-wide "perf".
> One of the things that surprised me was that Python spends a significant
> amount of time in fork() and
> execve() (in the kernel).
>
> Almost all of the time of fork() was spent in copy_page_range(), and of
> execve() in unmap_page_range().
> To my understanding, these 2 kernel functions are responsible to copy the
> process' page tables when
> fork()ing, and to remove existing mappings when execve()ing (as you switch
> to a new address space with
> new mappings). The larger resident set a process has, the more time those
> 2 functions will take.
>
> In that project, the SCons process grows very large in RAM (over 1GB);
> this is obviously specific to that
> project, and what it loads into Python. With a resident set that large,
> the fork()-execve() sequence incurs a
> significant cost.
>
> To overcome this I thought of 2 possible solutions:
> 1. Use vfork()-execve() in subprocess.Popen() - this is not supported in
> yet CPython, but it has been
>     suggested and a PR exists. See https://bugs.python.org/issue35823
> 2. Run a minimal "spawner" process and execute subprocess.Popen() requests
> via it. I've written a PoC, and
>     tested MongoDB's compilation using it. SCons doesn't hog too much
> CPU/memory in the MongoDB case,
>     but the spawner change still introduces a tiny improvement.
>     I also tried manually increasing the memory usage of SCons (by
> creating many Python objects), which
>     lead to much more visible effects for this change.
>     You can see it here:
> https://github.com/Jongy/scons/tree/process-spawner
>
> I hope the vfork() PR gets merged, and meanwhile I was wondering if the
> "spawner" approach sounds feasible to you.
> I'd be happy to push it towards mainlining.
>
> Thanks,
> Yonatan
> _______________________________________________
> Scons-users mailing list
> [email protected]
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>

_______________________________________________
Scons-users mailing list
[email protected]
https://pairlist4.pair.net/mailman/listinfo/scons-users