Re: Proposal for a more efficient version of the Parallel scheduler

--===============2145382232592946622==
Content-Type: multipart/alternative; boundary="000000000000224a4005ef41e4be"

--000000000000224a4005ef41e4be
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Adam,

We just pushed a new parallel job implementation by Andrew Morrow which
might be worth trying in your environment.
It's currently an experimental feature enabled by --experimental=3Dtm_v2 or
SetOption('experimental','tm_v2')

-Bill

On Tue, Jul 9, 2019 at 11:35 AM Adam Gross via Scons-dev <
[email protected]> wrote:

> In VMware builds, the sheer number of tasks (33000 leaf tasks in the firs=
t
> iteration of ESX builds, for example) means that Parallel.start can take =
up
> to 20 minutes simply collecting and preparing tasks over the course of a
> build. I started a project to look at batch task handling in order to
> support remote caching (e.g. asking the cache for 1000 nodes at a time
> instead of 1) and realized that the approach that I want to take actually
> makes normal builds more efficient as well. In this e-mail, I=E2=80=99d l=
ike to
> explain my proposal so I can get your thoughts on it.
>
>
>
> --- Current Parallel.start performance problems ---
>
>
>
> Reference:
> https://github.com/SCons/scons/blob/master/src/engine/SCons/Job.py#L369
>
>
>
> In the current implementation, SCons collects just enough tasks to
> dispatch to the thread pool such that the number of active jobs is equal =
to
> the max number of jobs. It then waits for at least one job to be done,
> gathers all finished jobs, then repeats the process of collecting enough
> tasks to have jobs=3D=3Dself.maxjobs.
>
>
>
> Waiting on at least one job to be done misses an opportunity to keep
> calling taskmaster.next_task() and task.prepare() while jobs are active.
> These calls are not cheap for many reasons, including that it initiates
> scanning of source nodes.
>
>
>
> --- Proposal ---
>
>
>
> A first rough draft is contained in draft pull request
> https://github.com/SCons/scons/pull/3404 . In this form it is an
> alternative child class of Parallel; it could just replace it if people
> felt strongly.
>
>
>
> I would like to implement an alternative to the Parallel class that only
> waits for jobs to complete if there are no tasks left (i.e.
> taskmaster.next_task() returns None). It is optimized for keeping
> jobs=3D=3Dself.maxjobs but otherwise, will keep looking for more tasks. I=
f
> there are no more tasks left, it waits for a job to complete and then
> rechecks whether there are any tasks left, just in case other tasks were
> unblocked by its completion.
>
>
>
> One very useful side effect is that this class will be collecting lists o=
f
> tasks instead of operating on one at a time, so it serves as a useful
> building block towards remote caching. The current one-at-a-time cache
> retrieval approach wouldn=E2=80=99t work for remote caching due to networ=
k latency
> but this approach can.
>
>
>
> Please let me know what you think either over e-mail or on the
> aforementioned pull request.
>
>
>
> Thanks,
>
> Adam Gross
> _______________________________________________
> Scons-dev mailing list
> [email protected]
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>

--000000000000224a4005ef41e4be
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Adam,</div><div><br></div><div>We just pushed a new p=
arallel job implementation by Andrew Morrow which might be worth trying in =
your environment.</div><div>It&#39;s currently an experimental feature enab=
led by --experimental=3Dtm_v2 or SetOption(&#39;experimental&#39;,&#39;tm_v=
2&#39;)</div><div><br></div><div>-Bill<br></div></div><br><div class=3D"gma=
il_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Tue, Jul 9, 2019 at 11:3=
5 AM Adam Gross via Scons-dev &lt;<a href=3D"mailto:[email protected]">sc=
[email protected]</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote"=
 style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);p=
adding-left:1ex"><div class=3D"msg-5696285289936337105">





<div lang=3D"EN-US">
<div class=3D"m_-5696285289936337105WordSection1">
<p class=3D"MsoNormal">In VMware builds, the sheer number of tasks (33000 l=
eaf tasks in the first iteration of ESX builds, for example) means that Par=
allel.start can take up to 20 minutes simply collecting and preparing tasks=
 over the course of a build. I started
 a project to look at batch task handling in order to support remote cachin=
g (e.g. asking the cache for 1000 nodes at a time instead of 1) and realize=
d that the approach that I want to take actually makes normal builds more e=
fficient as well. In this e-mail,
 I=E2=80=99d like to explain my proposal so I can get your thoughts on it.<=
u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal">--- Current Parallel.start performance problems ---<=
u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal">Reference: <a href=3D"https://github.com/SCons/scons=
/blob/master/src/engine/SCons/Job.py#L369" target=3D"_blank">
https://github.com/SCons/scons/blob/master/src/engine/SCons/Job.py#L369</a>=
<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal">In the current implementation, SCons collects just e=
nough tasks to dispatch to the thread pool such that the number of active j=
obs is equal to the max number of jobs. It then waits for at least one job =
to be done, gathers all finished jobs,
 then repeats the process of collecting enough tasks to have jobs=3D=3Dself=
.maxjobs.<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal">Waiting on at least one job to be done misses an opp=
ortunity to keep calling taskmaster.next_task() and task.prepare() while jo=
bs are active. These calls are not cheap for many reasons, including that i=
t initiates scanning of source nodes.<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal">--- Proposal ---<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal">A first rough draft is contained in draft pull reque=
st <a href=3D"https://github.com/SCons/scons/pull/3404" target=3D"_blank">
https://github.com/SCons/scons/pull/3404</a> . In this form it is an altern=
ative child class of Parallel; it could just replace it if people felt stro=
ngly.<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal">I would like to implement an alternative to the Para=
llel class that only waits for jobs to complete if there are no tasks left =
(i.e. taskmaster.next_task() returns None). It is optimized for keeping job=
s=3D=3Dself.maxjobs but otherwise, will
 keep looking for more tasks. If there are no more tasks left, it waits for=
 a job to complete and then rechecks whether there are any tasks left, just=
 in case other tasks were unblocked by its completion.<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal">One very useful side effect is that this class will =
be collecting lists of tasks instead of operating on one at a time, so it s=
erves as a useful building block towards remote caching. The current one-at=
-a-time cache retrieval approach wouldn=E2=80=99t
 work for remote caching due to network latency but this approach can.<u></=
u><u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal">Please let me know what you think either over e-mail=
 or on the aforementioned pull request.<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal">Thanks,<u></u><u></u></p>
<p class=3D"MsoNormal">Adam Gross<u></u><u></u></p>
</div>
</div>

_______________________________________________<br>
Scons-dev mailing list<br>
<a href=3D"mailto:[email protected]" target=3D"_blank">[email protected]=
g</a><br>
<a href=3D"https://pairlist2.pair.net/mailman/listinfo/scons-dev" rel=3D"no=
referrer" target=3D"_blank">https://pairlist2.pair.net/mailman/listinfo/sco=
ns-dev</a><br>
</div></blockquote></div>

--000000000000224a4005ef41e4be--

--===============2145382232592946622==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Scons-dev mailing list
[email protected]
https://pairlist2.pair.net/mailman/listinfo/scons-dev

--===============2145382232592946622==--