Re: Alternatives to -j parallel

--===============3926930319907716334==
Content-Type: multipart/alternative; boundary=94eb2c060266923bc3053bd3b0a7

--94eb2c060266923bc3053bd3b0a7
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Thanks,
Here's another way of describing my problem.

I have a configuration that allows me to substitute distcc for cc/cxx. This
works fine, and I can distribute compilation across the 4 nodes available
with circleCI, when I supply a -j N option to make.

However, it appears that the linking (ld) phase is carried out using a
separate mechanism - i.e. not from distcc. Thus the requirement of only one
job on the localhost is not respected by the link phase because distcc
isn't executing the linker. In the circleCI system this leads to problems
with memory.

I'd therefore like to be able to tell distcc to be parallel without telling
the rest of the makefile also being parallel. Obviously this would require
that there is some other mechanism (besides -j) - perhaps an environment
variable - that triggers parallel behaviour in distcc.

In short - the cc part of distcc works fine. However, in this build system,
ld is not being run from distcc. I don't want multiple ld jobs running, so
I'd like to be able to run make -j1 but instruct distcc to use a higher
level of j.




On Tue, Sep 6, 2016 at 6:42 PM, =C5=81ukasz Tasz <[email protected]> wrote:

> Could you please describe more what you would like to achieve?
> -j does net decide it compilation will be done with distcc or not,
> -j must take into consideration how many distcc agent you have.
>
> you have two dimensions, one is if distcc should be used, and second is
> about parallelism.
> first dimension is controlled in build system, second can be controlled b=
y
> build system, a but also may be limited by distcc.
>
> localslots and localslots_cpp controls parallelism from distcc point of
> view.
>
> regards
> Lukasz
>
> =C5=81ukasz Tasz
> RTKW
>
> 2016-09-06 4:11 GMT+02:00 Richard Beare <[email protected]>:
>
>> Hi,
>> I'm having a few problems using distcc for a large project on the
>> circleCI infrastructure, which has 4GB limits for the compilation hosts.
>>
>> Occasionally I see this limit being exceeded when 2 link jobs run on the
>> main server at the same time. This happens despite the --localslots=3D1 =
and
>> localhost/1 settings in the distcc hosts file.
>>
>> I suspect the problem is that the large and complex cmake project that i=
s
>> being built has its own link mechanism. So my question is whether there =
is
>> an alternative way to pass the parallel option to distcc besides make -j=
N?
>> This would allow the other compile steps that might not be under distcc
>> control to run serially on the local host.
>>
>> Thanks
>>
>> __
>> distcc mailing list            http://distcc.samba.org/
>> To unsubscribe or change options:
>> https://lists.samba.org/mailman/listinfo/distcc
>>
>
>

--94eb2c060266923bc3053bd3b0a7
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Thanks,<div>Here&#39;s another way of describing my proble=
m.</div><div><br></div><div>I have a configuration that allows me to substi=
tute distcc for cc/cxx. This works fine, and I can distribute compilation a=
cross the 4 nodes available with circleCI, when I supply a -j N option to m=
ake.</div><div><br></div><div>However, it appears that the linking (ld) pha=
se is carried out using a separate mechanism - i.e. not from distcc. Thus t=
he requirement of only one job on the localhost is not respected by the lin=
k phase because distcc isn&#39;t executing the linker. In the circleCI syst=
em this leads to problems with memory.</div><div><br></div><div>I&#39;d the=
refore like to be able to tell distcc to be parallel without telling the re=
st of the makefile also being parallel. Obviously this would require that t=
here is some other mechanism (besides -j) - perhaps an environment variable=
 - that triggers parallel behaviour in distcc.</div><div><br></div><div>In =
short - the cc part of distcc works fine. However, in this build system, ld=
 is not being run from distcc. I don&#39;t want multiple ld jobs running, s=
o I&#39;d like to be able to run make -j1 but instruct distcc to use a high=
er level of j.</div><div><br></div><div><br></div><div><br></div></div><div=
 class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Tue, Sep 6, 2016 a=
t 6:42 PM, =C5=81ukasz Tasz <span dir=3D"ltr">&lt;<a href=3D"mailto:lukasz@=
tasz.eu" target=3D"_blank">[email protected]</a>&gt;</span> wrote:<br><blockqu=
ote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc s=
olid;padding-left:1ex"><div dir=3D"ltr"><div><div><div><div><div><div>Could=
 you please describe more what you would like to achieve?<br></div>-j does =
net decide it compilation will be done with distcc or not, <br></div>-j mus=
t take into consideration how many distcc agent you have.<br><br></div>you =
have two dimensions, one is if distcc should be used, and second is about p=
arallelism. <br></div><div>first dimension is controlled in build system, s=
econd can be controlled by build system, a but also may be limited by distc=
c. <br></div><div><br></div>localslots and localslots_cpp controls parallel=
ism from distcc point of view.<br><br></div>regards<br></div>Lukasz<br></di=
v><div class=3D"gmail_extra"><br clear=3D"all"><div><div data-smartmail=3D"=
gmail_signature"><div dir=3D"ltr"><div>=C5=81ukasz Tasz<br></div><div>RTKW<=
br></div></div></div></div>
<br><div class=3D"gmail_quote"><div><div class=3D"h5">2016-09-06 4:11 GMT+0=
2:00 Richard Beare <span dir=3D"ltr">&lt;<a href=3D"mailto:richard.beare@gm=
ail.com" target=3D"_blank">[email protected]</a>&gt;</span>:<br></div=
></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-=
left:1px #ccc solid;padding-left:1ex"><div><div class=3D"h5"><div dir=3D"lt=
r">Hi,<div>I&#39;m having a few problems using distcc for a large project o=
n the circleCI infrastructure, which has 4GB limits for the compilation hos=
ts.</div><div><br></div><div>Occasionally I see this limit being exceeded w=
hen 2 link jobs run on the main server at the same time. This happens despi=
te the --localslots=3D1 and localhost/1 settings in the distcc hosts file.<=
/div><div><br></div><div>I suspect the problem is that the large and comple=
x cmake project that is being built has its own link mechanism. So my quest=
ion is whether there is an alternative way to pass the parallel option to d=
istcc besides make -jN? This would allow the other compile steps that might=
 not be under distcc control to run serially on the local host.</div><div><=
br></div><div>Thanks</div></div>
<br></div></div>__<br>
distcc mailing list=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a href=3D"htt=
p://distcc.samba.org/" rel=3D"noreferrer" target=3D"_blank">http://distcc.s=
amba.org/</a><br>
To unsubscribe or change options:<br>
<a href=3D"https://lists.samba.org/mailman/listinfo/distcc" rel=3D"noreferr=
er" target=3D"_blank">https://lists.samba.org/mailma<wbr>n/listinfo/distcc<=
/a><br></blockquote></div><br></div>
</blockquote></div><br></div>

--94eb2c060266923bc3053bd3b0a7--


--===============3926930319907716334==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

__
distcc mailing list            http://distcc.samba.org/
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/distcc
--===============3926930319907716334==--