Re: Alternatives to -j parallel

--===============0766985697066524343==
Content-Type: multipart/alternative; boundary=001a114453be2213ce053bd4a9bb

--001a114453be2213ce053bd4a9bb
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

I think I got your point,

playing with MKFLAGS with cmake will be a tricky part and complex at the
end.
I would recommend to use distcc and limit --localslots=3D1 and assure that
linking starts with distcc at the begining.
Make sure that your LD variable is prefixed with distcc.

In my opinion it will be the easiest way.

In case of pure make it is very easy, enough is to call $MAKE MKFLAGS=3D an=
d
-j then is 1, but it is not a case when talking about cmake buildsystem.

regards
L.

=C5=81ukasz Tasz
RTKW

2016-09-06 11:47 GMT+02:00 Richard Beare <[email protected]>:

> Thanks,
> Here's another way of describing my problem.
>
> I have a configuration that allows me to substitute distcc for cc/cxx.
> This works fine, and I can distribute compilation across the 4 nodes
> available with circleCI, when I supply a -j N option to make.
>
> However, it appears that the linking (ld) phase is carried out using a
> separate mechanism - i.e. not from distcc. Thus the requirement of only o=
ne
> job on the localhost is not respected by the link phase because distcc
> isn't executing the linker. In the circleCI system this leads to problems
> with memory.
>
> I'd therefore like to be able to tell distcc to be parallel without
> telling the rest of the makefile also being parallel. Obviously this woul=
d
> require that there is some other mechanism (besides -j) - perhaps an
> environment variable - that triggers parallel behaviour in distcc.
>
> In short - the cc part of distcc works fine. However, in this build
> system, ld is not being run from distcc. I don't want multiple ld jobs
> running, so I'd like to be able to run make -j1 but instruct distcc to us=
e
> a higher level of j.
>
>
>
>
> On Tue, Sep 6, 2016 at 6:42 PM, =C5=81ukasz Tasz <[email protected]> wrote:
>
>> Could you please describe more what you would like to achieve?
>> -j does net decide it compilation will be done with distcc or not,
>> -j must take into consideration how many distcc agent you have.
>>
>> you have two dimensions, one is if distcc should be used, and second is
>> about parallelism.
>> first dimension is controlled in build system, second can be controlled
>> by build system, a but also may be limited by distcc.
>>
>> localslots and localslots_cpp controls parallelism from distcc point of
>> view.
>>
>> regards
>> Lukasz
>>
>> =C5=81ukasz Tasz
>> RTKW
>>
>> 2016-09-06 4:11 GMT+02:00 Richard Beare <[email protected]>:
>>
>>> Hi,
>>> I'm having a few problems using distcc for a large project on the
>>> circleCI infrastructure, which has 4GB limits for the compilation hosts=
.
>>>
>>> Occasionally I see this limit being exceeded when 2 link jobs run on th=
e
>>> main server at the same time. This happens despite the --localslots=3D1=
 and
>>> localhost/1 settings in the distcc hosts file.
>>>
>>> I suspect the problem is that the large and complex cmake project that
>>> is being built has its own link mechanism. So my question is whether th=
ere
>>> is an alternative way to pass the parallel option to distcc besides mak=
e
>>> -jN? This would allow the other compile steps that might not be under
>>> distcc control to run serially on the local host.
>>>
>>> Thanks
>>>
>>> __
>>> distcc mailing list            http://distcc.samba.org/
>>> To unsubscribe or change options:
>>> https://lists.samba.org/mailman/listinfo/distcc
>>>
>>
>>
>

--001a114453be2213ce053bd4a9bb
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div><div><div><div><div><div><div>I think I got your=
 point, <br></div><br></div>playing with MKFLAGS with cmake will be a trick=
y part and complex at the end.<br></div>I would recommend to use distcc and=
 limit --localslots=3D1 and assure that linking starts with distcc at the b=
egining.<br></div>Make sure that your LD variable is prefixed with distcc.<=
br><br></div>In my opinion it will be the easiest way.<br><br></div>In case=
 of pure make it is very easy, enough is to call $MAKE MKFLAGS=3D and -j th=
en is 1, but it is not a case when talking about cmake buildsystem.<br><br>=
</div>regards<br></div>L.<br></div><div class=3D"gmail_extra"><br clear=3D"=
all"><div><div class=3D"gmail_signature" data-smartmail=3D"gmail_signature"=
><div dir=3D"ltr"><div>=C5=81ukasz Tasz<br></div><div>RTKW<br></div></div><=
/div></div>
<br><div class=3D"gmail_quote">2016-09-06 11:47 GMT+02:00 Richard Beare <sp=
an dir=3D"ltr">&lt;<a href=3D"mailto:[email protected]" target=3D"_bl=
ank">[email protected]</a>&gt;</span>:<br><blockquote class=3D"gmail_=
quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1=
ex"><div dir=3D"ltr">Thanks,<div>Here&#39;s another way of describing my pr=
oblem.</div><div><br></div><div>I have a configuration that allows me to su=
bstitute distcc for cc/cxx. This works fine, and I can distribute compilati=
on across the 4 nodes available with circleCI, when I supply a -j N option =
to make.</div><div><br></div><div>However, it appears that the linking (ld)=
 phase is carried out using a separate mechanism - i.e. not from distcc. Th=
us the requirement of only one job on the localhost is not respected by the=
 link phase because distcc isn&#39;t executing the linker. In the circleCI =
system this leads to problems with memory.</div><div><br></div><div>I&#39;d=
 therefore like to be able to tell distcc to be parallel without telling th=
e rest of the makefile also being parallel. Obviously this would require th=
at there is some other mechanism (besides -j) - perhaps an environment vari=
able - that triggers parallel behaviour in distcc.</div><div><br></div><div=
>In short - the cc part of distcc works fine. However, in this build system=
, ld is not being run from distcc. I don&#39;t want multiple ld jobs runnin=
g, so I&#39;d like to be able to run make -j1 but instruct distcc to use a =
higher level of j.</div><div><br></div><div><br></div><div><br></div></div>=
<div class=3D"HOEnZb"><div class=3D"h5"><div class=3D"gmail_extra"><br><div=
 class=3D"gmail_quote">On Tue, Sep 6, 2016 at 6:42 PM, =C5=81ukasz Tasz <sp=
an dir=3D"ltr">&lt;<a href=3D"mailto:[email protected]" target=3D"_blank">luka=
[email protected]</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=
=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=
=3D"ltr"><div><div><div><div><div><div>Could you please describe more what =
you would like to achieve?<br></div>-j does net decide it compilation will =
be done with distcc or not, <br></div>-j must take into consideration how m=
any distcc agent you have.<br><br></div>you have two dimensions, one is if =
distcc should be used, and second is about parallelism. <br></div><div>firs=
t dimension is controlled in build system, second can be controlled by buil=
d system, a but also may be limited by distcc. <br></div><div><br></div>loc=
alslots and localslots_cpp controls parallelism from distcc point of view.<=
br><br></div>regards<br></div>Lukasz<br></div><div class=3D"gmail_extra"><b=
r clear=3D"all"><div><div data-smartmail=3D"gmail_signature"><div dir=3D"lt=
r"><div>=C5=81ukasz Tasz<br></div><div>RTKW<br></div></div></div></div>
<br><div class=3D"gmail_quote"><div><div>2016-09-06 4:11 GMT+02:00 Richard =
Beare <span dir=3D"ltr">&lt;<a href=3D"mailto:[email protected]" targ=
et=3D"_blank">[email protected]</a>&gt;</span>:<br></div></div><block=
quote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc=
 solid;padding-left:1ex"><div><div><div dir=3D"ltr">Hi,<div>I&#39;m having =
a few problems using distcc for a large project on the circleCI infrastruct=
ure, which has 4GB limits for the compilation hosts.</div><div><br></div><d=
iv>Occasionally I see this limit being exceeded when 2 link jobs run on the=
 main server at the same time. This happens despite the --localslots=3D1 an=
d localhost/1 settings in the distcc hosts file.</div><div><br></div><div>I=
 suspect the problem is that the large and complex cmake project that is be=
ing built has its own link mechanism. So my question is whether there is an=
 alternative way to pass the parallel option to distcc besides make -jN? Th=
is would allow the other compile steps that might not be under distcc contr=
ol to run serially on the local host.</div><div><br></div><div>Thanks</div>=
</div>
<br></div></div>__<br>
distcc mailing list=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a href=3D"htt=
p://distcc.samba.org/" rel=3D"noreferrer" target=3D"_blank">http://distcc.s=
amba.org/</a><br>
To unsubscribe or change options:<br>
<a href=3D"https://lists.samba.org/mailman/listinfo/distcc" rel=3D"noreferr=
er" target=3D"_blank">https://lists.samba.org/mailma<wbr>n/listinfo/distcc<=
/a><br></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>

--001a114453be2213ce053bd4a9bb--


--===============0766985697066524343==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

__
distcc mailing list            http://distcc.samba.org/
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/distcc
--===============0766985697066524343==--