Re: Alternatives to -j parallel
Richard Beare <[email protected]> Tue, 6 Sep 2016 21:04:20 +1000
| Newsgroups | gmane.comp.compilers.distcc |
|---|---|
| Message-ID | <CA+V7QS8=Rphv8vtdP9E5MB1V_6Q88Wx0zxdLkbwWtVaCBN7vtQ@mail.gmail.com> |
--===============7410161918324144387== Content-Type: multipart/alternative; boundary=94eb2c0892c83310cc053bd4c298 --94eb2c0892c83310cc053bd4c298 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I'll have a look at that - perhaps ensuring LD is set correctly will be enough Thanks On Tue, Sep 6, 2016 at 8:56 PM, =C5=81ukasz Tasz <[email protected]> wrote: > I think I got your point, > > playing with MKFLAGS with cmake will be a tricky part and complex at the > end. > I would recommend to use distcc and limit --localslots=3D1 and assure tha= t > linking starts with distcc at the begining. > Make sure that your LD variable is prefixed with distcc. > > In my opinion it will be the easiest way. > > In case of pure make it is very easy, enough is to call $MAKE MKFLAGS=3D = and > -j then is 1, but it is not a case when talking about cmake buildsystem. > > regards > L. > > =C5=81ukasz Tasz > RTKW > > 2016-09-06 11:47 GMT+02:00 Richard Beare <[email protected]>: > >> Thanks, >> Here's another way of describing my problem. >> >> I have a configuration that allows me to substitute distcc for cc/cxx. >> This works fine, and I can distribute compilation across the 4 nodes >> available with circleCI, when I supply a -j N option to make. >> >> However, it appears that the linking (ld) phase is carried out using a >> separate mechanism - i.e. not from distcc. Thus the requirement of only = one >> job on the localhost is not respected by the link phase because distcc >> isn't executing the linker. In the circleCI system this leads to problem= s >> with memory. >> >> I'd therefore like to be able to tell distcc to be parallel without >> telling the rest of the makefile also being parallel. Obviously this wou= ld >> require that there is some other mechanism (besides -j) - perhaps an >> environment variable - that triggers parallel behaviour in distcc. >> >> In short - the cc part of distcc works fine. However, in this build >> system, ld is not being run from distcc. I don't want multiple ld jobs >> running, so I'd like to be able to run make -j1 but instruct distcc to u= se >> a higher level of j. >> >> >> >> >> On Tue, Sep 6, 2016 at 6:42 PM, =C5=81ukasz Tasz <[email protected]> wrote: >> >>> Could you please describe more what you would like to achieve? >>> -j does net decide it compilation will be done with distcc or not, >>> -j must take into consideration how many distcc agent you have. >>> >>> you have two dimensions, one is if distcc should be used, and second is >>> about parallelism. >>> first dimension is controlled in build system, second can be controlled >>> by build system, a but also may be limited by distcc. >>> >>> localslots and localslots_cpp controls parallelism from distcc point of >>> view. >>> >>> regards >>> Lukasz >>> >>> =C5=81ukasz Tasz >>> RTKW >>> >>> 2016-09-06 4:11 GMT+02:00 Richard Beare <[email protected]>: >>> >>>> Hi, >>>> I'm having a few problems using distcc for a large project on the >>>> circleCI infrastructure, which has 4GB limits for the compilation host= s. >>>> >>>> Occasionally I see this limit being exceeded when 2 link jobs run on >>>> the main server at the same time. This happens despite the --localslot= s=3D1 >>>> and localhost/1 settings in the distcc hosts file. >>>> >>>> I suspect the problem is that the large and complex cmake project that >>>> is being built has its own link mechanism. So my question is whether t= here >>>> is an alternative way to pass the parallel option to distcc besides ma= ke >>>> -jN? This would allow the other compile steps that might not be under >>>> distcc control to run serially on the local host. >>>> >>>> Thanks >>>> >>>> __ >>>> distcc mailing list http://distcc.samba.org/ >>>> To unsubscribe or change options: >>>> https://lists.samba.org/mailman/listinfo/distcc >>>> >>> >>> >> > --94eb2c0892c83310cc053bd4c298 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">I'll have a look at that - perhaps ensuring LD is set = correctly will be enough<div><br></div><div>Thanks</div></div><div class=3D= "gmail_extra"><br><div class=3D"gmail_quote">On Tue, Sep 6, 2016 at 8:56 PM= , =C5=81ukasz Tasz <span dir=3D"ltr"><<a href=3D"mailto:[email protected]" = target=3D"_blank">[email protected]</a>></span> wrote:<br><blockquote class= =3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd= ing-left:1ex"><div dir=3D"ltr"><div><div><div><div><div><div><div><div>I th= ink I got your point, <br></div><br></div>playing with MKFLAGS with cmake w= ill be a tricky part and complex at the end.<br></div>I would recommend to = use distcc and limit --localslots=3D1 and assure that linking starts with d= istcc at the begining.<br></div>Make sure that your LD variable is prefixed= with distcc.<br><br></div>In my opinion it will be the easiest way.<br><br= ></div>In case of pure make it is very easy, enough is to call $MAKE MKFLAG= S=3D and -j then is 1, but it is not a case when talking about cmake builds= ystem.<br><br></div>regards<br></div>L.<span class=3D"HOEnZb"><font color= =3D"#888888"><br></font></span></div><div class=3D"gmail_extra"><span class= =3D"HOEnZb"><font color=3D"#888888"><br clear=3D"all"><div><div data-smartm= ail=3D"gmail_signature"><div dir=3D"ltr"><div>=C5=81ukasz Tasz<br></div><di= v>RTKW<br></div></div></div></div></font></span><div><div class=3D"h5"> <br><div class=3D"gmail_quote">2016-09-06 11:47 GMT+02:00 Richard Beare <sp= an dir=3D"ltr"><<a href=3D"mailto:[email protected]" target=3D"_bl= ank">[email protected]</a>></span>:<br><blockquote class=3D"gmail_= quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1= ex"><div dir=3D"ltr">Thanks,<div>Here's another way of describing my pr= oblem.</div><div><br></div><div>I have a configuration that allows me to su= bstitute distcc for cc/cxx. This works fine, and I can distribute compilati= on across the 4 nodes available with circleCI, when I supply a -j N option = to make.</div><div><br></div><div>However, it appears that the linking (ld)= phase is carried out using a separate mechanism - i.e. not from distcc. Th= us the requirement of only one job on the localhost is not respected by the= link phase because distcc isn't executing the linker. In the circleCI = system this leads to problems with memory.</div><div><br></div><div>I'd= therefore like to be able to tell distcc to be parallel without telling th= e rest of the makefile also being parallel. Obviously this would require th= at there is some other mechanism (besides -j) - perhaps an environment vari= able - that triggers parallel behaviour in distcc.</div><div><br></div><div= >In short - the cc part of distcc works fine. However, in this build system= , ld is not being run from distcc. I don't want multiple ld jobs runnin= g, so I'd like to be able to run make -j1 but instruct distcc to use a = higher level of j.</div><div><br></div><div><br></div><div><br></div></div>= <div><div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Tue,= Sep 6, 2016 at 6:42 PM, =C5=81ukasz Tasz <span dir=3D"ltr"><<a href=3D"= mailto:[email protected]" target=3D"_blank">[email protected]</a>></span> wrot= e:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-l= eft:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div><div><div><div><= div><div>Could you please describe more what you would like to achieve?<br>= </div>-j does net decide it compilation will be done with distcc or not, <b= r></div>-j must take into consideration how many distcc agent you have.<br>= <br></div>you have two dimensions, one is if distcc should be used, and sec= ond is about parallelism. <br></div><div>first dimension is controlled in b= uild system, second can be controlled by build system, a but also may be li= mited by distcc. <br></div><div><br></div>localslots and localslots_cpp con= trols parallelism from distcc point of view.<br><br></div>regards<br></div>= Lukasz<br></div><div class=3D"gmail_extra"><br clear=3D"all"><div><div data= -smartmail=3D"gmail_signature"><div dir=3D"ltr"><div>=C5=81ukasz Tasz<br></= div><div>RTKW<br></div></div></div></div> <br><div class=3D"gmail_quote"><div><div>2016-09-06 4:11 GMT+02:00 Richard = Beare <span dir=3D"ltr"><<a href=3D"mailto:[email protected]" targ= et=3D"_blank">[email protected]</a>></span>:<br></div></div><block= quote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc= solid;padding-left:1ex"><div><div><div dir=3D"ltr">Hi,<div>I'm having = a few problems using distcc for a large project on the circleCI infrastruct= ure, which has 4GB limits for the compilation hosts.</div><div><br></div><d= iv>Occasionally I see this limit being exceeded when 2 link jobs run on the= main server at the same time. This happens despite the --localslots=3D1 an= d localhost/1 settings in the distcc hosts file.</div><div><br></div><div>I= suspect the problem is that the large and complex cmake project that is be= ing built has its own link mechanism. So my question is whether there is an= alternative way to pass the parallel option to distcc besides make -jN? Th= is would allow the other compile steps that might not be under distcc contr= ol to run serially on the local host.</div><div><br></div><div>Thanks</div>= </div> <br></div></div>__<br> distcc mailing list=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a href=3D"htt= p://distcc.samba.org/" rel=3D"noreferrer" target=3D"_blank">http://distcc.s= amba.org/</a><br> To unsubscribe or change options:<br> <a href=3D"https://lists.samba.org/mailman/listinfo/distcc" rel=3D"noreferr= er" target=3D"_blank">https://lists.samba.org/mailma<wbr>n/listinfo/distcc<= /a><br></blockquote></div><br></div> </blockquote></div><br></div> </div></div></blockquote></div><br></div></div></div> </blockquote></div><br></div> --94eb2c0892c83310cc053bd4c298-- --===============7410161918324144387== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline __ distcc mailing list http://distcc.samba.org/ To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/distcc --===============7410161918324144387==--