Re: DISTCC_POTENTIAL_HOSTS, localhost, and slots
Victor Norman <[email protected]> Thu, 15 May 2014 08:36:14 -0400
| Newsgroups | gmane.comp.compilers.distcc |
|---|---|
| Message-ID | <CAC-+LH+=hPK-SE-3bFmeqgtYf3qLUgZfKOymJk0F5Vu3GW+4FA@mail.gmail.com> |
--===============4423919684695262347== Content-Type: multipart/alternative; boundary="047d7b343448f2397404f96f8ae9" --047d7b343448f2397404f96f8ae9 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Wow! A reference to DMUCS! I'm the author of that code, and yes, it hasn't been touched since 2006... :-( New job, new location, new priorities, etc... It was made pre-pump-mode, so I doubt it works with it... But, it might... Perhaps I should really open-source that code so anyone could contribute to it. On Thu, May 15, 2014 at 7:37 AM, Dave Abrahams <[email protected]> wrote: > Wow, Jeff, that looks pretty awesome. It hasn't been touched since 2006, > though; does it work with pump mode? > > > On Wed, May 14, 2014 at 8:00 AM, Jeff Domogala <[email protected]>= wrote: > >> Another suggestion if you don=E2=80=99t want to dig into the code=E2=80= =A6 I ran across >> DMUCS (http://dmucs.sourceforge.net) about 8 years ago when I was >> building up a farm of machines to be used by a couple hundred users. DM= UCS >> will manage the compute resources and is able to handle unavailability o= f >> resources. I did make a few of my own tweaks to the source code for >> monitoring and other glitches I have run into along the way. I=E2=80=99= m still >> using DMUCS as my frontend for distcc and am very happy. If there are a= ny >> questions about my setup fire away. >> >> >> >> --Jeff >> >> >> >> *From:* Fergus Henderson [mailto:[email protected]] >> *Sent:* Wednesday, May 14, 2014 8:47 AM >> *To:* Dave Abrahams >> *Cc:* [email protected] >> *Subject:* Re: [distcc] DISTCC_POTENTIAL_HOSTS, localhost, and slots >> >> >> >> On Wed, May 14, 2014 at 6:16 AM, Dave Abrahams <[email protected]> wrote= : >> >> DISTCC_POTENTIAL_HOSTS seems like a great idea; as I scale up the build >> farm I=E2=80=99d like to be able to deal with unavailability, but I=E2= =80=99m having >> trouble getting it to work right. >> >> >> >> First, after much fooling around I surmise two undocumented facts(?) >> >> >> >> 1. it adds ,lzo,cpp to all the hosts implicitly >> >> >> >> Yes, that's done in ExportDISTCC_HOSTS function in the pump script >> (source file pump.in). >> >> >> >> 2. it only accepts bare host names, not the usual >> @?hostname/<slots>(,<extension>)* syntax >> >> Is that correct? >> >> >> >> Pretty much, yes. It just passes the variable's value to lsdistcc, and >> lsdistcc accepts either bare host names, or a printf format string >> containing %d, e.g. "mydistcc%d" or "distcc%d.my.own.org.com", not *host >> specifications* as described the distcc man page. >> >> >> >> If so, it would be good if these things were documented explicitly >> somewhere. >> >> >> >> Agreed. I would be happy to accept a patch to add such documentation. >> >> >> >> Another possibility would be to change lsdistcc to handle host >> specifications. >> >> >> >> Also, if I want localhost to be one of the hosts, it looks like I need t= o >> run distccd there. That doesn=E2=80=99t seem like the most efficient th= ing to do, >> since normal local compilation will presumably have fewer layers to work >> through. Am I mistaken? >> >> >> >> Well, you only need to run distccd for lsdistcc's initial query (lsdistc= c >> doesn't have any special casing for localhost). >> >> But any actual compilations done will not go via the distccd on >> localhost; distcc itself does have special casing for localhost. >> >> >> >> I'd be happy to accept a patch to src/lsdistcc.c to make it special-case >> "localhost". >> >> >> >> Lastly, I=E2=80=99ve only been able to get it to run the default 6 (?) s= lots per >> host using DISTCC_POTENTIAL_HOSTS. My .distcc/hosts file looks like thi= s, >> and has been working pretty well: >> >> >> >> --randomize >> >> localhost/10 >> >> bruford/10,lzo,cpp >> >> fripp/10,lzo,cpp >> >> >> >> Is there any way to get an effect like that using DISTCC_POTENTIAL_HOSTS= , >> i.e. 10 slots per host, no local distccd? >> >> >> >> Not currently. I imagine that it would require only very minor changes >> to the ExportDISTCC_HOSTS function in the pump script (source file >> pump.in), though. >> >> >> >> Cheers, >> >> Fergus. >> >> >> >> -- >> >> Fergus Henderson <[email protected]> >> >> >> >> "Defend the user, exclude no one, and create magic." -- Eric Schmidt. >> >> >> > > > > -- > Dave Abrahams > > > __ > distcc mailing list http://distcc.samba.org/ > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/distcc > --=20 Prof. Victor Norman Computer Science Dept., Calvin College [email protected] / 616 526-7805 =E2=80=9CA designer knows he has achieved perfection not when there is noth= ing left to add, but when there is nothing left to take away.=E2=80=9D -- Antoine de= Saint Exup=C3=A9ry --047d7b343448f2397404f96f8ae9 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Wow! =C2=A0A reference to DMUCS!<div><br></div><div>I'= m the author of that code, and yes, it hasn't been touched since 2006..= . :-( =C2=A0New job, new location, new priorities, etc...</div><div><br></d= iv><div>It was made pre-pump-mode, so I doubt it works with it... =C2=A0But= , it might...</div> <div><br></div><div>Perhaps I should really open-source that code so anyone= could contribute to it.</div></div><div class=3D"gmail_extra"><br><br><div= class=3D"gmail_quote">On Thu, May 15, 2014 at 7:37 AM, Dave Abrahams <span= dir=3D"ltr"><<a href=3D"mailto:[email protected]" target=3D"_blank">dav= [email protected]</a>></span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"><div dir=3D"ltr">Wow, Jeff, that looks prett= y awesome. =C2=A0It hasn't been touched since 2006, though; does it wor= k with pump mode?</div> <div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On Wed, May 1= 4, 2014 at 8:00 AM, Jeff Domogala <span dir=3D"ltr"><<a href=3D"mailto:j= [email protected]" target=3D"_blank">[email protected]</a>>= </span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"><div lang=3D"EN-US" link=3D"blue" vlink=3D"p= urple"><div><p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-fam= ily:"Calibri","sans-serif";color:#1f497d">Another sugge= stion if you don=E2=80=99t want to dig into the code=E2=80=A6=C2=A0 I ran a= cross DMUCS (<a href=3D"http://dmucs.sourceforge.net" target=3D"_blank">htt= p://dmucs.sourceforge.net</a>) about 8 years ago when I was building up a f= arm of machines to be used by a couple hundred users.=C2=A0 DMUCS will mana= ge the compute resources and is able to handle unavailability of resources.= =C2=A0 I did make a few of my own tweaks to the source code for monitoring = and other glitches I have run into along the way.=C2=A0 I=E2=80=99m still u= sing DMUCS as my frontend for distcc and am very happy.=C2=A0 If there are = any questions about my setup fire away.<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri","sans-serif";color:#1f497d"><u></u>=C2=A0<u></u></spa= n></p><p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:&q= uot;Calibri","sans-serif";color:#1f497d">--Jeff<u></u><u></u= ></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri","sans-serif";color:#1f497d"><u></u>=C2=A0<u></u></spa= n></p><p class=3D"MsoNormal"><b><span style=3D"font-size:10.0pt;font-family= :"Tahoma","sans-serif"">From:</span></b><span style=3D"= font-size:10.0pt;font-family:"Tahoma","sans-serif""> Fe= rgus Henderson [mailto:<a href=3D"mailto:[email protected]" target=3D"_blan= k">[email protected]</a>] <br> <b>Sent:</b> Wednesday, May 14, 2014 8:47 AM<br><b>To:</b> Dave Abrahams<br= ><b>Cc:</b> <a href=3D"mailto:[email protected]" target=3D"_blank">dis= [email protected]</a><br><b>Subject:</b> Re: [distcc] DISTCC_POTENTIAL_HO= STS, localhost, and slots<u></u><u></u></span></p> <div><div><p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p><div><div><div><p = class=3D"MsoNormal">On Wed, May 14, 2014 at 6:16 AM, Dave Abrahams <<a h= ref=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a>>= ; wrote:<u></u><u></u></p> <div><p class=3D"MsoNormal">DISTCC_POTENTIAL_HOSTS seems like a great idea;= as I scale up the build farm I=E2=80=99d like to be able to deal with unav= ailability, but I=E2=80=99m having trouble getting it to work right.<u></u>= <u></u></p><div> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p></div><div><p class=3D"MsoNo= rmal">First, after much fooling around I surmise two undocumented facts(?)<= u></u><u></u></p></div><div><p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>= </div><div> <p class=3D"MsoNormal"> 1. it adds ,lzo,cpp to all the hosts implicitly<br clear=3D"all"><u></u><u>= </u></p></div></div><div><p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p></d= iv><div><p class=3D"MsoNormal">Yes, that's done in=C2=A0ExportDISTCC_HO= STS function in the pump script (source file <a href=3D"http://pump.in" tar= get=3D"_blank">pump.in</a>).<u></u><u></u></p> </div><div><p class=3D"MsoNormal">=C2=A0<u></u><u></u></p></div><blockquote= style=3D"border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6= .0pt;margin-left:4.8pt;margin-right:0in"><div><div><div><p class=3D"MsoNorm= al">2. it only accepts bare host names, not the usual @?hostname/<slots&= gt;(,<extension>)* syntax=C2=A0<u></u><u></u></p> </div></div></div></blockquote><div><blockquote style=3D"border:none;border= -left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margi= n-right:0in"><div><p class=3D"MsoNormal">Is that correct?<u></u><u></u></p> </div></blockquote><div><p class=3D"MsoNormal">=C2=A0<u></u><u></u></p></di= v></div><div><p class=3D"MsoNormal">Pretty much, yes. =C2=A0It just passes = the variable's value to lsdistcc, and lsdistcc accepts either bare host= names, or a printf format string containing %d, e.g. "mydistcc%d"= ; or "distcc%<a href=3D"http://d.my.own.org.com" target=3D"_blank">d.m= y.own.org.com</a>", not <i>host specifications</i> as described the di= stcc man page.<u></u><u></u></p> </div><div><p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p></div><blockquote= style=3D"border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6= .0pt;margin-left:4.8pt;margin-right:0in"><div><div><div><p class=3D"MsoNorm= al">If so, it would be good if these things were documented explicitly some= where.<u></u><u></u></p> </div></div></div></blockquote><div><p class=3D"MsoNormal"><u></u>=C2=A0<u>= </u></p></div><div><p class=3D"MsoNormal">Agreed. =C2=A0I would be happy to= accept a patch to add such documentation.<u></u><u></u></p></div><div><p c= lass=3D"MsoNormal"> <u></u>=C2=A0<u></u></p></div><div><p class=3D"MsoNormal">Another possibili= ty would be to change lsdistcc to handle host specifications.<u></u><u></u>= </p></div><div><p class=3D"MsoNormal">=C2=A0<u></u><u></u></p></div><blockq= uote style=3D"border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0= in 6.0pt;margin-left:4.8pt;margin-right:0in"> <div><div><div><p class=3D"MsoNormal">Also, if I want localhost to be one o= f the hosts, it looks like I need to run distccd there. =C2=A0That doesn=E2= =80=99t seem like the most efficient thing to do, since normal local compil= ation will presumably have fewer layers to work through. =C2=A0Am I mistake= n?<u></u><u></u></p> </div></div></div></blockquote><div><p class=3D"MsoNormal"><u></u>=C2=A0<u>= </u></p></div><div><p class=3D"MsoNormal">Well, you only need to run distcc= d for lsdistcc's initial query (lsdistcc doesn't have any special c= asing for localhost).<u></u><u></u></p> </div><div><p class=3D"MsoNormal">But any actual compilations done will not= go via the distccd on localhost; distcc itself does have special casing fo= r localhost.=C2=A0<u></u><u></u></p></div><div><p class=3D"MsoNormal"><u></= u>=C2=A0<u></u></p> </div><div><p class=3D"MsoNormal">I'd be happy to accept a patch to src= /lsdistcc.c to make it special-case "localhost".<u></u><u></u></p= ></div><div><p class=3D"MsoNormal">=C2=A0<u></u><u></u></p></div><blockquot= e style=3D"border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in = 6.0pt;margin-left:4.8pt;margin-right:0in"> <div><div><div><p class=3D"MsoNormal">Lastly, I=E2=80=99ve only been able t= o get it to run the default 6 (?) slots per host using DISTCC_POTENTIAL_HOS= TS. =C2=A0My .distcc/hosts file looks like this, and has been working prett= y well:<u></u><u></u></p> </div><div><p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p></div></div><bloc= kquote style=3D"margin-left:30.0pt;margin-right:0in"><div><div><p class=3D"= MsoNormal">--randomize<u></u><u></u></p></div></div><div><div><p class=3D"M= soNormal"> localhost/10<u></u><u></u></p></div></div><div><div><p class=3D"MsoNormal">= bruford/10,lzo,cpp<u></u><u></u></p></div></div><div><div><p class=3D"MsoNo= rmal">fripp/10,lzo,cpp<u></u><u></u></p></div></div></blockquote><div><div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p></div><div><p class=3D"MsoNo= rmal">Is there any way to get an effect like that using DISTCC_POTENTIAL_HO= STS, i.e. 10 slots per host, no local distccd? =C2=A0<u></u><u></u></p></di= v></div></div> </blockquote><div><p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p></div><div= ><p class=3D"MsoNormal">Not currently. =C2=A0I imagine that it would requir= e only very minor changes to the ExportDISTCC_HOSTS function in the pump sc= ript (source file <a href=3D"http://pump.in" target=3D"_blank">pump.in</a>)= , though.<u></u><u></u></p> </div><div><p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p></div></div><p cl= ass=3D"MsoNormal">Cheers,<u></u><u></u></p></div><div><p class=3D"MsoNormal= ">=C2=A0 =C2=A0Fergus.<br clear=3D"all"><u></u><u></u></p><div><p class=3D"= MsoNormal"><u></u>=C2=A0<span class=3D"HOEnZb"><font color=3D"#888888"><u><= /u></font></span></p> <span class=3D"HOEnZb"><font color=3D"#888888"> </font></span></div><span class=3D"HOEnZb"><font color=3D"#888888"><p class= =3D"MsoNormal">-- <u></u><u></u></p><div><p class=3D"MsoNormal">Fergus Hend= erson <<a href=3D"mailto:[email protected]" target=3D"_blank">fergus@goo= gle.com</a>><u></u><u></u></p> </div><div><p class=3D"MsoNormal"> <u></u>=C2=A0<u></u></p></div><div><div><p class=3D"MsoNormal"><span style= =3D"font-size:10.0pt;font-family:"Arial","sans-serif"">= "Defend the user, exclude no one, and create magic." -- Eric Schm= idt.</span><u></u><u></u></p> </div></div><div><p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p></div></fon= t></span></div></div></div></div></div></div></blockquote></div><span class= =3D"HOEnZb"><font color=3D"#888888"><br><br clear=3D"all"><div><br></div>--= <br><div dir=3D"ltr"> Dave Abrahams<br><br></div> </font></span></div> <br>__<br> distcc mailing list =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<a href=3D"htt= p://distcc.samba.org/" target=3D"_blank">http://distcc.samba.org/</a><br> To unsubscribe or change options:<br> <a href=3D"https://lists.samba.org/mailman/listinfo/distcc" target=3D"_blan= k">https://lists.samba.org/mailman/listinfo/distcc</a><br></blockquote></di= v><br><br clear=3D"all"><div><br></div>-- <br>Prof. Victor Norman<div>Compu= ter Science Dept.,=C2=A0Calvin College</div> <div><a href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</= a> /=C2=A0616 526-7805</div><div><br></div><div><br></div><div>=E2=80=9CA d= esigner knows he has achieved perfection not when there is nothing left to = add, but when there is nothing left to take away.=E2=80=9D -- Antoine de Sa= int Exup=C3=A9ry</div> </div> --047d7b343448f2397404f96f8ae9-- --===============4423919684695262347== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline __ distcc mailing list http://distcc.samba.org/ To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/distcc --===============4423919684695262347==--