Re: small redesign...
Martin Pool <[email protected]> Fri, 24 Oct 2014 00:42:09 +0000
| Newsgroups | gmane.comp.compilers.distcc |
|---|---|
| Message-ID | <CAA9uavD5rWWB5yCrVZyRBdDwnFmg2g3kC3nBdk_eweoG16pgGQ@mail.gmail.com> |
--===============6589078591240809431== Content-Type: multipart/alternative; boundary=001a113970a08f6a41050620735c --001a113970a08f6a41050620735c Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable It seems like if there's nowhere to execute the job, we want the client program to just pause, before using too many resources, until it gets unqueued by a server ready to do the job. (Or, by a local slot being available.) On Thu Oct 16 2014 at 2:43:35 AM =C5=81ukasz Tasz <[email protected]> wrote: > Hi Martin, > > Lets assume that you can trigger more compilation tasks executors then yo= u > have. > In this scenario you are facing situation that cluster is saturated. > When such a compilation will be triggered by two developers, or two CI > (e.g jenkins) jobs, then cluster is saturated twice... > > Default behaviour is to lock locally slot, and try to connect three > times, if not, fallback, if fallback is disabled CI got failed build > (fallback is not the case, since local machine cannot handle -j > $(distcc -j)). > > consider scenario, I have 1000 objects, 500 executors, > - clean build on one machine takes > 1000 * 20 sec (one obj) =3D 20000 / 16 processors =3D 1000 sec, > - on cluster (1000/500) * 20 sec =3D 40 sec > > Saturating cluster was impossible without pump mode, but now with pump > mode after "warm up" effect, pump can dispatch many tasks, and I faced > situation that saturated cluster destroys almost every compilation. > > My expectation is that cluster wont reject my connect, or reject will > be handled, either by client, either by server. > > by server: > - accept every connetion, > - fork child if not accepted by child, > - in case of pump prepare local dir structure, receive headers > - --critical section starts here-- multi value semaphore with value > maxchild > - execute job > - release semaphore > > > Also what you suggested may be even better solution, since client will > pick first avaliable executor instead of entering queue, so distcc > could make connection already in function dcc_lock_one() > > I already tried to set DISTCC_DIR on a common nfs share, but in case > you are triggering so many jobs, this started to be bottle neck... I > won't tell about locking on nfs, and also scenario that somebody will > make a lock on nfs and machine will got crash - will not work by > design :) > > I know that scenario is not happening very often, and it has more or > less picks characteristic, but we should be happy that distcc cluster > is saturated and this case should be handled. > > hope it's more clear now! > br > LT > > > > > > > > > > =C5=81ukasz Tasz > > > 2014-10-16 1:39 GMT+02:00 Martin Pool <[email protected]>: > > Can you try to explain more clearly what difference in queueing behavio= r > you > > expect from this change? > > > > I think probably the main change that's needed is for the client to ask > all > > masters if they have space, to avoid needing to effectively poll by > > retrying, or getting stuck waiting for a particular server. > > > > On Wed, Oct 15, 2014 at 12:53 PM, =C5=81ukasz Tasz <[email protected]> wro= te: > >> > >> Hi Guys, > >> > >> please correct me if I'm wrong, > >> - currently distcc tries to connect server 3 times, with small delay, > >> - server forks x childs and all of them are trying to accept incoming > >> connection. > >> If server runs out of childs (all of them are busy), client will > >> fallback, and within next 60 sec will not try this machine. > >> > >> What do you think about redesigning distcc in a way that master server > >> will always accept inconing connection, fork a child, but in a same > >> time only x of them will be able to enter compilation > >> task(dcc_spawn_child)? (mayby preforking still could be used?) > >> > >> This may create kind of queue, client always can decide by his own, if > >> can wait some time, or maximum is DISTCC_IO_TIMEOUT, but still it's > >> faster to wait, since probably on a cluster side it's just a pick of > >> saturation then making falback to local machine. > >> > >> currently I'm facing situation that many jobs are making fallback, and > >> localmachine is being killed by make's -j calculated for distccd... > >> > >> other trick maybe to pick different machine, if current is busy, but > >> this may be much more complex in my opinion. > >> > >> what do you think? > >> regards > >> =C5=81ukasz Tasz > >> __ > >> distcc mailing list http://distcc.samba.org/ > >> To unsubscribe or change options: > >> https://lists.samba.org/mailman/listinfo/distcc > > > > > > > > > > -- > > Martin > --001a113970a08f6a41050620735c Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable It seems like if there's nowhere to execute the job, we want the client= program to just pause, before using too many resources, until it gets unqu= eued by a server ready to do the job. (Or, by a local slot being available.= )<br><br><div class=3D"gmail_quote">On Thu Oct 16 2014 at 2:43:35 AM =C5=81= ukasz Tasz <<a href=3D"mailto:[email protected]">[email protected]</a>> wro= te:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-= left:1px #ccc solid;padding-left:1ex">Hi Martin,<br> <br> Lets assume that you can trigger more compilation tasks executors then you = have.<br> In this scenario you are facing situation that cluster is saturated.<br> When such a compilation will be triggered by two developers, or two CI<br> (e.g jenkins) jobs, then cluster is saturated twice...<br> <br> Default behaviour is to lock locally slot, and try to connect three<br> times, if not, fallback, if fallback is disabled CI got failed build<br> (fallback is not the case, since local machine cannot handle -j<br> $(distcc -j)).<br> <br> consider scenario, I have 1000 objects, 500 executors,<br> - clean build on one machine takes<br> =C2=A0 1000 * 20 sec (one obj) =3D 20000 / 16 processors =3D 1000 sec,<br> - on cluster (1000/500) * 20 sec =3D 40 sec<br> <br> Saturating cluster was impossible without pump mode, but now with pump<br> mode after "warm up" effect, pump can dispatch many tasks, and I = faced<br> situation that saturated cluster destroys almost=C2=A0 every compilation.<b= r> <br> My expectation is that cluster wont reject my connect, or reject will<br> be handled, either by client, either by server.<br> <br> by server:<br> - accept every connetion,<br> - fork child if not accepted by child,<br> - in case of pump prepare local dir structure, receive headers<br> - --critical section starts here-- multi value semaphore with value maxchil= d<br> =C2=A0 - execute job<br> - release semaphore<br> <br> <br> Also what you suggested may be even better solution, since client will<br> pick first avaliable executor instead of entering queue, so distcc<br> could make connection already in function dcc_lock_one()<br> <br> I already tried to set DISTCC_DIR on a common nfs share, but in case<br> you are triggering so many jobs, this started to be bottle neck... I<br> won't tell about locking on nfs, and also scenario that somebody will<b= r> make a lock on nfs and machine will got crash - will not work by<br> design :)<br> <br> I know that scenario is not happening very often, and it has more or<br> less picks characteristic, but we should be happy that distcc cluster<br> is saturated and this case should be handled.<br> <br> hope it's more clear now!<br> br<br> LT<br> <br> <br> <br> <br> <br> <br> <br> <br> <br> =C5=81ukasz Tasz<br> <br> <br> 2014-10-16 1:39 GMT+02:00 Martin Pool <<a href=3D"mailto:mbp@sourcefrog.= net" target=3D"_blank">[email protected]</a>>:<br> > Can you try to explain more clearly what difference in queueing behavi= or you<br> > expect from this change?<br> ><br> > I think probably the main change that's needed is for the client t= o ask all<br> > masters if they have space, to avoid needing to effectively poll by<br= > > retrying, or getting stuck waiting for a particular server.<br> ><br> > On Wed, Oct 15, 2014 at 12:53 PM, =C5=81ukasz Tasz <<a href=3D"mail= to:[email protected]" target=3D"_blank">[email protected]</a>> wrote:<br> >><br> >> Hi Guys,<br> >><br> >> please correct me if I'm wrong,<br> >> - currently distcc tries to connect server 3 times, with small del= ay,<br> >> - server forks x childs and all of them are trying to accept incom= ing<br> >> connection.<br> >> If server runs out of childs (all of them are busy), client will<b= r> >> fallback, and within next 60 sec will not try this machine.<br> >><br> >> What do you think about redesigning distcc in a way that master se= rver<br> >> will always accept inconing connection, fork a child, but in a sam= e<br> >> time only x of them will be able to enter compilation<br> >> task(dcc_spawn_child)? (mayby preforking still could be used?)<br> >><br> >> This may create kind of queue, client always can decide by his own= , if<br> >> can wait some=C2=A0 time, or maximum is DISTCC_IO_TIMEOUT, but sti= ll it's<br> >> faster to wait, since probably on a cluster side it's just a p= ick of<br> >> saturation then making falback to local machine.<br> >><br> >> currently I'm facing situation that many jobs are making fallb= ack, and<br> >> localmachine is being killed by make's -j calculated for distc= cd...<br> >><br> >> other trick maybe to pick different machine, if current is busy, b= ut<br> >> this may be much more complex in my opinion.<br> >><br> >> what do you think?<br> >> regards<br> >> =C5=81ukasz Tasz<br> >> __<br> >> distcc mailing list=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a hr= ef=3D"http://distcc.samba.org/" target=3D"_blank">http://distcc.samba.org/<= /a><br> >> To unsubscribe or change options:<br> >> <a href=3D"https://lists.samba.org/mailman/listinfo/distcc" target= =3D"_blank">https://lists.samba.org/<u></u>mailman/listinfo/distcc</a><br> ><br> ><br> ><br> ><br> > --<br> > Martin<br> </blockquote></div> --001a113970a08f6a41050620735c-- --===============6589078591240809431== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline __ distcc mailing list http://distcc.samba.org/ To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/distcc --===============6589078591240809431==--