Re: small redesign...
Fergus Henderson <[email protected]> Sat, 1 Nov 2014 08:06:12 +0000
| Newsgroups | gmane.comp.compilers.distcc |
|---|---|
| Message-ID | <CAPXkjd9qcBPCy12eoQdFNP4u64QP=VsXg-008cNhadn72_J-PA@mail.gmail.com> |
--===============4078194226169139121== Content-Type: multipart/alternative; boundary=bcaec517198344648b0506c796cf --bcaec517198344648b0506c796cf Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Well, perhaps it would be a good idea to add a distccd flag or environment variable to control the queue length rather than hard-coding 10 or 256? On 31 Oct 2014 11:37, "=C5=81ukasz Tasz" <[email protected]> wrote: > Hi Guys, > > I'm very very happy, reasons of my failures are identified. > issue is in: > --- src/srvnet.c (wersja 177) > +++ src/srvnet.c (kopia robocza) > @@ -99,7 +99,7 @@ > rs_log_info("listening on %s", sa_buf ? sa_buf : "UNKNOWN"); > free(sa_buf); > > - if (listen(fd, 10)) { > + if (listen(fd, 256)) { > rs_log_error("listen failed: %s", strerror(errno)); > close(fd); > return EXIT_BIND_FAILED; > Index: src/io.c > > queue for new connetcion was minited to 10, that's why in case that > cluster is overloaded, many connection are reseted. > aim is to even wait 5 min for cluster availability, then compile localy. > > @Jarek, thanks for support! > > let's discuss if we should fix it or not. > > regards > Lukasz > > > =C5=81ukasz Tasz > > > 2014-10-24 10:27 GMT+02:00 =C5=81ukasz Tasz <[email protected]>: > > Hi Martin > > > > What I have noticed. > > Client tries to connect distccd 3 times with 500ms delays in between. > > Linux kernel by default accept 128 connection. > > If client creates connection, even if no executors are avaliable, > > connection is accepted and queued by kernel running distccd. > > This leads to situation that client thinks that distccd is reserved, > > but in fact connection still waits to be accepted by distccd server. > > I suspect that then client starts communication too fast, distcc wont > > receive DIST token, and both sides waits, communication is broken, and > > then timeouts are applied for client default is applied, for server > > there is no defaults. > > > > fail scenarion is: > > one distccd, and two distcc users, both of them will try to compile > > with DISTCC_HOSTS=3Ddistccd/1,cpp,lzo, both users have lot of big > > objects, cluster is overloaded with ratio 2. > > This still should be OK, that third, and forth user will join cluster. > > > > Easy reproducer is to set one distcc, and set distcc_hosts=3Ddistccd/20= , > > this is broken configuration, but simulates overload by 20 - 20 > > developers uses cluster in a same time. > > Please remember that those are exceptional situation, but developer > > can start compilation with -j 1000 from his laptop, and cluster will > > timeout, then receiving 1000 jobs on a laptop will end with memmory > > killer :D > > Those are exceptional situation, and somehow cluster should handle that= . > > > > In the attachement, next to some pump changes, you can find change > > which is moving making connection to very beginning, when distcc is > > picking host, also remote connection is made. if this will fail, discc > > follow default behaviour, goes sleep for one sec, and will pick host > > again. But this requires additional administration change on distccd > > machine: > > iptables -I INPUT -p tcp --dport 3632 -m connlimit --connlimit-above > > <NUMBER OF DISTCCD> --connlimit-mask 0 -j REJECT --reject-with > > tcp-reset > > which accept only number of connection which equals to number of > executors. > > > > So far so good! > > remark, patch is done on top of arankine_distcc_issue16-r335, since > > his pump changes are making pump mode working on my environment. > > But distccd allocation I tested also on latest official distcc release. > > > > let me know what you think! > > > > with best regards > > Lukasz > > > > > > > > =C5=81ukasz Tasz > > > > > > 2014-10-24 2:42 GMT+02:00 Martin Pool <[email protected]>: > >> It seems like if there's nowhere to execute the job, we want the clien= t > >> program to just pause, before using too many resources, until it gets > >> unqueued by a server ready to do the job. (Or, by a local slot being > >> available.) > >> > >> > >> On Thu Oct 16 2014 at 2:43:35 AM =C5=81ukasz Tasz <[email protected]> wro= te: > >>> > >>> Hi Martin, > >>> > >>> Lets assume that you can trigger more compilation tasks executors the= n > you > >>> have. > >>> In this scenario you are facing situation that cluster is saturated. > >>> When such a compilation will be triggered by two developers, or two C= I > >>> (e.g jenkins) jobs, then cluster is saturated twice... > >>> > >>> Default behaviour is to lock locally slot, and try to connect three > >>> times, if not, fallback, if fallback is disabled CI got failed build > >>> (fallback is not the case, since local machine cannot handle -j > >>> $(distcc -j)). > >>> > >>> consider scenario, I have 1000 objects, 500 executors, > >>> - clean build on one machine takes > >>> 1000 * 20 sec (one obj) =3D 20000 / 16 processors =3D 1000 sec, > >>> - on cluster (1000/500) * 20 sec =3D 40 sec > >>> > >>> Saturating cluster was impossible without pump mode, but now with pum= p > >>> mode after "warm up" effect, pump can dispatch many tasks, and I face= d > >>> situation that saturated cluster destroys almost every compilation. > >>> > >>> My expectation is that cluster wont reject my connect, or reject will > >>> be handled, either by client, either by server. > >>> > >>> by server: > >>> - accept every connetion, > >>> - fork child if not accepted by child, > >>> - in case of pump prepare local dir structure, receive headers > >>> - --critical section starts here-- multi value semaphore with value > >>> maxchild > >>> - execute job > >>> - release semaphore > >>> > >>> > >>> Also what you suggested may be even better solution, since client wil= l > >>> pick first avaliable executor instead of entering queue, so distcc > >>> could make connection already in function dcc_lock_one() > >>> > >>> I already tried to set DISTCC_DIR on a common nfs share, but in case > >>> you are triggering so many jobs, this started to be bottle neck... I > >>> won't tell about locking on nfs, and also scenario that somebody will > >>> make a lock on nfs and machine will got crash - will not work by > >>> design :) > >>> > >>> I know that scenario is not happening very often, and it has more or > >>> less picks characteristic, but we should be happy that distcc cluster > >>> is saturated and this case should be handled. > >>> > >>> hope it's more clear now! > >>> br > >>> LT > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> =C5=81ukasz Tasz > >>> > >>> > >>> 2014-10-16 1:39 GMT+02:00 Martin Pool <[email protected]>: > >>> > Can you try to explain more clearly what difference in queueing > behavior > >>> > you > >>> > expect from this change? > >>> > > >>> > I think probably the main change that's needed is for the client to > ask > >>> > all > >>> > masters if they have space, to avoid needing to effectively poll by > >>> > retrying, or getting stuck waiting for a particular server. > >>> > > >>> > On Wed, Oct 15, 2014 at 12:53 PM, =C5=81ukasz Tasz <[email protected]> > wrote: > >>> >> > >>> >> Hi Guys, > >>> >> > >>> >> please correct me if I'm wrong, > >>> >> - currently distcc tries to connect server 3 times, with small > delay, > >>> >> - server forks x childs and all of them are trying to accept > incoming > >>> >> connection. > >>> >> If server runs out of childs (all of them are busy), client will > >>> >> fallback, and within next 60 sec will not try this machine. > >>> >> > >>> >> What do you think about redesigning distcc in a way that master > server > >>> >> will always accept inconing connection, fork a child, but in a sam= e > >>> >> time only x of them will be able to enter compilation > >>> >> task(dcc_spawn_child)? (mayby preforking still could be used?) > >>> >> > >>> >> This may create kind of queue, client always can decide by his own= , > if > >>> >> can wait some time, or maximum is DISTCC_IO_TIMEOUT, but still it= 's > >>> >> faster to wait, since probably on a cluster side it's just a pick = of > >>> >> saturation then making falback to local machine. > >>> >> > >>> >> currently I'm facing situation that many jobs are making fallback, > and > >>> >> localmachine is being killed by make's -j calculated for distccd..= . > >>> >> > >>> >> other trick maybe to pick different machine, if current is busy, b= ut > >>> >> this may be much more complex in my opinion. > >>> >> > >>> >> what do you think? > >>> >> regards > >>> >> =C5=81ukasz Tasz > >>> >> __ > >>> >> distcc mailing list http://distcc.samba.org/ > >>> >> To unsubscribe or change options: > >>> >> https://lists.samba.org/mailman/listinfo/distcc > >>> > > >>> > > >>> > > >>> > > >>> > -- > >>> > Martin > __ > distcc mailing list http://distcc.samba.org/ > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/distcc --bcaec517198344648b0506c796cf Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <p dir=3D"ltr">Well, perhaps it would be a good idea to add a distccd flag = or environment variable to control the queue length rather than hard-coding= 10 or 256?</p> <div class=3D"gmail_quote">On 31 Oct 2014 11:37, "=C5=81ukasz Tasz&quo= t; <<a href=3D"mailto:[email protected]">[email protected]</a>> wrote:<br t= ype=3D"attribution"><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0= .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Guys,<br> <br> I'm very very happy, reasons of my failures are identified.<br> issue is in:<br> --- src/srvnet.c=C2=A0 =C2=A0 =C2=A0 =C2=A0 (wersja 177)<br> +++ src/srvnet.c=C2=A0 =C2=A0 =C2=A0 =C2=A0 (kopia robocza)<br> @@ -99,7 +99,7 @@<br> =C2=A0 =C2=A0 =C2=A0rs_log_info("listening on %s", sa_buf ? sa_bu= f : "UNKNOWN");<br> =C2=A0 =C2=A0 =C2=A0free(sa_buf);<br> <br> -=C2=A0 =C2=A0 if (listen(fd, 10)) {<br> +=C2=A0 =C2=A0 if (listen(fd, 256)) {<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rs_log_error("listen failed: %s"= ;, strerror(errno));<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0close(fd);<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0return EXIT_BIND_FAILED;<br> Index: src/io.c<br> <br> queue for new connetcion was minited to 10, that's why in case that<br> cluster is overloaded, many connection are reseted.<br> aim is to even wait 5 min for cluster availability, then compile localy.<br= > <br> @Jarek, thanks for support!<br> <br> let's discuss if we should fix it or not.<br> <br> regards<br> Lukasz<br> <br> <br> =C5=81ukasz Tasz<br> <br> <br> 2014-10-24 10:27 GMT+02:00 =C5=81ukasz Tasz <<a href=3D"mailto:lukasz@ta= sz.eu">[email protected]</a>>:<br> > Hi Martin<br> ><br> > What I have noticed.<br> > Client tries to connect distccd 3 times with 500ms delays in between.<= br> > Linux kernel by default accept 128 connection.<br> > If client creates connection, even if no executors are avaliable,<br> > connection is accepted and queued by kernel running distccd.<br> > This leads to situation that client thinks that distccd is reserved,<b= r> > but in fact connection still waits to be accepted by distccd server.<b= r> > I suspect that then client starts communication too fast, distcc wont<= br> > receive DIST token, and both sides waits, communication is broken, and= <br> > then timeouts are applied for client default is applied, for server<br= > > there is no defaults.<br> ><br> > fail scenarion is:<br> > one distccd, and two distcc users, both of them will try to compile<br= > > with DISTCC_HOSTS=3Ddistccd/1,cpp,lzo, both users have lot of big<br> > objects, cluster is overloaded with ratio 2.<br> > This still should be OK, that third, and forth user will join cluster.= <br> ><br> > Easy reproducer is to set one distcc, and set distcc_hosts=3Ddistccd/2= 0,<br> > this is broken configuration, but simulates overload by 20 - 20<br> > developers uses cluster in a same time.<br> > Please remember that those are exceptional situation, but developer<br= > > can start compilation with -j 1000 from his laptop, and cluster will<b= r> > timeout, then receiving 1000 jobs on a laptop will end with memmory<br= > > killer :D<br> > Those are exceptional situation, and somehow cluster should handle tha= t.<br> ><br> > In the attachement, next to some pump changes, you can find change<br> > which is moving making connection to very beginning, when distcc is<br= > > picking host, also remote connection is made. if this will fail, discc= <br> > follow default behaviour, goes sleep for one sec, and will pick host<b= r> > again. But this requires additional administration change on distccd<b= r> > machine:<br> > iptables -I INPUT -p tcp --dport 3632 -m connlimit --connlimit-above<b= r> > <NUMBER OF DISTCCD> --connlimit-mask 0 -j REJECT --reject-with<b= r> > tcp-reset<br> > which accept only number of connection which equals to number of execu= tors.<br> ><br> > So far so good!<br> > remark, patch is done on top of arankine_distcc_issue16-r335, since<br= > > his pump changes are making pump mode working on my environment.<br> > But distccd allocation I tested also on latest official distcc release= .<br> ><br> > let me know what you think!<br> ><br> > with best regards<br> > Lukasz<br> ><br> ><br> ><br> > =C5=81ukasz Tasz<br> ><br> ><br> > 2014-10-24 2:42 GMT+02:00 Martin Pool <<a href=3D"mailto:mbp@source= frog.net">[email protected]</a>>:<br> >> It seems like if there's nowhere to execute the job, we want t= he client<br> >> program to just pause, before using too many resources, until it g= ets<br> >> unqueued by a server ready to do the job. (Or, by a local slot bei= ng<br> >> available.)<br> >><br> >><br> >> On Thu Oct 16 2014 at 2:43:35 AM =C5=81ukasz Tasz <<a href=3D"m= ailto:[email protected]">[email protected]</a>> wrote:<br> >>><br> >>> Hi Martin,<br> >>><br> >>> Lets assume that you can trigger more compilation tasks execut= ors then you<br> >>> have.<br> >>> In this scenario you are facing situation that cluster is satu= rated.<br> >>> When such a compilation will be triggered by two developers, o= r two CI<br> >>> (e.g jenkins) jobs, then cluster is saturated twice...<br> >>><br> >>> Default behaviour is to lock locally slot, and try to connect = three<br> >>> times, if not, fallback, if fallback is disabled CI got failed= build<br> >>> (fallback is not the case, since local machine cannot handle -= j<br> >>> $(distcc -j)).<br> >>><br> >>> consider scenario, I have 1000 objects, 500 executors,<br> >>> - clean build on one machine takes<br> >>>=C2=A0 =C2=A01000 * 20 sec (one obj) =3D 20000 / 16 processors = =3D 1000 sec,<br> >>> - on cluster (1000/500) * 20 sec =3D 40 sec<br> >>><br> >>> Saturating cluster was impossible without pump mode, but now w= ith pump<br> >>> mode after "warm up" effect, pump can dispatch many = tasks, and I faced<br> >>> situation that saturated cluster destroys almost=C2=A0 every c= ompilation.<br> >>><br> >>> My expectation is that cluster wont reject my connect, or reje= ct will<br> >>> be handled, either by client, either by server.<br> >>><br> >>> by server:<br> >>> - accept every connetion,<br> >>> - fork child if not accepted by child,<br> >>> - in case of pump prepare local dir structure, receive headers= <br> >>> - --critical section starts here-- multi value semaphore with = value<br> >>> maxchild<br> >>>=C2=A0 =C2=A0- execute job<br> >>> - release semaphore<br> >>><br> >>><br> >>> Also what you suggested may be even better solution, since cli= ent will<br> >>> pick first avaliable executor instead of entering queue, so di= stcc<br> >>> could make connection already in function dcc_lock_one()<br> >>><br> >>> I already tried to set DISTCC_DIR on a common nfs share, but i= n case<br> >>> you are triggering so many jobs, this started to be bottle nec= k... I<br> >>> won't tell about locking on nfs, and also scenario that so= mebody will<br> >>> make a lock on nfs and machine will got crash - will not work = by<br> >>> design :)<br> >>><br> >>> I know that scenario is not happening very often, and it has m= ore or<br> >>> less picks characteristic, but we should be happy that distcc = cluster<br> >>> is saturated and this case should be handled.<br> >>><br> >>> hope it's more clear now!<br> >>> br<br> >>> LT<br> >>><br> >>><br> >>><br> >>><br> >>><br> >>><br> >>><br> >>><br> >>><br> >>> =C5=81ukasz Tasz<br> >>><br> >>><br> >>> 2014-10-16 1:39 GMT+02:00 Martin Pool <<a href=3D"mailto:mb= [email protected]">[email protected]</a>>:<br> >>> > Can you try to explain more clearly what difference in qu= eueing behavior<br> >>> > you<br> >>> > expect from this change?<br> >>> ><br> >>> > I think probably the main change that's needed is for= the client to ask<br> >>> > all<br> >>> > masters if they have space, to avoid needing to effective= ly poll by<br> >>> > retrying, or getting stuck waiting for a particular serve= r.<br> >>> ><br> >>> > On Wed, Oct 15, 2014 at 12:53 PM, =C5=81ukasz Tasz <<a= href=3D"mailto:[email protected]">[email protected]</a>> wrote:<br> >>> >><br> >>> >> Hi Guys,<br> >>> >><br> >>> >> please correct me if I'm wrong,<br> >>> >> - currently distcc tries to connect server 3 times, w= ith small delay,<br> >>> >> - server forks x childs and all of them are trying to= accept incoming<br> >>> >> connection.<br> >>> >> If server runs out of childs (all of them are busy), = client will<br> >>> >> fallback, and within next 60 sec will not try this ma= chine.<br> >>> >><br> >>> >> What do you think about redesigning distcc in a way t= hat master server<br> >>> >> will always accept inconing connection, fork a child,= but in a same<br> >>> >> time only x of them will be able to enter compilation= <br> >>> >> task(dcc_spawn_child)? (mayby preforking still could = be used?)<br> >>> >><br> >>> >> This may create kind of queue, client always can deci= de by his own, if<br> >>> >> can wait some=C2=A0 time, or maximum is DISTCC_IO_TIM= EOUT, but still it's<br> >>> >> faster to wait, since probably on a cluster side it&#= 39;s just a pick of<br> >>> >> saturation then making falback to local machine.<br> >>> >><br> >>> >> currently I'm facing situation that many jobs are= making fallback, and<br> >>> >> localmachine is being killed by make's -j calcula= ted for distccd...<br> >>> >><br> >>> >> other trick maybe to pick different machine, if curre= nt is busy, but<br> >>> >> this may be much more complex in my opinion.<br> >>> >><br> >>> >> what do you think?<br> >>> >> regards<br> >>> >> =C5=81ukasz Tasz<br> >>> >> __<br> >>> >> distcc mailing list=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 <a href=3D"http://distcc.samba.org/" target=3D"_blank">http://distc= c.samba.org/</a><br> >>> >> To unsubscribe or change options:<br> >>> >> <a href=3D"https://lists.samba.org/mailman/listinfo/d= istcc" target=3D"_blank">https://lists.samba.org/mailman/listinfo/distcc</a= ><br> >>> ><br> >>> ><br> >>> ><br> >>> ><br> >>> > --<br> >>> > Martin<br> __<br> distcc mailing list=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a href=3D"htt= p://distcc.samba.org/" target=3D"_blank">http://distcc.samba.org/</a><br> To unsubscribe or change options:<br> <a href=3D"https://lists.samba.org/mailman/listinfo/distcc" target=3D"_blan= k">https://lists.samba.org/mailman/listinfo/distcc</a></blockquote></div> --bcaec517198344648b0506c796cf-- --===============4078194226169139121== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline __ distcc mailing list http://distcc.samba.org/ To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/distcc --===============4078194226169139121==--