Re: small redesign...
Łukasz Tasz <[email protected]> Sat, 1 Nov 2014 16:08:47 +0100
| Newsgroups | gmane.comp.compilers.distcc |
|---|---|
| Message-ID | <CAGRr-Niy_bcDEFQeJNNMf2SJH4J9fG=gsnVv38SJFzLJ3RHs4Q@mail.gmail.com> |
--===============9179695028737457539== Content-Type: multipart/alternative; boundary=001a11c3f10684d82c0506cd7d94 --001a11c3f10684d82c0506cd7d94 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Sure, I just made quick fix to test my test case, and immediately share it with you. I will try to send more polite fix:) Regards lt 1 lis 2014 09:06 "Fergus Henderson" <[email protected]> napisa=C5=82(a): > Well, perhaps it would be a good idea to add a distccd flag or environmen= t > variable to control the queue length rather than hard-coding 10 or 256? > On 31 Oct 2014 11:37, "=C5=81ukasz Tasz" <[email protected]> wrote: > >> Hi Guys, >> >> I'm very very happy, reasons of my failures are identified. >> issue is in: >> --- src/srvnet.c (wersja 177) >> +++ src/srvnet.c (kopia robocza) >> @@ -99,7 +99,7 @@ >> rs_log_info("listening on %s", sa_buf ? sa_buf : "UNKNOWN"); >> free(sa_buf); >> >> - if (listen(fd, 10)) { >> + if (listen(fd, 256)) { >> rs_log_error("listen failed: %s", strerror(errno)); >> close(fd); >> return EXIT_BIND_FAILED; >> Index: src/io.c >> >> queue for new connetcion was minited to 10, that's why in case that >> cluster is overloaded, many connection are reseted. >> aim is to even wait 5 min for cluster availability, then compile localy. >> >> @Jarek, thanks for support! >> >> let's discuss if we should fix it or not. >> >> regards >> Lukasz >> >> >> =C5=81ukasz Tasz >> >> >> 2014-10-24 10:27 GMT+02:00 =C5=81ukasz Tasz <[email protected]>: >> > Hi Martin >> > >> > What I have noticed. >> > Client tries to connect distccd 3 times with 500ms delays in between. >> > Linux kernel by default accept 128 connection. >> > If client creates connection, even if no executors are avaliable, >> > connection is accepted and queued by kernel running distccd. >> > This leads to situation that client thinks that distccd is reserved, >> > but in fact connection still waits to be accepted by distccd server. >> > I suspect that then client starts communication too fast, distcc wont >> > receive DIST token, and both sides waits, communication is broken, and >> > then timeouts are applied for client default is applied, for server >> > there is no defaults. >> > >> > fail scenarion is: >> > one distccd, and two distcc users, both of them will try to compile >> > with DISTCC_HOSTS=3Ddistccd/1,cpp,lzo, both users have lot of big >> > objects, cluster is overloaded with ratio 2. >> > This still should be OK, that third, and forth user will join cluster. >> > >> > Easy reproducer is to set one distcc, and set distcc_hosts=3Ddistccd/2= 0, >> > this is broken configuration, but simulates overload by 20 - 20 >> > developers uses cluster in a same time. >> > Please remember that those are exceptional situation, but developer >> > can start compilation with -j 1000 from his laptop, and cluster will >> > timeout, then receiving 1000 jobs on a laptop will end with memmory >> > killer :D >> > Those are exceptional situation, and somehow cluster should handle tha= t. >> > >> > In the attachement, next to some pump changes, you can find change >> > which is moving making connection to very beginning, when distcc is >> > picking host, also remote connection is made. if this will fail, discc >> > follow default behaviour, goes sleep for one sec, and will pick host >> > again. But this requires additional administration change on distccd >> > machine: >> > iptables -I INPUT -p tcp --dport 3632 -m connlimit --connlimit-above >> > <NUMBER OF DISTCCD> --connlimit-mask 0 -j REJECT --reject-with >> > tcp-reset >> > which accept only number of connection which equals to number of >> executors. >> > >> > So far so good! >> > remark, patch is done on top of arankine_distcc_issue16-r335, since >> > his pump changes are making pump mode working on my environment. >> > But distccd allocation I tested also on latest official distcc release= . >> > >> > let me know what you think! >> > >> > with best regards >> > Lukasz >> > >> > >> > >> > =C5=81ukasz Tasz >> > >> > >> > 2014-10-24 2:42 GMT+02:00 Martin Pool <[email protected]>: >> >> It seems like if there's nowhere to execute the job, we want the clie= nt >> >> program to just pause, before using too many resources, until it gets >> >> unqueued by a server ready to do the job. (Or, by a local slot being >> >> available.) >> >> >> >> >> >> On Thu Oct 16 2014 at 2:43:35 AM =C5=81ukasz Tasz <[email protected]> wr= ote: >> >>> >> >>> Hi Martin, >> >>> >> >>> Lets assume that you can trigger more compilation tasks executors >> then you >> >>> have. >> >>> In this scenario you are facing situation that cluster is saturated. >> >>> When such a compilation will be triggered by two developers, or two = CI >> >>> (e.g jenkins) jobs, then cluster is saturated twice... >> >>> >> >>> Default behaviour is to lock locally slot, and try to connect three >> >>> times, if not, fallback, if fallback is disabled CI got failed build >> >>> (fallback is not the case, since local machine cannot handle -j >> >>> $(distcc -j)). >> >>> >> >>> consider scenario, I have 1000 objects, 500 executors, >> >>> - clean build on one machine takes >> >>> 1000 * 20 sec (one obj) =3D 20000 / 16 processors =3D 1000 sec, >> >>> - on cluster (1000/500) * 20 sec =3D 40 sec >> >>> >> >>> Saturating cluster was impossible without pump mode, but now with pu= mp >> >>> mode after "warm up" effect, pump can dispatch many tasks, and I fac= ed >> >>> situation that saturated cluster destroys almost every compilation. >> >>> >> >>> My expectation is that cluster wont reject my connect, or reject wil= l >> >>> be handled, either by client, either by server. >> >>> >> >>> by server: >> >>> - accept every connetion, >> >>> - fork child if not accepted by child, >> >>> - in case of pump prepare local dir structure, receive headers >> >>> - --critical section starts here-- multi value semaphore with value >> >>> maxchild >> >>> - execute job >> >>> - release semaphore >> >>> >> >>> >> >>> Also what you suggested may be even better solution, since client wi= ll >> >>> pick first avaliable executor instead of entering queue, so distcc >> >>> could make connection already in function dcc_lock_one() >> >>> >> >>> I already tried to set DISTCC_DIR on a common nfs share, but in case >> >>> you are triggering so many jobs, this started to be bottle neck... I >> >>> won't tell about locking on nfs, and also scenario that somebody wil= l >> >>> make a lock on nfs and machine will got crash - will not work by >> >>> design :) >> >>> >> >>> I know that scenario is not happening very often, and it has more or >> >>> less picks characteristic, but we should be happy that distcc cluste= r >> >>> is saturated and this case should be handled. >> >>> >> >>> hope it's more clear now! >> >>> br >> >>> LT >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> =C5=81ukasz Tasz >> >>> >> >>> >> >>> 2014-10-16 1:39 GMT+02:00 Martin Pool <[email protected]>: >> >>> > Can you try to explain more clearly what difference in queueing >> behavior >> >>> > you >> >>> > expect from this change? >> >>> > >> >>> > I think probably the main change that's needed is for the client t= o >> ask >> >>> > all >> >>> > masters if they have space, to avoid needing to effectively poll b= y >> >>> > retrying, or getting stuck waiting for a particular server. >> >>> > >> >>> > On Wed, Oct 15, 2014 at 12:53 PM, =C5=81ukasz Tasz <[email protected]= > >> wrote: >> >>> >> >> >>> >> Hi Guys, >> >>> >> >> >>> >> please correct me if I'm wrong, >> >>> >> - currently distcc tries to connect server 3 times, with small >> delay, >> >>> >> - server forks x childs and all of them are trying to accept >> incoming >> >>> >> connection. >> >>> >> If server runs out of childs (all of them are busy), client will >> >>> >> fallback, and within next 60 sec will not try this machine. >> >>> >> >> >>> >> What do you think about redesigning distcc in a way that master >> server >> >>> >> will always accept inconing connection, fork a child, but in a sa= me >> >>> >> time only x of them will be able to enter compilation >> >>> >> task(dcc_spawn_child)? (mayby preforking still could be used?) >> >>> >> >> >>> >> This may create kind of queue, client always can decide by his >> own, if >> >>> >> can wait some time, or maximum is DISTCC_IO_TIMEOUT, but still >> it's >> >>> >> faster to wait, since probably on a cluster side it's just a pick >> of >> >>> >> saturation then making falback to local machine. >> >>> >> >> >>> >> currently I'm facing situation that many jobs are making fallback= , >> and >> >>> >> localmachine is being killed by make's -j calculated for distccd.= .. >> >>> >> >> >>> >> other trick maybe to pick different machine, if current is busy, >> but >> >>> >> this may be much more complex in my opinion. >> >>> >> >> >>> >> what do you think? >> >>> >> regards >> >>> >> =C5=81ukasz Tasz >> >>> >> __ >> >>> >> distcc mailing list http://distcc.samba.org/ >> >>> >> To unsubscribe or change options: >> >>> >> https://lists.samba.org/mailman/listinfo/distcc >> >>> > >> >>> > >> >>> > >> >>> > >> >>> > -- >> >>> > Martin >> __ >> distcc mailing list http://distcc.samba.org/ >> To unsubscribe or change options: >> https://lists.samba.org/mailman/listinfo/distcc > > --001a11c3f10684d82c0506cd7d94 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <p dir=3D"ltr">Sure, I just made quick fix to test my test case,=C2=A0 and = immediately share it with you. I will try to send more polite fix:)<br> Regards<br> lt</p> <div class=3D"gmail_quote">1 lis 2014 09:06 "Fergus Henderson" &l= t;<a href=3D"mailto:[email protected]">[email protected]</a>> napisa=C5= =82(a):<br type=3D"attribution"><blockquote class=3D"gmail_quote" style=3D"= margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir=3D"lt= r">Well, perhaps it would be a good idea to add a distccd flag or environme= nt variable to control the queue length rather than hard-coding 10 or 256?<= /p> <div class=3D"gmail_quote">On 31 Oct 2014 11:37, "=C5=81ukasz Tasz&quo= t; <<a href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</= a>> wrote:<br type=3D"attribution"><blockquote class=3D"gmail_quote" sty= le=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Guy= s,<br> <br> I'm very very happy, reasons of my failures are identified.<br> issue is in:<br> --- src/srvnet.c=C2=A0 =C2=A0 =C2=A0 =C2=A0 (wersja 177)<br> +++ src/srvnet.c=C2=A0 =C2=A0 =C2=A0 =C2=A0 (kopia robocza)<br> @@ -99,7 +99,7 @@<br> =C2=A0 =C2=A0 =C2=A0rs_log_info("listening on %s", sa_buf ? sa_bu= f : "UNKNOWN");<br> =C2=A0 =C2=A0 =C2=A0free(sa_buf);<br> <br> -=C2=A0 =C2=A0 if (listen(fd, 10)) {<br> +=C2=A0 =C2=A0 if (listen(fd, 256)) {<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rs_log_error("listen failed: %s"= ;, strerror(errno));<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0close(fd);<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0return EXIT_BIND_FAILED;<br> Index: src/io.c<br> <br> queue for new connetcion was minited to 10, that's why in case that<br> cluster is overloaded, many connection are reseted.<br> aim is to even wait 5 min for cluster availability, then compile localy.<br= > <br> @Jarek, thanks for support!<br> <br> let's discuss if we should fix it or not.<br> <br> regards<br> Lukasz<br> <br> <br> =C5=81ukasz Tasz<br> <br> <br> 2014-10-24 10:27 GMT+02:00 =C5=81ukasz Tasz <<a href=3D"mailto:lukasz@ta= sz.eu" target=3D"_blank">[email protected]</a>>:<br> > Hi Martin<br> ><br> > What I have noticed.<br> > Client tries to connect distccd 3 times with 500ms delays in between.<= br> > Linux kernel by default accept 128 connection.<br> > If client creates connection, even if no executors are avaliable,<br> > connection is accepted and queued by kernel running distccd.<br> > This leads to situation that client thinks that distccd is reserved,<b= r> > but in fact connection still waits to be accepted by distccd server.<b= r> > I suspect that then client starts communication too fast, distcc wont<= br> > receive DIST token, and both sides waits, communication is broken, and= <br> > then timeouts are applied for client default is applied, for server<br= > > there is no defaults.<br> ><br> > fail scenarion is:<br> > one distccd, and two distcc users, both of them will try to compile<br= > > with DISTCC_HOSTS=3Ddistccd/1,cpp,lzo, both users have lot of big<br> > objects, cluster is overloaded with ratio 2.<br> > This still should be OK, that third, and forth user will join cluster.= <br> ><br> > Easy reproducer is to set one distcc, and set distcc_hosts=3Ddistccd/2= 0,<br> > this is broken configuration, but simulates overload by 20 - 20<br> > developers uses cluster in a same time.<br> > Please remember that those are exceptional situation, but developer<br= > > can start compilation with -j 1000 from his laptop, and cluster will<b= r> > timeout, then receiving 1000 jobs on a laptop will end with memmory<br= > > killer :D<br> > Those are exceptional situation, and somehow cluster should handle tha= t.<br> ><br> > In the attachement, next to some pump changes, you can find change<br> > which is moving making connection to very beginning, when distcc is<br= > > picking host, also remote connection is made. if this will fail, discc= <br> > follow default behaviour, goes sleep for one sec, and will pick host<b= r> > again. But this requires additional administration change on distccd<b= r> > machine:<br> > iptables -I INPUT -p tcp --dport 3632 -m connlimit --connlimit-above<b= r> > <NUMBER OF DISTCCD> --connlimit-mask 0 -j REJECT --reject-with<b= r> > tcp-reset<br> > which accept only number of connection which equals to number of execu= tors.<br> ><br> > So far so good!<br> > remark, patch is done on top of arankine_distcc_issue16-r335, since<br= > > his pump changes are making pump mode working on my environment.<br> > But distccd allocation I tested also on latest official distcc release= .<br> ><br> > let me know what you think!<br> ><br> > with best regards<br> > Lukasz<br> ><br> ><br> ><br> > =C5=81ukasz Tasz<br> ><br> ><br> > 2014-10-24 2:42 GMT+02:00 Martin Pool <<a href=3D"mailto:mbp@source= frog.net" target=3D"_blank">[email protected]</a>>:<br> >> It seems like if there's nowhere to execute the job, we want t= he client<br> >> program to just pause, before using too many resources, until it g= ets<br> >> unqueued by a server ready to do the job. (Or, by a local slot bei= ng<br> >> available.)<br> >><br> >><br> >> On Thu Oct 16 2014 at 2:43:35 AM =C5=81ukasz Tasz <<a href=3D"m= ailto:[email protected]" target=3D"_blank">[email protected]</a>> wrote:<br> >>><br> >>> Hi Martin,<br> >>><br> >>> Lets assume that you can trigger more compilation tasks execut= ors then you<br> >>> have.<br> >>> In this scenario you are facing situation that cluster is satu= rated.<br> >>> When such a compilation will be triggered by two developers, o= r two CI<br> >>> (e.g jenkins) jobs, then cluster is saturated twice...<br> >>><br> >>> Default behaviour is to lock locally slot, and try to connect = three<br> >>> times, if not, fallback, if fallback is disabled CI got failed= build<br> >>> (fallback is not the case, since local machine cannot handle -= j<br> >>> $(distcc -j)).<br> >>><br> >>> consider scenario, I have 1000 objects, 500 executors,<br> >>> - clean build on one machine takes<br> >>>=C2=A0 =C2=A01000 * 20 sec (one obj) =3D 20000 / 16 processors = =3D 1000 sec,<br> >>> - on cluster (1000/500) * 20 sec =3D 40 sec<br> >>><br> >>> Saturating cluster was impossible without pump mode, but now w= ith pump<br> >>> mode after "warm up" effect, pump can dispatch many = tasks, and I faced<br> >>> situation that saturated cluster destroys almost=C2=A0 every c= ompilation.<br> >>><br> >>> My expectation is that cluster wont reject my connect, or reje= ct will<br> >>> be handled, either by client, either by server.<br> >>><br> >>> by server:<br> >>> - accept every connetion,<br> >>> - fork child if not accepted by child,<br> >>> - in case of pump prepare local dir structure, receive headers= <br> >>> - --critical section starts here-- multi value semaphore with = value<br> >>> maxchild<br> >>>=C2=A0 =C2=A0- execute job<br> >>> - release semaphore<br> >>><br> >>><br> >>> Also what you suggested may be even better solution, since cli= ent will<br> >>> pick first avaliable executor instead of entering queue, so di= stcc<br> >>> could make connection already in function dcc_lock_one()<br> >>><br> >>> I already tried to set DISTCC_DIR on a common nfs share, but i= n case<br> >>> you are triggering so many jobs, this started to be bottle nec= k... I<br> >>> won't tell about locking on nfs, and also scenario that so= mebody will<br> >>> make a lock on nfs and machine will got crash - will not work = by<br> >>> design :)<br> >>><br> >>> I know that scenario is not happening very often, and it has m= ore or<br> >>> less picks characteristic, but we should be happy that distcc = cluster<br> >>> is saturated and this case should be handled.<br> >>><br> >>> hope it's more clear now!<br> >>> br<br> >>> LT<br> >>><br> >>><br> >>><br> >>><br> >>><br> >>><br> >>><br> >>><br> >>><br> >>> =C5=81ukasz Tasz<br> >>><br> >>><br> >>> 2014-10-16 1:39 GMT+02:00 Martin Pool <<a href=3D"mailto:mb= [email protected]" target=3D"_blank">[email protected]</a>>:<br> >>> > Can you try to explain more clearly what difference in qu= eueing behavior<br> >>> > you<br> >>> > expect from this change?<br> >>> ><br> >>> > I think probably the main change that's needed is for= the client to ask<br> >>> > all<br> >>> > masters if they have space, to avoid needing to effective= ly poll by<br> >>> > retrying, or getting stuck waiting for a particular serve= r.<br> >>> ><br> >>> > On Wed, Oct 15, 2014 at 12:53 PM, =C5=81ukasz Tasz <<a= href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a>> wr= ote:<br> >>> >><br> >>> >> Hi Guys,<br> >>> >><br> >>> >> please correct me if I'm wrong,<br> >>> >> - currently distcc tries to connect server 3 times, w= ith small delay,<br> >>> >> - server forks x childs and all of them are trying to= accept incoming<br> >>> >> connection.<br> >>> >> If server runs out of childs (all of them are busy), = client will<br> >>> >> fallback, and within next 60 sec will not try this ma= chine.<br> >>> >><br> >>> >> What do you think about redesigning distcc in a way t= hat master server<br> >>> >> will always accept inconing connection, fork a child,= but in a same<br> >>> >> time only x of them will be able to enter compilation= <br> >>> >> task(dcc_spawn_child)? (mayby preforking still could = be used?)<br> >>> >><br> >>> >> This may create kind of queue, client always can deci= de by his own, if<br> >>> >> can wait some=C2=A0 time, or maximum is DISTCC_IO_TIM= EOUT, but still it's<br> >>> >> faster to wait, since probably on a cluster side it&#= 39;s just a pick of<br> >>> >> saturation then making falback to local machine.<br> >>> >><br> >>> >> currently I'm facing situation that many jobs are= making fallback, and<br> >>> >> localmachine is being killed by make's -j calcula= ted for distccd...<br> >>> >><br> >>> >> other trick maybe to pick different machine, if curre= nt is busy, but<br> >>> >> this may be much more complex in my opinion.<br> >>> >><br> >>> >> what do you think?<br> >>> >> regards<br> >>> >> =C5=81ukasz Tasz<br> >>> >> __<br> >>> >> distcc mailing list=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 <a href=3D"http://distcc.samba.org/" target=3D"_blank">http://distc= c.samba.org/</a><br> >>> >> To unsubscribe or change options:<br> >>> >> <a href=3D"https://lists.samba.org/mailman/listinfo/d= istcc" target=3D"_blank">https://lists.samba.org/mailman/listinfo/distcc</a= ><br> >>> ><br> >>> ><br> >>> ><br> >>> ><br> >>> > --<br> >>> > Martin<br> __<br> distcc mailing list=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a href=3D"htt= p://distcc.samba.org/" target=3D"_blank">http://distcc.samba.org/</a><br> To unsubscribe or change options:<br> <a href=3D"https://lists.samba.org/mailman/listinfo/distcc" target=3D"_blan= k">https://lists.samba.org/mailman/listinfo/distcc</a></blockquote></div> </blockquote></div> --001a11c3f10684d82c0506cd7d94-- --===============9179695028737457539== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline __ distcc mailing list http://distcc.samba.org/ To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/distcc --===============9179695028737457539==--