Re: small redesign...
Fergus Henderson <[email protected]> Sun, 2 Nov 2014 10:24:34 +0000
| Newsgroups | gmane.comp.compilers.distcc |
|---|---|
| Message-ID | <CAPXkjd9jpPTtG+xfoAF_5qUWCGYAnan8YpSTGWwXqD4eZHuhCA@mail.gmail.com> |
--===============4759822127168289635== Content-Type: multipart/alternative; boundary=089e0122efeced27580506dda2ea --089e0122efeced27580506dda2ea Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 1 Nov 2014 15:08, "=C5=81ukasz Tasz" <[email protected]> wrote: > > Sure, I just made quick fix to test my test case, and immediately share it with you. Sure, understood -- that's great, thanks. > I will try to send more polite fix:) > Regards > lt > > 1 lis 2014 09:06 "Fergus Henderson" <[email protected]> napisa=C5=82(a): > >> Well, perhaps it would be a good idea to add a distccd flag or environment variable to control the queue length rather than hard-coding 10 or 256? >> >> On 31 Oct 2014 11:37, "=C5=81ukasz Tasz" <[email protected]> wrote: >>> >>> Hi Guys, >>> >>> I'm very very happy, reasons of my failures are identified. >>> issue is in: >>> --- src/srvnet.c (wersja 177) >>> +++ src/srvnet.c (kopia robocza) >>> @@ -99,7 +99,7 @@ >>> rs_log_info("listening on %s", sa_buf ? sa_buf : "UNKNOWN"); >>> free(sa_buf); >>> >>> - if (listen(fd, 10)) { >>> + if (listen(fd, 256)) { >>> rs_log_error("listen failed: %s", strerror(errno)); >>> close(fd); >>> return EXIT_BIND_FAILED; >>> Index: src/io.c >>> >>> queue for new connetcion was minited to 10, that's why in case that >>> cluster is overloaded, many connection are reseted. >>> aim is to even wait 5 min for cluster availability, then compile localy= . >>> >>> @Jarek, thanks for support! >>> >>> let's discuss if we should fix it or not. >>> >>> regards >>> Lukasz >>> >>> >>> =C5=81ukasz Tasz >>> >>> >>> 2014-10-24 10:27 GMT+02:00 =C5=81ukasz Tasz <[email protected]>: >>> > Hi Martin >>> > >>> > What I have noticed. >>> > Client tries to connect distccd 3 times with 500ms delays in between. >>> > Linux kernel by default accept 128 connection. >>> > If client creates connection, even if no executors are avaliable, >>> > connection is accepted and queued by kernel running distccd. >>> > This leads to situation that client thinks that distccd is reserved, >>> > but in fact connection still waits to be accepted by distccd server. >>> > I suspect that then client starts communication too fast, distcc wont >>> > receive DIST token, and both sides waits, communication is broken, an= d >>> > then timeouts are applied for client default is applied, for server >>> > there is no defaults. >>> > >>> > fail scenarion is: >>> > one distccd, and two distcc users, both of them will try to compile >>> > with DISTCC_HOSTS=3Ddistccd/1,cpp,lzo, both users have lot of big >>> > objects, cluster is overloaded with ratio 2. >>> > This still should be OK, that third, and forth user will join cluster= . >>> > >>> > Easy reproducer is to set one distcc, and set distcc_hosts=3Ddistccd/= 20, >>> > this is broken configuration, but simulates overload by 20 - 20 >>> > developers uses cluster in a same time. >>> > Please remember that those are exceptional situation, but developer >>> > can start compilation with -j 1000 from his laptop, and cluster will >>> > timeout, then receiving 1000 jobs on a laptop will end with memmory >>> > killer :D >>> > Those are exceptional situation, and somehow cluster should handle that. >>> > >>> > In the attachement, next to some pump changes, you can find change >>> > which is moving making connection to very beginning, when distcc is >>> > picking host, also remote connection is made. if this will fail, disc= c >>> > follow default behaviour, goes sleep for one sec, and will pick host >>> > again. But this requires additional administration change on distccd >>> > machine: >>> > iptables -I INPUT -p tcp --dport 3632 -m connlimit --connlimit-above >>> > <NUMBER OF DISTCCD> --connlimit-mask 0 -j REJECT --reject-with >>> > tcp-reset >>> > which accept only number of connection which equals to number of executors. >>> > >>> > So far so good! >>> > remark, patch is done on top of arankine_distcc_issue16-r335, since >>> > his pump changes are making pump mode working on my environment. >>> > But distccd allocation I tested also on latest official distcc release. >>> > >>> > let me know what you think! >>> > >>> > with best regards >>> > Lukasz >>> > >>> > >>> > >>> > =C5=81ukasz Tasz >>> > >>> > >>> > 2014-10-24 2:42 GMT+02:00 Martin Pool <[email protected]>: >>> >> It seems like if there's nowhere to execute the job, we want the client >>> >> program to just pause, before using too many resources, until it get= s >>> >> unqueued by a server ready to do the job. (Or, by a local slot being >>> >> available.) >>> >> >>> >> >>> >> On Thu Oct 16 2014 at 2:43:35 AM =C5=81ukasz Tasz <[email protected]> w= rote: >>> >>> >>> >>> Hi Martin, >>> >>> >>> >>> Lets assume that you can trigger more compilation tasks executors then you >>> >>> have. >>> >>> In this scenario you are facing situation that cluster is saturated= . >>> >>> When such a compilation will be triggered by two developers, or two CI >>> >>> (e.g jenkins) jobs, then cluster is saturated twice... >>> >>> >>> >>> Default behaviour is to lock locally slot, and try to connect three >>> >>> times, if not, fallback, if fallback is disabled CI got failed buil= d >>> >>> (fallback is not the case, since local machine cannot handle -j >>> >>> $(distcc -j)). >>> >>> >>> >>> consider scenario, I have 1000 objects, 500 executors, >>> >>> - clean build on one machine takes >>> >>> 1000 * 20 sec (one obj) =3D 20000 / 16 processors =3D 1000 sec, >>> >>> - on cluster (1000/500) * 20 sec =3D 40 sec >>> >>> >>> >>> Saturating cluster was impossible without pump mode, but now with pump >>> >>> mode after "warm up" effect, pump can dispatch many tasks, and I faced >>> >>> situation that saturated cluster destroys almost every compilation= . >>> >>> >>> >>> My expectation is that cluster wont reject my connect, or reject will >>> >>> be handled, either by client, either by server. >>> >>> >>> >>> by server: >>> >>> - accept every connetion, >>> >>> - fork child if not accepted by child, >>> >>> - in case of pump prepare local dir structure, receive headers >>> >>> - --critical section starts here-- multi value semaphore with value >>> >>> maxchild >>> >>> - execute job >>> >>> - release semaphore >>> >>> >>> >>> >>> >>> Also what you suggested may be even better solution, since client will >>> >>> pick first avaliable executor instead of entering queue, so distcc >>> >>> could make connection already in function dcc_lock_one() >>> >>> >>> >>> I already tried to set DISTCC_DIR on a common nfs share, but in cas= e >>> >>> you are triggering so many jobs, this started to be bottle neck... = I >>> >>> won't tell about locking on nfs, and also scenario that somebody will >>> >>> make a lock on nfs and machine will got crash - will not work by >>> >>> design :) >>> >>> >>> >>> I know that scenario is not happening very often, and it has more o= r >>> >>> less picks characteristic, but we should be happy that distcc cluster >>> >>> is saturated and this case should be handled. >>> >>> >>> >>> hope it's more clear now! >>> >>> br >>> >>> LT >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> =C5=81ukasz Tasz >>> >>> >>> >>> >>> >>> 2014-10-16 1:39 GMT+02:00 Martin Pool <[email protected]>: >>> >>> > Can you try to explain more clearly what difference in queueing behavior >>> >>> > you >>> >>> > expect from this change? >>> >>> > >>> >>> > I think probably the main change that's needed is for the client to ask >>> >>> > all >>> >>> > masters if they have space, to avoid needing to effectively poll by >>> >>> > retrying, or getting stuck waiting for a particular server. >>> >>> > >>> >>> > On Wed, Oct 15, 2014 at 12:53 PM, =C5=81ukasz Tasz <[email protected]= u> wrote: >>> >>> >> >>> >>> >> Hi Guys, >>> >>> >> >>> >>> >> please correct me if I'm wrong, >>> >>> >> - currently distcc tries to connect server 3 times, with small delay, >>> >>> >> - server forks x childs and all of them are trying to accept incoming >>> >>> >> connection. >>> >>> >> If server runs out of childs (all of them are busy), client will >>> >>> >> fallback, and within next 60 sec will not try this machine. >>> >>> >> >>> >>> >> What do you think about redesigning distcc in a way that master server >>> >>> >> will always accept inconing connection, fork a child, but in a same >>> >>> >> time only x of them will be able to enter compilation >>> >>> >> task(dcc_spawn_child)? (mayby preforking still could be used?) >>> >>> >> >>> >>> >> This may create kind of queue, client always can decide by his own, if >>> >>> >> can wait some time, or maximum is DISTCC_IO_TIMEOUT, but still it's >>> >>> >> faster to wait, since probably on a cluster side it's just a pick of >>> >>> >> saturation then making falback to local machine. >>> >>> >> >>> >>> >> currently I'm facing situation that many jobs are making fallback, and >>> >>> >> localmachine is being killed by make's -j calculated for distccd... >>> >>> >> >>> >>> >> other trick maybe to pick different machine, if current is busy, but >>> >>> >> this may be much more complex in my opinion. >>> >>> >> >>> >>> >> what do you think? >>> >>> >> regards >>> >>> >> =C5=81ukasz Tasz >>> >>> >> __ >>> >>> >> distcc mailing list http://distcc.samba.org/ >>> >>> >> To unsubscribe or change options: >>> >>> >> https://lists.samba.org/mailman/listinfo/distcc >>> >>> > >>> >>> > >>> >>> > >>> >>> > >>> >>> > -- >>> >>> > Martin >>> __ >>> distcc mailing list http://distcc.samba.org/ >>> To unsubscribe or change options: >>> https://lists.samba.org/mailman/listinfo/distcc --089e0122efeced27580506dda2ea Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <p dir=3D"ltr"><br> On 1 Nov 2014 15:08, "=C5=81ukasz Tasz" <<a href=3D"mailto:luk= [email protected]">[email protected]</a>> wrote:<br> ><br> > Sure, I just made quick fix to test my test case,=C2=A0 and immediatel= y share it with you.</p> <p dir=3D"ltr">Sure, understood -- that's great, thanks.</p> <p dir=3D"ltr">> I will try to send more polite fix:)<br> > Regards<br> > lt<br> ><br> > 1 lis 2014 09:06 "Fergus Henderson" <<a href=3D"mailto:fe= [email protected]">[email protected]</a>> napisa=C5=82(a):<br> ><br> >> Well, perhaps it would be a good idea to add a distccd flag or env= ironment variable to control the queue length rather than hard-coding 10 or= 256?<br> >><br> >> On 31 Oct 2014 11:37, "=C5=81ukasz Tasz" <<a href=3D"= mailto:[email protected]">[email protected]</a>> wrote:<br> >>><br> >>> Hi Guys,<br> >>><br> >>> I'm very very happy, reasons of my failures are identified= .<br> >>> issue is in:<br> >>> --- src/srvnet.c=C2=A0 =C2=A0 =C2=A0 =C2=A0 (wersja 177)<br> >>> +++ src/srvnet.c=C2=A0 =C2=A0 =C2=A0 =C2=A0 (kopia robocza)<br= > >>> @@ -99,7 +99,7 @@<br> >>> =C2=A0 =C2=A0 =C2=A0rs_log_info("listening on %s", s= a_buf ? sa_buf : "UNKNOWN");<br> >>> =C2=A0 =C2=A0 =C2=A0free(sa_buf);<br> >>><br> >>> -=C2=A0 =C2=A0 if (listen(fd, 10)) {<br> >>> +=C2=A0 =C2=A0 if (listen(fd, 256)) {<br> >>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rs_log_error("listen fa= iled: %s", strerror(errno));<br> >>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0close(fd);<br> >>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0return EXIT_BIND_FAILED;<br> >>> Index: src/io.c<br> >>><br> >>> queue for new connetcion was minited to 10, that's why in = case that<br> >>> cluster is overloaded, many connection are reseted.<br> >>> aim is to even wait 5 min for cluster availability, then compi= le localy.<br> >>><br> >>> @Jarek, thanks for support!<br> >>><br> >>> let's discuss if we should fix it or not.<br> >>><br> >>> regards<br> >>> Lukasz<br> >>><br> >>><br> >>> =C5=81ukasz Tasz<br> >>><br> >>><br> >>> 2014-10-24 10:27 GMT+02:00 =C5=81ukasz Tasz <<a href=3D"mai= lto:[email protected]">[email protected]</a>>:<br> >>> > Hi Martin<br> >>> ><br> >>> > What I have noticed.<br> >>> > Client tries to connect distccd 3 times with 500ms delays= in between.<br> >>> > Linux kernel by default accept 128 connection.<br> >>> > If client creates connection, even if no executors are av= aliable,<br> >>> > connection is accepted and queued by kernel running distc= cd.<br> >>> > This leads to situation that client thinks that distccd i= s reserved,<br> >>> > but in fact connection still waits to be accepted by dist= ccd server.<br> >>> > I suspect that then client starts communication too fast,= distcc wont<br> >>> > receive DIST token, and both sides waits, communication i= s broken, and<br> >>> > then timeouts are applied for client default is applied, = for server<br> >>> > there is no defaults.<br> >>> ><br> >>> > fail scenarion is:<br> >>> > one distccd, and two distcc users, both of them will try = to compile<br> >>> > with DISTCC_HOSTS=3Ddistccd/1,cpp,lzo, both users have lo= t of big<br> >>> > objects, cluster is overloaded with ratio 2.<br> >>> > This still should be OK, that third, and forth user will = join cluster.<br> >>> ><br> >>> > Easy reproducer is to set one distcc, and set distcc_host= s=3Ddistccd/20,<br> >>> > this is broken configuration, but simulates overload by 2= 0 - 20<br> >>> > developers uses cluster in a same time.<br> >>> > Please remember that those are exceptional situation, but= developer<br> >>> > can start compilation with -j 1000 from his laptop, and c= luster will<br> >>> > timeout, then receiving 1000 jobs on a laptop will end wi= th memmory<br> >>> > killer :D<br> >>> > Those are exceptional situation, and somehow cluster shou= ld handle that.<br> >>> ><br> >>> > In the attachement, next to some pump changes, you can fi= nd change<br> >>> > which is moving making connection to very beginning, when= distcc is<br> >>> > picking host, also remote connection is made. if this wil= l fail, discc<br> >>> > follow default behaviour, goes sleep for one sec, and wil= l pick host<br> >>> > again. But this requires additional administration change= on distccd<br> >>> > machine:<br> >>> > iptables -I INPUT -p tcp --dport 3632 -m connlimit --conn= limit-above<br> >>> > <NUMBER OF DISTCCD> --connlimit-mask 0 -j REJECT --= reject-with<br> >>> > tcp-reset<br> >>> > which accept only number of connection which equals to nu= mber of executors.<br> >>> ><br> >>> > So far so good!<br> >>> > remark, patch is done on top of arankine_distcc_issue16-r= 335, since<br> >>> > his pump changes are making pump mode working on my envir= onment.<br> >>> > But distccd allocation I tested also on latest official d= istcc release.<br> >>> ><br> >>> > let me know what you think!<br> >>> ><br> >>> > with best regards<br> >>> > Lukasz<br> >>> ><br> >>> ><br> >>> ><br> >>> > =C5=81ukasz Tasz<br> >>> ><br> >>> ><br> >>> > 2014-10-24 2:42 GMT+02:00 Martin Pool <<a href=3D"mail= to:[email protected]">[email protected]</a>>:<br> >>> >> It seems like if there's nowhere to execute the j= ob, we want the client<br> >>> >> program to just pause, before using too many resource= s, until it gets<br> >>> >> unqueued by a server ready to do the job. (Or, by a l= ocal slot being<br> >>> >> available.)<br> >>> >><br> >>> >><br> >>> >> On Thu Oct 16 2014 at 2:43:35 AM =C5=81ukasz Tasz <= ;<a href=3D"mailto:[email protected]">[email protected]</a>> wrote:<br> >>> >>><br> >>> >>> Hi Martin,<br> >>> >>><br> >>> >>> Lets assume that you can trigger more compilation= tasks executors then you<br> >>> >>> have.<br> >>> >>> In this scenario you are facing situation that cl= uster is saturated.<br> >>> >>> When such a compilation will be triggered by two = developers, or two CI<br> >>> >>> (e.g jenkins) jobs, then cluster is saturated twi= ce...<br> >>> >>><br> >>> >>> Default behaviour is to lock locally slot, and tr= y to connect three<br> >>> >>> times, if not, fallback, if fallback is disabled = CI got failed build<br> >>> >>> (fallback is not the case, since local machine ca= nnot handle -j<br> >>> >>> $(distcc -j)).<br> >>> >>><br> >>> >>> consider scenario, I have 1000 objects, 500 execu= tors,<br> >>> >>> - clean build on one machine takes<br> >>> >>>=C2=A0 =C2=A01000 * 20 sec (one obj) =3D 20000 / 1= 6 processors =3D 1000 sec,<br> >>> >>> - on cluster (1000/500) * 20 sec =3D 40 sec<br> >>> >>><br> >>> >>> Saturating cluster was impossible without pump mo= de, but now with pump<br> >>> >>> mode after "warm up" effect, pump can d= ispatch many tasks, and I faced<br> >>> >>> situation that saturated cluster destroys almost= =C2=A0 every compilation.<br> >>> >>><br> >>> >>> My expectation is that cluster wont reject my con= nect, or reject will<br> >>> >>> be handled, either by client, either by server.<b= r> >>> >>><br> >>> >>> by server:<br> >>> >>> - accept every connetion,<br> >>> >>> - fork child if not accepted by child,<br> >>> >>> - in case of pump prepare local dir structure, re= ceive headers<br> >>> >>> - --critical section starts here-- multi value se= maphore with value<br> >>> >>> maxchild<br> >>> >>>=C2=A0 =C2=A0- execute job<br> >>> >>> - release semaphore<br> >>> >>><br> >>> >>><br> >>> >>> Also what you suggested may be even better soluti= on, since client will<br> >>> >>> pick first avaliable executor instead of entering= queue, so distcc<br> >>> >>> could make connection already in function dcc_loc= k_one()<br> >>> >>><br> >>> >>> I already tried to set DISTCC_DIR on a common nfs= share, but in case<br> >>> >>> you are triggering so many jobs, this started to = be bottle neck... I<br> >>> >>> won't tell about locking on nfs, and also sce= nario that somebody will<br> >>> >>> make a lock on nfs and machine will got crash - w= ill not work by<br> >>> >>> design :)<br> >>> >>><br> >>> >>> I know that scenario is not happening very often,= and it has more or<br> >>> >>> less picks characteristic, but we should be happy= that distcc cluster<br> >>> >>> is saturated and this case should be handled.<br> >>> >>><br> >>> >>> hope it's more clear now!<br> >>> >>> br<br> >>> >>> LT<br> >>> >>><br> >>> >>><br> >>> >>><br> >>> >>><br> >>> >>><br> >>> >>><br> >>> >>><br> >>> >>><br> >>> >>><br> >>> >>> =C5=81ukasz Tasz<br> >>> >>><br> >>> >>><br> >>> >>> 2014-10-16 1:39 GMT+02:00 Martin Pool <<a href= =3D"mailto:[email protected]">[email protected]</a>>:<br> >>> >>> > Can you try to explain more clearly what dif= ference in queueing behavior<br> >>> >>> > you<br> >>> >>> > expect from this change?<br> >>> >>> ><br> >>> >>> > I think probably the main change that's = needed is for the client to ask<br> >>> >>> > all<br> >>> >>> > masters if they have space, to avoid needing= to effectively poll by<br> >>> >>> > retrying, or getting stuck waiting for a par= ticular server.<br> >>> >>> ><br> >>> >>> > On Wed, Oct 15, 2014 at 12:53 PM, =C5=81ukas= z Tasz <<a href=3D"mailto:[email protected]">[email protected]</a>> wrote:<= br> >>> >>> >><br> >>> >>> >> Hi Guys,<br> >>> >>> >><br> >>> >>> >> please correct me if I'm wrong,<br> >>> >>> >> - currently distcc tries to connect serv= er 3 times, with small delay,<br> >>> >>> >> - server forks x childs and all of them = are trying to accept incoming<br> >>> >>> >> connection.<br> >>> >>> >> If server runs out of childs (all of the= m are busy), client will<br> >>> >>> >> fallback, and within next 60 sec will no= t try this machine.<br> >>> >>> >><br> >>> >>> >> What do you think about redesigning dist= cc in a way that master server<br> >>> >>> >> will always accept inconing connection, = fork a child, but in a same<br> >>> >>> >> time only x of them will be able to ente= r compilation<br> >>> >>> >> task(dcc_spawn_child)? (mayby preforking= still could be used?)<br> >>> >>> >><br> >>> >>> >> This may create kind of queue, client al= ways can decide by his own, if<br> >>> >>> >> can wait some=C2=A0 time, or maximum is = DISTCC_IO_TIMEOUT, but still it's<br> >>> >>> >> faster to wait, since probably on a clus= ter side it's just a pick of<br> >>> >>> >> saturation then making falback to local = machine.<br> >>> >>> >><br> >>> >>> >> currently I'm facing situation that = many jobs are making fallback, and<br> >>> >>> >> localmachine is being killed by make'= ;s -j calculated for distccd...<br> >>> >>> >><br> >>> >>> >> other trick maybe to pick different mach= ine, if current is busy, but<br> >>> >>> >> this may be much more complex in my opin= ion.<br> >>> >>> >><br> >>> >>> >> what do you think?<br> >>> >>> >> regards<br> >>> >>> >> =C5=81ukasz Tasz<br> >>> >>> >> __<br> >>> >>> >> distcc mailing list=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 <a href=3D"http://distcc.samba.org/">http://distcc.sam= ba.org/</a><br> >>> >>> >> To unsubscribe or change options:<br> >>> >>> >> <a href=3D"https://lists.samba.org/mailm= an/listinfo/distcc">https://lists.samba.org/mailman/listinfo/distcc</a><br> >>> >>> ><br> >>> >>> ><br> >>> >>> ><br> >>> >>> ><br> >>> >>> > --<br> >>> >>> > Martin<br> >>> __<br> >>> distcc mailing list=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <= a href=3D"http://distcc.samba.org/">http://distcc.samba.org/</a><br> >>> To unsubscribe or change options:<br> >>> <a href=3D"https://lists.samba.org/mailman/listinfo/distcc">ht= tps://lists.samba.org/mailman/listinfo/distcc</a></p> --089e0122efeced27580506dda2ea-- --===============4759822127168289635== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline __ distcc mailing list http://distcc.samba.org/ To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/distcc --===============4759822127168289635==--