Re: Broken Pipe when agents report to console
olivier merlin <[email protected]> Tue, 01 Nov 2016 19:43:18 +0000
| Newsgroups | gmane.comp.java.grinder.user |
|---|---|
| Message-ID | <CAFjxuZM1HKVV2em3F2G+CF4HPk0q3v7H8+UFRe5z5hio4Ve84A@mail.gmail.com> |
--===============2932796684269248455== Content-Type: multipart/alternative; boundary=f403045ea63adddd0505404289b0 --f403045ea63adddd0505404289b0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hello, We have got the Broken pipe at startup because we have a lot of data to initialized. The console close the connections after 30 second if there is no activity coming from the worker only before the first run. We have so recompiled the code replacing the default value of 30 seconds by 5 minutes. If you have the problem sometimes at startup you have this issue. Cheers Olivier Le mar. 1 nov. 2016 20:26, Joel Lucuik <[email protected]> a =C3=A9crit= : > Thanks Darren. > > I am about to try > > 1. Adding connector and address to SocketWrapper, and adding another > constructor to set them. > 2. Adding method SocketWrapper reconnectToSocket() > 3. Calling from ClientSender.blockingSend() > > Thinking about using this pattern in other places too. > > That said, I think your idea sounds like it's worth a try, especially > considering it's already worked for you. And I am not even sure > which methods to change!! Just kind of looking and hoping, and will do > some logging to hopefully reproduce it. > > Joel > > > > On Tue, Nov 1, 2016 at 3:11 PM, Darren Ball <[email protected]> wrote= : > > Hi Joel, > > I've had a lot of issues with client sender timing out. > > Here is an approach (patch below) that I've taken. Adding a keepAlive to > sender as it is created in GrinderProcess > > Not sure if it will help, but it did for me in my case. I did this as it > was timing out way to early in my process and I was getting 'whilst' > communicating with console errors. > > Patch:: > > Index: grinder-core/src/main/java/net/grinder/engine/process/GrinderProce= ss.java > IDEA additional info: > Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP > <+>UTF-8 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- grinder-core/src/main/java/net/grinder/engine/process/GrinderProcess.= java (revision ffb7bca73ea3190eaab12d1eccf73bf31464b58f) > +++ grinder-core/src/main/java/net/grinder/engine/process/GrinderProcess.= java (revision ) > @@ -95,6 +95,9 @@ > import java.util.Map; > import java.util.Timer; > import java.util.TimerTask; > +import java.util.concurrent.Executors; > +import java.util.concurrent.ScheduledExecutorService; > +import java.util.concurrent.TimeUnit; > > > /** > @@ -141,6 +144,10 @@ > // Guarded by m_eventSynchronisation. > private String m_shutdownReason; > > + private ClientSender sender; > + > + private final ScheduledExecutorService m_executor =3D Executors.newSin= gleThreadScheduledExecutor(); > + > /** > * Creates a new {@code GrinderProcess} instance. > * > @@ -184,13 +191,34 @@ > > final BarrierGroups barrierGroups; > > + sender =3D ClientSender.connect( > + new ConnectorFactory(ConnectionType.WORKER).create(propertie= s), > + new WorkerAddress(workerIdentity)); > + > + final Runnable keepAlive =3D new Runnable() { > + @Override > + public void run() { > + try { > + sender.sendKeepAlive(); > + } > + catch (CommunicationException e) { > + // Connection is dead, terminate task. > + throw new RuntimeException(e); > + } > + } > + }; > + > + m_executor.scheduleWithFixedDelay(keepAlive, > + 0, > + 1, > + TimeUnit.SECONDS); > + > + > if (m_initialisationMessage.getReportToConsole()) { > m_consoleSender =3D > new FirstHurdleSender( > - new QueuedSenderDecorator( > - ClientSender.connect( > - new ConnectorFactory(ConnectionType.WORKER).create(propert= ies), > - new WorkerAddress(workerIdentity)))); > + new QueuedSenderDecorator(sender > + )); > > barrierGroups =3D > new ClientBarrierGroups(m_consoleSender, > > > On Tue, Nov 1, 2016 at 1:33 PM, Joel Lucuik <[email protected]> wrote= : > > Hi, > > Grinder version is 3-11. > 4 vms, each VM has an agent, and the console shares a VM with the first > agent. 10,0000 users (threads). > > Problem: We are occasionally seeing broken pipes when the agent(s) report > anything to the console. > > My Investigation: > > I have been looking at the Grinder source. There is some code which traps > CommunicationException such as > > AbstractSender.send() or > ClientSender.blockingSend() > > but doesn't do anything with it. Such as attempt a reconnect if it's a > Broken Pipe. > > Has this been fixed in newer versions? More specifically if the agent > tries to send stats to the console, or a status update, and encounters a > broken pipe, can it reconnect? > > Suggestions on which code/class/method to fix? > > > We have very good unix people on our team. They have already upped the OS > files, so we are (fairly) sure this is not the issue. > > Right now we are debating on a port to JMeter. > > Any advice would be appreciated. > > Joel > > > > > -------------------------------------------------------------------------= ----- > Developer Access Program for Intel Xeon Phi Processors > Access to Intel Xeon Phi processor-based developer platforms. > With one year of Intel Parallel Studio XE. > Training and support from Colfax. > Order your platform today. http://sdm.link/xeonphi > _______________________________________________ > grinder-use mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/grinder-use > > > > > -------------------------------------------------------------------------= ----- > Developer Access Program for Intel Xeon Phi Processors > Access to Intel Xeon Phi processor-based developer platforms. > With one year of Intel Parallel Studio XE. > Training and support from Colfax. > Order your platform today. http://sdm.link/xeonphi > _______________________________________________ > grinder-use mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/grinder-use > > > > -------------------------------------------------------------------------= ----- > Developer Access Program for Intel Xeon Phi Processors > Access to Intel Xeon Phi processor-based developer platforms. > With one year of Intel Parallel Studio XE. > Training and support from Colfax. > Order your platform today. http://sdm.link/xeonphi > _______________________________________________ > grinder-use mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/grinder-use > --f403045ea63adddd0505404289b0 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <p dir=3D"ltr">Hello,<br> We have got the Broken pipe at startup because we have a lot of data to ini= tialized.<br> The console close the connections after 30 second if there is no activity c= oming from the worker only before the first run. We have so recompiled the = code replacing the default=C2=A0 value of 30 seconds by 5 minutes.<br> If you have the problem sometimes at startup you have this issue.<br> Cheers<br> Olivier<br> </p> <br><div class=3D"gmail_quote"><div dir=3D"ltr">Le mar. 1 nov. 2016 20:26, = Joel Lucuik <<a href=3D"mailto:[email protected]">joel.lucuik@gmail.= com</a>> a =C3=A9crit=C2=A0:<br></div><blockquote class=3D"gmail_quote" = style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><di= v dir=3D"ltr" class=3D"gmail_msg"><div class=3D"gmail_msg"><div class=3D"gm= ail_msg"><div class=3D"gmail_msg"><div class=3D"gmail_msg"><div class=3D"gm= ail_msg"><div class=3D"gmail_msg"><div class=3D"gmail_msg">Thanks Darren.<b= r class=3D"gmail_msg"><br class=3D"gmail_msg"></div>I am about to try <br c= lass=3D"gmail_msg"><br class=3D"gmail_msg"></div>1. Adding connector and ad= dress to SocketWrapper, and adding another constructor to set them.<br clas= s=3D"gmail_msg"></div>2. Adding method SocketWrapper reconnectToSocket()<br= class=3D"gmail_msg"></div>3. Calling from ClientSender.blockingSend()<br c= lass=3D"gmail_msg"><br class=3D"gmail_msg"></div>Thinking about using this = pattern in other places too.<br class=3D"gmail_msg"><br class=3D"gmail_msg"= ></div>That said, I think your idea sounds like it's worth a try, espec= ially considering it's already worked for you. And I am not even sure<b= r class=3D"gmail_msg"></div><div class=3D"gmail_msg">which methods to chang= e!! Just kind of looking and hoping, and will do some logging to hopefully = reproduce it.<br class=3D"gmail_msg"></div></div><div dir=3D"ltr" class=3D"= gmail_msg"><div class=3D"gmail_msg"><br class=3D"gmail_msg"></div>Joel<br c= lass=3D"gmail_msg"><div class=3D"gmail_msg"><div class=3D"gmail_msg"><div c= lass=3D"gmail_msg"><br class=3D"gmail_msg"><div class=3D"gmail_msg"><div cl= ass=3D"gmail_msg"><br class=3D"gmail_msg"></div></div></div></div></div></d= iv><div class=3D"gmail_extra gmail_msg"><br class=3D"gmail_msg"><div class= =3D"gmail_quote gmail_msg">On Tue, Nov 1, 2016 at 3:11 PM, Darren Ball <spa= n dir=3D"ltr" class=3D"gmail_msg"><<a href=3D"mailto:[email protected]= om" class=3D"gmail_msg" target=3D"_blank">[email protected]</a>></sp= an> wrote:<br class=3D"gmail_msg"><blockquote class=3D"gmail_quote gmail_ms= g" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">= <div dir=3D"ltr" class=3D"gmail_msg">Hi Joel,<div class=3D"gmail_msg"><br c= lass=3D"gmail_msg"></div><div class=3D"gmail_msg">I've had a lot of iss= ues with client sender timing out.</div><div class=3D"gmail_msg"><br class= =3D"gmail_msg"></div><div class=3D"gmail_msg">Here is an approach (patch be= low) that I've taken.=C2=A0 Adding a keepAlive to sender as it is creat= ed in GrinderProcess</div><div class=3D"gmail_msg"><br class=3D"gmail_msg">= </div><div class=3D"gmail_msg">Not sure if it will help, but it did for me = in my case.=C2=A0 I did this as it was timing out way to early in my proces= s and I was getting 'whilst' communicating with console errors.</di= v><div class=3D"gmail_msg"><br class=3D"gmail_msg"></div><div class=3D"gmai= l_msg">Patch::</div><div class=3D"gmail_msg"><br class=3D"gmail_msg"></div>= <div class=3D"gmail_msg"><pre style=3D"color:rgb(0,0,0);font-family:menlo;f= ont-size:9pt" class=3D"gmail_msg">Index: grinder-core/src/main/java/net/gri= nder/engine/process/GrinderProcess.java<br class=3D"gmail_msg">IDEA additio= nal info:<br class=3D"gmail_msg">Subsystem: com.intellij.openapi.diff.impl.= patch.CharsetEP<br class=3D"gmail_msg"><+>UTF-8<br class=3D"gmail_msg= ">=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br class=3D"gmail_ms= g">--- grinder-core/src/main/java/net/grinder/engine/process/GrinderProcess= .java (revision ffb7bca73ea3190eaab12d1eccf73bf31464b58f)<br class=3D"gmai= l_msg">+++ grinder-core/src/main/java/net/grinder/engine/process/GrinderPro= cess.java (revision )<br class=3D"gmail_msg">@@ -95,6 +95,9 @@<br class=3D= "gmail_msg"> import java.util.Map;<br class=3D"gmail_msg"> import java.util= .Timer;<br class=3D"gmail_msg"> import java.util.TimerTask;<br class=3D"gma= il_msg">+import java.util.concurrent.Executors;<br class=3D"gmail_msg">+imp= ort java.util.concurrent.ScheduledExecutorService;<br class=3D"gmail_msg">+= import java.util.concurrent.TimeUnit;<br class=3D"gmail_msg"> <br class=3D"= gmail_msg"> <br class=3D"gmail_msg"> /**<br class=3D"gmail_msg">@@ -141,6 += 144,10 @@<br class=3D"gmail_msg"> // Guarded by m_eventSynchronisation.<b= r class=3D"gmail_msg"> private String m_shutdownReason;<br class=3D"gmail= _msg"> <br class=3D"gmail_msg">+ private ClientSender sender;<br class=3D= "gmail_msg">+<br class=3D"gmail_msg">+ private final ScheduledExecutorServ= ice m_executor =3D Executors.newSingleThreadScheduledExecutor();<br class= =3D"gmail_msg">+<br class=3D"gmail_msg"> /**<br class=3D"gmail_msg"> *= Creates a new {@code GrinderProcess} instance.<br class=3D"gmail_msg"> = *<br class=3D"gmail_msg">@@ -184,13 +191,34 @@<br class=3D"gmail_msg"> <br = class=3D"gmail_msg"> final BarrierGroups barrierGroups;<br class=3D"gma= il_msg"> <br class=3D"gmail_msg">+ sender =3D ClientSender.connect(<br c= lass=3D"gmail_msg">+ new ConnectorFactory(ConnectionType.WORKER)= .create(properties),<br class=3D"gmail_msg">+ new WorkerAddress(= workerIdentity));<br class=3D"gmail_msg">+<br class=3D"gmail_msg">+ fina= l Runnable keepAlive =3D new Runnable() {<br class=3D"gmail_msg">+ @Ov= erride<br class=3D"gmail_msg">+ public void run() {<br class=3D"gmail_= msg">+ try {<br class=3D"gmail_msg">+ sender.sendKeepAlive(= );<br class=3D"gmail_msg">+ }<br class=3D"gmail_msg">+ catch = (CommunicationException e) {<br class=3D"gmail_msg">+ // Connectio= n is dead, terminate task.<br class=3D"gmail_msg">+ throw new Runt= imeException(e);<br class=3D"gmail_msg">+ }<br class=3D"gmail_msg">+= }<br class=3D"gmail_msg">+ };<br class=3D"gmail_msg">+<br class=3D= "gmail_msg">+ m_executor.scheduleWithFixedDelay(keepAlive,<br class=3D"g= mail_msg">+ 0,<br class=3D"gmail_msg">+ 1,<br class= =3D"gmail_msg">+ TimeUnit.SECONDS);<br class=3D"gmail_msg">+<br = class=3D"gmail_msg">+<br class=3D"gmail_msg"> if (m_initialisationMessa= ge.getReportToConsole()) {<br class=3D"gmail_msg"> m_consoleSender = =3D<br class=3D"gmail_msg"> new FirstHurdleSender(<br class=3D"gmai= l_msg">- new QueuedSenderDecorator(<br class=3D"gmail_msg">- = ClientSender.connect(<br class=3D"gmail_msg">- new Conne= ctorFactory(ConnectionType.WORKER).create(properties),<br class=3D"gmail_ms= g">- new WorkerAddress(workerIdentity))));<br class=3D"gmail_m= sg">+ new QueuedSenderDecorator(sender<br class=3D"gmail_msg">+ = ));<br class=3D"gmail_msg"> <br class=3D"gmail_msg"> barrierGr= oups =3D<br class=3D"gmail_msg"> new ClientBarrierGroups(m_consoleS= ender,<br class=3D"gmail_msg"></pre></div><div class=3D"gmail_extra gmail_m= sg"><br class=3D"gmail_msg"><div class=3D"gmail_quote gmail_msg">On Tue, No= v 1, 2016 at 1:33 PM, Joel Lucuik <span dir=3D"ltr" class=3D"gmail_msg"><= ;<a href=3D"mailto:[email protected]" class=3D"gmail_msg" target=3D"_bl= ank">[email protected]</a>></span> wrote:<br class=3D"gmail_msg"><bl= ockquote class=3D"gmail_quote gmail_msg" style=3D"margin:0 0 0 .8ex;border-= left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr" class=3D"gmail_msg">= <div class=3D"gmail_msg"><div class=3D"gmail_msg"><div class=3D"gmail_msg">= <div class=3D"gmail_msg">Hi,<br class=3D"gmail_msg"><br class=3D"gmail_msg"= >Grinder version is 3-11.<br class=3D"gmail_msg">4 vms, each VM has an agen= t, and the console shares a VM with the first agent. 10,0000 users (threads= ). <br class=3D"gmail_msg"><br class=3D"gmail_msg"></div>Problem: We are oc= casionally seeing broken pipes when the agent(s) report anything to the con= sole. <br class=3D"gmail_msg"></div></div><div class=3D"gmail_msg"><br clas= s=3D"gmail_msg">My Investigation:<br class=3D"gmail_msg"><br class=3D"gmail= _msg">I have been looking at the Grinder source. There is some code which t= raps CommunicationException such as <br class=3D"gmail_msg"><br class=3D"gm= ail_msg">AbstractSender.send() or <br class=3D"gmail_msg">ClientSender.bloc= kingSend()<br class=3D"gmail_msg"><br class=3D"gmail_msg">but doesn't d= o anything with it. Such as attempt a reconnect if it's a Broken Pipe.<= br class=3D"gmail_msg"><br class=3D"gmail_msg"></div><div class=3D"gmail_ms= g">Has this been fixed in newer versions? More specifically if the agent tr= ies to send stats to the console, or a status update, and encounters a brok= en pipe, can it reconnect?<br class=3D"gmail_msg"><br class=3D"gmail_msg"><= /div><div class=3D"gmail_msg">Suggestions on which code/class/method to fix= ?<br class=3D"gmail_msg"><br class=3D"gmail_msg"></div><div class=3D"gmail_= msg"><br class=3D"gmail_msg"></div><div class=3D"gmail_msg">We have very go= od unix people on our team. They have already upped the OS files, so we are= (fairly) sure this is not the issue.<br class=3D"gmail_msg"><br class=3D"g= mail_msg"></div><div class=3D"gmail_msg">Right now we are debating on a por= t to JMeter.<br class=3D"gmail_msg"><br class=3D"gmail_msg"></div><div clas= s=3D"gmail_msg">Any advice would be appreciated.<span class=3D"m_1140134821= 490960840m_-1466682822665794981HOEnZb gmail_msg"><font color=3D"#888888" cl= ass=3D"gmail_msg"><br class=3D"gmail_msg"></font></span></div><span class= =3D"m_1140134821490960840m_-1466682822665794981HOEnZb gmail_msg"><font colo= r=3D"#888888" class=3D"gmail_msg"><div class=3D"gmail_msg"><br class=3D"gma= il_msg"></div><div class=3D"gmail_msg">Joel<br class=3D"gmail_msg"></div></= font></span></div><br class=3D"gmail_msg"><br class=3D"gmail_msg"></div> <br class=3D"gmail_msg">---------------------------------------------------= ---------------------------<br class=3D"gmail_msg"> Developer Access Program for Intel Xeon Phi Processors<br class=3D"gmail_ms= g"> Access to Intel Xeon Phi processor-based developer platforms.<br class=3D"g= mail_msg"> With one year of Intel Parallel Studio XE.<br class=3D"gmail_msg"> Training and support from Colfax.<br class=3D"gmail_msg"> Order your platform today. <a href=3D"http://sdm.link/xeonphi" rel=3D"noref= errer" class=3D"gmail_msg" target=3D"_blank">http://sdm.link/xeonphi</a><br= class=3D"gmail_msg">_______________________________________________<br cla= ss=3D"gmail_msg"> grinder-use mailing list<br class=3D"gmail_msg"> <a href=3D"mailto:[email protected]" class=3D"gmail_msg" ta= rget=3D"_blank">[email protected]</a><br class=3D"gmail_msg= "> <a href=3D"https://lists.sourceforge.net/lists/listinfo/grinder-use" rel=3D= "noreferrer" class=3D"gmail_msg" target=3D"_blank">https://lists.sourceforg= e.net/lists/listinfo/grinder-use</a><br class=3D"gmail_msg"> <br class=3D"gmail_msg"></blockquote></div><br class=3D"gmail_msg"></div></= div> <br class=3D"gmail_msg">---------------------------------------------------= ---------------------------<br class=3D"gmail_msg"> Developer Access Program for Intel Xeon Phi Processors<br class=3D"gmail_ms= g"> Access to Intel Xeon Phi processor-based developer platforms.<br class=3D"g= mail_msg"> With one year of Intel Parallel Studio XE.<br class=3D"gmail_msg"> Training and support from Colfax.<br class=3D"gmail_msg"> Order your platform today. <a href=3D"http://sdm.link/xeonphi" rel=3D"noref= errer" class=3D"gmail_msg" target=3D"_blank">http://sdm.link/xeonphi</a><br= class=3D"gmail_msg">_______________________________________________<br cla= ss=3D"gmail_msg"> grinder-use mailing list<br class=3D"gmail_msg"> <a href=3D"mailto:[email protected]" class=3D"gmail_msg" ta= rget=3D"_blank">[email protected]</a><br class=3D"gmail_msg= "> <a href=3D"https://lists.sourceforge.net/lists/listinfo/grinder-use" rel=3D= "noreferrer" class=3D"gmail_msg" target=3D"_blank">https://lists.sourceforg= e.net/lists/listinfo/grinder-use</a><br class=3D"gmail_msg"> <br class=3D"gmail_msg"></blockquote></div><br class=3D"gmail_msg"></div> ---------------------------------------------------------------------------= ---<br class=3D"gmail_msg"> Developer Access Program for Intel Xeon Phi Processors<br class=3D"gmail_ms= g"> Access to Intel Xeon Phi processor-based developer platforms.<br class=3D"g= mail_msg"> With one year of Intel Parallel Studio XE.<br class=3D"gmail_msg"> Training and support from Colfax.<br class=3D"gmail_msg"> Order your platform today. <a href=3D"http://sdm.link/xeonphi" rel=3D"noref= errer" class=3D"gmail_msg" target=3D"_blank">http://sdm.link/xeonphi</a>___= ____________________________________________<br class=3D"gmail_msg"> grinder-use mailing list<br class=3D"gmail_msg"> <a href=3D"mailto:[email protected]" class=3D"gmail_msg" ta= rget=3D"_blank">[email protected]</a><br class=3D"gmail_msg= "> <a href=3D"https://lists.sourceforge.net/lists/listinfo/grinder-use" rel=3D= "noreferrer" class=3D"gmail_msg" target=3D"_blank">https://lists.sourceforg= e.net/lists/listinfo/grinder-use</a><br class=3D"gmail_msg"> </blockquote></div> --f403045ea63adddd0505404289b0-- --===============2932796684269248455== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------------ Developer Access Program for Intel Xeon Phi Processors Access to Intel Xeon Phi processor-based developer platforms. With one year of Intel Parallel Studio XE. Training and support from Colfax. Order your platform today. http://sdm.link/xeonphi --===============2932796684269248455== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ grinder-use mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/grinder-use --===============2932796684269248455==--