Re: Broken Pipe when agents report to console
Joel Lucuik <[email protected]> Tue, 1 Nov 2016 15:38:21 -0400
| Newsgroups | gmane.comp.java.grinder.user |
|---|---|
| Message-ID | <CAA4qVTLiDMbyMZCr79r4Kv1rEOd8PQWZJWGMhtmZwhWdPHNEbg@mail.gmail.com> |
--===============0809589350501480077== Content-Type: multipart/alternative; boundary=94eb2c140a1495e74d054042772d --94eb2c140a1495e74d054042772d Content-Type: text/plain; charset=UTF-8 What is FirstHurdleSender? Did you create this? (Have Source?) Thanks. Joel On Tue, Nov 1, 2016 at 3:11 PM, Darren Ball <[email protected]> wrote: > Hi Joel, > > I've had a lot of issues with client sender timing out. > > Here is an approach (patch below) that I've taken. Adding a keepAlive to > sender as it is created in GrinderProcess > > Not sure if it will help, but it did for me in my case. I did this as it > was timing out way to early in my process and I was getting 'whilst' > communicating with console errors. > > Patch:: > > Index: grinder-core/src/main/java/net/grinder/engine/process/GrinderProcess.java > IDEA additional info: > Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP > <+>UTF-8 > =================================================================== > --- grinder-core/src/main/java/net/grinder/engine/process/GrinderProcess.java (revision ffb7bca73ea3190eaab12d1eccf73bf31464b58f) > +++ grinder-core/src/main/java/net/grinder/engine/process/GrinderProcess.java (revision ) > @@ -95,6 +95,9 @@ > import java.util.Map; > import java.util.Timer; > import java.util.TimerTask; > +import java.util.concurrent.Executors; > +import java.util.concurrent.ScheduledExecutorService; > +import java.util.concurrent.TimeUnit; > > > /** > @@ -141,6 +144,10 @@ > // Guarded by m_eventSynchronisation. > private String m_shutdownReason; > > + private ClientSender sender; > + > + private final ScheduledExecutorService m_executor = Executors.newSingleThreadScheduledExecutor(); > + > /** > * Creates a new {@code GrinderProcess} instance. > * > @@ -184,13 +191,34 @@ > > final BarrierGroups barrierGroups; > > + sender = ClientSender.connect( > + new ConnectorFactory(ConnectionType.WORKER).create(properties), > + new WorkerAddress(workerIdentity)); > + > + final Runnable keepAlive = new Runnable() { > + @Override > + public void run() { > + try { > + sender.sendKeepAlive(); > + } > + catch (CommunicationException e) { > + // Connection is dead, terminate task. > + throw new RuntimeException(e); > + } > + } > + }; > + > + m_executor.scheduleWithFixedDelay(keepAlive, > + 0, > + 1, > + TimeUnit.SECONDS); > + > + > if (m_initialisationMessage.getReportToConsole()) { > m_consoleSender = > new FirstHurdleSender( > - new QueuedSenderDecorator( > - ClientSender.connect( > - new ConnectorFactory(ConnectionType.WORKER).create(properties), > - new WorkerAddress(workerIdentity)))); > + new QueuedSenderDecorator(sender > + )); > > barrierGroups = > new ClientBarrierGroups(m_consoleSender, > > > On Tue, Nov 1, 2016 at 1:33 PM, Joel Lucuik <[email protected]> wrote: > >> Hi, >> >> Grinder version is 3-11. >> 4 vms, each VM has an agent, and the console shares a VM with the first >> agent. 10,0000 users (threads). >> >> Problem: We are occasionally seeing broken pipes when the agent(s) report >> anything to the console. >> >> My Investigation: >> >> I have been looking at the Grinder source. There is some code which traps >> CommunicationException such as >> >> AbstractSender.send() or >> ClientSender.blockingSend() >> >> but doesn't do anything with it. Such as attempt a reconnect if it's a >> Broken Pipe. >> >> Has this been fixed in newer versions? More specifically if the agent >> tries to send stats to the console, or a status update, and encounters a >> broken pipe, can it reconnect? >> >> Suggestions on which code/class/method to fix? >> >> >> We have very good unix people on our team. They have already upped the OS >> files, so we are (fairly) sure this is not the issue. >> >> Right now we are debating on a port to JMeter. >> >> Any advice would be appreciated. >> >> Joel >> >> >> >> ------------------------------------------------------------ >> ------------------ >> Developer Access Program for Intel Xeon Phi Processors >> Access to Intel Xeon Phi processor-based developer platforms. >> With one year of Intel Parallel Studio XE. >> Training and support from Colfax. >> Order your platform today. http://sdm.link/xeonphi >> _______________________________________________ >> grinder-use mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/grinder-use >> >> > > ------------------------------------------------------------ > ------------------ > Developer Access Program for Intel Xeon Phi Processors > Access to Intel Xeon Phi processor-based developer platforms. > With one year of Intel Parallel Studio XE. > Training and support from Colfax. > Order your platform today. http://sdm.link/xeonphi > _______________________________________________ > grinder-use mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/grinder-use > > --94eb2c140a1495e74d054042772d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div><div>What is FirstHurdleSender? Did you create this? = (Have Source?)<br><br></div>Thanks.<br></div>Joel<br><div><div><div class= =3D"gmail_extra"><br><div class=3D"gmail_quote">On Tue, Nov 1, 2016 at 3:11= PM, Darren Ball <span dir=3D"ltr"><<a href=3D"mailto:balldarrens@gmail.= com" target=3D"_blank">[email protected]</a>></span> wrote:<br><bloc= kquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #cc= c solid;padding-left:1ex"><div dir=3D"ltr">Hi Joel,<div><br></div><div>I= 9;ve had a lot of issues with client sender timing out.</div><div><br></div= ><div>Here is an approach (patch below) that I've taken.=C2=A0 Adding a= keepAlive to sender as it is created in GrinderProcess</div><div><br></div= ><div>Not sure if it will help, but it did for me in my case.=C2=A0 I did t= his as it was timing out way to early in my process and I was getting '= whilst' communicating with console errors.</div><div><br></div><div>Pat= ch::</div><div><br></div><div><pre style=3D"color:rgb(0,0,0);font-family:me= nlo;font-size:9pt">Index: grinder-core/src/main/java/<wbr>net/grinder/engin= e/process/<wbr>GrinderProcess.java<br>IDEA additional info:<br>Subsystem: c= om.intellij.openapi.diff.<wbr>impl.patch.CharsetEP<br><+>UTF-8<br>=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D<wbr>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<wbr>=3D=3D=3D=3D=3D=3D=3D<br>--- grinder-= core/src/main/java/<wbr>net/grinder/engine/process/<wbr>GrinderProcess.java= (revision ffb7bca73ea3190eaab12d1eccf73b<wbr>f31464b58f)<br>+++ grinder-c= ore/src/main/java/<wbr>net/grinder/engine/process/<wbr>GrinderProcess.java = (revision )<br>@@ -95,6 +95,9 @@<br> import java.util.Map;<br> import java= .util.Timer;<br> import java.util.TimerTask;<br>+import java.util.concurren= t.<wbr>Executors;<br>+import java.util.concurrent.<wbr>ScheduledExecutorSer= vice;<br>+import java.util.concurrent.TimeUnit;<br> <br> <br> /**<br>@@ -14= 1,6 +144,10 @@<br> // Guarded by m_eventSynchronisation.<br> private St= ring m_shutdownReason;<br> <br>+ private ClientSender sender;<br>+<br>+ = private final ScheduledExecutorService m_executor =3D Executors.<wbr>newSin= gleThreadScheduledExecut<wbr>or();<br>+<br> /**<br> * Creates a new {@= code GrinderProcess} instance.<br> *<br>@@ -184,13 +191,34 @@<br> <br> = final BarrierGroups barrierGroups;<br> <br>+ sender =3D ClientSender.= connect(<br>+ new ConnectorFactory(<wbr>ConnectionType.WORKER).c= reate(<wbr>properties),<br>+ new WorkerAddress(workerIdentity))<= wbr>;<br>+<br>+ final Runnable keepAlive =3D new Runnable() {<br>+ = @Override<br>+ public void run() {<br>+ try {<br>+ sen= der.sendKeepAlive();<br>+ }<br>+ catch (CommunicationExceptio= n e) {<br>+ // Connection is dead, terminate task.<br>+ t= hrow new RuntimeException(e);<br>+ }<br>+ }<br>+ };<br>+<br>= + m_executor.<wbr>scheduleWithFixedDelay(<wbr>keepAlive,<br>+ = 0,<br>+ 1,<br>+ TimeUnit.SECONDS);<br>+<br>+<br> = if (m_initialisationMessage.<wbr>getReportToConsole()) {<br> m_conso= leSender =3D<br> new FirstHurdleSender(<br>- new QueuedSen= derDecorator(<br>- ClientSender.connect(<br>- new C= onnectorFactory(<wbr>ConnectionType.WORKER).create(<wbr>properties),<br>- = new WorkerAddress(workerIdentity))<wbr>));<br>+ new Qu= euedSenderDecorator(sender<br>+ ));<br> <br> barrierGroups = =3D<br> new ClientBarrierGroups(m_<wbr>consoleSender,<br></pre></di= v><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Tue, Nov 1, = 2016 at 1:33 PM, Joel Lucuik <span dir=3D"ltr"><<a href=3D"mailto:joel.l= [email protected]" target=3D"_blank">[email protected]</a>></span> wro= te:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-= left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div><div><div><div>= Hi,<br><br>Grinder version is 3-11.<br>4 vms, each VM has an agent, and the= console shares a VM with the first agent. 10,0000 users (threads). <br><br= ></div>Problem: We are occasionally seeing broken pipes when the agent(s) r= eport anything to the console. <br></div></div><div><br>My Investigation:<b= r><br>I have been looking at the Grinder source. There is some code which t= raps CommunicationException such as <br><br>AbstractSender.send() or <br>Cl= ientSender.blockingSend()<br><br>but doesn't do anything with it. Such = as attempt a reconnect if it's a Broken Pipe.<br><br></div><div>Has thi= s been fixed in newer versions? More specifically if the agent tries to sen= d stats to the console, or a status update, and encounters a broken pipe, c= an it reconnect?<br><br></div><div>Suggestions on which code/class/method t= o fix?<br><br></div><div><br></div><div>We have very good unix people on ou= r team. They have already upped the OS files, so we are (fairly) sure this = is not the issue.<br><br></div><div>Right now we are debating on a port to = JMeter.<br><br></div><div>Any advice would be appreciated.<span class=3D"m_= -8631182522115003179HOEnZb"><font color=3D"#888888"><br></font></span></div= ><span class=3D"m_-8631182522115003179HOEnZb"><font color=3D"#888888"><div>= <br></div><div>Joel<br></div></font></span></div><br><br></div> <br>------------------------------<wbr>------------------------------<wbr>-= -----------------<br> Developer Access Program for Intel Xeon Phi Processors<br> Access to Intel Xeon Phi processor-based developer platforms.<br> With one year of Intel Parallel Studio XE.<br> Training and support from Colfax.<br> Order your platform today. <a href=3D"http://sdm.link/xeonphi" rel=3D"noref= errer" target=3D"_blank">http://sdm.link/xeonphi</a><br>___________________= ___________<wbr>_________________<br> grinder-use mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank">grin= [email protected]<wbr>net</a><br> <a href=3D"https://lists.sourceforge.net/lists/listinfo/grinder-use" rel=3D= "noreferrer" target=3D"_blank">https://lists.sourceforge.net/<wbr>lists/lis= tinfo/grinder-use</a><br> <br></blockquote></div><br></div></div> <br>------------------------------<wbr>------------------------------<wbr>-= -----------------<br> Developer Access Program for Intel Xeon Phi Processors<br> Access to Intel Xeon Phi processor-based developer platforms.<br> With one year of Intel Parallel Studio XE.<br> Training and support from Colfax.<br> Order your platform today. <a href=3D"http://sdm.link/xeonphi" rel=3D"noref= errer" target=3D"_blank">http://sdm.link/xeonphi</a><br>___________________= ___________<wbr>_________________<br> grinder-use mailing list<br> <a href=3D"mailto:[email protected]">[email protected]= ceforge.<wbr>net</a><br> <a href=3D"https://lists.sourceforge.net/lists/listinfo/grinder-use" rel=3D= "noreferrer" target=3D"_blank">https://lists.sourceforge.net/<wbr>lists/lis= tinfo/grinder-use</a><br> <br></blockquote></div><br></div></div></div></div> --94eb2c140a1495e74d054042772d-- --===============0809589350501480077== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------------ Developer Access Program for Intel Xeon Phi Processors Access to Intel Xeon Phi processor-based developer platforms. With one year of Intel Parallel Studio XE. Training and support from Colfax. Order your platform today. http://sdm.link/xeonphi --===============0809589350501480077== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ grinder-use mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/grinder-use --===============0809589350501480077==--