Re: Broken Pipe when agents report to console
Joel Lucuik <[email protected]> Tue, 1 Nov 2016 15:23:04 -0400
| Newsgroups | gmane.comp.java.grinder.user |
|---|---|
| Message-ID | <CAA4qVTLGqyKMJPu4DRFzw9i_axEr7RNh=f7s=iwsnVJpzrqAyg@mail.gmail.com> |
--===============7536165341987099786== Content-Type: multipart/alternative; boundary=001a1137a8baf512ef05404240b5 --001a1137a8baf512ef05404240b5 Content-Type: text/plain; charset=UTF-8 Thanks Darren. I am about to try 1. Adding connector and address to SocketWrapper, and adding another constructor to set them. 2. Adding method SocketWrapper reconnectToSocket() 3. Calling from ClientSender.blockingSend() Thinking about using this pattern in other places too. That said, I think your idea sounds like it's worth a try, especially considering it's already worked for you. And I am not even sure which methods to change!! Just kind of looking and hoping, and will do some logging to hopefully reproduce it. Joel On Tue, Nov 1, 2016 at 3:11 PM, Darren Ball <[email protected]> wrote: > Hi Joel, > > I've had a lot of issues with client sender timing out. > > Here is an approach (patch below) that I've taken. Adding a keepAlive to > sender as it is created in GrinderProcess > > Not sure if it will help, but it did for me in my case. I did this as it > was timing out way to early in my process and I was getting 'whilst' > communicating with console errors. > > Patch:: > > Index: grinder-core/src/main/java/net/grinder/engine/process/GrinderProcess.java > IDEA additional info: > Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP > <+>UTF-8 > =================================================================== > --- grinder-core/src/main/java/net/grinder/engine/process/GrinderProcess.java (revision ffb7bca73ea3190eaab12d1eccf73bf31464b58f) > +++ grinder-core/src/main/java/net/grinder/engine/process/GrinderProcess.java (revision ) > @@ -95,6 +95,9 @@ > import java.util.Map; > import java.util.Timer; > import java.util.TimerTask; > +import java.util.concurrent.Executors; > +import java.util.concurrent.ScheduledExecutorService; > +import java.util.concurrent.TimeUnit; > > > /** > @@ -141,6 +144,10 @@ > // Guarded by m_eventSynchronisation. > private String m_shutdownReason; > > + private ClientSender sender; > + > + private final ScheduledExecutorService m_executor = Executors.newSingleThreadScheduledExecutor(); > + > /** > * Creates a new {@code GrinderProcess} instance. > * > @@ -184,13 +191,34 @@ > > final BarrierGroups barrierGroups; > > + sender = ClientSender.connect( > + new ConnectorFactory(ConnectionType.WORKER).create(properties), > + new WorkerAddress(workerIdentity)); > + > + final Runnable keepAlive = new Runnable() { > + @Override > + public void run() { > + try { > + sender.sendKeepAlive(); > + } > + catch (CommunicationException e) { > + // Connection is dead, terminate task. > + throw new RuntimeException(e); > + } > + } > + }; > + > + m_executor.scheduleWithFixedDelay(keepAlive, > + 0, > + 1, > + TimeUnit.SECONDS); > + > + > if (m_initialisationMessage.getReportToConsole()) { > m_consoleSender = > new FirstHurdleSender( > - new QueuedSenderDecorator( > - ClientSender.connect( > - new ConnectorFactory(ConnectionType.WORKER).create(properties), > - new WorkerAddress(workerIdentity)))); > + new QueuedSenderDecorator(sender > + )); > > barrierGroups = > new ClientBarrierGroups(m_consoleSender, > > > On Tue, Nov 1, 2016 at 1:33 PM, Joel Lucuik <[email protected]> wrote: > >> Hi, >> >> Grinder version is 3-11. >> 4 vms, each VM has an agent, and the console shares a VM with the first >> agent. 10,0000 users (threads). >> >> Problem: We are occasionally seeing broken pipes when the agent(s) report >> anything to the console. >> >> My Investigation: >> >> I have been looking at the Grinder source. There is some code which traps >> CommunicationException such as >> >> AbstractSender.send() or >> ClientSender.blockingSend() >> >> but doesn't do anything with it. Such as attempt a reconnect if it's a >> Broken Pipe. >> >> Has this been fixed in newer versions? More specifically if the agent >> tries to send stats to the console, or a status update, and encounters a >> broken pipe, can it reconnect? >> >> Suggestions on which code/class/method to fix? >> >> >> We have very good unix people on our team. They have already upped the OS >> files, so we are (fairly) sure this is not the issue. >> >> Right now we are debating on a port to JMeter. >> >> Any advice would be appreciated. >> >> Joel >> >> >> >> ------------------------------------------------------------ >> ------------------ >> Developer Access Program for Intel Xeon Phi Processors >> Access to Intel Xeon Phi processor-based developer platforms. >> With one year of Intel Parallel Studio XE. >> Training and support from Colfax. >> Order your platform today. http://sdm.link/xeonphi >> _______________________________________________ >> grinder-use mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/grinder-use >> >> > > ------------------------------------------------------------ > ------------------ > Developer Access Program for Intel Xeon Phi Processors > Access to Intel Xeon Phi processor-based developer platforms. > With one year of Intel Parallel Studio XE. > Training and support from Colfax. > Order your platform today. http://sdm.link/xeonphi > _______________________________________________ > grinder-use mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/grinder-use > > --001a1137a8baf512ef05404240b5 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div><div><div><div><div><div><div>Thanks Darren.<br><br><= /div>I am about to try <br><br></div>1. Adding connector and address to Soc= ketWrapper, and adding another constructor to set them.<br></div>2. Adding = method SocketWrapper reconnectToSocket()<br></div>3. Calling from ClientSen= der.blockingSend()<br><br></div>Thinking about using this pattern in other = places too.<br><br></div>That said, I think your idea sounds like it's = worth a try, especially considering it's already worked for you. And I = am not even sure<br></div><div>which methods to change!! Just kind of looki= ng and hoping, and will do some logging to hopefully reproduce it.<br></div= ><div><br></div>Joel<br><div><div><div><br><div><div><br></div></div></div>= </div></div></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote"= >On Tue, Nov 1, 2016 at 3:11 PM, Darren Ball <span dir=3D"ltr"><<a href= =3D"mailto:[email protected]" target=3D"_blank">[email protected]</= a>></span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0= 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">Hi = Joel,<div><br></div><div>I've had a lot of issues with client sender ti= ming out.</div><div><br></div><div>Here is an approach (patch below) that I= 've taken.=C2=A0 Adding a keepAlive to sender as it is created in Grind= erProcess</div><div><br></div><div>Not sure if it will help, but it did for= me in my case.=C2=A0 I did this as it was timing out way to early in my pr= ocess and I was getting 'whilst' communicating with console errors.= </div><div><br></div><div>Patch::</div><div><br></div><div><pre style=3D"co= lor:rgb(0,0,0);font-family:menlo;font-size:9pt">Index: grinder-core/src/mai= n/java/<wbr>net/grinder/engine/process/<wbr>GrinderProcess.java<br>IDEA add= itional info:<br>Subsystem: com.intellij.openapi.diff.<wbr>impl.patch.Chars= etEP<br><+>UTF-8<br>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<wbr>=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<wbr>=3D=3D= =3D=3D=3D=3D=3D<br>--- grinder-core/src/main/java/<wbr>net/grinder/engine/p= rocess/<wbr>GrinderProcess.java (revision ffb7bca73ea3190eaab12d1eccf73b<w= br>f31464b58f)<br>+++ grinder-core/src/main/java/<wbr>net/grinder/engine/pr= ocess/<wbr>GrinderProcess.java (revision )<br>@@ -95,6 +95,9 @@<br> import= java.util.Map;<br> import java.util.Timer;<br> import java.util.TimerTask;= <br>+import java.util.concurrent.<wbr>Executors;<br>+import java.util.concu= rrent.<wbr>ScheduledExecutorService;<br>+import java.util.concurrent.TimeUn= it;<br> <br> <br> /**<br>@@ -141,6 +144,10 @@<br> // Guarded by m_eventSy= nchronisation.<br> private String m_shutdownReason;<br> <br>+ private C= lientSender sender;<br>+<br>+ private final ScheduledExecutorService m_exe= cutor =3D Executors.<wbr>newSingleThreadScheduledExecut<wbr>or();<br>+<br> = /**<br> * Creates a new {@code GrinderProcess} instance.<br> *<br>@= @ -184,13 +191,34 @@<br> <br> final BarrierGroups barrierGroups;<br> <b= r>+ sender =3D ClientSender.connect(<br>+ new ConnectorFactor= y(<wbr>ConnectionType.WORKER).create(<wbr>properties),<br>+ new = WorkerAddress(workerIdentity))<wbr>;<br>+<br>+ final Runnable keepAlive = =3D new Runnable() {<br>+ @Override<br>+ public void run() {<br>+= try {<br>+ sender.sendKeepAlive();<br>+ }<br>+ = catch (CommunicationException e) {<br>+ // Connection is dead, = terminate task.<br>+ throw new RuntimeException(e);<br>+ }<= br>+ }<br>+ };<br>+<br>+ m_executor.<wbr>scheduleWithFixedDelay(= <wbr>keepAlive,<br>+ 0,<br>+ 1,<br>+ TimeU= nit.SECONDS);<br>+<br>+<br> if (m_initialisationMessage.<wbr>getReportT= oConsole()) {<br> m_consoleSender =3D<br> new FirstHurdleSend= er(<br>- new QueuedSenderDecorator(<br>- ClientSender.c= onnect(<br>- new ConnectorFactory(<wbr>ConnectionType.WORKER).= create(<wbr>properties),<br>- new WorkerAddress(workerIdentity= ))<wbr>));<br>+ new QueuedSenderDecorator(sender<br>+ ));= <br> <br> barrierGroups =3D<br> new ClientBarrierGroups(m_<wb= r>consoleSender,<br></pre></div><div class=3D"gmail_extra"><br><div class= =3D"gmail_quote">On Tue, Nov 1, 2016 at 1:33 PM, Joel Lucuik <span dir=3D"l= tr"><<a href=3D"mailto:[email protected]" target=3D"_blank">joel.luc= [email protected]</a>></span> wrote:<br><blockquote class=3D"gmail_quote" st= yle=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div = dir=3D"ltr"><div><div><div><div>Hi,<br><br>Grinder version is 3-11.<br>4 vm= s, each VM has an agent, and the console shares a VM with the first agent. = 10,0000 users (threads). <br><br></div>Problem: We are occasionally seeing = broken pipes when the agent(s) report anything to the console. <br></div></= div><div><br>My Investigation:<br><br>I have been looking at the Grinder so= urce. There is some code which traps CommunicationException such as <br><br= >AbstractSender.send() or <br>ClientSender.blockingSend()<br><br>but doesn&= #39;t do anything with it. Such as attempt a reconnect if it's a Broken= Pipe.<br><br></div><div>Has this been fixed in newer versions? More specif= ically if the agent tries to send stats to the console, or a status update,= and encounters a broken pipe, can it reconnect?<br><br></div><div>Suggesti= ons on which code/class/method to fix?<br><br></div><div><br></div><div>We = have very good unix people on our team. They have already upped the OS file= s, so we are (fairly) sure this is not the issue.<br><br></div><div>Right n= ow we are debating on a port to JMeter.<br><br></div><div>Any advice would = be appreciated.<span class=3D"m_-1466682822665794981HOEnZb"><font color=3D"= #888888"><br></font></span></div><span class=3D"m_-1466682822665794981HOEnZ= b"><font color=3D"#888888"><div><br></div><div>Joel<br></div></font></span>= </div><br><br></div> <br>------------------------------<wbr>------------------------------<wbr>-= -----------------<br> Developer Access Program for Intel Xeon Phi Processors<br> Access to Intel Xeon Phi processor-based developer platforms.<br> With one year of Intel Parallel Studio XE.<br> Training and support from Colfax.<br> Order your platform today. <a href=3D"http://sdm.link/xeonphi" rel=3D"noref= errer" target=3D"_blank">http://sdm.link/xeonphi</a><br>___________________= ___________<wbr>_________________<br> grinder-use mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank">grin= [email protected]<wbr>net</a><br> <a href=3D"https://lists.sourceforge.net/lists/listinfo/grinder-use" rel=3D= "noreferrer" target=3D"_blank">https://lists.sourceforge.net/<wbr>lists/lis= tinfo/grinder-use</a><br> <br></blockquote></div><br></div></div> <br>------------------------------<wbr>------------------------------<wbr>-= -----------------<br> Developer Access Program for Intel Xeon Phi Processors<br> Access to Intel Xeon Phi processor-based developer platforms.<br> With one year of Intel Parallel Studio XE.<br> Training and support from Colfax.<br> Order your platform today. <a href=3D"http://sdm.link/xeonphi" rel=3D"noref= errer" target=3D"_blank">http://sdm.link/xeonphi</a><br>___________________= ___________<wbr>_________________<br> grinder-use mailing list<br> <a href=3D"mailto:[email protected]">[email protected]= ceforge.<wbr>net</a><br> <a href=3D"https://lists.sourceforge.net/lists/listinfo/grinder-use" rel=3D= "noreferrer" target=3D"_blank">https://lists.sourceforge.net/<wbr>lists/lis= tinfo/grinder-use</a><br> <br></blockquote></div><br></div> --001a1137a8baf512ef05404240b5-- --===============7536165341987099786== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------------ Developer Access Program for Intel Xeon Phi Processors Access to Intel Xeon Phi processor-based developer platforms. With one year of Intel Parallel Studio XE. Training and support from Colfax. Order your platform today. http://sdm.link/xeonphi --===============7536165341987099786== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ grinder-use mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/grinder-use --===============7536165341987099786==--