Re: Broken Pipe when agents report to console
Darren Ball <[email protected]> Tue, 1 Nov 2016 15:11:54 -0400
| Newsgroups | gmane.comp.java.grinder.user |
|---|---|
| Message-ID | <CA+YPNhfzccd-Q1aOS2KRWyaSa_zaROYUr0V9-ZfEcjbv_K4=nQ@mail.gmail.com> |
--===============1636039117841590564==
Content-Type: multipart/alternative; boundary=001a114ac51a2f2ec00540421a7d
--001a114ac51a2f2ec00540421a7d
Content-Type: text/plain; charset=UTF-8
Hi Joel,
I've had a lot of issues with client sender timing out.
Here is an approach (patch below) that I've taken. Adding a keepAlive to
sender as it is created in GrinderProcess
Not sure if it will help, but it did for me in my case. I did this as it
was timing out way to early in my process and I was getting 'whilst'
communicating with console errors.
Patch::
Index: grinder-core/src/main/java/net/grinder/engine/process/GrinderProcess.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- grinder-core/src/main/java/net/grinder/engine/process/GrinderProcess.java
(revision ffb7bca73ea3190eaab12d1eccf73bf31464b58f)
+++ grinder-core/src/main/java/net/grinder/engine/process/GrinderProcess.java
(revision )
@@ -95,6 +95,9 @@
import java.util.Map;
import java.util.Timer;
import java.util.TimerTask;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
/**
@@ -141,6 +144,10 @@
// Guarded by m_eventSynchronisation.
private String m_shutdownReason;
+ private ClientSender sender;
+
+ private final ScheduledExecutorService m_executor =
Executors.newSingleThreadScheduledExecutor();
+
/**
* Creates a new {@code GrinderProcess} instance.
*
@@ -184,13 +191,34 @@
final BarrierGroups barrierGroups;
+ sender = ClientSender.connect(
+ new ConnectorFactory(ConnectionType.WORKER).create(properties),
+ new WorkerAddress(workerIdentity));
+
+ final Runnable keepAlive = new Runnable() {
+ @Override
+ public void run() {
+ try {
+ sender.sendKeepAlive();
+ }
+ catch (CommunicationException e) {
+ // Connection is dead, terminate task.
+ throw new RuntimeException(e);
+ }
+ }
+ };
+
+ m_executor.scheduleWithFixedDelay(keepAlive,
+ 0,
+ 1,
+ TimeUnit.SECONDS);
+
+
if (m_initialisationMessage.getReportToConsole()) {
m_consoleSender =
new FirstHurdleSender(
- new QueuedSenderDecorator(
- ClientSender.connect(
- new ConnectorFactory(ConnectionType.WORKER).create(properties),
- new WorkerAddress(workerIdentity))));
+ new QueuedSenderDecorator(sender
+ ));
barrierGroups =
new ClientBarrierGroups(m_consoleSender,
On Tue, Nov 1, 2016 at 1:33 PM, Joel Lucuik <[email protected]> wrote:
> Hi,
>
> Grinder version is 3-11.
> 4 vms, each VM has an agent, and the console shares a VM with the first
> agent. 10,0000 users (threads).
>
> Problem: We are occasionally seeing broken pipes when the agent(s) report
> anything to the console.
>
> My Investigation:
>
> I have been looking at the Grinder source. There is some code which traps
> CommunicationException such as
>
> AbstractSender.send() or
> ClientSender.blockingSend()
>
> but doesn't do anything with it. Such as attempt a reconnect if it's a
> Broken Pipe.
>
> Has this been fixed in newer versions? More specifically if the agent
> tries to send stats to the console, or a status update, and encounters a
> broken pipe, can it reconnect?
>
> Suggestions on which code/class/method to fix?
>
>
> We have very good unix people on our team. They have already upped the OS
> files, so we are (fairly) sure this is not the issue.
>
> Right now we are debating on a port to JMeter.
>
> Any advice would be appreciated.
>
> Joel
>
>
>
> ------------------------------------------------------------
> ------------------
> Developer Access Program for Intel Xeon Phi Processors
> Access to Intel Xeon Phi processor-based developer platforms.
> With one year of Intel Parallel Studio XE.
> Training and support from Colfax.
> Order your platform today. http://sdm.link/xeonphi
> _______________________________________________
> grinder-use mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/grinder-use
>
>
--001a114ac51a2f2ec00540421a7d
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">Hi Joel,<div><br></div><div>I've had a lot of issues w=
ith client sender timing out.</div><div><br></div><div>Here is an approach =
(patch below) that I've taken.=C2=A0 Adding a keepAlive to sender as it=
is created in GrinderProcess</div><div><br></div><div>Not sure if it will =
help, but it did for me in my case.=C2=A0 I did this as it was timing out w=
ay to early in my process and I was getting 'whilst' communicating =
with console errors.</div><div><br></div><div>Patch::</div><div><br></div><=
div><pre style=3D"color:rgb(0,0,0);font-family:menlo;font-size:9pt">Index: =
grinder-core/src/main/java/net/grinder/engine/process/GrinderProcess.java<b=
r>IDEA additional info:<br>Subsystem: com.intellij.openapi.diff.impl.patch.=
CharsetEP<br><+>UTF-8<br>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D<br>--- grinder-core/src/main/java/net/grinder/engine/process/Grin=
derProcess.java (revision ffb7bca73ea3190eaab12d1eccf73bf31464b58f)<br>+++=
grinder-core/src/main/java/net/grinder/engine/process/GrinderProcess.java =
(revision )<br>@@ -95,6 +95,9 @@<br> import java.util.Map;<br> import java=
.util.Timer;<br> import java.util.TimerTask;<br>+import java.util.concurren=
t.Executors;<br>+import java.util.concurrent.ScheduledExecutorService;<br>+=
import java.util.concurrent.TimeUnit;<br> <br> <br> /**<br>@@ -141,6 +144,1=
0 @@<br> // Guarded by m_eventSynchronisation.<br> private String m_shu=
tdownReason;<br> <br>+ private ClientSender sender;<br>+<br>+ private fi=
nal ScheduledExecutorService m_executor =3D Executors.newSingleThreadSchedu=
ledExecutor();<br>+<br> /**<br> * Creates a new {@code GrinderProcess}=
instance.<br> *<br>@@ -184,13 +191,34 @@<br> <br> final BarrierGrou=
ps barrierGroups;<br> <br>+ sender =3D ClientSender.connect(<br>+ =
new ConnectorFactory(ConnectionType.WORKER).create(properties),<br>+ =
new WorkerAddress(workerIdentity));<br>+<br>+ final Runnable k=
eepAlive =3D new Runnable() {<br>+ @Override<br>+ public void run=
() {<br>+ try {<br>+ sender.sendKeepAlive();<br>+ }<=
br>+ catch (CommunicationException e) {<br>+ // Connection =
is dead, terminate task.<br>+ throw new RuntimeException(e);<br>+ =
}<br>+ }<br>+ };<br>+<br>+ m_executor.scheduleWithFixedDe=
lay(keepAlive,<br>+ 0,<br>+ 1,<br>+ TimeUn=
it.SECONDS);<br>+<br>+<br> if (m_initialisationMessage.getReportToConso=
le()) {<br> m_consoleSender =3D<br> new FirstHurdleSender(<br=
>- new QueuedSenderDecorator(<br>- ClientSender.connect=
(<br>- new ConnectorFactory(ConnectionType.WORKER).create(prop=
erties),<br>- new WorkerAddress(workerIdentity))));<br>+ =
new QueuedSenderDecorator(sender<br>+ ));<br> <br> barri=
erGroups =3D<br> new ClientBarrierGroups(m_consoleSender,<br></pre>=
</div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Tue, Nov=
1, 2016 at 1:33 PM, Joel Lucuik <span dir=3D"ltr"><<a href=3D"mailto:jo=
[email protected]" target=3D"_blank">[email protected]</a>></span>=
wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bor=
der-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div><div><div><=
div>Hi,<br><br>Grinder version is 3-11.<br>4 vms, each VM has an agent, and=
the console shares a VM with the first agent. 10,0000 users (threads). <br=
><br></div>Problem: We are occasionally seeing broken pipes when the agent(=
s) report anything to the console. <br></div></div><div><br>My Investigatio=
n:<br><br>I have been looking at the Grinder source. There is some code whi=
ch traps CommunicationException such as <br><br>AbstractSender.send() or <b=
r>ClientSender.blockingSend()<br><br>but doesn't do anything with it. S=
uch as attempt a reconnect if it's a Broken Pipe.<br><br></div><div>Has=
this been fixed in newer versions? More specifically if the agent tries to=
send stats to the console, or a status update, and encounters a broken pip=
e, can it reconnect?<br><br></div><div>Suggestions on which code/class/meth=
od to fix?<br><br></div><div><br></div><div>We have very good unix people o=
n our team. They have already upped the OS files, so we are (fairly) sure t=
his is not the issue.<br><br></div><div>Right now we are debating on a port=
to JMeter.<br><br></div><div>Any advice would be appreciated.<span class=
=3D"HOEnZb"><font color=3D"#888888"><br></font></span></div><span class=3D"=
HOEnZb"><font color=3D"#888888"><div><br></div><div>Joel<br></div></font></=
span></div><br><br></div>
<br>------------------------------<wbr>------------------------------<wbr>-=
-----------------<br>
Developer Access Program for Intel Xeon Phi Processors<br>
Access to Intel Xeon Phi processor-based developer platforms.<br>
With one year of Intel Parallel Studio XE.<br>
Training and support from Colfax.<br>
Order your platform today. <a href=3D"http://sdm.link/xeonphi" rel=3D"noref=
errer" target=3D"_blank">http://sdm.link/xeonphi</a><br>___________________=
___________<wbr>_________________<br>
grinder-use mailing list<br>
<a href=3D"mailto:[email protected]">[email protected]=
ceforge.<wbr>net</a><br>
<a href=3D"https://lists.sourceforge.net/lists/listinfo/grinder-use" rel=3D=
"noreferrer" target=3D"_blank">https://lists.sourceforge.net/<wbr>lists/lis=
tinfo/grinder-use</a><br>
<br></blockquote></div><br></div></div>
--001a114ac51a2f2ec00540421a7d--
--===============1636039117841590564==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
--===============1636039117841590564==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
grinder-use mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/grinder-use
--===============1636039117841590564==--