Re: [jgroups-users] Hijacking the JGroups channel inside Infinispan and not getting away with it :)
Questions/problems related to using JGroups <[email protected]> Tue, 22 Nov 2022 02:29:18 +0100
| Newsgroups | gmane.comp.java.javagroups.general |
|---|---|
| Message-ID | <mailman.20459.1669080579.1459.javagroups-users@lists.sourceforge.net> |
--===============9145970667175078134==
Content-Type: multipart/alternative; boundary="0000000000008c475305ee051ae3"
--0000000000008c475305ee051ae3
Content-Type: text/plain; charset="UTF-8"
Thanks for the confirmation Bela, I was easily able to modify the
Infinispan configuration to include the FORK [1] and the application is no
longer prone to random OOMs during bootstrap.
Cheers,
Johnathan
[1]
<config xmlns="urn:org:jgroups"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:org:jgroups
http://www.jgroups.org/schema/jgroups-4.0.xsd">
...
<FORK>
<fork-stacks>
<fork-stack id="hijack-stack"/>
</fork-stacks>
</FORK>
<FRAG3 />
</config>
if (cacheManager.getTransport() instanceof JGroupsTransport) {
JGroupsTransport jGroupsTransport = (JGroupsTransport)
cacheManager.getTransport();
ProtocolStack stack = jGroupsTransport.getChannel().getProtocolStack();
Class<? extends Protocol> neighborProtocol =
stack.findProtocol(FRAG2.class) != null ?
FRAG2.class : FRAG3.class;
channel = new ForkChannel(jGroupsTransport.getChannel(),
"hijack-stack",
"lead-hijacker",
false,
ProtocolStack.Position.ABOVE,
neighborProtocol);
On Fri, 21 Oct 2022, 16:40 Questions/problems related to using JGroups via
javagroups-users, <[email protected]> wrote:
> Hi Jonathan
>
> yes, the best solution is to define FORK in the JGroups section of the
> Infinispan configuration, as you mentioned.
>
> If you don't control the configuration, then it gets a bit more
> tricky... you could (*before sending any brodcast traffic*) insert FORK
> dynamically into every JChannel instance.
> To do that, you need to get the JChannel; IIRC, via (paraphrased)
> cache.getExtendedCache().getRpcManager().getTransport(), downcast it to
> JGroupsTransport, then call getChannel().
>
> Once you have JChannel, call
> channel.getProtocolStack().insertProtocolAtTop(new FORK(...));
> Hope this helps,
>
> [1] http://www.jgroups.org/manual5/index.html#ForkChannel
>
> On 21.10.22 16:02, Questions/problems related to using JGroups wrote:
> > Hi all,
> >
> > We have run into an interesting race condition when attempting to use a
> > fork channel in our application, we more or less follow what Bela wrote
> > here
> >
> http://belaban.blogspot.com/2013/08/how-to-hijack-jgroups-channel-inside.html
> <
> http://belaban.blogspot.com/2013/08/how-to-hijack-jgroups-channel-inside.html>
> .
> >
> > When the cluster comes up and receives views the node broadcasts
> > information about itself over the fork channel so the cluster knows what
> > each node is capable of handling.
> >
> > Unfortunately from what I can see there is a race condition on bootstrap
> > where the JGroups stack is started and receives a fork message before
> > the fork is inserted into the stack (fork not present in stack trace)
> > [1] which results in garbage / unknown data passing through the
> > Infinispan marshaller.. if you are lucky enough it will read an
> > extremely large int and try and allocate that into a byte array
> > resulting in the JVM to throw an OOM or NegativeArraySizeException
> >
> > I believe one possible solution is to define the fork inside the
> > jgroups.xml which is used to create the initial jgroups stack which
> > would hopefully discard fork channel messages (until the message
> > listener is registered) and not pass them up the stack resulting in
> > undefined behaviour.
> >
> > I had a look at the fork documentation but there are not many examples,
> > does my possible solution seem feasible or does someone have alternative
> > solutions? I am currently looking through the Infinispan code to see if
> > there is any way to decorate jgroups before it starts.
> >
> > Thanks in advance,
> >
> > Johnathan
> >
> > [1]
> >
> > 2022-10-20 10:42:28,515 ERROR [jgroups-89,service-2]
> > (org.infinispan.CLUSTER) ISPN000474: Error processing request 0@service-2
> > java.lang.NegativeArraySizeException: -436207616
> > at
> >
> org.infinispan.marshall.core.GlobalMarshaller.readUnknown(GlobalMarshaller.java:904)
> ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> > at
> >
> org.infinispan.marshall.core.GlobalMarshaller.readUnknown(GlobalMarshaller.java:891)
> ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> > at
> >
> org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:715)
> ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> > at
> >
> org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:358)
> ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> > at
> >
> org.infinispan.marshall.core.GlobalMarshaller.objectFromObjectInput(GlobalMarshaller.java:192)
> ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> > at
> >
> org.infinispan.marshall.core.GlobalMarshaller.objectFromByteBuffer(GlobalMarshaller.java:221)
> ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> > at
> >
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1361)
> ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> > at
> >
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1301)
> ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> > at
> >
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$300(JGroupsTransport.java:130)
> ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> > at
> >
> org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.lambda$up$0(JGroupsTransport.java:1450)
> ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> > at org.jgroups.util.MessageBatch.forEach(MessageBatch.java:318)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at
> >
> org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1450)
> ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> > at org.jgroups.JChannel.up(JChannel.java:796)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:903)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.protocols.FRAG3.up(FRAG3.java:187)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.stack.Protocol.up(Protocol.java:338)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:297)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.protocols.UNICAST3.deliverBatch(UNICAST3.java:1071)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.protocols.UNICAST3.removeAndDeliver(UNICAST3.java:886)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.protocols.UNICAST3.handleBatchReceived(UNICAST3.java:852)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:501)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:689)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.stack.Protocol.up(Protocol.java:338)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:197)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.stack.Protocol.up(Protocol.java:338)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.stack.Protocol.up(Protocol.java:338)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.stack.Protocol.up(Protocol.java:338)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.protocols.TP.passBatchUp(TP.java:1408)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at
> >
> org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.passBatchUp(MaxOneThreadPerSender.java:284)
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at
> >
> org.jgroups.util.SubmitToThreadPool$BatchHandler.run(SubmitToThreadPool.java:136)
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at
> >
> org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.run(MaxOneThreadPerSender.java:273)
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> ~[?:?]
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> ~[?:?]
> > at java.lang.Thread.run(Thread.java:829) ~[?:?]
> >
> >
> > _______________________________________________
> > javagroups-users mailing list
> > [email protected]
> > https://lists.sourceforge.net/lists/listinfo/javagroups-users
>
> --
> Bela Ban | http://www.jgroups.org
>
>
>
> _______________________________________________
> javagroups-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>
--0000000000008c475305ee051ae3
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div dir=3D"auto"><div>Thanks for the confirmation Bela, I=
was easily able to modify the Infinispan configuration to include the FORK=
[1] and the application is no longer prone to random OOMs during bootstrap=
.</div><div dir=3D"auto"><br></div><div dir=3D"auto">Cheers,</div><div dir=
=3D"auto"><br></div><div dir=3D"auto">Johnathan=C2=A0</div><div dir=3D"auto=
"><br></div><div dir=3D"auto">[1]</div><div dir=3D"auto"><br></div><div dir=
=3D"auto"><pre style=3D"background-color:rgb(43,43,43);color:rgb(169,183,19=
8);font-family:"JetBrains Mono",monospace;font-size:9.8pt"><span =
style=3D"color:rgb(232,191,106)"><config </span><span style=3D"color:rgb=
(186,186,186)">xmlns</span><span style=3D"color:rgb(106,135,89)">=3D"u=
rn:org:jgroups"<br></span><span style=3D"color:rgb(106,135,89)"> =
</span><span style=3D"color:rgb(186,186,186)">xmlns:</span><span style=3D=
"color:rgb(152,118,170)">xsi</span><span style=3D"color:rgb(106,135,89)">=
=3D"<a href=3D"http://www.w3.org/2001/XMLSchema-instance">http://www.w=
3.org/2001/XMLSchema-instance</a>"<br></span><span style=3D"color:rgb(=
106,135,89)"> </span><span style=3D"color:rgb(152,118,170)">xsi</spa=
n><span style=3D"color:rgb(186,186,186)">:schemaLocation</span><span style=
=3D"color:rgb(106,135,89)">=3D"urn:org:jgroups <a href=3D"http://www.j=
groups.org/schema/jgroups-4.0.xsd">http://www.jgroups.org/schema/jgroups-4.=
0.xsd</a>"</span><span style=3D"color:rgb(232,191,106)">><br></span=
><span style=3D"color:rgb(232,191,106)"> </span>...<br> <span style=
=3D"color:rgb(232,191,106)"><FORK><br></span><span style=3D"color:rgb=
(232,191,106)"> <fork-stacks><br></span><span style=3D"color:r=
gb(232,191,106)"> <fork-stack </span><span style=3D"color:rgb=
(186,186,186)">id</span><span style=3D"color:rgb(106,135,89)">=3D"hija=
ck-stack"</span><span style=3D"color:rgb(232,191,106)">/><br></span=
><span style=3D"color:rgb(232,191,106)"> </fork-stacks><br></s=
pan><span style=3D"color:rgb(232,191,106)"> </FORK><br></span><spa=
n style=3D"color:rgb(232,191,106)"> <FRAG3 /><br></span><span styl=
e=3D"color:rgb(232,191,106)"></config></span></pre></div><div dir=3D"=
auto"><br></div><div dir=3D"auto"><pre style=3D"background-color:rgb(43,43,=
43);color:rgb(169,183,198);font-family:"JetBrains Mono",monospace=
;font-size:9.8pt"><span style=3D"color:rgb(204,120,50)">if </span>(<span st=
yle=3D"color:rgb(152,118,170)">cacheManager</span>.getTransport() <span sty=
le=3D"color:rgb(204,120,50)">instanceof </span>JGroupsTransport) {<br> J=
GroupsTransport jGroupsTransport =3D (JGroupsTransport) <span style=3D"colo=
r:rgb(152,118,170)">cacheManager</span>.getTransport()<span style=3D"color:=
rgb(204,120,50)">;<br></span><span style=3D"color:rgb(204,120,50)"><br></sp=
an><span style=3D"color:rgb(204,120,50)"> </span>ProtocolStack stack =3D=
jGroupsTransport.getChannel().getProtocolStack()<span style=3D"color:rgb(2=
04,120,50)">;<br></span><span style=3D"color:rgb(204,120,50)"><br></span><s=
pan style=3D"color:rgb(204,120,50)"> </span>Class<? <span style=3D"co=
lor:rgb(204,120,50)">extends </span>Protocol> neighborProtocol =3D stack=
.findProtocol(FRAG2.<span style=3D"color:rgb(204,120,50)">class</span>) !=
=3D <span style=3D"color:rgb(204,120,50)">null </span>?<br> FRAG=
2.<span style=3D"color:rgb(204,120,50)">class </span>: FRAG3.<span style=3D=
"color:rgb(204,120,50)">class;<br></span><span style=3D"color:rgb(204,120,5=
0)"> </span><span style=3D"color:rgb(152,118,170)">channel </span>=3D <s=
pan style=3D"color:rgb(204,120,50)">new </span>ForkChannel(jGroupsTransport=
.getChannel()<span style=3D"color:rgb(204,120,50)">,<br></span><span style=
=3D"color:rgb(204,120,50)"> </span><span style=
=3D"color:rgb(106,135,89)">"hijack-stack"</span><span style=3D"co=
lor:rgb(204,120,50)">,<br></span><span style=3D"color:rgb(204,120,50)"> =
</span><span style=3D"color:rgb(106,135,89)">&quo=
t;lead-hijacker"</span><span style=3D"color:rgb(204,120,50)">,<br></sp=
an><span style=3D"color:rgb(204,120,50)"> fals=
e,<br></span><span style=3D"color:rgb(204,120,50)"> =
</span>ProtocolStack.Position.<span style=3D"color:rgb(152,118,170);f=
ont-style:italic">ABOVE</span><span style=3D"color:rgb(204,120,50)">,<br></=
span><span style=3D"color:rgb(204,120,50)"> </=
span>neighborProtocol)<span style=3D"color:rgb(204,120,50)">;
<br></span></pre><br><div class=3D"gmail_quote" dir=3D"auto"><div dir=3D"lt=
r" class=3D"gmail_attr">On Fri, 21 Oct 2022, 16:40 Questions/problems relat=
ed to using JGroups via javagroups-users, <<a href=3D"mailto:javagroups-=
[email protected]" target=3D"_blank">[email protected]=
eforge.net</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=
=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Jonat=
han<br>
<br>
yes, the best solution is to define FORK in the JGroups section of the <br>
Infinispan configuration, as you mentioned.<br>
<br>
If you don't control the configuration, then it gets a bit more <br>
tricky... you could (*before sending any brodcast traffic*) insert FORK <br=
>
dynamically into every JChannel instance.<br>
To do that, you need to get the JChannel; IIRC, via (paraphrased) <br>
cache.getExtendedCache().getRpcManager().getTransport(), downcast it to <br=
>
JGroupsTransport, then call getChannel().<br>
<br>
Once you have JChannel, call <br>
channel.getProtocolStack().insertProtocolAtTop(new FORK(...));<br>
Hope this helps,<br>
<br>
[1] <a href=3D"http://www.jgroups.org/manual5/index.html#ForkChannel" rel=
=3D"noreferrer noreferrer" target=3D"_blank">http://www.jgroups.org/manual5=
/index.html#ForkChannel</a><br>
<br>
On 21.10.22 16:02, Questions/problems related to using JGroups wrote:<br>
> Hi all,<br>
> <br>
> We have run into an interesting race condition when attempting to use =
a <br>
> fork channel in our application, we more or less follow what Bela wrot=
e <br>
> here <br>
> <a href=3D"http://belaban.blogspot.com/2013/08/how-to-hijack-jgroups-c=
hannel-inside.html" rel=3D"noreferrer noreferrer" target=3D"_blank">http://=
belaban.blogspot.com/2013/08/how-to-hijack-jgroups-channel-inside.html</a> =
<<a href=3D"http://belaban.blogspot.com/2013/08/how-to-hijack-jgroups-ch=
annel-inside.html" rel=3D"noreferrer noreferrer" target=3D"_blank">http://b=
elaban.blogspot.com/2013/08/how-to-hijack-jgroups-channel-inside.html</a>&g=
t; .<br>
> <br>
> When the cluster comes up and receives views the node broadcasts <br>
> information about itself over the fork channel so the cluster knows wh=
at <br>
> each node is capable of handling.<br>
> <br>
> Unfortunately from what I can see there is a race condition on bootstr=
ap <br>
> where the JGroups stack is started and receives a fork message before =
<br>
> the fork is inserted into the stack (fork not present in stack trace) =
<br>
> [1] which results in garbage / unknown data passing through the <br>
> Infinispan marshaller.. if you are lucky enough it will read an <br>
> extremely large int and try and allocate that into a byte array <br>
> resulting in the JVM to throw an OOM or=C2=A0NegativeArraySizeExceptio=
n<br>
> <br>
> I believe one possible solution is to define the fork inside the <br>
> jgroups.xml which is used to create the initial jgroups stack which <b=
r>
> would hopefully discard fork channel messages (until the message <br>
> listener is registered) and not pass them up the stack resulting in <b=
r>
> undefined behaviour.<br>
> <br>
> I had a look at the fork documentation but there are not many examples=
, <br>
> does my possible solution seem feasible or does someone have alternati=
ve <br>
> solutions? I am currently looking through the Infinispan code to see i=
f <br>
> there is any way to decorate jgroups before it starts.<br>
> <br>
> Thanks in advance,<br>
> <br>
> Johnathan<br>
> <br>
> [1]<br>
> <br>
> 2022-10-20 10:42:28,515 ERROR [jgroups-89,service-2] <br>
> (org.infinispan.CLUSTER) ISPN000474: Error processing request 0@servic=
e-2<br>
> java.lang.NegativeArraySizeException: -436207616<br>
> at <br>
> org.infinispan.marshall.core.GlobalMarshaller.readUnknown(GlobalMarsha=
ller.java:904) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]<br>
> at <br>
> org.infinispan.marshall.core.GlobalMarshaller.readUnknown(GlobalMarsha=
ller.java:891) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]<br>
> at <br>
> org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(Gl=
obalMarshaller.java:715) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]<b=
r>
> at <br>
> org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(Globa=
lMarshaller.java:358) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]<br>
> at <br>
> org.infinispan.marshall.core.GlobalMarshaller.objectFromObjectInput(Gl=
obalMarshaller.java:192) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]<b=
r>
> at <br>
> org.infinispan.marshall.core.GlobalMarshaller.objectFromByteBuffer(Glo=
balMarshaller.java:221) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]<br=
>
> at <br>
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequ=
est(JGroupsTransport.java:1361) ~[infinispan-core-11.0.1.Final.jar:11.0.1.F=
inal]<br>
> at <br>
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMess=
age(JGroupsTransport.java:1301) ~[infinispan-core-11.0.1.Final.jar:11.0.1.F=
inal]<br>
> at <br>
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$300(=
JGroupsTransport.java:130) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]=
<br>
> at <br>
> org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCall=
backs.lambda$up$0(JGroupsTransport.java:1450) ~[infinispan-core-11.0.1.Fina=
l.jar:11.0.1.Final]<br>
> at org.jgroups.util.MessageBatch.forEach(MessageBatch.java:318) <br>
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>
> at <br>
> org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCall=
backs.up(JGroupsTransport.java:1450) ~[infinispan-core-11.0.1.Final.jar:11.=
0.1.Final]<br>
> at org.jgroups.JChannel.up(JChannel.java:796) <br>
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:903) <br>
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>
> at org.jgroups.protocols.FRAG3.up(FRAG3.java:187) <br>
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) <br>
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>
> at org.jgroups.stack.Protocol.up(Protocol.java:338) <br>
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:297) <br>
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>
> at org.jgroups.protocols.UNICAST3.deliverBatch(UNICAST3.java:1071) <br=
>
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>
> at org.jgroups.protocols.UNICAST3.removeAndDeliver(UNICAST3.java:886) =
<br>
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>
> at org.jgroups.protocols.UNICAST3.handleBatchReceived(UNICAST3.java:85=
2) <br>
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:501) <br>
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:689) <br>
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>
> at org.jgroups.stack.Protocol.up(Protocol.java:338) <br>
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>
> at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:197=
) <br>
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>
> at org.jgroups.stack.Protocol.up(Protocol.java:338) <br>
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>
> at org.jgroups.stack.Protocol.up(Protocol.java:338) <br>
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>
> at org.jgroups.stack.Protocol.up(Protocol.java:338) <br>
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>
> at org.jgroups.protocols.TP.passBatchUp(TP.java:1408) <br>
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>
> at <br>
> org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.passBatchUp(Ma=
xOneThreadPerSender.java:284) ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>
> at <br>
> org.jgroups.util.SubmitToThreadPool$BatchHandler.run(SubmitToThreadPoo=
l.java:136) ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>
> at <br>
> org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.run(MaxOneThre=
adPerSender.java:273) ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>
> at <br>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j=
ava:1128) ~[?:?]<br>
> at <br>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.=
java:628) ~[?:?]<br>
> at java.lang.Thread.run(Thread.java:829) ~[?:?]<br>
> <br>
> <br>
> _______________________________________________<br>
> javagroups-users mailing list<br>
> <a href=3D"mailto:[email protected]" rel=3D"noref=
errer" target=3D"_blank">[email protected]</a><br>
> <a href=3D"https://lists.sourceforge.net/lists/listinfo/javagroups-use=
rs" rel=3D"noreferrer noreferrer" target=3D"_blank">https://lists.sourcefor=
ge.net/lists/listinfo/javagroups-users</a><br>
<br>
-- <br>
Bela Ban | <a href=3D"http://www.jgroups.org" rel=3D"noreferrer noreferrer"=
target=3D"_blank">http://www.jgroups.org</a><br>
<br>
<br>
<br>
_______________________________________________<br>
javagroups-users mailing list<br>
<a href=3D"mailto:[email protected]" rel=3D"noreferrer=
" target=3D"_blank">[email protected]</a><br>
<a href=3D"https://lists.sourceforge.net/lists/listinfo/javagroups-users" r=
el=3D"noreferrer noreferrer" target=3D"_blank">https://lists.sourceforge.ne=
t/lists/listinfo/javagroups-users</a><br>
</blockquote></div></div></div>
</div>
--0000000000008c475305ee051ae3--
--===============9145970667175078134==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
--===============9145970667175078134==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users
--===============9145970667175078134==--