[jgroups-users] Hijacking the JGroups channel inside Infinispan and not getting away with it :)

--===============0935754855032308116==
Content-Type: multipart/alternative; boundary="000000000000bb111305eb8be4d1"

--000000000000bb111305eb8be4d1
Content-Type: text/plain; charset="UTF-8"

Hi all,

We have run into an interesting race condition when attempting to use a
fork channel in our application, we more or less follow what Bela wrote
here
http://belaban.blogspot.com/2013/08/how-to-hijack-jgroups-channel-inside.html
.

When the cluster comes up and receives views the node broadcasts
information about itself over the fork channel so the cluster knows what
each node is capable of handling.

Unfortunately from what I can see there is a race condition on bootstrap
where the JGroups stack is started and receives a fork message before the
fork is inserted into the stack (fork not present in stack trace) [1] which
results in garbage / unknown data passing through the Infinispan
marshaller.. if you are lucky enough it will read an extremely large int
and try and allocate that into a byte array resulting in the JVM to throw
an OOM or NegativeArraySizeException

I believe one possible solution is to define the fork inside the
jgroups.xml which is used to create the initial jgroups stack which would
hopefully discard fork channel messages (until the message listener is
registered) and not pass them up the stack resulting in undefined behaviour.

I had a look at the fork documentation but there are not many examples,
does my possible solution seem feasible or does someone have alternative
solutions? I am currently looking through the Infinispan code to see if
there is any way to decorate jgroups before it starts.

Thanks in advance,

Johnathan

[1]

2022-10-20 10:42:28,515 ERROR [jgroups-89,service-2]
(org.infinispan.CLUSTER) ISPN000474: Error processing request 0@service-2
java.lang.NegativeArraySizeException: -436207616
at
org.infinispan.marshall.core.GlobalMarshaller.readUnknown(GlobalMarshaller.java:904)
~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
at
org.infinispan.marshall.core.GlobalMarshaller.readUnknown(GlobalMarshaller.java:891)
~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
at
org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:715)
~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
at
org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:358)
~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
at
org.infinispan.marshall.core.GlobalMarshaller.objectFromObjectInput(GlobalMarshaller.java:192)
~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
at
org.infinispan.marshall.core.GlobalMarshaller.objectFromByteBuffer(GlobalMarshaller.java:221)
~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1361)
~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1301)
~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$300(JGroupsTransport.java:130)
~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.lambda$up$0(JGroupsTransport.java:1450)
~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
at org.jgroups.util.MessageBatch.forEach(MessageBatch.java:318)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1450)
~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
at org.jgroups.JChannel.up(JChannel.java:796)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:903)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.protocols.FRAG3.up(FRAG3.java:187)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:338)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:297)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.protocols.UNICAST3.deliverBatch(UNICAST3.java:1071)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.protocols.UNICAST3.removeAndDeliver(UNICAST3.java:886)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.protocols.UNICAST3.handleBatchReceived(UNICAST3.java:852)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:501)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:689)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:338)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:197)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:338)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:338)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.stack.Protocol.up(Protocol.java:338)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at org.jgroups.protocols.TP.passBatchUp(TP.java:1408)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at
org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.passBatchUp(MaxOneThreadPerSender.java:284)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at
org.jgroups.util.SubmitToThreadPool$BatchHandler.run(SubmitToThreadPool.java:136)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at
org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.run(MaxOneThreadPerSender.java:273)
~[jgroups-4.2.1.Final.jar:4.2.1.Final]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]

--000000000000bb111305eb8be4d1
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"auto"><div dir=3D"ltr">Hi all,<div><br></div><div>We have run i=
nto an interesting race condition when attempting to use a fork channel in =
our application, we more or less follow what Bela wrote here=C2=A0<a href=
=3D"http://belaban.blogspot.com/2013/08/how-to-hijack-jgroups-channel-insid=
e.html" target=3D"_blank" rel=3D"noreferrer">http://belaban.blogspot.com/20=
13/08/how-to-hijack-jgroups-channel-inside.html</a> .</div><div><br></div><=
div>When the cluster comes up and receives views the node broadcasts inform=
ation about itself over the fork channel so the cluster knows what each nod=
e is capable of handling.</div><div><br></div><div>Unfortunately from what =
I can see there is a race condition on bootstrap where the JGroups stack is=
 started and receives a fork message before the fork is inserted into the s=
tack (fork not present in stack trace) [1] which results in garbage / unkno=
wn data passing through the Infinispan marshaller.. if you are lucky enough=
 it will read an extremely large int and try and allocate that into a byte =
array resulting in the JVM to throw an OOM or=C2=A0NegativeArraySizeExcepti=
on</div><div><br></div><div>I believe one possible solution is to define th=
e fork inside the jgroups.xml which is used to create the initial jgroups s=
tack which would hopefully discard fork channel messages (until the message=
 listener is registered) and not pass them up the stack resulting in undefi=
ned behaviour.</div><div dir=3D"auto"><br></div><div dir=3D"auto">I had a l=
ook at the fork documentation but there are not many examples, does my poss=
ible solution seem feasible or does someone have alternative solutions? I a=
m currently looking through the Infinispan code to see if there is any way =
to decorate jgroups before it starts.</div><div dir=3D"auto"><br></div><div=
 dir=3D"auto">Thanks in advance,</div><div dir=3D"auto"><br></div><div dir=
=3D"auto">Johnathan=C2=A0</div><div><br></div><div>[1]</div><div><br></div>=
<div>2022-10-20 10:42:28,515 ERROR [jgroups-89,service-2] (org.infinispan.C=
LUSTER) ISPN000474: Error processing request 0@service-2<br>java.lang.Negat=
iveArraySizeException: -436207616<br>	at org.infinispan.marshall.core.Globa=
lMarshaller.readUnknown(GlobalMarshaller.java:904) ~[infinispan-core-11.0.1=
.Final.jar:11.0.1.Final]<br>	at org.infinispan.marshall.core.GlobalMarshall=
er.readUnknown(GlobalMarshaller.java:891) ~[infinispan-core-11.0.1.Final.ja=
r:11.0.1.Final]<br>	at org.infinispan.marshall.core.GlobalMarshaller.readNo=
nNullableObject(GlobalMarshaller.java:715) ~[infinispan-core-11.0.1.Final.j=
ar:11.0.1.Final]<br>	at org.infinispan.marshall.core.GlobalMarshaller.readN=
ullableObject(GlobalMarshaller.java:358) ~[infinispan-core-11.0.1.Final.jar=
:11.0.1.Final]<br>	at org.infinispan.marshall.core.GlobalMarshaller.objectF=
romObjectInput(GlobalMarshaller.java:192) ~[infinispan-core-11.0.1.Final.ja=
r:11.0.1.Final]<br>	at org.infinispan.marshall.core.GlobalMarshaller.object=
FromByteBuffer(GlobalMarshaller.java:221) ~[infinispan-core-11.0.1.Final.ja=
r:11.0.1.Final]<br>	at org.infinispan.remoting.transport.jgroups.JGroupsTra=
nsport.processRequest(JGroupsTransport.java:1361) ~[infinispan-core-11.0.1.=
Final.jar:11.0.1.Final]<br>	at org.infinispan.remoting.transport.jgroups.JG=
roupsTransport.processMessage(JGroupsTransport.java:1301) ~[infinispan-core=
-11.0.1.Final.jar:11.0.1.Final]<br>	at org.infinispan.remoting.transport.jg=
roups.JGroupsTransport.access$300(JGroupsTransport.java:130) ~[infinispan-c=
ore-11.0.1.Final.jar:11.0.1.Final]<br>	at org.infinispan.remoting.transport=
.jgroups.JGroupsTransport$ChannelCallbacks.lambda$up$0(JGroupsTransport.jav=
a:1450) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]<br>	at org.jgroups=
.util.MessageBatch.forEach(MessageBatch.java:318) ~[jgroups-4.2.1.Final.jar=
:4.2.1.Final]<br>	at org.infinispan.remoting.transport.jgroups.JGroupsTrans=
port$ChannelCallbacks.up(JGroupsTransport.java:1450) ~[infinispan-core-11.0=
.1.Final.jar:11.0.1.Final]<br>	at org.jgroups.JChannel.up(JChannel.java:796=
) ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>	at org.jgroups.stack.ProtocolS=
tack.up(ProtocolStack.java:903) ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>	=
at org.jgroups.protocols.FRAG3.up(FRAG3.java:187) ~[jgroups-4.2.1.Final.jar=
:4.2.1.Final]<br>	at org.jgroups.protocols.FlowControl.up(FlowControl.java:=
418) ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>	at org.jgroups.stack.Protoc=
ol.up(Protocol.java:338) ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>	at org.=
jgroups.protocols.pbcast.STABLE.up(STABLE.java:297) ~[jgroups-4.2.1.Final.j=
ar:4.2.1.Final]<br>	at org.jgroups.protocols.UNICAST3.deliverBatch(UNICAST3=
.java:1071) ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>	at org.jgroups.proto=
cols.UNICAST3.removeAndDeliver(UNICAST3.java:886) ~[jgroups-4.2.1.Final.jar=
:4.2.1.Final]<br>	at org.jgroups.protocols.UNICAST3.handleBatchReceived(UNI=
CAST3.java:852) ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>	at org.jgroups.p=
rotocols.UNICAST3.up(UNICAST3.java:501) ~[jgroups-4.2.1.Final.jar:4.2.1.Fin=
al]<br>	at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:689) ~[jgro=
ups-4.2.1.Final.jar:4.2.1.Final]<br>	at org.jgroups.stack.Protocol.up(Proto=
col.java:338) ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>	at org.jgroups.pro=
tocols.FailureDetection.up(FailureDetection.java:197) ~[jgroups-4.2.1.Final=
.jar:4.2.1.Final]<br>	at org.jgroups.stack.Protocol.up(Protocol.java:338) ~=
[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>	at org.jgroups.stack.Protocol.up(=
Protocol.java:338) ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>	at org.jgroup=
s.stack.Protocol.up(Protocol.java:338) ~[jgroups-4.2.1.Final.jar:4.2.1.Fina=
l]<br>	at org.jgroups.protocols.TP.passBatchUp(TP.java:1408) ~[jgroups-4.2.=
1.Final.jar:4.2.1.Final]<br>	at org.jgroups.util.MaxOneThreadPerSender$Batc=
hHandlerLoop.passBatchUp(MaxOneThreadPerSender.java:284) ~[jgroups-4.2.1.Fi=
nal.jar:4.2.1.Final]<br>	at org.jgroups.util.SubmitToThreadPool$BatchHandle=
r.run(SubmitToThreadPool.java:136) ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<b=
r>	at org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.run(MaxOneThr=
eadPerSender.java:273) ~[jgroups-4.2.1.Final.jar:4.2.1.Final]<br>	at java.u=
til.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~=
[?:?]<br>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE=
xecutor.java:628) ~[?:?]<br>	at java.lang.Thread.run(Thread.java:829) ~[?:?=
]</div></div></div>

--000000000000bb111305eb8be4d1--


--===============0935754855032308116==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline


--===============0935754855032308116==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users

--===============0935754855032308116==--