[jgroups-users] OOM error after long run with jgroups
Questions/problems related to using JGroups <[email protected]> Mon, 6 May 2019 13:51:34 -0400
| Newsgroups | gmane.comp.java.javagroups.general |
|---|---|
| Message-ID | <mailman.45043.1557165138.1518.javagroups-users@lists.sourceforge.net> |
--===============0639019060822780452==
Content-Type: multipart/alternative; boundary="0000000000006ee3ed05883bc2fc"
--0000000000006ee3ed05883bc2fc
Content-Type: text/plain; charset="UTF-8"
[Sorry, this is as short as I thought I could make it. :) ]
Hi again,
I asked about this at the end of last year just in case, but didn't have
any good info at the time. After a longevity test ~4 months, I'm seeing my
jgroups cluster members hit an OOM issue. I ran two clusters, one with JVM
option to get a memory dump when needed:
-agentlib:hprof=heap=sites,depth=20,file=/tmp/<file name>
...and one without, both with jvm option -Xmx128m set. Then I watched the
memory with 'jstat -gc' and checked in on them every now and then. There
were only a couple full GCs in the first couple months, but eventually the
FGCs were happening minutes apart and eventually the nodes started to hit
OOM. I can share the full jstat output if you're curious.
This is the top of the heap allocation near the end of the run, when FGCs
were <1 minute apart (I can share the whole memory file, but it's over 300
meg):
--- begin ---
SITES BEGIN (ordered by live bytes) Thu May 2 23:46:35 2019
percent live alloc'ed stack class
rank self accum bytes objs bytes objs trace name
1 39.77% 39.77% 38014944 1187967 38014944 1187967 415369
java.lang.ref.WeakReference
2 29.83% 69.60% 28511208 1187967 28511208 1187967 415370
java.util.concurrent.ConcurrentLinkedQueue$Node
3 13.68% 83.28% 13077664 408677 13077664 408677 387456
java.lang.ref.WeakReference
4 10.26% 93.54% 9808248 408677 9808248 408677 387458
java.util.concurrent.ConcurrentLinkedQueue$Node
5 0.59% 94.13% 562432 17576 562432 17576 416748
java.lang.ref.WeakReference
6 0.44% 94.57% 421824 17576 421824 17576 416749
java.util.concurrent.ConcurrentLinkedQueue$Node
--- end ---
And these are the traces for those, though the last two are negligible. I
just wanted to include them because they're also jgroups allocations:
--- begin ---
TRACE 415369:
java.lang.ref.Reference.<init>(Reference.java:306)
java.lang.ref.Reference.<init>(Reference.java:303)
java.lang.ref.WeakReference.<init>(WeakReference.java:57)
org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:36)
org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:23)
java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:619)
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932)
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
org.jgroups.protocols.TP.submitToThreadPool(TP.java:1380)
org.jgroups.protocols.TP.submitToThreadPool(TP.java:1375)
org.jgroups.util.SubmitToThreadPool.loopback(SubmitToThreadPool.java:30)
org.jgroups.util.MaxOneThreadPerSender.loopback(MaxOneThreadPerSender.java:54)
org.jgroups.protocols.TP.loopback(TP.java:1188)
org.jgroups.protocols.TP.down(TP.java:1107)
org.jgroups.stack.Protocol.down(Protocol.java:318)
org.jgroups.stack.Protocol.down(Protocol.java:318)
org.jgroups.protocols.FD_ALL$HeartbeatSender.run(FD_ALL.java:413)
org.jgroups.util.TimeScheduler3$Task.run(TimeScheduler3.java:324)
org.jgroups.util.TimeScheduler3$RecurringTask.run(TimeScheduler3.java:358)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
TRACE 415370:
java.util.concurrent.ConcurrentLinkedQueue$Node.<init>(ConcurrentLinkedQueue.java:188)
java.util.concurrent.ConcurrentLinkedQueue.offer(ConcurrentLinkedQueue.java:328)
java.util.concurrent.ConcurrentLinkedQueue.add(ConcurrentLinkedQueue.java:297)
org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:36)
org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:23)
java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:619)
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932)
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
org.jgroups.protocols.TP.submitToThreadPool(TP.java:1380)
org.jgroups.protocols.TP.submitToThreadPool(TP.java:1375)
org.jgroups.util.SubmitToThreadPool.loopback(SubmitToThreadPool.java:30)
org.jgroups.util.MaxOneThreadPerSender.loopback(MaxOneThreadPerSender.java:54)
org.jgroups.protocols.TP.loopback(TP.java:1188)
org.jgroups.protocols.TP.down(TP.java:1107)
org.jgroups.stack.Protocol.down(Protocol.java:318)
org.jgroups.stack.Protocol.down(Protocol.java:318)
org.jgroups.protocols.FD_ALL$HeartbeatSender.run(FD_ALL.java:413)
org.jgroups.util.TimeScheduler3$Task.run(TimeScheduler3.java:324)
org.jgroups.util.TimeScheduler3$RecurringTask.run(TimeScheduler3.java:358)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
TRACE 387456:
java.lang.ref.Reference.<init>(Reference.java:306)
java.lang.ref.Reference.<init>(Reference.java:303)
java.lang.ref.WeakReference.<init>(WeakReference.java:57)
org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:36)
org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:23)
java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:619)
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932)
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
org.jgroups.protocols.TP.submitToThreadPool(TP.java:1380)
org.jgroups.protocols.TP.submitToThreadPool(TP.java:1375)
org.jgroups.util.SubmitToThreadPool.process(SubmitToThreadPool.java:34)
org.jgroups.util.MaxOneThreadPerSender.process(MaxOneThreadPerSender.java:63)
org.jgroups.protocols.TP.handleSingleMessage(TP.java:1352)
org.jgroups.protocols.TP.receive(TP.java:1323)
org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:295)
java.lang.Thread.run(Thread.java:748)
TRACE 387458:
java.util.concurrent.ConcurrentLinkedQueue$Node.<init>(ConcurrentLinkedQueue.java:188)
java.util.concurrent.ConcurrentLinkedQueue.offer(ConcurrentLinkedQueue.java:328)
java.util.concurrent.ConcurrentLinkedQueue.add(ConcurrentLinkedQueue.java:297)
org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:36)
org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:23)
java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:619)
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932)
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
org.jgroups.protocols.TP.submitToThreadPool(TP.java:1380)
org.jgroups.protocols.TP.submitToThreadPool(TP.java:1375)
org.jgroups.util.SubmitToThreadPool.process(SubmitToThreadPool.java:34)
org.jgroups.util.MaxOneThreadPerSender.process(MaxOneThreadPerSender.java:63)
org.jgroups.protocols.TP.handleSingleMessage(TP.java:1352)
org.jgroups.protocols.TP.receive(TP.java:1323)
org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:295)
java.lang.Thread.run(Thread.java:748)
TRACE 416748:
java.lang.ref.Reference.<init>(Reference.java:306)
java.lang.ref.Reference.<init>(Reference.java:303)
java.lang.ref.WeakReference.<init>(WeakReference.java:57)
org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:36)
org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:23)
java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:619)
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932)
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
org.jgroups.util.TimeScheduler3.submitToPool(TimeScheduler3.java:231)
org.jgroups.util.TimeScheduler3.run(TimeScheduler3.java:182)
java.lang.Thread.run(Thread.java:748)
TRACE 416749:
java.util.concurrent.ConcurrentLinkedQueue$Node.<init>(ConcurrentLinkedQueue.java:188)
java.util.concurrent.ConcurrentLinkedQueue.offer(ConcurrentLinkedQueue.java:328)
java.util.concurrent.ConcurrentLinkedQueue.add(ConcurrentLinkedQueue.java:297)
org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:36)
org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:23)
java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:619)
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932)
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
org.jgroups.util.TimeScheduler3.submitToPool(TimeScheduler3.java:231)
org.jgroups.util.TimeScheduler3.run(TimeScheduler3.java:182)
java.lang.Thread.run(Thread.java:748)
--- end ---
Below is the code we use to create the jchannel. We're using jgroups
version 4.0.13. Can you tell me if this is something that can be fixed, or
if there's any way my code could be doing something to prevent these
objects from being collected?
This is a product we sell, rather than something running within our
company, so I don't have the ability to restart it every couple months
unless I add something in the next version to try to do so.
List<Protocol> stack = new ArrayList<>();
stack.add(new TCP()
.setValue("bind_addr",
InetAddress.getByName(getBindingAddress()))
.setValue("bind_port", bindingPort)
.setValue("thread_pool_min_threads", 1)
.setValue("thread_pool_keep_alive_time", 5000)
.setValue("send_buf_size", 640000)
.setValue("sock_conn_timeout", 300)
.setValue("recv_buf_size", 5000000));
stack.add(new TCPPING()
.setValue("initial_hosts", parseHostList())
.setValue("send_cache_on_join", true)
.setValue("port_range", 0));
stack.add(new MERGE3()
.setValue("min_interval", 10000)
.setValue("max_interval", 30000));
stack.add(new FD_ALL()
.setValue("timeout", getJGroupsTotalTimeout() * 1000L));
stack.add(new VERIFY_SUSPECT()
.setValue("timeout", 1500));
stack.add(new BARRIER());
stack.add(new NAKACK2()
.setValue("use_mcast_xmit", false));
stack.add(new UNICAST3());
stack.add(new STABLE()
.setValue("desired_avg_gossip", 50000)
.setValue("max_bytes", 4000000));
stack.add(createAuthProtocol(....));
stack.add(new GMS()
.setValue("join_timeout", 3000));
stack.add(new MFC()
.setValue("max_credits", 2000000)
.setValue("min_credits", 800000));
stack.add(new FRAG2());
stack.add(new STATE_TRANSFER());
return new JChannel(stack);
Thank you,
Bobby
--0000000000006ee3ed05883bc2fc
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div dir=3D"ltr"><div dir=3D"ltr"><div dir=3D"ltr"><div di=
r=3D"ltr"><div dir=3D"ltr"><div>[Sorry, this is as short as I thought I cou=
ld make it. :) ]</div><div><br></div><div dir=3D"ltr">Hi again,<div><br></d=
iv><div>I asked about this at the end of last year just in case, but didn&#=
39;t have any good info at the time. After a longevity test ~4 months, I=
9;m seeing my jgroups cluster members hit an OOM issue. I ran two clusters,=
one with JVM option to get a memory dump when needed:</div><div><br></div>=
<div>-agentlib:hprof=3Dheap=3Dsites,depth=3D20,file=3D/tmp/<file name>=
;<br></div><div><br></div><div>...and one without, both with=C2=A0jvm optio=
n -Xmx128m set. Then I watched the memory with 'jstat -gc' and chec=
ked in on them every now and then. There were only a couple full GCs in the=
first couple months, but eventually the FGCs were happening minutes apart =
and eventually the nodes started to hit OOM. I can share the full jstat out=
put if you're curious.</div><div><br></div><div>This is the top of the =
heap allocation near the end of the run, when FGCs were <1 minute apart =
(I can share the whole memory file, but it's over 300 meg):</div><div><=
br></div><div>--- begin ---</div><div><div>SITES BEGIN (ordered by live byt=
es) Thu May=C2=A0 2 23:46:35 2019</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 percent=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 live=C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 alloc'ed=C2=A0 stack class</div><div>=C2=A0rank=C2=A0 =C2=A0=
self=C2=A0 accum=C2=A0 =C2=A0 =C2=A0bytes objs=C2=A0 =C2=A0 =C2=A0bytes=C2=
=A0 objs trace name</div><div>=C2=A0 =C2=A0 1 39.77% 39.77%=C2=A0 38014944 =
1187967=C2=A0 38014944 1187967 415369 java.lang.ref.WeakReference</div><div=
>=C2=A0 =C2=A0 2 29.83% 69.60%=C2=A0 28511208 1187967=C2=A0 28511208 118796=
7 415370 java.util.concurrent.ConcurrentLinkedQueue$Node</div><div>=C2=A0 =
=C2=A0 3 13.68% 83.28%=C2=A0 13077664 408677=C2=A0 13077664 408677 387456 j=
ava.lang.ref.WeakReference</div><div>=C2=A0 =C2=A0 4 10.26% 93.54%=C2=A0 =
=C2=A09808248 408677=C2=A0 =C2=A09808248 408677 387458 java.util.concurrent=
.ConcurrentLinkedQueue$Node</div><div>=C2=A0 =C2=A0 5=C2=A0 0.59% 94.13%=C2=
=A0 =C2=A0 562432 17576=C2=A0 =C2=A0 562432 17576 416748 java.lang.ref.Weak=
Reference</div><div>=C2=A0 =C2=A0 6=C2=A0 0.44% 94.57%=C2=A0 =C2=A0 421824 =
17576=C2=A0 =C2=A0 421824 17576 416749 java.util.concurrent.ConcurrentLinke=
dQueue$Node</div></div><div>--- end ---</div><div><br></div><div>And these =
are the traces for those, though the last two are negligible. I just wanted=
to include them because they're also jgroups allocations:</div><div><b=
r></div><div>--- begin ---</div><div><div>TRACE 415369:</div><div><span sty=
le=3D"white-space:pre"> </span>java.lang.ref.Reference.<init>(Referen=
ce.java:306)</div><div><span style=3D"white-space:pre"> </span>java.lang.re=
f.Reference.<init>(Reference.java:303)</div><div><span style=3D"white=
-space:pre"> </span>java.lang.ref.WeakReference.<init>(WeakReference.=
java:57)</div><div><span style=3D"white-space:pre"> </span>org.jgroups.util=
.LazyThreadFactory.newThread(LazyThreadFactory.java:36)</div><div><span sty=
le=3D"white-space:pre"> </span>org.jgroups.util.LazyThreadFactory.newThread=
(LazyThreadFactory.java:23)</div><div><span style=3D"white-space:pre"> </sp=
an>java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolEx=
ecutor.java:619)</div><div><span style=3D"white-space:pre"> </span>java.uti=
l.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932)</div=
><div><span style=3D"white-space:pre"> </span>java.util.concurrent.ThreadPo=
olExecutor.execute(ThreadPoolExecutor.java:1378)</div><div><span style=3D"w=
hite-space:pre"> </span>org.jgroups.protocols.TP.submitToThreadPool(TP.java=
:1380)</div><div><span style=3D"white-space:pre"> </span>org.jgroups.protoc=
ols.TP.submitToThreadPool(TP.java:1375)</div><div><span style=3D"white-spac=
e:pre"> </span>org.jgroups.util.SubmitToThreadPool.loopback(SubmitToThreadP=
ool.java:30)</div><div><span style=3D"white-space:pre"> </span>org.jgroups.=
util.MaxOneThreadPerSender.loopback(MaxOneThreadPerSender.java:54)</div><di=
v><span style=3D"white-space:pre"> </span>org.jgroups.protocols.TP.loopback=
(TP.java:1188)</div><div><span style=3D"white-space:pre"> </span>org.jgroup=
s.protocols.TP.down(TP.java:1107)</div><div><span style=3D"white-space:pre"=
> </span>org.jgroups.stack.Protocol.down(Protocol.java:318)</div><div><span=
style=3D"white-space:pre"> </span>org.jgroups.stack.Protocol.down(Protocol=
.java:318)</div><div><span style=3D"white-space:pre"> </span>org.jgroups.pr=
otocols.FD_ALL$HeartbeatSender.run(FD_ALL.java:413)</div><div><span style=
=3D"white-space:pre"> </span>org.jgroups.util.TimeScheduler3$Task.run(TimeS=
cheduler3.java:324)</div><div><span style=3D"white-space:pre"> </span>org.j=
groups.util.TimeScheduler3$RecurringTask.run(TimeScheduler3.java:358)</div>=
<div><span style=3D"white-space:pre"> </span>java.util.concurrent.ThreadPoo=
lExecutor.runWorker(ThreadPoolExecutor.java:1149)</div><div>TRACE 415370:</=
div><div><span style=3D"white-space:pre"> </span>java.util.concurrent.Concu=
rrentLinkedQueue$Node.<init>(ConcurrentLinkedQueue.java:188)</div><di=
v><span style=3D"white-space:pre"> </span>java.util.concurrent.ConcurrentLi=
nkedQueue.offer(ConcurrentLinkedQueue.java:328)</div><div><span style=3D"wh=
ite-space:pre"> </span>java.util.concurrent.ConcurrentLinkedQueue.add(Concu=
rrentLinkedQueue.java:297)</div><div><span style=3D"white-space:pre"> </spa=
n>org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:36)</=
div><div><span style=3D"white-space:pre"> </span>org.jgroups.util.LazyThrea=
dFactory.newThread(LazyThreadFactory.java:23)</div><div><span style=3D"whit=
e-space:pre"> </span>java.util.concurrent.ThreadPoolExecutor$Worker.<ini=
t>(ThreadPoolExecutor.java:619)</div><div><span style=3D"white-space:pre=
"> </span>java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecu=
tor.java:932)</div><div><span style=3D"white-space:pre"> </span>java.util.c=
oncurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)</div><di=
v><span style=3D"white-space:pre"> </span>org.jgroups.protocols.TP.submitTo=
ThreadPool(TP.java:1380)</div><div><span style=3D"white-space:pre"> </span>=
org.jgroups.protocols.TP.submitToThreadPool(TP.java:1375)</div><div><span s=
tyle=3D"white-space:pre"> </span>org.jgroups.util.SubmitToThreadPool.loopba=
ck(SubmitToThreadPool.java:30)</div><div><span style=3D"white-space:pre"> <=
/span>org.jgroups.util.MaxOneThreadPerSender.loopback(MaxOneThreadPerSender=
.java:54)</div><div><span style=3D"white-space:pre"> </span>org.jgroups.pro=
tocols.TP.loopback(TP.java:1188)</div><div><span style=3D"white-space:pre">=
</span>org.jgroups.protocols.TP.down(TP.java:1107)</div><div><span style=
=3D"white-space:pre"> </span>org.jgroups.stack.Protocol.down(Protocol.java:=
318)</div><div><span style=3D"white-space:pre"> </span>org.jgroups.stack.Pr=
otocol.down(Protocol.java:318)</div><div><span style=3D"white-space:pre"> <=
/span>org.jgroups.protocols.FD_ALL$HeartbeatSender.run(FD_ALL.java:413)</di=
v><div><span style=3D"white-space:pre"> </span>org.jgroups.util.TimeSchedul=
er3$Task.run(TimeScheduler3.java:324)</div><div><span style=3D"white-space:=
pre"> </span>org.jgroups.util.TimeScheduler3$RecurringTask.run(TimeSchedule=
r3.java:358)</div><div><span style=3D"white-space:pre"> </span>java.util.co=
ncurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)</div><d=
iv>TRACE 387456:</div><div><span style=3D"white-space:pre"> </span>java.lan=
g.ref.Reference.<init>(Reference.java:306)</div><div><span style=3D"w=
hite-space:pre"> </span>java.lang.ref.Reference.<init>(Reference.java=
:303)</div><div><span style=3D"white-space:pre"> </span>java.lang.ref.WeakR=
eference.<init>(WeakReference.java:57)</div><div><span style=3D"white=
-space:pre"> </span>org.jgroups.util.LazyThreadFactory.newThread(LazyThread=
Factory.java:36)</div><div><span style=3D"white-space:pre"> </span>org.jgro=
ups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:23)</div><div><=
span style=3D"white-space:pre"> </span>java.util.concurrent.ThreadPoolExecu=
tor$Worker.<init>(ThreadPoolExecutor.java:619)</div><div><span style=
=3D"white-space:pre"> </span>java.util.concurrent.ThreadPoolExecutor.addWor=
ker(ThreadPoolExecutor.java:932)</div><div><span style=3D"white-space:pre">=
</span>java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.=
java:1378)</div><div><span style=3D"white-space:pre"> </span>org.jgroups.pr=
otocols.TP.submitToThreadPool(TP.java:1380)</div><div><span style=3D"white-=
space:pre"> </span>org.jgroups.protocols.TP.submitToThreadPool(TP.java:1375=
)</div><div><span style=3D"white-space:pre"> </span>org.jgroups.util.Submit=
ToThreadPool.process(SubmitToThreadPool.java:34)</div><div><span style=3D"w=
hite-space:pre"> </span>org.jgroups.util.MaxOneThreadPerSender.process(MaxO=
neThreadPerSender.java:63)</div><div><span style=3D"white-space:pre"> </spa=
n>org.jgroups.protocols.TP.handleSingleMessage(TP.java:1352)</div><div><spa=
n style=3D"white-space:pre"> </span>org.jgroups.protocols.TP.receive(TP.jav=
a:1323)</div><div><span style=3D"white-space:pre"> </span>org.jgroups.block=
s.cs.BaseServer.receive(BaseServer.java:171)</div><div><span style=3D"white=
-space:pre"> </span>org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpCon=
nection.java:295)</div><div><span style=3D"white-space:pre"> </span>java.la=
ng.Thread.run(Thread.java:748)</div><div>TRACE 387458:</div><div><span styl=
e=3D"white-space:pre"> </span>java.util.concurrent.ConcurrentLinkedQueue$No=
de.<init>(ConcurrentLinkedQueue.java:188)</div><div><span style=3D"wh=
ite-space:pre"> </span>java.util.concurrent.ConcurrentLinkedQueue.offer(Con=
currentLinkedQueue.java:328)</div><div><span style=3D"white-space:pre"> </s=
pan>java.util.concurrent.ConcurrentLinkedQueue.add(ConcurrentLinkedQueue.ja=
va:297)</div><div><span style=3D"white-space:pre"> </span>org.jgroups.util.=
LazyThreadFactory.newThread(LazyThreadFactory.java:36)</div><div><span styl=
e=3D"white-space:pre"> </span>org.jgroups.util.LazyThreadFactory.newThread(=
LazyThreadFactory.java:23)</div><div><span style=3D"white-space:pre"> </spa=
n>java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExe=
cutor.java:619)</div><div><span style=3D"white-space:pre"> </span>java.util=
.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932)</div>=
<div><span style=3D"white-space:pre"> </span>java.util.concurrent.ThreadPoo=
lExecutor.execute(ThreadPoolExecutor.java:1378)</div><div><span style=3D"wh=
ite-space:pre"> </span>org.jgroups.protocols.TP.submitToThreadPool(TP.java:=
1380)</div><div><span style=3D"white-space:pre"> </span>org.jgroups.protoco=
ls.TP.submitToThreadPool(TP.java:1375)</div><div><span style=3D"white-space=
:pre"> </span>org.jgroups.util.SubmitToThreadPool.process(SubmitToThreadPoo=
l.java:34)</div><div><span style=3D"white-space:pre"> </span>org.jgroups.ut=
il.MaxOneThreadPerSender.process(MaxOneThreadPerSender.java:63)</div><div><=
span style=3D"white-space:pre"> </span>org.jgroups.protocols.TP.handleSingl=
eMessage(TP.java:1352)</div><div><span style=3D"white-space:pre"> </span>or=
g.jgroups.protocols.TP.receive(TP.java:1323)</div><div><span style=3D"white=
-space:pre"> </span>org.jgroups.blocks.cs.BaseServer.receive(BaseServer.jav=
a:171)</div><div><span style=3D"white-space:pre"> </span>org.jgroups.blocks=
.cs.TcpConnection$Receiver.run(TcpConnection.java:295)</div><div><span styl=
e=3D"white-space:pre"> </span>java.lang.Thread.run(Thread.java:748)</div><d=
iv>TRACE 416748:</div><div><span style=3D"white-space:pre"> </span>java.lan=
g.ref.Reference.<init>(Reference.java:306)</div><div><span style=3D"w=
hite-space:pre"> </span>java.lang.ref.Reference.<init>(Reference.java=
:303)</div><div><span style=3D"white-space:pre"> </span>java.lang.ref.WeakR=
eference.<init>(WeakReference.java:57)</div><div><span style=3D"white=
-space:pre"> </span>org.jgroups.util.LazyThreadFactory.newThread(LazyThread=
Factory.java:36)</div><div><span style=3D"white-space:pre"> </span>org.jgro=
ups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:23)</div><div><=
span style=3D"white-space:pre"> </span>java.util.concurrent.ThreadPoolExecu=
tor$Worker.<init>(ThreadPoolExecutor.java:619)</div><div><span style=
=3D"white-space:pre"> </span>java.util.concurrent.ThreadPoolExecutor.addWor=
ker(ThreadPoolExecutor.java:932)</div><div><span style=3D"white-space:pre">=
</span>java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.=
java:1378)</div><div><span style=3D"white-space:pre"> </span>org.jgroups.ut=
il.TimeScheduler3.submitToPool(TimeScheduler3.java:231)</div><div><span sty=
le=3D"white-space:pre"> </span>org.jgroups.util.TimeScheduler3.run(TimeSche=
duler3.java:182)</div><div><span style=3D"white-space:pre"> </span>java.lan=
g.Thread.run(Thread.java:748)</div><div>TRACE 416749:</div><div><span style=
=3D"white-space:pre"> </span>java.util.concurrent.ConcurrentLinkedQueue$Nod=
e.<init>(ConcurrentLinkedQueue.java:188)</div><div><span style=3D"whi=
te-space:pre"> </span>java.util.concurrent.ConcurrentLinkedQueue.offer(Conc=
urrentLinkedQueue.java:328)</div><div><span style=3D"white-space:pre"> </sp=
an>java.util.concurrent.ConcurrentLinkedQueue.add(ConcurrentLinkedQueue.jav=
a:297)</div><div><span style=3D"white-space:pre"> </span>org.jgroups.util.L=
azyThreadFactory.newThread(LazyThreadFactory.java:36)</div><div><span style=
=3D"white-space:pre"> </span>org.jgroups.util.LazyThreadFactory.newThread(L=
azyThreadFactory.java:23)</div><div><span style=3D"white-space:pre"> </span=
>java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExec=
utor.java:619)</div><div><span style=3D"white-space:pre"> </span>java.util.=
concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932)</div><=
div><span style=3D"white-space:pre"> </span>java.util.concurrent.ThreadPool=
Executor.execute(ThreadPoolExecutor.java:1378)</div><div><span style=3D"whi=
te-space:pre"> </span>org.jgroups.util.TimeScheduler3.submitToPool(TimeSche=
duler3.java:231)</div><div><span style=3D"white-space:pre"> </span>org.jgro=
ups.util.TimeScheduler3.run(TimeScheduler3.java:182)</div><div><span style=
=3D"white-space:pre"> </span>java.lang.Thread.run(Thread.java:748)</div></d=
iv><div>--- end ---</div><div><br></div><div>Below is the code we use to cr=
eate the jchannel. We're using jgroups version=C2=A04.0.13. Can you tel=
l me if this is something that can be fixed, or if there's any way my c=
ode could be doing something to prevent these objects from being collected?=
</div><div><br></div><div>This is a product we sell, rather than something =
running within our company, so I don't have the ability to restart it e=
very couple months unless I add something in the next version to try to do =
so.</div><div><br></div><div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 List<Proto=
col> stack =3D new ArrayList<>();</div><div>=C2=A0 =C2=A0 =C2=A0 =
=C2=A0 stack.add(new TCP()</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 .setValue("bind_addr", InetAddress.getByName(getBindingAddres=
s()))</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("b=
ind_port", bindingPort)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 .setValue("thread_pool_min_threads", 1)</div><div>=C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("thread_pool_keep_alive_t=
ime", 5000)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setVa=
lue("send_buf_size", 640000)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 .setValue("sock_conn_timeout", 300)</div><div>=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("recv_buf_size&quo=
t;, 5000000));</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 stack.add(new TCPPING(=
)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("initi=
al_hosts", parseHostList())</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 .setValue("send_cache_on_join", true)</div><div>=C2=A0=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("port_range", 0));<=
/div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 stack.add(new MERGE3()</div><div>=C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("min_interval", =
10000)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("=
max_interval", 30000));</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 stack.ad=
d(new FD_ALL()</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValu=
e("timeout", getJGroupsTotalTimeout() * 1000L));</div><div>=C2=A0=
=C2=A0 =C2=A0 =C2=A0 stack.add(new VERIFY_SUSPECT()</div><div>=C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("timeout", 1500));</div=
><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 stack.add(new BARRIER());</div><div>=C2=
=A0 =C2=A0 =C2=A0 =C2=A0 stack.add(new NAKACK2()</div><div>=C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("use_mcast_xmit", false));<=
/div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 stack.add(new UNICAST3());</div><div>=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 stack.add(new STABLE()</div><div>=C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("desired_avg_gossip", 50000=
)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("max_b=
ytes", 4000000));</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 stack.add(crea=
teAuthProtocol(....));</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 stack.add(new =
GMS()</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("j=
oin_timeout", 3000));</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 stack.add(=
new MFC()</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue(&qu=
ot;max_credits", 2000000)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=
=C2=A0 .setValue("min_credits", 800000));</div><div>=C2=A0 =C2=
=A0 =C2=A0 =C2=A0 stack.add(new FRAG2());</div><div>=C2=A0 =C2=A0 =C2=A0 =
=C2=A0 stack.add(new STATE_TRANSFER());</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=
=A0 return new JChannel(stack);</div></div><div><br></div><div>Thank you,</=
div><div>Bobby</div><div><br></div></div></div></div></div></div></div></di=
v>
--0000000000006ee3ed05883bc2fc--
--===============0639019060822780452==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
--===============0639019060822780452==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users
--===============0639019060822780452==--