Re: [jgroups-users] OOM error after long run with jgroups
Questions/problems related to using JGroups <[email protected]> Tue, 7 May 2019 12:00:24 -0400
| Newsgroups | gmane.comp.java.javagroups.general |
|---|---|
| Message-ID | <mailman.45752.1557244871.1518.javagroups-users@lists.sourceforge.net> |
--===============8468180602418256384== Content-Type: multipart/alternative; boundary="000000000000c1e14005884e520e" --000000000000c1e14005884e520e Content-Type: text/plain; charset="UTF-8" Awesome, that looks like it. When https://issues.jboss.org/browse/JGRP-2343 is also resolved will pick up the next jgroups. Will that be 4.1.0? Thanks again, Bobby On Mon, May 6, 2019 at 2:44 PM Questions/problems related to using JGroups < [email protected]> wrote: > Hi Bobby > > you might have hit [1] which was fixed in 4.0.16 > > [1] https://issues.jboss.org/browse/JGRP-2312 > > On 06.05.19 19:51, Questions/problems related to using JGroups wrote: > > [Sorry, this is as short as I thought I could make it. :) ] > > > > Hi again, > > > > I asked about this at the end of last year just in case, but didn't have > > any good info at the time. After a longevity test ~4 months, I'm seeing > > my jgroups cluster members hit an OOM issue. I ran two clusters, one > > with JVM option to get a memory dump when needed: > > > > -agentlib:hprof=heap=sites,depth=20,file=/tmp/<file name> > > > > ...and one without, both with jvm option -Xmx128m set. Then I watched > > the memory with 'jstat -gc' and checked in on them every now and then. > > There were only a couple full GCs in the first couple months, but > > eventually the FGCs were happening minutes apart and eventually the > > nodes started to hit OOM. I can share the full jstat output if you're > > curious. > > > > This is the top of the heap allocation near the end of the run, when > > FGCs were <1 minute apart (I can share the whole memory file, but it's > > over 300 meg): > > > > --- begin --- > > SITES BEGIN (ordered by live bytes) Thu May 2 23:46:35 2019 > > percent live alloc'ed stack class > > rank self accum bytes objs bytes objs trace name > > 1 39.77% 39.77% 38014944 1187967 38014944 1187967 415369 > > java.lang.ref.WeakReference > > 2 29.83% 69.60% 28511208 1187967 28511208 1187967 415370 > > java.util.concurrent.ConcurrentLinkedQueue$Node > > 3 13.68% 83.28% 13077664 408677 13077664 408677 387456 > > java.lang.ref.WeakReference > > 4 10.26% 93.54% 9808248 408677 9808248 408677 387458 > > java.util.concurrent.ConcurrentLinkedQueue$Node > > 5 0.59% 94.13% 562432 17576 562432 17576 416748 > > java.lang.ref.WeakReference > > 6 0.44% 94.57% 421824 17576 421824 17576 416749 > > java.util.concurrent.ConcurrentLinkedQueue$Node > > --- end --- > > > > And these are the traces for those, though the last two are negligible. > > I just wanted to include them because they're also jgroups allocations: > > > > --- begin --- > > TRACE 415369: > > java.lang.ref.Reference.<init>(Reference.java:306) > > java.lang.ref.Reference.<init>(Reference.java:303) > > java.lang.ref.WeakReference.<init>(WeakReference.java:57) > > org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:36) > > org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:23) > > > java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:619) > > > java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) > > > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378) > > org.jgroups.protocols.TP.submitToThreadPool(TP.java:1380) > > org.jgroups.protocols.TP.submitToThreadPool(TP.java:1375) > > org.jgroups.util.SubmitToThreadPool.loopback(SubmitToThreadPool.java:30) > > > org.jgroups.util.MaxOneThreadPerSender.loopback(MaxOneThreadPerSender.java:54) > > org.jgroups.protocols.TP.loopback(TP.java:1188) > > org.jgroups.protocols.TP.down(TP.java:1107) > > org.jgroups.stack.Protocol.down(Protocol.java:318) > > org.jgroups.stack.Protocol.down(Protocol.java:318) > > org.jgroups.protocols.FD_ALL$HeartbeatSender.run(FD_ALL.java:413) > > org.jgroups.util.TimeScheduler3$Task.run(TimeScheduler3.java:324) > > > org.jgroups.util.TimeScheduler3$RecurringTask.run(TimeScheduler3.java:358) > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > TRACE 415370: > > > java.util.concurrent.ConcurrentLinkedQueue$Node.<init>(ConcurrentLinkedQueue.java:188) > > > java.util.concurrent.ConcurrentLinkedQueue.offer(ConcurrentLinkedQueue.java:328) > > > java.util.concurrent.ConcurrentLinkedQueue.add(ConcurrentLinkedQueue.java:297) > > org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:36) > > org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:23) > > > java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:619) > > > java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) > > > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378) > > org.jgroups.protocols.TP.submitToThreadPool(TP.java:1380) > > org.jgroups.protocols.TP.submitToThreadPool(TP.java:1375) > > org.jgroups.util.SubmitToThreadPool.loopback(SubmitToThreadPool.java:30) > > > org.jgroups.util.MaxOneThreadPerSender.loopback(MaxOneThreadPerSender.java:54) > > org.jgroups.protocols.TP.loopback(TP.java:1188) > > org.jgroups.protocols.TP.down(TP.java:1107) > > org.jgroups.stack.Protocol.down(Protocol.java:318) > > org.jgroups.stack.Protocol.down(Protocol.java:318) > > org.jgroups.protocols.FD_ALL$HeartbeatSender.run(FD_ALL.java:413) > > org.jgroups.util.TimeScheduler3$Task.run(TimeScheduler3.java:324) > > > org.jgroups.util.TimeScheduler3$RecurringTask.run(TimeScheduler3.java:358) > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > TRACE 387456: > > java.lang.ref.Reference.<init>(Reference.java:306) > > java.lang.ref.Reference.<init>(Reference.java:303) > > java.lang.ref.WeakReference.<init>(WeakReference.java:57) > > org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:36) > > org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:23) > > > java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:619) > > > java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) > > > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378) > > org.jgroups.protocols.TP.submitToThreadPool(TP.java:1380) > > org.jgroups.protocols.TP.submitToThreadPool(TP.java:1375) > > org.jgroups.util.SubmitToThreadPool.process(SubmitToThreadPool.java:34) > > > org.jgroups.util.MaxOneThreadPerSender.process(MaxOneThreadPerSender.java:63) > > org.jgroups.protocols.TP.handleSingleMessage(TP.java:1352) > > org.jgroups.protocols.TP.receive(TP.java:1323) > > org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171) > > org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:295) > > java.lang.Thread.run(Thread.java:748) > > TRACE 387458: > > > java.util.concurrent.ConcurrentLinkedQueue$Node.<init>(ConcurrentLinkedQueue.java:188) > > > java.util.concurrent.ConcurrentLinkedQueue.offer(ConcurrentLinkedQueue.java:328) > > > java.util.concurrent.ConcurrentLinkedQueue.add(ConcurrentLinkedQueue.java:297) > > org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:36) > > org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:23) > > > java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:619) > > > java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) > > > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378) > > org.jgroups.protocols.TP.submitToThreadPool(TP.java:1380) > > org.jgroups.protocols.TP.submitToThreadPool(TP.java:1375) > > org.jgroups.util.SubmitToThreadPool.process(SubmitToThreadPool.java:34) > > > org.jgroups.util.MaxOneThreadPerSender.process(MaxOneThreadPerSender.java:63) > > org.jgroups.protocols.TP.handleSingleMessage(TP.java:1352) > > org.jgroups.protocols.TP.receive(TP.java:1323) > > org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171) > > org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:295) > > java.lang.Thread.run(Thread.java:748) > > TRACE 416748: > > java.lang.ref.Reference.<init>(Reference.java:306) > > java.lang.ref.Reference.<init>(Reference.java:303) > > java.lang.ref.WeakReference.<init>(WeakReference.java:57) > > org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:36) > > org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:23) > > > java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:619) > > > java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) > > > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378) > > org.jgroups.util.TimeScheduler3.submitToPool(TimeScheduler3.java:231) > > org.jgroups.util.TimeScheduler3.run(TimeScheduler3.java:182) > > java.lang.Thread.run(Thread.java:748) > > TRACE 416749: > > > java.util.concurrent.ConcurrentLinkedQueue$Node.<init>(ConcurrentLinkedQueue.java:188) > > > java.util.concurrent.ConcurrentLinkedQueue.offer(ConcurrentLinkedQueue.java:328) > > > java.util.concurrent.ConcurrentLinkedQueue.add(ConcurrentLinkedQueue.java:297) > > org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:36) > > org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:23) > > > java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:619) > > > java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) > > > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378) > > org.jgroups.util.TimeScheduler3.submitToPool(TimeScheduler3.java:231) > > org.jgroups.util.TimeScheduler3.run(TimeScheduler3.java:182) > > java.lang.Thread.run(Thread.java:748) > > --- end --- > > > > Below is the code we use to create the jchannel. We're using jgroups > > version 4.0.13. Can you tell me if this is something that can be fixed, > > or if there's any way my code could be doing something to prevent these > > objects from being collected? > > > > This is a product we sell, rather than something running within our > > company, so I don't have the ability to restart it every couple months > > unless I add something in the next version to try to do so. > > > > List<Protocol> stack = new ArrayList<>(); > > stack.add(new TCP() > > .setValue("bind_addr", > > InetAddress.getByName(getBindingAddress())) > > .setValue("bind_port", bindingPort) > > .setValue("thread_pool_min_threads", 1) > > .setValue("thread_pool_keep_alive_time", 5000) > > .setValue("send_buf_size", 640000) > > .setValue("sock_conn_timeout", 300) > > .setValue("recv_buf_size", 5000000)); > > stack.add(new TCPPING() > > .setValue("initial_hosts", parseHostList()) > > .setValue("send_cache_on_join", true) > > .setValue("port_range", 0)); > > stack.add(new MERGE3() > > .setValue("min_interval", 10000) > > .setValue("max_interval", 30000)); > > stack.add(new FD_ALL() > > .setValue("timeout", getJGroupsTotalTimeout() * 1000L)); > > stack.add(new VERIFY_SUSPECT() > > .setValue("timeout", 1500)); > > stack.add(new BARRIER()); > > stack.add(new NAKACK2() > > .setValue("use_mcast_xmit", false)); > > stack.add(new UNICAST3()); > > stack.add(new STABLE() > > .setValue("desired_avg_gossip", 50000) > > .setValue("max_bytes", 4000000)); > > stack.add(createAuthProtocol(....)); > > stack.add(new GMS() > > .setValue("join_timeout", 3000)); > > stack.add(new MFC() > > .setValue("max_credits", 2000000) > > .setValue("min_credits", 800000)); > > stack.add(new FRAG2()); > > stack.add(new STATE_TRANSFER()); > > return new JChannel(stack); > > > > Thank you, > > Bobby > > > > > > > > _______________________________________________ > > javagroups-users mailing list > > [email protected] > > https://lists.sourceforge.net/lists/listinfo/javagroups-users > > > > -- > Bela Ban | http://www.jgroups.org > > > > _______________________________________________ > javagroups-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/javagroups-users > --000000000000c1e14005884e520e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr">Awesome, that looks like it. When=C2=A0<a= href=3D"https://issues.jboss.org/browse/JGRP-2343" target=3D"_blank">https= ://issues.jboss.org/browse/JGRP-2343</a> is also resolved will pick up the = next jgroups. Will that be 4.1.0?</div><div dir=3D"ltr"><br></div><div>Than= ks again,</div><div>Bobby</div><div><br></div></div><br><div class=3D"gmail= _quote"><div dir=3D"ltr" class=3D"gmail_attr">On Mon, May 6, 2019 at 2:44 P= M Questions/problems related to using JGroups <<a href=3D"mailto:javagro= [email protected]" target=3D"_blank">[email protected]= ourceforge.net</a>> wrote:<br></div><blockquote class=3D"gmail_quote" st= yle=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padd= ing-left:1ex">Hi Bobby<br> <br> you might have hit [1] which was fixed in 4.0.16<br> <br> [1] <a href=3D"https://issues.jboss.org/browse/JGRP-2312" rel=3D"noreferrer= " target=3D"_blank">https://issues.jboss.org/browse/JGRP-2312</a><br> <br> On 06.05.19 19:51, Questions/problems related to using JGroups wrote:<br> > [Sorry, this is as short as I thought I could make it. :) ]<br> > <br> > Hi again,<br> > <br> > I asked about this at the end of last year just in case, but didn'= t have <br> > any good info at the time. After a longevity test ~4 months, I'm s= eeing <br> > my jgroups cluster members hit an OOM issue. I ran two clusters, one <= br> > with JVM option to get a memory dump when needed:<br> > <br> > -agentlib:hprof=3Dheap=3Dsites,depth=3D20,file=3D/tmp/<file name>= ;<br> > <br> > ...and one without, both with=C2=A0jvm option -Xmx128m set. Then I wat= ched <br> > the memory with 'jstat -gc' and checked in on them every now a= nd then. <br> > There were only a couple full GCs in the first couple months, but <br> > eventually the FGCs were happening minutes apart and eventually the <b= r> > nodes started to hit OOM. I can share the full jstat output if you'= ;re <br> > curious.<br> > <br> > This is the top of the heap allocation near the end of the run, when <= br> > FGCs were <1 minute apart (I can share the whole memory file, but i= t's <br> > over 300 meg):<br> > <br> > --- begin ---<br> > SITES BEGIN (ordered by live bytes) Thu May=C2=A0 2 23:46:35 2019<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 percent=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 live=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 alloc'ed=C2=A0 sta= ck class<br> >=C2=A0 =C2=A0rank=C2=A0 =C2=A0self=C2=A0 accum=C2=A0 =C2=A0 =C2=A0bytes= objs=C2=A0 =C2=A0 =C2=A0bytes=C2=A0 objs trace name<br> >=C2=A0 =C2=A0 =C2=A0 1 39.77% 39.77%=C2=A0 38014944 1187967=C2=A0 38014= 944 1187967 415369 <br> > java.lang.ref.WeakReference<br> >=C2=A0 =C2=A0 =C2=A0 2 29.83% 69.60%=C2=A0 28511208 1187967=C2=A0 28511= 208 1187967 415370 <br> > java.util.concurrent.ConcurrentLinkedQueue$Node<br> >=C2=A0 =C2=A0 =C2=A0 3 13.68% 83.28%=C2=A0 13077664 408677=C2=A0 130776= 64 408677 387456 <br> > java.lang.ref.WeakReference<br> >=C2=A0 =C2=A0 =C2=A0 4 10.26% 93.54%=C2=A0 =C2=A09808248 408677=C2=A0 = =C2=A09808248 408677 387458 <br> > java.util.concurrent.ConcurrentLinkedQueue$Node<br> >=C2=A0 =C2=A0 =C2=A0 5=C2=A0 0.59% 94.13%=C2=A0 =C2=A0 562432 17576=C2= =A0 =C2=A0 562432 17576 416748 <br> > java.lang.ref.WeakReference<br> >=C2=A0 =C2=A0 =C2=A0 6=C2=A0 0.44% 94.57%=C2=A0 =C2=A0 421824 17576=C2= =A0 =C2=A0 421824 17576 416749 <br> > java.util.concurrent.ConcurrentLinkedQueue$Node<br> > --- end ---<br> > <br> > And these are the traces for those, though the last two are negligible= . <br> > I just wanted to include them because they're also jgroups allocat= ions:<br> > <br> > --- begin ---<br> > TRACE 415369:<br> > java.lang.ref.Reference.<init>(Reference.java:306)<br> > java.lang.ref.Reference.<init>(Reference.java:303)<br> > java.lang.ref.WeakReference.<init>(WeakReference.java:57)<br> > org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:36= )<br> > org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:23= )<br> > java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPool= Executor.java:619)<br> > java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.j= ava:932)<br> > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.jav= a:1378)<br> > <a href=3D"http://org.jgroups.protocols.TP" target=3D"_blank">org.jgro= ups.protocols.TP</a>.submitToThreadPool(TP.java:1380)<br> > <a href=3D"http://org.jgroups.protocols.TP" target=3D"_blank">org.jgro= ups.protocols.TP</a>.submitToThreadPool(TP.java:1375)<br> > org.jgroups.util.SubmitToThreadPool.loopback(SubmitToThreadPool.java:3= 0)<br> > org.jgroups.util.MaxOneThreadPerSender.loopback(MaxOneThreadPerSender.= java:54)<br> > <a href=3D"http://org.jgroups.protocols.TP" target=3D"_blank">org.jgro= ups.protocols.TP</a>.loopback(TP.java:1188)<br> > org.jgroups.protocols.TP.down(TP.java:1107)<br> > org.jgroups.stack.Protocol.down(Protocol.java:318)<br> > org.jgroups.stack.Protocol.down(Protocol.java:318)<br> > org.jgroups.protocols.FD_ALL$HeartbeatSender.run(FD_ALL.java:413)<br> > org.jgroups.util.TimeScheduler3$Task.run(TimeScheduler3.java:324)<br> > org.jgroups.util.TimeScheduler3$RecurringTask.run(TimeScheduler3.java:= 358)<br> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j= ava:1149)<br> > TRACE 415370:<br> > java.util.concurrent.ConcurrentLinkedQueue$Node.<init>(Concurren= tLinkedQueue.java:188)<br> > java.util.concurrent.ConcurrentLinkedQueue.offer(ConcurrentLinkedQueue= .java:328)<br> > java.util.concurrent.ConcurrentLinkedQueue.add(ConcurrentLinkedQueue.j= ava:297)<br> > org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:36= )<br> > org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:23= )<br> > java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPool= Executor.java:619)<br> > java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.j= ava:932)<br> > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.jav= a:1378)<br> > <a href=3D"http://org.jgroups.protocols.TP" target=3D"_blank">org.jgro= ups.protocols.TP</a>.submitToThreadPool(TP.java:1380)<br> > <a href=3D"http://org.jgroups.protocols.TP" target=3D"_blank">org.jgro= ups.protocols.TP</a>.submitToThreadPool(TP.java:1375)<br> > org.jgroups.util.SubmitToThreadPool.loopback(SubmitToThreadPool.java:3= 0)<br> > org.jgroups.util.MaxOneThreadPerSender.loopback(MaxOneThreadPerSender.= java:54)<br> > <a href=3D"http://org.jgroups.protocols.TP" target=3D"_blank">org.jgro= ups.protocols.TP</a>.loopback(TP.java:1188)<br> > org.jgroups.protocols.TP.down(TP.java:1107)<br> > org.jgroups.stack.Protocol.down(Protocol.java:318)<br> > org.jgroups.stack.Protocol.down(Protocol.java:318)<br> > org.jgroups.protocols.FD_ALL$HeartbeatSender.run(FD_ALL.java:413)<br> > org.jgroups.util.TimeScheduler3$Task.run(TimeScheduler3.java:324)<br> > org.jgroups.util.TimeScheduler3$RecurringTask.run(TimeScheduler3.java:= 358)<br> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j= ava:1149)<br> > TRACE 387456:<br> > java.lang.ref.Reference.<init>(Reference.java:306)<br> > java.lang.ref.Reference.<init>(Reference.java:303)<br> > java.lang.ref.WeakReference.<init>(WeakReference.java:57)<br> > org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:36= )<br> > org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:23= )<br> > java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPool= Executor.java:619)<br> > java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.j= ava:932)<br> > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.jav= a:1378)<br> > <a href=3D"http://org.jgroups.protocols.TP" target=3D"_blank">org.jgro= ups.protocols.TP</a>.submitToThreadPool(TP.java:1380)<br> > <a href=3D"http://org.jgroups.protocols.TP" target=3D"_blank">org.jgro= ups.protocols.TP</a>.submitToThreadPool(TP.java:1375)<br> > org.jgroups.util.SubmitToThreadPool.process(SubmitToThreadPool.java:34= )<br> > org.jgroups.util.MaxOneThreadPerSender.process(MaxOneThreadPerSender.j= ava:63)<br> > <a href=3D"http://org.jgroups.protocols.TP" target=3D"_blank">org.jgro= ups.protocols.TP</a>.handleSingleMessage(TP.java:1352)<br> > <a href=3D"http://org.jgroups.protocols.TP" target=3D"_blank">org.jgro= ups.protocols.TP</a>.receive(TP.java:1323)<br> > org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)<br> > org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:29= 5)<br> > java.lang.Thread.run(Thread.java:748)<br> > TRACE 387458:<br> > java.util.concurrent.ConcurrentLinkedQueue$Node.<init>(Concurren= tLinkedQueue.java:188)<br> > java.util.concurrent.ConcurrentLinkedQueue.offer(ConcurrentLinkedQueue= .java:328)<br> > java.util.concurrent.ConcurrentLinkedQueue.add(ConcurrentLinkedQueue.j= ava:297)<br> > org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:36= )<br> > org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:23= )<br> > java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPool= Executor.java:619)<br> > java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.j= ava:932)<br> > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.jav= a:1378)<br> > <a href=3D"http://org.jgroups.protocols.TP" target=3D"_blank">org.jgro= ups.protocols.TP</a>.submitToThreadPool(TP.java:1380)<br> > <a href=3D"http://org.jgroups.protocols.TP" target=3D"_blank">org.jgro= ups.protocols.TP</a>.submitToThreadPool(TP.java:1375)<br> > org.jgroups.util.SubmitToThreadPool.process(SubmitToThreadPool.java:34= )<br> > org.jgroups.util.MaxOneThreadPerSender.process(MaxOneThreadPerSender.j= ava:63)<br> > <a href=3D"http://org.jgroups.protocols.TP" target=3D"_blank">org.jgro= ups.protocols.TP</a>.handleSingleMessage(TP.java:1352)<br> > <a href=3D"http://org.jgroups.protocols.TP" target=3D"_blank">org.jgro= ups.protocols.TP</a>.receive(TP.java:1323)<br> > org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)<br> > org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:29= 5)<br> > java.lang.Thread.run(Thread.java:748)<br> > TRACE 416748:<br> > java.lang.ref.Reference.<init>(Reference.java:306)<br> > java.lang.ref.Reference.<init>(Reference.java:303)<br> > java.lang.ref.WeakReference.<init>(WeakReference.java:57)<br> > org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:36= )<br> > org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:23= )<br> > java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPool= Executor.java:619)<br> > java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.j= ava:932)<br> > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.jav= a:1378)<br> > org.jgroups.util.TimeScheduler3.submitToPool(TimeScheduler3.java:231)<= br> > org.jgroups.util.TimeScheduler3.run(TimeScheduler3.java:182)<br> > java.lang.Thread.run(Thread.java:748)<br> > TRACE 416749:<br> > java.util.concurrent.ConcurrentLinkedQueue$Node.<init>(Concurren= tLinkedQueue.java:188)<br> > java.util.concurrent.ConcurrentLinkedQueue.offer(ConcurrentLinkedQueue= .java:328)<br> > java.util.concurrent.ConcurrentLinkedQueue.add(ConcurrentLinkedQueue.j= ava:297)<br> > org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:36= )<br> > org.jgroups.util.LazyThreadFactory.newThread(LazyThreadFactory.java:23= )<br> > java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPool= Executor.java:619)<br> > java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.j= ava:932)<br> > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.jav= a:1378)<br> > org.jgroups.util.TimeScheduler3.submitToPool(TimeScheduler3.java:231)<= br> > org.jgroups.util.TimeScheduler3.run(TimeScheduler3.java:182)<br> > java.lang.Thread.run(Thread.java:748)<br> > --- end ---<br> > <br> > Below is the code we use to create the jchannel. We're using jgrou= ps <br> > version=C2=A04.0.13. Can you tell me if this is something that can be = fixed, <br> > or if there's any way my code could be doing something to prevent = these <br> > objects from being collected?<br> > <br> > This is a product we sell, rather than something running within our <b= r> > company, so I don't have the ability to restart it every couple mo= nths <br> > unless I add something in the next version to try to do so.<br> > <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 List<Protocol> stack =3D new A= rrayList<>();<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 stack.add(new TCP()<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("bind_a= ddr", <br> > InetAddress.getByName(getBindingAddress()))<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("bind_p= ort", bindingPort)<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("thread= _pool_min_threads", 1)<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("thread= _pool_keep_alive_time", 5000)<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("send_b= uf_size", 640000)<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("sock_c= onn_timeout", 300)<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("recv_b= uf_size", 5000000));<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 stack.add(new TCPPING()<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("initia= l_hosts", parseHostList())<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("send_c= ache_on_join", true)<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("port_r= ange", 0));<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 stack.add(new MERGE3()<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("min_in= terval", 10000)<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("max_in= terval", 30000));<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 stack.add(new FD_ALL()<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("timeou= t", getJGroupsTotalTimeout() * 1000L));<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 stack.add(new VERIFY_SUSPECT()<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("timeou= t", 1500));<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 stack.add(new BARRIER());<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 stack.add(new NAKACK2()<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("use_mc= ast_xmit", false));<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 stack.add(new UNICAST3());<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 stack.add(new STABLE()<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("desire= d_avg_gossip", 50000)<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("max_by= tes", 4000000));<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 stack.add(createAuthProtocol(....));= <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 stack.add(new GMS()<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("join_t= imeout", 3000));<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 stack.add(new MFC()<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("max_cr= edits", 2000000)<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .setValue("min_cr= edits", 800000));<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 stack.add(new FRAG2());<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 stack.add(new STATE_TRANSFER());<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 return new JChannel(stack);<br> > <br> > Thank you,<br> > Bobby<br> > <br> > <br> > <br> > _______________________________________________<br> > javagroups-users mailing list<br> > <a href=3D"mailto:[email protected]" target=3D"_b= lank">[email protected]</a><br> > <a href=3D"https://lists.sourceforge.net/lists/listinfo/javagroups-use= rs" rel=3D"noreferrer" target=3D"_blank">https://lists.sourceforge.net/list= s/listinfo/javagroups-users</a><br> > <br> <br> -- <br> Bela Ban | <a href=3D"http://www.jgroups.org" rel=3D"noreferrer" target=3D"= _blank">http://www.jgroups.org</a><br> <br> <br> <br> _______________________________________________<br> javagroups-users mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank"= >[email protected]</a><br> <a href=3D"https://lists.sourceforge.net/lists/listinfo/javagroups-users" r= el=3D"noreferrer" target=3D"_blank">https://lists.sourceforge.net/lists/lis= tinfo/javagroups-users</a><br> </blockquote></div> --000000000000c1e14005884e520e-- --===============8468180602418256384== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --===============8468180602418256384== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ javagroups-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/javagroups-users --===============8468180602418256384==--