Re: [jgroups-users] TCP transport is blocked
Questions/problems related to using JGroups <[email protected]>
| Newsgroups | gmane.comp.java.javagroups.general |
|---|---|
| Message-ID | <CAB1b6fGM+fKfOzEnM213-84b2qdXFwE7FNkg-ebtPZN1kzoK_w@mail.gmail.com> |
Seems, the patch helped get rid of blocking on a monitor. But, unfortunately, that's not a cause, why nodes re-joins:-( On Mon, Apr 18, 2016 at 5:02 PM, Denis Gabaydulin <[email protected]> wrote: > Thank you! > Have built the 3.9.10-SNAPSHOT from 3.6 branch and installed it on the > test env. with monitoring. > > > On Mon, Apr 18, 2016 at 3:10 PM, Questions/problems related to using > JGroups <[email protected]> wrote: > >> OK, so while I didn't detecta deadlock, the connection receiver thread >> would close a connection in the same synchronized clause, so I pushed >> Connection.close() outside the synchronized scope, see [2]. This means >> Connection.close() does now not compete with Connection.send(), e.g. >> executed in your stack trace by the TransferQueueBundler thread. >> >> This should speed things up. The JIRA is [1]. >> Try it out and let me know if this helps, >> >> [1] https://issues.jboss.org/browse/JGRP-2053 >> [2] >> >> https://github.com/belaban/JGroups/commit/38b91a6b07532f8893337b01e80267c8bbe3a89e >> >> On 17/04/16 15:18, Questions/problems related to using JGroups wrote: >> > Something wrong with formatting :-/ >> > I mean this piece of the stacktrace: >> > >> > >> https://gist.github.com/sherman/1623d539f29bd32d4cf627633e2a9c95#file-gistfile1-txt-L201-L218 >> > >> > On Sun, Apr 17, 2016 at 1:11 PM, Denis Gabaydulin <[email protected] >> > <mailto:[email protected]>> wrote: >> > >> > Also, pls have a look, at Connection.Receiver thread. >> > >> > Connection.Receiver [172.16.42.4:7800 <http://172.16.42.4:7800> - >> > 172.16.42.5:42120 <http://172.16.42.5:42120>],puten,node2" - Thread >> > t@11221 java.lang.Thread.State: WAITING at >> > sun.misc.Unsafe.park(Native Method) - waiting to lock <43ea8685> (a >> > java.util.concurrent.locks.ReentrantLock$NonfairSync) owned by >> > "TransferQueueBundler,puten,node2" t@53 at >> > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at >> > >> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) >> > at >> > >> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) >> > at >> > >> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) >> > at >> > >> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) >> > at >> > >> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at >> > org.jgroups.blocks.cs.TcpConnection.close(TcpConnection.java:429) at >> > org.jgroups.util.Util.close(Util.java:405) at >> > >> org.jgroups.blocks.cs.BaseServer.removeConnectionIfPresent(BaseServer.java:362) >> > - locked <4a73a364> (a org.jgroups.blocks.cs.TcpServer) at >> > >> org.jgroups.blocks.cs.BaseServer.connectionClosed(BaseServer.java:223) >> > at >> > >> org.jgroups.blocks.cs.BaseServer.notifyConnectionClosed(BaseServer.java:393) >> > at >> > >> org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:325) >> > at java.lang.Thread.run(Thread.java:745) Locked ownable >> > synchronizers: - None >> > >> > As you can see we have two monitors here: >> > >> > 43ea8685 and 4a73a364. >> > >> > I'm not familiar with this code, but it can be situation when lock >> > 4a73a364, is waiting for release of 43ea8685. And the last one is >> > released by a timeout. >> > >> > On Apr 17, 2016 12:57 PM, "Denis Gabaydulin" <[email protected] >> > <mailto:[email protected]>> wrote: >> > >> > Here's the full stacktrace. >> > >> > >> https://gist.github.com/sherman/1623d539f29bd32d4cf627633e2a9c95 >> > >> > Yes, it doesn't look like a deadlock. After a few minutes, a >> > monitor is released (nodes has joined back). >> > >> > On Apr 17, 2016 12:42 PM, "Questions/problems related to using >> > JGroups" <[email protected] >> > <mailto:[email protected]>> wrote: >> > >> > OK, so the BaseServer.hasConnection() and >> > BaseServer.retainAll() both >> > block on 4a73a364, but both methods cannot block forever, so >> > this cannot >> > lead to a deadlock. I still need the full stacktrace (like >> > to a paste, >> > not as attachment to an email!)... >> > >> > On 14/04/16 18:52, Questions/problems related to using >> > JGroups wrote: >> > > One more example: >> > > >> > > "ViewHandler,puten,node2" - Thread t@11655 >> > > java.lang.Thread.State: BLOCKED >> > > at >> > >> org.jgroups.blocks.cs.BaseServer.retainAll(BaseServer.java:381) >> > > - waiting to lock <4a73a364> (a >> > org.jgroups.blocks.cs.TcpServer) owned >> > > by "Connection.Receiver [172.16.42.4:7800 >> > <http://172.16.42.4:7800> <http://172.16.42.4:7800> - >> > > 172.16.42.5:42120 <http://172.16.42.5:42120> >> > <http://172.16.42.5:42120>],puten,node2" t@11221 >> > > at org.jgroups.protocols.TCP.retainAll(TCP.java:64) >> > > at >> > >> org.jgroups.protocols.BasicTCP.handleDownEvent(BasicTCP.java:160) >> > > at org.jgroups.protocols.TP.down(TP.java:1452) >> > > at >> org.jgroups.protocols.Discovery.down(Discovery.java:403) >> > > at org.jgroups.protocols.TCPPING.down(TCPPING.java:103) >> > > at org.jgroups.protocols.MERGE3.down(MERGE3.java:227) >> > > at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:374) >> > > at org.jgroups.protocols.FD_ALL.down(FD_ALL.java:220) >> > > at >> > >> org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:92) >> > > at >> > org.jgroups.protocols.pbcast.NAKACK2.down(NAKACK2.java:589) >> > > at org.jgroups.protocols.UNICAST3.down(UNICAST3.java:669) >> > > at >> org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:327) >> > > at >> org.jgroups.protocols.pbcast.GMS.installView(GMS.java:732) >> > > - locked <7f555696> (a org.jgroups.protocols.pbcast.GMS) >> > > at >> > > >> > >> org.jgroups.protocols.pbcast.CoordGmsImpl.handleViewChange(CoordGmsImpl.java:244) >> > > at >> > >> org.jgroups.protocols.pbcast.GMS.castViewChange(GMS.java:617) >> > > at >> > > >> > >> org.jgroups.protocols.pbcast.CoordGmsImpl.handleMembershipChange(CoordGmsImpl.java:221) >> > > at >> > >> org.jgroups.protocols.pbcast.GMS$ViewHandler.process(GMS.java:1697) >> > > at >> > >> org.jgroups.protocols.pbcast.GMS$ViewHandler.run(GMS.java:1651) >> > > at java.lang.Thread.run(Thread.java:745) >> > > >> > > "TcpServer.Acceptor [7800],null,null" - Thread t@51 >> > > java.lang.Thread.State: BLOCKED >> > > at >> > >> org.jgroups.blocks.cs.BaseServer.hasConnection(BaseServer.java:235) >> > > - waiting to lock <4a73a364> (a >> > org.jgroups.blocks.cs.TcpServer) owned >> > > by "Connection.Receiver [172.16.42.4:7800 >> > <http://172.16.42.4:7800> <http://172.16.42.4:7800> - >> > > 172.16.42.5:42120 <http://172.16.42.5:42120> >> > <http://172.16.42.5:42120>],puten,node2" t@11221 >> > > at >> > >> org.jgroups.blocks.cs.TcpServer$Acceptor.handleAccept(TcpServer.java:135) >> > > - locked <2ae2e471> (a >> > org.jgroups.blocks.cs.TcpServer$Acceptor) >> > > at >> > >> org.jgroups.blocks.cs.TcpServer$Acceptor.run(TcpServer.java:117) >> > > at java.lang.Thread.run(Thread.java:745) >> > > >> > > Only these two thread are BLOCKED. >> > > >> > > >> > > >> > > On Mon, Apr 11, 2016 at 7:54 PM, Denis Gabaydulin >> > <[email protected] <mailto:[email protected]> >> > > <mailto:[email protected] <mailto:[email protected]>>> >> wrote: >> > > >> > > Full thread-dump in the attached file. >> > > >> > > On Mon, Apr 11, 2016 at 7:44 PM, Denis Gabaydulin >> > <[email protected] <mailto:[email protected]> >> > > <mailto:[email protected] >> > <mailto:[email protected]>>> wrote: >> > > >> > > Sometimes, I see the following picture in the >> > Thread dump: >> > > >> > > "ViewHandler,puten,node2" - Thread t@11063 >> > > java.lang.Thread.State: BLOCKED >> > > at >> > >> org.jgroups.blocks.cs.BaseServer.retainAll(BaseServer.java:381) >> > > - waiting to lock <4a73a364> (a >> > org.jgroups.blocks.cs.TcpServer) >> > > owned by "Connection.Receiver [172.16.42.4:42606 >> > <http://172.16.42.4:42606> >> > > <http://172.16.42.4:42606> - 172.16.42.3:7800 >> > <http://172.16.42.3:7800> >> > > <http://172.16.42.3:7800>],puten,node2" t@11058 >> > > at >> org.jgroups.protocols.TCP.retainAll(TCP.java:64) >> > > at >> > >> org.jgroups.protocols.BasicTCP.handleDownEvent(BasicTCP.java:160) >> > > at org.jgroups.protocols.TP.down(TP.java:1452) >> > > at >> > org.jgroups.protocols.Discovery.down(Discovery.java:403) >> > > at >> > org.jgroups.protocols.TCPPING.down(TCPPING.java:103) >> > > at >> org.jgroups.protocols.MERGE3.down(MERGE3.java:227) >> > > at >> > org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:374) >> > > at >> org.jgroups.protocols.FD_ALL.down(FD_ALL.java:220) >> > > at >> > >> org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:92) >> > > at >> > org.jgroups.protocols.pbcast.NAKACK2.down(NAKACK2.java:589) >> > > at >> > org.jgroups.protocols.UNICAST3.down(UNICAST3.java:669) >> > > at >> > org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:327) >> > > at >> > org.jgroups.protocols.pbcast.GMS.installView(GMS.java:732) >> > > - locked <7f555696> (a >> > org.jgroups.protocols.pbcast.GMS) >> > > at >> > > >> > >> org.jgroups.protocols.pbcast.CoordGmsImpl.handleViewChange(CoordGmsImpl.java:244) >> > > at >> > >> org.jgroups.protocols.pbcast.GMS.castViewChange(GMS.java:617) >> > > at >> > > >> > >> org.jgroups.protocols.pbcast.CoordGmsImpl.handleMembershipChange(CoordGmsImpl.java:221) >> > > at >> > > >> > >> org.jgroups.protocols.pbcast.GMS$ViewHandler.process(GMS.java:1697) >> > > at >> > >> org.jgroups.protocols.pbcast.GMS$ViewHandler.run(GMS.java:1651) >> > > at java.lang.Thread.run(Thread.java:745) >> > > >> > > When it happens, a cluster loses a node for a >> while. >> > > How can it be fixed? >> > > >> > > I use the following config: >> > > >> > > <config xmlns="urn:org:jgroups" >> > > >> > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >> > > xsi:schemaLocation="urn:org:jgroups >> > > http://www.jgroups.org/schema/JGroups-3.6.1.xsd"> >> > > <TCP >> > > bind_port="7800" >> > > timer_type="new3" >> > > timer.min_threads="4" >> > > timer.max_threads="10" >> > > timer.keep_alive_time="3000" >> > > timer.queue_max_size="500" >> > > thread_pool.enabled="true" >> > > thread_pool.min_threads="4" >> > > thread_pool.max_threads="64" >> > > thread_pool.keep_alive_time="20000" >> > > thread_pool.queue_enabled="false" >> > > thread_pool.queue_max_size="100" >> > > thread_pool.rejection_policy="discard" >> > > >> > > oob_thread_pool.enabled="true" >> > > oob_thread_pool.min_threads="2" >> > > oob_thread_pool.max_threads="8" >> > > oob_thread_pool.keep_alive_time="20000" >> > > oob_thread_pool.queue_enabled="false" >> > > oob_thread_pool.queue_max_size="100" >> > > >> oob_thread_pool.rejection_policy="discard" >> > > /> >> > > <TCPPING >> > > >> > >> initial_hosts="${jgroups.tcpping.initial_hosts:172.16.42.3[7800],172.16.42.4[7800],172.16.42.5[7800]}" >> > > port_range="0"/> >> > > <MERGE3/> >> > > <FD_SOCK/> >> > > <FD_ALL timeout="90000"/> >> > > <VERIFY_SUSPECT/> >> > > <pbcast.NAKACK2 use_mcast_xmit="false"/> >> > > <UNICAST3/> >> > > <pbcast.STABLE/> >> > > <pbcast.GMS/> >> > > <MFC/> >> > > <FRAG2/> >> > > <pbcast.STATE_TRANSFER /> >> > > <CENTRAL_LOCK/> >> > > </config> >> > > >> > > Jgroups version is: 3.6.7 FINAL. >> > > >> > > When I trying to probe the problem node at that >> > moment (when it >> > > blocked), I see: Found no valid hosts - >> terminating >> > > >> > > >> > > >> > > >> > > >> > > >> > >> ------------------------------------------------------------------------------ >> > > Find and fix application performance issues faster with >> > Applications Manager >> > > Applications Manager provides deep performance insights >> > into multiple tiers of >> > > your business applications. It resolves application >> > problems quickly and >> > > reduces your MTTR. Get your free trial! >> > > https://ad.doubleclick.net/ddm/clk/302982198;130105516;z >> > > >> > > >> > > >> > > _______________________________________________ >> > > javagroups-users mailing list >> > > [email protected] >> > <mailto:[email protected]> >> > > >> https://lists.sourceforge.net/lists/listinfo/javagroups-users >> > > >> > >> > -- >> > Bela Ban, JGroups lead (http://www.jgroups.org) >> > >> > >> > >> ------------------------------------------------------------------------------ >> > Find and fix application performance issues faster with >> > Applications Manager >> > Applications Manager provides deep performance insights into >> > multiple tiers of >> > your business applications. It resolves application problems >> > quickly and >> > reduces your MTTR. Get your free trial! >> > https://ad.doubleclick.net/ddm/clk/302982198;130105516;z >> > _______________________________________________ >> > javagroups-users mailing list >> > [email protected] >> > <mailto:[email protected]> >> > >> https://lists.sourceforge.net/lists/listinfo/javagroups-users >> > >> > >> > >> > >> > >> ------------------------------------------------------------------------------ >> > Find and fix application performance issues faster with Applications >> Manager >> > Applications Manager provides deep performance insights into multiple >> tiers of >> > your business applications. It resolves application problems quickly and >> > reduces your MTTR. Get your free trial! >> > https://ad.doubleclick.net/ddm/clk/302982198;130105516;z >> > >> > >> > >> > _______________________________________________ >> > javagroups-users mailing list >> > [email protected] >> > https://lists.sourceforge.net/lists/listinfo/javagroups-users >> > >> >> -- >> Bela Ban, JGroups lead (http://www.jgroups.org) >> >> >> >> ------------------------------------------------------------------------------ >> Find and fix application performance issues faster with Applications >> Manager >> Applications Manager provides deep performance insights into multiple >> tiers of >> your business applications. It resolves application problems quickly and >> reduces your MTTR. Get your free trial! >> https://ad.doubleclick.net/ddm/clk/302982198;130105516;z >> _______________________________________________ >> javagroups-users mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/javagroups-users >> > > ------------------------------------------------------------------------------ Find and fix application performance issues faster with Applications Manager Applications Manager provides deep performance insights into multiple tiers of your business applications. It resolves application problems quickly and reduces your MTTR. Get your free trial! https://ad.doubleclick.net/ddm/clk/302982198;130105516;z _______________________________________________ javagroups-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/javagroups-users