Re: [jgroups-users] TCP transport is blocked

Questions/problems related to using JGroups <[email protected]>
Newsgroups gmane.comp.java.javagroups.general
Message-ID <CAB1b6fFBk2VpuLd7zu+p469F1TcuLGQXbrdRz-AM36iyD+FQvQ@mail.gmail.com> 
Here's the full stacktrace.

https://gist.github.com/sherman/1623d539f29bd32d4cf627633e2a9c95

Yes, it doesn't look like a deadlock. After a few minutes, a monitor is
released (nodes has joined back).
On Apr 17, 2016 12:42 PM, "Questions/problems related to using JGroups" <
[email protected]> wrote:

> OK, so the BaseServer.hasConnection() and BaseServer.retainAll() both
> block on 4a73a364, but both methods cannot block forever, so this cannot
> lead to a deadlock. I still need the full stacktrace (like to a paste,
> not as attachment to an email!)...
>
> On 14/04/16 18:52, Questions/problems related to using JGroups wrote:
> > One more example:
> >
> > "ViewHandler,puten,node2" - Thread t@11655
> >     java.lang.Thread.State: BLOCKED
> > at org.jgroups.blocks.cs.BaseServer.retainAll(BaseServer.java:381)
> > - waiting to lock <4a73a364> (a org.jgroups.blocks.cs.TcpServer) owned
> > by "Connection.Receiver [172.16.42.4:7800 <http://172.16.42.4:7800> -
> > 172.16.42.5:42120 <http://172.16.42.5:42120>],puten,node2" t@11221
> > at org.jgroups.protocols.TCP.retainAll(TCP.java:64)
> > at org.jgroups.protocols.BasicTCP.handleDownEvent(BasicTCP.java:160)
> > at org.jgroups.protocols.TP.down(TP.java:1452)
> > at org.jgroups.protocols.Discovery.down(Discovery.java:403)
> > at org.jgroups.protocols.TCPPING.down(TCPPING.java:103)
> > at org.jgroups.protocols.MERGE3.down(MERGE3.java:227)
> > at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:374)
> > at org.jgroups.protocols.FD_ALL.down(FD_ALL.java:220)
> > at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:92)
> > at org.jgroups.protocols.pbcast.NAKACK2.down(NAKACK2.java:589)
> > at org.jgroups.protocols.UNICAST3.down(UNICAST3.java:669)
> > at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:327)
> > at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:732)
> > - locked <7f555696> (a org.jgroups.protocols.pbcast.GMS)
> > at
> >
> org.jgroups.protocols.pbcast.CoordGmsImpl.handleViewChange(CoordGmsImpl.java:244)
> > at org.jgroups.protocols.pbcast.GMS.castViewChange(GMS.java:617)
> > at
> >
> org.jgroups.protocols.pbcast.CoordGmsImpl.handleMembershipChange(CoordGmsImpl.java:221)
> > at org.jgroups.protocols.pbcast.GMS$ViewHandler.process(GMS.java:1697)
> > at org.jgroups.protocols.pbcast.GMS$ViewHandler.run(GMS.java:1651)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "TcpServer.Acceptor [7800],null,null" - Thread t@51
> >     java.lang.Thread.State: BLOCKED
> > at org.jgroups.blocks.cs.BaseServer.hasConnection(BaseServer.java:235)
> > - waiting to lock <4a73a364> (a org.jgroups.blocks.cs.TcpServer) owned
> > by "Connection.Receiver [172.16.42.4:7800 <http://172.16.42.4:7800> -
> > 172.16.42.5:42120 <http://172.16.42.5:42120>],puten,node2" t@11221
> > at
> org.jgroups.blocks.cs.TcpServer$Acceptor.handleAccept(TcpServer.java:135)
> > - locked <2ae2e471> (a org.jgroups.blocks.cs.TcpServer$Acceptor)
> > at org.jgroups.blocks.cs.TcpServer$Acceptor.run(TcpServer.java:117)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > Only these two thread are BLOCKED.
> >
> >
> >
> > On Mon, Apr 11, 2016 at 7:54 PM, Denis Gabaydulin <[email protected]
> > <mailto:[email protected]>> wrote:
> >
> >     Full thread-dump in the attached file.
> >
> >     On Mon, Apr 11, 2016 at 7:44 PM, Denis Gabaydulin <[email protected]
> >     <mailto:[email protected]>> wrote:
> >
> >         Sometimes, I see the following picture in the Thread dump:
> >
> >         "ViewHandler,puten,node2" - Thread t@11063
> >             java.lang.Thread.State: BLOCKED
> >         at
> org.jgroups.blocks.cs.BaseServer.retainAll(BaseServer.java:381)
> >         - waiting to lock <4a73a364> (a org.jgroups.blocks.cs.TcpServer)
> >         owned by "Connection.Receiver [172.16.42.4:42606
> >         <http://172.16.42.4:42606> - 172.16.42.3:7800
> >         <http://172.16.42.3:7800>],puten,node2" t@11058
> >         at org.jgroups.protocols.TCP.retainAll(TCP.java:64)
> >         at
> org.jgroups.protocols.BasicTCP.handleDownEvent(BasicTCP.java:160)
> >         at org.jgroups.protocols.TP.down(TP.java:1452)
> >         at org.jgroups.protocols.Discovery.down(Discovery.java:403)
> >         at org.jgroups.protocols.TCPPING.down(TCPPING.java:103)
> >         at org.jgroups.protocols.MERGE3.down(MERGE3.java:227)
> >         at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:374)
> >         at org.jgroups.protocols.FD_ALL.down(FD_ALL.java:220)
> >         at
> org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:92)
> >         at org.jgroups.protocols.pbcast.NAKACK2.down(NAKACK2.java:589)
> >         at org.jgroups.protocols.UNICAST3.down(UNICAST3.java:669)
> >         at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:327)
> >         at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:732)
> >         - locked <7f555696> (a org.jgroups.protocols.pbcast.GMS)
> >         at
> >
>  org.jgroups.protocols.pbcast.CoordGmsImpl.handleViewChange(CoordGmsImpl.java:244)
> >         at org.jgroups.protocols.pbcast.GMS.castViewChange(GMS.java:617)
> >         at
> >
>  org.jgroups.protocols.pbcast.CoordGmsImpl.handleMembershipChange(CoordGmsImpl.java:221)
> >         at
> >
>  org.jgroups.protocols.pbcast.GMS$ViewHandler.process(GMS.java:1697)
> >         at
> org.jgroups.protocols.pbcast.GMS$ViewHandler.run(GMS.java:1651)
> >         at java.lang.Thread.run(Thread.java:745)
> >
> >         When it happens, a cluster loses a node for a while.
> >         How can it be fixed?
> >
> >         I use the following config:
> >
> >         <config xmlns="urn:org:jgroups"
> >                  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> >                  xsi:schemaLocation="urn:org:jgroups
> >         http://www.jgroups.org/schema/JGroups-3.6.1.xsd">
> >               <TCP
> >                  bind_port="7800"
> >                  timer_type="new3"
> >                  timer.min_threads="4"
> >                  timer.max_threads="10"
> >                  timer.keep_alive_time="3000"
> >                  timer.queue_max_size="500"
> >                  thread_pool.enabled="true"
> >                  thread_pool.min_threads="4"
> >                  thread_pool.max_threads="64"
> >                  thread_pool.keep_alive_time="20000"
> >                  thread_pool.queue_enabled="false"
> >                  thread_pool.queue_max_size="100"
> >                  thread_pool.rejection_policy="discard"
> >
> >                  oob_thread_pool.enabled="true"
> >                  oob_thread_pool.min_threads="2"
> >                  oob_thread_pool.max_threads="8"
> >                  oob_thread_pool.keep_alive_time="20000"
> >                  oob_thread_pool.queue_enabled="false"
> >                  oob_thread_pool.queue_max_size="100"
> >                  oob_thread_pool.rejection_policy="discard"
> >               />
> >              <TCPPING
> >
>  initial_hosts="${jgroups.tcpping.initial_hosts:172.16.42.3[7800],172.16.42.4[7800],172.16.42.5[7800]}"
> >         port_range="0"/>
> >              <MERGE3/>
> >              <FD_SOCK/>
> >              <FD_ALL timeout="90000"/>
> >              <VERIFY_SUSPECT/>
> >              <pbcast.NAKACK2 use_mcast_xmit="false"/>
> >              <UNICAST3/>
> >              <pbcast.STABLE/>
> >              <pbcast.GMS/>
> >              <MFC/>
> >              <FRAG2/>
> >              <pbcast.STATE_TRANSFER />
> >              <CENTRAL_LOCK/>
> >         </config>
> >
> >         Jgroups version is: 3.6.7 FINAL.
> >
> >         When I trying to probe the problem node at that moment (when it
> >         blocked), I see: Found no valid hosts - terminating
> >
> >
> >
> >
> >
> >
> ------------------------------------------------------------------------------
> > Find and fix application performance issues faster with Applications
> Manager
> > Applications Manager provides deep performance insights into multiple
> tiers of
> > your business applications. It resolves application problems quickly and
> > reduces your MTTR. Get your free trial!
> > https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
> >
> >
> >
> > _______________________________________________
> > javagroups-users mailing list
> > [email protected]
> > https://lists.sourceforge.net/lists/listinfo/javagroups-users
> >
>
> --
> Bela Ban, JGroups lead (http://www.jgroups.org)
>
>
>
> ------------------------------------------------------------------------------
> Find and fix application performance issues faster with Applications
> Manager
> Applications Manager provides deep performance insights into multiple
> tiers of
> your business applications. It resolves application problems quickly and
> reduces your MTTR. Get your free trial!
> https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
> _______________________________________________
> javagroups-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>

------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z

_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.