Re: [jgroups-users] TCP transport is blocked

Questions/problems related to using JGroups <[email protected]>
Newsgroups gmane.comp.java.javagroups.general
Message-ID <CAB1b6fHEfU8mi4aBEagrNmWK-QAXLYdoDNTktRYSd+X7fTcm-g@mail.gmail.com> 
One more example:

"ViewHandler,puten,node2" - Thread t@11655
   java.lang.Thread.State: BLOCKED
at org.jgroups.blocks.cs.BaseServer.retainAll(BaseServer.java:381)
- waiting to lock <4a73a364> (a org.jgroups.blocks.cs.TcpServer) owned by
"Connection.Receiver [172.16.42.4:7800 - 172.16.42.5:42120],puten,node2"
t@11221
at org.jgroups.protocols.TCP.retainAll(TCP.java:64)
at org.jgroups.protocols.BasicTCP.handleDownEvent(BasicTCP.java:160)
at org.jgroups.protocols.TP.down(TP.java:1452)
at org.jgroups.protocols.Discovery.down(Discovery.java:403)
at org.jgroups.protocols.TCPPING.down(TCPPING.java:103)
at org.jgroups.protocols.MERGE3.down(MERGE3.java:227)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:374)
at org.jgroups.protocols.FD_ALL.down(FD_ALL.java:220)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:92)
at org.jgroups.protocols.pbcast.NAKACK2.down(NAKACK2.java:589)
at org.jgroups.protocols.UNICAST3.down(UNICAST3.java:669)
at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:327)
at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:732)
- locked <7f555696> (a org.jgroups.protocols.pbcast.GMS)
at
org.jgroups.protocols.pbcast.CoordGmsImpl.handleViewChange(CoordGmsImpl.java:244)
at org.jgroups.protocols.pbcast.GMS.castViewChange(GMS.java:617)
at
org.jgroups.protocols.pbcast.CoordGmsImpl.handleMembershipChange(CoordGmsImpl.java:221)
at org.jgroups.protocols.pbcast.GMS$ViewHandler.process(GMS.java:1697)
at org.jgroups.protocols.pbcast.GMS$ViewHandler.run(GMS.java:1651)
at java.lang.Thread.run(Thread.java:745)

"TcpServer.Acceptor [7800],null,null" - Thread t@51
   java.lang.Thread.State: BLOCKED
at org.jgroups.blocks.cs.BaseServer.hasConnection(BaseServer.java:235)
- waiting to lock <4a73a364> (a org.jgroups.blocks.cs.TcpServer) owned by
"Connection.Receiver [172.16.42.4:7800 - 172.16.42.5:42120],puten,node2"
t@11221
at org.jgroups.blocks.cs.TcpServer$Acceptor.handleAccept(TcpServer.java:135)
- locked <2ae2e471> (a org.jgroups.blocks.cs.TcpServer$Acceptor)
at org.jgroups.blocks.cs.TcpServer$Acceptor.run(TcpServer.java:117)
at java.lang.Thread.run(Thread.java:745)

Only these two thread are BLOCKED.



On Mon, Apr 11, 2016 at 7:54 PM, Denis Gabaydulin <[email protected]> wrote:

> Full thread-dump in the attached file.
>
> On Mon, Apr 11, 2016 at 7:44 PM, Denis Gabaydulin <[email protected]>
> wrote:
>
>> Sometimes, I see the following picture in the Thread dump:
>>
>> "ViewHandler,puten,node2" - Thread t@11063
>>    java.lang.Thread.State: BLOCKED
>> at org.jgroups.blocks.cs.BaseServer.retainAll(BaseServer.java:381)
>> - waiting to lock <4a73a364> (a org.jgroups.blocks.cs.TcpServer) owned by
>> "Connection.Receiver [172.16.42.4:42606 - 172.16.42.3:7800],puten,node2"
>> t@11058
>> at org.jgroups.protocols.TCP.retainAll(TCP.java:64)
>> at org.jgroups.protocols.BasicTCP.handleDownEvent(BasicTCP.java:160)
>> at org.jgroups.protocols.TP.down(TP.java:1452)
>> at org.jgroups.protocols.Discovery.down(Discovery.java:403)
>> at org.jgroups.protocols.TCPPING.down(TCPPING.java:103)
>> at org.jgroups.protocols.MERGE3.down(MERGE3.java:227)
>> at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:374)
>> at org.jgroups.protocols.FD_ALL.down(FD_ALL.java:220)
>> at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:92)
>> at org.jgroups.protocols.pbcast.NAKACK2.down(NAKACK2.java:589)
>> at org.jgroups.protocols.UNICAST3.down(UNICAST3.java:669)
>> at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:327)
>> at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:732)
>> - locked <7f555696> (a org.jgroups.protocols.pbcast.GMS)
>> at
>> org.jgroups.protocols.pbcast.CoordGmsImpl.handleViewChange(CoordGmsImpl.java:244)
>> at org.jgroups.protocols.pbcast.GMS.castViewChange(GMS.java:617)
>> at
>> org.jgroups.protocols.pbcast.CoordGmsImpl.handleMembershipChange(CoordGmsImpl.java:221)
>> at org.jgroups.protocols.pbcast.GMS$ViewHandler.process(GMS.java:1697)
>> at org.jgroups.protocols.pbcast.GMS$ViewHandler.run(GMS.java:1651)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> When it happens, a cluster loses a node for a while.
>> How can it be fixed?
>>
>> I use the following config:
>>
>> <config xmlns="urn:org:jgroups"
>>         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>         xsi:schemaLocation="urn:org:jgroups
>> http://www.jgroups.org/schema/JGroups-3.6.1.xsd">
>>      <TCP
>>         bind_port="7800"
>>         timer_type="new3"
>>         timer.min_threads="4"
>>         timer.max_threads="10"
>>         timer.keep_alive_time="3000"
>>         timer.queue_max_size="500"
>>
>>
>>
>>         thread_pool.enabled="true"
>>         thread_pool.min_threads="4"
>>         thread_pool.max_threads="64"
>>         thread_pool.keep_alive_time="20000"
>>         thread_pool.queue_enabled="false"
>>         thread_pool.queue_max_size="100"
>>         thread_pool.rejection_policy="discard"
>>
>>         oob_thread_pool.enabled="true"
>>         oob_thread_pool.min_threads="2"
>>         oob_thread_pool.max_threads="8"
>>         oob_thread_pool.keep_alive_time="20000"
>>         oob_thread_pool.queue_enabled="false"
>>         oob_thread_pool.queue_max_size="100"
>>         oob_thread_pool.rejection_policy="discard"
>>      />
>>     <TCPPING
>> initial_hosts="${jgroups.tcpping.initial_hosts:172.16.42.3[7800],172.16.42.4[7800],172.16.42.5[7800]}"
>> port_range="0"/>
>>     <MERGE3/>
>>     <FD_SOCK/>
>>     <FD_ALL timeout="90000"/>
>>     <VERIFY_SUSPECT/>
>>     <pbcast.NAKACK2 use_mcast_xmit="false"/>
>>     <UNICAST3/>
>>     <pbcast.STABLE/>
>>     <pbcast.GMS/>
>>     <MFC/>
>>     <FRAG2/>
>>     <pbcast.STATE_TRANSFER />
>>     <CENTRAL_LOCK/>
>> </config>
>>
>> Jgroups version is: 3.6.7 FINAL.
>>
>> When I trying to probe the problem node at that moment (when it blocked),
>> I see: Found no valid hosts - terminating
>>
>
>

------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z

_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.