Re: [jgroups-users] TCP transport is blocked

Questions/problems related to using JGroups <[email protected]>
Newsgroups gmane.comp.java.javagroups.general
Message-ID <CAB1b6fFkkygzAJgbXrPwTFEQF9BAZfTc5mYVW+L0rd8BU_9zDw@mail.gmail.com> 
Something wrong with formatting :-/
I mean this piece of the stacktrace:

https://gist.github.com/sherman/1623d539f29bd32d4cf627633e2a9c95#file-gistfile1-txt-L201-L218

On Sun, Apr 17, 2016 at 1:11 PM, Denis Gabaydulin <[email protected]> wrote:

> Also, pls have a look, at Connection.Receiver thread.
>
> Connection.Receiver [172.16.42.4:7800 - 172.16.42.5:42120],puten,node2" -
> Thread t@11221 java.lang.Thread.State: WAITING at
> sun.misc.Unsafe.park(Native Method) - waiting to lock <43ea8685> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync) owned by
> "TransferQueueBundler,puten,node2" t@53 at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
> at
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
> at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at
> org.jgroups.blocks.cs.TcpConnection.close(TcpConnection.java:429) at
> org.jgroups.util.Util.close(Util.java:405) at
> org.jgroups.blocks.cs.BaseServer.removeConnectionIfPresent(BaseServer.java:362)
> - locked <4a73a364> (a org.jgroups.blocks.cs.TcpServer) at
> org.jgroups.blocks.cs.BaseServer.connectionClosed(BaseServer.java:223) at
> org.jgroups.blocks.cs.BaseServer.notifyConnectionClosed(BaseServer.java:393)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:325)
> at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: -
> None
>
> As you can see we have two monitors here:
>
> 43ea8685 and 4a73a364.
>
> I'm not familiar with this code, but it can be situation when lock
> 4a73a364, is waiting for release of 43ea8685. And the last one is released
> by a timeout.
> On Apr 17, 2016 12:57 PM, "Denis Gabaydulin" <[email protected]> wrote:
>
>> Here's the full stacktrace.
>>
>> https://gist.github.com/sherman/1623d539f29bd32d4cf627633e2a9c95
>>
>> Yes, it doesn't look like a deadlock. After a few minutes, a monitor is
>> released (nodes has joined back).
>> On Apr 17, 2016 12:42 PM, "Questions/problems related to using JGroups" <
>> [email protected]> wrote:
>>
>>> OK, so the BaseServer.hasConnection() and BaseServer.retainAll() both
>>> block on 4a73a364, but both methods cannot block forever, so this cannot
>>> lead to a deadlock. I still need the full stacktrace (like to a paste,
>>> not as attachment to an email!)...
>>>
>>> On 14/04/16 18:52, Questions/problems related to using JGroups wrote:
>>> > One more example:
>>> >
>>> > "ViewHandler,puten,node2" - Thread t@11655
>>> >     java.lang.Thread.State: BLOCKED
>>> > at org.jgroups.blocks.cs.BaseServer.retainAll(BaseServer.java:381)
>>> > - waiting to lock <4a73a364> (a org.jgroups.blocks.cs.TcpServer) owned
>>> > by "Connection.Receiver [172.16.42.4:7800 <http://172.16.42.4:7800> -
>>> > 172.16.42.5:42120 <http://172.16.42.5:42120>],puten,node2" t@11221
>>> > at org.jgroups.protocols.TCP.retainAll(TCP.java:64)
>>> > at org.jgroups.protocols.BasicTCP.handleDownEvent(BasicTCP.java:160)
>>> > at org.jgroups.protocols.TP.down(TP.java:1452)
>>> > at org.jgroups.protocols.Discovery.down(Discovery.java:403)
>>> > at org.jgroups.protocols.TCPPING.down(TCPPING.java:103)
>>> > at org.jgroups.protocols.MERGE3.down(MERGE3.java:227)
>>> > at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:374)
>>> > at org.jgroups.protocols.FD_ALL.down(FD_ALL.java:220)
>>> > at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:92)
>>> > at org.jgroups.protocols.pbcast.NAKACK2.down(NAKACK2.java:589)
>>> > at org.jgroups.protocols.UNICAST3.down(UNICAST3.java:669)
>>> > at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:327)
>>> > at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:732)
>>> > - locked <7f555696> (a org.jgroups.protocols.pbcast.GMS)
>>> > at
>>> >
>>> org.jgroups.protocols.pbcast.CoordGmsImpl.handleViewChange(CoordGmsImpl.java:244)
>>> > at org.jgroups.protocols.pbcast.GMS.castViewChange(GMS.java:617)
>>> > at
>>> >
>>> org.jgroups.protocols.pbcast.CoordGmsImpl.handleMembershipChange(CoordGmsImpl.java:221)
>>> > at org.jgroups.protocols.pbcast.GMS$ViewHandler.process(GMS.java:1697)
>>> > at org.jgroups.protocols.pbcast.GMS$ViewHandler.run(GMS.java:1651)
>>> > at java.lang.Thread.run(Thread.java:745)
>>> >
>>> > "TcpServer.Acceptor [7800],null,null" - Thread t@51
>>> >     java.lang.Thread.State: BLOCKED
>>> > at org.jgroups.blocks.cs.BaseServer.hasConnection(BaseServer.java:235)
>>> > - waiting to lock <4a73a364> (a org.jgroups.blocks.cs.TcpServer) owned
>>> > by "Connection.Receiver [172.16.42.4:7800 <http://172.16.42.4:7800> -
>>> > 172.16.42.5:42120 <http://172.16.42.5:42120>],puten,node2" t@11221
>>> > at
>>> org.jgroups.blocks.cs.TcpServer$Acceptor.handleAccept(TcpServer.java:135)
>>> > - locked <2ae2e471> (a org.jgroups.blocks.cs.TcpServer$Acceptor)
>>> > at org.jgroups.blocks.cs.TcpServer$Acceptor.run(TcpServer.java:117)
>>> > at java.lang.Thread.run(Thread.java:745)
>>> >
>>> > Only these two thread are BLOCKED.
>>> >
>>> >
>>> >
>>> > On Mon, Apr 11, 2016 at 7:54 PM, Denis Gabaydulin <[email protected]
>>> > <mailto:[email protected]>> wrote:
>>> >
>>> >     Full thread-dump in the attached file.
>>> >
>>> >     On Mon, Apr 11, 2016 at 7:44 PM, Denis Gabaydulin <
>>> [email protected]
>>> >     <mailto:[email protected]>> wrote:
>>> >
>>> >         Sometimes, I see the following picture in the Thread dump:
>>> >
>>> >         "ViewHandler,puten,node2" - Thread t@11063
>>> >             java.lang.Thread.State: BLOCKED
>>> >         at
>>> org.jgroups.blocks.cs.BaseServer.retainAll(BaseServer.java:381)
>>> >         - waiting to lock <4a73a364> (a
>>> org.jgroups.blocks.cs.TcpServer)
>>> >         owned by "Connection.Receiver [172.16.42.4:42606
>>> >         <http://172.16.42.4:42606> - 172.16.42.3:7800
>>> >         <http://172.16.42.3:7800>],puten,node2" t@11058
>>> >         at org.jgroups.protocols.TCP.retainAll(TCP.java:64)
>>> >         at
>>> org.jgroups.protocols.BasicTCP.handleDownEvent(BasicTCP.java:160)
>>> >         at org.jgroups.protocols.TP.down(TP.java:1452)
>>> >         at org.jgroups.protocols.Discovery.down(Discovery.java:403)
>>> >         at org.jgroups.protocols.TCPPING.down(TCPPING.java:103)
>>> >         at org.jgroups.protocols.MERGE3.down(MERGE3.java:227)
>>> >         at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:374)
>>> >         at org.jgroups.protocols.FD_ALL.down(FD_ALL.java:220)
>>> >         at
>>> org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:92)
>>> >         at org.jgroups.protocols.pbcast.NAKACK2.down(NAKACK2.java:589)
>>> >         at org.jgroups.protocols.UNICAST3.down(UNICAST3.java:669)
>>> >         at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:327)
>>> >         at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:732)
>>> >         - locked <7f555696> (a org.jgroups.protocols.pbcast.GMS)
>>> >         at
>>> >
>>>  org.jgroups.protocols.pbcast.CoordGmsImpl.handleViewChange(CoordGmsImpl.java:244)
>>> >         at
>>> org.jgroups.protocols.pbcast.GMS.castViewChange(GMS.java:617)
>>> >         at
>>> >
>>>  org.jgroups.protocols.pbcast.CoordGmsImpl.handleMembershipChange(CoordGmsImpl.java:221)
>>> >         at
>>> >
>>>  org.jgroups.protocols.pbcast.GMS$ViewHandler.process(GMS.java:1697)
>>> >         at
>>> org.jgroups.protocols.pbcast.GMS$ViewHandler.run(GMS.java:1651)
>>> >         at java.lang.Thread.run(Thread.java:745)
>>> >
>>> >         When it happens, a cluster loses a node for a while.
>>> >         How can it be fixed?
>>> >
>>> >         I use the following config:
>>> >
>>> >         <config xmlns="urn:org:jgroups"
>>> >                  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> >                  xsi:schemaLocation="urn:org:jgroups
>>> >         http://www.jgroups.org/schema/JGroups-3.6.1.xsd">
>>> >               <TCP
>>> >                  bind_port="7800"
>>> >                  timer_type="new3"
>>> >                  timer.min_threads="4"
>>> >                  timer.max_threads="10"
>>> >                  timer.keep_alive_time="3000"
>>> >                  timer.queue_max_size="500"
>>> >                  thread_pool.enabled="true"
>>> >                  thread_pool.min_threads="4"
>>> >                  thread_pool.max_threads="64"
>>> >                  thread_pool.keep_alive_time="20000"
>>> >                  thread_pool.queue_enabled="false"
>>> >                  thread_pool.queue_max_size="100"
>>> >                  thread_pool.rejection_policy="discard"
>>> >
>>> >                  oob_thread_pool.enabled="true"
>>> >                  oob_thread_pool.min_threads="2"
>>> >                  oob_thread_pool.max_threads="8"
>>> >                  oob_thread_pool.keep_alive_time="20000"
>>> >                  oob_thread_pool.queue_enabled="false"
>>> >                  oob_thread_pool.queue_max_size="100"
>>> >                  oob_thread_pool.rejection_policy="discard"
>>> >               />
>>> >              <TCPPING
>>> >
>>>  initial_hosts="${jgroups.tcpping.initial_hosts:172.16.42.3[7800],172.16.42.4[7800],172.16.42.5[7800]}"
>>> >         port_range="0"/>
>>> >              <MERGE3/>
>>> >              <FD_SOCK/>
>>> >              <FD_ALL timeout="90000"/>
>>> >              <VERIFY_SUSPECT/>
>>> >              <pbcast.NAKACK2 use_mcast_xmit="false"/>
>>> >              <UNICAST3/>
>>> >              <pbcast.STABLE/>
>>> >              <pbcast.GMS/>
>>> >              <MFC/>
>>> >              <FRAG2/>
>>> >              <pbcast.STATE_TRANSFER />
>>> >              <CENTRAL_LOCK/>
>>> >         </config>
>>> >
>>> >         Jgroups version is: 3.6.7 FINAL.
>>> >
>>> >         When I trying to probe the problem node at that moment (when it
>>> >         blocked), I see: Found no valid hosts - terminating
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> ------------------------------------------------------------------------------
>>> > Find and fix application performance issues faster with Applications
>>> Manager
>>> > Applications Manager provides deep performance insights into multiple
>>> tiers of
>>> > your business applications. It resolves application problems quickly
>>> and
>>> > reduces your MTTR. Get your free trial!
>>> > https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > javagroups-users mailing list
>>> > [email protected]
>>> > https://lists.sourceforge.net/lists/listinfo/javagroups-users
>>> >
>>>
>>> --
>>> Bela Ban, JGroups lead (http://www.jgroups.org)
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Find and fix application performance issues faster with Applications
>>> Manager
>>> Applications Manager provides deep performance insights into multiple
>>> tiers of
>>> your business applications. It resolves application problems quickly and
>>> reduces your MTTR. Get your free trial!
>>> https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
>>> _______________________________________________
>>> javagroups-users mailing list
>>> [email protected]
>>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>>>
>>

------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z

_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.