Re: [jgroups-dev] Threads blocked in TCP
Bela Ban <[email protected]> Thu, 02 May 2013 15:40:51 +0200
| Newsgroups | gmane.comp.java.javagroups.devel |
|---|---|
| Message-ID | <[email protected]> |
Sorry, I don't support 2.12.x On 5/1/13 2:58 AM, Sundararajan, Mukund wrote: > Hi Bela, > > I have an 18 node cluster running jgroups 2.12.Final. The transport > layer protocol for jgroups is TCP. > > I have configured for FD, maxTries = 30 and timeout = 60000 to give me > 30 minutes to remove a crashed node from the view. > > Out of the 18 nodes 9 nodes are in one data center and nine in another. > > When we disconnect the first data center from the network, leaving the > other nine to detect their crashed nine peers through FD and FD_SOCK, I > see that some of the nodes seem to drop or discard the view change > event, especially under load. This is causing stuck threads in that node > and jgroups is not able to send or receive any messages. This same test > gives me better results ( more resiliency ) when I set maxTries to 30 > and timeout to 10000 in FD ( 10 minute failure detection interval) and > even better results with maxTries = 5 and timeout = 10000. > > This is my TCP config. Do you think I have stuck threads because I do > not have enough threads in the pool? For the nodes that have stuck > threads, the View contains nodes from the disconnected datacenter while > the View on the other nodes don’t. > > <config > > xmlns="urn:org:jgroups" > > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > > xsi:schemaLocation="urn:org:jgroups > http://www.jgroups.org/schema/JGroups-3.0.xsd" > > > <TCP > > bind_port="9802" > > loopback="true" > > recv_buf_size="20000000" > > send_buf_size="640000" > > discard_incompatible_packets="true" > > max_bundle_size="64000" > > max_bundle_timeout="30" > > enable_bundling="true" > > use_send_queues="true" > > sock_conn_timeout="300" > > thread_pool.enabled="true" > > thread_pool.min_threads="10" > > thread_pool.max_threads="100" > > thread_pool.keep_alive_time="5000" > > thread_pool.queue_enabled="false" > > thread_pool.queue_max_size="100" > > > thread_pool.rejection_policy="Discard" > > oob_thread_pool.enabled="true" > > oob_thread_pool.min_threads="10" > > oob_thread_pool.max_threads="50" > > > oob_thread_pool.keep_alive_time="5000" > > > oob_thread_pool.queue_enabled="false" > > > oob_thread_pool.queue_max_size="100" > > > oob_thread_pool.rejection_policy="Discard" /> > > <TCPPING > > timeout="6000" > > initial_hosts="" > > port_range="0" > > num_initial_members="1" > > num_ping_requests="6" /> > > The thread that blocks all other threads is always executing this code : > > "Timer-8,EnterpriseRBS,ukrom1unyacm02-48208(25.221.12.6,25.221.12.5,Schema_Version > 0.190,1)" id=344863 idx=0x1c0 tid=12949 prio=5 alive, in native, daemon > > at > java/net/PlainSocketImpl.socketConnect(Ljava/net/InetAddress;II)V(Native > Method) > > at java/net/PlainSocketImpl.doConnect(PlainSocketImpl.java:351) > > ^-- Holding lock: java/net/SocksSocketImpl@0x54972000[biased lock] > > at java/net/PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) > > at java/net/PlainSocketImpl.connect(PlainSocketImpl.java:200) > > at > java/net/SocksSocketImpl.connect(SocksSocketImpl.java:366)[optimized] > > at java/net/Socket.connect(Socket.java:529)[optimized] > > at org/jgroups/util/Util.connect(Util.java:276) > > at > org/jgroups/blocks/TCPConnectionMap$TCPConnection.<init>(TCPConnectionMap.java:375) > > at > org/jgroups/blocks/TCPConnectionMap$Mapper.getConnection(TCPConnectionMap.java:755)[optimized] > > at > org/jgroups/blocks/TCPConnectionMap.send(TCPConnectionMap.java:162)[inlined] > > at org/jgroups/protocols/TCP.send(TCP.java:56)[optimized] > > at > org/jgroups/protocols/BasicTCP.sendUnicast(BasicTCP.java:95)[optimized] > > at org/jgroups/protocols/TP.sendToSingleMember(TP.java:1214)[optimized] > > at org/jgroups/protocols/TP.doSend(TP.java:1198)[inlined] > > at org/jgroups/protocols/TP.send(TP.java:1184)[optimized] > > at org/jgroups/protocols/TP.down(TP.java:1041)[optimized] > > at org/jgroups/protocols/Discovery.down(Discovery.java:499)[inlined] > > at org/jgroups/protocols/TCPPING.down(TCPPING.java:149)[optimized] > > at org/jgroups/protocols/MERGE2.down(MERGE2.java:185)[optimized] > > at org/jgroups/protocols/FD_SOCK.down(FD_SOCK.java:340)[optimized] > > at org/jgroups/protocols/FD.down(FD.java:310) > > at org/jgroups/protocols/VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72) > > at org/jgroups/protocols/BARRIER.down(BARRIER.java:93) > > at org/jgroups/protocols/pbcast/NAKACK.down(NAKACK.java:634)[optimized] > > at org/jgroups/protocols/UNICAST.retransmit(UNICAST.java:490) > > at > org/jgroups/stack/AckSenderWindow.retransmit(AckSenderWindow.java:168) > > at > org/jgroups/stack/DefaultRetransmitter$SeqnoTask.callRetransmissionCommand(DefaultRetransmitter.java:125) > > at org/jgroups/stack/Retransmitter$Task.run(Retransmitter.java:142) > > at org/jgroups/util/TimeScheduler2$MyTask.run(TimeScheduler2.java:524) > > at > org/jgroups/util/TimeScheduler2$Entry.execute(TimeScheduler2.java:429) > > at org/jgroups/util/TimeScheduler2$1.run(TimeScheduler2.java:295) > > at > java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)[inlined] > > The blocked threads are blocked here -à > > "OOB-7847,EnterpriseRBS,ukrom1unyacm02-48208(25.221.12.6,25.221.12.5,Schema_Version > 0.190,1)" id=336920 idx=0x84 tid=31697 prio=5 alive, parked, native_blocked > > -- Parking to wait for: > java/util/concurrent/locks/ReentrantLock$NonfairSync@0x407c5988 > > at jrockit/vm/Locks.park0(J)V(Native Method) > > at jrockit/vm/Locks.park(Locks.java:2230)[inlined] > > at jrockit/proxy/sun/misc/Unsafe.park(Unsafe.java:616)[inlined] > > at > java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)[inlined] > > at > java/util/concurrent/locks/AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)[inlined] > > at > java/util/concurrent/locks/AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)[inlined] > > at > java/util/concurrent/locks/AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)[inlined] > > at > java/util/concurrent/locks/ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)[optimized] > > at > java/util/concurrent/locks/ReentrantLock.lock(ReentrantLock.java:262)[optimized] > > at > org/jgroups/blocks/TCPConnectionMap$Mapper.getConnection(TCPConnectionMap.java:749)[optimized] > > at > org/jgroups/blocks/TCPConnectionMap.send(TCPConnectionMap.java:162)[inlined] > > at org/jgroups/protocols/TCP.send(TCP.java:56)[optimized] > > at > org/jgroups/protocols/BasicTCP.sendUnicast(BasicTCP.java:95)[optimized] > > at org/jgroups/protocols/TP.sendToSingleMember(TP.java:1214)[optimized] > > at org/jgroups/protocols/TP.doSend(TP.java:1198)[inlined] > > at org/jgroups/protocols/TP.send(TP.java:1184)[optimized] > > at org/jgroups/protocols/TP.down(TP.java:1041)[optimized] > > at org/jgroups/protocols/Discovery.down(Discovery.java:499)[inlined] > > at org/jgroups/protocols/TCPPING.down(TCPPING.java:149)[optimized] > > at org/jgroups/protocols/MERGE2.down(MERGE2.java:185)[optimized] > > at org/jgroups/protocols/FD_SOCK.down(FD_SOCK.java:340)[optimized] > > at org/jgroups/protocols/FD.down(FD.java:310) > > at org/jgroups/protocols/VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72) > > at org/jgroups/protocols/BARRIER.down(BARRIER.java:93) > > at org/jgroups/protocols/pbcast/NAKACK.sendXmitRsp(NAKACK.java:986) > > at org/jgroups/protocols/pbcast/NAKACK.handleXmitReq(NAKACK.java:903) > > at org/jgroups/protocols/pbcast/NAKACK.up(NAKACK.java:677)[optimized] > > at org/jgroups/protocols/BARRIER.up(BARRIER.java:101)[optimized] > > at > org/jgroups/protocols/VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:135)[optimized] > > at org/jgroups/protocols/FD.up(FD.java:275)[optimized] > > at org/jgroups/protocols/FD_SOCK.up(FD_SOCK.java:275)[optimized] > > at org/jgroups/protocols/MERGE2.up(MERGE2.java:210)[optimized] > > at org/jgroups/protocols/Discovery.up(Discovery.java:294)[optimized] > > at org/jgroups/protocols/TP.passMessageUp(TP.java:1109)[optimized] > > at > org/jgroups/protocols/TP$IncomingPacket.handleMyMessage(TP.java:1665)[inlined] > > at org/jgroups/protocols/TP$IncomingPacket.run(TP.java:1647)[optimized] > > at > java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)[inlined] > > at > java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)[optimized] > > at java/lang/Thread.run(Thread.java:662)[optimized] > > at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method) > > Appreciate your help. > > Mukund > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > > DISCLAIMER: This e-mail may contain information that is confidential, > privileged or otherwise protected from disclosure. If you are not an > intended recipient of this e-mail, do not duplicate or redistribute it > by any means. Please delete it and any attachments and notify the sender > that you have received it in error. Unintended recipients are prohibited > from taking action on the basis of information in this e-mail.E-mail > messages may contain computer viruses or other defects, may not be > accurately replicated on other systems, or may be intercepted, deleted > or interfered with without the knowledge of the sender or the intended > recipient. If you are not comfortable with the risks associated with > e-mail messages, you may decide not to use e-mail to communicate with > IPC. IPC reserves the right, to the extent and under circumstances > permitted by applicable law, to retain, monitor and intercept e-mail > messages to and from its systems. > > -- Bela Ban, JGroups lead (http://www.jgroups.org) ------------------------------------------------------------------------------ Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET Get 100% visibility into your production application - at no cost. Code-level diagnostics for performance bottlenecks with <2% overhead Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap1 _______________________________________________ Javagroups-development mailing list