Re: [jgroups-dev] Issue with TCPConnectionMap in jgroups 2.12.2.Final

"Sundararajan, Mukund" <[email protected]> Thu, 28 Feb 2013 03:27:07 +0000
Newsgroups	gmane.comp.java.javagroups.devel
Message-ID	<1159DDC505B322408341C5F00AB950DF4E3FDFE5@STSNYEXMBX2.corp.root.ipc.com>
Bela,

Is TCP_NIO supported with jgroups 2.12.2.Final? I have been reading bug reports and have seen people reporting that the Connection_Sender thread disappears when the issue below happens. I don't know what was the resolution to this issue.

https://issues.jboss.org/browse/JGRP-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12475091#action_12475091


Mukund

From: Sundararajan, Mukund
Sent: Wednesday, February 27, 2013 5:23 PM
To: [email protected]
Cc: [email protected]
Subject: Issue with TCPConnectionMap in jgroups 2.12.2.Final

Hi Bela,

I am running into an issue that is very similar to what was reported here ---> https://issues.jboss.org/browse/JGRP-1157

Basically during application startup, jgroups forms a cluster with 9 other nodes. What I am seeing is sometimes all our threads get stuck in org/jgroups/blocks/TCPConnectionMap$TCPConnection$Sender.addToQueue(TCPConnectionMap.java:619). When this condition happens I have no other way out but to restart the application.

I have thread dumps and have analyzed it thoroughly. What I see is there are 9 Sender threads waiting on LinkedBlockingQueue.take(). However the threads that are sending messages out are getting blocked on LinkedBlockingQueue.put() for a Connection for which the Sender thread has most probably terminated and hence there is no code waiting on take(). I say this because the Condition object that the 9 Sender threads are waiting on is different from the Condition object that the writers are blocked on!


The nine Sender threads are the following -


"Connection.Sender [25.221.5.230:59891 - 25.221.12.6:9800],Enterprise-,ukwgc7unybcm02-41676(2)" id=14668 idx=0x6e8 tid=21005 prio=5 alive, parked, native_blocked
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x9f88f638

Connection.Sender [25.221.5.230:9800 - 25.221.13.101:35061],Enterprise-,ukwgc7unybcm02-41676(2)" id=14660 idx=0x928 tid=20997 prio=5 alive, parked, native_blocked
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x9f7dfd80

"Connection.Sender [25.221.5.230:43781 - 25.221.4.6:9800],Enterprise-,ukwgc7unybcm02-41676(2)" id=14670 idx=0x978 tid=21007 prio=5 alive, parked, native_blocked
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x9f4cd3b0

"Connection.Sender [25.221.5.230:52571 - 25.221.42.7:9800],Enterprise-,ukwgc7unybcm02-41676(2)" id=14674 idx=0x9c8 tid=21011 prio=5 alive, parked, native_blocked
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x9f741e40

"Connection.Sender [25.221.5.230:48129 - 25.221.42.6:9800],Enterprise-,ukwgc7unybcm02-41676(2)" id=14672 idx=0xa48 tid=21009 prio=5 alive, parked, native_blocked
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x9f474138


"Connection.Sender [25.221.5.230:41259 - 25.221.12.166:9800],Enterprise-,ukwgc7unybcm02-41676(2)" id=14664 idx=0xa6c tid=21001 prio=5 alive, parked, native_blocked
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x9f772cd8


"Connection.Sender [25.221.5.230:9800 - 25.221.2.5:35354],Enterprise-,ukwgc7unybcm02-41676(2)" id=14644 idx=0xa98 tid=19682 prio=5 alive, parked, native_blocked
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x90bb3790

"Connection.Sender [25.221.5.230:59870 - 25.221.13.230:9800],Enterprise-,ukwgc7unybcm02-41676(2)" id=14662 idx=0xb24 tid=20999 prio=5 alive, parked, native_blocked
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x9f6bca10


"Connection.Sender [25.221.5.230:36122 - 25.221.4.198:9800],Enterprise-,ukwgc7unybcm02-41676(2)" id=14666 idx=0xb44 tid=21003 prio=5 alive, parked, native_blocked
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x9f5b0618


All my application threads are stuck in Sender.addToQueue() all waiting on the same Condition object AbstractQueuedSynchronizer$ConditionObject@0x58590740. If you notice there is no Sender thread above that has a blocking take() call on this condition object.

[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" id=20 idx=0x8c tid=26789 prio=1 alive, parked, native_blocked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x58590740

[STUCK] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" id=23 idx=0x98 tid=26792 prio=1 alive, parked, native_blocked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x58590740

"[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" id=30 idx=0xac tid=26802 prio=1 alive, parked, native_blocked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x58590740

.....and so on. All my application threads (30 of them) are stuck in the same place.

Here is the complete stack trace.

[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" id=20 idx=0x8c tid=26789 prio=1 alive, parked, native_blocked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x58590740
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2230)[inlined]
    at jrockit/proxy/sun/misc/Unsafe.park(Unsafe.java:616)[inlined]
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)[inlined]
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)[optimized]
    at java/util/concurrent/LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)[optimized]
    at org/jgroups/blocks/TCPConnectionMap$TCPConnection$Sender.addToQueue(TCPConnectionMap.java:619)[inlined]
    at org/jgroups/blocks/TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:446)[inlined]
    at org/jgroups/blocks/TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:354)[inlined]
    at org/jgroups/blocks/TCPConnectionMap.send(TCPConnectionMap.java:167)[inlined]
    at org/jgroups/protocols/TCP.send(TCP.java:56)[optimized]
    at org/jgroups/protocols/BasicTCP.sendUnicast(BasicTCP.java:95)[optimized]
    at org/jgroups/protocols/TP.sendToSingleMember(TP.java:1214)[optimized]
    at org/jgroups/protocols/TP.doSend(TP.java:1198)[inlined]
    at org/jgroups/protocols/TP.send(TP.java:1184)[optimized]
    at org/jgroups/protocols/TP.down(TP.java:1041)[optimized]
    at org/jgroups/protocols/Discovery.down(Discovery.java:499)[inlined]
    at org/jgroups/protocols/TCPPING.down(TCPPING.java:149)[optimized]
    at org/jgroups/protocols/MERGE2.down(MERGE2.java:185)
    at org/jgroups/protocols/FD_SOCK.down(FD_SOCK.java:340)
    at org/jgroups/protocols/FD.down(FD.java:310)[optimized]
    at org/jgroups/protocols/VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)[optimized]
    at org/jgroups/protocols/BARRIER.down(BARRIER.java:93)[optimized]
    at org/jgroups/protocols/pbcast/NAKACK.down(NAKACK.java:634)[optimized]
    at org/jgroups/protocols/UNICAST.send(UNICAST.java:450)[inlined]
    at org/jgroups/protocols/UNICAST.down(UNICAST.java:401)[optimized]
    at org/jgroups/protocols/pbcast/STABLE.down(STABLE.java:330)
    at org/jgroups/protocols/pbcast/GMS.down(GMS.java:977)
    at org/jgroups/protocols/UFC.handleDownMessage(UFC.java:123)
    at org/jgroups/protocols/FlowControl.down(FlowControl.java:341)
    at org/jgroups/protocols/FlowControl.sendCreditRequest(FlowControl.java:498)
    at org/jgroups/protocols/MFC.handleDownMessage(MFC.java:112)
    at org/jgroups/protocols/FlowControl.down(FlowControl.java:341)
    at org/jgroups/protocols/FRAG2.down(FRAG2.java:155)
    at org/jgroups/protocols/pbcast/STATE_TRANSFER.down(STATE_TRANSFER.java:214)
    at org/jgroups/stack/ProtocolStack.down(ProtocolStack.java:1081)
    at org/jgroups/JChannel.down(JChannel.java:1654)
    at org/jgroups/JChannel.send(JChannel.java:734)
    at com/ipc/ds/interzone/channel/IZChannel.sendMessage(IZChannel.java:138)
    at com/ipc/ds/interzone/channel/IZChannel.sendMessage(IZChannel.java:120)
    at com/ipc/ds/interzone/notification/LocalLineStatusNotificationListener.notify(LocalLineStatusNotificationListener.java:207)
    at com/ipc/ds/notification/adaptors/JNDIAdaptor.tell(Ljava/lang/String;Ljava/util/List;Lcom/ipc/ds/notification/Publishable;)V(Unknown Source)
    at com/ipc/ds/notification/core/NotificationEvent.fire()V(Unknown Source)
    at com/ipc/ds/notification/core/NotificationJob.run()V(Unknown Source)
    at com/ipc/ds/notification/ejb/LineStatusNotificationMDB.onMessage(Ljavax/jms/Message;)V(Unknown Source)
    at sun/reflect/GeneratedMethodAccessor552.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
    at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]
    at java/lang/reflect/Method.invoke(Method.java:597)[optimized]
    at com/bea/core/repackaged/springframework/aop/support/AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)[inlined]
    at com/bea/core/repackaged/springframework/aop/framework/ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)[inlined]
    at com/bea/core/repackaged/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)[optimized]
    at com/bea/core/repackaged/springframework/aop/support/DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    at com/bea/core/repackaged/springframework/aop/support/DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
    at com/bea/core/repackaged/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)[optimized]
    at com/bea/core/repackaged/springframework/aop/interceptor/ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
    at com/bea/core/repackaged/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)[optimized]
    at com/bea/core/repackaged/springframework/aop/support/DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    at com/bea/core/repackaged/springframework/aop/support/DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
    at com/bea/core/repackaged/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)[optimized]
    at com/bea/core/repackaged/springframework/aop/framework/JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)[optimized]
    at $Proxy195.onMessage(Ljavax/jms/Message;)V(Unknown Source)
    at weblogic/ejb/container/internal/MDListener.execute(MDListener.java:466)
    at weblogic/ejb/container/internal/MDListener.transactionalOnMessage(MDListener.java:371)
    at weblogic/ejb/container/internal/MDListener.onMessage(MDListener.java:327)
    at weblogic/jms/client/JMSSession.onMessage(JMSSession.java:4659)
    at weblogic/jms/client/JMSSession.execute(JMSSession.java:4345)[optimized]
    at weblogic/jms/client/JMSSession.executeMessage(JMSSession.java:3821)
    at weblogic/jms/client/JMSSession.access$000(JMSSession.java:115)
    at weblogic/jms/client/JMSSession$UseForRunnable.run(JMSSession.java:5170)
    at weblogic/work/SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)[optimized]
    at weblogic/work/ExecuteThread.execute(ExecuteThread.java:201)[optimized]
    at weblogic/work/ExecuteThread.run(ExecuteThread.java:173)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace


Is there a way out? This happens during application startup only under unknown circumstances. I would say 1 out of 10 times it happens.

Our application starts sending out messages as soon as the JChannel gets connected. From netstat what I see is out of the 9 TCP connections, about 7 of them were initiated by this node to other nodes and 2 of them were initiated by other nodes. My bind port is 9800. For some connections I see the connection is from an ephemeral port on the remote side versus 9800. See example below. My server IP us 25.221.5.230. This connection was initiated by the remote side.

Connection.Sender [25.221.5.230:9800 - 25.221.13.101:35061],Enterprise-,ukwgc7unybcm02-41676(2)" id=14660 idx=0x928 tid=20997 prio=5 alive, parked, native_blocked
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x9f7dfd80


I am trying to find a way out. Would setting use_send_queues to false help? At this point I have to stay with 2.12.2.Final since this code is running in the field but if an upgrade is inevitable I can look into it.

Thanks
Mukund

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

DISCLAIMER: This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unintended recipients are prohibited from taking action on the basis of information in this e-mail.E-mail messages may contain computer viruses or other defects, may not be accurately replicated on other systems, or may be intercepted, deleted or interfered with without the knowledge of the sender or the intended recipient. If you are not comfortable with the risks associated with e-mail messages, you may decide not to use e-mail to communicate with IPC. IPC reserves the right, to the extent and under circumstances permitted by applicable law, to retain, monitor and intercept e-mail messages to and from its systems.

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_feb

_______________________________________________
Javagroups-development mailing list