Zombi state of OpenJMS 0.7.6.1 server ?
Antoine PICOLET <antoine.picolet-W30auxM/[email protected]>
| Newsgroups | gmane.comp.java.openjms.user |
|---|---|
| Message-ID | <OFBBCBBF0E.B16526B2-ONC125701B.005D4559-C125701B.00604971@sagem.com> |
Hello, I am running OpenJMS 0.7.6.1 in a production environment,with an Oracle 9i database. The server itself operates on a Linux platform (SuSE Linux Entreprise Server 8). I implemented two administrative scripts : - a periodic server status checker, which opens and closes a QueueConnection and a QueueSession with the server, and restarts it when it fails or times out (it does sometimes happen, generally when a database problem occurs). - an OpenJMS administration script, which creates/deletes Administered Objects in the server, according to a configuration file giving requested destinations (so that we have neither to restart the server, nor to use the administration GUI when we want to add/remove destinations - everything can be done through a centralized web interface). Now I have some clients to this OpenJMS Server, which I do not totally master. It occurs that one of these clients, when connectiong to the OpenJMS Server, makes it turn to a weird state, where Connections can be made with it, sometimes even Sessions, but any other operations hang forever (createReceiver, removeDestination,etc). When this problem happens, I sometimes tries to kill this client, but the OpenJMS Server would stay in that weird state. I have to restart it in order to make it become available again. Note that this 'weird state' is quite difficult to see automatically; I could always enhace my periodic status script, to try to perform a more heavy operation than merley opening a Connection, so that the server could be restarted automatically; however, this would not correct this problem ! The problem I face is that my JMS provider should be tolerant to a client doing very weird things (using the openjms client library), since our company does not master all the JMS clients do. I have gone as far of my knowledge of Java and the OpenJMS source code (quite little) can go. Does anyone have an explanation for what is happening ? Here are the traces I could have about this problem. I can reproduce this bug quite often (not always, of course...), so I might be able to give other kind of infos if needed. I start the server : Jun 8 09:49:37 SERVER OUT/ERR : Starting Notification/Message Service (OpenJMS Server): Jun 8 09:49:37 SERVER OUT/ERR : OpenJMS 0.7.6.1 Jun 8 09:49:37 SERVER OUT/ERR : The OpenJMS Group. (C) 1999-2004. All rights reserved. Jun 8 09:49:37 SERVER OUT/ERR : http://openjms.sourceforge.net Jun 8 09:49:39 SERVER LOGGER : INFO DatabaseService Creating RdbmsAdapter for oracle.jdbc.driver.OracleDriver Jun 8 09:49:39 SERVER LOGGER : INFO ServiceGroup Started service [ThreadPoolManager] Jun 8 09:49:39 SERVER LOGGER : INFO ServiceGroup Started service [EventManagerThread] Jun 8 09:49:39 SERVER LOGGER : INFO DatabaseService Removed expired messages. Jun 8 09:49:39 SERVER LOGGER : INFO ServiceGroup Started service [DatabaseService] Jun 8 09:49:39 SERVER LOGGER : INFO ServiceGroup Started service [Scheduler] Jun 8 09:49:39 SERVER LOGGER : INFO ServiceGroup Started service [LeaseManagerReaper] Jun 8 09:49:39 SERVER LOGGER : INFO GarbageCollectionService Registering Garbage Collection every 60000 for memory. Jun 8 09:49:39 SERVER LOGGER : INFO GarbageCollectionService Registering Garbage Collection every 120000 for other resources. Jun 8 09:49:39 SERVER LOGGER : INFO ServiceGroup Started service [GCCollectionService] Jun 8 09:49:39 SERVER LOGGER : INFO ServiceGroup Started service [AuthenticationManager] Jun 8 09:49:39 SERVER LOGGER : DEBUG ConsumerManager addToConsumerCache(name=OIT-CAPT, dest=owsTopic-true, durable=true) Jun 8 09:49:39 SERVER LOGGER : DEBUG ConsumerManager addToConsumerCache(name=OMT, dest=owsTopic-true, durable=true) Jun 8 09:49:39 SERVER LOGGER : DEBUG ConsumerManager addToConsumerCache(name=OIT-FO, dest=owsTopic-true, durable=true) Jun 8 09:49:39 SERVER LOGGER : DEBUG ConsumerManager addToConsumerCache(name=DurableConsumer1, dest=TestTopic1-true, durable=true) Jun 8 09:49:39 SERVER LOGGER : DEBUG ConsumerManager addToConsumerCache(name=DurableConsumer2, dest=TestTopic2-true, durable=true) Jun 8 09:49:39 SERVER LOGGER : INFO ServiceGroup Started service [MessageManager] Jun 8 09:49:39 SERVER LOGGER : INFO JmsServer Creating server interface for the tcp connector Jun 8 09:49:39 SERVER LOGGER : DEBUG JmsServer Created an instance of org.exolab.jms.server.mipc.IpcJmsServer as a server interface Jun 8 09:49:39 SERVER LOGGER : DEBUG MultiplexConnectionServer MultiplexConnectionServer is ready, on port 3030 Jun 8 09:49:39 SERVER LOGGER : DEBUG ConnectionFactoryHelper Bound connection factory JmsQueueConnectionFactory Jun 8 09:49:39 SERVER LOGGER : DEBUG ConnectionFactoryHelper Bound connection factory JmsTopicConnectionFactory Jun 8 09:49:39 SERVER LOGGER : INFO JmsServer Creating JNDI interface for the tcp connector Jun 8 09:49:39 SERVER LOGGER : DEBUG Server IPC Service Started Jun 8 09:49:39 SERVER LOGGER : DEBUG TcpServerService Server is ready, on port 3035 Jun 8 09:49:39 SERVER LOGGER : INFO IpcJndiServer Started TCP JNDI server on port 3035 Jun 8 09:49:39 SERVER LOGGER : DEBUG JmsServer Created an instance of org.exolab.jms.server.mipc.IpcJndiServer as a JNDI provider interface Quite soon, two clients connect, and try to create receivers on queue 'testprobe001' and topic 'owsTopic' (the latter poses no problem; I had the exact same behavior with this client off). Jun 8 09:49:41 SERVER LOGGER : DEBUG MultiplexConnectionServer MultiplexConnectionServer accept()'ed connection Jun 8 09:49:42 SERVER LOGGER : DEBUG JmsServerSession createReceiver(queue=probe001-true, clientId=1, selector=null) [sessionId=4430dd3d-bd0a-1004-8168-54de4800cd17] Jun 8 09:49:42 SERVER LOGGER : DEBUG ConsumerManager createConsumerEndpoint(session=[sessionId=4430dd3d-bd0a-1004-8168-54de4800cd17], clientId=1, destination=probe001-true, selector=null) Jun 8 09:49:42 SERVER LOGGER : DEBUG RDBMSAdapter getMessageHandles,15 Jun 8 09:49:42 SERVER LOGGER : DEBUG ConsumerManager addToConsumerCache(name=443104b0-bd0a-1004-8168-54de4800cd17, dest=probe001-true, durable=false) Jun 8 09:49:42 SERVER LOGGER : DEBUG JmsServerSession start() [sessionId=4430dd3d-bd0a-1004-8168-54de4800cd17] Jun 8 09:49:42 SERVER LOGGER : DEBUG RDBMSAdapter getMessage,51 Jun 8 09:49:42 SERVER LOGGER : DEBUG MultiplexConnectionServer MultiplexConnectionServer accept()'ed connection Jun 8 09:49:42 SERVER LOGGER : DEBUG JmsServerSession createSubscriber(topic=owsTopic-true, name=OIT-FO, clientId=1, selector=null, noLocal=false) [sessionId=4431a280-bd0a-1004-8168-54de4800cd17] Jun 8 09:49:42 SERVER LOGGER : DEBUG ConsumerManager createDurableConsumerEndpoint(session=[sessionId=4431a280-bd0a-1004-8168-54de4800cd17], topic=owsTopic-true, name=OIT-FO, clientId=1, selector=null Jun 8 09:49:42 SERVER LOGGER : DEBUG RDBMSAdapter getMessageHandles,1 Jun 8 09:49:43 SERVER LOGGER : DEBUG JmsServerSession start() [sessionId=4431a280-bd0a-1004-8168-54de4800cd17] Now the same client as before retries to create a receiver on the same queue 'probe001'; the 'createReceiver()' method of this client hangs forever. Jun 8 09:49:47 SERVER LOGGER : DEBUG MultiplexConnectionServer MultiplexConnectionServer accept()'ed connection Jun 8 09:49:47 SERVER LOGGER : DEBUG JmsServerSession createReceiver(queue=probe001-true, clientId=1, selector=null) [sessionId=44394274-bd0a-1004-8168-54de4800cd17] Jun 8 09:49:47 SERVER LOGGER : DEBUG ConsumerManager createConsumerEndpoint(session=[sessionId=44394274-bd0a-1004-8168-54de4800cd17], clientId=1, destination=probe001-true, selector=null) some times later, a periodic status scripts tries to open a QueueConnection and a TopicConnection and to close them, and succeeds; it then declares the server OK. Jun 8 09:49:59 SERVER LOGGER : DEBUG MultiplexConnectionServer MultiplexConnectionServer accept()'ed connection Jun 8 09:50:00 SERVER LOGGER : DEBUG Demultiplexer Demultiplexer terminating on exception Jun 8 09:50:00 SERVER LOGGER : java.io.EOFException Jun 8 09:50:00 SERVER LOGGER : at java.io.DataInputStream.readInt(Unknown Source) Jun 8 09:50:00 SERVER LOGGER : at org.exolab.core.mipc.MessageInputStream.receive(MessageInputStream.java:95) Jun 8 09:50:00 SERVER LOGGER : at org.exolab.core.mipc.Demultiplexer.run(Demultiplexer.java:153) Jun 8 09:50:00 SERVER LOGGER : at org.exolab.core.mipc.MultiplexConnection.run(MultiplexConnection.java:240) Jun 8 09:50:00 SERVER LOGGER : DEBUG IpcServerChannel java.io.InterruptedIOException: Empty packet was returned Jun 8 09:50:00 SERVER LOGGER : java.io.InterruptedIOException: Empty packet was returned Jun 8 09:50:00 SERVER LOGGER : at org.exolab.core.mipc.QueueInputStream.receive(QueueInputStream.java:95) Jun 8 09:50:00 SERVER LOGGER : at org.exolab.core.mipc.ObjectChannel.receive(ObjectChannel.java:141) Jun 8 09:50:00 SERVER LOGGER : at org.exolab.jms.server.mipc.IpcServerChannel.run(IpcServerChannel.java:160) Jun 8 09:50:00 SERVER LOGGER : DEBUG MultiplexConnectionServer MultiplexConnectionServer accept()'ed connection Jun 8 09:50:00 SERVER LOGGER : DEBUG Demultiplexer Demultiplexer terminating on exception Jun 8 09:50:00 SERVER LOGGER : java.io.EOFException Jun 8 09:50:00 SERVER LOGGER : at java.io.DataInputStream.readInt(Unknown Source) Jun 8 09:50:00 SERVER LOGGER : at org.exolab.core.mipc.MessageInputStream.receive(MessageInputStream.java:95) Jun 8 09:50:00 SERVER LOGGER : at org.exolab.core.mipc.Demultiplexer.run(Demultiplexer.java:153) Jun 8 09:50:00 SERVER LOGGER : at org.exolab.core.mipc.MultiplexConnection.run(MultiplexConnection.java:240) Jun 8 09:50:00 SERVER OUT/ERR : Checking for Notification/Message Service (OpenJMS Server): 0 some times later, an administration script tries to delete the 'probe004' queue; the method call to 'removeDestination' hangs forever Jun 8 09:50:18 ADMIN LOG : Change in requested configuration Jun 8 09:50:18 ADMIN LOG : Launching Notification/Message Service Adaptation script Jun 8 09:50:18 ADMIN LOG : Waiting for Notification/Message Service Adaptation script to finish (max 30 seconds) Jun 8 09:50:18 SERVER LOGGER : DEBUG MultiplexConnectionServer MultiplexConnectionServer accept()'ed connection Jun 8 09:50:19 SERVER LOGGER : DEBUG DestinationManager deleteAdministeredDestination(dest=probe004-true) Jun 8 09:50:39 SERVER LOGGER : DEBUG GarbageCollectionService GC Threshold=406320 Free=122000 Jun 8 09:50:39 SERVER LOGGER : INFO GarbageCollectionService GC Collecting Garbage Free Heap below 20 30 seconds after, the administration script is killed (kill -9 on its JVM). Its log shows that it was blocked on its first 'removeDestination' method call Jun 8 09:50:48 ADMIN LOG : The Notification/Message Service Administration script has not finished in 30 seconds, killing it. Output : Jun 8 09:50:49 ADMIN LOG : LEVEL 3 : Starting Java Notification/Message Administration script Jun 8 09:50:50 ADMIN LOG : LEVEL 5 : Topic owsTopic requested Jun 8 09:50:50 ADMIN LOG : LEVEL 5 : Durable consumer OMT requested on topic owsTopic Jun 8 09:50:50 ADMIN LOG : LEVEL 5 : Durable consumer OIT-CAPT requested on topic owsTopic Jun 8 09:50:50 ADMIN LOG : LEVEL 5 : Durable consumer OIT-FO requested on topic owsTopic Jun 8 09:50:50 ADMIN LOG : LEVEL 5 : Queue probe001 requested Jun 8 09:50:50 ADMIN LOG : LEVEL 5 : Queue TestQueue1 requested Jun 8 09:50:50 ADMIN LOG : LEVEL 5 : Queue TestQueue2 requested Jun 8 09:50:50 ADMIN LOG : LEVEL 5 : Queue refappli requested Jun 8 09:50:50 ADMIN LOG : LEVEL 5 : Topic TestTopic1 requested Jun 8 09:50:50 ADMIN LOG : LEVEL 5 : Topic TestTopic2 requested Jun 8 09:50:50 ADMIN LOG : LEVEL 5 : Durable consumer DurableConsumer1 requested on topic TestTopic1 Jun 8 09:50:50 ADMIN LOG : LEVEL 5 : Durable consumer DurableConsumer2 requested on topic TestTopic2 Jun 8 09:50:50 ADMIN LOG : LEVEL 5 : Queue probe002 requested Jun 8 09:50:50 ADMIN LOG : LEVEL 5 : Queue probe003 requested Jun 8 09:50:50 ADMIN LOG : LEVEL 5 : Queue probe005 requested Jun 8 09:50:50 ADMIN LOG : LEVEL 5 : Queue probe006 requested Jun 8 09:50:50 ADMIN LOG : LEVEL 5 : Queue probe007 requested Jun 8 09:50:50 ADMIN LOG : LEVEL 5 : Queue probe008 requested Jun 8 09:50:51 ADMIN LOG : LEVEL 4 : Retrieving OpenJMS Server Administration interface Jun 8 09:50:51 ADMIN LOG : LEVEL 3 : Configuring OpenJMS Server for requested Destinations Jun 8 09:50:51 ADMIN LOG : LEVEL 4 : Retrieving existing Destinations Jun 8 09:50:51 ADMIN LOG : LEVEL 4 : Already existing queues : [probe004] [probe008] [probe003] [TestQueue2] [probe007] [TestQueue1] [probe002] [probe006] [probe001] [refappli] [probe005] Jun 8 09:50:51 ADMIN LOG : LEVEL 4 : Already existing topics : [owsTopic] [TestTopic2] [TestTopic1] Jun 8 09:50:51 ADMIN LOG : LEVEL 4 : Destroying queue probe004 Meanwhile, the same periodic status script launches itself every 30 seconds (opening and closing QueueConnections and TopicConnections), and always succeeds I then generated a thread dump of the OpenJMS Server (kill -QUIT); it shows that several threads are waiting to lock objects hold by other threads EventManager-Worker-0 waits to acquire lock on <0x44a1f500> server-5 locked <0x44a1f500>, and waits to acquire lock on <0x44a1f490> server-2 locked <0x44a1f490>, and waits to acquire lock on <0x44a4f860> Scheduler-Worker-0 locked <0x44a4f860>, and waits on object <0x44a4f810> Jun 8 09:52:55 SERVER OUT/ERR : Full thread dump Java HotSpot(TM) Client VM (1.4.2_02-b03 mixed mode): Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "EventManager-Worker-4" daemon prio=1 tid=0x0823b6b8 nid=0x1994 in Object.wait() [4ee94000..4ee948b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : - waiting on <0x444b0130> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.util.FifoQueue.waitWhileEmpty(FifoQueue.java:308) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x444b0130> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.util.FifoQueue.get(FifoQueue.java:207) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x444b0130> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.runWork(ThreadPoolWorker.java:176) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.access$000(ThreadPoolWorker.java:67) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker$1.run(ThreadPoolWorker.java:122) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Thread.run(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "EventManager-Worker-3" daemon prio=1 tid=0x0823ab28 nid=0x1993 in Object.wait() [4ee13000..4ee138b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : - waiting on <0x444b01c8> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.util.FifoQueue.waitWhileEmpty(FifoQueue.java:308) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x444b01c8> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.util.FifoQueue.get(FifoQueue.java:207) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x444b01c8> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.runWork(ThreadPoolWorker.java:176) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.access$000(ThreadPoolWorker.java:67) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker$1.run(ThreadPoolWorker.java:122) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Thread.run(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "EventManager-Worker-2" daemon prio=1 tid=0x0823a398 nid=0x1992 in Object.wait() [4ed92000..4ed928b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : - waiting on <0x444b0260> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.util.FifoQueue.waitWhileEmpty(FifoQueue.java:308) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x444b0260> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.util.FifoQueue.get(FifoQueue.java:207) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x444b0260> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.runWork(ThreadPoolWorker.java:176) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.access$000(ThreadPoolWorker.java:67) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker$1.run(ThreadPoolWorker.java:122) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Thread.run(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "EventManager-Worker-1" daemon prio=1 tid=0x0823a008 nid=0x1991 waiting for monitor entry [4ed11000..4ed118b8] Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.server.mipc.IpcJmsSessionSender.onMessage(IpcJmsSessionSender.java:111) Jun 8 09:52:55 SERVER OUT/ERR : - waiting to lock <0x44a4f860> (a org.exolab.jms.server.mipc.IpcJmsSessionSender) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.server.JmsServerSession.isClientEndpointActive(JmsServerSession.java:986) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.server.JmsServerConnection.isClientEndpointActive(JmsServerConnection.java:271) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.server.JmsServerConnectionManager.handleEvent(JmsServerConnectionManager.java:242) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.events.BasicEventManager$QueueEntry.run(BasicEventManager.java:383) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.runIt(ThreadPoolWorker.java:191) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.runWork(ThreadPoolWorker.java:178) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.access$000(ThreadPoolWorker.java:67) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker$1.run(ThreadPoolWorker.java:122) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Thread.run(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "EventManager-Worker-0" daemon prio=1 tid=0x08239cb0 nid=0x1990 waiting for monitor entry [4ec90000..4ec908b8] Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.messagemgr.DestinationManager.collectGarbage(DestinationManager.java:739) Jun 8 09:52:55 SERVER OUT/ERR : - waiting to lock <0x44a1f500> (a org.exolab.jms.messagemgr.DestinationManager) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.gc.GarbageCollectionService.collectGarbage(GarbageCollectionService.java:412) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.gc.GarbageCollectionService.handleEvent(GarbageCollectionService.java:357) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.events.BasicEventManager$QueueEntry.run(BasicEventManager.java:383) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.runIt(ThreadPoolWorker.java:191) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.runWork(ThreadPoolWorker.java:178) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.access$000(ThreadPoolWorker.java:67) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker$1.run(ThreadPoolWorker.java:122) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Thread.run(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "MessageCopier-5" daemon prio=1 tid=0x08235c28 nid=0x172f in Object.wait() [4ec0f000..4ec0f8b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : - waiting on <0x444b0438> (a java.util.LinkedList) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.Queue.remove(Queue.java:93) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x444b0438> (a java.util.LinkedList) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.QueueInputStream.receive(QueueInputStream.java:93) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.MessageCopier.copy(MessageCopier.java:172) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.MessageCopier.run(MessageCopier.java:150) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "MultiplexConnection-bsgroup3.nss-airframer-op.aircraft.airliner.tld:3030" daemon prio=1 tid=0x08235898 nid=0x172e runnable [4eb8e000..4eb8e8b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.net.SocketInputStream.socketRead0(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : at java.net.SocketInputStream.read(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at java.net.SocketInputStream.read(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at java.io.DataInputStream.readInt(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.MessageInputStream.receive(MessageInputStream.java:95) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x444b04e0> (a java.net.SocketInputStream) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.Demultiplexer.run(Demultiplexer.java:153) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.MultiplexConnection.run(MultiplexConnection.java:240) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "server-5" prio=1 tid=0x082355d8 nid=0x172d waiting for monitor entry [4e582000..4e5828b8] Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.messagemgr.ConsumerManager.getEndpointsForDest(ConsumerManager.java:878) Jun 8 09:52:55 SERVER OUT/ERR : - waiting to lock <0x44a1f490> (a org.exolab.jms.messagemgr.ConsumerManager) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.messagemgr.DestinationManager.deleteAdministeredDestination(DestinationManager.java:487) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a1f500> (a org.exolab.jms.messagemgr.DestinationManager) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.server.AdminConnection.removeDestination(AdminConnection.java:436) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.server.mipc.IpcJmsAdminConnection.removeDestination(IpcJmsAdminConnection.java:331) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.server.mipc.IpcJmsAdminConnection.notify(IpcJmsAdminConnection.java:117) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.server.mipc.IpcJmsReceiver.notify(IpcJmsReceiver.java:100) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.server.mipc.IpcServerChannel.run(IpcServerChannel.java:161) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "MessageCopier-2" daemon prio=1 tid=0x0806e1b0 nid=0x1630 in Object.wait() [4eb0d000..4eb0d8b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : - waiting on <0x444b0738> (a java.util.LinkedList) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.Queue.remove(Queue.java:93) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x444b0738> (a java.util.LinkedList) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.QueueInputStream.receive(QueueInputStream.java:93) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.MessageCopier.copy(MessageCopier.java:172) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.MessageCopier.run(MessageCopier.java:150) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "MultiplexConnection-ANSU-OPS-2.nss-airframer-op.aircraft.airliner.tld:3030" daemon prio=1 tid=0x0806d440 nid=0x162f runnable [4ea8c000..4ea8c8b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.net.SocketInputStream.socketRead0(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : at java.net.SocketInputStream.read(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at java.net.SocketInputStream.read(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at java.io.DataInputStream.readInt(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.MessageInputStream.receive(MessageInputStream.java:95) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x444b07e0> (a java.net.SocketInputStream) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.Demultiplexer.run(Demultiplexer.java:153) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.MultiplexConnection.run(MultiplexConnection.java:240) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "server-2" prio=1 tid=0x0806c8b0 nid=0x162e waiting for monitor entry [4ea0b000..4ea0b8b8] Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.server.mipc.IpcJmsSessionSender.onMessage(IpcJmsSessionSender.java:111) Jun 8 09:52:55 SERVER OUT/ERR : - waiting to lock <0x44a4f860> (a org.exolab.jms.server.mipc.IpcJmsSessionSender) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.server.JmsServerSession.isClientEndpointActive(JmsServerSession.java:986) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.messagemgr.ConsumerManager.cleanUnreferencedEndpoints(ConsumerManager.java:859) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.messagemgr.ConsumerManager.createConsumerEndpoint(ConsumerManager.java:356) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a1f490> (a org.exolab.jms.messagemgr.ConsumerManager) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.server.JmsServerSession.createReceiver(JmsServerSession.java:593) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.server.mipc.IpcJmsSessionConnection.createReceiver(IpcJmsSessionConnection.java:458) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.server.mipc.IpcJmsSessionConnection.notify(IpcJmsSessionConnection.java:147) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.server.mipc.IpcJmsReceiver.notify(IpcJmsReceiver.java:100) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.server.mipc.IpcServerChannel.run(IpcServerChannel.java:161) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "Thread-5" prio=1 tid=0x084d7d28 nid=0x162d runnable [4e98a000..4e98a8b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.net.SocketInputStream.socketRead0(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : at java.net.SocketInputStream.read(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at java.net.SocketInputStream.read(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at java.io.DataInputStream.readInt(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.ipc.TcpService.receive(TcpService.java:208) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.ipc.Notifier.start(Notifier.java:149) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.ipc.Notifier.access$000(Notifier.java:79) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.ipc.Notifier$1.run(Notifier.java:120) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Thread.run(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "MessageCopier-1" daemon prio=1 tid=0x084d76e8 nid=0x15b2 in Object.wait() [4e909000..4e9098b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : - waiting on <0x44a5e628> (a java.util.LinkedList) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.Queue.remove(Queue.java:93) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a5e628> (a java.util.LinkedList) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.QueueInputStream.receive(QueueInputStream.java:93) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.MessageCopier.copy(MessageCopier.java:172) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.MessageCopier.run(MessageCopier.java:150) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "MultiplexConnection-ANSU-OPS-2.nss-airframer-op.aircraft.airliner.tld:3030" daemon prio=1 tid=0x084d92b8 nid=0x15b1 runnable [4e888000..4e8888b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.net.SocketInputStream.socketRead0(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : at java.net.SocketInputStream.read(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at java.net.SocketInputStream.read(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at java.io.DataInputStream.readInt(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.MessageInputStream.receive(MessageInputStream.java:95) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a5e6d0> (a java.net.SocketInputStream) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.Demultiplexer.run(Demultiplexer.java:153) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.MultiplexConnection.run(MultiplexConnection.java:240) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "server-1" prio=1 tid=0x084d96d8 nid=0x15b0 in Object.wait() [4e807000..4e8078b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : - waiting on <0x44a5e798> (a java.util.LinkedList) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.Queue.remove(Queue.java:93) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a5e798> (a java.util.LinkedList) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.QueueInputStream.receive(QueueInputStream.java:93) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.ObjectChannel.receive(ObjectChannel.java:141) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.server.mipc.IpcServerChannel.run(IpcServerChannel.java:160) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "Clock Daemon" daemon prio=1 tid=0x084aeee8 nid=0x15af waiting on condition [4e786000..4e7868b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Thread.sleep(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.util.Clock.run(Clock.java:210) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "MessageCopier-0" daemon prio=1 tid=0x084a9c08 nid=0x15ae in Object.wait() [4e705000..4e7058b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : - waiting on <0x44a1e8f8> (a java.util.LinkedList) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.Queue.remove(Queue.java:93) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a1e8f8> (a java.util.LinkedList) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.QueueInputStream.receive(QueueInputStream.java:93) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.MessageCopier.copy(MessageCopier.java:172) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.MessageCopier.run(MessageCopier.java:150) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "MultiplexConnection-ANSU-OPS-2.nss-airframer-op.aircraft.airliner.tld:3030" daemon prio=1 tid=0x084a9758 nid=0x15ad runnable [4e684000..4e6848b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.net.SocketInputStream.socketRead0(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : at java.net.SocketInputStream.read(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at java.net.SocketInputStream.read(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at java.io.DataInputStream.readInt(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.MessageInputStream.receive(MessageInputStream.java:95) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a1e990> (a java.net.SocketInputStream) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.Demultiplexer.run(Demultiplexer.java:153) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.MultiplexConnection.run(MultiplexConnection.java:240) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "server-0" prio=1 tid=0x084ac138 nid=0x15ac in Object.wait() [4e603000..4e6038b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : - waiting on <0x44a4a488> (a java.util.LinkedList) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.Queue.remove(Queue.java:93) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a4a488> (a java.util.LinkedList) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.QueueInputStream.receive(QueueInputStream.java:93) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.ObjectChannel.receive(ObjectChannel.java:141) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.server.mipc.IpcServerChannel.run(IpcServerChannel.java:160) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "Thread-3" prio=1 tid=0x080f2950 nid=0x15aa runnable [4e501000..4e5018b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.net.SocketInputStream.socketRead0(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : at java.net.SocketInputStream.read(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at java.net.SocketInputStream.read(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at java.io.DataInputStream.readInt(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.ipc.TcpService.receive(TcpService.java:208) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.ipc.Notifier.start(Notifier.java:149) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.ipc.Notifier.access$000(Notifier.java:79) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.ipc.Notifier$1.run(Notifier.java:120) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Thread.run(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "DestroyJavaVM" prio=1 tid=0x4e301150 nid=0x1582 waiting on condition [0..bfffd404] Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "Thread-2" prio=1 tid=0x4e304e90 nid=0x15a7 runnable [4e480000..4e4808b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.net.PlainSocketImpl.socketAccept(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : at java.net.PlainSocketImpl.accept(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a1ec30> (a java.net.PlainSocketImpl) Jun 8 09:52:55 SERVER OUT/ERR : at java.net.ServerSocket.implAccept(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at java.net.ServerSocket.accept(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.ipc.TcpServerService.start(TcpServerService.java:154) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.ipc.Server.run(Server.java:147) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Thread.run(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "Thread-1" prio=1 tid=0x4e304a68 nid=0x15a6 runnable [4e2be000..4e2be8b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.net.PlainSocketImpl.socketAccept(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : at java.net.PlainSocketImpl.accept(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a1ed90> (a java.net.PlainSocketImpl) Jun 8 09:52:55 SERVER OUT/ERR : at java.net.ServerSocket.implAccept(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at java.net.ServerSocket.accept(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.MultiplexConnectionServer.run(MultiplexConnectionServer.java:152) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "LeaseManagerReaper" prio=1 tid=0x08498fc8 nid=0x15a4 in Object.wait() [4e23d000..4e23d8b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : - waiting on <0x44a1baa0> (a java.lang.Object) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.lease.LeaseManager.run(LeaseManager.java:251) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a1baa0> (a java.lang.Object) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Thread.run(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "Scheduler" prio=1 tid=0x08498da0 nid=0x15a3 in Object.wait() [4e1bc000..4e1bc8b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : - waiting on <0x44a1bac8> (a java.util.LinkedList) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.scheduler.Scheduler.next(Scheduler.java:270) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a1bac8> (a java.util.LinkedList) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.scheduler.Scheduler.run(Scheduler.java:240) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Thread.run(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "EventManagerThread" prio=1 tid=0x084969e0 nid=0x15a2 in Object.wait() [4e13b000..4e13b8b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : - waiting on <0x449b0118> (a java.lang.Object) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.events.BasicEventManager.run(BasicEventManager.java:277) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x449b0118> (a java.lang.Object) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Thread.run(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "ThreadPoolManager" prio=1 tid=0x08496730 nid=0x15a1 in Object.wait() [4e0ba000..4e0ba8b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : - waiting on <0x449ac118> (a org.exolab.jms.threads.ThreadPoolManager) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.threads.ThreadPoolManager.run(ThreadPoolManager.java:195) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x449ac118> (a org.exolab.jms.threads.ThreadPoolManager) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Thread.run(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "Scheduler-Worker-9" daemon prio=1 tid=0x083c4128 nid=0x15a0 in Object.wait() [4e039000..4e0398b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : - waiting on <0x44a1bb50> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.util.FifoQueue.waitWhileEmpty(FifoQueue.java:308) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a1bb50> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.util.FifoQueue.get(FifoQueue.java:207) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a1bb50> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.runWork(ThreadPoolWorker.java:176) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.access$000(ThreadPoolWorker.java:67) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker$1.run(ThreadPoolWorker.java:122) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Thread.run(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "Scheduler-Worker-8" daemon prio=1 tid=0x083dc2e8 nid=0x159f in Object.wait() [4dfb8000..4dfb88b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : - waiting on <0x44a1bbe8> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.util.FifoQueue.waitWhileEmpty(FifoQueue.java:308) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a1bbe8> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.util.FifoQueue.get(FifoQueue.java:207) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a1bbe8> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.runWork(ThreadPoolWorker.java:176) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.access$000(ThreadPoolWorker.java:67) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker$1.run(ThreadPoolWorker.java:122) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Thread.run(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "Scheduler-Worker-7" daemon prio=1 tid=0x08305590 nid=0x159e in Object.wait() [4df37000..4df378b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : - waiting on <0x44a1bc80> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.util.FifoQueue.waitWhileEmpty(FifoQueue.java:308) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a1bc80> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.util.FifoQueue.get(FifoQueue.java:207) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a1bc80> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.runWork(ThreadPoolWorker.java:176) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.access$000(ThreadPoolWorker.java:67) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker$1.run(ThreadPoolWorker.java:122) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Thread.run(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "Scheduler-Worker-6" daemon prio=1 tid=0x083d2f70 nid=0x159d in Object.wait() [4deb6000..4deb68b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : - waiting on <0x44a1bd18> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.util.FifoQueue.waitWhileEmpty(FifoQueue.java:308) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a1bd18> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.util.FifoQueue.get(FifoQueue.java:207) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a1bd18> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.runWork(ThreadPoolWorker.java:176) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.access$000(ThreadPoolWorker.java:67) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker$1.run(ThreadPoolWorker.java:122) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Thread.run(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "Scheduler-Worker-5" daemon prio=1 tid=0x083d2df0 nid=0x159c in Object.wait() [4de35000..4de358b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : - waiting on <0x44a1bdb0> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.util.FifoQueue.waitWhileEmpty(FifoQueue.java:308) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a1bdb0> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.util.FifoQueue.get(FifoQueue.java:207) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a1bdb0> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.runWork(ThreadPoolWorker.java:176) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.access$000(ThreadPoolWorker.java:67) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker$1.run(ThreadPoolWorker.java:122) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Thread.run(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "Scheduler-Worker-4" daemon prio=1 tid=0x0843b520 nid=0x159b in Object.wait() [4ddb4000..4ddb48b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : - waiting on <0x44a1be48> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.util.FifoQueue.waitWhileEmpty(FifoQueue.java:308) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a1be48> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.util.FifoQueue.get(FifoQueue.java:207) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a1be48> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.runWork(ThreadPoolWorker.java:176) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.access$000(ThreadPoolWorker.java:67) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker$1.run(ThreadPoolWorker.java:122) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Thread.run(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "Scheduler-Worker-3" daemon prio=1 tid=0x0848ce90 nid=0x159a in Object.wait() [4dd33000..4dd338b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : - waiting on <0x44a1bee0> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.util.FifoQueue.waitWhileEmpty(FifoQueue.java:308) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a1bee0> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.util.FifoQueue.get(FifoQueue.java:207) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a1bee0> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.runWork(ThreadPoolWorker.java:176) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.access$000(ThreadPoolWorker.java:67) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker$1.run(ThreadPoolWorker.java:122) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Thread.run(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "Scheduler-Worker-2" daemon prio=1 tid=0x083d2b68 nid=0x1599 in Object.wait() [4dcb2000..4dcb28b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : - waiting on <0x44a1bf78> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.util.FifoQueue.waitWhileEmpty(FifoQueue.java:308) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a1bf78> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.util.FifoQueue.get(FifoQueue.java:207) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a1bf78> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.runWork(ThreadPoolWorker.java:176) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.access$000(ThreadPoolWorker.java:67) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker$1.run(ThreadPoolWorker.java:122) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Thread.run(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "Scheduler-Worker-1" daemon prio=1 tid=0x083cef18 nid=0x1598 in Object.wait() [4dc31000..4dc318b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : - waiting on <0x44a1c010> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.util.FifoQueue.waitWhileEmpty(FifoQueue.java:308) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a1c010> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.util.FifoQueue.get(FifoQueue.java:207) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a1c010> (a org.exolab.core.util.FifoQueue) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.runWork(ThreadPoolWorker.java:176) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.access$000(ThreadPoolWorker.java:67) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker$1.run(ThreadPoolWorker.java:122) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Thread.run(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "Scheduler-Worker-0" daemon prio=1 tid=0x083d4ac0 nid=0x1597 in Object.wait() [4dbb0000..4dbb08b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : - waiting on <0x44a4f810> (a java.util.LinkedList) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.Queue.remove(Queue.java:93) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a4f810> (a java.util.LinkedList) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.QueueInputStream.receive(QueueInputStream.java:93) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.mipc.ObjectChannel.receive(ObjectChannel.java:141) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.server.mipc.IpcJmsSessionSender.onMessage(IpcJmsSessionSender.java:123) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a4f860> (a org.exolab.jms.server.mipc.IpcJmsSessionSender) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.server.JmsServerSession.onMessage(JmsServerSession.java:942) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.messagemgr.QueueConsumerEndpoint.deliverMessages(QueueConsumerEndpoint.java:177) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.jms.messagemgr.ConsumerEndpoint.run(ConsumerEndpoint.java:425) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44a4f950> (a java.lang.Object) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.runIt(ThreadPoolWorker.java:191) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.runWork(ThreadPoolWorker.java:178) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker.access$000(ThreadPoolWorker.java:67) Jun 8 09:52:55 SERVER OUT/ERR : at org.exolab.core.threadPool.ThreadPoolWorker$1.run(ThreadPoolWorker.java:122) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Thread.run(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "Thread-0" daemon prio=1 tid=0x081eaa50 nid=0x1592 waiting on condition [4d04d000..4d04d8b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Thread.sleep(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : at org.apache.commons.pool.impl.GenericObjectPool$Evictor.run(GenericObjectPool.java:1122) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Thread.run(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "Signal Dispatcher" daemon prio=1 tid=0x0808eaa0 nid=0x1589 waiting on condition [0..0] Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "Finalizer" daemon prio=1 tid=0x0808b810 nid=0x1587 in Object.wait() [4c632000..4c6328b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : - waiting on <0x44920c88> (a java.lang.ref.ReferenceQueue$Lock) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.ref.ReferenceQueue.remove(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44920c88> (a java.lang.ref.ReferenceQueue$Lock) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.ref.ReferenceQueue.remove(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "Reference Handler" daemon prio=1 tid=0x08089cf0 nid=0x1586 in Object.wait() [4c5b1000..4c5b18b8] Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Native Method) Jun 8 09:52:55 SERVER OUT/ERR : - waiting on <0x44920cf0> (a java.lang.ref.Reference$Lock) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.Object.wait(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source) Jun 8 09:52:55 SERVER OUT/ERR : - locked <0x44920cf0> (a java.lang.ref.Reference$Lock) Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "VM Thread" prio=1 tid=0x08086aa8 nid=0x1585 runnable Jun 8 09:52:55 SERVER OUT/ERR : Jun 8 09:52:55 SERVER OUT/ERR : "VM Periodic Task Thread" prio=1 tid=0x080911c8 nid=0x158b waiting on condition Jun 8 09:52:55 SERVER OUT/ERR : "Suspend Checker Thread" prio=1 tid=0x0808e068 nid=0x1588 runnable ------------------------------------------------------- This SF.Net email is sponsored by: NEC IT Guy Games. How far can you shotput a projector? How fast can you ride your desk chair down the office luge track? If you want to score the big prize, get to know the little guy. Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20