RE: Help on JVM hang
"Veda N Ponnusamy" <[email protected]>
| Newsgroups | gmane.comp.java.linux.general |
|---|---|
| Message-ID | <[email protected]> |
Please find attached the thread dump of the JVM. I didn't try running this app on 1.5 jvm Reg Veda -----Original Message----- From: Joseph Shraibman [mailto:[email protected]] Sent: Friday, June 11, 2004 1:26 PM To: Calvin Austin Cc: Veda N Ponnusamy; [email protected]; [email protected] Subject: Re: Help on JVM hang Calvin Austin wrote: > Does it ever resume. 1.5beta2 has more diagnostics in it and you may be > able to even attached with jdb (hey tim :*) > > Without seeing the other threads its difficult to see what is happening, > however there was a bug fix in 1.5 to stop the cache growing out of > control. A quick test with 1.5 would help a great deal. > I was never able to reproduce the hang on a test machine, it only happened in production under load. My boss would kill me if I ran a beta jvm in production. I also haven't seen the bug for a couple of weeks since I limited the work done by that jvm in order to avoid the bug. And I'm not going to risk freezing our system in order to hit the bug again. If I did attach to the jvm with jdb, what good would it do? What would I see that the SIGQUIT thread dump wouldn't show?
1arc-crash-204-4[1].log
(application/octet-stream, 98.4 KB)
Full thread dump Java HotSpot(TM) Client VM (1.4.2_03-b02 mixed mode): "Thread-217" prio=1 tid=0x080c32c0 nid=0x7b88 waiting on condition [6e1a1000..6e1a2854] at java.lang.Class.getMethod(Class.java:975) at com.sun.management.jmx.MetaData.findMethod(MetaData.java:181) at com.sun.management.jmx.MetaData.findGetter(MetaData.java:215) at com.sun.management.jmx.MBeanServerImpl.getAttribute(MBeanServerImpl.java:1192) at com.sun.management.jmx.MBeanServerImpl.getAttribute(MBeanServerImpl.java:1151) at com.silver.base.jmxmanager.mbbroker.MBeanBroker.getAttribute(MBeanBroker.java:446) at com.silver.base.jmxmanager.mbbroker.MBeanBroker.listMBeansInAState(MBeanBroker.java:708) at com.silver.base.jmxmanager.mbbroker.MBeanBroker.listFailedMBeans(MBeanBroker.java:734) at com.silver.base.jmxmanager.JmxManager.getFailedMBeans(JmxManager.java:225) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at com.sun.management.jmx.MBeanServerImpl.getAttribute(MBeanServerImpl.java:1205) at com.sun.management.jmx.MBeanServerImpl.getAttribute(MBeanServerImpl.java:1151) at com.sun.jdmk.comm.HtmlObjectPage.getStringValue(HtmlObjectPage.java:1023) at com.sun.jdmk.comm.HtmlObjectPage.analyseObj(HtmlObjectPage.java:662) at com.sun.jdmk.comm.HtmlObjectPage.buildPage(HtmlObjectPage.java:81) at com.sun.jdmk.comm.HtmlRequestHandler.processGetRequest(HtmlRequestHandler.java:258) at com.sun.jdmk.comm.HtmlRequestHandler.processRequest(HtmlRequestHandler.java:152) at com.sun.jdmk.comm.HtmlRequestHandler.doRun(HtmlRequestHandler.java:79) at com.sun.jdmk.comm.ClientHandler.run(ClientHandler.java:84) at java.lang.Thread.run(Thread.java:534) "Thread-211" prio=1 tid=0x080c6868 nid=0x7b88 waiting on condition [6e121000..6e121854] at javax.management.ObjectInstance.<init>(ObjectInstance.java:73) at com.sun.management.jmx.MBeanServerImpl.filterListOfObjects(MBeanServerImpl.java:2583) at com.sun.management.jmx.MBeanServerImpl.searchMBeans(MBeanServerImpl.java:1047) at com.sun.management.jmx.MBeanServerImpl.searchNames(MBeanServerImpl.java:1073) at com.sun.jdmk.comm.HtmlMasterPage.buildPage(HtmlMasterPage.java:93) at com.sun.jdmk.comm.HtmlRequestHandler.processGetRequest(HtmlRequestHandler.java:332) at com.sun.jdmk.comm.HtmlRequestHandler.processRequest(HtmlRequestHandler.java:152) at com.sun.jdmk.comm.HtmlRequestHandler.doRun(HtmlRequestHandler.java:79) at com.sun.jdmk.comm.ClientHandler.run(ClientHandler.java:84) at java.lang.Thread.run(Thread.java:534) "Thread-210" prio=1 tid=0x087b8cb8 nid=0x7b88 waiting for monitor entry [6e09f000..6e0a0854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2410) at com.silver.base.database.ConnectionPool.<init>(ConnectionPool.java:67) at com.silver.base.indexer.error.ErrorAccessor.getConnection(ErrorAccessor.java:585) - locked <0x4717a028> (a java.lang.Object) at com.silver.base.indexer.error.ErrorAccessor.getNumberEntries(ErrorAccessor.java:391) at com.silver.base.indexer.mbean.Indexer.getNFailedDocs(Indexer.java:788) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at com.sun.management.jmx.MBeanServerImpl.getAttribute(MBeanServerImpl.java:1205) at com.sun.management.jmx.MBeanServerImpl.getAttribute(MBeanServerImpl.java:1151) at com.sun.jdmk.comm.HtmlObjectPage.getStringValue(HtmlObjectPage.java:1023) at com.sun.jdmk.comm.HtmlObjectPage.analyseObj(HtmlObjectPage.java:662) at com.sun.jdmk.comm.HtmlObjectPage.buildPage(HtmlObjectPage.java:81) at com.sun.jdmk.comm.HtmlRequestHandler.processGetRequest(HtmlRequestHandler.java:258) at com.sun.jdmk.comm.HtmlRequestHandler.processRequest(HtmlRequestHandler.java:152) at com.sun.jdmk.comm.HtmlRequestHandler.doRun(HtmlRequestHandler.java:79) at com.sun.jdmk.comm.ClientHandler.run(ClientHandler.java:84) at java.lang.Thread.run(Thread.java:534) "Thread-206" prio=1 tid=0x087b7d18 nid=0x7b88 waiting on condition [6e01e000..6e01f854] at java.util.Stack.pop(Stack.java:59) at org.apache.log4j.NDC.pop(NDC.java:274) at com.silver.base.jmxmanager.mbbroker.MBeanBroker.invoke(MBeanBroker.java:405) at com.silver.base.cellstate.ZProxy.subnetRange(ZProxy.java:515) at com.silver.base.cellstate.ZProxy.lookupSubnet(ZProxy.java:484) at com.silver.base.cellstate.SmartCellStateController.getSubnetRange(SmartCellStateController.java:165) at com.silver.base.cellstate.SmartCellStateControllerSC.getSubnetRange(SmartCellStateControllerSC.java:231) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at com.sun.management.jmx.MBeanServerImpl.getAttribute(MBeanServerImpl.java:1205) at com.sun.management.jmx.MBeanServerImpl.getAttribute(MBeanServerImpl.java:1151) at com.sun.jdmk.comm.HtmlObjectPage.getStringValue(HtmlObjectPage.java:1023) at com.sun.jdmk.comm.HtmlObjectPage.analyseObj(HtmlObjectPage.java:662) at com.sun.jdmk.comm.HtmlObjectPage.buildPage(HtmlObjectPage.java:81) at com.sun.jdmk.comm.HtmlRequestHandler.processGetRequest(HtmlRequestHandler.java:258) at com.sun.jdmk.comm.HtmlRequestHandler.processRequest(HtmlRequestHandler.java:152) at com.sun.jdmk.comm.HtmlRequestHandler.doRun(HtmlRequestHandler.java:79) at com.sun.jdmk.comm.ClientHandler.run(ClientHandler.java:84) at java.lang.Thread.run(Thread.java:534) "Thread-204" prio=1 tid=0x08738c10 nid=0x7b88 waiting on condition [6df9d000..6df9e854] at java.util.Stack.push(Stack.java:44) at org.apache.log4j.NDC.push(NDC.java:302) at com.silver.base.jmxmanager.mbbroker.MBeanBroker.getAttribute(MBeanBroker.java:429) at com.silver.base.jmxmanager.mbbroker.MBeanBroker.listMBeansInAState(MBeanBroker.java:695) at com.silver.base.jmxmanager.mbbroker.MBeanBroker.listUnhealthyMBeans(MBeanBroker.java:735) at com.silver.base.jmxmanager.JmxManager.getUnhealthyMBeans(JmxManager.java:230) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at com.sun.management.jmx.MBeanServerImpl.getAttribute(MBeanServerImpl.java:1205) at com.sun.management.jmx.MBeanServerImpl.getAttribute(MBeanServerImpl.java:1151) at com.sun.jdmk.comm.HtmlObjectPage.getStringValue(HtmlObjectPage.java:1023) at com.sun.jdmk.comm.HtmlObjectPage.analyseObj(HtmlObjectPage.java:662) at com.sun.jdmk.comm.HtmlObjectPage.buildPage(HtmlObjectPage.java:81) at com.sun.jdmk.comm.HtmlRequestHandler.processGetRequest(HtmlRequestHandler.java:258) at com.sun.jdmk.comm.HtmlRequestHandler.processRequest(HtmlRequestHandler.java:152) at com.sun.jdmk.comm.HtmlRequestHandler.doRun(HtmlRequestHandler.java:79) at com.sun.jdmk.comm.ClientHandler.run(ClientHandler.java:84) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(87)-10.0.71.4" daemon prio=1 tid=0x08738950 nid=0x7b88 waiting for monitor entry [6df1c000..6df1d854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2410) at com.silver.base.archiver.ArchiverCore.createFile(ArchiverCore.java:484) at com.silver.base.archiver.RemoteArchiverImpl.createFile(RemoteArchiverImpl.java:90) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "Thread-202" prio=1 tid=0x08736fa8 nid=0x7b88 waiting on condition [6dd99000..6dd9a854] at org.apache.log4j.NDC.pop(NDC.java:271) at com.silver.base.jmxmanager.mbbroker.MBeanBroker.getAttribute(MBeanBroker.java:485) at com.silver.base.jmxmanager.mbbroker.MBeanBroker.listMBeansInAState(MBeanBroker.java:695) at com.silver.base.jmxmanager.mbbroker.MBeanBroker.listUnhealthyMBeans(MBeanBroker.java:735) at com.silver.base.jmxmanager.JmxManager.getUnhealthyMBeans(JmxManager.java:230) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at com.sun.management.jmx.MBeanServerImpl.getAttribute(MBeanServerImpl.java:1205) at com.sun.management.jmx.MBeanServerImpl.getAttribute(MBeanServerImpl.java:1151) at com.sun.jdmk.comm.HtmlObjectPage.getStringValue(HtmlObjectPage.java:1023) at com.sun.jdmk.comm.HtmlObjectPage.analyseObj(HtmlObjectPage.java:662) at com.sun.jdmk.comm.HtmlObjectPage.buildPage(HtmlObjectPage.java:81) at com.sun.jdmk.comm.HtmlRequestHandler.processGetRequest(HtmlRequestHandler.java:258) at com.sun.jdmk.comm.HtmlRequestHandler.processRequest(HtmlRequestHandler.java:152) at com.sun.jdmk.comm.HtmlRequestHandler.doRun(HtmlRequestHandler.java:79) at com.sun.jdmk.comm.ClientHandler.run(ClientHandler.java:84) at java.lang.Thread.run(Thread.java:534) "Thread-200" prio=1 tid=0x0868ea58 nid=0x7b88 waiting on condition [6de9b000..6de9c854] at org.apache.log4j.NDC.push(NDC.java:289) at com.silver.base.jmxmanager.mbbroker.MBeanBroker.getAttribute(MBeanBroker.java:428) at com.silver.base.jmxmanager.mbbroker.MBeanBroker.listMBeansInAState(MBeanBroker.java:708) at com.silver.base.jmxmanager.mbbroker.MBeanBroker.listFailedMBeans(MBeanBroker.java:734) at com.silver.base.jmxmanager.JmxManager.getFailedMBeans(JmxManager.java:225) at com.silver.base.jmxmanager.JmxManager.getCellState(JmxManager.java:203) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at com.sun.management.jmx.MBeanServerImpl.getAttribute(MBeanServerImpl.java:1205) at com.sun.management.jmx.MBeanServerImpl.getAttribute(MBeanServerImpl.java:1151) at com.sun.jdmk.comm.HtmlObjectPage.getStringValue(HtmlObjectPage.java:1023) at com.sun.jdmk.comm.HtmlObjectPage.analyseObj(HtmlObjectPage.java:662) at com.sun.jdmk.comm.HtmlObjectPage.buildPage(HtmlObjectPage.java:81) at com.sun.jdmk.comm.HtmlRequestHandler.processGetRequest(HtmlRequestHandler.java:258) at com.sun.jdmk.comm.HtmlRequestHandler.processRequest(HtmlRequestHandler.java:152) at com.sun.jdmk.comm.HtmlRequestHandler.doRun(HtmlRequestHandler.java:79) at com.sun.jdmk.comm.ClientHandler.run(ClientHandler.java:84) at java.lang.Thread.run(Thread.java:534) "Thread-191" prio=1 tid=0x0868d368 nid=0x7b88 waiting on condition [6de1b000..6de1b854] at com.silver.base.jmxmanager.mbbroker.MBeanBroker.listMBeansInAState(MBeanBroker.java:684) at com.silver.base.jmxmanager.mbbroker.MBeanBroker.listFailedMBeans(MBeanBroker.java:734) at com.silver.base.jmxmanager.JmxManager.getFailedMBeans(JmxManager.java:225) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at com.sun.management.jmx.MBeanServerImpl.getAttribute(MBeanServerImpl.java:1205) at com.sun.management.jmx.MBeanServerImpl.getAttribute(MBeanServerImpl.java:1151) at com.sun.jdmk.comm.HtmlObjectPage.getStringValue(HtmlObjectPage.java:1023) at com.sun.jdmk.comm.HtmlObjectPage.analyseObj(HtmlObjectPage.java:662) at com.sun.jdmk.comm.HtmlObjectPage.buildPage(HtmlObjectPage.java:81) at com.sun.jdmk.comm.HtmlRequestHandler.processGetRequest(HtmlRequestHandler.java:258) at com.sun.jdmk.comm.HtmlRequestHandler.processRequest(HtmlRequestHandler.java:152) at com.sun.jdmk.comm.HtmlRequestHandler.doRun(HtmlRequestHandler.java:79) at com.sun.jdmk.comm.ClientHandler.run(ClientHandler.java:84) at java.lang.Thread.run(Thread.java:534) "Thread-180" prio=1 tid=0x0868e150 nid=0x7b88 waiting on condition [6dd19000..6dd19854] at javax.management.ObjectName.toString(ObjectName.java:260) at com.sun.jdmk.comm.HtmlMasterPage.buildPage(HtmlMasterPage.java:145) at com.sun.jdmk.comm.HtmlRequestHandler.processGetRequest(HtmlRequestHandler.java:332) at com.sun.jdmk.comm.HtmlRequestHandler.processRequest(HtmlRequestHandler.java:152) at com.sun.jdmk.comm.HtmlRequestHandler.doRun(HtmlRequestHandler.java:79) at com.sun.jdmk.comm.ClientHandler.run(ClientHandler.java:84) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(86)-10.0.71.1" daemon prio=1 tid=0x0868c5d0 nid=0x7b88 waiting for monitor entry [6dc97000..6dc98854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2410) at com.silver.base.archiver.ArchiverCore.createFile(ArchiverCore.java:484) at com.silver.base.archiver.RemoteArchiverImpl.createFile(RemoteArchiverImpl.java:90) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(85)-10.0.71.2" daemon prio=1 tid=0x0868b870 nid=0x7b88 waiting for monitor entry [6dc16000..6dc17854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2410) at com.silver.base.archiver.ArchiverCore.createFile(ArchiverCore.java:484) at com.silver.base.archiver.RemoteArchiverImpl.createFile(RemoteArchiverImpl.java:90) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(84)-10.0.71.2" daemon prio=1 tid=0x0868aa58 nid=0x7b88 waiting for monitor entry [6db95000..6db96854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2410) at com.silver.base.archiver.ArchiverCore.createFile(ArchiverCore.java:484) at com.silver.base.archiver.RemoteArchiverImpl.createFile(RemoteArchiverImpl.java:90) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(83)-10.0.71.3" daemon prio=1 tid=0x080c9a68 nid=0x7b88 waiting for monitor entry [6db14000..6db15854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2410) at com.silver.base.archiver.ArchiverCore.createFile(ArchiverCore.java:484) at com.silver.base.archiver.RemoteArchiverImpl.createFile(RemoteArchiverImpl.java:90) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(82)-10.0.71.4" daemon prio=1 tid=0x080c8c98 nid=0x7b88 waiting for monitor entry [6da93000..6da94854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2410) at com.silver.base.archiver.ArchiverCore.createFile(ArchiverCore.java:484) at com.silver.base.archiver.RemoteArchiverImpl.createFile(RemoteArchiverImpl.java:90) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(81)-10.0.71.3" daemon prio=1 tid=0x080c7f90 nid=0x7b88 waiting for monitor entry [6da12000..6da13854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2410) at com.silver.base.archiver.ArchiverCore.createFile(ArchiverCore.java:484) at com.silver.base.archiver.RemoteArchiverImpl.createFile(RemoteArchiverImpl.java:90) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(80)-10.0.71.4" daemon prio=1 tid=0x086373a8 nid=0x7b88 waiting for monitor entry [6a9bb000..6a9bc854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2410) at com.silver.base.archiver.ArchiverCore.createFile(ArchiverCore.java:484) at com.silver.base.archiver.RemoteArchiverImpl.createFile(RemoteArchiverImpl.java:90) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(79)-10.0.71.1" daemon prio=1 tid=0x080c7240 nid=0x7b88 waiting for monitor entry [6d991000..6d992854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2410) at com.silver.base.archiver.ArchiverCore.createFile(ArchiverCore.java:484) at com.silver.base.archiver.RemoteArchiverImpl.createFile(RemoteArchiverImpl.java:90) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(78)-10.0.71.1" daemon prio=1 tid=0x08689b28 nid=0x7b88 waiting for monitor entry [6d88f000..6d890854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2410) at com.silver.base.archiver.ArchiverCore.createFile(ArchiverCore.java:484) at com.silver.base.archiver.RemoteArchiverImpl.createFile(RemoteArchiverImpl.java:90) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(77)-10.0.71.1" daemon prio=1 tid=0x08358d18 nid=0x7b88 waiting for monitor entry [6d910000..6d911854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2410) at com.silver.base.archiver.ArchiverCore.createFile(ArchiverCore.java:484) at com.silver.base.archiver.RemoteArchiverImpl.createFile(RemoteArchiverImpl.java:90) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(76)-10.0.71.1" daemon prio=1 tid=0x08357fe8 nid=0x7b88 waiting for monitor entry [6d80e000..6d80f854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2410) at com.silver.base.archiver.ArchiverCore.createFile(ArchiverCore.java:484) at com.silver.base.archiver.RemoteArchiverImpl.createFile(RemoteArchiverImpl.java:90) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(75)-10.0.71.2" daemon prio=1 tid=0x083571a0 nid=0x7b88 waiting for monitor entry [6d78d000..6d78e854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2410) at com.silver.base.archiver.ArchiverCore.createFile(ArchiverCore.java:484) at com.silver.base.archiver.RemoteArchiverImpl.createFile(RemoteArchiverImpl.java:90) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(74)-10.0.71.1" daemon prio=1 tid=0x083563a0 nid=0x7b88 waiting for monitor entry [6d70c000..6d70d854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2410) at com.silver.base.archiver.ArchiverCore.createFile(ArchiverCore.java:484) at com.silver.base.archiver.RemoteArchiverImpl.createFile(RemoteArchiverImpl.java:90) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(73)-10.0.71.4" daemon prio=1 tid=0x08355640 nid=0x7b88 waiting for monitor entry [6d68b000..6d68c854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2410) at com.silver.base.archiver.ArchiverCore.createFile(ArchiverCore.java:484) at com.silver.base.archiver.RemoteArchiverImpl.createFile(RemoteArchiverImpl.java:90) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(72)-10.0.71.2" daemon prio=1 tid=0x08354ab0 nid=0x7b88 waiting for monitor entry [6d60a000..6d60b854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2410) at com.silver.base.archiver.ArchiverCore.createFile(ArchiverCore.java:484) at com.silver.base.archiver.RemoteArchiverImpl.createFile(RemoteArchiverImpl.java:90) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(71)-10.0.71.3" daemon prio=1 tid=0x080965d8 nid=0x7b88 waiting for monitor entry [6d508000..6d509854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2410) at com.silver.base.archiver.ArchiverCore.createFile(ArchiverCore.java:484) at com.silver.base.archiver.RemoteArchiverImpl.createFile(RemoteArchiverImpl.java:90) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(70)-10.0.71.1" daemon prio=1 tid=0x080973f0 nid=0x7b88 waiting for monitor entry [6d589000..6d58a854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2439) at com.silver.base.archiver.FileProxy.close(FileProxy.java:150) at com.silver.base.archiver.ArchiverCore.close(ArchiverCore.java:618) at com.silver.base.archiver.RemoteArchiverImpl.close(RemoteArchiverImpl.java:138) at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(69)-10.0.71.2" daemon prio=1 tid=0x08095960 nid=0x7b88 waiting for monitor entry [6d487000..6d488854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2410) at com.silver.base.archiver.ArchiverCore.createFile(ArchiverCore.java:484) at com.silver.base.archiver.RemoteArchiverImpl.createFile(RemoteArchiverImpl.java:90) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(68)-10.0.71.4" daemon prio=1 tid=0x08094c30 nid=0x7b88 waiting for monitor entry [6d406000..6d407854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2410) at com.silver.base.archiver.ArchiverCore.createFile(ArchiverCore.java:484) at com.silver.base.archiver.RemoteArchiverImpl.createFile(RemoteArchiverImpl.java:90) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(67)-10.0.71.3" daemon prio=1 tid=0x080948b0 nid=0x7b88 waiting for monitor entry [6d385000..6d386854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2410) at com.silver.base.archiver.ArchiverCore.createFile(ArchiverCore.java:484) at com.silver.base.archiver.RemoteArchiverImpl.createFile(RemoteArchiverImpl.java:90) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(66)-10.0.71.3" daemon prio=1 tid=0x08093ca8 nid=0x7b88 waiting for monitor entry [6d283000..6d284854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2439) at com.silver.base.archiver.FileProxy.close(FileProxy.java:150) at com.silver.base.archiver.ArchiverCore.close(ArchiverCore.java:618) at com.silver.base.archiver.RemoteArchiverImpl.close(RemoteArchiverImpl.java:138) at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(65)-10.0.71.4" daemon prio=1 tid=0x080987d8 nid=0x7b88 waiting for monitor entry [6d304000..6d305854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2410) at com.silver.base.archiver.ArchiverCore.createFile(ArchiverCore.java:484) at com.silver.base.archiver.RemoteArchiverImpl.createFile(RemoteArchiverImpl.java:90) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(64)-10.0.71.1" daemon prio=1 tid=0x08786a98 nid=0x7b88 waiting for monitor entry [6d202000..6d203854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2410) at com.silver.base.archiver.ArchiverCore.createFile(ArchiverCore.java:484) at com.silver.base.archiver.RemoteArchiverImpl.createFile(RemoteArchiverImpl.java:90) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(63)-10.0.71.3" daemon prio=1 tid=0x08373308 nid=0x7b88 waiting for monitor entry [6d100000..6d101854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2439) at com.silver.base.archiver.FileProxy.close(FileProxy.java:150) at com.silver.base.archiver.ArchiverCore.close(ArchiverCore.java:618) at com.silver.base.archiver.RemoteArchiverImpl.close(RemoteArchiverImpl.java:138) at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI ConnectionExpiration-[10.0.97.1:34079]" daemon prio=1 tid=0x08785bb0 nid=0x7b88 waiting for monitor entry [6a476000..6a476854] at sun.rmi.transport.tcp.TCPChannel.freeCachedConnections(TCPChannel.java:407) - waiting to lock <0x46ff86e0> (a java.util.ArrayList) at sun.rmi.transport.tcp.TCPChannel.access$200(TCPChannel.java:35) at sun.rmi.transport.tcp.TCPChannel$Reaper.run(TCPChannel.java:452) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(62)-10.0.71.4" daemon prio=1 tid=0x08099090 nid=0x7b88 waiting on condition [6c993000..6c994854] at sun.util.calendar.ZoneInfo.getOffsets(ZoneInfo.java:210) at sun.util.calendar.ZoneInfo.getOffsets(ZoneInfo.java:201) at java.util.GregorianCalendar.computeFieldsImpl(GregorianCalendar.java:1319) at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:1298) at java.util.Calendar.setTimeInMillis(Calendar.java:927) at java.util.Calendar.setTime(Calendar.java:902) at org.apache.log4j.helpers.ISO8601DateFormat.format(ISO8601DateFormat.java:55) at java.text.DateFormat.format(DateFormat.java:314) at org.apache.log4j.helpers.PatternParser$DatePatternConverter.convert(PatternParser.java:434) at org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:56) at org.apache.log4j.PatternLayout.format(PatternLayout.java:547) at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:295) at org.apache.log4j.RollingFileAppender.subAppend(RollingFileAppender.java:566) at org.apache.log4j.WriterAppender.append(WriterAppender.java:157) at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251) - locked <0x471bfd28> (a org.apache.log4j.RollingFileAppender) at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:57) at org.apache.log4j.Category.callAppenders(Category.java:380) - locked <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2439) at com.silver.base.archiver.FileProxyTable.removeProxy(FileProxyTable.java:135) at com.silver.base.archiver.FileProxy.evict(FileProxy.java:325) at com.silver.base.archiver.FileProxy.commit(FileProxy.java:282) at com.silver.base.archiver.ArchiverCore.commit(ArchiverCore.java:702) at com.silver.base.archiver.ArchiverCore.close(ArchiverCore.java:619) at com.silver.base.archiver.RemoteArchiverImpl.close(RemoteArchiverImpl.java:138) at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(61)-10.0.71.4" daemon prio=1 tid=0x08099fe8 nid=0x7b88 waiting on condition [6d080000..6d080854] at dictionary.ZBaseLogsServiceBase.getDictionaryHash(ZBaseLogsServiceBase.java:382) at org.apache.log4j.Category.debugParamArray(Category.java:1596) at org.apache.log4j.Category.debug(Category.java:2439) at com.silver.base.archiver.FileProxyTable.putProxy(FileProxyTable.java:110) at com.silver.base.archiver.FileProxy.<init>(FileProxy.java:121) at com.silver.base.archiver.ArchiverCore.createFile(ArchiverCore.java:495) at com.silver.base.archiver.RemoteArchiverImpl.createFile(RemoteArchiverImpl.java:90) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(60)-10.0.71.2" daemon prio=1 tid=0x0809a950 nid=0x7b88 waiting for monitor entry [6a577000..6a578854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2439) at com.silver.base.archiver.FileProxy.close(FileProxy.java:150) at com.silver.base.archiver.ArchiverCore.close(ArchiverCore.java:618) at com.silver.base.archiver.RemoteArchiverImpl.close(RemoteArchiverImpl.java:138) at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(58)-10.0.71.1" daemon prio=1 tid=0x080e8da0 nid=0x7b88 waiting on condition [6d182000..6d182854] at com.silver.base.archiver.PropertyMgr.getInstance(PropertyMgr.java:280) at com.silver.base.archiver.FileProxyTable.removeProxy(FileProxyTable.java:134) at com.silver.base.archiver.FileProxy.evict(FileProxy.java:325) at com.silver.base.archiver.FileProxy.commit(FileProxy.java:282) at com.silver.base.archiver.ArchiverCore.commit(ArchiverCore.java:702) at com.silver.base.archiver.ArchiverCore.close(ArchiverCore.java:619) at com.silver.base.archiver.RemoteArchiverImpl.close(RemoteArchiverImpl.java:138) at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "Thread-134" prio=1 tid=0x08099d78 nid=0x7b88 waiting on condition [6c790000..6c790854] at javax.management.MBeanInfo.getOperations(MBeanInfo.java:177) at com.sun.management.jmx.MBeanServerImpl.getMBeanInfo(MBeanServerImpl.java:1897) at com.sun.jdmk.comm.HtmlObjectPage.analyseObj(HtmlObjectPage.java:614) at com.sun.jdmk.comm.HtmlObjectPage.buildPage(HtmlObjectPage.java:81) at com.sun.jdmk.comm.HtmlRequestHandler.processGetRequest(HtmlRequestHandler.java:258) at com.sun.jdmk.comm.HtmlRequestHandler.processRequest(HtmlRequestHandler.java:152) at com.sun.jdmk.comm.HtmlRequestHandler.doRun(HtmlRequestHandler.java:79) at com.sun.jdmk.comm.ClientHandler.run(ClientHandler.java:84) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(55)-10.0.71.3" daemon prio=1 tid=0x08098b28 nid=0x7b88 waiting on condition [6cd99000..6cd99854] at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1284) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1247) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1052) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:278) at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:265) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:271) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(53)-10.0.71.2" daemon prio=1 tid=0x0874d550 nid=0x7b88 waiting on condition [6aabd000..6aabe854] at com.silver.base.database.PrepStmtParam.<init>(PrepStmtParam.java:13) at com.silver.base.database.PreparedStatementWrapper.setString(PreparedStatementWrapper.java:191) at com.silver.base.bitfilequeue.BitfileQueueDB.insertBitfileQueue(BitfileQueueDB.java:1301) at com.silver.base.bitfilequeue.BitfileQueueDB.addQueueEntry(BitfileQueueDB.java:157) at com.silver.base.archiver.SecondaryStorageArchiver.closeForRole(SecondaryStorageArchiver.java:86) at com.silver.base.archiver.ArchiverCore.commit(ArchiverCore.java:700) at com.silver.base.archiver.ArchiverCore.close(ArchiverCore.java:619) at com.silver.base.archiver.RemoteArchiverImpl.close(RemoteArchiverImpl.java:138) at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(48)-10.0.71.3" daemon prio=1 tid=0x086ba668 nid=0x7b88 waiting on condition [6c892000..6c892854] at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1318) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1304) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1247) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1052) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:278) at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:265) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:271) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "RMI TCP Connection(13)-10.0.71.4" daemon prio=1 tid=0x083aefe0 nid=0x7b88 waiting on condition [6c913000..6c913854] at java.io.ObjectStreamClass$FieldReflector.getObjFieldValues(ObjectStreamClass.java:1841) at java.io.ObjectStreamClass.getObjFieldValues(ObjectStreamClass.java:1067) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1330) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1304) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1247) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1052) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:278) at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:265) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:271) at sun.rmi.transport.Transport$1.run(Transport.java:148) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:144) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701) at java.lang.Thread.run(Thread.java:534) "Archiver.BitfileThrotter" daemon prio=1 tid=0x084153b8 nid=0x7b88 waiting on condition [6a372000..6a372854] at java.lang.Thread.sleep(Native Method) at com.silver.base.archiver.bitfile.BitfileThrottler.doContinuousThrottling(BitfileThrottler.java:118) at com.silver.base.archiver.bitfile.BitfileThrottler.run(BitfileThrottler.java:53) "/boston/searchServices/holdingQueue" daemon prio=1 tid=0x0806f088 nid=0x7b88 waiting for monitor entry [6ae45000..6ae45854] at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:142) - waiting to lock <0x46ff86e0> (a java.util.ArrayList) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:101) at org.jnp.server.NamingServer_Stub.lookup(Unknown Source) at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:349) at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:333) at com.silver.base.servicebroker.ServiceBroker.getItemsFromJNDI(ServiceBroker.java:151) - locked <0x471da0c0> (a com.silver.base.servicebroker.ServiceBroker) at com.silver.base.servicebroker.ServiceBroker.run(ServiceBroker.java:349) - locked <0x471da0c0> (a com.silver.base.servicebroker.ServiceBroker) at java.lang.Thread.run(Thread.java:534) "DestroyJavaVM" prio=1 tid=0x69fa9a90 nid=0x7b88 waiting on condition [0..bfffca04] "StoreIndexerStatsCollector" prio=1 tid=0x69fa8508 nid=0x7b88 waiting on condition [6c811000..6c811854] at java.lang.FloatingDecimal.countBits(FloatingDecimal.java:77) at java.lang.FloatingDecimal.dtoa(FloatingDecimal.java:509) at java.lang.FloatingDecimal.<init>(FloatingDecimal.java:496) at java.lang.Float.toString(Float.java:139) at com.silver.base.performancemonitor.StoreIndexStatsCollector.run(StoreIndexStatsCollector.java:88) at com.silver.base.performancemonitor.StartRunnableTimerTask.run(StartRunnableTimerTask.java:50) at java.util.TimerThread.mainLoop(Timer.java:432) at java.util.TimerThread.run(Timer.java:382) "AuditQueryThread9" prio=1 tid=0x69fa5ad0 nid=0x7b88 in Object.wait() [6c70f000..6c70f854] at java.lang.Object.wait(Native Method) - waiting on <0x471904e0> (a com.silver.base.scaudit.audit.SCAuditQueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:86) - locked <0x471904e0> (a com.silver.base.scaudit.audit.SCAuditQueryQueue) at com.silver.base.scaudit.audit.AuditQueryThread.run(AuditQueryThread.java:119) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "T0" prio=1 tid=0x69fa4f80 nid=0x7b88 in Object.wait() [6c68e000..6c68e854] at java.lang.Object.wait(Native Method) - waiting on <0x4718fa28> (a com.silver.base.utils.threads.Blocking_queue) at com.silver.base.utils.threads.Blocking_queue.dequeue(Blocking_queue.java:255) - locked <0x4718fa28> (a com.silver.base.utils.threads.Blocking_queue) at com.silver.base.utils.threads.Blocking_queue.dequeue(Blocking_queue.java:298) - locked <0x4718fa28> (a com.silver.base.utils.threads.Blocking_queue) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:173) "AuditQueryThread0" prio=1 tid=0x69fa4430 nid=0x7b88 in Object.wait() [6c60d000..6c60d854] at java.lang.Object.wait(Native Method) - waiting on <0x471904e0> (a com.silver.base.scaudit.audit.SCAuditQueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:86) - locked <0x471904e0> (a com.silver.base.scaudit.audit.SCAuditQueryQueue) at com.silver.base.scaudit.audit.AuditQueryThread.run(AuditQueryThread.java:119) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "AuditQueryThread1" prio=1 tid=0x69fa2720 nid=0x7b88 in Object.wait() [6c58c000..6c58c854] at java.lang.Object.wait(Native Method) - waiting on <0x471904e0> (a com.silver.base.scaudit.audit.SCAuditQueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:86) - locked <0x471904e0> (a com.silver.base.scaudit.audit.SCAuditQueryQueue) at com.silver.base.scaudit.audit.AuditQueryThread.run(AuditQueryThread.java:119) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "AuditQueryThread2" prio=1 tid=0x69fa23d0 nid=0x7b88 in Object.wait() [6c50b000..6c50b854] at java.lang.Object.wait(Native Method) - waiting on <0x471904e0> (a com.silver.base.scaudit.audit.SCAuditQueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:86) - locked <0x471904e0> (a com.silver.base.scaudit.audit.SCAuditQueryQueue) at com.silver.base.scaudit.audit.AuditQueryThread.run(AuditQueryThread.java:119) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "AuditQueryThread3" prio=1 tid=0x69fa1880 nid=0x7b88 in Object.wait() [6c48a000..6c48a854] at java.lang.Object.wait(Native Method) - waiting on <0x471904e0> (a com.silver.base.scaudit.audit.SCAuditQueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:86) - locked <0x471904e0> (a com.silver.base.scaudit.audit.SCAuditQueryQueue) at com.silver.base.scaudit.audit.AuditQueryThread.run(AuditQueryThread.java:119) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "AuditQueryThread4" prio=1 tid=0x69fa0d30 nid=0x7b88 in Object.wait() [6c409000..6c409854] at java.lang.Object.wait(Native Method) - waiting on <0x471904e0> (a com.silver.base.scaudit.audit.SCAuditQueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:86) - locked <0x471904e0> (a com.silver.base.scaudit.audit.SCAuditQueryQueue) at com.silver.base.scaudit.audit.AuditQueryThread.run(AuditQueryThread.java:119) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "AuditQueryThread5" prio=1 tid=0x69fa0268 nid=0x7b88 in Object.wait() [6c388000..6c388854] at java.lang.Object.wait(Native Method) - waiting on <0x471904e0> (a com.silver.base.scaudit.audit.SCAuditQueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:86) - locked <0x471904e0> (a com.silver.base.scaudit.audit.SCAuditQueryQueue) at com.silver.base.scaudit.audit.AuditQueryThread.run(AuditQueryThread.java:119) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "AuditQueryThread6" prio=1 tid=0x69fa2e58 nid=0x7b88 in Object.wait() [6c307000..6c307854] at java.lang.Object.wait(Native Method) - waiting on <0x471904e0> (a com.silver.base.scaudit.audit.SCAuditQueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:86) - locked <0x471904e0> (a com.silver.base.scaudit.audit.SCAuditQueryQueue) at com.silver.base.scaudit.audit.AuditQueryThread.run(AuditQueryThread.java:119) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "AuditQueryThread7" prio=1 tid=0x69faa7d8 nid=0x7b88 in Object.wait() [6c286000..6c286854] at java.lang.Object.wait(Native Method) - waiting on <0x471904e0> (a com.silver.base.scaudit.audit.SCAuditQueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:86) - locked <0x471904e0> (a com.silver.base.scaudit.audit.SCAuditQueryQueue) at com.silver.base.scaudit.audit.AuditQueryThread.run(AuditQueryThread.java:119) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "AuditQueryThread8" prio=1 tid=0x69faa5b0 nid=0x7b88 in Object.wait() [6c205000..6c205854] at java.lang.Object.wait(Native Method) - waiting on <0x471904e0> (a com.silver.base.scaudit.audit.SCAuditQueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:86) - locked <0x471904e0> (a com.silver.base.scaudit.audit.SCAuditQueryQueue) at com.silver.base.scaudit.audit.AuditQueryThread.run(AuditQueryThread.java:119) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "IndexQueryThread_19" prio=1 tid=0x69f9ec98 nid=0x7b88 in Object.wait() [6c184000..6c184854] at java.lang.Object.wait(Native Method) - waiting on <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:107) - locked <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.IndexQueryThread.run(IndexQueryThread.java:115) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "IndexQueryThread_0" prio=1 tid=0x69f9e148 nid=0x7b88 in Object.wait() [6c103000..6c103854] at java.lang.Object.wait(Native Method) - waiting on <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:86) - locked <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.IndexQueryThread.run(IndexQueryThread.java:115) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "IndexQueryThread_1" prio=1 tid=0x69f9d6f0 nid=0x7b88 in Object.wait() [6c082000..6c082854] at java.lang.Object.wait(Native Method) - waiting on <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:107) - locked <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.IndexQueryThread.run(IndexQueryThread.java:115) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "IndexQueryThread_2" prio=1 tid=0x69f9c898 nid=0x7b88 in Object.wait() [6c001000..6c001854] at java.lang.Object.wait(Native Method) - waiting on <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:107) - locked <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.IndexQueryThread.run(IndexQueryThread.java:115) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "IndexQueryThread_3" prio=1 tid=0x69f9bd48 nid=0x7b88 in Object.wait() [6bf80000..6bf80854] at java.lang.Object.wait(Native Method) - waiting on <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:107) - locked <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.IndexQueryThread.run(IndexQueryThread.java:115) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "IndexQueryThread_4" prio=1 tid=0x69f9b1f8 nid=0x7b88 in Object.wait() [6beff000..6beff854] at java.lang.Object.wait(Native Method) - waiting on <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:107) - locked <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.IndexQueryThread.run(IndexQueryThread.java:115) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "IndexQueryThread_5" prio=1 tid=0x69f9a6a8 nid=0x7b88 in Object.wait() [6be7e000..6be7e854] at java.lang.Object.wait(Native Method) - waiting on <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:107) - locked <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.IndexQueryThread.run(IndexQueryThread.java:115) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "IndexQueryThread_6" prio=1 tid=0x69f99b58 nid=0x7b88 in Object.wait() [6bdfd000..6bdfd854] at java.lang.Object.wait(Native Method) - waiting on <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:107) - locked <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.IndexQueryThread.run(IndexQueryThread.java:115) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "IndexQueryThread_7" prio=1 tid=0x69f99008 nid=0x7b88 in Object.wait() [6bd7c000..6bd7c854] at java.lang.Object.wait(Native Method) - waiting on <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:107) - locked <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.IndexQueryThread.run(IndexQueryThread.java:115) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "IndexQueryThread_8" prio=1 tid=0x69f984b8 nid=0x7b88 in Object.wait() [6bcfb000..6bcfb854] at java.lang.Object.wait(Native Method) - waiting on <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:107) - locked <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.IndexQueryThread.run(IndexQueryThread.java:115) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "IndexQueryThread_9" prio=1 tid=0x69f97968 nid=0x7b88 in Object.wait() [6bc7a000..6bc7a854] at java.lang.Object.wait(Native Method) - waiting on <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:107) - locked <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.IndexQueryThread.run(IndexQueryThread.java:115) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "IndexQueryThread_10" prio=1 tid=0x69f96e18 nid=0x7b88 in Object.wait() [6bbf9000..6bbf9854] at java.lang.Object.wait(Native Method) - waiting on <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:107) - locked <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.IndexQueryThread.run(IndexQueryThread.java:115) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "IndexQueryThread_11" prio=1 tid=0x69f962c8 nid=0x7b88 in Object.wait() [6bb78000..6bb78854] at java.lang.Object.wait(Native Method) - waiting on <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:107) - locked <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.IndexQueryThread.run(IndexQueryThread.java:115) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "IndexQueryThread_12" prio=1 tid=0x69f95778 nid=0x7b88 in Object.wait() [6baf7000..6baf7854] at java.lang.Object.wait(Native Method) - waiting on <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:107) - locked <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.IndexQueryThread.run(IndexQueryThread.java:115) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "IndexQueryThread_13" prio=1 tid=0x69f94c28 nid=0x7b88 in Object.wait() [6ba76000..6ba76854] at java.lang.Object.wait(Native Method) - waiting on <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:107) - locked <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.IndexQueryThread.run(IndexQueryThread.java:115) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "IndexQueryThread_14" prio=1 tid=0x69f940f0 nid=0x7b88 in Object.wait() [6b9f5000..6b9f5854] at java.lang.Object.wait(Native Method) - waiting on <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:107) - locked <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.IndexQueryThread.run(IndexQueryThread.java:115) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "IndexQueryThread_15" prio=1 tid=0x69f8cf48 nid=0x7b88 in Object.wait() [6b974000..6b974854] at java.lang.Object.wait(Native Method) - waiting on <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:107) - locked <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.IndexQueryThread.run(IndexQueryThread.java:115) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "IndexQueryThread_16" prio=1 tid=0x69f8d940 nid=0x7b88 in Object.wait() [6b8f3000..6b8f3854] at java.lang.Object.wait(Native Method) - waiting on <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:107) - locked <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.IndexQueryThread.run(IndexQueryThread.java:115) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "IndexQueryThread_17" prio=1 tid=0x69f8d7d8 nid=0x7b88 in Object.wait() [6b872000..6b872854] at java.lang.Object.wait(Native Method) - waiting on <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:86) - locked <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.IndexQueryThread.run(IndexQueryThread.java:115) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "IndexQueryThread_18" prio=1 tid=0x69f8d270 nid=0x7b88 in Object.wait() [6b7f1000..6b7f1854] at java.lang.Object.wait(Native Method) - waiting on <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:107) - locked <0x47189ae8> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.IndexQueryThread.run(IndexQueryThread.java:115) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "QueryThread_2" prio=1 tid=0x69f8b1e8 nid=0x7b88 in Object.wait() [6b770000..6b770854] at java.lang.Object.wait(Native Method) - waiting on <0x47189428> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:107) - locked <0x47189428> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.QueryThread.run(QueryThread.java:141) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "QueryThread_1" prio=1 tid=0x69f8a698 nid=0x7b88 in Object.wait() [6b6ef000..6b6ef854] at java.lang.Object.wait(Native Method) - waiting on <0x47189428> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:86) - locked <0x47189428> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.QueryThread.run(QueryThread.java:141) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "QueryThread_0" prio=1 tid=0x69f89b48 nid=0x7b88 in Object.wait() [6b66e000..6b66e854] at java.lang.Object.wait(Native Method) - waiting on <0x47189428> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:107) - locked <0x47189428> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.QueryThread.run(QueryThread.java:141) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "MCListener_0" prio=1 tid=0x69f89010 nid=0x7b88 runnable [6b5ed000..6b5ed854] at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.net.SocketInputStream.read(SocketInputStream.java:182) at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2133) at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2423) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2433) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1245) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:324) at com.silver.base.scsearch.scsearchserver.MCQueryListener.handleQuery(MCQueryListener.java:268) - locked <0x471897c0> (a java.lang.Object) at com.silver.base.scsearch.scsearchserver.MCQueryListener.run(MCQueryListener.java:407) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "QueryThread_9" prio=1 tid=0x69f885e8 nid=0x7b88 in Object.wait() [6b56c000..6b56c854] at java.lang.Object.wait(Native Method) - waiting on <0x47189428> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:107) - locked <0x47189428> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.QueryThread.run(QueryThread.java:141) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "QueryThread_8" prio=1 tid=0x69f8fc28 nid=0x7b88 in Object.wait() [6b4eb000..6b4eb854] at java.lang.Object.wait(Native Method) - waiting on <0x47189428> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:107) - locked <0x47189428> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.QueryThread.run(QueryThread.java:141) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "QueryThread_7" prio=1 tid=0x69f8f0d8 nid=0x7b88 in Object.wait() [6b46a000..6b46a854] at java.lang.Object.wait(Native Method) - waiting on <0x47189428> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:107) - locked <0x47189428> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.QueryThread.run(QueryThread.java:141) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "QueryThread_6" prio=1 tid=0x69f85468 nid=0x7b88 in Object.wait() [6b3e9000..6b3e9854] at java.lang.Object.wait(Native Method) - waiting on <0x47189428> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:86) - locked <0x47189428> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.QueryThread.run(QueryThread.java:141) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "QueryThread_5" prio=1 tid=0x69f84a70 nid=0x7b88 in Object.wait() [6b368000..6b368854] at java.lang.Object.wait(Native Method) - waiting on <0x47189428> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:107) - locked <0x47189428> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.QueryThread.run(QueryThread.java:141) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "QueryThread_4" prio=1 tid=0x69f90b80 nid=0x7b88 in Object.wait() [6b2e7000..6b2e7854] at java.lang.Object.wait(Native Method) - waiting on <0x47189428> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:107) - locked <0x47189428> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.QueryThread.run(QueryThread.java:141) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "QueryThread_3" prio=1 tid=0x69f90928 nid=0x7b88 in Object.wait() [6b266000..6b266854] at java.lang.Object.wait(Native Method) - waiting on <0x47189428> (a com.silver.base.searchcommon.QueryQueue) at java.lang.Object.wait(Object.java:429) at com.silver.base.searchcommon.QueryQueue.removeFirst(QueryQueue.java:86) - locked <0x47189428> (a com.silver.base.searchcommon.QueryQueue) at com.silver.base.scsearch.scsearchserver.QueryThread.run(QueryThread.java:141) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "Indexer_1" prio=1 tid=0x69f92d30 nid=0x7b88 in Object.wait() [6b1e5000..6b1e5854] at java.lang.Object.wait(Native Method) - waiting on <0x47114058> (a java.lang.Object) at com.silver.base.indexer.batch.QueueMgr.getFilesToIndex(QueueMgr.java:257) - locked <0x47114058> (a java.lang.Object) at com.silver.base.indexer.batch.IndexerThread.run(IndexerThread.java:167) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "Indexer_0" prio=1 tid=0x69f80a38 nid=0x7b88 waiting on condition [6b164000..6b164854] at java.io.Reader.<init>(Reader.java:59) at java.io.BufferedReader.<init>(BufferedReader.java:76) at java.io.BufferedReader.<init>(BufferedReader.java:91) at com.silver.base.metadataretriever.BitfileIndexableDataRetriever.getTypedReaders(BitfileIndexableDataRetriever.java:857) at com.silver.base.metadataretriever.BitfileIndexableDataRetriever.getTypedReaders(BitfileIndexableDataRetriever.java:567) at com.silver.base.metadataretriever.BitfileIndexableDataRetriever.getTypedReaders(BitfileIndexableDataRetriever.java:494) at com.silver.base.indexinterface.indexmgr.IndexerDocument.initialize(IndexerDocument.java:232) at com.silver.base.indexer.batch.IndexerThread.processDocument(IndexerThread.java:495) at com.silver.base.indexer.batch.IndexerThread.indexFiles(IndexerThread.java:434) at com.silver.base.indexer.batch.IndexerThread.run(IndexerThread.java:206) at com.silver.base.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:193) "Thread-17" prio=1 tid=0x69f811d8 nid=0x7b88 in Object.wait() [6b0e3000..6b0e3854] at java.lang.Object.wait(Native Method) - waiting on <0x4717a000> (a java.util.TaskQueue) at java.util.TimerThread.mainLoop(Timer.java:429) - locked <0x4717a000> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:382) "ReadersReaperTimer" prio=1 tid=0x69f81510 nid=0x7b88 waiting for monitor entry [6b062000..6b062854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2410) at com.silver.base.indexinterface.indexmgr.Index.flushSearchInterface(Index.java:850) - locked <0x4713c538> (a java.lang.Object) at com.silver.base.indexinterface.indexmgr.Index.reapSearcher(Index.java:867) at com.silver.base.indexinterface.indexmgr.IndexMgr.reapSearchers(IndexMgr.java:809) at com.silver.base.indexinterface.indexmgr.IndexMgr.access$000(IndexMgr.java:49) at com.silver.base.indexinterface.indexmgr.IndexMgr$ReadersReaperTask.run(IndexMgr.java:885) at java.util.TimerThread.mainLoop(Timer.java:432) at java.util.TimerThread.run(Timer.java:382) "TimeoutManager-Thread" daemon prio=1 tid=0x69f92b68 nid=0x7b88 waiting on condition [6afc8000..6afc8854] at java.lang.Object.wait(Native Method) - waiting on <0x47113d30> (a com.silver.base.utils.threads.TimeoutManager) at com.silver.base.utils.threads.TimeoutManager.run(TimeoutManager.java:173) - locked <0x47113d30> (a com.silver.base.utils.threads.TimeoutManager) "Thread-15" prio=1 tid=0x69f88090 nid=0x7b88 waiting for monitor entry [6af47000..6af47854] at org.apache.log4j.Category.callAppenders(Category.java:376) - waiting to lock <0x46eb4598> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.debug(Category.java:451) at org.apache.log4j.Category.debugParamArray(Category.java:1644) at org.apache.log4j.Category.debug(Category.java:2439) at com.silver.base.archiver.FileProxy.isLeaseRenewed(FileProxy.java:367) at com.silver.base.archiver.FileProxy.evict(FileProxy.java:303) at com.silver.base.utils.LeaseEvictor.run(LeaseEvictor.java:45) at java.util.TimerThread.mainLoop(Timer.java:432) at java.util.TimerThread.run(Timer.java:382) "/boston/DBLookup_Services/MSDBLookupHolder" daemon prio=1 tid=0x69f82190 nid=0x7b88 waiting on condition [6aec6000..6aec6854] at java.security.AccessControlContext.hashCode(AccessControlContext.java:477) at java.util.HashMap.hash(HashMap.java:261) at java.util.WeakHashMap.put(WeakHashMap.java:392) at sun.rmi.transport.tcp.TCPChannel.checkConnectPermission(TCPChannel.java:120) at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:148) - locked <0x46ff86e0> (a java.util.ArrayList) at sun.rmi.transport.ConnectionInputStream.done(ConnectionInputStream.java:112) at sun.rmi.transport.StreamRemoteCall.releaseInputStream(StreamRemoteCall.java:139) at sun.rmi.transport.StreamRemoteCall.done(StreamRemoteCall.java:284) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:172) at org.jnp.server.NamingServer_Stub.lookup(Unknown Source) at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:349) at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:333) at com.silver.base.servicebroker.ServiceBroker.getItemsFromJNDI(ServiceBroker.java:151) - locked <0x470b8e10> (a com.silver.base.servicebroker.ServiceBroker) at com.silver.base.servicebroker.ServiceBroker.run(ServiceBroker.java:349) - locked <0x470b8e10> (a com.silver.base.servicebroker.ServiceBroker) at java.lang.Thread.run(Thread.java:534) "SESSION-Thread-14" prio=1 tid=0x0863ce18 nid=0x7b88 runnable [6a8ba000..6a8ba854] at java.net.PlainDatagramSocketImpl.peekData(Native Method) at java.net.DatagramSocket.receive(DatagramSocket.java:661) - locked <0x46ffc220> (a java.net.DatagramPacket) - locked <0x46ffc240> (a java.net.DatagramSocket) at com.adventnet.snmp.snmp2.IpTSocket.receive(IpTSocket.java:30) - locked <0x46ffc260> (a com.adventnet.snmp.snmp2.IpTSocket) at com.adventnet.snmp.snmp2.SnmpSession.receiveFromIP(SnmpSession.java:2277) at com.adventnet.snmp.snmp2.SnmpSession.receiver(SnmpSession.java:1956) at com.adventnet.snmp.snmp2.SnmpSession.run(SnmpSession.java:719) "ZGarbageCollector" prio=1 tid=0x69f757e8 nid=0x7b88 waiting on condition [6adc4000..6adc4854] at java.lang.Thread.sleep(Native Method) at com.silver.base.resourcemonitor.ZGarbageCollector.run(ZGarbageCollector.java:30) "ResourceMonitorHistoryThread" prio=1 tid=0x69f75ac0 nid=0x7b88 waiting on condition [6ad43000..6ad43854] at sun.util.calendar.ZoneInfo.getLastRawOffset(ZoneInfo.java:384) at sun.util.calendar.ZoneInfo.getOffsets(ZoneInfo.java:211) at sun.util.calendar.ZoneInfo.getOffsets(ZoneInfo.java:201) at java.util.GregorianCalendar.computeFieldsImpl(GregorianCalendar.java:1319) at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:1298) at java.util.Calendar.setTimeInMillis(Calendar.java:927) at java.util.Calendar.setTime(Calendar.java:902) at org.apache.log4j.helpers.ISO8601DateFormat.format(ISO8601DateFormat.java:55) at java.text.DateFormat.format(DateFormat.java:314) at org.apache.log4j.helpers.PatternParser$DatePatternConverter.convert(PatternParser.java:434) at org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:56) at org.apache.log4j.PatternLayout.format(PatternLayout.java:547) at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:295) at org.apache.log4j.RollingFileAppender.subAppend(RollingFileAppender.java:566) at org.apache.log4j.WriterAppender.append(WriterAppender.java:157) at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251) - locked <0x471bfb48> (a org.apache.log4j.RollingFileAppender) at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:57) at org.apache.log4j.Category.callAppenders(Category.java:380) - locked <0x46eb6a40> (a org.apache.log4j.Category) at org.apache.log4j.Category.forcedLog(Category.java:609) at org.apache.log4j.Category.info(Category.java:886) at com.silver.base.resourcemonitor.ResourceHistoryThread.run(ResourceHistoryThread.java:88) "[SYSREPORTER]" prio=1 tid=0x69f76338 nid=0x7b88 waiting on condition [6acc2000..6acc2854] at java.lang.Thread.sleep(Native Method) at com.silver.base.sysreporter.SysReporter.run(SysReporter.java:247) at java.lang.Thread.run(Thread.java:534) "VMStatHistoryThread" prio=1 tid=0x69f76138 nid=0x7b88 waiting on condition [6ac41000..6ac41854] at org.apache.log4j.Priority.isGreaterOrEqual(Priority.java:115) at org.apache.log4j.Category.info(Category.java:885) at org.apache.log4j.Category.infoParamArray(Category.java:1577) at org.apache.log4j.Category.info(Category.java:2328) at com.silver.base.systemmonitor.VMStatHistoryThread.run(VMStatHistoryThread.java:62) "SERVICEMONITOR:HEALTH" prio=1 tid=0x08640e00 nid=0x7b88 waiting on condition [6abc0000..6abc0854] at java.util.Collections.reverse(Collections.java:355) at java.io.ObjectStreamClass.getClassDataLayout0(ObjectStreamClass.java:1020) at java.io.ObjectStreamClass.getClassDataLayout(ObjectStreamClass.java:972) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1284) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1247) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1052) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:278) at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:265) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:124) at org.jnp.server.NamingServer_Stub.rebind(Unknown Source) at org.jnp.interfaces.NamingContext.rebind(NamingContext.java:267) at org.jnp.interfaces.NamingContext.rebind(NamingContext.java:240) at javax.naming.InitialContext.rebind(InitialContext.java:363) at com.silver.base.servicemonitor.ServiceMonitor.serverHealth(ServiceMonitor.java:455) - locked <0x46eeef98> (a com.silver.base.servicemonitor.ServiceMonitor) at com.silver.base.servicemonitor.ServerHealthCheckup.run(ServiceMonitor.java:1232) "ServiceMonitor" daemon prio=1 tid=0x08639078 nid=0x7b88 in Object.wait() [6ab3f000..6ab3f854] at java.lang.Object.wait(Native Method) - waiting on <0x46eef808> (a java.util.Vector) at com.silver.base.servicemonitor.ServiceMonitor.run(ServiceMonitor.java:1117) - locked <0x46eef808> (a java.util.Vector) at java.lang.Thread.run(Thread.java:534) "RMI RenewClean-[10.0.97.1:34079]" daemon prio=1 tid=0x08641cc0 nid=0x7b88 waiting for monitor entry [6aa3d000..6aa3d854] at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:142) - waiting to lock <0x46ff86e0> (a java.util.ArrayList) at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:313) at sun.rmi.transport.DGCImpl_Stub.dirty(Unknown Source) at sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:328) at sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:144) at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:539) at java.lang.Thread.run(Thread.java:534) "Thread-8" prio=1 tid=0x0863a790 nid=0x7b88 runnable [6a93b000..6a93b854] at java.net.PlainDatagramSocketImpl.peekData(Native Method) at java.net.DatagramSocket.receive(DatagramSocket.java:661) - locked <0x46fb4c60> (a java.net.DatagramPacket) - locked <0x46fb4c80> (a java.net.DatagramSocket) at com.adventnet.snmp.snmp2.IpTSocket.receive(IpTSocket.java:30) - locked <0x46fb4ca0> (a com.adventnet.snmp.snmp2.IpTSocket) at com.adventnet.snmp.snmp2.SnmpSession.receiveFromIP(SnmpSession.java:2277) at com.adventnet.snmp.snmp2.SnmpSession.receiver(SnmpSession.java:1956) at com.adventnet.snmp.snmp2.SnmpSession.run(SnmpSession.java:719) "API-Thread-6" prio=1 tid=0x08645e20 nid=0x7b88 waiting on condition [6a67a000..6a67a854] at java.lang.Thread.sleep(Native Method) at com.adventnet.snmp.snmp2.SnmpAPI.run(SnmpAPI.java:141) "Adaptor:name=html" prio=1 tid=0x084333c0 nid=0x7b88 runnable [6a5f9000..6a5f9854] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353) - locked <0x46ecac50> (a java.net.PlainSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:448) at java.net.ServerSocket.accept(ServerSocket.java:419) at com.sun.jdmk.comm.HtmlAdaptorServer.doReceive(HtmlAdaptorServer.java:935) at com.sun.jdmk.comm.CommunicatorServer.run(CommunicatorServer.java:512) at java.lang.Thread.run(Thread.java:534) "RMI LeaseChecker" daemon prio=1 tid=0x0842f220 nid=0x7b88 waiting on condition [6a4f7000..6a4f7854] at java.lang.Thread.sleep(Native Method) at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:307) at java.lang.Thread.run(Thread.java:534) "Thread-5" prio=1 tid=0x69f77348 nid=0x7b88 runnable [6a3f3000..6a3f3854] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353) - locked <0x46ecad68> (a java.net.PlainSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:448) at java.net.ServerSocket.accept(ServerSocket.java:419) at org.jnp.server.Main.run(Main.java:145) at java.lang.Thread.run(Thread.java:534) "GC Daemon" daemon prio=1 tid=0x69f72808 nid=0x7b88 in Object.wait() [6a2f1000..6a2f1854] at java.lang.Object.wait(Native Method) - waiting on <0x46ec20b0> (a sun.misc.GC$LatencyLock) at sun.misc.GC$Daemon.run(GC.java:100) - locked <0x46ec20b0> (a sun.misc.GC$LatencyLock) "RMI Reaper" prio=1 tid=0x69f64c20 nid=0x7b88 in Object.wait() [6a270000..6a270854] at java.lang.Object.wait(Native Method) - waiting on <0x46ec14c0> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111) - locked <0x46ec14c0> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127) at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:330) at java.lang.Thread.run(Thread.java:534) "Thread-3" daemon prio=1 tid=0x69f64900 nid=0x7b88 in Object.wait() [6a1ef000..6a1ef854] at java.lang.Object.wait(Native Method) - waiting on <0x46ec1868> (a java.util.TaskQueue) at java.lang.Object.wait(Object.java:429) at java.util.TimerThread.mainLoop(Timer.java:403) - locked <0x46ec1868> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:382) "RMI TCP Accept-0" daemon prio=1 tid=0x69f71418 nid=0x7b88 runnable [6a16e000..6a16e854] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353) - locked <0x46ec1518> (a java.net.PlainSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:448) at java.net.ServerSocket.accept(ServerSocket.java:419) at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334) at java.lang.Thread.run(Thread.java:534) "Thread-0" daemon prio=1 tid=0x69f65498 nid=0x7b88 waiting on condition [6a0cc000..6a0cc854] at java.lang.Thread.sleep(Native Method) at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:95) "Log time updater" daemon prio=1 tid=0x081a4358 nid=0x7b88 waiting on condition [6936c000..6936c854] at java.lang.Thread.sleep(Native Method) at org.jboss.logging.Logger.run(Logger.java:139) at java.lang.Thread.run(Thread.java:534) "Signal Dispatcher" daemon prio=1 tid=0x0808c920 nid=0x7b88 waiting on condition [0..0] "Finalizer" daemon prio=1 tid=0x08088040 nid=0x7b88 in Object.wait() [68ac1000..68ac1854] at java.lang.Object.wait(Native Method) - waiting on <0x46e7eb08> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111) - locked <0x46e7eb08> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) "Reference Handler" daemon prio=1 tid=0x08087460 nid=0x7b88 in Object.wait() [68a40000..68a40854] at java.lang.Object.wait(Native Method) - waiting on <0x46e7eb70> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:429) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:115) - locked <0x46e7eb70> (a java.lang.ref.Reference$Lock) "VM Thread" prio=1 tid=0x08086198 nid=0x7b88 runnable "VM Periodic Task Thread" prio=1 tid=0x0808f130 nid=0x7b88 waiting on condition "Suspend Checker Thread" prio=1 tid=0x0808bf20 nid=0x7b88 runnable Shutting down [Service Control] Stopping 39 MBeans