Hung Threads Issue in DTMDefaultBase.findNamespaceContext
"Ilyas, Numan" <[email protected]> Thu, 22 Aug 2013 20:53:17 +0000
| Newsgroups | gmane.text.xml.xalan.java.user |
|---|---|
| Message-ID | <00B51801CFD8264DB508EFBF348798FF0570068C@smtp_mail.bankofamerica.com> |
--Boundary_(ID_HXL7Hwueg8/ABSS1dXhBuA)
Content-type: text/plain; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT
Hi,
We are having hung threads in our JVM logs and these threads are consuming all the CPU resources. We've to restart the JVMs everytime this issue occur. Unfortunately we aren't able to reproduce the issue and hence we can't provide the test case to reproduce. Please see following stack trace copied from JVM logs:
[8/20/13 10:29:57:982 EDT] 0000001a ThreadMonitor W WSVR0605W: Thread "Migrated.Servlet.Engine.Transports : 1605" (00000670) has been active for 646127 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung.
at org.apache.xml.dtm.ref.DTMDefaultBase.findNamespaceContext(Unknown Source)
at org.apache.xml.dtm.ref.DTMDefaultBase.declareNamespaceInContext(Unknown Source)
at org.apache.xml.dtm.ref.dom2dtm.DOM2DTM.addNode(Unknown Source)
at org.apache.xml.dtm.ref.dom2dtm.DOM2DTM.processNamespacesAndAttributes(Unknown Source)
at org.apache.xml.dtm.ref.dom2dtm.DOM2DTM.nextNode(Unknown Source)
at org.apache.xml.dtm.ref.DTMDefaultBase._nextsib(Unknown Source)
at org.apache.xml.dtm.ref.DTMDefaultBase.getNextSibling(Unknown Source)
at org.apache.xml.dtm.ref.DTMDefaultBaseTraversers$ChildTraverser.next(Unknown Source)
at org.apache.xpath.axes.AxesWalker.getNextNode(Unknown Source)
at org.apache.xpath.axes.AxesWalker.nextNode(Unknown Source)
at org.apache.xpath.axes.WalkingIterator.nextNode(Unknown Source)
at org.apache.xpath.axes.NodeSequence.nextNode(Unknown Source)
at org.apache.xpath.axes.NodeSequence.runTo(Unknown Source)
at org.apache.xpath.axes.NodeSequence.setRoot(Unknown Source)
at org.apache.xpath.axes.LocPathIterator.execute(Unknown Source)
at org.apache.xpath.XPath.execute(Unknown Source)
at org.apache.xpath.jaxp.XPathExpressionImpl.evaluate(Unknown Source)
at our code reference
at com.ibm._jsp._GetCustomerViewExclusionDetails._jspService(_GetCustomerViewExclusionDetails.java:152)
at com.ibm.ws.jsp.runtime.HttpJspBase.service(HttpJspBase.java:98)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1655)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1595)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:131)
at com.savvion.sbm.websecurity.xss.filter.XSSFilter.doFilter(Unknown Source)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
at com.savvion.sbm.bpmportal.util.GZIPFilter.doFilter(Unknown Source)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
at com.savvion.sbm.bpmportal.util.EncodingFilter.doFilter(Unknown Source)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:77)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:908)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:932)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:500)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.wsspi.webcontainer.servlet.GenericServletWrapper.handleRequest(GenericServletWrapper.java:121)
at com.ibm.ws.jsp.webcontainerext.AbstractJSPExtensionServletWrapper.handleRequest(AbstractJSPExtensionServletWrapper.java:239)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:864)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1583)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:455)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:384)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:83)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1550)
Regards,
Numan Ilyas
----------------------------------------------------------------------
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
--Boundary_(ID_HXL7Hwueg8/ABSS1dXhBuA)
Content-type: text/html; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT
<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:black">We are having hung threads in our JVM logs and these threads are consuming all the CPU resources. We've to restart the JVMs everytime this issue occur. Unfortunately
we aren't able to reproduce the issue and hence we can't provide the test case to reproduce. Please see following stack trace copied from JVM logs:
<br>
<br>
<br>
[8/20/13 10:29:57:982 EDT] 0000001a ThreadMonitor W WSVR0605W: Thread "Migrated.Servlet.Engine.Transports : 1605" (00000670) has been active for 646127 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung.
<br>
at org.apache.xml.dtm.ref.DTMDefaultBase.findNamespaceContext(Unknown Source) <br>
at org.apache.xml.dtm.ref.DTMDefaultBase.declareNamespaceInContext(Unknown Source)
<br>
at org.apache.xml.dtm.ref.dom2dtm.DOM2DTM.addNode(Unknown Source) <br>
at org.apache.xml.dtm.ref.dom2dtm.DOM2DTM.processNamespacesAndAttributes(Unknown Source)
<br>
at org.apache.xml.dtm.ref.dom2dtm.DOM2DTM.nextNode(Unknown Source) <br>
at org.apache.xml.dtm.ref.DTMDefaultBase._nextsib(Unknown Source) <br>
at org.apache.xml.dtm.ref.DTMDefaultBase.getNextSibling(Unknown Source) <br>
at org.apache.xml.dtm.ref.DTMDefaultBaseTraversers$ChildTraverser.next(Unknown Source)
<br>
at org.apache.xpath.axes.AxesWalker.getNextNode(Unknown Source) <br>
at org.apache.xpath.axes.AxesWalker.nextNode(Unknown Source) <br>
at org.apache.xpath.axes.WalkingIterator.nextNode(Unknown Source) <br>
at org.apache.xpath.axes.NodeSequence.nextNode(Unknown Source) <br>
at org.apache.xpath.axes.NodeSequence.runTo(Unknown Source) <br>
at org.apache.xpath.axes.NodeSequence.setRoot(Unknown Source) <br>
at org.apache.xpath.axes.LocPathIterator.execute(Unknown Source) <br>
at org.apache.xpath.XPath.execute(Unknown Source) <br>
at org.apache.xpath.jaxp.XPathExpressionImpl.evaluate(Unknown Source) <br>
at our code reference <br>
at com.ibm._jsp._GetCustomerViewExclusionDetails._jspService(_GetCustomerViewExclusionDetails.java:152)
<br>
at com.ibm.ws.jsp.runtime.HttpJspBase.service(HttpJspBase.java:98) <br>
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831) <br>
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1655)
<br>
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1595)
<br>
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:131)
<br>
at com.savvion.sbm.websecurity.xss.filter.XSSFilter.doFilter(Unknown Source) <br>
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
<br>
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
<br>
at com.savvion.sbm.bpmportal.util.GZIPFilter.doFilter(Unknown Source) <br>
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
<br>
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
<br>
at com.savvion.sbm.bpmportal.util.EncodingFilter.doFilter(Unknown Source) <br>
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
<br>
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
<br>
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:77)
<br>
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:908)
<br>
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:932)
<br>
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:500)
<br>
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
<br>
at com.ibm.wsspi.webcontainer.servlet.GenericServletWrapper.handleRequest(GenericServletWrapper.java:121)
<br>
at com.ibm.ws.jsp.webcontainerext.AbstractJSPExtensionServletWrapper.handleRequest(AbstractJSPExtensionServletWrapper.java:239)
<br>
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
<br>
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:864) <br>
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1583)
<br>
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186) <br>
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:455)
<br>
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:384)
<br>
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:83)
<br>
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
<br>
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
<br>
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
<br>
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138) <br>
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204) <br>
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775) <br>
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905) <br>
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1550) <br>
<br>
<br>
Regards, <br>
Numan Ilyas</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<HR>This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.<BR>
</body>
</html>
--Boundary_(ID_HXL7Hwueg8/ABSS1dXhBuA)--