Re: Something is killing my CPU

Thanks for the detailed advice David. I'll certainly apply Some of it as best practice, however, I don't think it's Disk IO that's the issue.

Some background:

1. I'm running in Azure with SSD disks attached for the /rrd/snmp and /rrd/response data so lot of the hardware config is moot.

2. I see a sudden spike in CPU utilisation at a point in time and it never comes back down. I'd expect to see sustained high load averages if it were a disk bottleneck. Add to that I only have 800- nodes, 700 of which are 5-min StrafePing services only, and I don't think I'm really taxing a 4-core 32Gb RAM machine.

3. I had 4Gb allocated to the HEAP, I've upped that to 16Gb.

4. Is there a way to automate a "thread dump" at regular intervals as you suggest?

5. My install is only 1 year old

6. I am running RRD and not Jrobin

7. PostgreSQL spikes to about 2000 transactions a second every poll interval, other than that it's idling at less than 50.

It's obvious there is one thread that is the major culprit, I rand a thread dump and here is the extract of the thread that is consuming the majority of CPU - my untrained eye suggests it's something to do with the Jetty server and thus the GUI or REST interface:

"qtp1403511311-2390" - Thread t@2390
java.lang.Thread.State: RUNNABLE
at java.util.zip.Deflater.deflateBytes(Native Method)
at java.util.zip.Deflater.deflate(Deflater.java:444)
- locked <21d266e8> (a java.util.zip.ZStreamRef)
at java.util.zip.Deflater.deflate(Deflater.java:366)
at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:251)
at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:211)
at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:145)
- locked <4e3f2d41> (a java.util.zip.GZIPOutputStream)
at org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:51)
at org.apache.cxf.io.AbstractThresholdOutputStream.write(AbstractThresholdOutputStream.java:69)
at org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:60)
at org.apache.cxf.jaxrs.provider.StringTextProvider.writeTo(StringTextProvider.java:73)
at org.apache.cxf.jaxrs.provider.StringTextProvider.writeTo(StringTextProvider.java:36)
at org.apache.cxf.jaxrs.utils.JAXRSUtils.writeMessageBody(JAXRSUtils.java:1386)
at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.serializeMessage(JAXRSOutInterceptor.java:244)
at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.handleWriteException(JAXRSOutInterceptor.java:397)
at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.serializeMessage(JAXRSOutInterceptor.java:270)
at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.processResponse(JAXRSOutInterceptor.java:120)
at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.handleMessage(JAXRSOutInterceptor.java:83)
at org.apache.cxf.jaxrs.interceptor.JAXRSDefaultFaultOutInterceptor.handleMessage(JAXRSDefaultFaultOutInterceptor.java:68)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
- locked <7d5a3131> (a org.apache.cxf.phase.PhaseInterceptorChain)
at org.apache.cxf.interceptor.AbstractFaultChainInitiatorObserver.onMessage(AbstractFaultChainInitiatorObserver.java:112)
at org.apache.cxf.phase.PhaseInterceptorChain.wrapExceptionAsFault(PhaseInterceptorChain.java:366)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:324)
- locked <7ba34bfd> (a org.apache.cxf.phase.PhaseInterceptorChain)
at org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(OutgoingChainInterceptor.java:83)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
- locked <247c7bcf> (a org.apache.cxf.phase.PhaseInterceptorChain)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:262)
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:180)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:299)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:218)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:274)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
at org.opennms.container.web.felix.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:53)
at org.opennms.container.web.felix.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:35)
at org.opennms.container.web.felix.base.internal.dispatch.FilterPipeline.dispatch(FilterPipeline.java:50)
at org.opennms.container.web.felix.base.internal.dispatch.Dispatcher.dispatch(Dispatcher.java:42)
at org.opennms.container.web.bridge.internal.DispatcherFilter.doFilter(DispatcherFilter.java:72)
at org.opennms.container.web.ProxyFilter.doFilter(ProxyFilter.java:56)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621)
at org.extremecomponents.table.filter.AbstractExportFilter.doFilter(AbstractExportFilter.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621)
at org.opennms.web.servlet.StoreRequestPropertiesFilter.doFilter(StoreRequestPropertiesFilter.java:79)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.opennms.web.springframework.security.AuthFilterEnabler.doFilter(AuthFilterEnabler.java:115)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication. www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621)
at org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:231)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:121)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1613)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:541)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1592)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1239)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:481)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1561)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1141)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:118)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.opennms.netmgt.jetty.MDCHandler.handle(MDCHandler.java:46)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:564)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Thread.java:748)

Locked ownable synchronizers:
- None

--

Gavin Hill | Director

m: +27 83 601 8181

t: +27 10 020 5959

mailto:%28%21Email%21%29 https://www.linkedin.com/company/highpeak-pty-ltd/ skype:%28%21Email%21%29 https://www.highpeak.co.za/

-----Original Message-----
From: David Hustace <[email protected]>
Sent: Tuesday, 09 October 2018 15:48
To: DiscussionList List <[email protected]>
Subject: Re: [opennms-discuss] Something is killing my CPU

> On Oct 9, 2018, at 08:50, Gavin Hill <[email protected]> wrote:
>
> If I expand the thread all the way down (and boy does it go a long way) I finally get here:

Really need to see the entire stack trace. It would help if you posted a thread dump somewhere at the time of increased CPU would help. Probably a few of them about 5 minutes apart.

Also, the thing I've noticed about high CPU vs. high load averages in my time with OpenNMS:

1) Correctly reading load average when you have multiple cores is important.
2) High load average usually is indicative of poor performance writing performance data to the RRDs and/or PostgreSQL performance if PostgreSQL is on the same system.
3) High CPU is usually due to either the HEAP being exhausted and the JVM trying to find memory (usually indicative of 100% CPU) or due to there being a lot of serialization from running 100s - 1000s of Remote Pollers (yes I've seen this) and, now I can assume Minions, too (but I've not personally witnessed this).

Making sure that your storage configuration is optimized to not be using the same disk channels for high volume writes from where contention for IOps will run up your load averages. We've spent entire careers care and feeding systems for this and it's just something that you have to do when you have applications with very high volume of writes as we do with OpenNMS. There are dozens of performance related tunings built into OpenNMS to help you with this problem along with a lot of optimizations you can do for PostgreSQL as well as the underlying System kernel and storage settings. Dozens of not hundreds.

Quick wins:

1) Move PostgreSQL to its own server.
2) Use a RAID configuration optimized for high volume writes
3) Insure the RAID controller is properly configured (w/battery)
4) Make sure that the file systems are optimized and mounted with recommend mount options
5) Put the /var/opennms/rrd/snmp and /var/lib/opennms/rrd/response are using different disk channels (partition/file systems at a minimum)
6) PostgreSQL:
a) put the write ahead logs on a separate disk channel/partition/file system from the data files
b) put the indexes on a separate (requires creating separate table spaces moving the indexes to those table spaces)
7) Logging:
a) Make sure that OpenNMS logging is not in DEBUG mode
b) Make sure that logging is not on the same file system as RRDs or PostgreSQL
8) OpenNMS Tuning:
a) Don't run services that you don't need.
b) Use RRD vs. JRobin
c) if you've been upgrading from installs originating more than 3 years ago, I'd make sure and compare opennms.properties and rod-configuration.properties with etc-pristine to make sure you've not lost optimization such as "#org.opennms.rrd.strategyClass=org.opennms.netmgt.rrd.rrdtool.MultithreadedJniRrdStrategy"

etc.

David

_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss