Re: Something is killing my CPU

David Hustace <[email protected]> Tue, 9 Oct 2018 09:48:03 -0400
Newsgroups	gmane.network.opennms.general
Message-ID	<[email protected]>

> On Oct 9, 2018, at 08:50, Gavin Hill <[email protected]> wrote:
> 
> If I expand the thread all the way down (and boy does it go a long way) I finally get here:

Really need to see the entire stack trace.  It would help if you posted a thread dump somewhere at the time of increased CPU would help.  Probably a few of them about 5 minutes apart.

Also, the thing I've noticed about high CPU vs. high load averages in my time with OpenNMS:

1) Correctly reading load average when you have multiple cores is important.
2) High load average usually is indicative of poor performance writing performance data to the RRDs and/or PostgreSQL performance if PostgreSQL is on the same system.
3) High CPU is usually due to either the HEAP being exhausted and the JVM trying to find memory (usually indicative of 100% CPU) or due to there being a lot of serialization from running 100s - 1000s of Remote Pollers (yes I've seen this) and, now I can assume Minions, too (but I've not personally witnessed this).

Making sure that your storage configuration is optimized to not be using the same disk channels for high volume writes from where contention for IOps will run up your load averages.  We've spent entire careers care and feeding systems for this and it's just something that you have to do when you have applications with very high volume of writes as we do with OpenNMS.  There are dozens of performance related tunings built into OpenNMS to help you with this problem along with a lot of optimizations you can do for PostgreSQL as well as the underlying System kernel and storage settings.  Dozens of not hundreds.

Quick wins:

1) Move PostgreSQL to its own server.
2) Use a RAID configuration optimized for high volume writes
3) Insure the RAID controller is properly configured (w/battery)
4) Make sure that the file systems are optimized and mounted with recommend mount options
5) Put the /var/opennms/rrd/snmp and /var/lib/opennms/rrd/response are using different disk channels (partition/file systems at a minimum)
6) PostgreSQL:
   a) put the write ahead logs on a separate disk channel/partition/file system from the data files
   b) put the indexes on a separate (requires creating separate table spaces moving the indexes to those table spaces)
7) Logging:
   a) Make sure that OpenNMS logging is not in DEBUG mode
   b) Make sure that logging is not on the same file system as RRDs or PostgreSQL
8) OpenNMS Tuning:
   a) Don't run services that you don't need.
   b) Use RRD vs. JRobin
   c) if you've been upgrading from installs originating more than 3 years ago, I'd make sure and compare opennms.properties and rod-configuration.properties with etc-pristine to make sure you've not lost optimization such as "#org.opennms.rrd.strategyClass=org.opennms.netmgt.rrd.rrdtool.MultithreadedJniRrdStrategy"

etc.



David

_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
signature.asc (application/pgp-signature, 873 B)