This Week in OpenNMS: Learning from Mistakes
Benjamin Reed <[email protected]> Mon, 28 Feb 2011 17:33:09 -0500
| Newsgroups | gmane.network.opennms.announce |
|---|---|
| Message-ID | <4D6C22A5.2080303__44750.7249810316$1298932701$gmane$org@opennms.org> |
It's time for This Week in OpenNMS <http://www.opennms.org/>. In the
last few weeks we worked on reports, IPv6, a ton of bugs, and JRobin.
Project Updates
* 1.8: Current Release is 1.8.10 (Loggerhead Shrike)
1.8.10
<http://www.opennms.org/documentation/ReleaseNotesStable.html#opennms-1.8.10> is
the current stable release, tagged 24 February, 2011. For a
complete list of changes and updates, see the "New and Noteworthy"
page on the OpenNMS wiki
<http://www.opennms.org/wiki/New_and_Noteworthy#New_in_OpenNMS_1.8.10>.
As always, it is recommended that you back up your database before
upgrading. A large number of bugs were fixed since 1.8.7, it is
highly recommended that you upgrade to this release.
* 1.9: Current release is 1.9.6 (Barbat)
1.9.3 is the current unstable release, tagged 24 February, 2010.
This release is not recommended for production use, but for
developers and users who want to try out the very latest features.
* A massive number of bugfixes since the last TWiO were fixed. See
below for details.
* A ton of work on optimizing JRobin has been going on over the last
week, more on that below as well.
* Donald, our resident reporting expert, did more work on reports,
including fixing PDF generation on statsd reports and support for
storeByGroup.
* Alejandro expanded the RRD support in Jasper reports to work with
RRDtool databases as well as JRobin.
* Seth did more IPv6 work.
Issues Fixed
* [IPL-17 <http://issues.opennms.org/browse/IPL-17>] -- Add IPv6
support to IPLIKE
* [JRB-3 <http://issues.opennms.org/browse/JRB-3>] -- jrobin
converter will not convert 64-bit rrdtool files
* [JRB-9 <http://issues.opennms.org/browse/JRB-9>] -- Jrobin
conversion errors
* [JRB-15 <http://issues.opennms.org/browse/JRB-15>] -- Change
license to LGPL (approved by original copyright holder)
* [NMS-113 <http://issues.opennms.org/browse/NMS-113>] -- NMS-112
Organize Scanning code into 'NodeScan' class and nested classes
* [NMS-114 <http://issues.opennms.org/browse/NMS-114>] -- NMS-112
Use callback or event mechanism to schedule 'subscans'
* [NMS-115 <http://issues.opennms.org/browse/NMS-115>] -- NMS-112
Refactor out DefaultProvisionSerivce into separate svc layer
* [NMS-116 <http://issues.opennms.org/browse/NMS-116>] -- NMS-112
Abstract 'Requistion' to cover 'suspect ip address'
* [NMS-117 <http://issues.opennms.org/browse/NMS-117>] -- NMS-112
Enhance scanning so discovered agents result in agentScan
* [NMS-118 <http://issues.opennms.org/browse/NMS-118>] -- NMS-112
handle 'deleteNode' and related events
* [NMS-119 <http://issues.opennms.org/browse/NMS-119>] -- NMS-112
handle 'newSuspect' and 'forceRescan"" events
* [NMS-120 <http://issues.opennms.org/browse/NMS-120>] -- NMS-112
Enable scanning of a single node rather than an entire requisition
* [NMS-121 <http://issues.opennms.org/browse/NMS-121>] -- NMS-112
handle 'population; of serivce table from foreighSource files
* [NMS-122 <http://issues.opennms.org/browse/NMS-122>] -- NMS-112
enable typing a new service name in foreignsource editor
* [NMS-123 <http://issues.opennms.org/browse/NMS-123>] -- NMS-112
Rename foreign source to something that has more meaning
* [NMS-124 <http://issues.opennms.org/browse/NMS-124>] -- NMS-112
use serivceRegistry for discovery of plugins and policies
* [NMS-125 <http://issues.opennms.org/browse/NMS-125>] -- NMS-112 Be
sure that a scheduled import blocks a scheduled or forced rescan
of a foreign source (a.k.a. group)
* [NMS-126 <http://issues.opennms.org/browse/NMS-126>] -- NMS-112 Be
sure that a rescan blocks a scheduled re-import of a foreign
source (a.k.a. group)
* [NMS-2572 <http://issues.opennms.org/browse/NMS-2572>] -- Stop
distributing the non-Jetty webapp as part of the base distribution
* [NMS-2640 <http://issues.opennms.org/browse/NMS-2640>] --
[syslogd] OpenNMS's syslogd implementation does not handle all
syslog facilities
* [NMS-3068 <http://issues.opennms.org/browse/NMS-3068>] --
rrd-configuration.properties error in comment
* [NMS-3465 <http://issues.opennms.org/browse/NMS-3465>] --
Liquibase: ERROR: there is no unique constraint matching given
keys for referenced table ""datalinkinterface""
* [NMS-3469 <http://issues.opennms.org/browse/NMS-3469>] -- webapps
won't deploy with Tomcat in 1.7.7 and 1.7.8-SNAPSHOT
* [NMS-3717 <http://issues.opennms.org/browse/NMS-3717>] -- Can't
proceed in Reports Database Run
* [NMS-3736 <http://issues.opennms.org/browse/NMS-3736>] -- Resource
Graphs and custom time period error while refreshing
* [NMS-3855 <http://issues.opennms.org/browse/NMS-3855>] --
statistics report pdf creation does not work
* [NMS-3882 <http://issues.opennms.org/browse/NMS-3882>] -- Case
sensitive inconsistency between capsd and poller for
HostResourceSwRunMonitor
* [NMS-3922 <http://issues.opennms.org/browse/NMS-3922>] --
Installer still has RCS-style $Id$ tag in banner
* [NMS-4057 <http://issues.opennms.org/browse/NMS-4057>] -- database
reports ui enhancement request
* [NMS-4058 <http://issues.opennms.org/browse/NMS-4058>] -- stored
report data in non-unique named files
* [NMS-4118 <http://issues.opennms.org/browse/NMS-4118>] -- null
pointer exception from Admin ? Instrumentation Log Reader jsp page
* [NMS-4150 <http://issues.opennms.org/browse/NMS-4150>] --
memBuffer should count in threshold limit
* [NMS-4253 <http://issues.opennms.org/browse/NMS-4253>] -- Date
offsets incorrectly applied when batch reports
* [NMS-4269 <http://issues.opennms.org/browse/NMS-4269>] -- calender
report shows two month back outages
* [NMS-4294 <http://issues.opennms.org/browse/NMS-4294>] --
Node-Availability-Report failing -- possibly due to a parameter
that should be marked notForPrompting
* [NMS-4299 <http://issues.opennms.org/browse/NMS-4299>] -- Default
categories for CIORPRTS
* [NMS-4314 <http://issues.opennms.org/browse/NMS-4314>] --
Dashboard loading image tag missing
* [NMS-4318 <http://issues.opennms.org/browse/NMS-4318>] --
reportd-configuration.xml, update quartz URL
* [NMS-4319 <http://issues.opennms.org/browse/NMS-4319>] --
JasperReport time range helper
* [NMS-4322 <http://issues.opennms.org/browse/NMS-4322>] -- Fonts
Very Large on KSC Reports and Admin Surveillance Categories
* [NMS-4359 <http://issues.opennms.org/browse/NMS-4359>] --
Calculations in new Jasper JRB Traffic-Reports are wrong
* [NMS-4365 <http://issues.opennms.org/browse/NMS-4365>] -- Early
Morning Report -- Limit Top 20 events also to 24 hours
* [NMS-4368 <http://issues.opennms.org/browse/NMS-4368>] --
Javascript error in IE7 on 1.8.7
* [NMS-4378 <http://issues.opennms.org/browse/NMS-4378>] -- OpenNMS
will not start if unable to connect to the database as
""postgres"" (admin) user
* [NMS-4379 <http://issues.opennms.org/browse/NMS-4379>] --
Provisioner always punts on deleteService events when discovery
not enabled
* [NMS-4384 <http://issues.opennms.org/browse/NMS-4384>] -- Make
sure tests pass on bamboo
* [NMS-4386 <http://issues.opennms.org/browse/NMS-4386>] -- Add a
build script to clean everything (top-level and full assembly)
* [NMS-4389 <http://issues.opennms.org/browse/NMS-4389>] -- OpenNMS
1.9.x needs to depend on PostgreSQL (>= 8.1) and IPLIKE (>= 2.0.0)
* [NMS-4391 <http://issues.opennms.org/browse/NMS-4391>] --
Configure SNMP Data Collection per Interface generates
org.postgresql.util.PSQLException
* [NMS-4392 <http://issues.opennms.org/browse/NMS-4392>] -- OpenNMS
unable to connect to opennms database with opennms username
* [NMS-4393 <http://issues.opennms.org/browse/NMS-4393>] -- Delete
Nodes Button is a no-op
* [NMS-4394 <http://issues.opennms.org/browse/NMS-4394>] -- Clicking
the Delete Icon next to a entity (node, interface, service)
defined in the Edit provisioning group page causes exception
* [NMS-4397 <http://issues.opennms.org/browse/NMS-4397>] -- rancid
provisioning adapter jar still included in opennms-core RPM
* [NMS-4398 <http://issues.opennms.org/browse/NMS-4398>] -- Unable
to see Telnet/HTTP/OpenManage links in node.jsp
* [NMS-4403 <http://issues.opennms.org/browse/NMS-4403>] -- 1.8.7 >
1.8.9 failed to startup -- 1.8.9 factory statsd-configuration.xml
* [NMS-4406 <http://issues.opennms.org/browse/NMS-4406>] -- KSC
reports editor broken in IE7
* [NMS-4408 <http://issues.opennms.org/browse/NMS-4408>] --
installer should make sure IPLIKE has IPv6 support
* [NMS-4409 <http://issues.opennms.org/browse/NMS-4409>] -- OpenNMS
not installing on RHEL6 x86_64
* [NMS-4410 <http://issues.opennms.org/browse/NMS-4410>] -- Node
Availability report query incorrect
* [NMS-4411 <http://issues.opennms.org/browse/NMS-4411>] --
opennms-webapp is built and contains all of the jars though they
are not needed
* [NMS-4414 <http://issues.opennms.org/browse/NMS-4414>] -- REST
calls for iPhone and iPad App broken
* [NMS-4415 <http://issues.opennms.org/browse/NMS-4415>] -- Deleting
Foreign Source policy throws java.lang.NoSuchMethodException
* [NMS-4416 <http://issues.opennms.org/browse/NMS-4416>] --
Documentation can't be build off-line
* [NMS-4417 <http://issues.opennms.org/browse/NMS-4417>] --
Interface Availability report query incorrect
* [NMS-4420 <http://issues.opennms.org/browse/NMS-4420>] -- Need to
update PostgreSQL JDBC JAR to 9.0
* [NMS-4421 <http://issues.opennms.org/browse/NMS-4421>] -- Change
event-label name ""OpeNMS"" for restartPollingInterface event in
eventconf.xml
* [NMS-4425 <http://issues.opennms.org/browse/NMS-4425>] -- Only
role.admin users are able to use the iphone app
* [NMS-4426 <http://issues.opennms.org/browse/NMS-4426>] --
Acknowledging alarms results in an error
* [NMS-4428 <http://issues.opennms.org/browse/NMS-4428>] -- Increase
default max file descriptors setting
* [NMS-4429 <http://issues.opennms.org/browse/NMS-4429>] --
runInPlace.sh script gives unclear instructions
* [NMS-4431 <http://issues.opennms.org/browse/NMS-4431>] -- upgrade
to JRobin 1.5.10
* [NMS-4432 <http://issues.opennms.org/browse/NMS-4432>] -- null
pointer exception when attempting to enable snmp collection.
* [NMS-4438 <http://issues.opennms.org/browse/NMS-4438>] -- WebUI
Broken When Deleting Parameters in Foreign Sources
* [NMS-4450 <http://issues.opennms.org/browse/NMS-4450>] -- create a
tool for migrating RRDs
* [NMS-4466 <http://issues.opennms.org/browse/NMS-4466>] --
Provisioning WebUI: Adding multiple fields fails
* [NMS-4467 <http://issues.opennms.org/browse/NMS-4467>] -- WebUI:
Foreign-Sources -- only first policy has a delete button
* [NMS-4469 <http://issues.opennms.org/browse/NMS-4469>] --
interface.jsp gives NPE
iPhone App
If you've been having issues accessing OpenNMS with the iPhone/iPad app,
please upgrade to OpenNMS 1.8.10 or 1.9.6, it contains a significant
number of REST fixes (and a new suite of unit tests) to make sure iOS
interoperability works.
Issues with "Issues with Issues"
If you'll recall, in the last TWiO I talked about the way we've been
trying to restructure the way we deal with issues. OpenNMS 1.8.8 and
1.9.4 were the first releases after we started moving to dealing with
JIRA in a more timely manner and, well, hey, look at that, we're at
1.8.10 and 1.9.6 already?!? What the heck?!?!
So, while we are dealing with bugs and enhancements in a *much* better
manner, we exposed our next big problem: bake time. For 1.8.8, we didn't
really have this big "rush" to get some crazy critical bug out, but in
fixing bugs right up to the day before release day, we created a similar
problem: all that great stuff we fixed and merged just a day or two
before release never really got any sanity checking. Sure, we do some
basic build tests, and of course, the unit tests all have to pass, but
that doesn't solve issues like NMS-4392
<http://issues.opennms.org/browse/NMS-4392> where a bug was fixed, but
in doing so it introduced a race condition at initialization that caused
a number of our users to get a fatal error on OpenNMS startup. The three
machines I tested it on did fine, but for plenty of folks who upgraded
to 1.8.8 (and 1.9.4) it did not.
1.8.9/1.9.5 came out shortly afterwards, and fixed the immediate issue,
but still had some issues that, while not showstoppers
for /most/ OpenNMS users, were significant enough to warrant another
release off the monthly release schedule. This time we'd learned our
lesson. As tempting as it was to immediately put out fixes for those
issues, we let things bake for a week. I asked people on the OpenNMS
lists to try out the release candidate branch, we gave time for people
to try out snapshots, and we got some good feedback, and a few more
corner case bugs related to the things we'd cleaned up. So now we have a
template for going forward.
So, in short... sorry about the couple of broken releases. A perfect
storm of adjusting the way we handle releases, and unexpected bugs
caused things to go worse than we could have imagined. Both of those
releases still worked fine for plenty of folks, but "plenty of folks" is
not exactly the bar to target. ;) Going forward, we want things to look
like 1.8.10, which is shaping up to be a solid release, and not like
1.8.8, which, well, was only /half/-baked.
Working with JRobin
While The OpenNMS Group has been the stewards of the JRobin code for
quite some time now, it has only sporadically had work done, mostly just
to scratch a particular itch. This last week, we had a big itch to
scratch that will probably interest a lot of people: the conversion to
storeByGroup.
The Backstory
If you're not familiar with storeByGroup, I'll give you a little
history. Originally, OpenNMS stored every type of collected data in it's
own RRD file. Every time you collected ifInOctets on a MIB-2 device, it
went in ifInOctets.rrd, ciscoMemoryPoolUsed on a Cisco device became
ciscoMemoryPoolUsed.rrd, and so on. This was ultimately flexible,
because MIBs could change over time, you could decide to collect more or
less data, and RRDs would go in or out of use as you made those changes.
The problem with this is: disk I/O. On a well-tuned OpenNMS system, I/O
is the single tightest bottleneck. As our user-base grew and we had
folks with 10s or even 100s of thousands of interfaces, the process of
open, read, write, close becomes very expensive, especially when you get
onto large disk arrays with large block sizes. "Store By Group" was the
solution to this. You lose the flexibility of being able to change your
"schema" of your RRDs, but in return you're often only doing a single
block open-read-write-close for the entirety of MIB-2 data for an
interface, rather than one for each varbind.
Unfortunately, there was no good way to convert this data, if you have a
lot of history you don't want to lose. If you keep everything the same
and just convert to storeByGroup, you could probably get away with doing
some heavy massaging of RRD xml dump data, but it got very sticky very
quickly.
So finally, we had a customer who, a few weeks ago, was coming to us for
help with OpenNMS being unable to keep up with data collection before.
We converted them to storeByGroup as it's generally the single most
performant thing you can do when it comes to data collection, and we
also switched them from the old OpenNMS 1.2 RRA definitions which stored
something crazy like 6 months of 5-minute data. They most definitely
needed their old data, though, as it is customer facing. So, I spent the
last week and a half, with some help from Matt and Jeff, writing an
engine for converting their data.
The JRobin Converter
So before trying to abstract everything, it made sense to at least make
everything work with JRobin, where we have good control over the APIs
used to pull the data out, and put it back in. Since their RRAs changed,
we can't just take the raw data from the old file, and put it in the
new, because it wouldn't match. The steps wouldn't line up. The best
architecture is one that lets you query any part of an RRD for it's
data, in the step size of your choosing, for a date range of your
choosing. If the step sizes don't match, it should interpolate as best
it can, and if there's no data (either because there's none to begin
with, or you're outside of the time range) it should just consider that
a NaN (not-a-number).
This interface ended up working out really well, because it applies just
as easily to an individual collection of values (one dsName), an archive
(one collection of dsNames), or a whole RRD (a collection of archives).
On top of that, I built an "AggregateTimeSeriesData" object which wraps
multiple RRD files and creates a virtual "sample" out of the best data
from each of the RRDs, wherever it's available.
After creating that framework, writing the actual converter was simple.
Scan a directory for RRD files, ask them for their dsNames. If they have
more than one dsName, it's a storeByGroup RRD file. Look in that same
directory for individual files matching the dsNames in the storeByGroup
one, and you have your source data. Slurp it up into a giant table of
data in the step we want (5 minutes, in this case) using the "aggregate"
interface, and all we need to do is write it back out into a new RRD.
For more details on using the JRobin Converter, see the JRobin Converter
wiki page <http://www.opennms.org/wiki/JRobin_Converter>.
Optimization
Since I was already digging in the JRobin code, I spent some time in a
profiler <http://www.yourkit.com/> trying to figure out if there were
things that could be sped up in JRobin. With 88k nodes, any small
optimization I could make would cut hours or even days off the
conversion. I found a number of places where deep in loops, it would be
getting, or re-calculating, the same data multiple times. Additionally,
I finalized a bunch of variables, which helps Java give hints to the
garbage collector about how long various data is going to stick around
in memory, or be reused. In all, I was able to get anywhere from a 20-
to 35-percent increase in speed. Mike Huot, another OGP member,
confirmed that his IOPS (number of I/O operations) dropped from 1800 to
450. This code change is in what will become JRobin 1.5.11, but for now,
it's only available in CVS. If you're interested in helping test, please
let me know <mailto:[email protected]>, or stop by in IRC
<irc://irc.freenode.net/%23opennms> and I'll let you know how to try it out.
Future Plans
In the future, I want to expand it to supporting both RRDtool and JRobin
archives, to allow for easy transition back and forth, and also to
support exploding storeByGroup RRDs *back* into individual ones (or even
just transitioning data into an RRD with more or less dsNames if you
change settings), but I'm not sure when I will have time to do it, the
immediate need is met, and I've been watching it chug along through 88k
nodes since Friday. =)
OpenNMS Users Conference Americas
One of our OGP <http://www.opennms.org/wiki/OGP> members, Mike Huot, has
been spearheading a campaign to create an OpenNMS Users Conference
Americas, on June 17th and 18th at the Unniversity of Minnesota, Twin
Cities, just before Dev-Jam 2011
<http://www.opennms.org/wiki/Dev-Jam_2011>. We are still working on
finalizing arrangements, but should have this completed by mid-March.
The fee will be $65.
We have a minimum threshold of attendees we need to get to before we can
hold this event, so please sign up here
<http://ouca2011.eventbrite.com/> to register your interest (no fee will
be charged) and we can gauge whether we have enough folks to put
together the event.
Upcoming Events
* 26 March 2011: Tarus to keynote the inaugural Indiana LinuxFest
<http://www.indianalinux.org/cms/>
* 26 through 27 May 2011: OpenNMS Users Conference Europe
<http://www.opennms.org/wiki/UCE#OpenNMS_User_Conference_Europe_2011>
* 17 through 18 June 2011: OpenNMS UCE Americas (Tentative)
* 19 through 24 June 2011: OpenNMS Dev-Jam 6
<http://www.opennms.org/wiki/Dev-Jam_2011>
Until Next Week...
As always, if there's anything you'd like me to talk about in a future
TWiO, or you just have a comment or criticism that you'd like to share,
don't hesitate to say hi <mailto:[email protected]>.
--
Benjamin Reed
The OpenNMS Group
http://www.opennms.org/
------------------------------------------------------------------------------
Free Software Download: Index, Search & Analyze Logs and other IT data in
Real-Time with Splunk. Collect, index and harness all the fast moving IT data
generated by your applications, servers and devices whether physical, virtual
or in the cloud. Deliver compliance at lower cost and gain new business
insights. http://p.sf.net/sfu/splunk-dev2dev
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/wiki/index.php?page=MailingListFaq
opennms-announce mailing list
To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-announce