RRD, Newts or both?

Hello,

I am running an ONMS instance which stores collected metrics to RRD files
(JRobin). Currently, we update 150 000 RRDs every 5 minutes. To be able to
run such a load of RRD updates, we must have moved RRDs to a RAM disk. (It
is a Linux ZRAM, i.e. LZMA compressed RAM disk formatted as XFS, because
XFS supports freeze operation, which is needed to dump whole RAM disk to
HDD every hour in a consistent way). This way, it works very fast even with
150 k RRDs.

However, we would like to shorten colletion interval, our target is to
lower it down to 10 seconds. With the RAM disk trick, it is doable on a
single computer even with RRD. But, isn't it time to switch to Newts? It
does not "loose precision" of old data (hard disk space should not be a
problem with current HDD prices) and could potentially scale even more than
RRD-on-RAM-disk.

I believe a single Cassandra/Newts node will accept our collection load (it
would be 15 k values per second), *BUT how will the aggregation work?*

You guys are using Newts with 5 minute interval, so it must get approx. 10
k (100 k) values to show an aggregated graph spanning one month (one year).
Without knowing Cassandra/Newts internals, I still believe it is feasible
in "near real time" (i.e. aggregate data on-demand of visualization). But
what if the data are gathered every 10 seconds? How long will it take to
load a graph aggregating 270 k (3 M) data points?

And what are disk space requirements of Newts/Cassandra? With 150 k metrics
and 10 seconds interval, it is 480 G data points per year... With RRD's 8
bytes per data point, one year would fit into 4 TB...

Kouli

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss