Re: rrdtool memory usage and the Raspberry Pi
Tobias Oetiker <[email protected]>
| Newsgroups | gmane.comp.db.rrdtool.user |
|---|---|
| Message-ID | <alpine.DEB.2.20.1510190833080.14465@engelberg> |
Hi Jared, are you using 1.5.4 ? cheers tobi Today Jared Henley wrote: > > > On 16/10/15 17:19, Tobias Oetiker wrote: > > Hi Jared, > > > > Today Jared Henley wrote: > > > > > Hi, > > > > > > I've been working on a logging application that uses rrdtool. It works > > > brilliantly on my PC, not so good on the Raspberry Pi model B+. But > > > before I > > > go there, there's something interesting I noticed while working on the PC. > > > > > > I'm creating a database with the following command (in rrdpython) > > > > > > rrdtool.create(filepath, > > > '--step', '1s', > > > 'DS:progreset:GAUGE:1000s:0:1', > > > 'RRA:MAX:0.1:5s:5000m', > > > > > > <snip - there are 29 definitions all up, all the same> > > > > > > This creates a 386MB file. > > > > > > I didn't think too hard about it, until I did a graph to a CSV file. Of > > > course the CSV would be less space-efficient than the binary rrd file, > > > right? > > > But the generated CSV turns out to be only 15MB? > > > > > > rrdtool seems to use 64-bit integers for everything, so I figure the rrd > > > file > > > above should use: > > > 8 bytes per RRA entry * 29 RRAs + 1 timestamp * 60,000 locations in the > > > round-robin (5000 minutes / 5 seconds) = 14.4MB. I'm confused. Why is the > > > rrd > > > file 30 times bigger than I'd expect? I did experiment with a step size > > > of 5 > > > seconds, but the created file was the same size. But I can live with > > > largish > > > files. > > when I create this database on my intel box it looks like this: > > > > oetiker>./rrdtool create /tmp/demo.rrd --step 1s > > DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m > > oetiker>ls -l /tmp/demo.rrd > > -rw-r--r-- 1 oetiker oep 480584 Oct 16 08:17 /tmp/demo.rrd > > > > so something looks rather odd here ... are you using 1.5.4 on your > > raspy ? > > > > > > cheers > > tobi > > Hi Tobi, > > I can't actually create the database on the rpi - the process gets killed. > Results mentioned are on my AMD 64-bit box, as are all tests below. > > I just ran your test: > > [08:32:39 pe@hope] $ rrdtool create /tmp/demo.rrd --step 1s > DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m > 0 ~ > [08:32:49 pe@hope] $ ls -l /tmp/demo.rrd > -rw-rw-r-- 1 pe pe 480584 Oct 19 08:32 /tmp/demo.rrd > > Results are identical and obviously what I would expect. > > I'm having another issue with rrdpython, (but my mail to Hye-Shik Chang is > bouncing), so I just tried my database creation directly in rrdtool with the > following command (complete): > > [08:32:57 pe@hope] $ rrdtool create test.rrd \ > > --step 1s \ > > DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \ > > DS:tpm:GAUGE:1000s:0:1 RRA:LAST:0.1:5s:5000m \ > > DS:voltcontrol:GAUGE:1000s:0:1 RRA:LAST:0.1:5s:5000m \ > > DS:freqcontrol:GAUGE:1000s:0:1 RRA:LAST:0.1:5s:5000m \ > > DS:fault:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \ > > DS:uff:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \ > > DS:off:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \ > > DS:uvf:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \ > > DS:ovf:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \ > > DS:svopen:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \ > > DS:svclose:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \ > > DS:bvopen:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \ > > DS:bvclose:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \ > > DS:syncen:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \ > > DS:busconnen:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \ > > DS:wmfull:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \ > > DS:wmmed:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \ > > DS:wmlow:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \ > > DS:vred:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \ > > DS:exciter:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \ > > DS:ired:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \ > > DS:iwhite:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \ > > DS:iblue:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \ > > DS:f:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \ > > DS:isp:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \ > > DS:dlred:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \ > > DS:dlwhite:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \ > > DS:dlblue:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \ > > DS:svpos:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m > 0 ~ > [08:45:55 pe@hope] $ ls -l test.rrd > -rw-rw-r-- 1 pe pe 403757864 Oct 19 08:45 test.rrd > > Obviously something funny is going on between the one datasource/RRA test and > the 29 datasource/RRA test, so I tried with just two DS/RRAs: > > [08:47:32 pe@hope] $ rrdtool create test.rrd \ > > --step 1s \ > > DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \ > > DS:tpm:GAUGE:1000s:0:1 RRA:LAST:0.1:5s:5000m > 0 ~ > [08:47:52 pe@hope] $ ls -l test.rrd > -rw-rw-r-- 1 pe pe 1921184 Oct 19 08:47 test.rrd > > This file also seems too large. 1921184/480584 = 3.997... so it naively looks > like the filesize is following a parabolic function against DS/RRA pairs > rather than linear. I then created the following two files with almost no > data, so I could look at them in a hex editor easily. > > [08:55:59 pe@hope] $ rrdtool create test.rrd \ > > --step 1s \ > > DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:30s > 0 ~ > [08:56:01 pe@hope] $ ls -l test.rrd > -rw-rw-r-- 1 pe pe 632 Oct 19 08:56 test.rrd > 0 ~ > [08:56:03 pe@hope] $ rrdtool create test2.rrd \ > > --step 1s \ > > DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:30s \ > > DS:tpm:GAUGE:1000s:0:1 RRA:LAST:0.1:5s:30s > 0 ~ > [08:58:23 pe@hope] $ ls -l test2.rrd > -rw-rw-r-- 1 pe pe 1376 Oct 19 08:58 test2.rrd > > I really don't understand the structure of an rrd file, but there's a section > at the end populated with "00 00 00 00 00 00 F8 FF" multiple times over. I > assume this is where the RRA data is stored, and that the first section is the > definitions and most-recently-updated values. Working on that understanding, > the RRA data section of test.rrd is: > > 0000:0240 | 00 00 00 00 00 00 F8 FF | > ......øÿ > 0000:0250 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | > ......øÿ......øÿ > 0000:0260 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | > ......øÿ......øÿ > 0000:0270 | 00 00 00 00 00 00 F8 FF | ......øÿ > > > (as expected, 6 64-bit values) > > > And the RRA data section of test2.rrd is: > > 0000:04A0 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | > ......øÿ......øÿ > 0000:04B0 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | > ......øÿ......øÿ > 0000:04C0 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | > ......øÿ......øÿ > 0000:04D0 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | > ......øÿ......øÿ > 0000:04E0 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | > ......øÿ......øÿ > 0000:04F0 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | > ......øÿ......øÿ > 0000:0500 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | > ......øÿ......øÿ > 0000:0510 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | > ......øÿ......øÿ > 0000:0520 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | > ......øÿ......øÿ > 0000:0530 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | > ......øÿ......øÿ > 0000:0540 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | > ......øÿ......øÿ > 0000:0550 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | > ......øÿ......øÿ > > (I would expect 12 64-bit values, there are 24 here.) > > I did some updates to these files. test.rrd made some sense to me. Updates > were 0.1, 0.2, 0.3, 0.4, 0.5 in 5 second intervals. > > 0000:0240 | 9A 99 99 99 99 99 D9 3F | > ......Ù? > 0000:0250 | 00 00 00 00 00 00 E0 3F 33 33 33 33 33 33 E3 3F | > ......à?333333ã? > 0000:0260 | 9A 99 99 99 99 99 B9 3F 9A 99 99 99 99 99 C9 3F | > ......¹?......É? > 0000:0270 | 33 33 33 33 33 33 D3 3F | 333333Ó? > > and again, with updates 0.7, 0.7, 0.8, 0.9, 1: > > 0000:0240 | CD CC CC CC CC CC EC 3F | > ÍÌÌÌÌÌì? > 0000:0250 | 00 00 00 00 00 00 F0 3F 33 33 33 33 33 33 E3 3F | > ......ð?333333ã? > 0000:0260 | 66 66 66 66 66 66 E6 3F 66 66 66 66 66 66 E6 3F | > ffffffæ?ffffffæ? > 0000:0270 | 9A 99 99 99 99 99 E9 3F | ......é? > > > Updates to test2.rrd were 0.1:0.2, 0.3:0.4, 0.5:0.6, 0.7:0.9, 0.9:1 in 5 > second intervals. > > 0000:04A0 | 66 66 66 66 66 66 E6 3F CD CC CC CC CC CC EC 3F | > ffffffæ?ÍÌÌÌÌÌì? > 0000:04B0 | CD CC CC CC CC CC EC 3F 00 00 00 00 00 00 F0 3F | > ÍÌÌÌÌÌì?......ð? > 0000:04C0 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | > ......øÿ......øÿ > 0000:04D0 | 9A 99 99 99 99 99 B9 3F 9A 99 99 99 99 99 C9 3F | > ......¹?......É? > 0000:04E0 | 33 33 33 33 33 33 D3 3F 9A 99 99 99 99 99 D9 3F | > 333333Ó?......Ù? > 0000:04F0 | 00 00 00 00 00 00 E0 3F 33 33 33 33 33 33 E3 3F | > ......à?333333ã? > 0000:0500 | 9A 99 99 99 99 99 B9 3F 9A 99 99 99 99 99 C9 3F | > ......¹?......É? > 0000:0510 | 33 33 33 33 33 33 D3 3F 9A 99 99 99 99 99 D9 3F | > 333333Ó?......Ù? > 0000:0520 | 00 00 00 00 00 00 E0 3F 33 33 33 33 33 33 E3 3F | > ......à?333333ã? > 0000:0530 | 66 66 66 66 66 66 E6 3F CD CC CC CC CC CC EC 3F | > ffffffæ?ÍÌÌÌÌÌì? > 0000:0540 | CD CC CC CC CC CC EC 3F 00 00 00 00 00 00 F0 3F | > ÍÌÌÌÌÌì?......ð? > 0000:0550 | 00 00 00 00 00 00 F8 FF 00 00 00 00 00 00 F8 FF | > ......øÿ......øÿ > > Using the info from test.rrd's hex dump, it looks like the values in test2.rrd > are: > > 0.7 0.9 > 0.9 1 > UNKNOWN UNKNOWN > 0.4 0.5 > 0.6 0.1 > 0.2 0.3 > 0.4 0.5 > 0.6 0.1 > 0.2 0.3 > 0.7 0.9 > 0.9 1 > UNKNOWN UNKNOWN > > It seems like the file test2.rrd contains two sets of RRA data like this: > > first-half data > second-half data > second-half data > first-half data > > Jared. > > > > > > > > > > > > > -- Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland www.oetiker.ch [email protected] +41 62 775 9902 _______________________________________________ rrd-users mailing list [email protected] https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users