Re: RRDcached performance issues (from Users mailinglist)

--===============3908946910658881283==
Content-Type: multipart/alternative; boundary=089e010d879cf194cf05116325d7

--089e010d879cf194cf05116325d7
Content-Type: text/plain; charset=UTF-8

Hi,

I'm a dev guy from the Munin project(*). We have some heavy RRD among our
users.

Some hints from the trenches:

* I discovered that having multiple rrdcached daemon is actually very bad
for performance. The only way to shard them efficiently is to have them
handling a different FS with different disk subsystem. (I didn't try with
SSD btw)
* For some workloads tmpfs+sync (note that a simple cp might be more
efficient than a rsync) is simply the best.
* You have to avoid doing reads on rrds, as it flushes. (1.4.x, don't know
about 1.5)
* A very big -w and -z setting (think 3600) helps, as it give rrdcached the
most flexibility to reorder writes
* Always flush manually before closing (FLUSHALL on the socket), to avoid
too long restart times. The restarts journal read is way too slow, a you
noticed.

Hopes it helps.

(*) http://mm0.eu/

On 17:59, Sun, Mar 15, 2015 Tobias Oetiker <[email protected]> wrote:

Hi Gaby,

I guess that very few people have rrdtool setup at your scale ...
so chances for someone coming forward with useful hints may be
rather slim.

  Things to look at:

a) try rrdtool 1.5rc2 since this is the latest and greates code

b) go into the cached code and add some debugging output

c) strace/truss may provide some insight in to what rrdcached when
   it seems busy

d) attach gdb and break processing to see where in the source this
   is happening.

cheers
tobi

Yesterday Gaby wrote:

> Dear developers, could you take a look at my post in the Users forum?
>
>
http://rrd-mailinglists.937164.n2.nabble.com/RRDcached-performance-issues-tp7582783.html;cid=1426345040002-291
>
> It has no replies, and I'm still very interested in shedding some light on
> it.
>
> Gaby
>
>
>
> --
> View this message in context:
http://rrd-mailinglists.937164.n2.nabble.com/RRDcached-performance-issues-from-Users-mailinglist-tp7582860.html
> Sent from the RRDtool Developers Mailinglist mailing list archive at
Nabble.com.
>
> _______________________________________________
> rrd-developers mailing list
> [email protected]
> https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers
>
>

--
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
www.oetiker.ch [email protected] +41 62 775 9902

_______________________________________________
rrd-developers mailing list
[email protected]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

--089e010d879cf194cf05116325d7
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<p dir=3D"ltr">Hi, </p>
<p dir=3D"ltr">I&#39;m a dev guy from the Munin project(*). We have some he=
avy RRD among our users.</p>
<p dir=3D"ltr">Some hints from the trenches:</p>
<p dir=3D"ltr">* I discovered that having multiple rrdcached daemon is actu=
ally very bad for performance. The only way to shard them efficiently is to=
 have them handling a different FS with different disk subsystem. (I didn&#=
39;t try with SSD btw)<br>
* For some workloads tmpfs+sync (note that a simple cp might be more effici=
ent than a rsync) is simply the best.<br>
* You have to avoid doing reads on rrds, as it flushes. (1.4.x, don&#39;t k=
now about 1.5)<br>
* A very big -w and -z setting (think 3600) helps, as it give rrdcached the=
 most flexibility to reorder writes<br>
* Always flush manually before closing (FLUSHALL on the socket), to avoid t=
oo long restart times. The restarts journal read is way too slow, a you not=
iced.</p>
<p dir=3D"ltr">Hopes it helps.</p>
<p dir=3D"ltr">(*) <a href=3D"http://mm0.eu/">http://mm0.eu/</a> </p>
<p dir=3D"ltr"></p>
<p dir=3D"ltr">On 17:59, Sun, Mar 15, 2015=C2=A0Tobias Oetiker &lt;<a href=
=3D"mailto:[email protected]">[email protected]</a>&gt; wrote:</p>
<blockquote><p dir=3D"ltr">Hi Gaby,</p>
<p dir=3D"ltr">I guess that very few people have rrdtool setup at your scal=
e ...<br>
so chances for someone coming forward with useful hints may be<br>
rather slim.</p>
</blockquote>
<p dir=3D"ltr"> </p>
<blockquote><p dir=3D"ltr"></p>
<p dir=3D"ltr">Things to look at:</p>
<p dir=3D"ltr">a) try rrdtool 1.5rc2 since this is the latest and greates c=
ode</p>
<p dir=3D"ltr">b) go into the cached code and add some debugging output</p>
<p dir=3D"ltr">c) strace/truss may provide some insight in to what rrdcache=
d when<br>
=C2=A0 =C2=A0it seems busy</p>
<p dir=3D"ltr">d) attach gdb and break processing to see where in the sourc=
e this<br>
=C2=A0 =C2=A0is happening.</p>
<p dir=3D"ltr">cheers<br>
tobi</p>
<p dir=3D"ltr">Yesterday Gaby wrote:</p>
<p dir=3D"ltr">&gt; Dear developers, could you take a look at my post in th=
e Users forum?<br>
&gt;<br>
&gt; <a href=3D"http://rrd-mailinglists.937164.n2.nabble.com/RRDcached-perf=
ormance-issues-tp7582783.html;cid=3D1426345040002-291">http://rrd-mailingli=
sts.937164.n2.nabble.com/RRDcached-performance-issues-tp7582783.html;cid=3D=
1426345040002-291</a><br>
&gt;<br>
&gt; It has no replies, and I&#39;m still very interested in shedding some =
light on<br>
&gt; it.<br>
&gt;<br>
&gt; Gaby<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt; --<br>
&gt; View this message in context: <a href=3D"http://rrd-mailinglists.93716=
4.n2.nabble.com/RRDcached-performance-issues-from-Users-mailinglist-tp75828=
60.html">http://rrd-mailinglists.937164.n2.nabble.com/RRDcached-performance=
-issues-from-Users-mailinglist-tp7582860.html</a><br>
&gt; Sent from the RRDtool Developers Mailinglist mailing list archive at N=
abble.com.<br>
&gt;<br>
&gt; _______________________________________________<br>
&gt; rrd-developers mailing list<br>
&gt; <a href=3D"mailto:[email protected]">rrd-developers@list=
s.oetiker.ch</a><br>
&gt; <a href=3D"https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers">h=
ttps://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers</a><br>
&gt;<br>
&gt;</p>
<p dir=3D"ltr">--<br>
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland<br>
<a href=3D"http://www.oetiker.ch">www.oetiker.ch</a> <a href=3D"mailto:tobi=
@oetiker.ch">[email protected]</a> +41 62 775 9902</p>
<p dir=3D"ltr">_______________________________________________<br>
rrd-developers mailing list<br>
<a href=3D"mailto:[email protected]">[email protected]=
iker.ch</a><br>
<a href=3D"https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers">https:=
//lists.oetiker.ch/cgi-bin/listinfo/rrd-developers</a><br>
</p>
</blockquote>
<p dir=3D"ltr"><br>
</p>

--089e010d879cf194cf05116325d7--


--===============3908946910658881283==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
rrd-developers mailing list
[email protected]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

--===============3908946910658881283==--