NaN values in percentile calculations
Jacques du Toit <[email protected]> Thu, 25 May 2017 14:15:18 +0200
| Newsgroups | gmane.comp.db.rrdtool.user |
|---|---|
| Message-ID | <[email protected]> |
--===============3685835466674657983== Content-Type: multipart/alternative; boundary="Apple-Mail=_E76AF78A-56C5-46F1-BB41-BA77371BAF68" --Apple-Mail=_E76AF78A-56C5-46F1-BB41-BA77371BAF68 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi guys I'm having this issue with some of our data for a customer=E2=80=99s = billing. Wondering if there is an elegant solution? What I really want = is the ability to exclude NaN values from percentile calculations. = Consider the following data series: foo =3D 1,NaN,2,NaN,3,NaN,4,NaN,5,NaN,6,NaN,7,NaN,8,NaN,9,NaN,10,NaN VDEF:90perc=3Dfoo,90,PERCENT Seems like rrdtool includes the NaN values in the calculation. So I get = 90perc=3D8. While technically correct according to the series, its not = really useful in determining real 90th percentile values in an every-day = use-case. Particularly with billing, just because you have no data = it=E2=80=99s not reasonable to assume it=E2=80=99s 0. Most likely the = traffic profile by =E2=80=9Cconnecting-the-dots=E2=80=9D would have = actually looked like this: 1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8,9,9,10,10 In this case I get 90perc=3D9 which means the billing would be skewed = because of the loss of data. Now, it=E2=80=99s debatable what is = mathematically really the right thing to do here. But the bottom line is = that that we don=E2=80=99t know if the NaN values would have been above = or below the 90th percentile value, it=E2=80=99s better to exclude them = rather than assume they are below, IMHO. The customer would also not be too happy as NaN values being included = always pushes the percentile value down by definition and this mean they = might end up with slightly =E2=80=9Cincorrect=E2=80=9D billing. So anyone know of a way to exclude those NaN values from the PERCENT = calculation? Thanks, Jacques= --Apple-Mail=_E76AF78A-56C5-46F1-BB41-BA77371BAF68 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D"">Hi guys<div class=3D""><br class=3D""></div><div class=3D"">I'm= having this issue with some of our data for a customer=E2=80=99s = billing. Wondering if there is an elegant solution? What I really want = is the ability to exclude NaN values from percentile calculations. = Consider the following data series:</div><div class=3D""><br = class=3D""></div><div class=3D""><font face=3D"Courier New" class=3D"">foo= =3D = 1,NaN,2,NaN,3,NaN,4,NaN,5,NaN,6,NaN,7,NaN,8,NaN,9,NaN,10,NaN</font></div><= div class=3D""><font face=3D"Courier New" class=3D""><br = class=3D""></font></div><div class=3D""><font face=3D"Courier New" = class=3D"">VDEF:90perc=3Dfoo,90,PERCENT</font></div><div class=3D""><br = class=3D""></div><div class=3D"">Seems like rrdtool includes the NaN = values in the calculation. So I get <font face=3D"Courier New" = class=3D"">90perc=3D8</font>. While technically correct according to the = series, its not really useful in determining real 90th percentile values = in an every-day use-case. Particularly with billing, just because you = have no data it=E2=80=99s not reasonable to assume it=E2=80=99s 0. Most = likely the traffic profile by =E2=80=9Cconnecting-the-dots=E2=80=9D = would have actually looked like this:</div><div class=3D""><br = class=3D""></div><div class=3D""><div class=3D""><font face=3D"Courier = New" = class=3D"">1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8,9,9,10,10</font></div></div><di= v class=3D""><font face=3D"Courier New" class=3D""><br = class=3D""></font></div><div class=3D"">In this case I get <font = face=3D"Courier New" class=3D"">90perc=3D9</font> which means the = billing would be skewed because of the loss of data. Now, it=E2=80=99s = debatable what is mathematically really the right thing to do here. But = the bottom line is that that we don=E2=80=99t know if the NaN values = would have been above or below the 90th percentile value, it=E2=80=99s = better to exclude them rather than assume they are below, = IMHO.</div><div class=3D""><br class=3D""></div><div class=3D"">The = customer would also not be too happy as NaN values being included always = pushes the percentile value down by definition and this mean they might = end up with slightly =E2=80=9Cincorrect=E2=80=9D billing.</div><div = class=3D""><br class=3D""></div><div class=3D"">So anyone know of a way = to exclude those NaN values from the PERCENT calculation?</div><div = class=3D""><br class=3D""></div><div class=3D"">Thanks,</div><div = class=3D""> Jacques</div></body></html>= --Apple-Mail=_E76AF78A-56C5-46F1-BB41-BA77371BAF68-- --===============3685835466674657983== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ rrd-users mailing list [email protected] https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users --===============3685835466674657983==--