Re: Has anyone added median to the summary statistics?
Gary Mulder <[email protected]> Thu, 5 Nov 2015 21:47:24 +0000
| Newsgroups | gmane.comp.java.grinder.user |
|---|---|
| Message-ID | <CAKy9zTCz+qT-er-c_8yhtnPTH4Uu0vLWpohLzEiwiaZfktGKsg@mail.gmail.com> |
--===============7051855653767787982== Content-Type: multipart/alternative; boundary=001a114e3d46e66f710523d2147e --001a114e3d46e66f710523d2147e Content-Type: text/plain; charset=UTF-8 Darren, What is perhaps more ironic is the is the kool aid Gil has identified that most Performance Testers are drinking and sharing with management when producing statistics. I've started to move on from perf. testing to data analytics and what I'm learning has put into question many of the stats most test tools report. I always sort of knew that standard deviations > means were indicative of invalid statistical conclusions based around the assumption of Gaussian (bell curve) distributed random processes. Most response times are not Gaussian distributed, unless you have a very well-behaved application that isn't loaded near capacity. I suspect that Poison distributions rather than Gaussian distributions are what we should be measuring and modelling - namely counts of events over an interval of time. Add in Gil's coordinated omission and the complete lack of statistical significance analysis when reporting results and the only conclusion you can really make about the numbers coming out of our performance tests is that "they bear some general indication on what *might* occur under ideal real world load conditions". In the end we can thank developers for often delivering very non-performant code which provides for a target rich environment for load testing! Simply throwing somewhat representative load an an application and watching what starts to smoke usually provides enough evidence that the application needs some performance redesign and re-factoring. While there will always be a role for performance testing, I propose that A/B testing and concomitant rigorous statistical comparison between the performance of A and B is likely the future of cloudy big-data apps. Regards, Gary On 5 November 2015 at 21:19, Darren Ball <[email protected]> wrote: > It is funny Gary... > > Gil just gave a talk at Intuit (CA) where I work, on this subject, entitle > the same as the youtube video. > > Ironic! > > -Darren > > On Thu, Oct 8, 2015 at 7:21 AM, Darren Ball <[email protected]> wrote: > >> Thanks Gary, >> But I am not just satisfying myself with these numbers. :) >> >> I have requests for min, max, mode and median regardless of the fact that >> I think they may not matter. >> >> I was just wondering if anyone has had luck getting these into summary >> stats. >> On Oct 8, 2015 2:21 AM, "Gary Mulder" <[email protected]> wrote: >> >>> Anything is possible, but the 99th, 99.9th and 99.99th percentiles might >>> tell you more about the actual performance of your system. >>> >>> Have a watch of this when you have a spare 45 minutes: >>> >>> https://www.youtube.com/watch?v=lJ8ydIuPFeU >>> >>> Regards, >>> Gary >>> >>> On 8 October 2015 at 05:17, Darren Ball <[email protected]> wrote: >>> >>>> Is it possible to add median to summary statistics? >>>> >>>> >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> >>>> _______________________________________________ >>>> grinder-use mailing list >>>> [email protected] >>>> https://lists.sourceforge.net/lists/listinfo/grinder-use >>>> >>>> >>> >>> >>> ------------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> grinder-use mailing list >>> [email protected] >>> https://lists.sourceforge.net/lists/listinfo/grinder-use >>> >>> > > > ------------------------------------------------------------------------------ > > _______________________________________________ > grinder-use mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/grinder-use > > --001a114e3d46e66f710523d2147e Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>Darren,</div><div><br></div>What is perhaps more iron= ic is the is the kool aid Gil has identified that most Performance Testers = are drinking and sharing with management when producing statistics. I'v= e started to move on from perf. testing to data analytics and what I'm = learning has put into question many of the stats most test tools report.<di= v><br></div><div>I always sort of knew that standard deviations > means = were indicative of invalid statistical conclusions based around the assumpt= ion of Gaussian (bell curve) distributed random processes. Most response ti= mes are not Gaussian distributed, unless you have a very well-behaved appli= cation that isn't loaded near capacity.</div><div><br></div><div>I susp= ect that Poison distributions rather than Gaussian distributions are what w= e should be measuring and modelling - namely counts of events over an inter= val of time. Add in Gil's coordinated omission and the complete lack of= statistical significance analysis when reporting results and the only conc= lusion you can really make about the numbers coming out of our performance = tests is that "they bear some general indication on what <i>might</i> = occur under ideal real world load conditions".<br><div class=3D"gmail_= extra"><br></div><div class=3D"gmail_extra">In the end we can thank develop= ers for often delivering very non-performant code which provides for a targ= et rich environment for load testing! Simply throwing somewhat representati= ve load an an application and watching what starts to smoke usually provide= s enough evidence that the application needs some performance redesign and = re-factoring.</div><div class=3D"gmail_extra"><br></div><div class=3D"gmail= _extra">While there will always be a role for performance testing, I propos= e that A/B testing and concomitant rigorous statistical comparison between = the performance of A and B is likely the future of cloudy big-data apps.</d= iv><div class=3D"gmail_extra"><br></div><div class=3D"gmail_extra">Regards,= </div><div class=3D"gmail_extra">Gary</div><div class=3D"gmail_extra"><br><= /div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On 5 Novembe= r 2015 at 21:19, Darren Ball <span dir=3D"ltr"><<a href=3D"mailto:ballda= [email protected]" target=3D"_blank">[email protected]</a>></span> wro= te:<br><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;= border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:= solid;padding-left:1ex"><div dir=3D"ltr">It is funny Gary...<div><br></div>= <div>Gil just gave a talk at Intuit (CA) where I work, on this subject, ent= itle the same as the youtube video.</div><div><br></div><div>Ironic!</div><= div><br></div><div>-Darren</div></div><div class=3D"gmail_extra"><br><div c= lass=3D"gmail_quote">On Thu, Oct 8, 2015 at 7:21 AM, Darren Ball <span dir= =3D"ltr"><<a href=3D"mailto:[email protected]" target=3D"_blank">bal= [email protected]</a>></span> wrote:<br><blockquote class=3D"gmail_quot= e" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-colo= r:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><p dir=3D"ltr"= >Thanks Gary,<br> But I am not just satisfying myself with these numbers.=C2=A0 :)</p> <p dir=3D"ltr">I have requests for min, max, mode and median regardless of = the fact that I think they may not matter.</p> <p dir=3D"ltr">I was just wondering if anyone has had luck getting these in= to summary stats.</p><div><div> <div class=3D"gmail_quote">On Oct 8, 2015 2:21 AM, "Gary Mulder" = <<a href=3D"mailto:[email protected]" target=3D"_blank">flyingkiwi= [email protected]</a>> wrote:<br type=3D"attribution"><blockquote class=3D"g= mail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;border-= left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div = dir=3D"ltr">Anything is possible, but the 99th, 99.9th and 99.99th percenti= les might tell you more about the actual performance of your system.<div><b= r></div><div>Have a watch of this when you have a spare 45 minutes:</div><d= iv><br></div><div><a href=3D"https://www.youtube.com/watch?v=3DlJ8ydIuPFeU"= target=3D"_blank">https://www.youtube.com/watch?v=3DlJ8ydIuPFeU</a></div><= div><br></div><div>Regards,</div><div>Gary=C2=A0</div></div><div class=3D"g= mail_extra"><br><div class=3D"gmail_quote">On 8 October 2015 at 05:17, Darr= en Ball <span dir=3D"ltr"><<a href=3D"mailto:[email protected]" targ= et=3D"_blank">[email protected]</a>></span> wrote:<br><blockquote cl= ass=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px= ;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1e= x"><div dir=3D"ltr">Is it possible to add median to summary statistics?<div= ><br></div><div><br></div></div> <br>-----------------------------------------------------------------------= -------<br> <br>_______________________________________________<br> grinder-use mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank">grin= [email protected]</a><br> <a href=3D"https://lists.sourceforge.net/lists/listinfo/grinder-use" rel=3D= "noreferrer" target=3D"_blank">https://lists.sourceforge.net/lists/listinfo= /grinder-use</a><br> <br></blockquote></div><br></div> <br>-----------------------------------------------------------------------= -------<br> <br>_______________________________________________<br> grinder-use mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank">grin= [email protected]</a><br> <a href=3D"https://lists.sourceforge.net/lists/listinfo/grinder-use" rel=3D= "noreferrer" target=3D"_blank">https://lists.sourceforge.net/lists/listinfo= /grinder-use</a><br> <br></blockquote></div> </div></div></blockquote></div><br></div> <br>-----------------------------------------------------------------------= -------<br> <br>_______________________________________________<br> grinder-use mailing list<br> <a href=3D"mailto:[email protected]">[email protected]= ceforge.net</a><br> <a href=3D"https://lists.sourceforge.net/lists/listinfo/grinder-use" rel=3D= "noreferrer" target=3D"_blank">https://lists.sourceforge.net/lists/listinfo= /grinder-use</a><br> <br></blockquote></div><br></div></div></div> --001a114e3d46e66f710523d2147e-- --===============7051855653767787982== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------------ --===============7051855653767787982== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ grinder-use mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/grinder-use --===============7051855653767787982==--