Mean vs Median

Hello BMWG,

Following some of the discussion we had in IETF93 about using either mean or median as a summarizing function for the results of multiple test iterations, I added the following section in http://tools.ietf.org/html/draft-ietf-bmwg-ipv6-tran-tech-benchmarking-00 
.

10(http://tools.ietf.org/html/draft-ietf-bmwg-ipv6-tran-tech-benchmarking-00#section-10). Summarizing function and repeatability To ensure the stability of the benchmarking scores obtained using the tests presented in Sections 6(http://tools.ietf.org/html/draft-ietf-bmwg-ipv6-tran-tech-benchmarking-00#section-6)-9(http://tools.ietf.org/html/draft-ietf-bmwg-ipv6-tran-tech-benchmarking-00#section-9), multiple test iterations are recommended. Following the recommendations of RFC2544(http://tools.ietf.org/html/rfc2544), the average was chosen to be the summarizing function for the reported values. While median can be an alternative summarizing function, a rationale for using one or the other is needed. The median can be useful for summarizing especially when outliers are not a desired quantity. However, in the overall performance of a network device the outliers can represent a malfunction or misconfiguration in the DUT, which should be taken into account. The average is a more inclusive summarizing function. Moreover, as underlined in [DeNijs(http://tools.ietf.org/html/draft-ietf-bmwg-ipv6-tran-tech-benchmarking-00#ref-DeNijs)], the average is less exposed to statistical uncertainty. These reasons make it the RECOMMENDED summarizing function for the results of different test iterations, unless stated otherwise. To express the repeatability of the benchmarking tests through a number, the Margin of error (MoE) can be used. Of course, other functions, such as standard error could be employed as well. The advantage the MoE has is expressing an associated confidence interval by using the alpha parameter. The recommended formula for calculating the MoE is presented in 

 Section 6.3.1(http://tools.ietf.org/html/draft-ietf-bmwg-ipv6-tran-tech-benchmarking-00#section-6.3.1).

After discussing this rationale with Al (Morton) and Kostas (Pentakousis), I am tending to lean towards using median. One of the reasons is non-normal probability distribution cases (e.g. bimodal distribution), where the Mean might not mean much (trying to paraphrase Al). One could add a step in the procedure like "analyze the probability distribution of the 20 measurements after deciding the summarizing function", but this might be an undesired over-complication. In any case, I think a measure of variance should be provided with the summarized results, in order to express the stability/repeatability of the results.

Since the rationale for using Mean or Median (or ...) could be reused in other documents produced by this WG, I would like to ask for more feedback on this subject.

Best regards,
Marius

_______________________________________________
bmwg mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/bmwg