comments on draft-ietf-bmwg-bgp-basic-convergence-00

This is the authors response to the comments raised in the Berlin IETF 
WG meeting  regarding the repetition of tests
and the number of required  iterations.

- We definitely acknowledge the importance of repetition of tests during 
Convergence Benchmarking and we had already taken that into
    consideration in our draft.
- We had also socialized our methodology and testcases with various 
Service Providers and had incorporated their feedback.
- Here are the sections where we have information regarding repeatability:

<<<
4.7. Measurement Statistics

     The benchmark measurements may vary for each trial, due to the
     statistical nature of timer expirations, CPU scheduling, etc. It is
     recommended to repeat the test multiple times.  Evaluation of the
     test data must be done with an understanding of generally accepted
     testing practices regarding repeatability, variance and statistical
     significance of a small number of trials.

     For any repeated tests that are averaged to remove variance, all
     parameters MUST remain the same.
  >>>

- Under various testcases we have also included (ex. Sec.5.1.1, 5.1.5,5.2.1)

<<<
     Note: It is recommended that a single test with the same route
     mixture be repeated several times.  A report should provide the Stand
     Deviation of all tests and the Average.
  >>>

- Although, we do recommend in our text that repeatability is expected 
and users will carry out a number of iterations to characterize the
results to maximize the accuracy of observed results, but there is a 
trade-off between the number of iterations versus subjecting the DUT
into a stress situation. For instance, we had received a comment 
recommending that at least 100 iterations be carried out, we believe 
that this
number is extremely high and if these many iterations are carried out in 
quick succession, the purpose of convergence testing will convert
into protocol scale stress testing and we may not get reliable results. 
We want the results to characterize the platform's convergence in a
real world environment as well rather than produce results which could 
rarely mimic the deployment conditions

Please let us know if this addresses the comments or anything else needs 
to be added/modified. We can also further go over this in Friday's
meeting.

thanks,
Bhavani Parise