Re: SIP benchmarking drafts

Thanks for the update Vijay, it's nice to see. See you in London!

/S

On 2/18/14, 7:00 AM, "Vijay K. Gurbani" <[email protected]> wrote:

>Folks: Carol, Scott and I have submitted the SIP benchmarking drafts
>for discussion in London.
>
>These drafts were in IETF LC [1,2] back in January 2013.  As part
>of IETF LC, Robert Sparks performed an ind-epth review of the drafts
>[3].  The revised version of the drafts are based on attending to
>his review.
>
>We will follow up with a detailed list of changes.  Here is a high-
>level list of changes outside the editorial changes to improve
>readability:
>
>- Simplified the testing of SIP devices.  We no longer maintain
>   test setup parameters (Terminology, or T) related to forking and
>   loop detection.  While these add time for each SIP transaction
>   to complete, tests for these would be in essence protocol conformance
>   tests, not benchmarking tests.
>
>   Similarly, we have removed the test suites related to loop detection
>   and forking from the Methodology (M) document as well.
>
>- We have simplified the benchmarks being sought in T and M to three:
>   Session Establishment Rate, Registration Rate, and Registration
>   Attempt Rate.  Earlier sesison-related benchmarks (Session Capacity,
>   Session Overload Capacity, Session Establishment Performance, and
>   Session Attempt Delay) have been taken out for the simple reason that
>   our implementation experience indicated that these additional
>   benchmarks do not provide much benefit beyond what is provided by
>   the singularly important Session Establishment Rate benchmark.
>   Furthermore, reviewers also had questions on the nature of these
>   benchmarks and their uniform interpretation.
>
>- We have taken out benchmark related to IM Rate due to the
>   variabilities inherent in benchmarking it; for instance, size of
>   payload, fragmentation potential on large payloads, varying user
>   behaviour in the real world (an IM may pend until the subscriber
>   actually reads it and replies), etc.
>
>   The original intent of using IM was to benchmark a simple non-INVITE
>   transaction.  The current version of the draft takes the tact that
>   the REGISTER transaction better serves this purpose.
>
>- We have expanded the test reporting template to include artifacts
>   related to TLS ciphersuites (for TLS-based benchmarks) and IPSec
>   profiles (for IPSec-based benchmarks).
>
>In sum, we believe that the reduced focus of the benchmarking results
>in a vastly more tractable system whose properties the testing
>organizations can understand (and control) much better.  The results
>from the improved benchmarks will provide an authoritative answer for
>comparing different vendor offerings or understanding the behaviour of
>the device under test.
>
>-09 of terminology is available in [4] and methodology is available
>in [5].
>
>Comments are welcome.
>
>[1] http://www.ietf.org/mail-archive/web/bmwg/current/msg02717.html
>[2] http://www.ietf.org/mail-archive/web/bmwg/current/msg02718.html
>[3] http://www.ietf.org/mail-archive/web/bmwg/current/msg02719.html
>[4] http://tools.ietf.org/html/draft-ietf-bmwg-sip-bench-term-09
>[5] http://tools.ietf.org/html/draft-ietf-bmwg-sip-bench-meth-09
>
>Cheers,
>
>- vijay
>-- 
>Vijay K. Gurbani, Bell Laboratories, Alcatel-Lucent
>1960 Lucent Lane, Rm. 9C-533, Naperville, Illinois 60563 (USA)
>Email: vkg@{bell-labs.com,acm.org} / [email protected]
>Web: http://ect.bell-labs.com/who/vkg/  | Calendar: http://goo.gl/x3Ogq
>
>_______________________________________________
>bmwg mailing list
>[email protected]
>https://www.ietf.org/mailman/listinfo/bmwg