Re: My comments on draft-ietf-bmwg-sip-bench-meth-09

Thanks.  We will review as soon as possible.

Regards,

Carol

Carol Davids
Professor &  Director RTC Lab
Illinois Institute of Technology
Office: 630-682-6024
Mobile: 630-292-9417
Email: [email protected]
Skype: caroldavids1
Web: rtc-lab.itm.iit.edu








On 4/7/14, 10:06 AM, "William Cerveny" <[email protected]> wrote:

>Below are my high level comments on draft-ietf-bmwg-sip-bench-meth-09
>denoted by /* comment */ ; I'm sending more detailed (mostly grammatical
>comments) directly to the authors. I've identified the section and
>general line number in which the comment is made.
>
>Of note, I had trouble following the variables in the pseudocode, but I
>may not have been following close enough attention to the code.
>
>Bill Cerveny
>
><begin comments on draft-ietf-bmwg-sip-bench-meth-09>
>
>desktop-10-36:scratch wcerveny$ cat grep-output-meth-2014-04-17.txt
>1. Abstract:
>
>30-   objective comparison of the capacity of SIP devices.  Test setup
>31-   parameters and a methodology are necessary because SIP allows a
>wide
>32-   range of configuration and operational conditions that can
>influence
>33:/* In my opinion, sentence beginning with "A standard terminology
>..."
>34:   is an assumed and I'm not sure it should be in the abstract */
>35-   performance benchmark measurements.  A standard terminology and
>36-   methodology will ensure that benchmarks have consistent definition
>37-   and were obtained following the same procedures.
>--
>2. Introduction:
>--
>200-   only.
>201-
>202-   The device-under-test (DUT) is a SIP server, which may be any SIP
>203:/* Capitalization in "Benchmarks can be ..." is inconsistant */
>204-   conforming [RFC3261] device.  Benchmarks can be obtained and
>compared
>205-   for different types of devices such as SIP Proxy Server, Session
>206-   Border Controllers (SBC), SIP registrars and SIP proxy server
>paired
>--
>--
>208-
>209-   The test cases provide metrics for benchmarking the maximum 'SIP
>210-   Registration Rate' and maximum 'SIP Session Establishment Rate'
>that
>211:/* Is "extended period" defined? */
>212-   the DUT can sustain over an extended period of time without
>failures.
>213-   Some cases are included to cover Encrypted SIP.  The test
>topologies
>214-   that can be used are described in the Test Setup section.
>Topologies
>--
>--
>219-
>220-   SIP permits a wide range of configuration options that are
>explained
>221-   in Section 4 and Section 2 of [I-D.sip-bench-term].  Benchmark
>values
>222:/* Is associated media defined */
>223-   could possibly be impacted by Associated Media.  The selected
>values
>224-   for Session Duration and Media Streams per Session enable
>benchmark
>225-
>--
>3. Benchmarking Topologies
>--
>259-
>260-   There are two test topologies; one in which the DUT does not
>process
>261-   the media (Figure 1) and the other in which it does process media
>262:/* EA defined? */
>263-   (Figure 2).  In both cases, the tester or EA sends traffic into
>the
>264-   DUT and absorbs traffic from the DUT.  The diagrams in Figure 1
>and
>265-   Figure 2 represent the logical flow of information and do not
>dictate
>--
>4.3 Associated Media
>--
>346-4.3.  Associated Media
>347-
>348-   Some tests require Associated Media to be present for each SIP
>349:/* Is this redundant? */
>350-   session.  The test topologies to be used when benchmarking DUT
>351-   performance for Associated Media are shown in Figure 1 and Figure
>2.
>352-
>--
>4.6 Session Duration
>--
>370-
>371-   The value of the DUT's performance benchmarks may vary with the
>372-   duration of SIP sessions.  Session Duration MUST be reported with
>373:/* I'm not sure if this sentence ("A Session Duration ...") is
>properly
>374-formed (it might be), but I had difficulty following the logic of
>the
>375:sentence */
>376-   benchmarking results.  A Session Duration of zero seconds
>indicates
>377-   transmission of a BYE immediately following successful SIP
>378-   establishment indicate by receipt of a 200 OK.  An infinite
>Session
>--
>4.8 Benchmarking algorithm
>--
>409-
>410-   During the Candidate Identification phase, the test runs until n
>411-   sessions have been attempted, at session attempt rates, r, which
>vary
>412:/* Upper case N and lower case n are different variables?? Same with
>"R" */
>413-   according to the algorithm below, where n is also a parameter of
>test
>414-   and is a relatively large number, but an order of magnitude
>smaller
>415-   than N. If no errors occur during the time it takes to attempt n
>--
>--
>415-   than N. If no errors occur during the time it takes to attempt n
>416-   sessions, we increment r according to the algorithm.  If errors
>are
>417-   encountered during the test, we decrement r according to the
>418:/* sentence "The algorithm provides ..." needs clarification
>419:Is the word "how" unnecessary? */
>420-   algorithm.  The algorithm provides a variable, G, that allows us
>to
>421-   control how the accuracy, in sessions per second, that we require
>of
>422-   the test.
>--
>--
>422-   the test.
>423-
>424-   After this candidate rate has been discovered, the test enters
>the
>425:/* Is N consistent with N in pseudocode? */
>426-   Steady State phase.  In the Steady State phase, N session
>Attempts
>427-   are made at the candidate rate.  The goal is to find a rate at
>which
>428-   the DUT can process calls "forever" with no errors and the test
>--
>--
>432-   the steady-state phase is entered again until a final (new)
>steady-
>433-   state rate is achieved.
>434-
>435:/* Would this process be clearer if presented as a list? */
>436-   The iterative process itself is defined as follows: A starting
>rate
>437-   of r = 100 sessions per second is used and we place calls at that
>438-   rate until n = 5000 calls have been placed.  If all n calls are
>--
>--
>436-   The iterative process itself is defined as follows: A starting
>rate
>437-   of r = 100 sessions per second is used and we place calls at that
>438-   rate until n = 5000 calls have been placed.  If all n calls are
>439:/* sps defined? This said, it's easy to figure out */
>440-   successful, the rate is increased to 150 sps and again we place
>calls
>441-   at that rate until n = 5000 calls have been placed.  The attempt
>rate
>442-   is continuously ramped up until a failure is encountered before n
>=
>--
>--
>449-   between the rate at which failures occurred and the last
>successful
>450-   rate.  Continuing in this way, an attempt rate without errors is
>451-   found.  The tester can specify a margin of error using the
>parameter G,
>452:/* units? */
>453-   measured in units of sessions per second.
>454-
>455-   The pseudo-code corresponding to the description above follows.
>--
>--
>478-      G  := 5      ; granularity of results - the margin of error in
>479-                   ; sps
>480-      C  := 0.05   ; calibration amount: How much to back down if we
>481:/* using "s" before definition, in my opinion; consider "found
>candidate rate
>482:s but cannot send at s" (Still not right, though ... */
>483-                   ; have found candidate s but cannot send at rate
>s
>484-                   ; for time T without failures
>485-
>--
>--
>487-      ; ---- Initialization of flags, candidate values and upper
>bounds
>488-
>489-      f  := false  ; indicates a success after the upper limit
>490:/* Capital F never used in pseudocode */
>491-      F  := false  ; indicates that test is done
>492-      c  := 0      ; indicates that we have found an upper limit
>493-
>--
>--
>499-                                   ; characteristics until n
>500-                                   ; requests have been sent
>501-             if (all requests succeeded) {
>502:/* undefined variable r'?  does this matter? */
>503-                r' := r ; save candidate value of metric
>504-                if ( c == 0 ) {
>--
>6.3 Session Establishment Rate with Media not on DUT
>--
>649-          be recorded using any pertinent parameters as shown in the
>650-          reporting format of Section 5.1.
>651-
>652:/* Long sentence in general, but minimally last part of sentence
>doesn't
>653:conclude */
>654-   Expected Results:  Session Establishment Rate results obtained
>with
>655-      Associated Media with any number of media streams per SIP
>session
>656-      are expected to be identical to the Session Establishment Rate
>
><end comments on draft-ietf-bmwg-sip-bench-meth-09>
>
>_______________________________________________
>bmwg mailing list
>[email protected]
>https://www.ietf.org/mailman/listinfo/bmwg