Re: Fwd: Review: draft-ietf-bmwg-sip-bench-term-10 and draft-ietf-bmwg-sip-bench-meth-10

Robert,

Thank you very much for you careful reviews.  Your question and comments
have greatly helped us to restructure and cleanup the documents. The
versions 11 of the documents are now available.  A summary of changes is
below. Details of the resolution of each item in the review are in­line in
our reply.

Summary of changes:
*Made it explicit that we do not intend for the EA server to add delay
before it initiates its 200 response. (M)
*Clarified the need for distinct AoR¹s in the REGISTER tests (M)
*Clarified SIP Transport Protocol issues (M)
*Added a section called Connection-oriented Transport Management to address
the question of the number of such connections to be used. (M)
*Added rows in the Test Setup Report for Connection-oriented Transport as
well as for Codec Type (M)
*Clarified use of the parameter, Œmedia packet size¹ is for audio codecs.
(T)

*Removed references to IS/NS.  (M), (T)
*Removed references to session(sig,medc,med) and the associated diagram. (T)
*Added a new figure and explanatory material for registration tests,
including Œexpected results.' (M)
*Removed reference to RTSP

*Cleaned up typos and other nits.

See detailed resolutions in-line below.

Best Regards,

Carol Davids 
Vijay Gurbani 


From:  Robert Sparks <[email protected]>
Date:  Wednesday, June 18, 2014 at 4:26 PM
To:  BMWG mailing list BMWG mailing list <[email protected]>
Subject:  [bmwg] Fwd: Review: draft-ietf-bmwg-sip-bench-term-10 and
draft-ietf-bmwg-sip-bench-meth-10

    
 Oops - meant to copy the list. Forwarding...
 

 
 -------- Original Message --------
 Subject:  Review: draft-ietf-bmwg-sip-bench-term-10 and
draft-ietf-bmwg-sip-bench-meth-10
 Date:  Wed, 18 Jun 2014 16:19:06 -0500
 From:  Robert Sparks <[email protected]> <mailto:[email protected]>
 To:  MORTON, ALFRED C (AL) <[email protected]> <mailto:[email protected]> ,
Vijay Gurbani <[email protected]> <mailto:[email protected]> , joel jaeggli
<[email protected]> <mailto:[email protected]> , Carol Davids ([email protected])
<[email protected]> <mailto:[email protected]> , [email protected]
<[email protected]> <mailto:[email protected]>
 CC:  Banks, Sarah ([email protected]) <[email protected]>
<mailto:[email protected]>
 
 
Reviews of draft-ietf-bmwg-sip-bench-term-10 and
draft-ietf-bmwg-sip-bench-meth-10

Thank you for the restructure and cleanup work on these drafts.

Thank you for your help with this.


Most of the comments I made on version -08 of each of them have been
addressed.
Since there were so many changes, rather than revisit the thread from
the -08 review,
I'll start a new thread here, bringing up unaddressed points from the
earlier review
as necessary.

Again, thank you for the adjustment to the title and the text to address
the scope
of the documents. The result is much more straightforward and clear than
what I
had earlier reviewed.

I have a few remaining points and questions and a few nits to call out.

Major technical points:

1) I've read through this enough times now that maybe I've become blind
to where
it's discussed, but for the INVITE tests, I think you have an unstated
assumption
that the responding EA is configured to send 200s as quickly as it can,
and that
whatever delay it has in responding is fairly constant. Otherwise, your
tests
will have widely varying results due to retransmissions of the INVITE.
Is it your
intent that the EA would ever retransmit? If not, the assumptions above
should
be spelled out in the methodology document.

Added new text  in Methodology Section 4.9.


2) The documents still don't make it clear that each register request needs
to be to a distinct AOR (otherwise, there is no sense in having a separate
re-registration test).

Added text in Methodology Sections 6.7 and 6.8.

3) The documents (particularly the report forms) assume you will use the
same transport on both sides of the DUT. Please state that explicitly, or
if allowing for (for instance) UDP on one side of a proxy and TCP on the
other was intended, please adjust the document to talk about it.

Added text in Methodology Section 5.1 under the SIP Transport Protocol line.
4) I think the documents are assuming that an EA will make one connection
if it's using a connection oriented transport (like TLS). The results of
the test will be very different if it opens a new connection for each
sent message. Similarly, I think you're assuming that a connection
gets set up between the DUT and the EA that will respond to an INVITE
once. Those should be called out explicitly. If you're not assuming
that, there needs to be more discussion about how connections get
established, and whether that's a parameter that needs to be captured
as part of the test.

Added a new section, 4.2 in Methodology, called Connection-oriented
Transport Management.

Added corresponding line in Methodology, Section 5.1, called Connection
Management Strategy.


5) Is it the intent to only test with media that acts like G.711 over
RTP? The configuration parameters you have for EA imply that. If its
not the case, are you missing configuration parameters for, say, video
codecs (an EA won't be able to use what you have in terminology's 3.3.4
for that stream), or for an msrp media session?

Added Section 4.7 in Methodology, entitled ŒCodec Type.¹

Added text to Section 3.3.4 Definition and Discussion, in Terminology.


Major editorial points:

1) The terms IS and NS are holdovers from when the document was trying to
do more than it is now. NS is not accurately defined and is at cross
purposes with the tests you _do_ define in the document (The discussion
of MESSAGE and SUBSCRIBE that's still in 3.1.1 of the terminology document
does not help this document at all.) I know these words took effort to
write,
but they really should just be removed. A very short introduction to the two
types of session you are actually testing would leave much less room for
confusion.

Removed reference to IS and NS.  Replaced those terms by 'invitation to
audio/video communication' and Œregistration¹ respectively.


2) I challenged the usefulness of the definition of session
(sig,medc,med) and
the diagram trying to show these as points in some three-dimensional
space in
my previous review.  I did see Carol's explanation of it in the summary
to that
review, but I disagree that it is helping this document. I think it's
hurting
by adding confusion.  If you deleted it, the rest of the document means
what
it meant before, and the reader is no less informed.  Please remove it.
If the primary point is to make sure that the tester considers covering
every
permutation of the test parameters, just say that. The graphic implies that
some things have more sess.sig than others, and that you might talk about
the distance between points in this vector space. Save all this aside and
bring it back in a document that actually uses it if you must, but please
take it out of this one.

Agreed.  See changes in Methodology Section 3.1.1.



3) The code in Appendix A of the methodology doc should be identified as
a "Code Component" as described in the TLP
(http://trustee.ietf.org/license-info/IETF-TLP-4.htm)
and a license block should be added.

Added to Appendix A.


More minor points.

* The methodology document says the DUT can be any conforming 3261 device.
The terminology document says the device can't be user equipment (see
bullet 1
in section 1.1) . Neither are correct. (A presence server, for instance, is
not a reasonable thing for a DUT for the tests in this document). I
suggest in
both documents you just explicitly list what the DUT can be. That list
should
not include "User Agent Server" as it currently does in 3.2.2 of the
terminology
document. A UAS is just a role that any of these devices, including
end-user
terminals, can hold.

Agreed. 
Changes made in Terminology, Section 2.1 bullets 2 and 3; and Section 3.2.2,
Discussion. 
Changes made in Methodology, Section 2, the paragraph beginning,"The
device-under-test (DUT) is.."
 


* If a Registrar is the DUT, then neither of the topology figures are
correct.
(Unless you're intending for an EA to be the registrar, and the DUT is just
a proxy). Consider adding a figure that more clearly shows what the topology
for your registration test is intended to be, and making it clear that you
are only testing INVITE through devices that forward messages.
Changes made in Methodology Section 3, Benchmarking Topology: The new
figure, Figure 3,  and explanatory material were added.

* There are several places that call out RTSP as a media protocol.
RTSP is a control protocol, not a media protocol - it doesn't make sense
being listed with RTP and SRTP. MSRP would make more sense.

Agreed. Reference to RTSP was removed.


* The Expected Results in Section 6.8 of the methodology claim that
the rate should not be more than what was in 6.7. How do you come to
that conclusion? There are several valid implementation choices that
could lead to reregistration taking slightly longer than an initial
registration.

See changes in Expected Results in Methodology Section 6.8, as well as
clarification of the procedure and reference to the new Figure 3.

NITS:

Methodology document:

Introduction paragraph 1 s/in Terminology document/in the Terminology
Document/
done
Introduction paragraph 5. This points to section 4 (the IANA
consideration section)
and section 2 (the introduction) of the terminology document for an
explanation
of configuration options. Neither of those sections explain
configuration options.
Where did you mean to point?

Deleted reference.

Terminology document:

Security Considerations: Please remove "and various other drafts". If
you know of
other important documents to point to, ase add them as references.

Removed
The definition of Stateful Proxy and Stateless Proxy copied the words
"defined
by this specification" from RFC3261. This literal copy introduces ambiguity.
Please replace "by this specification" with "by [RFC3261]".

Done


Introduction paragraph 4, last sentence: By calling out devices that include
the UAS and UAC functions, you have eliminated stateless proxies, which
contain
neither.

We excluded stateless proxies from the scope of the documents, and explained
that in Section 2,  paragraph 2 (Introduction) to the Methodology document.


In section 3, when you use the templates, you are careful to say None under
issues when there are no issues. Please use the same care with See Also.
Right now, you have empty See Also: sections that could be misread to
take up whatever content follows (particularly by a text-to-speech engine).

Added Œnone' as needed.


3.1.6: s/tie interval/time interval/

Done



 
 
 
 
_______________________________________________ bmwg mailing list
[email protected] https://www.ietf.org/mailman/listinfo/bmwg

_______________________________________________
bmwg mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/bmwg