Re: Adam Roach's No Objection on draft-ietf-bmwg-vswitch-opnfv-03: (with COMMENT)

Thanks. Your proposed clarification looks good to me. 

/a

> On Jun 7, 2017, at 18:20, MORTON, ALFRED C (AL) <[email protected]> wrote:
> 
> Hi Adam,
> Thanks for your review, please see reply below.
> Al (for the co-authors)
> 
>> -----Original Message-----
>> From: Adam Roach [mailto:[email protected]]
>> Sent: Wednesday, June 07, 2017 6:36 PM
>> To: The IESG
>> Cc: [email protected]; Sarah Banks; bmwg-
>> [email protected]; [email protected]; [email protected]
>> Subject: Adam Roach's No Objection on draft-ietf-bmwg-vswitch-opnfv-03:
>> (with COMMENT)
>> 
>> Adam Roach has entered the following ballot position for
>> draft-ietf-bmwg-vswitch-opnfv-03: No Objection
>> 
> ...
>> 
>> ----------------------------------------------------------------------
>> COMMENT:
>> ----------------------------------------------------------------------
>> 
>> I think this is a variation on the issue that Alissa calls out, but I'm
>> having a hard time reconciling:
>> 
>>   It's unlikely that the virtual switch will be the only application
>>   running on the System Under Test (SUT), so CPU utilization, Cache
>>   utilization, and Memory footprint should also be recorded for the
>>   virtual implementations of internetworking functions.
>> 
>> ...with...
>> 
>>   Further, benchmarking is performed on a "black-box" basis, relying
>>   solely on measurements observable external to the DUT/SUT.
>> 
>> Please add text that clarifies how the metrics that 3.1 says should be recorded
>> in section 3.1 relate to benchmarking.
>> 
> [ACM] 
> Ok, this is a somewhat different issue.
> There are metrics like CPU, Cache, and Memory Utilization
> which are reported by measurement processes running on the 
> Device Under Test. These are useful for the user/operations community
> to know, and may also be useful if there is a problem encountered 
> during testing. However, these metrics cannot be promoted to
> "Benchmark" status, because we cannot measure any of them 
> external to the DUT or SUT.  In fact, only the most important 
> externally observable metrics are designated as Benchmarks
> in our consensus process, and BMWG refers to internally measured
> metrics as "white-box metrics" to clearly distinguish them
> from the others.
> 
> So, I suggest adding the new sentence below:
>>   It's unlikely that the virtual switch will be the only application
>>   running on the System Under Test (SUT), so CPU utilization, Cache
>>   utilization, and Memory footprint should also be recorded for the
>>   virtual implementations of internetworking functions.
> +    However, internally-measured metrics such as these are not
> +    benchmarks; they should be useful for the audience (operations)
> +    to know, and may also be useful if there is a problem encountered 
> +    during testing.
> 
>