Re: draft-ietf-bmwg-virtual-net / CPU & memory utilization should be test configurations or test results?

Hi Saurabh,

Thanks for your question and for continuing this discussion.

In my mind, any server-oriented measurement that informs us
about resources consumed would be a fair result to collect.
But as we said during the meeting, Only as an Auxilliary metric
while other benchmarking is in-progress.

One of the challenges I mentioned adding to my draft is for
benchmarking metrics to assist deployment designers, or perhaps
to provide input for some form of resource model so that your
question about adding VNFs below might be answered.

We have a draft that begins to address the resource sharing
aspect of benchmarking here:
https://tools.ietf.org/html/draft-vsperf-bmwg-vswitch-opnfv-00
prepared by the OPNFV vsperf project, and Maryam Tahhan presented
additional material on this topic in slides (see the IETF-93 materails).

Sorry for the brief response and the delay responding, I've
been travelling all weekend and just arrived at another meeting. :(

regards,
Al

From: Saurabh Chattopadhyay - ERS, HCL Tech [mailto:[email protected]]
Sent: Friday, July 24, 2015 5:58 PM
To: [email protected]
Cc: [email protected]
Subject: draft-ietf-bmwg-virtual-net / CPU & memory utilization should be test configurations or test results?

Hello Al,

First, thank you for writing this draft. It is an excellent guideline for us who are working in VNF Benchmarking area.

In the BMWG meeting session yesterday, there were some interesting discussions around whether to consider the CPU / memory utilization (and similar parameters) as test configurations or test results. I think we couldn't discuss this in detail due to time constrain, thus I thought of bringing this up in the list to have your and other experts' views.

My own understanding is, for VNF benchmarking specifically, this issue becomes little delicate. Considering the fact that VNF Black-box is reliant on certain soft integration and soft partitioning of the underlying hardware, all benchmark results are dependent on the load imposed over the entire hardware, in addition to be dependent on the load directed towards the VNF specifically. For example, if a VNF is masked over four core, and the particular hardware provides 12 more core, VNF's response on a fixed load condition will vary while the other cores are put under different load conditions. Now at this point we can consider creating certain fixed type of load profiles (let's say, a combination % of compute, storage and networking load) over remaining hardware, and benchmark the VNF under Test against its own load conditions as originally planned. However, during the real deployment, these fixed type of load profiles (combination % of compute, storage and networking load) don't correlate well unless we qualify every VNF's performance profiles against these parameters. So this essentially means that even though a benchmark data is produced for the VNF for the particular type of hardware-part and for certain load conditions over remaining hardware, the deployment folks are not clear on how to leverage this intelligence especially when they are planning to deploy the other VNFs in the remaining hardware.

I'm not sure what will be an appropriate way to solve this. Measuring CPU / memory utilization (and similar parameters for the shared assets) may be an option, but not sure if this truly aligns with the black-box benchmarking methodologies. Kindly advice.

Warm Regards,
Saurabh



::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------

_______________________________________________
bmwg mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/bmwg