[ippm] Re: [bmwg] Re: Introducing: AI Fabric Benchma rking Methodology Suite (Training, Inference, Terminology )

Hi Kehan,

This is a good question, not only for this specific topic.

I see that BMWG developed in the past some work on technologies developed by other organization (virtualization, for example).

The only part I found relevant in the charter is the following:

“When possible, the
benchmarks and other terminologies will be developed jointly with
organizations that are willing to share their expertise.”

This is the kind of clarification we can include in future rechartering (either as part of the joint future with IPPM or separately).

Cheers,
Med

De : kehan yao <[email protected]>
Envoyé : mardi 3 mars 2026 16:59
À : BOUCADAIR Mohamed INNOV/NET <[email protected]>
Cc : Fernando Calabria (fcalabri) <[email protected]>; [email protected]; [email protected]
Objet : Re: [bmwg] Re: Introducing: AI Fabric Benchmarking Methodology Suite (Training, Inference, Terminology)


Hi Med, all,

Just a clarification question.
Can IETF standardize or publish benchmarking methodologies specifications which describe techniques designed outside of IETF? For example, RoCEv2, UET, etc.

Overall, I think the document is very useful and hit hot topics.

Best regards,
Kehan

<[email protected]<mailto:[email protected]>> 于2026年3月3日周二 23:50写道：
Hi Fernando,

Thank you for sharing this.

I had troubles to find the drafts by the names indicated below. I guess the correct files are the following:


  *   https://datatracker.ietf.org/doc/draft-calabria-bmwg-ai-fabric-terminology/
  *   https://datatracker.ietf.org/doc/draft-calabria-bmwg-ai-fabric-training-bench/
  *   https://datatracker.ietf.org/doc/draft-calabria-bmwg-ai-fabric-inference-bench/

Cheers,
Med

De : Fernando Calabria (fcalabri) <[email protected]<mailto:[email protected]>>
Envoyé : vendredi 27 février 2026 15:14
À : [email protected]<mailto:[email protected]>
Objet : [bmwg] Introducing: AI Fabric Benchmarking Methodology Suite (Training, Inference, Terminology)



Dear BMWG Participants,

We are pleased to introduce three companion individual submissions addressing a benchmarking methodology gap for AI/ML network fabrics:

  (1) draft-calabria-pignataro-bmwg-ai-fabric-terms-00
      Terminology for AI Fabric Benchmarking

  (2) draft-calabria-pignataro-bmwg-ai-fabric-training-bench-00
      Benchmarking Methodology for AI Training Fabric Networks

  (3) draft-calabria-pignataro-bmwg-ai-fabric-inference-bench-00
      Benchmarking Methodology for AI Inference Fabric Networks

WHY AI FABRICS REQUIRE NEW BMWG WORK

AI/ML workloads impose network behaviors with no analog in existing BMWG methodology documents:

  Training: Bulk synchronous collective operations (AllReduce, AllGather, ReduceScatter) over RoCEv2 fabrics create synchronized incast-like traffic bursts, making tail latency and congestion management (PFC/ECN/DCQCN) the dominant performance determinants — not throughput or packet loss in isolation.

  Inference: Disaggregated prefill/decode architectures and Mixture-of-Experts (MoE) routing generate highly asymmetric, bursty point-to-point KV cache transfer patterns. SLA-relevant KPIs are Time to First Token (TTFT) and Inter-Token Latency (ITL), which are jointly determined by compute and fabric behavior in ways that require careful test isolation methodology.

DOCUMENT STRUCTURE

The three documents follow established BMWG convention: a terminology companion paired with separate methodology documents for the two principal AI workload classes. All documents are scoped to controlled laboratory environments, maintain strict vendor neutrality, and define no acceptance thresholds.

We welcome Working Group review and would appreciate Chair guidance on presentation opportunities.
We are also interested in coordinating with authors of draft-gaikwad-llm-benchmarking-methodology and any related BMWG efforts.

Thank you for your time and consideration.


Fernando Calabria (Cisco) ,  Carlos Pignataro (Blue Fern Consulting) , Giuseppe  Fioccola and Qin Wu (Huawei)


____________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc

pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler

a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,

Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.



This message and its attachments may contain confidential or privileged information that may be protected by law;

they should not be distributed, used or copied without authorisation.

If you have received this email in error, please notify the sender and delete this message and its attachments.

As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.

Thank you.
_______________________________________________
bmwg mailing list -- [email protected]<mailto:[email protected]>
To unsubscribe send an email to [email protected]<mailto:[email protected]>
____________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

_______________________________________________
ippm mailing list -- [email protected]
To unsubscribe send an email to [email protected]