|
GROUP SPECIFICATION
Network Functions Virtualisation (NFV) Release 3;
Testing;
Specification of Networking Benchmarks and
Measurement Methods for NFVI
Disclaimer
The present document has been produced and approved by the Network Functions Virtualisation (NFV) ETSI Industry
Specification Group (ISG) and represents the views of those members who participated in this ISG.
It does not necessarily represent the views of the entire ETSI membership.
---------------------- Page: 1 ----------------------
2 ETSI GS NFV-TST 009 V3.1.1 (2018-10)
Reference
DGS/NFV-TST009
Keywords
benchmarking, measurement, NFV, NFVI
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE
Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16
Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88
Important notice
The present document can be downloaded from:
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the only prevailing document is the
print of the Portable Document Format (PDF) version kept on a specific network drive within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
If you find errors in the present document, please send your comment to one of the following services:
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.
© ETSI 2018.
All rights reserved.
TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are trademarks of ETSI registered for the benefit of its Members.
TM TM
3GPP and LTE are trademarks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
oneM2M logo is protected for the benefit of its Members.
GSM and the GSM logo are trademarks registered and owned by the GSM Association.
ETSI
---------------------- Page: 2 ----------------------
3 ETSI GS NFV-TST 009 V3.1.1 (2018-10)
Contents
Intellectual Property Rights . 5
Foreword . 5
Modal verbs terminology . 5
Executive summary . 5
Introduction . 5
1 Scope . 6
2 References . 6
2.1 Normative references . 6
2.2 Informative references . 6
3 Definitions of terms and abbreviations. 8
3.1 Terms . 8
3.2 Abbreviations . 9
4 Time and Time Intervals for Metrics and Benchmarks . 10
5 Framework for Metric and Benchmark Definitions . 10
6 Test Set-ups and Configuration . 11
6.1 Goals of Benchmarking and Use Cases . 11
6.2 Test Setups . 11
6.3 Configuration . 14
7 Test Device/Function Capabilities . 15
7.1 Traffic Generator Requirements . 15
7.2 Traffic Receiver Requirements . 16
7.3 Test Device/Function Requirements . 16
8 Throughput . 17
8.1 Background . 17
8.2 Name . 17
8.3 Parameters . 17
8.4 Scope . 17
8.5 Units of Measure . 17
8.6 Definition . 18
8.7 Method of Measurement . 18
8.8 Sources of Error . 18
8.9 Discussion . 18
8.10 Reporting Format . 18
9 Latency . 19
9.1 Background . 19
9.2 Name . 19
9.3 Parameters . 19
9.4 Scope . 20
9.5 Units of Measure . 20
9.6 Definition . 20
9.7 Method of Measurement . 20
9.8 Sources of Error . 20
9.9 Discussion . 21
9.10 Reporting Format . 21
10 Delay Variation . 21
10.1 Background . 21
10.2 Name . 22
10.3 Parameters . 22
10.4 Scope . 22
ETSI
---------------------- Page: 3 ----------------------
4 ETSI GS NFV-TST 009 V3.1.1 (2018-10)
10.5 Units of Measure . 22
10.6 Definition . 23
10.7 Method of Measurement . 23
10.8 Sources of Error . 23
10.9 Discussion . 23
10.10 Reporting Format . 24
11 Loss . 24
11.1 Background . 24
11.2 Name . 25
11.3 Parameters . 25
11.4 Scope . 25
11.5 Units of Measure . 25
11.6 Definition . 25
11.7 Method of Measurement . 26
11.8 Sources of Error . 26
11.9 Discussion . 27
11.10 Reporting Format . 27
12 Methods of Measurement . 27
12.1 Pre-Test and Measurement Procedure . 27
12.2 Core Procedures . 28
12.3 Search Algorithms . 31
12.3.1 Introduction. 31
12.3.2 Binary search . 32
12.3.3 Binary search with loss verification . 33
12.3.4 Binary search with NDR and PDR Loss Thresholds . 35
12.4 Long Duration Testing . 36
13 Follow-on Activities . 36
Annex A (informative): Survey of Current Benchmarking Campaigns . 37
A.1 Overall Summary of Key Issues and Points of Learning . 37
A.1.1 Introduction . 37
A.1.2 Test Conditions with Non-zero Packet Loss . 37
A.1.3 Repeatable Results Depend on many Config. Variables . 37
A.1.4 Generation of Multiple Streams . 38
A.1.5 Test Stream Variations over Size and Protocol . 38
A.1.6 Testing during dynamic flow establishment . 39
A.1.7 Monitor Operational Infrastructure Metrics During Tests . 39
Annex B (informative): Development of New Search Strategies. 40
B.1 Mitigating background processes that cause infrequent loss . 40
Annex C (informative): Authors & contributors . 43
History . 44
ETSI
---------------------- Page: 4 ----------------------
5 ETSI GS NFV-TST 009 V3.1.1 (2018-10)
Intellectual Property Rights
Essential patents
IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (https://ipr.etsi.org/).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Trademarks
The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.
ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no
right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does
not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
Foreword
This Group Specification (GS) has been produced by ETSI Industry Specification Group (ISG) Network Functions
Virtualisation (NFV).
Modal verbs terminology
In the present document "shall", "shall not", "should", "should not", "may", "need not", "will", "will not", "can" and
"cannot" are to be interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of
provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
Executive summary
This unnumbered clause, if present, appears after the "Modal verbs terminology" and before the "Introduction". It is an
optional informative element and shall not contain requirements.
The "Executive summary" is used, if required, to summarize the ETSI deliverable. It contains enough information for
the readers to become acquainted with the full document without reading it. It is usually one page or shorter.
Introduction
The widespread adoption of virtualised implementation of functions has brought about many changes and challenges for
the testing and benchmarking industries. The subjects of the tests perform their functions within a virtualisation system
for additional convenience and flexibility, but virtualised implementations also bring challenges to measure their
performance in a reliable and repeatable way, now that the natural boundaries and dedicated connectivity of physical
network functions are gone. Even the hardware testing systems have virtualised counterparts, presenting additional
factors to consider in the pursuit of accurate results.
The present document draws on learnings from many early benchmarking campaigns and years of benchmarking
physical network functions to develop and specify new normative benchmarks and methods of measurement to
characterize the performance of networks in the Network Function Virtualisation Infrastructure.
ETSI
---------------------- Page: 5 ----------------------
6 ETSI GS NFV-TST 009 V3.1.1 (2018-10)
1 Scope
The present document specifies vendor-agnostic definitions of performance metrics and the associated methods of
measurement for Benchmarking networks supported in the NFVI. The Benchmarks and Methods will take into account
the communication-affecting aspects of the compute/networking/virtualisation environment (such as the transient
interrupts that block other processes or the ability to dedicate variable amounts of resources to communication
processes). These Benchmarks are intended to serve as a basis for fair comparison of different implementations of
NFVI, (composed of various hardware and software components) according to each individual Benchmark and
networking configuration evaluated. Note that a Virtual Infrastructure Manager (VIM) may play a supporting role in
configuring the network under test. Examples of existing Benchmarks include IETF RFC 2544 [1] Throughput and
Latency (developed for physical network functions).
2 References
2.1 Normative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
Referenced documents which are not found to be publicly available in the expected location might be found at
https://docbox.etsi.org/Reference.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are necessary for the application of the present document.
[1] IETF RFC 2544 (March 1999): "Benchmarking Methodology for Network Interconnect Devices".
[2] IETF RFC 2285 (February 1998): "Benchmarking Terminology for LAN Switching Devices".
[3] IETF RFC 2889 (August 2000): "Benchmarking Methodology for LAN Switching Devices".
[4] IETF RFC 6985 (July 2013): "IMIX Genome: Specification of Variable Packet Sizes for
Additional Testing".
[5] ETSI GS NFV-TST 008 (V3.1.1): "Network Functions Virtualisation (NFV) Release 2; Testing;
NFVI Compute and Network Metrics Specification".
[6] ETSI GS NFV 003 (V1.3.1): "Network Functions Virtualisation (NFV); Terminology for Main
Concepts in NFV".
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ETSI GS NFV-INF 003 (V.1.1.1): "Network Functions Virtualisation (NFV); Infrastructure;
Compute Domain".
ETSI
---------------------- Page: 6 ----------------------
7 ETSI GS NFV-TST 009 V3.1.1 (2018-10)
[i.2] Bregni, Stefano, (October 1996): "Measurement of Maximum Time Interval Error for
Telecommunications Clock Stability Characterization", IEEE Transactions on Instrumentation and
Measurement.
NOTE: Available at http://home.deib.polimi.it/bregni/papers/mtiemeas.pdf.
[i.3] ETSI GS NFV-TST 001 (V1.1.1): "Network Functions Virtualisation (NFV);
Pre-deployment Testing; Report on Validation of NFV Environments and Services".
[i.4] FD.io VPP Developer Documentation: "Shared Memory Packet Interface (memif) Library".
NOTE: Available at https://docs.fd.io/vpp/17.10/libmemif_doc.html.
[i.5] FD.io VPP pma-tools: "Software tools for performance and efficiency measurements".
NOTE: Available at https://git.fd.io/pma_tools/tree/jitter.
[i.6] John D. McCalpin: "STREAM: Sustainable Memory Bandwidth in High Performance
Computers".
NOTE: Available at https://www.cs.virginia.edu/stream/.
[i.7] Intel Software Developer Zone: "Intel Memory Latency Checker v3.5".
NOTE: Available at https://software.intel.com/en-us/articles/intelr-memory-latency-checker.
[i.8] IETF RFC 8172 (July 2017): "Considerations for Benchmarking Virtual Network Functions and
Their Infrastructure".
[i.9] IETF RFC 8204 (July 2017): "Benchmarking Virtual Switches in the Open Platform for NFV
(OPNFV)".
[i.10] Cooper, T., Morton, A., Rao, S., OPNFV Summit Presentation: "Dataplane Performance,
Capacity, and Benchmarking in OPNFV", June 2017.
NOTE: Available at https://wiki.opnfv.org/download/attachments/10293193/VSPERF-Dataplane-Perf-Cap-
Bench.pptx?api=v2.
[i.11] IETF RFC 5481 (March 2009): "Packet Delay Variation Applicability Statement".
[i.12] Wikipedia: "Binary Search".
NOTE: Available at https://en.wikipedia.org/wiki/Binary_search_algorithm.
[i.13] GeeksforGeeks Computer Science Portal: "Searching Algorithms, Binary Search".
NOTE: Available at https://www.geeksforgeeks.org/binary-search/ (including code implementations).
[i.14] Andrzej Pelc: "Searching games with errors - fifty years of coping with liars", Theoretical
Computer Science 270 (2002) pp 71-109.
NOTE: Available at https://www.gwern.net/docs/statistics/comparison/2002-pelc.pdf.
[i.15] Presentation to IETF-102 BMWG Session: "Evolution of Repeatability in Benchmarking: Fraser
Plugfest (Summary for IETF BMWG)", Sridhar Rao, Al Morton, and OPNFV VSPERF Project
Team.
NOTE: Available at https://datatracker.ietf.org/meeting/102/materials/slides-102-bmwg-evolution-of-
repeatability-in-benchmarking-fraser-plugfest-summary-for-ietf-bmwg-00.
[i.16] LightReading/EANTC NFV Tests and Trials: "Validating Cisco's NFV Infrastructure",
October 2015.
NOTE: Available at http://www.lightreading.com/nfv/nfv-tests-and-trials/validating-ciscos-nfv-infrastructure-pt-
1/d/d-id/718684.
ETSI
---------------------- Page: 7 ----------------------
8 ETSI GS NFV-TST 009 V3.1.1 (2018-10)
[i.17] Wikipedia: "OPNFV NFVbench Project Description".
NOTE: Available at https://wiki.opnfv.org/display/nfvbench.
[i.18] Python Package Index MLResearch 0.1.1.0: "Library for speeding up binary search using shorter
measurements".
NOTE: Available at https://pypi.org/project/MLRsearch/ and
https://docs.fd.io/csit/master/report/introduction/methodology.html#mlrsearch-algorithm.
[i.19] Recommendation ITU-T Y.1541 (December 2011): "Network performance objectives for IP-based
services".
[i.20] IETF RFC 8337 (March 2018): "Model-based Metrics".
[i.21] IETF RFC 7348: "Virtual eXtensible Local Area Network (VXLAN): A Framework for
Overlaying Virtualised Layer 2 Networks over Layer 3 Networks".
3 Definitions of terms and abbreviations
3.1 Terms
For the purposes of the present document, the terms given in ETSI GS NFV 003 [6] and the following apply:
NOTE: A term defined in the present document takes precedence over the definition of the same term, if any, in
ETSI GS NFV 003 [6].
burst: stream of two or more frames transmitted with the minimum allowable inter-frame gap, as in a burst of frames
bursty traffic rate: stream consisting of repeated bursts that maintains a specified frequency of transmitted frames per
second, such that the frequency equals the reciprocal of the sum of the constant inter-burst gap and the burst
serialization time (for a constant frame size and minimum allowable inter-frame gaps between frames in the burst)
NOTE: See section 21 of IETF RFC 2544 [1].
constant frame rate stream: stream that maintains a specified frequency of transmitted frames per second, such that
the frequency equals the reciprocal of the sum of the constant inter-frame gap and the frame serialization time (for a
constant frame size)
flow: set of frames or packets with the same n-tuple of designated header fields that (when held constant) result in
identical treatment in a multi-path decision (such as the decision taken in load balancing)
frame size: fixed length of a frame in octets (or 8-bit bytes), all headers included
NOTE: For example, Ethernet frame size includes the frame CRC, but exclude the transmission overhead per
frame of 20 octets (the preamble and the inter frame gap).
measurement goal: specific criteria that a measurement result is expected to meet to satisfy the requirements of a
benchmark definition
method: series of one or more Sets of Tests conducted to achieve a measurement goal
offered load: both the count (in frames) and transmission rate (in frames per second) generated by the measurement
system during a trial, including both directions of transmission with bi-directional streams
pod: partition of a compute node that provides an isolated virtualised computation environment, for one or more
virtualisation containers in the context of an Operating System Container virtualisation layer
set: series of one or more tests conducted to achieve a measurement goal
stream: population of frames or packets with various header attributes, that contain one or more flows and
...