|
TECHNICAL REPORT
Speech and Multimedia Transmission Quality (STQ);
Guidelines on OTT Video Streaming;
Service Quality Evaluation Procedures
---------------------- Page: 1 ----------------------
2 ETSI TR 103 488 V1.1.1 (2019-01)
Reference
DTR/STQ-00215m
Keywords
3G, data, GSM, network, QoE, QoS, service,
video
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE
Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16
Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88
Important notice
The present document can be downloaded from:
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the prevailing version of an ETSI
deliverable is the one made publicly available in PDF format at www.etsi.org/deliver.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
If you find errors in the present document, please send your comment to one of the following services:
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.
© ETSI 2019.
All rights reserved.
TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are trademarks of ETSI registered for the benefit of its Members.
TM TM
3GPP and LTE are trademarks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
oneM2M™ logo is a trademark of ETSI registered for the benefit of its Members and
of the oneM2M Partners.
GSM and the GSM logo are trademarks registered and owned by the GSM Association.
ETSI
---------------------- Page: 2 ----------------------
3 ETSI TR 103 488 V1.1.1 (2019-01)
Contents
Intellectual Property Rights . 4
Foreword . 4
Modal verbs terminology . 4
Executive summary . 4
Introduction . 4
1 Scope . 6
2 References . 6
2.1 Normative references . 6
2.2 Informative references . 6
3 Definition of terms, symbols and abbreviations . 7
3.1 Terms . 7
3.2 Symbols . 7
3.3 Abbreviations . 7
4 Passive, non-intrusive network monitoring of SSL/QUIC OTT video services bitstreams . 8
5 Categories of OTT video streaming session QoS parameters . 8
5.1 Introduction . 8
5.2 Triggers of main QoS parameters . 10
5.2.1 Session . 10
5.2.2 Video Access Time and Pre-playout buffering time . 10
5.3 Parameters relevance for an effective Root Cause Analysis . 10
5.3.1 Transport parameters relevance . 10
5.3.2 QoE-Video/Audio parameters relevance . 11
5.3.3 Streaming session parameters relevance . 11
5.3.4 Service centric parameters relevance . 11
6 Thresholds for OTT video streaming quality . 11
7 Figure of Merit of the OTT video streaming service . 12
7.1 Introduction . 12
7.2 Calculation of Figure of Merit . 13
7.3 Quality thresholds used for FoM calculation . 14
7.4 Weightings . 14
7.5 Video streaming sessions' quality classification . 14
8 Reporting . 15
Annex A: Example of FoM calculation . 16
Annex B: Bibliography . 18
History . 19
ETSI
---------------------- Page: 3 ----------------------
4 ETSI TR 103 488 V1.1.1 (2019-01)
Intellectual Property Rights
Essential patents
IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (https://ipr.etsi.org/).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Trademarks
The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.
ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no
right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does
not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
Foreword
This Technical Report (TR) has been produced by ETSI Technical Committee Speech and multimedia Transmission
Quality (STQ).
Modal verbs terminology
In the present document "should", "should not", "may", "need not", "will", "will not", "can" and "cannot" are to be
interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
Executive summary
The fast increase of the variety, technology complexity, and dynamic changes of the OTT video streaming services'
delivery, as well as their consumptions by the users on a wide range of smart devices, suggests a consistent and robust
testing approach that can allow an application transparent and meaningful evaluation of these services' quality
management and control. The present document offers guidance for such a testing approach, while aligning the content
with all other ETSI WG STQ Mobile documents as well as work related to the topic ongoing in other organizations.
Introduction
Several published STQM TSs and TRs cover the area of the video streaming end to end QoS/QoE evaluation. ETSI
TS 102 250-2 [i.1] covers video streaming session performance evaluation in the case of real time streaming (e.g. RTP,
RTSP, RTCP). ETSI TR 101 578 [i.2] complements ETSI TS 102 250-2 [i.1] with QoS/QoE for the specific case of
TCP based streaming (YouTube™ application), which is the de-facto delivery technique for the OTT video streaming
applications. Last, but not least, ETSI TR 102 493 [i.3] provides guidance on available QoE algorithms that can be used
in various video streaming testing scenarios.
ETSI
---------------------- Page: 4 ----------------------
5 ETSI TR 103 488 V1.1.1 (2019-01)
However, the OTT video streaming services' arena has seen video technology evolution (e.g. H.265 codec, 1440
resolution, new protocols such as QUIC), as well as dynamic technology changes (e.g. encryption schemes, encoding
schemes of various profiles, adaptations schemes at the server and/or client side within the context of various throttling
techniques and policies). Last, but not least, the variety of devices, namely operation system based video clients, do see
the same dynamic change. Therefore, guidance for a transparent and flexible testing approach is required by today's
OTT video streaming sessions' quality evaluation. And, consequently, a set of the most meaningful QoE centric QoS
parameters is also necessary, which can complement MOS estimators (QoE algorithms/models; Recommendations
ITU-T J series [i.4] to [i.9], P series [i.10] and [i.11]) whenever these are available, but also go beyond a single quality
score.
ETSI
---------------------- Page: 5 ----------------------
6 ETSI TR 103 488 V1.1.1 (2019-01)
1 Scope
The present document's scope is to provide guidance on OTT video streaming testing approach with a set of minimum
desired and most meaningful QoE centric QoS parameters along with recommendations to create a figure of merit
quantifying the OTT video streaming session quality, where possible. In addition, the set of introduced QoE centric
QoS parameters aim to help with the identification of the possible roots of video quality degradation. The present
document also offers means to understand aspects related with network and services optimization and troubleshooting,
such as the trade-off between bandwidth usage or controlled throttling and end-to-end video quality.
The scope of the present document complements ETSI TS 102 250-2 [i.1] and ETSI TR 101 578 [i.2] while not being as
exhaustive, but rather focused on QoE centric characterization and an end-to-end view on the video streaming session
as a whole. Furthermore, the present document takes into consideration QoE centric evaluation by means of passive,
non-intrusive network monitoring of SSL/QUIC OTT Video Services bitstreams. In addition, the present document
aims to complement the scope of ETSI TR 102 493 [i.3] with respect to QoE models for video streaming integrity as
perceived by users.
2 References
2.1 Normative references
Normative references are not applicable in the present document.
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ETSI TS 102 250-2: "Speech and multimedia Transmission Quality (STQ); QoS aspects for
popular services in mobile networks; Part 2: Definition of Quality of Service parameters and their
computation".
[i.2] ETSI TR 101 578: "Speech and multimedia Transmission Quality (STQ); QoS aspects of TCP-
based video services like YouTube™".
[i.3] ETSI TR 102 493: "Speech and multimedia Transmission Quality (STQ); Guidelines for the use of
Video Quality Algorithms for Mobile Applications".
[i.4] Recommendation ITU-T P.1201: "Parametric non-intrusive assessment of audiovisual media
streaming quality".
[i.5] Recommendation ITU-T P.1202: "Parametric non-intrusive bitstream assessment of video media
streaming quality".
[i.6] Recommendation ITU-T P.1203: "Parametric bitstream-based quality assessment of progressive
download and adaptive audiovisual streaming services over reliable transport".
[i.7] Recommendation ITU-T P.343: "Hybrid perceptual bitstream models for objective video quality
measurements".
[i.8] Recommendation ITU-T P.341: "Objective perceptual multimedia video quality measurement of
HDTV for digital cable television in the presence of a full reference".
ETSI
---------------------- Page: 6 ----------------------
7 ETSI TR 103 488 V1.1.1 (2019-01)
[i.9] Recommendation ITU-T P.342: "Objective multimedia video quality measurement of HDTV for
digital cable television in the presence of a reduced reference signal".
[i.10] Recommendation ITU-T J.247: "Objective perceptual multimedia video quality measurement in
the presence of a full reference".
[i.11] Recommendation ITU-T J.246: "Perceptual visual quality measurement techniques for multimedia
services over digital cable television networks in the presence of a reduced bandwidth reference".
[i.12] Recommendation ITU-T P.1401: "Methods, metrics and procedures for statistical evaluation,
qualification and comparison of objective quality prediction models".
[i.13] Larry Stephens: "Schaum's Outline of Statistics" series, McGraw-Hill Trade, January 1989.
3 Definition of terms, symbols and abbreviations
3.1 Terms
Void.
3.2 Symbols
Void.
3.3 Abbreviations
For the purposes of the present document, the following abbreviations apply:
API Application Programming Interface
CDN Content Delivery Network
DASH Dynamic Adaptive Streaming over HTTP
DL DownLoad
FoM Figure of Merit
IP Internet Protocol
MOS Mean Opinion Score
OS Operating System
OTT Over The Top
QoE Quality of Experience
QoS Quality of Services
QUIC Quick UDP Internet Connections
RCA Root Cause Analysis
RTCP Real Time Control Protocol
RTP Real Time Protocol
RTSP Real Time Streaming Protocol
SSL Secure Sockets Layer
TCP Transport Control Protocol
TH THreshold
UDP User Datagram Protocol
UE User Equipment
WG Working Group
ETSI
---------------------- Page: 7 ----------------------
8 ETSI TR 103 488 V1.1.1 (2019-01)
4 Passive, non-intrusive network monitoring of
SSL/QUIC OTT video services bitstreams
OTT Video payloads transported by SSL and QUIC, together with metadata encryption, reduce the applicability of QoE
algorithms/models referred by ETSI TR 102 493 [i.3] and Recommendation ITU-T J series, P series [i.4], [i.5], [i.6],
[i.7], [i.8], [i.9], [i.10] and [i.11] for non-intrusive network monitoring at mid-point solutions and prevent them to
provide the MOS identified by such standards.
The approach described in the present document is based on the figure of merit and on identification of the possible
causes of video quality degradation, complements the one depicted by ETSI TR 102 493 [i.3] for non-intrusive network
monitoring solutions and provides a reference for them when payloads are transported by SSL and QUIC.
Note that such approach can be also extended to all the passive monitoring solutions, including any possible ones on the
User Equipment, when only transported payloads by SSL and QUIC are available.
5 Categories of OTT video streaming session QoS
parameters
5.1 Introduction
The whole end-to-end OTT video streaming session quality is determined and impacted by four categories of QoS
parameters:
• Transport/delivery QoS parameters.
• Audio/video integrity QoS parameters.
• Streaming session QoS parameters.
• Service centric QoS parameters.
Each category can cover a set of QoS parameters which can be either measured and/or calculated directly, or can be
estimated based on inferences (that are out of the scope of the present document) made on client's events, service's
states and transport packet analysis. The latter is needed due to various types and levels of encryptions. At the time of
publication of the present document, there are scenarios within which even these inferences techniques are challenged
by fully encrypted streams, such as QUIC protocol, which leaves no metadata available. Work is going on in 3GPP,
IETF and DASH Industry Forum in order to offer an open API interface which would allow access to minimum
required metadata.
Table 1 provides a list of minimum required and most meaningful QoS parameters which can be used for end-to-end
characterization (e.g. CDNs Id, CDN protocol, subscriber IP address, session id, video provider) and quantification
(e.g. video-audio QoS parameters, streaming QoS parameters) of the OTT video streaming session' s quality.Ultimately,
these QoS parameters can be further used to troubleshoot and identify more likely root causes of possible quality
problems.
The columns in table 1 refer to:
• Category: Transport; audio/video; streaming session; Service centric.
• Name: It provides the parameter name.
• Type: It refers to QoS or Auxiliary (Aux) aspects.
ETSI
---------------------- Page: 8 ----------------------
9 ETSI TR 103 488 V1.1.1 (2019-01)
Table 1: QoS parameters
Category Name Description Type
Transport Number of CDN Media Number of participating CDN Media Server inside a Aux
Servers session.
CDN Media ServerX Name of the used CDN Media Server X. Aux
Name X is an integer number that ranges from 1 to Number of
CDN Media Servers.
CDN Media ServerX IP IP Address of the used CDN Media Server X. Aux
address X is an integer number that ranges from 1 to Number of
CDN Media Servers.
CDN Media ServerX Bytes downloaded from CDN Media server X Aux
downloaded bytes X is an integer number that ranges from 1 to Number of
CDN Media Servers.
CDN Media ServerX Average time between the last packet of the player QoS
Average Time to first request till the first packet of the related CDN Media
packet (CDN delay) server response for all the player requests with a
response.
X is an integer number that ranges from 1 to Number of
CDN Media Servers.
CDN Media ServerX Rate of Player Requests with no answer towards CDN QoS
Failure rate (%) Media ServerX.
X is an integer number that ranges from 1 to Number of
CDN Media Servers.
CDN Downlink It is the overall Downlink Application Throughput related QoS
Application throughput to media content downloaded from CDN Media Servers
(Kbps) (Kbps).
Video/Audio Avg. video buffer (s) Average Player video buffer size during play time in Aux
second.
Avg. audio buffer (s) Average Player Audio buffer size during play time in Aux
second.
Avg. video bit rate Average video bit rate during play time (Kbps). QoS
(Kbps)
Avg. video bit rate Average video bit rate range during play time (low, fair, QoS
Range excellent). See clause 6 for Range definition.
Avg. audio bit rate Avg. audio bit rate during play time (low, fair, excellent). QoS
Range See clause 6 for range definition.
Low Video bit rate % Play time % at Low Video Bit Rate range. See clause 6 QoS
for range definition.
Low Audio bit rate % Play time % at Low Audio Bit Rate range. See clause 6 QoS
for range definition.
Good Video bit rate % Play time % at Good Video Bit Rate range. See clause 6 QoS
for range definition.
Good Audio bit rate % Play time % at Good Audio Bit Rate range. See clause 6 QoS
for range definition.
Excellent Video bit rate Play time % at Excellent Video bit rate range. See QoS
% clause 6 for range definition.
Excellent Audio bit rate Play time % at Excellent Audio bit rate range. See QoS
% clause 6 for range definition.
Video bit rate range Number of Video bit rate range switches. See QoS
switches clause 6 for range definition.
Audio bit rate range Number of Audio bit rate range switches. See QoS
switches clause 6 for range definition.
Positive Video bit rate Number of switches to higher video bit rate range. See QoS
range switches Clause 6 for range definition.
Positive Audio bit rate Number of switches to higher audio bit rate range. See QoS
range switches clause 6 for range definition.
Negative Video bit rate Number of switches to lower video bit rate range. See QoS
range switches clause 6 for range definition.
Negative Audio bit rate Number of switches to lower audio bit rate range. See QoS
range switches clause 6 for range definition.
Video DL Mbytes Video downloaded Mbytes. Aux
Audio DL Mbytes Audio downloaded Mbytes. Aux
Video Quality It refers to ETSI TR 101 578 [i.2] Video Quality QoS
parameter.
ETSI
---------------------- Page: 9 ----------------------
10 ETSI TR 103 488 V1.1.1 (2019-01)
Category Name Description Type
streaming Video Access Time (s) It refers to ETSI TR 101 578 [i.2] Video Access Time (s) QoS
session parameter, see also clause 5.1.
Pre-Playout Buffering It refers to ETSI TR 101 578 [i.2] Pre-Playout Buffering
time (s) time (s) parameter, see also clause 5.1.
Video Playout Duration It refers to ETSI TR 101 578 [i.2] Video Playout Duration QoS
(s) (s) parameter.
Number of freezes Video Freeze Occurrences. QoS
Accumulated Video It refers to ETSI TR 101 578 [i.2] Accumulated Video QoS
Freezing duration (s) Freezing Duration (s) parameter.
Video Freezing Time It refers to ETSI TR 101 578 [i.2] Video Freezing Time QoS
proportion Proportion parameter.
Video Playout cut-off It refers to ETSI TR 101 578 [i.2] Video Playout cut-off QoS
parameter.
Terminated during In addition to ETSI TR 101 578 [i.2] Video Playout QoS
freeze cut-off, it indicates if the video was terminated during a
freeze (0 if no, 1 if yes).
Video Access Failure It refers to ETSI TR 101 578 [i.2] Video Access Failure QoS
parameter.
Terminated before Differently from ETSI TR 101 578 [i.2] Video Access Aux
video reproduction Failure or Pre-Playout buffering failure, it does not
start distinguish if the termination was due to the subscriber
that exits before the video start or it was due to a player
exit due to bad network condition.
Video transfer time (s) Time from the first to the last data packet from CDNs in Aux
second.
Starving time (s) Time in second with no responses to player requests. Aux
Service centric Subscriber IP address UE IP address. Aux
Video provider YouTube™, Netflix™, etc. QoS
OS Client OS client: Android™, iOS™, etc. Aux
Player Type browser, App. Aux
CDN transport protocol HTTP, QUIC, SSL, etc. Aux
Parameters that are common with ETSI TR 101 578 [i.2] should use the timeout specified therein.
5.2 Triggers of main QoS parameters
5.2.1 Session
A video streaming session is the interval time during which a subscriber executed one or more consecutive videos from
the same OTT provider.
A session should start when the first contact from the player is detected towards a CDN Media server to play one or
more consecutive videos. It should end after an inactivity time between the Player and the CDN Media Servers.
5.2.2 Video Access Time and Pre-playout buffering time
Monitoring solutions based on player API observation should characterize the video access phase from the video clip
request till the playout start as described in figure 1.
Non-intrusive passive monitoring solutions should evaluate the Pre-playout buffering time portion of the Video Access
Time, being the Video clip request times not observable because of SSL and QUIC encryption.
5.3 Parameters relevance for an effective Root Cause Analysis
5.3.1 Transport parameters relevance
CDN QoS parameters allow to identify the involved media servers and enable to detect the ones related to low QoE and
to address a further analysis about them.
ETSI
---------------------- Page: 10 ----------------------
11 ETSI TR 103 488 V1.1.1 (2019-01)
"CDN downlink application throughput" parameter characterizes the connectivity performances experienced during the
video reproduction and allows to identify low QoE related to it.
5.3.2 QoE-Video/Audio parameters relevance
Avg. video/audio buffer allows an historical trend analysis on the buffer size strategy adopted by the different players.
In line of principle, lower this number, higher the probability that freezes are experienced.
Video/Audio DL Mbyte enable to get a view of the network usage in term of volume. When it is referred to different
network elements, it could identify possible bottleneck related to them.
The value of Low, Good, Excellent Audio/video bit rate % parameters is explained in clause 6.
Audio/Video switches indicators could contribute to detect sessions with a high number of switches that lower the
streaming service's quality from the user's perspective point of view.
5.3.3 Streaming session parameters relevance
Long "Video Access Time" or "Pre-Playout Buffering Time", "Freezes", "Accumulated Freeze duration", "Video
Freeze Time Proportion", "Terminating During Freeze" identify events that lower user experience. Therefore, they
should be the main driver to start Root Cause Analysis (RCA). The interval time when such event happened could also
be relevant for RCA. They could identify Critical Interval Times when the user experience was not good. Common
Critical Interval Times between different video sessions could detect problems affecting more users.
High concentrations of "Terminated Before Video Reproduction Start"/"Video Access Failure" and "Starving Time" in
an interval time when Video Access Phase/Pre-Playout buffering time are high or Freezes are present could be related
to a more general issue than lowered QoS due to a main problem on the transport network or on the CDN side. By
analyzing the distribution of all these parameters by "Video provider", it should be possible to address RCA towards the
operator network or towards the CDN Media server side, as explained in clause 5.3.4.
"Video Transfer Time" analysis, when aggregated by CDN, should be useful to detect CDN Media server utilization
and identify overload of them that result on low QoE parameters (Long Video Access Time/Pre-Playout Buffering Time
or Freezes).
5.3.4 Service centric parameters relevance
"Video provider" allows to get different FoM for different providers and to benchmark them too. If, in the same interval
times, low QoS is detected in all the video provider, this may suggest an issue in the operator network that delivers the
service. If it is more related to a specific OTT video provider, that may suggest an issue on the CDN Media server side.
Player and OS type enable to address RCA of low QoS streaming session parameters to specific OS (Android™, iOS™,
etc.) or Player types (Browser, App).
"Transport Protocol" allows to aggregate the QoS Audio/Video and streaming session parameters to compare
performances of the video delivered by QUIC, SSL and HTTP, and to address specific Root Cause Analysis on the
network elements related to them.
6 Thresholds for OTT video streaming quality
In order to use the Qos parameters (
...