|
TECHNICAL REPORT
Universal Mobile Telecommunications System (UMTS);
LTE;
Evaluation of Additional Acoustic Tests for Speech Telephony
(3GPP TR 26.931 version 15.0.0 Release 15)
---------------------- Page: 1 ----------------------
3GPP TR 26.931 version 15.0.0 Release 15 1 ETSI TR 126 931 V15.0.0 (2018-07)
Reference
RTR/TSGS-0426931vf00
Keywords
LTE,UMTS
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE
Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16
Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88
Important notice
The present document can be downloaded from:
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the only prevailing document is the
print of the Portable Document Format (PDF) version kept on a specific network drive within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
If you find errors in the present document, please send your comment to one of the following services:
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.
© ETSI 2018.
All rights reserved.
TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are trademarks of ETSI registered for the benefit of its Members.
TM TM
3GPP and LTE are trademarks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
oneM2M logo is protected for the benefit of its Members.
GSM and the GSM logo are trademarks registered and owned by the GSM Association.
ETSI
---------------------- Page: 2 ----------------------
3GPP TR 26.931 version 15.0.0 Release 15 2 ETSI TR 126 931 V15.0.0 (2018-07)
Intellectual Property Rights
Essential patents
IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (https://ipr.etsi.org/).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Trademarks
The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.
ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no
right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does
not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
Foreword
This Technical Report (TR) has been produced by ETSI 3rd Generation Partnership Project (3GPP).
The present document may refer to technical specifications or reports using their 3GPP identities, UMTS identities or
GSM identities. These should be interpreted as being references to the corresponding ETSI deliverables.
The cross reference between GSM, UMTS, 3GPP and ETSI identities can be found under
.
Modal verbs terminology
In the present document "should", "should not", "may", "need not", "will", "will not", "can" and "cannot" are to be
interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
ETSI
---------------------- Page: 3 ----------------------
3GPP TR 26.931 version 15.0.0 Release 15 3 ETSI TR 126 931 V15.0.0 (2018-07)
Contents
Intellectual Property Rights . 2
Foreword . 2
Modal verbs terminology . 2
Foreword . 5
Introduction . 5
1 Scope . 6
2 References . 6
3 Definitions and abbreviations . 7
3.1 Definitions . 7
3.2 Abbreviations . 8
4 Release 11 "For Further Study" Items . 8
4.1 Stability loss, Headset UE . 8
4.2 UE Delay, NB & WB Wireless Headset . 8
4.3 NB & WB Echo control ("double-talk") characteristics . 8
4.3.1 Results from a study on NB Echo control ("double-talk") characteristics using P.835 methodology . 8
4.3.1.1 Background . 8
4.3.1.2 Test Method & Results . 8
4.3.1.2.1 Methods . 8
4.3.1.2.2 Results . 12
4.3.2 Comparison of predictions from objective metrics to subjective double talk ratings . 17
4.3.2.1 Background and metrics . 17
4.3.2.1.1 Method for P.863 computations . 17
4.3.2.2 Comparison of metrics to subjective SIG ratings . 17
4.3.2.3 Comparison of metrics to subjective BAK ratings . 24
4.3.2.4 Comparison of metrics to subjective OVRL ratings . 25
4.3.2.5 Summary of comparison of metrics to subjective ratings for double-talk . 26
4.4 Free-field measurements for vehicle-mounted hands-free . 26
4.5 Idle Channel Noise, Sending/Receiving of test signal . 26
5 New Acoustic Tests . 26
5.1 Time-variant user behaviour. 26
5.2 Additional UE usage environments . 26
5.3 Results from a study on positional robustness tests and background noise simulations . 27
5.3.1 General . 27
5.3.2 Setup . 27
5.3.2.0 General . 27
5.3.2.1 Handset Mounting . 28
5.3.2.2 Background Noise Systems . 29
5.3.2.2.0 General . 29
5.3.2.2.1 Equalization process according to ETSI ES 202 396-1 . 30
5.3.2.2.2 Equalization process according to ETSI TS 103 224 . 30
5.3.2.2.3 Background noises. 31
5.3.3 Measurement Results . 31
5.3.3.1 Measurements in Silence Condition . 31
5.3.3.2 Measurements with Ambient Noise . 32
5.3.4 Summary . 35
5.4 Results from a study on objective measures with noise suppression and background noise . 35
5.4.1 Comparison of P.862 to subjective results for noise suppression . 35
5.4.2 Experiment 1 - NB P.835 versus P.862.1 . 36
5.4.2.1 Setup . 36
5.4.2.2 Results . 36
5.4.3 Experiment 2 - Problems with tuning for P.862.1 . 38
5.4.3.1 Setup . 38
5.4.3.2 Results . 38
ETSI
---------------------- Page: 4 ----------------------
3GPP TR 26.931 version 15.0.0 Release 15 4 ETSI TR 126 931 V15.0.0 (2018-07)
5.4.4 Experiment 3: WB P.835 v P.862.2, P.863 and TS 103 106 . 39
5.4.4.1 Setup . 39
5.4.4.2 WB Correlation Results . 39
5.4.4.3 WB Rank Order Results . 40
5.4.5 Experiment 4 SWB P.835 v P.862.2, P.863 and TS 103 106 . 42
5.4.5.1 Setup . 42
5.4.5.2 SWB Correlation Results . 42
5.4.5.3 SWB Rank Order Results . 43
5.4.6 Conclusions. 45
5.5 Validation results for combination of model A and B according to ETSI TS 103 281 . 46
5.5.1 Introduction. 46
5.5.2 Description of combination of model predictions . 46
5.5.3 Validation database 3 (DES-25): Results for combined model . 47
5.5.4 Validation database 4 (DES-26): Results for combined model . 49
5.5.5 Validation database 5 (DES-27): Results for combined model . 51
5.5.6 Conclusions. 53
6 Conclusions . 53
Annex A: Change history . 54
History . 55
ETSI
---------------------- Page: 5 ----------------------
3GPP TR 26.931 version 15.0.0 Release 15 5 ETSI TR 126 931 V15.0.0 (2018-07)
Foreword
This Technical Report has been produced by the 3rd Generation Partnership Project (3GPP).
The contents of the present document are subject to continuing work within the TSG and may change following formal
TSG approval. Should the TSG modify the contents of the present document, it will be re-released by the TSG with an
identifying change of release date and an increase in version number as follows:
Version x.y.z
where:
x the first digit:
1 presented to TSG for information;
2 presented to TSG for approval;
3 or greater indicates TSG approved document under change control.
y the second digit is incremented for all changes of substance, i.e. technical enhancements, corrections,
updates, etc.
z the third digit is incremented when editorial only changes have been incorporated in the document.
Introduction
Mobile telephony devices and voice services continue to develop and evolve and their associated minimum
performance requirements and test methodologies also need to stay relevant and representative of quality demands.
While many advances were made in Release 11 to the acoustic requirements and test specifications in TS 26.131 [13]
and TS 26.132 [8] many items therein were left marked "for further study" and require a final decision by SA4.
Additionally, there are new acoustic requirements and emerging tests worth considering in a future release, but that
require further study before incorporation into specifications.
This technical report will, first and foremost, address the remaining items presently designated as "for further study" in
TS 26.131 [13] and TS 26.132 [8].
The present document will also examine opportunities for new acoustic tests and requirements that help us to better
characterize the UE acoustic experience, opportunities to replace existing test methods with others that are more
accurate or more efficient and make specific recommendations for their inclusion in existing or new specifications.
ETSI
---------------------- Page: 6 ----------------------
3GPP TR 26.931 version 15.0.0 Release 15 6 ETSI TR 126 931 V15.0.0 (2018-07)
1 Scope
The scope of the present document is to investigate, first and foremost, the existing items presently designated as "for
further study" in TS 26.131 [13] and TS 26.132 [8].
The investigation will additionally identify, examine and evaluate opportunities for new acoustic tests and requirements
that better help characterize the UE acoustic experience, opportunities to replace existing test methods with others that
are more accurate or more efficient and to make specific recommendations for their inclusion in existing or new
specifications.
While many advances were made in Release 11 to the acoustic requirements and test specifications in TS 26.131 [13]
and TS 26.132 [8] many items therein were left marked "for further study" and require a final disposition by SA4
including:
- NB & WB Stability loss, Headset UE (TS 26.131 [13], subclauses 5.6 & 6.6).
- NB & WB Delay, Wireless Headset (TS 26.131 [13], subclauses 5.12.2.2 & 6.11.2.2).
- NB & WB Echo control ("double-talk") characteristics (TS 26.131 [13], subclauses 5.13 & 6.12, TS 26.132 [8],
subclause 8.11).
- Handset, Headset, Handheld hands-free, Desktop and vehicle mounted hands-free are all marked FFS.
- NB& WB Free-field measurements for vehicle-mounted hands-free (TS 26.132 [8], subclauses 7.2.3 & 8.2.3).
- NB & WB Idle Channel Noise, Sending/Receiving of test signal (TS 26.132 [8], subclauses 7.3.1, 7.3.2, 8.3.1 &
8.3.2).
Additionally, there are new acoustic requirements and emerging tests that may be considered in a future release, but
require further study before incorporation to our specifications. It has been anticipated that topics in this area would
include, but would not be limited to, an evaluation of:
- Time-variant user behaviour.
- Additional UE usage environments.
- New or refined test methods for existing requirements.
- Acceptance of updates (if any) to existing ETSI and ITU-T dependencies.
2 References
The following documents contain provisions which, through reference in this text, constitute provisions of the present
document.
- References are either specific (identified by date of publication, edition number, version number, etc.) or
non-specific.
- For a specific reference, subsequent revisions do not apply.
- For a non-specific reference, the latest version applies. In the case of a reference to a 3GPP document (including
a GSM document), a non-specific reference implicitly refers to the latest version of that document in the same
Release as the present document.
[1] Recommendation ITU-T P.862 (02/2001): "Perceptual evaluation of speech quality (PESQ): An
objective method for end-to-end speech quality assessment of narrow-band telephone networks
and speech codecs".
[2] Recommendation ITU-T P.800 (08/1996): "Methods for subjective determination of transmission
quality".
[3] Recommendation ITU-T P.862.3 (11/2007): "Application guide for objective quality measurement
based on Recommendations P.862, P.862.1 and P.862.2".
ETSI
---------------------- Page: 7 ----------------------
3GPP TR 26.931 version 15.0.0 Release 15 7 ETSI TR 126 931 V15.0.0 (2018-07)
[4] Recommendation ITU-T P.835 (11/2003): "Subjective test methodology for evaluating speech
communication systems that include noise suppression algorithm".
[5] ETSI TS 103 106 (V1.3.1): "Speech quality performance in the presence of background noise:
Background noise transmission for mobile terminals - objective test methods".
[6] Recommendation ITU-T P.863 (09/2014): "Perceptual objective listening quality assessment".
[7] ETSI ES 202 396-1 (V1.4.1): "Speech quality performance in the presence of background noise;
Part 1: Background noise simulation technique and background noise database".
[8] 3GPP TS 26.132: "Technical Specification Group Services and System Aspects; Speech and video
telephony terminal acoustic test specification".
[9] Recommendation ITU-T P.862.1 (11/2003): "Mapping function for transforming P.862 raw result
scores to MOS-LQO".
[10] Recommendation ITU-T P.862.2 (11/2007): "Wideband extensions to Recommendation P.862 for
the assessment of wideband telephone networks and speech codecs".
[11] ETSI EG 202 396-3 (V1.5.1): "Speech quality performance in the presence of background noise;
Part 3: Background noise transmission - Objective test methods".
[12] Recommendation ITU-T P.64, Annex E (11/2007): "Determination of sensitivity/frequency
characteristics of local telephone systems".
[13] 3GPP TS 26.131: "Technical Specification Group Services and System Aspects; Terminal acoustic
characteristics for telephony".
[14] ETSI TS 103 224 (V1.3.1): "Speech and multimedia Transmission Quality (STQ); A sound field
reproduction method for terminal testing including a background noise database".
[15] Recommendation ITU-T P.58: "Head and Torso simulator for Telephonometry".
[16] Recommendation ITU-T P.502 (05/2000): "Objective test methods for speech communication
systems using complex test signals".
[17] Recommendation ITU-T P.863.1 (09/2014): "Application guide for Recommendation ITU-T
P.863".
[18] Recommendation ITU-T P.1401 (07/2012): "Methods, metrics and procedures for statistical
evaluation, qualification and comparison of objective quality prediction models".
[19] Recommendation ITU-T P. 79 (11/2007): "Calculation of loudness ratings for telephone sets".
[20] ETSI TS 103 281 (V1.1.1): "Speech quality in the presence of background noise: objective test
methods for SWB and FB terminals".
[21] Recommendation ITU-T P.501: "Test signals for use in telephonometry".
[22] ETSI TS 103 557 (V1.1.1): "Methods for reproducing reverberation for communication device
measurements".
[23] Recommendation ITU-T P.862 (2001) Corrigendum 1 (10/2017).
[24] Recommendation ITU-T P.831: "Subjective performance evaluation of network echo cancellers"
(12/1998).
3 Definitions and abbreviations
3.1 Definitions
For the purposes of the present document, the terms and definitions given in TR 21.905 [1] and the following apply.
A term defined in the present document takes precedence over the definition of the same term, if any, in TR 21.905 [1].
ETSI
---------------------- Page: 8 ----------------------
3GPP TR 26.931 version 15.0.0 Release 15 8 ETSI TR 126 931 V15.0.0 (2018-07)
3.2 Abbreviations
For the purposes of the present document, the abbreviations given in TR 21.905 [1] and the following apply.
An abbreviation defined in the present document takes precedence over the definition of the same abbreviation, if any,
in TR 21.905 [1].
HHS Hand-held speakerphone (= Hand-held hands-free)
4 Release 11 "For Further Study" Items
4.1 Stability loss, Headset UE
Void
4.2 UE Delay, NB & WB Wireless Headset
Void
4.3 NB & WB Echo control ("double-talk") characteristics
4.3.1 Results from a study on NB Echo control ("double-talk")
characteristics using P.835 methodology
4.3.1.1 Background
In Release 11 of TS 26.132 [8], new methods for evaluation of echo control characteristics were introduced in
Clauses 7.11 and 8.11. However, corresponding requirements were not defined in TS 26.131 [13].
A subjective listening test based on methods from Recommendation ITU-T P.835 [4] was conducted in order to provide
some data for purposes of investigating possible requirements.
Instead of a conversational test, or talking and listening test, the present document provides results from listening only
test, so participants did not experience echo while talking, only while passively listening. Below are presented results of
the subjective evaluation of real speech double talk test, for 12 devices, in both handset and handheld speakerphone for
narrow band.
4.3.1.2 Test Method & Results
4.3.1.2.1 Methods
The categories defined in Clauses 7.11, Figure 17b5, (copied below in Figure 1 for convenience) and Table 1 (Table 2c,
and 8.11, Figure 19b5, and Table 2g), are described in perceptually-relevant terms.
ETSI
---------------------- Page: 9 ----------------------
3GPP TR 26.931 version 15.0.0 Release 15 9 ETSI TR 126 931 V15.0.0 (2018-07)
Level
difference
[dB]
E F G
4
25 150
A1 Duration [ms]
-4
A2
25 150
-15
B C D
Figure 1: Classification of echo canceller performance
Table 1: Categories for echo canceller performance classification
Category Level difference (ΔL) Duration (D) Description
A1 -4 dB ≤ ΔL < 4 dB Full-duplex and full transparency
A2 -15 dB ≤ ΔL < -4 dB Full-duplex with level loss in Tx
B ΔL < -15 dB D < 25 ms Very short clipping
C ΔL < -15 dB 25 ms ≤ D < 150 ms Short clipping resulting in loss of syllables
D
ΔL < -15 dB D ≥ 150 ms Clipping resulting in loss of words
E
ΔL ≥ 4 dB D < 25 ms Very short residual echo
F
ΔL ≥ 4 dB 25 ms ≤ D < 150 ms Echo bursts
G ΔL ≥ 4 dB D ≥ 150 ms Continuous echo
4.3.1.2.1.1 Rating scales
The impairments in categories A2, B, C, and D can be understood as distortions of the uplink speech. In contrast, the
impairments in categories E, F, and G can be understood as intrusions of residual or continuous echo. Based on these
observations, the rating scales of P.835 [4], SIG, BAK, and OVRL, as shown in Figures 2 below, were adopted for this
listening evaluation. In this study, the BAK rating scale was used to quantify the level of intrusiveness of any echo. In
other P.835 studies, the BAK rating scale has been more typically used to quantify the level of intrusiveness of
background noise.
ETSI
---------------------- Page: 10 ----------------------
3GPP TR 26.931 version 15.0.0 Release 15 10 ETSI TR 126 931 V15.0.0 (2018-07)
Figure 2a (Figure 5/P.835): Speech signal rating scale
Figure 2b (Figure 6/P.835): Background rating scale
Figure 2c (Figure 7/P.835): Overall quality rating scale
ETSI
---------------------- Page: 11 ----------------------
3GPP TR 26.931 version 15.0.0 Release 15 11 ETSI TR 126 931 V15.0.0 (2018-07)
It was anticipated that the impairments in categories A2, B, C and D, would be related to ratings on the SIG (speech
distortion) scale, and that impairments in categories E, F, and G would be related to the ratings on the BAK background
intrusiveness scale.
4.3.1.2.1.2 Speech Source
The speech source used is Segment 2 (four sentences) of the current double talk test, British English from
Recommendation ITU-T P.501 [21]. This includes two male and two female talkers. While this is rather limited in
comparison to some subjective tests, the exact signal and conditions are used to facilitate direct comparisons with the
objective measures. For each presentation and rating, a single sentence was presented (total of four sentences, from two
male and two female speakers).
4.3.1.2.1.3 Reference Signals
For the SIG dimension, the Wiener-filter based reference system proposed in [5] and used in [6] was used. While this
reference system has been primarily used as a reference for noise suppression, as many echo control systems provide
echo suppression using a multi-band attenuation mechanism, it seems reasonable to use that reference system in this
context. Expert listening to the reference system and the distortions introduced by the devices exhibiting higher levels
of impairments in the A2, B, C and D categories indicated qualitatively similar perceptions. Four levels of Wiener-filter-
based distortion, similar to those used in P.835 tests for noise suppression and judged by expert listeners to span the
range from 1 to 5, were used.
For the BAK dimension, recordings of echo were made on a device with the ability to disable the AEC system, and to
capture signals at the microphone. T
...