|
TECHNICAL SPECIFICATION
Methods for Testing and Specification (MTS);
TTCN-3 Conformance Test Suite for use of XML schema;
Abstract Test Suite & IXIT
---------------------- Page: 1 ----------------------
2 ETSI TS 103 255 V1.4.1 (2018-11)
Reference
RTS/MTS-103255ed141
Keywords
ATS, IXIT, testing, TTCN, XML
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE
Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16
Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88
Important notice
The present document can be downloaded from:
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the only prevailing document is the
print of the Portable Document Format (PDF) version kept on a specific network drive within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
If you find errors in the present document, please send your comment to one of the following services:
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.
© ETSI 2018.
All rights reserved.
TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are trademarks of ETSI registered for the benefit of its Members.
TM TM
3GPP and LTE are trademarks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
oneM2M™ logo is a trademark of ETSI registered for the benefit of its Members and
of the oneM2M Partners.
GSM and the GSM logo are trademarks registered and owned by the GSM Association.
ETSI
---------------------- Page: 2 ----------------------
3 ETSI TS 103 255 V1.4.1 (2018-11)
Contents
Intellectual Property Rights . 4
Foreword . 4
Modal verbs terminology . 4
1 Scope . 5
2 References . 5
2.1 Normative references . 5
2.2 Informative references . 6
3 Definition of terms and abbreviations . 6
3.1 Terms . 6
3.2 Abbreviations . 7
4 Abstract Test Method (ATM) . 7
5 The ATS development process . . 8
5.1 Requirements and test purposes . 8
5.2 ATS structure . 8
5.2.0 Introduction. 8
5.2.1 Test case grouping . 9
5.2.2 Test case identifiers . 10
5.2.3 Naming convention inside XSD files . 11
5.3 ATS specification framework . 11
5.3.1 Use of TTCN-3 . 11
5.3.1.1 General . 11
5.3.1.2 TTCN-3 naming conventions . 12
5.3.1.3 TTCN-3 comment tags . 13
5.3.2 Module parameters . 16
5.3.3 Test case structure . 16
5.3.4 External functions . 18
5.4 ATS archive . 18
6 PIXIT conformance . 19
7 ATS conformance . 19
Annex A (normative): Abstract Test Suite (ATS) . 20
A.1 The ATS in TTCN-3 core (text) format . 20
Annex B (normative): Partial IXIT pro forma . 21
B.1 The right to copy . 21
B.2 Overview . 21
History . 22
ETSI
---------------------- Page: 3 ----------------------
4 ETSI TS 103 255 V1.4.1 (2018-11)
Intellectual Property Rights
Essential patents
IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (https://ipr.etsi.org/).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Trademarks
The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.
ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no
right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does
not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
Foreword
This Technical Specification (TS) has been produced by ETSI Technical Committee Methods for Testing and
Specification (MTS).
Modal verbs terminology
In the present document "shall", "shall not", "should", "should not", "may", "need not", "will", "will not", "can" and
"cannot" are to be interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of
provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
ETSI
---------------------- Page: 4 ----------------------
5 ETSI TS 103 255 V1.4.1 (2018-11)
1 Scope
The present document specifies the Abstract Test Suite (ATS) for the conformance test suite for using XML Schema
with TTCN-3, as defined in ETSI ES 201 873-9 [1].
The objective of the present document is to provide a basis for conformance tests for TTCN-3 tools supporting "Using
XML Schema with TTCN-3" extension [1]. The conformance test suite should give a high probability of standard
conformance with respect to TTCN-3 tools from different vendors. In the present document only using XML Schema
with TTCN-3, specified in ETSI ES 201 873-9 [1] have been considered but not the core language [9], tool
implementation (see ETSI ES 201 873-5 [i.1] and ETSI ES 201 873-6 [i.2]), language mapping (see [i.3] and [i.4]) and
language extension (see e.g. ETSI ES 202 781 [i.5], ETSI ES 202 784 [i.6] and ETSI ES 202 785 [i.7]) aspects. The test
notation used in the ATS attached in a zipped file is in TTCN-3 and it is part of the present document.
Annex A provides the Tree and Tabular Combined Notation (TTCN-3) part of the ATS.
Annex B provides the Partial Implementation Extra Information for Testing (PIXIT) Pro forma of the ATS.
2 References
2.1 Normative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
Referenced documents which are not found to be publicly available in the expected location might be found at
https://docbox.etsi.org/Reference/.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are necessary for the application of the present document.
[1] ETSI ES 201 873-9 (V4.9.1): "Methods for Testing and Specification (MTS); The Testing and
Test Control Notation version 3; Part 9: Using XML schema with TTCN-3".
[2] ETSI ES 201 873-10 (V4.4.1): "Methods for Testing and Specification (MTS); The Testing and
Test Control Notation version 3; Part 10: TTCN-3 Documentation Comment Specification".
[3] ETSI TS 102 351: "Methods for Testing and Specification (MTS); Internet Protocol Testing (IPT);
IPv6 Testing: Methodology and Framework".
[4] ISO/IEC 9646-1 (1992): "Information Technology - Open Systems Interconnection - Conformance
Testing Methodology and Framework - Part 1: General concepts".
[5] ISO/IEC 9646-4: "Information technology - Open Systems Interconnection - Conformance testing
methodology and framework - Part 4: Test realization".
[6] ISO/IEC 9646-5: "Information technology - Open Systems Interconnection - Conformance testing
methodology and framework - Part 5: Requirements on test laboratories and clients for the
conformance assessment process".
[7] ISO/IEC 9646-7 (1994): "Conformance testing methodology and framework - Part 7:
Implementation Conformance Statements".
[8] ETSI TS 103 253: "Methods for Testing and Specification (MTS); TTCN-3 Conformance Test
Suite for use of XML schema; Implementation Conformance Statement".
[9] ETSI ES 201 873-1 (V4.9.1): "Methods for Testing and Specification (MTS); The Testing and
Test Control Notation version 3; Part 1: TTCN-3 Core Language".
ETSI
---------------------- Page: 5 ----------------------
6 ETSI TS 103 255 V1.4.1 (2018-11)
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ETSI ES 201 873-5: "Methods for Testing and Specification (MTS); The Testing and Test Control
Notation version 3; Part 5: TTCN-3 Runtime Interface (TRI)".
[i.2] ETSI ES 201 873-6: "Methods for Testing and Specification (MTS); The Testing and Test Control
Notation version 3; Part 6: TTCN-3 Control Interface (TCI)".
[i.3] ETSI ES 201 873-7: "Methods for Testing and Specification (MTS); The Testing and Test Control
Notation version 3; Part 7: Using ASN.1 with TTCN-3".
[i.4] ETSI ES 201 873-8: "Methods for Testing and Specification (MTS); The Testing and Test Control
Notation version 3; Part 8: The IDL to TTCN-3 Mapping".
[i.5] ETSI ES 202 781: "Methods for Testing and Specification (MTS); The Testing and Test Control
Notation version 3; TTCN-3 Language Extensions: Configuration and Deployment Support".
[i.6] ETSI ES 202 784: "Methods for Testing and Specification (MTS); The Testing and Test Control
Notation version 3; TTCN-3 Language Extensions: Advanced Parameterization".
[i.7] ETSI ES 202 785: "Methods for Testing and Specification (MTS); The Testing and Test Control
Notation version 3; TTCN-3 Language Extensions: Behaviour Types".
3 Definition of terms and abbreviations
3.1 Terms
For the purposes of the present document, the terms given in ISO/IEC 9646-1 [4], ISO/IEC 9646-7 [7], ETSI
ES 201 873-1 [9] (TTCN-3) and the following apply:
Abstract Test Method (ATM): description of how an IUT is to be tested, given at an appropriate level of abstraction to
make the description independent of any particular realization of a Means of Testing, but with enough detail to enable
abstract test cases to be specified for this method
Abstract Test Suite (ATS): test suite composed of abstract test cases
ICS pro forma: document, in the form of a questionnaire, which when completed for an implementation or system
becomes an ICS
Implementation Conformance Statement (ICS): statement made by the supplier of an implementation claimed to
conform to a given specification, stating which capabilities have been implemented
Implementation eXtra Information for Testing (IXIT): statement made by a supplier or implementor of an IUT
which contains or references all of the information related to the IUT and its testing environment, which will enable the
test laboratory to run an appropriate test suite against the IUT
IXIT pro forma: document, in the form of a questionnaire, which when completed for the IUT becomes the IXIT
Implementation Under Test (IUT): implementation of one or more OSI protocols in an adjacent user/provider
relationship, being part of a real open system which is to be studied by testing
ETSI
---------------------- Page: 6 ----------------------
7 ETSI TS 103 255 V1.4.1 (2018-11)
Means Of Testing (MOT): combination of equipment and procedures that can perform the derivation, selection,
parameterization and execution of test cases, in conformance with a reference standardized ATS and can produce a
conformance log
3.2 Abbreviations
For the purposes of the present document, the following abbreviations apply:
ATM Abstract Test Method
ATS Abstract Test Suite
ETS Executable Test Suite
ICS Implementation Conformance Statement
IUT Implementation Under Test
IXIT Implementation eXtra Information for Testing
MOT Means Of Testing
TC Test Case
TCI TTCN-3 Control Interface
TP Test Purpose
TRI TTCN-3 Runtime Interface
TS Test System
TSS Test Suite Structure
TSS&TP Test Suite Structure and Test Purposes
TTCN-3 Testing and Test Control Notation edition 3
UTF Unicode Transformation Format
XML eXtensible Markup Language
XSD XML Schema Definition
4 Abstract Test Method (ATM)
This clause describes the ATM used to test the conformance of TTCN-3 tool implementations as described in ETSI
ES 201 873-9 [1]. Hereafter the TTCN-3 extension "Using XML Schema with TTCN-3" will be referred to as
TTCN-XML.
In the ATM, the work is performed on two levels:
• The TTCN-3 tool level. In TTCN-XML conformance tests, it is the TTCN-3 tool which is under test, i.e. the
IUT. However, unlike in protocol conformance testing, it is not standardized how test inputs, i.e. TTCN-3
modules and XML Schema, are provided. Neither are there any standardized interfaces to monitor the reaction
of the TTCN-3 tool to the test input. Outputs can only be observed indirectly by monitoring tool outputs such
as tool specific command line information, graphical user interfaces, or test execution logs. The tool output is
processed further in the tool output evaluation level in order to derive the tool conformance verdicts.
• The TTCN-3 tool output evaluation level. Here, the output of a TTCN-3 tool is indirectly observed,
e.g. rejection of TTCN-3 code due to a compile-time error in a command line notification, logging of one or
multiple test verdicts in a tool specific window or an execution trace. The observation is evaluated to assess
the tool conformance as a result of stimulating the tool with the TTCN-3 modules. Compliance or support of
the logging interface specified as part of the TTCN-3 Control Interface standard (TCI) is not required.
NOTE: The loading of the TTCN-3 modules and presentation of the output by the TTCN-3 tools is beyond the
scope of the present document.
The ATS document contains the test inputs, i.e. TTCN-3 modules and XML Schemas, for TTCN-3 tools do not
automate the execution of TTCN-3 tool conformance tests. TTCN-3 tool conformance test decisions shall be made on
the basis of expected outputs as specified in the test purposes provided in the documentation and as part of the
documentation of TTCN-3 tests in the ATS. Three different tool output classifications for TTCN-3 inputs exist:
• Rejection as invalid, i.e. the TTCN-3 input is declared syntactically or semantically incorrect by the tool. This
can either happen at compile-time or at runtime.
• Rejection to execute, i.e. an ETS is produced from the test input, but an execution does not take place.
ETSI
---------------------- Page: 7 ----------------------
8 ETSI TS 103 255 V1.4.1 (2018-11)
• Execution with results, i.e. the compiled or interpreted TTCN-3 code is executed and different kinds of outputs
are produced that can be subject of an evaluation, for example, a logged TTCN-3 test verdict in a test
execution trace (none, pass, fail, inconc) in a file or the console output. The respective tool outputs shall
specify the expected execution results in order to be able to evaluate whether the conformance test is
successful.
A conformance test for XML-supporting TTCN-3 tool can attempt to trigger every kind of such outputs in a controlled
way, i.e. a test input that is rejected as invalid does not imply a failing conformance test verdict, but instead results in a
pass verdict for the conformance test if the test is designed to trigger the rejection. More generally: a TTCN-3 tool
conformance test passes if the tool output corresponds to the expected output. The range of expected outputs is
described by the tool output classification above.
For a detailed description on how test verdict and test purposes are encoded and how they shall be evaluated with the
ATS of annex A, please refer to clause 5.3.1.3 and the descriptions for the document tags @verdict and @purpose.
5 The ATS development process
5.1 Requirements and test purposes
For each test purpose there is a table defined in clause A.2 of ETSI TS 103 253 [8]. The requirements applicable to this
TP are given by a reference to ETSI ES 201 873-9 [1]. There are no explicit formulations of requirements.
5.2 ATS structure
5.2.0 Introduction
The ATS is split into folders, where each folder represents a clause in ETSI ES 201 873-9 [1]. Clauses on a lower scope
or hierarchy are mapped into subfolders. The names of the folders are derived from the clause names in the following
way:
1) All clause and subclause numbers are converted to a two digit format: if the number consists of a single digit,
it is prefixed with zero.
2) All spaces and dots in the clause number are removed and all digits are concatenated.
3) The clause name is transformed by converting all upper case letters to lower case and replacing spaces with
low lines.
4) The transformed clause number and clause name are concatenated with a low line character inserted between
them.
5) If a clause contains subclauses and there are also a test case defined for the requirements defined in this clause,
a special subfolder is created to accommodate these test cases. The folder name consists of a transformed
clause number according to the above specified rules and the string "_top_level".
EXAMPLE: Clause 5 "Mapping XML schemas" of ETSI ES 201 873-9 [1] contains clauses 5.1 "Namespaces
and document references" and 5.2 "Name conversion". It is mapped to the following folder
structure:
+ 05_mapping_xml_schemas
+ 05_top_level
+ 0501_namespaces_and_document_references
+ 0502_name_conversion
…
ETSI
---------------------- Page: 8 ----------------------
9 ETSI TS 103 255 V1.4.1 (2018-11)
5.2.1 Test case grouping
A test case typically checks a single requirement specified in ETSI ES 201 873-9 [1]. However, tests for multiple
requirements are possible, especially in cases when the requirements are interconnected and testing them individually
would not be feasible.
Test cases consist of several files that are wrapped into folders in the lowest scope of the ATS hierarchy. The test case
folders are created in the location that corresponds to their position in ETSI ES 201 873-9 [1]. A folder containing test
cases cannot contain folders for subclauses; top_level folders are created for testing requirements of clauses that include
numbered subclauses.
Test case folder name is derived from the clause folder name in the following way:
1) The clause name is prefixed with "Pos_" in case of test cases that shall compile successfully and execute
without runtime errors. These test cases are called positive test cases.
Positive semantic tests shall be correct with respect to the TTCN-3BNF, the static semantics of TTCN-3,
should include correct XML schema inputs, and meet the respective text clauses of ETSI ES 201 873-9 [1].
They shall produce an ETS. If an ETS is produced and if it contains a control-part or a test case, it should be
executed.
2) The clause name is prefixed with "Neg_" in case of test cases that either shall produce compilation errors or
whose execution shall lead to a runtime error. These test cases are called negative test cases.
Negative tests shall be correct with respect to the TTCN-3BNF and the static semantics of TTCN-3, and
should include correct XML schema inputs, but violate the semantics one specific text clause of ETSI
ES 201 873-9 [1]. They may produce an ETS. If an ETS is produced and if it contains a control-part or a test
case, it should be executed.
3) The clause name shall be suffixed with the low line character followed by a three-digit ordinal number of the
test case. The ordinal number of the first test case defined inside a (sub)clause folder is 001 and for each
th
following test case it increases by 1 (e.g. the ordinal number of the 4 test case is 004).
4) Positive and negative test cases are numbered separately, so there can be positive and negative test cases with
the same ordinal number.
Tables 1 and 2 present test sample test case structure for clause 6.1 "Mapping of facets" of the clause 6 "Basic types" of
ETSI ES 201 873-9 [1].
The test cases shall conform to the following correctness rules:
• The test case identifiers and their group index do not imply the correct execution order of a TTCN-3 tool
conformance test.
• Grouping and subgrouping in the ATS is realized with the help of the ATS directory structure.
Table 1: Example ATS structure of positive tests
Group Subgroup Group Index
Mapping of facets Length Pos_060101_length
Clause 6.1 Clause 6.1.1
Mininclusive Pos_060107_mininclusive
Clause 6.1.7
Maxinclusive Pos_060108_maxinclusive
Clause 6.1.8
Maxexclusive Pos_060110_maxexclusive
Clause 6.1.10
ETSI
---------------------- Page: 9 ----------------------
10 ETSI TS 103 255 V1.4.1 (2018-11)
Table 2: Example ATS structure of negative tests
Group Subgroup Group Index
Mapping of facets Mininclusive Neg_060101_length_001
Clause 6.1 Clause 6.1.1
Maxexclusive Neg_060110_maxexclusive_001
Clause 6.1.10
Test case folders contain several files with the testing code. Two of these files are mandatory:
• XSD file: specifies the schema with declarations that are the subject of testing.
• TTCN-3 file: contains test description, expected result declaration and TTCN-3 module used for validation of
the imported XSD declarations.
Depending on a test case type, additional files can be present as well:
• XML file: used for validation of messages generated from XSD declarations; it contains the expected encoding
result and it is mandatory for positive test cases.
• Additional XSD files: used in case of various import scenarios and namespace tests.
TTCN-3 and XML file names and names of the mandatory XSD file consist of the name of the enclosing test case
folder and a mandatory extension:
• .ttcn (for TTCN-3 files)
• .xml (for XML files)
• .xsd (for XSD files)
In case of additional XSD files, the file name is derived from the name of the main XSD file, with low line and an
additional single-digit ordinal number inserted between the test case name and the file extension:
• _1.xml (for extra XML file)
• _1.xsd (for extra XSD file)
EXAMPLE: Folder structure for a test case Pos_050103_imports_001:
• Containing folder: 05_mapping_xml_schemas\0501_namespaces_and_document_references\050103_imports\:
- Folder: Pos_050103_imports_001; contains files:
Pos_050103_imports_001.ttcn
Pos_050103_imports_001.xsd
Pos_050103_imports_001_1.xsd
Pos_050103_imports_001.xml
5.2.2 Test case identifiers
The test case names are built up according to the following schema:
<"TC">"_""_""
where:
a) double quotes (") are used to enclose literal strings;
b) containing positive and negative syntactic and semantic test, refers to ETSI ES 201 873-1 [9]
clause numbers and names;
ETSI
---------------------- Page: 10 ----------------------
11 ETSI TS 103 255 V1.4.1 (2018-11)
c) is a running 3-digit decimal number, starting in each subgroup path with "001".
EXAMPLE: TC_Pos_060101_length_001
i) The example refers to a positive test case for string with length restriction.
ii) It is the first test case of this group/subgroup.
NOTE: The TP identifier of TC_Pos_060101_length_001 is TP_Pos_060101_length_001.
5.2.3 Naming convention inside XSD files
The XSD files shall define a target namespace unless it contains a dedicated test for rules connected with handling no
target namespace in which case the target namespace shall be omitted. The target namespace shall consist of the string
"schema:" followed by the schema file name without extension:
<"schema">"_"
where:
a) double quotes (") are used to enclose literal strings;
b) XSD_file_name is the name of the XSD file name without extension.
EXAMPLE: XSD file Pos_050103_imports_001.xsd defines the target namespace
"schema:Pos_050103_imports_001"
The tests that check module name transformation are the only exception to this rule. These tests might define a different
namespace name according to the test case purpose.
In the XSD file, the default unprefixed namespace shall be the schema n
...