ITU-T Workshop on
“New challenges for Telecommunication
Security Standardizations"
Geneva, 9(pm)-10 February 2009
Providing testability for ITU
Recommendations
Ostap Monkewich,
OMCI
Geneva, 9(pm)-10 February 2009
International
Telecommunication
Union
Theme
“What is missing from your
Recommendations that is needed for
testing?”
Geneva, 9(pm)-10 February 2009
International
Telecommunication
Union
2
Why do we need to test?
To show the customer that the product
meets the requirements
To show that the product is likely to
interoperate with other vendors
To demonstrate that Recommendations
were implemented inside the product
To improve the quality of ITU-T
Recommendations
Geneva, 9(pm)-10 February 2009
International
Telecommunication
Union
3
Need high-quality test results
Repeatable by any competent test
laboratory
Can be used to compare with test
results obtained for similar products
Recognized by all and accepted in all
geographical market regions
Geneva, 9(pm)-10 February 2009
International
Telecommunication
Union
4
What kind of testing?
Conformance Testing
verifies if the implementation does what the
Recommendation says it is supposed to do
Product
Test Suite (Gold Standard)
Interoperability Testing
checks if two implementation communicate at
functionalities level
• Functionalities
• Every pair
Geneva, 9(pm)-10 February 2009
•
•
•
•
N = 6 products or 15 pairs
Each product is tested 15 times
N = 100 or ~5000 pairs
Each product is tested 5000 times
International
Telecommunication
Union
5
Frequently Asked Question
Why not do only interoperability testing?
Answer: We don’t know what to do when
two implementations do not interoperate:
If we change something – are we closer to the
Recommendation or farther from it?
Non-conforming changes destroy interoperability
with other vendors
Serious problems will not be discovered when
observing functionalities only
example: turning lights on and off in a new house
is a good “interoperability” or functionality test.
The house may burn down at a later time because
the wiring did not conform to the standard
Geneva, 9(pm)-10 February 2009
International
Telecommunication
Union
6
Sources of Interoperability Problems
Recommendations
Errors in Recommendations
Ambiguities in natural language
Unverified or invalid behaviours described
Implementers’ different interpretations of
text
Requirements in text, more than one meaning
No standardized questionnaire for supplier
Incompatible Recommendations/implementations
Different choices of options
Device incompatibilities
Different host system configurations
Geneva, 9(pm)-10 February 2009
International
Telecommunication
Union
7
In addition to Base Recommendation
Requirements Clause
Extracted from text, need no interpretation
Implementation Conformance Statement (ICS)
Proforma
Supplier declares what pieces were
implemented from Recommendation
Implementation eXtra Information for Testing
(IXIT) Proforma
Identifies non-standardized items, how
architecture, interfaces are packaged
Test Suite Structure
Logically groups test cases
Test Purposes
one test purpose/verdict per test case
Abstract Test Suite
Set of tests that covers the Recommendation
Geneva, 9(pm)-10 February 2009
International
Telecommunication
Union
8
Recommendations to support testing
TC1
R2
TP2
TC2
TP3
TC3
R3
.
.
.
Rn
.
.
.
TPn
.
.
.
TCn
TSS & TP - Test Suite Structure and Test Purposes
ATS:
- Abstract Test Suite
ICS:
- Implementation Conformance Statement
Geneva, 9(pm)-10 February 2009
Execution
Test Cases
IXIT
TC1
TC3
.
.
.
IXIT Information
TP1
ICS
ICS Answers
ATS
R1
ICS Questions
Base Recommendation
Requirements ICS
TSS &TP
Clause
Proforma
TCn
TP - Test Purpose
TC - Test Case
R - Requirement
International
Telecommunication
Union
9
Requirements – NGN Draft Rec. Y.2702
Sections 7.2.3 and 8.2.1 – NGN User authentication for service access and
network attachment
(R-40) Each user/subscriber associated with an application
service subscription is required to be uniquely addressable
(R-41) It is required that it be possible for the end user to access a
service simultaneously multiple times and/or from multiple
devices.
(R-42) It is required that it be possible to support multiple
subscription profiles for an individual end-user.
(O-1) It is an option that network access points supporting NGN
TE and TE-BE support capabilities to allow the end user to
uniquely identify the NGN provider.
(O-2) It is an option that network access points supporting NGN
TE and TE-BE support capabilities to allow the user to
authenticate and authorize the NGN provider.
Geneva, 9(pm)-10 February 2009
International
Telecommunication
Union
10
Implementation Conformance
Statement (ICS) Proforma
Appendix IV to NGN Draft Rec. Y.2702 on Identity
management (IdM)
Index
Text
3.5.1
When User Equipment (UE) sends HTTP
Request does Service Provider (SP) return
http Response with <lib:AuthnRequest >
3.5.2
On receipt of HTTP Request from UE does
SP send a redirect http Response with
<lib:AuthnRequest > in the URL
3.5.3
When UE sends HTTP Request does SP
inform the Identity Provider (IdP) of the
receipt
Geneva, 9(pm)-10 February 2009
Status Ref.
Val.
Support
M
IV.
2.1.2
1)
_Yes _No
M
IV.
2.1.2
2)
_Yes _No
O
IV.
2.1.2
x)
_Yes _No
International
Telecommunication
Union
11
Examples of Test Purposes in
SIP Security Testing
Ensure that the IUT, after sending the
initial REGISTER request to the Registrar,
ignores Registrar OK response by sending
a second REGISTER request
Ensure that the IUT re-sends the initial
REGISTER request on receipt from the
Registrar of a 401 Unauthorized response
in which WWW-Authenticate header does
not contain the nonce= field
Geneva, 9(pm)-10 February 2009
International
Telecommunication
Union
12
Implementation Conformance
Statement (ICS)
ICS = ICS Proforma with answers
A list of requirements and
options claimed to have been
implemented
Used for
Shopping for products to match
for interoperability
selecting Test Cases from test
suite for test execution
Geneva, 9(pm)-10 February 2009
ICS
International
Telecommunication
Union
13
Test Suite Structure
Test Suite
Test Group
Test Group
Test Group
Test Group
Test Case
Test Case
Test Case
Geneva, 9(pm)-10 February 2009
Test Case
International
Telecommunication
Union
14
Test Case Structure
Test Case
Test Step
Test Step
Test Step
Test Step
Test Event
Test Event
Test Event
Geneva, 9(pm)-10 February 2009
Test Event
International
Telecommunication
Union
15
Conclusion
First, Conformance testing then
Interoperability testing
High-quality Recommendations are
needed
Base Recommendations require
additional parts to produce widely
accepted test results
Methodology Recommendations for
testing are available – attached charts
Geneva, 9(pm)-10 February 2009
International
Telecommunication
Union
16
Additional Slides
All you need to develop high-quality
Recommendations and Test Specifications
Geneva, 9(pm)-10 February 2009
International
Telecommunication
Union
17
Conformance and Interoperability Testing
Methodology Recommendations
X.290 - General Concepts
X.291 - Abstract Test Suite Specification
X.292 - (Superseded by Z.160/170 series)
X.293 - Test Realization
X.294 - Requirements on Test Laboratories and
Clients
X.295 - Protocol Profile Test Specification
X.296 - Implementation Conformance Statements
Supplement 1 to X.290 series - Generic approach
to interoperability testing
Supplement 2 to X.290 series - Interoperability
testing framework and methodology
Geneva, 9(pm)-10 February 2009
International
Telecommunication
Union
18
Testing and Test Control Notation
(TTCN-3)
Z.161
Z.162
Z.163
Z.164
Z.165
Z.166
Z.167
Z.168
Z.169
Z.170
(Z.140 Revised): Core Language
(Z.141 Revised): Tabular Format
(Z.142 Revised): Graphical Format
(Z.143 Revised): Operational semantics
(Z.144 Revised): Runtime interface
(Z.145 Revised): Control interface
(New): Using ASN.1 with TTCN-3
(New): The IDL to TTCN-3 mapping
(New): Using XML Schema with TTCN-3
(New): TTCN-3 documentation tags
Geneva, 9(pm)-10 February 2009
International
Telecommunication
Union
19
Specification and Description
Language
Z.100 Overview of SDL-2008
Z.101 Basic SDL-2008
Z.102 Comprehensive SDL-2008
Z.103 Shorthand notation and annotation in
SDL-2008
Z.104 Data and action language in SDL-2008
Z.105 SDL-2008 combined with ASN.1 modules
Z.106 Common Interchange Format for SDL2008
Geneva, 9(pm)-10 February 2009
International
Telecommunication
Union
20
Abstract Syntax Notation One (ASN.1)
X.680 (07/02) Information technology - Abstract
Syntax Notation One (ASN.1): Specification of
basic notation
X.680-X.693 (07/02) Information Technology Abstract Syntax Notation One (ASN.1) & ASN.1
encoding rules
X.681 (07/02) Information technology - Abstract
Syntax Notation One (ASN.1): Information object
specification
X.682 (07/02) Information technology - Abstract
Syntax Notation One (ASN.1): Constraint
specification
X.683 (07/02) Information technology - Abstract
Syntax Notation One (ASN.1): Parameterization
of ASN.1 specifications
Geneva, 9(pm)-10 February 2009
International
Telecommunication
Union
21
Abstract Syntax Notation One (ASN.1)
X.690 (07/02)Information technology - ASN.1
encoding rules: Specification of Basic Encoding
Rules (BER), Canonical Encoding Rules (CER) and
Distinguished Encoding Rules (DER)
X.691 (07/02) Information technology - ASN.1
encoding rules: Specification of Packed Encoding
Rules (PER)
X.692 (03/02) Information technology - ASN.1
encoding rules: Specification of Encoding Control
Notation (ECN)
X.693 (12/01) Information technology - ASN.1
encoding rules: XML encoding rules
Geneva, 9(pm)-10 February 2009
International
Telecommunication
Union
22
Supporting Recommendations
Z.120 Message
Sequence Chart (MSC)
Z.150 User
Requirements
Notation - Language
Requirements and
Framework
Z.151 User
Requirements
Notation - Language
Definition
Geneva, 9(pm)-10 February 2009
Z.110 Application of
Formal Description
Techniques
Z.450 Quality aspects
of protocol-related
Recommendations
International
Telecommunication
Union
23