Competence
Center
Modelling and Testing
An Overview on TTCN-3
Ina Schieferdecker
Technical University Berlin/Fraunhofer FOKUS
Tel.: +49 30 3463 7241, Email: schieferdecker@fokus.fraunhofer.de
Outline
ƒ
Overview
ƒ
Main Concepts
ƒ
TTCN-3 in a Nutshell
ƒ
TTCN-3 based test systems
ƒ
An Example: IMS Benchmarking
ƒ
Summary
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
Outline
ƒ
Overview
ƒ
Main Concepts
ƒ
TTCN-3 in a Nutshell
ƒ
TTCN-3 based test systems
ƒ
An Example: IMS Benchmarking
ƒ
Summary
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
ƒ
The Testing and Test Control Notation
ƒ
A standardized alternative to proprietary test systems
ƒ Developed by a large group of testing experts
ƒ Used by a growing community
ƒ Proven by tools
ƒ Maintained at ETSI
ƒ
A test specification and implementation language
ƒ
A multipart standard covering
ƒ textual TTCN-3 core
ƒ graphical TTCN-3
ƒ execution interfaces TRI and TCI
ƒ language mappings to TTCN-3, e.g. for IDL
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
ƒ
ƒ
Areas of Testing
ƒ
Regression Testing
ƒ
Conformance and Functionality Testing
ƒ
Interoperability and Integration Testing
ƒ
Load/ Stress Testing
Applications
ƒ
Mobile communications (GSM, GPRS, UMTS, TETRA)
ƒ
Wireless LANs and MANs (Hiperlan, Hiperaccess), cordless phones (DECT)
ƒ
Broadband technologies (B-ISDN, ATM)
ƒ
Internet protocols (IPv6, SIP, Voice over IP)
ƒ
Middleware platforms (CORBA, CCM, EJB, Web services)
ƒ
Smart Card Testing, MOST, CAN, Powertrain
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
Placement of TTCN-3
ƒ
One test technology for different tests
ƒ
Distributed, platform-independent testing
ƒ
Integrated graphical test development, documentation and analysis
ƒ
Adaptable, open test environment
Product s
B) Com ponent Provider
C) Solut ion Provider
Funct ionalit y Int erop./Int egr.
Development
Process
Solut ion
Deployment
ƒ
Test
Execut ion
Specif icat ion
Specif icat ion
The testing middleware
ƒ
unifying the documentation and
definition of tests
ƒ
unifying the tests in IT, Internetand Telco-based systems
(supporting their convergence)
ƒ
unifying the test infrastructure
St andardizat ion
Product /Component
Specif icat ion
A) Forum
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
History (1)
1984
ƒ TTCN (1992)
ƒ Published as an ISO standard
ƒ Tree and Tabular Combined Notation
1992
ƒ Used for protocol testing
(GSM, N-ISDN, B-ISDN)
TTCN
1994
ƒ TTCN-2/2++ (1997)
TTCN-2
ƒ Written by TC MTS
ƒ Published by ISO
1997
ƒ Concurrent tests
TTCN-2++
ƒ Modularization
ƒ Manipulate external data
ƒ Rather for conformance testing
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
History (2)
ƒ TTCN-3 (2000)
ƒ Testing and Test Control Notation
ƒ Written by TC MTS
1998
ƒ Published by ETSI and ITU
2000
ƒ Proper language
(well defined syntax and semantics)
‹
ƒ Enhanced communication, configuration and control
ƒ Standard test specification
(SIP, SCTP, HiperLan, HiperAccess, IPv6 etc.)
ƒ TTCN-3 (2006): version 3
2006
© FOKUS, 2006
Æ Ongoing maintenance for change requests and extension
proposals
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
TTCN-3 Definition
ASN.1
TTCN-3
Core
Language
Tabular
Format
IDL
XML
Other Types
& Values n
© FOKUS, 2006
Graphical
Format
Presentation
Format n
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
TTCN-3 Standards
ƒ
ETSI ES 201 873-1 TTCN-3 Core Language (CL)
ƒ
ETSI ES 201 873-2 TTCN-3 Tabular Presentation Format (TFT)
ƒ
ETSI ES 201 873-3 TTCN-3 Graphical Presentation Format (GFT)
ƒ
ETSI ES 201 873-4 TTCN-3 Semantics
ƒ
ETSI ES 201 873-5 TTCN-3 Runtime Interfaces (TRI)
ƒ
ETSI ES 201 873-6 TTCN-3 Control Interfaces (TCI)
ƒ
ETSI ES 201 873-7 ASN.1 to TTCN-3 Mapping
ƒ
ETSI ES 201 873-8 IDL to TTCN-3 Mapping
ƒ
ETSI ES 201 873-9 XML Schema to TTCN-3 Mapping
ƒ
ETSI ES 201 873-10 TTCN-3 Documentation
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
Outline
ƒ
Overview
ƒ
Main Concepts
ƒ
TTCN-3 in a Nutshell
ƒ
TTCN-3 based test systems
ƒ
An Example: IMS Benchmarking
ƒ
Summary
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
TTCN-3 – Based Black-Box Testing
TTCN-3 Test Case
Port.send(Stimulus)
Port.receive(Response)
• Assignment
of a
Test Verdict
Port
System Under Test
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
TTCN-3 – Test Configuration
SUT
TTCN-3 Test Case
create
start
TC
create
start
© FOKUS, 2006
MTC
TC
TCs
create
start
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
TTCN-3 – Test Verdicts
ƒ
Test verdicts: none < pass < inconc < fail < error
ƒ
Each test component has its own local verdict, which can be set (setverdict) and
read (getverdict).
ƒ
A test case returns a global verdict
Verdict returned by the test
case when it terminates
P
F
I
MTC
F
setverdict(fail)
© FOKUS, 2006
PTC1
P
setverdict(pass)
ITU-T SG17 IOP Testing Workshop, Dec. 2006
PTCN
I
setverdict(inconc)
www.fokus.fraunhofer.de/motion
TTCN-3 Module
•
•
Module
•
Module
Module
Definitions
Definitions
The top level entity of TTCN-3 is module.
A module can import definitions from other
modules.
A module contains a definition part and a
control part.
module MyModule {
// definition part
control {
// test execution logic
}
Module
ModuleControl
Control
}
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
Major elements of TTCN-3
module definitions
Importing definitions from other modules
defined in TTCN-3 or other languages
IImports
mports
User defined data types (messages, PDUs,
information elements, …)
DData
ata TTypes
ypes
Test data transmitted/received during test
execution (templates, values)
TTest
estD
Data
ata
TTest
estC
Configuration
onfiguration
TTest
estB
Behavior
ehavior
© FOKUS, 2006
Definition of the test components and
communication ports
Specification of the dynamic test behavior
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
Outline
ƒ
Overview
ƒ
Main Concepts
ƒ
TTCN-3 in a Nutshell
ƒ
TTCN-3 based test systems
ƒ
An Example: IMS Benchmarking
ƒ
Summary
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
Example: Test Configuration
Test Component (MTC)
UA
SA
System Under
Test (SUT)
Medium Service
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
Example: Test Behavior
msc Example
UA
loop<100>
SUT
SA
DATreq("data")
DATind(DT,nr,"data")
DATind(RE,nr,“resp")
DATind()
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
TTCN-3 Test Case Example
testcase Example( )
runs on MTC_Type {
var default mydefault := activate (DefaultDef());
T1.start;
for (integer i:=1; i<=100; i:=i+1) {
UA.send(DATreq:{"data"});
SA.receive(DATind:{DT, nr, "data"});
SA.send(DATind:{RE,nr, “resp”});
UA.receive(DATind :{});
}
setverdict(pass);
T1.stop;
}
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
The same graphically
testcase Example() runs on MTC_Type
mtc
UA
UA_Type
MTC_Type
SA
SA_Type
var default mydefault :=
activate(MyDefault())
T1
for (i:=1; i<=100; i:=i+1)
DATreq
("data")
DATind
(DT,nr,"data")
DATind
(RE,nr,"resp")
DATind
pass
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
Outline
ƒ
Overview
ƒ
Main Concepts
ƒ
TTCN-3 in a Nutshell
ƒ
TTCN-3 based test systems
ƒ
An Example: IMS Benchmarking
ƒ
Summary
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
Aspects of TTCN-3 Tooling
Test Automation
TTCN-3
development
tools
© FOKUS, 2006
TTCN-3
based
solutions
ITU-T SG17 IOP Testing Workshop, Dec. 2006
TTCN-3
community
www.fokus.fraunhofer.de/motion
Aspects of TTCN-3 Tooling
Test Automation
Editing
Closed-Box
Standard
Bodies
TTCN-3
Implementing
tools
TTCN-3
Test
Device
based
Support
solutions
TTCN-3
Users
community
Executing
Test Suite
Frameworks
Educational
Institutions
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
TTCN-3 Development Tools
The Foundation
Test Automation
ƒ Editing
ƒ Textual
ƒ Graphical
ƒ Tabular
Editing
Closed-Box
Standard
Bodies
TTCN-3
Implementing
Test Device
Support
Users
Executing
Test Suite
Frameworks
Educational
Institutions
Tools
ƒ Implementing
ƒ Compilers / Interpreters
ƒ Standardized Implementation Interfaces (TRI / TCI)
ƒ Multiple Platform Support
( Java / C / C++)
ƒ Executing
ƒ Interactive: Graphical user interfaces
ƒ Fully automated: Command line user interfaces
ƒ Distributed: Distributed execution environments
ƒ Recently, test generation tools towards TTCN-3
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
TTCN-3 Solutions
The Nuts and Bolts
Test Automation
ƒ Closed-Box Solutions
ƒ Ready-to-Run
Editing
Closed-Box
Standard
Bodies
Implementing
Test Device
Support
Users
Executing
Test Suite
Frameworks
Educational
Institutions
ƒ Optimized application
ƒ Conformance test scenario
ƒ Test Device Support
ƒ Test scripting language
ƒ Multiple interface support
ƒ User defined test scenarios
ƒ Test Suite Frameworks
ƒ Ready-to-Run
ƒ Modifiable and extensible
ƒ Combining compliance and development tests
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
Overall Picture
Data
Interface
Generation of
test data
Generation of
test behavior
Generation of
test data structure
Adaptor
acc. to the
mapping rules
System
© FOKUS, 2006
A
D
A
P
T
O
R
Compilation
to Executable Tests
Test
Test
Test
Component
Test
Component
Component
Component
Test
System
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
The Execution Interfaces
Test System User
TM: Management
Test System
TE: Test Execution
CD: CoDec
CH:
Component
Handling
TCI
TRI
SA: System Adaptor
PA : Platform Adaptor
SUT: System Under Test
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
Local and distributed test setups
TM
TE
TE
CD
CD
CH
Special TE:
Initiating StartTestCase &
Calculating Final Verdict
SA
PA
SA
SA
© FOKUS, 2006
CD
PA
PA
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
TTCN-3 Community
Resources and Users
Test Automation
ƒ
Standard Bodies
Editing
Closed-Box
Standard
Bodies
Implementing
Test Device
Support
Users
Test Suite
Frameworks
Educational
Institutions
Executing
ƒ ETSI, ITU, WiMAX, 3GPP, others
ƒ Standard bodies produce TTCN-3 test suites
ƒ TTCN-3 Web Site/Forum at ETSI (www.ttcn-3.org)
ƒ
Educational Institutions
ƒ Universities, Research Institutes
ƒ European Research Projects
ƒ Free academic licenses available
ƒ
Users
ƒ Different Domains: Telco, IT, Automotive, Embedded Systems
ƒ Education Tracks available
ƒ TTCN-3 Users Conference
ƒ TTCN-3 Certificate
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
An Example: TTworkbench – textual test design
Compile Button
Problems
The TTCN-3
Outline
view
File Overview
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
TTworkbench - graphical test design
3. Data Import
2. Full GFT Support
1. Project View
4. TTCN-3 Code Im- and Export
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
TTworkbench – TTCN-3 Execution
3.Test Data View
1. Test Case Management
6. Test Report Generation
4. Detail Logging
5. Graphical Logging
2. Parameterization
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
Outline
ƒ
Overview
ƒ
Main Concepts
ƒ
TTCN-3 in a Nutshell
ƒ
TTCN-3 based test systems
ƒ
An Example: IMS Benchmarking
ƒ
Summary
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
IMS in a Nutshell
ƒ
IMS = IP Multimedia Subsystem
Messaging Server
Presence Server
Application Server
IMS
Network Core
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
IMS Testing
ƒ
The importance of testing
ƒ ... to validate the key IMS subsystems
¾ Service-oriented
ƒ Protocol
conformance
Protocol
Conformance
testing
¾ Combines research and
industrial interests
ƒ ... to ensure
interoperability
between
¾ Metrics
for the
correctness,
reliability and scalability of IMS
components
Protocol
Interoperability
solution
ƒ
© FOKUS, 2006
¾ Objective performance
comparison for network/service
under real world conditions
providers
... to verify the performance of IMS solutions
ITU-T SG17 IOP Testing Workshop, Dec. 2006
Service Conformance
and Performance
www.fokus.fraunhofer.de/motion
IMS Performance Testing
T
E
S
T
ƒ
IMS
SYSTEM
Benchmarking
ƒ for comparison
ƒ
Load/stress
ƒ how system performs under load conditions
ƒ
Capacity testing
ƒ max load the system can handle before failing
ƒ
Scalability testing
ƒ to plan capacity improvements
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
Benchmark Procedure
DOC
(Design Objective Capacity of the SuT)
Estimated Maximum Load
Load
Step
Load
Ramp
Initial
Call
Load
Stability
Or Settling
Time
Rate
Increase
No
Load
© FOKUS, 2006
Time To
Stop
Initial
Load
Plus
Load
Plus
Load
Plus
Load
ITU-T SG17 IOP Testing Workshop, Dec. 2006
Plus
Load
www.fokus.fraunhofer.de/motion
IMS Test Bodies
ƒ ETSI
ƒ Methodology for IMS network integration testing
ƒ SIP testing
ƒ SIG
ƒ IMS benchmarking methodology
ƒ TISPAN
ƒ Standardisation of converged networks
ƒ IMS benchmark
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
FOKUS Open Source IMS
ƒ
a Home Subscriber Server
ƒ
3GPP compliant IMS Call Session Control Functions (CSCFs)
ƒ
a SIP2IMS gateway
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
FOKUS Test System
• Scenario Description Æ
• Standardized implementation-independent test notation
• Execution Platform Æ TTworkbench Enterprise
• Distributed Test Execution Platform
• Deploys, creates and coordinates distributed parallel
test components emulating user equipment on several hosts
• Synchronized traffic load, measurement and logging of results
• Visualization Æ TraVis
• Merge CSV files logged separately by Test Daemons
• Various graphs & statistics
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
First Benchmark Examples
5.1.2.1
5.2.2.4
5.3.2.1
sce
na r
ser
io
ver
Scenario 1.1 Successful
Initial Registration
Scenario 2.4-
Scenario 3.1 Successful
Succesful Call
Message Exchange
(SAPS)
(SAPS)
(SAPS)
kennicot
80
200
500
DTI
140
310
690
damascus
110
390
900
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
Example: Visualization of Fail Rate
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
Outline
ƒ
Overview
ƒ
Main Concepts
ƒ
TTCN-3 in a Nutshell
ƒ
TTCN-3 based test systems
ƒ
An Example: IMS Benchmarking
ƒ
Summary
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
TTCN-3
ƒ
Is
ƒ a test specification and implementation technique
ƒ open to various domains
ƒ open to various development and test processes
ƒ
For conformance, interoperability and performance testing!
ƒ
TTCN-3 is widely supported by tools
¾ TTCN-3 is the choice for technical testing and automated test execution
¾ It supports the development of telecommunication, Internet and IT
standards via well-defined test suites
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
TTCN-3 is ...
ƒ
adopted by ITU-T
ƒ
taken for numerous ETSI test suites
ƒ SIP
ƒ IPv6
ƒ SIP/ISUP
ƒ Hiperlan/Hiperman
ƒ
taken for tests of other consortia like AutoSar, MOST Forum, WiMax Forum
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
Gains from Using TTCN-3 Tools
What Users Say
ƒ
Users of TTCN-3 report that ...
ƒ productivity,
ƒ system quality and
... are doubled
ƒ test reuse
ƒ
SUT coverage is substantially increased
ƒ
Testers can concentrate on the test logic
ƒ i.e. what to test
ƒ
The test execution is fully automated
ƒ i.e. how to test technically
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
At the End:
Standardized Test Specifications
ƒ
Well-defined test notation for many testing applications
ƒ
Universally understood syntax and operational semantics
ƒ
Off-the-shelf tools
ƒ
Cheaper education and training costs
ƒ
Standardization, exchange and reuse of test suites
ƒ
Easier maintenance of test suites
¾ Transparency for the test process
¾ Increase of the objectiveness of tests
¾ Comparability of test results
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion
Thank You!
Any Questions?
Please be invited to the
4th TTCN-3 User Conference
@ Ericsson, Sweden, May 2007
© FOKUS, 2006
ITU-T SG17 IOP Testing Workshop, Dec. 2006
www.fokus.fraunhofer.de/motion