Protocol Testing
Lesson 5
IHA præsentation
1
Outline for today
• Guidelines for testing protocols
• TTCN – A Standard Language For Testing Protocols
• Motivation
• The Language
• The usage
IHA præsentation
2
Guidelines for protocol testing
IHA præsentation
3
Guidelines
• Unit Testing
• tests if individual units of source code are fit for use.
• A unit is the smallest testable part of an application
• Integration Testing
• individual software modules are combined and tested
as a group
• Occurs after unit testing
• Coverage Testing
• describes the degree to which the source code has
been tested
IHA præsentation
4
Guidelines
• Smoke Testing
• Collection of written tests that are performed on a
system prior to being accepted for further testing
• Regression Testing
• seeks to uncover software regressions
• previously working software functionality stops working
as intended
• Conformance Testing
• to determine whether a system meets some specified
standard
IHA præsentation
5
Conformance Testing Framework
• Conformance testing is the process of testing the
extent to which implementations adhere to the
requirements stated in relevant standard or
specification
• Conformance Testing is functional black-box testing
• Functional refers to the correct functional behavior of
an Implementation Under Test (IUT)
• Black-box means that the internal structure of the IUT
remains hidden
IHA præsentation
6
Conformance Testing Framework
IUT – Implementation Under Test
UT – Upper Tester function
LT – Lower Tester function
PCO – Point of Control & Observation
TCP – Test Coordination Procedures
ASP – Abstract Service Primitive
IHA præsentation
7
Conformance Testing Framework
UT plays the role of a user that
makes use of the service
provided by the IUT
LT plays the role of the peer
entity of the IUT, i.e. the LT and
the IUT communicate in order to
provide the service to the IUT
IHA præsentation
8
Conformance Testing Framework
PCO is a standardized interface.
Typically the lower interface of an
IUT is accessible only from the
remote
Communication is always meant
to be asynchronous => PCO is
modeled as two FIFO queues
IHA præsentation
9
Conformance Testing Framework
TCP can be used to coordinate
the actions of LT and UT.
LT and UT can be on separate
testers.
IUT and LT communicates by
means of ASP.
PDUs are encoded in ASPs of
the underlying service.
IHA præsentation
10
Conformance Testing Framework
• Test case development starts with:
• Identification of a test purpose
• Description of a single requirement or set of
related requirements
• A test case is an implementation of a test purpose
IHA præsentation
11
Conformance Testing Framework
Test Case Scheme
A test case starts and ends in stable
testing states.
Preamble to take the IUT from stable
testing state to test state.
Test body is performed in order to
check the test purpose.
Verification to check test body
Postamble to drive the IUT into a
stable testing state again
IHA præsentation
12
Protocol Testing
• Until now we have:
• Developed our protocol layer including:
– Timers
– State - event model
– PDUs to be exchanged
• Interfaces to upper and lower layers
HOW DO WE TEST OUR PROTOCOL LAYER?
IHA præsentation
13
Protocol Testing
• Protocol Testing is mainly about:
• Testing the state – event model.
Our protocol layer
Test System
PDU (message)
??
Observe response
IHA præsentation
14
Protocol Testing
• Testing the state – event model
• Difficult part is when we send an unexpected message into our layer
Our protocol layer
Test System
Unexpected PDU (message)
??
•Ignore – take no action
•Reject – return an explicit denial to the sender
•Allow – take action to process the event
IHA præsentation
15
Test & Testing Control Notation Edition 3
(TTCN-3)
IHA præsentation
16
• Introduction
• What is TTCN?
• Why use TTCN?
• The TTCN-3 Language
• Background
• Example
• TTCN-3 Usage
• Where is it used?
• Future Outlook
IHA præsentation
17
What is TTCN?
•
Internationally standardized testing language for formally defining test
scenarios and their implementation.
•
Designed purely for testing
•
Tool independent
•
Flexible and powerful language
• Combined use with other languages like ASN.1
• Well-defined syntax, interchange format
• Different presentation formats (e.g. tabular and graphical)
• Support text-based protocols
• Support for IP protocols
IHA præsentation
18
What is TTCN?
•
In TTCN we can….
• Define Test Suites
• Define Test Cases
• Define Test steps
• Declare variables (test suite & test case variables)
• Declare timers
• Create PDUs
• Etc..
IHA præsentation
19
What is TTCN?
• TTCN Edition 3 is documented in a serie of ETSI Specification
• ES 201 873-1 (TTCN-3 Core Language)
• ES 201 873-2 (Tabular presentation format)
• ES 201 873-3 (Graphical presentation format)
• etc.
• etc.
IHA præsentation
20
Why Use TTCN?
•
Quality
• a well-tested product increases confidence in the product’s quality
•
Time/Cost Improvement
• TTCN is a language designed for protocol conformance testing
• TTCN is standardized
• TTCN is fairly simple
•
Formal
• strict rules, complete BNF and operational semantics
•
Abstract
• TTCN test suites are independent of an implementation
IHA præsentation
21
Application Areas
• New application areas
• Software Testing
• Text-based protocols…
• Additional communication paradigm
• Message-based communication
• Procedure-based communication
•
Different kinds of testing
• Functional testing
• Conformance testing (behavioral testing)
• Scalability testing…
IHA præsentation
22
TTCN Basics
•
Behavioral testing
• send stimulus, observe response
• was response received within specified time?
SUT
Test System
TTCN
Test Suite
Send Stimuli
Observe Responses
IUT
Implementation
Under Test
Result
PASS, FAIL or
INCONCLUSIVE
IHA præsentation
23
TTCN – Example
Minimal Test Configuration
query = (www.iha.dk, A)
SUT
P
P
MTC
Verdict = pass
answer = (www.iha.dk, 193.65.199.110,A)
Tester
testcase MyTestCase() runs on DNSTester {
P.send(query);
P.receive(answer);
setverdict(pass);
stop;
}
IHA præsentation
24
TTCN-3 test systems in a nutshell
•
•
•
TTCN-3 specifies a test but a test
system is needed for test execution
TRI and TCI standards define test
system architecture
• TTCN- 3 tools are required to
support internal interfaces
• Allows reuse of test platforms
with different tools but also for
different SUTs
A test system requires
• A TTCN-3 tool = TTCN-3
compiler and execution
environment (
)
• A test platform for a specific
device under test
Test System Executor
TTCN-3 Test System
Test Control
Logging
Codecs
TCI
[compiled]
TTCN-3 Executable
TRI
(
s+
s)
Note: Tools come with default
Test Control & Logging
IHA præsentation
SUT Adapter
Platform Adapter
System Under Test (SUT)
TCI = TTCN-3 Control Interface
TRI = TTCN-3 Runtime Interface
25
An example adaptation: A IPv6 test system
Test System Executor
TTCN-3 Test System
Parameter
File
TCI
Tool
Logging
IPv6
Codecs
[compiled]
IPv6 ATS
TTCN-3 Executable
TRI
Ethernet Adapter
Real-time Adapter
Open Source FreeBSD Router
IHA præsentation
26
User’s view of TTCN-3
ASN.1 Types &
Values
TTCN-3
Core
Language
Text format
Tabular
format
IIDL Types
XML Types
Graphical
format
TTCN-3 User
C/C++ Types
Presentation
formatn
Other Types &
Valuesn
IHA præsentation
27
The Core Language and Other Presentation Formats
Text format
Tabular
TTCN-3
Core
Language
Format
Graphical
Format
Presentation
Format3
Presentation
Formatn
IHA præsentation
•
Core format is text based
(most popular)
•
TTCN-3 can be edited or
viewed in other formats
• Tabular format
(for TTCN-2 people)
• Graphical format
(good for visual
overview)
• Other standardized
formats in the
future?
• Proprietary formats
possible
28
Example Core (Text) Format
testcase TC_resolveEtsiWww() runs on DnsClient
{
timer t_ack;
serverPort.send(m_dnsQuestion("www.etsi.org"));
t_ack.start(1.0);
alt {
[] serverPort.receive(mw_dnsAnswer("172.26.1.17")) {
setverdict (pass);
}
[] serverPort.receive { // any other message
setverdict(fail);
}
[] t_ack.timeout {
setverdict(inconc);
}
}
t_ack.stop;
}
IHA præsentation
29
Example Graphical Format
testcase TC_resolveEtsiWww()
runs on DnsClient
mtc
serverPort
DnsClient
DnsPort
timer t_ack
m_dnsQuestion("www.etsi.org")
t_ack
alt
mw_dnsAnswer("172.26.1.17")
pass
fail
?
t_ack
inconc
t_ack
IHA præsentation
30
Example Tabular Format
Testcase
Name
TC_resolveEtsiWww()
Group
Purpose
System Interface
MTC Type
DnsClient
Comments
Local Def Name
Type
t_ack
timer
Initial value
Comments
Behavior
serverPort.send(m_dnsQuestion("www.etsi.org"));
t_ack.start(1.0);
alt {
[] serverPort.receive(mw_dnsAnswer("172.26.1.17"))
{
setverdict (pass); }
[] serverPort.receive // any other message
{
setverdict(fail); }
[] t_ack.timeout
{
setverdict(inconc); }
}
t_ack.stop;
Detailed Comments:
IHA præsentation
31
Use of TTCN-3 With Other Languages
ASN.1 Types
•
TTCN can be integrated
with types systems of
other languages
•
Fully harmonized with
ASN.1 (1997)
•
Harmonized with other
languages
& Values
XML Types &
Values
IDL Types &
Values
TTCN-3
Core
Language
• IDL, XML, C/C++
Other types
& Valuesn
IHA præsentation
32
ASN.1
•
Abstract Syntax Notation One
• language for describing structured information
• standardized internationally (ISO/IEC 8824, ITU-T X.680)
•
TTCN includes ASN.1
• used for creating data descriptions and constraints
• can be part of TTCN script or imported as a module
•
SDL also includes ASN.1
• ITU-T Z.105
•
ASN.1
ASN.1 can be re-used across design and test!
TTCN
IHA præsentation
SDL
33
TTCN – Example
query = (www.iha.dk, A)
SUT
P
P
MTC
Verdict = ???
???
testcase MyTestCase() runs on DNSTester {
P.send(query);
P.receive(answer);
setverdict(pass);
stop;
}
Tester
• P.receive(answer) blocks until it receives a message that matches answer
• Any other message does not unblock the tester, which then blocks forever
• If no message is received, the tester will also block forever
IHA præsentation
34
TTCN Test Case Example
testcase MyTestCase2() runs on DNSTester {
testcase MyTestCase2() runs on DnsTester
timer t := 5.0;
P.send(query);
serverPort
Tester
SUT
timer t_ack
t.start;
query("www.iha.dk")
t_ack
alt {
[] P.receive(answer); {
setverdict(pass);
}
[] P.receive { // any message
setverdict(fail);
}
[] t.timeout {
setverdict(inconc);
}
}
stop;
mtc
alt
answer("172.26.1.17")
pass
fail
?
t_ack
inconc
t_ack
}
IHA præsentation
35
Building blocks of a TTCN-3 Test Suite
Test Suite
Test Data
Types
Data types which specify
• Structure of messages or calls and their
information elements (fields, parameters)
• Internal data structures (e.g., for computation)
• Possibly encoding or display information
Built-in basic types
integer, boolean, float,
bitstring, hexstring, octetstring,
charstring, universal charstring
... and structured types
record, record of, set, set of
union, enumerated
... and special types such as
component, port, verdicttype, default, etc
IHA præsentation
36
Building blocks of a TTCN-3 Test Suite
Test Suite
Test Data
Types
Actual
Test Data
Actual test data (values) used during
testing
• Constants or Templates for specific
message or call parameter values
• Matching expressions for allowing multiple
message or call parameter values
• value range, value list, wildcards,
presence, length, size, permutation
• regular expressions
• Using also template decomposition,
parameterization and modification
IHA præsentation
37
Building blocks of a TTCN-3 Test Suite
Test Suite
Test Data
Types
Actual
Test Data
Test
Configuration
Static aspects
• Test component and port types
Dynamic aspects
• Dynamic instantiation and
management of test components
• Mappings of test components to
abstract test system interfaces
• Connections between test
component interfaces
• Management of test components
IHA præsentation
38
Building blocks of a TTCN-3 Test Suite
Test Suite
Test Data
Types
Actual
Test Data
Test
Configuration
Test
Behaviour
test cases
• specify sending/receiving messages,
computation (e.g., checksums),
and verdict assignment
• can be decomposed with functions and
altsteps
• can (re)use default behaviour
• can use timers and timeouts
test execution control (optional)
• order, repetition, conditions, etc
IHA præsentation
39
TTCN-3 Module
Module (…)
module EtsiDnsTests
{
// Test definition part
Module
Definitions
control
{
// Test execution part
// (optional)
Module
Control
}
}
IHA præsentation
40
Definitions Part
module EtsiDnsTests
{
// Message structure
// Actual test data
// Test configuration
// Test Case definitions
}
IHA præsentation
41
Structured Definitions Part
module EtsiDnsTests
{
group MessageStructure
{
// Defintions of message types
}
group TestData
{
// Templates for messages instances
}
group TestSystemConfiguration
{
// Port and component types and mappings
}
group TestCases
{
// Test case definitions
}
}
IHA præsentation
42
Message Structure and Test Data
type record DnsMsg // simplified message structure!
{
DnsMsgKind kind,
charstring question,
charstring answer optional
}
type enumerated DnsMsgKind {e_query, e_response}
template DnsMsg m_dnsQuestion( charstring p_question )
{
kind := e_query,
question := p_question,
answer := omit // no answer
}
template DnsMsg mw_dnsAnswer( charstring p_answer )
{
kind := e_answer,
question := ?, // any question ok
answer := p_answer
}
IHA præsentation
43
Test Configuration
type port DnsPort message
{
inout DnsMsg
}
// Note: port types may also allow multiple different
//
message types or restrict the direction
type component DnsClient
{
port DnsPort serverPort
}
// Note: component types can also define multiple port
//
instances of the same or different port type and
//
declare timers, constants or variables
IHA præsentation
44
Test Behaviour
query
mtc
serverPort
response
testcase TC_resolveEtsiWww() runs on DnsClient
{
timer t_ack;
serverPort.send(m_dnsQuestion("www.etsi.org"));
t_ack.start(1.0);
alt {
[] serverPort.receive(mw_dnsAnswer("172.26.1.17")) {
setverdict(pass);
}
[] serverPort.receive { // any other message
setverdict(fail);
}
[] t_ack.timeout {
setverdict(inconc);
}
}
t_ack.stop;
}
IHA præsentation
45
The Control Part
Module (…)
module EtsiDnsTests
{
// Test definition part
modulepar boolean mp_example;
Module
Definitions
testcase TC_resolveEtsiWww()
runs on DnsClient
{
// .. as in previous slide
}
Module
Control
// Test execution part
control {
if (mp_example) {
execute(TC_resolveEtsiWww());
}
}
}
IHA præsentation
46
TTCN-3 Usage
IHA præsentation
47
TTCN in 3GPP
•
Lessons from GSM
• TTCN ATSs contribute to interoperability
• powerful competitive tool for entry to the global GSM mobile market
• TTCN ATSs widely used within members’ development processes
•
3GPP
• TTCN in Protocol Conformance Specs
• 34.123-1
Prose description of tests
• 34.123-2
Implementation Conformance Statement (ICS)
• 34.123-3
TTCN Abstract Test Suite
• TTCN will be the normative specification
IHA præsentation
48
TTCN and Bluetooth
•
Bluetooth SIG selected TTCN for...
• definition of Protocol Requirements tests
• interoperability
• definition of Profile Requirements tests
• application interoperability
• end-user expectations
Applications
TCP/IP
HID
RFCOMM
Data
Audio
L2CAP
Link Manager
Baseband
RF
IHA præsentation
49