A conformance test suite for TTCN-3 tools
Benjamin Zeiss, T-Systems
Andras Kovacs, Broadbit
Bogdan Stanca-Kaposta, Testing Technologies
Agenda
 Motivation
 STF 409
 Methodology
 Discussion
 Challenges
 Conclusion
A conformance test suite for TTCN-3 tools
Slide 2
Motivation (1/2)
 Many TTCN-3 compilers (~7 commercial, 2 internal)
 High interest in the language
 Complex standard: hundreds of pages
 Estimation of ~5000 requirements
 No way to check the compliance of TTCN-3 tools
 TTCN-3 is based on ETSI standards
 A conformance test suite for TTCN-3 tools is needed!
 Quality of a test tool is critical
 Errors in the test tool may influece test results.
A conformance test suite for TTCN-3 tools
Slide 3
Motivation (2/2)
 Relevance:
 For users:
 Is my tool standards compliant?
 Can I switch tools easily / vendor lock-in?
 For tool vendors:
 Can I improve the standard compliance of my tool?
 Possibly later: compliance certification
 For standard developers:
 Where is need for clarification?
A conformance test suite for TTCN-3 tools
Slide 4
STF 409 (1/2)
 Specialist Task Force initiated by ETSI MTS
 Manpower: 3 persons w. 20 days each (60 days total)
 Benjamin Zeiss (T-Systems/University of Göttingen)
 Andras Kovacs (Broadbit)
 Bogdan Stanca-Kaposta (Testing Technologies)
 Dates: October 2010 – Early 2011
 Work is based on a framework given by TTCN-3 STF.
 Task:
 Devel. of a conformance test suite for TTCN-3 tools.
 Creation of a thorough documentation for the ATS.
A conformance test suite for TTCN-3 tools
Slide 5
STF 409 (2/2)
 Three deliverables:
 ETSI TS 102 950-1 V1.1.1 (2011-04) – ICS
 ETSI TS 102 950-1 V1.1.1 (2011-04) – TSS & TP
 ETSI TS 102 950-2 V1.1.1 (2011-04) – ATS & IXIT
 Approved by MTS in April 2011.
 Downloadable from the ETSI download area right now!
 Time spent:
 10 days each (30 work days)
 Development of conformance tests.
 10 days each + extra time (30+ work days):
 Validation, fixing of problems, documentation, tool
development.
A conformance test suite for TTCN-3 tools
Slide 6
Methodology (1/7)
 Five main steps:
1.
2.
3.
4.
5.
Create clause priority list. Discuss with MTS.
Develop conformance tests.
Create documentation framework.
Generate documentation content.
Validate and fix.
 Step 1 (creation of clause priority list):


Not all clauses can be covered. Selection has to be
made.
How to prioritize?
 Vendor feedback + most basic functionality.

There is no perfect selection!
A conformance test suite for TTCN-3 tools
Slide 7
Methodology (2/7)
 Step 2 (development of conformance tests):
 The ATS is not the test automation, but the test input!
 In most cases: 1 module = 1 conformance test.
 Two steps:
1.
2.
Specification of the test purpose: @purpose + @verdict
Implementation of the test in TTCN-3.
 Conformance verdict:

Expected tool output is specified in the document tags (@verdict).
 Purpose:

Also specified in the document tags (@purpose).
 Test automation is not provided by the ATS!


Can be scripted easily by extracting the document tags.
Example Java source code can be provided!
 Test suite structure: organized according to clauses.
A conformance test suite for TTCN-3 tools
Slide 8
Methodology (3/7)
 Step 2 (development of conformance tests) cont.:
 Expected output for verdict determination:
 @verdict pass accept/reject [expectedoutput]
 Examples:
 @verdict pass reject
 @verdict pass accept, ttcn3verdict:pass
 @verdict pass accept, noexecution
 Purpose specification:
 @purpose documentreference, description
 Documentreference: part:clause
 Examples:
 @purpose 1:5, Ensure that when the IUT loads a module
containing some definitions before the module declaration
then the module is rejected.
A conformance test suite for TTCN-3 tools
Slide 9
Methodology (4/7)
 Step 2 (development of conformance tests) cont.:
 Complete TTCN-3 test input example:
/***************************************************
** @purpose 1:5.2.2, Ensure that the IUT correctly
handles the uniqueness of
variable names in its scope.
** @verdict pass accept, ttcn3verdict:pass
***************************************************/
module Sem_050202_Uniqueness_001 {
import from Sem_050202_Uniqueness_001_import {
const all;
}
type component GeneralComp {}
function f_funcScope() {
var boolean repeatedIdentifier := true;
if(repeatedIdentifier==true) { setverdict(pass);
}
}
testcase TC_Sem_050202_Uniqueness_001()
runs on GeneralComp { f_funcScope(); }
control { execute(TC_Sem_050202_Uniqueness_001()); }
}
module Sem_050202_Uniqueness_001_import {
// repeated indentifier
// from imported module
// is allowed
const integer repeatedIdentifier := 0;
}
A conformance test suite for TTCN-3 tools
Slide 10
Methodology (5/7)
 Step 3 (creation of documentation framework):
 Already provided by STF393 (Proforma).
 Split single document into three documents.
 Step 4 (generate documentation content):
 Only 60 days manpower, little time for documentation.
 Working on a single source saves time.
 Solution:
 Write test purposes and expected tool outputs directly into
the test inputs.
 Generate tables for the documentation from the ATS:
 Test purpose tables.
 Test suite structure.
 Much easier to maintain!
 Documentation generation tools developed as part of
STF409.
A conformance test suite for TTCN-3 tools
Slide 11
Methodology (6/7)
 Step 5 (validation and fixing):
 Validation and fixing took more than half of the
development time!
 STF internal validation with 4 tools:




TTworkbench: compile-time and execution.
TestCast: compile-time and execution.
IBM: some initial compile-time validation.
MTP: as reports are sent in.
 Lots of feedback from Elvior and MTP!
A conformance test suite for TTCN-3 tools
Slide 12
Methodology (7/7)
 Executing the ATS:
TTCN-3
File
TTCN-3
File
TTCN-3 File
ATS
Sem_060101_TopLevel_001.ttcn
NegSem_060303_component_types_001.ttcn
…
Sem_060101_TopLevel_001.ttcn
** @verdict
pass accept, ttcn3verdict:pass
Test input
TTCN-3 Tool (test subject)
Expected
output
Tool output
Rejection as invalid
Execution with results
(e.g., pass/inconc/fail)
Rejection to execute
Tool output evaluation
pass
fail
A conformance test suite for TTCN-3 tools
Slide 13
Discussion
TTCN-3
Conformance ATS
TTCN-3 Tool
pass
Meaning?
 Compliance does not imply safety from a vendor lock-in!
 Testing is never complete!
 With v1.1.1 of the ATS, about 32% of the core standard clauses
are covered with at least one test.
 Ideally, we should have many tests for each clause.
 Compliance means less likelihood for vendor lock-in.
 With better coverage and higher number of test cases, we
improve the quality of this statement.
 Compliance reduces the likelihood for major tool errors
 Tests range from simple arithmetic operations to
communication operations and verdict handling.
 ATS checks syntax and semantics.
 ATS does not check the correctness of every single tool
message, but of the overall output.
A conformance test suite for TTCN-3 tools
Slide 14
Challenges (1/2)
 Various standard clarifications were necessary:
 9 CRs for TTCN-3 core languages have been written.
 Unclarified test cases have been excluded from the delivery
("on-hold").
 Tools sometimes handle the standard differently.
 Tests need to be expressed as tool independent as possible.
 Time management:
 Proper validation and fixing is extremely time consuming.
 More than 10 voluntary days have been actually spent.
 Responsibilities:
 Everyone mixes up STF393 and STF409 responsibilities
(proforma).
A conformance test suite for TTCN-3 tools
Slide 15
Challenges (2/2)
 Version numbers:
 To which core language standard version does the
conformance test suite relate to?
 How to maintain compatibility across different language
versions?
 Against which version should the conformance test
suite be developed in the future?
 Target quality:
 The target quality of the test suite should be fixed right
in the beginning (Terms of Reference).
A conformance test suite for TTCN-3 tools
Slide 16
Conclusion
 The conformance test suite is available now!
 ATS Metrics:







Total number of conformance tests: 733
Number of positive syntactic conformance tests: 110
Number of negative syntactic conformance tests: 34
Number of positive semantic conformance tests: 335
Number of negative semantic conformance tests: 254
Total number of clauses in the TTCN-3 standard: 342
Total number of clauses (at least partially) covered: 110
 About 140 pages of test purposes.
 Free to use for tool developers. Use it to improve your tool!
 Bug reports: http://t-ort.etsi.org
A conformance test suite for TTCN-3 tools
Slide 17
Special thanks:
Tool validation and comments
Elvior, MTP, Conformiq, Ericsson, ISPRAS
Jens Grabowski / University of Göttingen
Questions? Feedback?
A conformance test suite for TTCN-3 tools
Slide 18