LTE CONFORMANCE TESTING: EXPERIENCES IN A LONG TERM PROJECT
TTCN-3 User Conference 2011 – Bled, Slovenia
Presented by MCC TF160: Wolfgang Seka
Authors: Wolfgang Seka, Hellen Griffiths, Shicheng Hu
wolfgang.seka@etsi.org , hellen.griffiths@etsi.org , shicheng.hu@etsi.org
© ETSI 2011. All rights reserved
LTE Conformance Testing:
Experiences in a long term project
The Project
Technical Issues
Common Aspects of Long Term Projects
Conclusions
2
© ETSI 2011. All rights reserved
THE PROJECT
3
© ETSI 2011. All rights reserved
MCC TF160 - General
Task Force – Mobile Competence Centre: Project Group at ETSI
• Pool of TTCN expertise used by 3GPP
3GPP: 3rd Generation Partnership Project (http://www.3gpp.org)
→ Telecommunication Standardisation Bodies
→ TSG RAN: Radio Access Network
→ WG RAN5: Mobile terminal conformance testing
Conformance Tests
• Specification (Prose): RAN5
• Implementation (TTCN): MCC TF160
• Verification and Validation: Test Industry
MCC TF160: Signalling Conformance Tests for 3GPP (RAN5: Testing)
• Task: Develop Conformance Test Suites for UE world-wide certification
• since 2000
Conformance Tests for UMTS Signalling (TTCN-2)
since 2006
2007..2008
2008..now
4
© ETSI 2011. All rights reserved
Conformance Tests for IMS (TTCN-3)
Pre-evaluation of TTCN-3 for LTE Signalling
3GPP LTE/SAE UE Conformance Test
MCC TF160 – LTE/SAE Project
Size:
Duration:
Deliveries
Test cases:
17 experts all over the world
since 2008 and at least until 2015 (LTE-advanced will follow)
2011: 8 deliveries planned
264 approved
391 fully implemented
468 in total
80..100 planned for IMS
Code size:
• nearly 200 Modules
• nearly 190 000 Lines of code (TTCN-3)
Type Definitions
• 25 TTCN-3 modules, 3 ASN.1 modules, 6 XML modules
• TTCN-3: > 17 500 lines of code
• ASN.1: > 38 000 lines of code
• XML:
< 1 000 lines of code
Tools:
• 6 different compilers (all available at ETSI)
• quality checks (naming conventions, template restrictions etc.)
• code generation (top-level test case definitions, parameters, etc.)
• code analysis (structure, approved objects)
 TTCN-3 code is officially published and widely used
5
© ETSI 2011. All rights reserved
Component Structure - 1
MTC
• Start of PTCs, monitoring of “done” and “killed”
• RAT independent interfaces (e.g. AT-/MMI-commands to control the UE)
• In general no pass/fail verdicts
PTCs for each RAT (radio access technology)
• LTE, UMTS, GSM/GPRS, CDMA2000
• May be connected to any other RAT PTC
• Only RAT specific interfaces
• Assignment of pass/fail verdicts
PTCs for other protocols
• IP data (e.g. DHCP, ICMPv6), IMS and “protocol layer” underneath LTE PTC (NAS Emulator)
• To keep system interface simple and deal with “parallel behaviour”
• In general no test characteristic ( no pass/fail verdicts)
Ports and interfaces
• Connections: only one-to-one
•
•
6
( no “send to”, “receive from”; no use of address data type)
No duplicated interfaces: e.g. only one interface for AT-/MMI-commands hosted by the MTC
Message based communication only
© ETSI 2011. All rights reserved
7 7
UTRAN
PTC
Coordination
C2K_Ut
CDMA2000
PTC
Upper Tester
(AT , MMI)
G_Ut
DRB
SYSIND
SYS
NASCTRL
SRB
GERAN
PTC
CTRL
TC_SRB
E_Ut
C2K
GERAN
PTC
UT
(LTE)
GERAN
EUTRA PTC
CDMA2000
PTC
UTRAN
Component Structure - 2
MTC
Configuration
U_Ut
Signalling
NAS Emulator
E_DRB
E_SYSIND
E_SYS
SYS_SRB
Ut
E_SRB
Ut
System Interface
© ETSI 2011. All rights reserved
UTRAN
PTC
User Data
7
Test System
Test Control,
Logging
Host PC
SUT
System Simulator HW
RF
complex configuration
delay of messages
no matter what test
purpose is
Test Executable
UE
system
Codec
System/
Platform
Adaptor
specific
AT/MMI
 e.g. requirements regarding real-time behaviour for System Simulator and TTCN-3 code
8
© ETSI 2011. All rights reserved
TECHNICAL ISSUES
9
© ETSI 2011. All rights reserved
Real Time Issues - 1
Requirements
• Time resolution at the air interface: 1ms
• Due to significant delay between air interface and test case executable:
Timing needs to be monitored/controlled at the air interface (not at the TRI)
• All test equipment shall have similar behaviour at any time
• Test results shall not depend on
• how fast or slow a UE or system simulator is
• specific combinations of UEs and system simulators
System Restrictions
• Delay by codec and matching: up to some milliseconds;
•
•
typically < 5ms
Host system: typically Windows or Linux
 no real-time OS
Architecture: Host-PC + System Simulator
Estimation
• delay between test executable and air interface:
1ms < delay < 80ms
10
© ETSI 2011. All rights reserved
Real Time Issues - 2
Impact on System Interface
LTE timing information
• is part of all LTE primitives
• is provided by the System Simulator for all received messages
• is used to schedule sending of messages in the system simulator
• can be retrieved from the system simulator
Impact on writing TTCN-3
• Scheduled and non-scheduled sending shall not be mixed up
• When there is no reaction (e.g. from the UE) it is hard to know in the TTCN code when the
message has been sent out
• Scheduling vs. TTCN timers
 TTCN timers are not as accurate as scheduling
 Rules are required when to use TTCN timers or scheduled sending
Restrictions and Issues
• No common system time: uncorrelated timing between different RATs
• Legacy primitive definitions for UMTS and GSM/GPRS: less support of timing information
• Short duration only: For LTE, wrap-around after 10s
• No “bell” mechanism (yet):
to schedule a “wake-up call” in the system simulator to send back a trigger after a given time
11
© ETSI 2011. All rights reserved
Race Conditions
Race conditions occur when
• messages appear at the same time at different ports
• messages appear in any order at one port
Impact on TTCN implementation and Design considerations
• TTCN implementation
• interleave (in general not more than two messages)
• check operation
• evaluation of timing information
• Clever port definitions (few ports rather than many)
• Component structure
(e.g. signalling and data at different PTCs)
12
© ETSI 2011. All rights reserved
COMMON ASPECTS OF LONG TERM PROJECTS
13
© ETSI 2011. All rights reserved
Extendibility
Extensions of TTCN-3 code
• Modification of common objects shall not cause changes in the whole test suite
•
•
 reduced maintenance effort
Impact on writing TTCN-3
• foresighted TTCN implementation
• introduction of additional optional parameters
Example:
template MyType c_MyType :=
{
field1 := 1,
field2 := 2
};
template MyType c_MyType(integer p_Int := 1) :=
{
field1 := p_Int,
field2 := 2
};
Note: Optional Parameters vs. Modified Templates
• Optional parameters need to be added to modified templates too
• modification of templates and addition of optional parameters may serve the
same purpose
 possible conflicts, potential contradiction of both approaches
14
© ETSI 2011. All rights reserved
Backward Compatibility - 1
Backward Compatibility at the System Interface
• Modifications are caused by
• Release upgrade for signalling messages may have impact on configuration of
the system simulator
• New requirements (due to new test cases)
• Corrections
• Backward Compatibility Aspects
• Old TTCN-3 code on new system simulator implementation
• New TTCN-3 code on old system simulator implementation
• Impact
• system adaptor and codec
• interface definition
• top-level records or unions (even when not needed in the first place)
• definitions of place holders
15
© ETSI 2011. All rights reserved
Backward Compatibility - 2
Examples: Backward Compatibility at System Interface
type union MyUnion_Type := {
R8_Type R8
};
type union MyUnion_Type := {
R8_Type R8,
R9_Type R9
};
 New branch: branches are mutual exclusive; requires new templates
type record MyRecord_Type := {
R8_Type R8Only
};
type record MyRecord_Type := {
R8_Type R8Only,
R9Extension_Type R9Ext optional
};
 Extension: templates may be enhanced by optional parameter
16
© ETSI 2011. All rights reserved
Tools
Large, long-term project
 code reviews and manual checks are not sufficient anymore
Requirements
• Quality checks
• Approved objects: objects used by approved test cases shall not be changed
•
•
without change request
Analysis of module dependencies
Recursive selection
Enhanced tool support
•
•
•
•
17
Quality checks by T3Q (on behalf of ETSI)
Common tool to get dependencies based on T3D (on behalf of ETSI)
Project-specific front-ends (approved objects, recursive selection)
Further project specific tools:
replace tabs, find non-ASCII characters in comments
© ETSI 2011. All rights reserved
CONCLUSIONS
18
© ETSI 2011. All rights reserved
Conclusions (TTCN-3 language)
Important TTCN-3 language features
• optional parameters
 backward compatibility, maintenance
• template restrictions
 in combination with naming conventions and appropriate tools
 no problems with receive templates used in send direction anymore
• encvalue/decvalue
 maintenance (no external functions needed)
• check operation, interleave
 race conditions
• testcase.stop
 well-defined termination of a test case in case of unrecoverable error situations
• visibility: "private", "friend“
 not used yet (may improve quality regarding module dependencies)
19
© ETSI 2011. All rights reserved
Common Conclusions
Feedback of long-terms projects to the core language is vital
Tool Compatibility is needed
Shortened release cycle of TTCN-3 standards
TTCN-3 development is SW Development
 SW Engineering
 System Engineering
Long-term characteristic of a project needs to be considered from the beginning
 Conceptual errors will be expensive
Tool support is vital
 Common and project-specific tools
20
© ETSI 2011. All rights reserved
ADDITIONAL SLIDES
21
© ETSI 2011. All rights reserved
Release Upgrades - 1
Release Upgrade for Protocol and Signalling
(„Baseline Moving“)
• Typically every year
• In general changes are backward compatible
• Critical and non-critical extensions (TS 36.331 Annex A)
• Critical extensions
• redefinition of message content or parts of it
(branching, e.g. r8 and r9 branch)
• sender needs to know which release the receiver supports
• Non-critical extensions
• Provision of additional information
• e.g. new fields at the end of a record, new enumerated value
• when not supported receiver shall skip the information
• Messages in general contain hooks for critical and non-critical
extensions
22
© ETSI 2011. All rights reserved
Release Upgrades - 2
Impact on writing TTCN-3
• Non-critical extensions
• In general no code duplication is necessary
• Enhancement of existing templates by new optional parameters
(initialised with “*” for receiving and “omit” for sending)
• Critical extensions
• New templates are needed
• Challenge: minimised code duplication
• Example:
template of a plain RRC PDU for UMTS
 up to several hundred lines of code
 clever template structure reduces duplication of code
• parameterised templates
• modified templates
23
© ETSI 2011. All rights reserved