ETSI Testing Projects
ICT-OSA/Parlay Workshop
Brazil, March 2006
Anthony Wiles
Manager
ETSI Protocol and Testing Competence Centre
The ETSI PTCC
 Protocol and Testing Competence Centre
 In-house team of experts providing direct support to ETSI
Technical Bodies (for over 12 years)
 Development of protocols and profiles and the application
of modern specification techniques (35%)
 ‘Standards Engineering’
 Validation of standards (5%)
 E.g., testbed activities (IPv6, SIP/H.323)
 Development of test specifications (60%)




Conformance tests
Interoperability tests
Network Integration tests
Others (PHY, performance …)
 http://www.etsi.org/ptcc
ETSI Testing Projects - ICT Workshop, Brazil March 2006
3
Typical ETSI Test Specifications
 Cellular: GSM and 3G (UMTS) terminals
 Now includes IMS
 WiFi: HiperMAN, HiperACCESS, WiMax
 VoIP: H.323, SIP, SIGTRAN
 Service Creation: OSA/Parlay (API, IDL)
 IPv6: Core, Security, Mobility, v4-v6
 Cordless phones: DECT
 Radio communications: TETRA, DMR
 Access terminals: FSK, SMS
 Broadband: ISDN, DSL
 Smartcards: Readers, cards, security modules
 Intelligent Transport Systems (ITS): DSRC
 Future: More Security, more ITS, GRID ...
ETSI Testing Projects - ICT Workshop, Brazil March 2006
5
IPv6 Test Specifications
STF 276
 Joint ETSI – eEurope project
 Supported by the ETSI membership
 Mainly vendors
 Require IPv6 test specifications using ETSI methods
and test languages
 Designed to be part of 3GPP and TISPAN NGN IPv6
testing strategy (where applicable)
 European Commission:
 Mandate for IPv6 device interoperability
 Amplify Europe’s voice in IPv6 testing
 Liaising with the IPv6 Logo Program
 For more info visit: www.ipt.etsi.org
ETSI Testing Projects - ICT Workshop, Brazil March 2006
6
Two Phases (2005 – 2007)
 Phase 1: Core IPv6
 Total value approx. 0,5M€
 Jan – Dec 2005
 Phase2: IPSec, Mobile IPv6, IPv4-6 Transitioning
 Total value approx. 1,0M€
 Jan 2006 – May 2007
 Provided as




Funding
Provision of experts
Hosting costs
Travel
 All technical work done at ETSI by STF276
 Currently 11 participants in the STF
ETSI Testing Projects - ICT Workshop, Brazil March 2006
7
Outputs
 IPv6 Testing Framework
 Published (update due end 2006)
 IPv6 Requirements Catalogue
 On line as part of IPT Library
 Test specifications




Interoperability test specifications
Conformance test specifications
Library of TTCN-3 test code freely available
On-line, extensible and reusable
 All tests are validated (probably 30% of the effort)
 On ETSI IPv6 Testbed
 By member companies
 For more info visit: www.ipt.etsi.org
ETSI Testing Projects - ICT Workshop, Brazil March 2006
8
Results Phase 1
 Over 1100 Core IPv6 Requirements identified
 Including 3GPP
 300 Core IPv6 conformance tests
 fully validated
 120 Core IPv6 Interoperability Tests
 fully validated
 All these test specifications including TTCN-3
code are free and open source
ETSI Testing Projects - ICT Workshop, Brazil March 2006
9
STF276@Work
ETSI Testing Projects - ICT Workshop, Brazil March 2006
10
3GPP UMTS (terminal) Test
Specifications (STF 160)
 Nobody can make money without interoperable handsets (UE)!
 UE Interoperability towards access technologies is the key issue
for mobility
 Conformant UE = Interoperability (of UE in 3G networks)
 Good experience from GSM
 Solution: conformance testing
 Very detailed test specifications
 Based on the 3GPP core specs
 Before commercial products enter the market
 Voluntary declaration, but certified
• One of the most important UE quality benchmarks
ETSI Testing Projects - ICT Workshop, Brazil March 2006
11
Methodology for 3G Testing
Mobile Under
Test
SS
testing
Executable
Test Suite
(e.g., C++)
implementation
Core
specs
25.331,
25.321, 2
24.008,
44.018,
44.060
PICS
(capability
applicability)
34.123-2,
51.010-2
logging and
analysis
Test
Report
compilation
Test
Purposes,
34.123-1
Descript
(prose),
34.108,
51.010-1
Test Cases
(TTCN)
34.123-3,
51.010-5
ETSI Testing Projects - ICT Workshop, Brazil March 2006
12
3G Test Industry Commitment is Vital
 The TTCN test cases are implemented in the System
Simulators (SS) on different platforms
 Must secure the commitment of the SS vendors
 SS vendors organise verification and regression testing
 Each and every test case is validated
 Bug-free tests have a high real value
 Important to ensure that a TC will execute correctly even
after updates
 Early identification of clients who deploy the tests
 Prioritising TCs
 Scheduling the deployment
 Certifications
ETSI Testing Projects - ICT Workshop, Brazil March 2006
13
Funding
 Test specs developed at ETSI by STF160
 Duration of several years
 Shared costs to ensure the funding
 3GPP funding from Organisation Partners
 Main source, approx 1M€ (61 mm) /year)
 Voluntary
partners
contributions
from
the
marketing
 Range from 65k€ - 150k€ (5 – 12 mm) /year
 Man-power (free-of-charge)
 Range from 15 - 18 mm /year
ETSI Testing Projects - ICT Workshop, Brazil March 2006
14
Typical 3GPP UMTS UE Tests
 Protocol and signalling tests via radio
interface
 Largest part of the testing process
 Full conformance tests
 USIM functional tests
 Identification, PIN, account charging, network priority, phonebook
 UICC tests (SIM electrical test)
 power on/off, 1.8v/3v detection, presence detection
 bit/character transmitting duration, command processing and error
handling
 Acoustic tests
 Loudness rating, idle noise, echo …
 Radio Frequency & Radio Resource Management tests
 Characteristics and performance tests of transmitter & receiver
ETSI Testing Projects - ICT Workshop, Brazil March 2006
15
Users of 3GPP Tests
 Global Certification Forum (GCF)
 PCS Type Certification Review Board (PTCRB )
 GCF + PTCRB are the major clients for W-CDMA (FDD) &
GSM/GPRS test specs
 TD-SCDMA Industry Alliance (TDIA)
 TDIA is the major clients for TD-SCDMA (LCR TDD) test
specs
 GCF WI-10 selected 400 TCs as high priority
 Divided in 4 packages
 Progressing sequentially one by one (milestone 80%)
 UMTS mobile certification will be started if 80% of
each package are validated
ETSI Testing Projects - ICT Workshop, Brazil March 2006
16
Approval Process
 Test cases are approved by e-mail
 No quality compromise
 Two week consultation period
 TCs are approvedone by one
 Each test must run on at least one SS platform
against two independent commercial mobiles
 Log files and radio conditions as support information
for approval
 Reason for this
 TTCN TC, SS and UE are developed at the same time
 None of them is perfect
ETSI Testing Projects - ICT Workshop, Brazil March 2006
17
Results
 400 TCs in 8 ATSs, 25 MB code
 Platform independent
 Running on > 4 platforms
 We have reached the GCF 80% validated milestone
 3G mobile certification starting at beginning of 2005
 The light is green for mass 3G mobiles entering the markets
 4 types of 3G mobiles are ready to go
 Begun the real 3G test business
 4 validated 3G SS platforms available in the markets
 4 test houses in Europe opening 3G test services
 3G TTCN experts becoming hot in the labour market
 We know the price of each 3G coded TC
 10 k€ /TC
 3G investment for 4 years: 4 mil.€ for 400 TCs
 All TTCN code can be downloaded free of charge
 Code is continuously maintained and delivered
 Every three weeks for 3G members
 Every three months for everybody
ETSI Testing Projects - ICT Workshop, Brazil March 2006
18
VoIP-related Test Specifications
STF 269, STF 270 and others
 SIP (RFC 3261)
 Over 400 conformance test cases
 Fully validated (executables available on commercial tools)
 STF 270 is doing the maintenance
 H.225
 About 300 conformance test cases
 Fully validated (executables available on commercial tools)
 No ongoing maintenance
 SIGTRAN
 Test purposes for M3UA, M2UA, SCTP
 TTCN-3 available commercially (not ETSI)
 IMS SIP (together with SDP)




3GPP RAN5 (STF 160)
TTCN-3 will be used
Work started March 2006
Based on the generic SIP test suite (see above) augmented to
concentrate on 3GPP SIP specific requirements
ETSI Testing Projects - ICT Workshop, Brazil March 2006
19
ETSI – WiMAX Test Specifications
STF 252
 WiMAX Forum Board meeting in January 2005
 Agreed to harmonize the WiMAX/HiperMAN base
specifications
 Agreed to harmonize the WiMAX/HiperMAN test
specifications
 Agreed to exploit ETSI’s testing expertise
 Agreed to accelerate the work using the ETSI STFs
 Agreed that validation of the test specifications needed
• complex interplay of all parties involved
• WiMAX Forum, ETSI BRAN, equipment vendors, ETSI STF,
test tool developers, test labs, etc.
 ETSI WiMAX Co-operation agreement signed in 2005
ETSI Testing Projects - ICT Workshop, Brazil March 2006
20
Results
 Wave 1: Joint test specifications approved and published




Completed Dec. 2005
PICS – Protocol Implementation Conformance Statement
TSS&TP – Test Suite Structure and Test Purposes
ATS – Approx 60 test cases implemented, validated and used
in certification
 First WiMAX products have been certified
 Wave 2: New drafts reaching approval
 PICS – new updated version with corrections
 TSS&TP – new draft with close to 1400 Test Purposes
 ATS – new draft with close to 300 additional test cases
developed and entering validation
ETSI Testing Projects - ICT Workshop, Brazil March 2006
21
IPT Library
STF 295
 IP Testing Library




Developed by ETSI TC MTS
All MTS IP and IP-related conformance and IOP tests
PICS, TPs, Test Suites etc.
Due to be published April 2006
 SIP, IPv6, H.323, H.248
 Includes IPv6 Requirements Catalogue
 Linked to TPs and TTCN-3 code
 Powerful search and selection possibilities
 E.g., allows test suite tailored to specific search criteria
ETSI Testing Projects - ICT Workshop, Brazil March 2006
22
Screen Shot of IPT Library
ETSI Testing Projects - ICT Workshop, Brazil March 2006
23
PlugtestsTM
ETSI Interoperability Service
 Created in 1999, specialized in the organization of
test events
 Open to all organizations
 Experts from ETSI and other technical communities
 Skilled ETSI Staff managing test events
In 6 years, 66 Events
3500 engineers
1400 Companies
Usefulness &
Service Satisfaction
ranked at
4.35 / 5
ETSI Testing Projects - ICT Workshop, Brazil March 2006
24
Why?
 Provides useful feedback to the standardization process
 Accelerates time to market via quick product debugging
 Improves both quality and features of implementations
 Helps to ensure end users’ satisfaction
 Reduces deployment costs
 Allows networking with the market key players
COST & TIME SAVING!
ETSI Testing Projects - ICT Workshop, Brazil March 2006
25
Upcoming events in 2006
Q1 – 2006
Q2 – 2006
SIM/Handset
Lawful Interception
3rd WiMAX Plugfest
Digital Identity
Q3 – 2006
OSA Parlay
UWB
SIGTRAN
IPv6 Asia
4th SQTE EU round
NGN
Mobile Application
NGN@Home
Q4 – 2006
Human Factors (HF)
IPv6 French Summit &
Interop (FR)
Powerline
RFID
ITS ecall
ETSI Testing Projects - ICT Workshop, Brazil March 2006
26
Thank You!
Phone +33 (0)4 92 94 43 26
e-mail ptcchelp@etsi.org
Website www.etsi.org/ptcc
Anthony Wiles
ETSI Protocol and Testing Competence Centre