principles of
Test Automation
Linköping University
TDDD04 Software Testing, 2015-05-06
Thomas Thunell, Senior Specialist Test Automation, Ericsson
Who am I?
› Thomas Thunell, working in Ericsson at Product
Development, Mobile Systems in Linköping, Sweden.
› Working with Test Automation since ~2000.
Senior Specialist Test Automation since 2006.
› Current assignments:
2010 - Test Automation Area Manager
 2013 - Team lead GIC Orchestration
Thomas Thunell, Senior Specialist Test Automation
Ericsson AB
Datalinjen 3-4, P.O. Box 1248
S-581 12 LINKÖPING, Sweden
Phone: +46 10 7114566
mailto: thomas.thunell@ericsson.com
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 2
Abstract
Ericsson is one of the largest software companies in the
world and the use of Continuous Integration (CI) is high on
the agenda. One of the most important aspects of CI is
Test Automation. So how can we be successful in Test
Automation?
In this session Principles of Test Automation will be
presented and also the rationale behind them. These
principles covers for example tools, methods and test
environments within Ericsson, but can most likely be
applied and used in other companies. The principles can
guide You in the rather complex area of automating your
tests, from unit test up-to end-to-end testing.
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 3
Agenda
 Ericsson
 Principles of Test Automation
 Challenges today and beyond…
 Sum up
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 4
Ericsson
› Vision – the “Networked Society” , where every person and
every industry is empowered to reach their full potential.
› Founded in 1876 in Stockholm, Sweden; 120,000 employees
worldwide
› 40 percent of global mobile traffic runs through Ericsson
networks. More than 1 billion subscribers around the world rely
every day on networks managed by Ericsson.
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 5
More connections
50
15 years
50 billion connected
devices
Connections (billion)
40
30
25 years
5 billion connected
people
20
10
1875
100 years
1 billion connected
places
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 6
1975
2000
2020
Mobile Radio Networks
Laptop
Tablets
Internet
Radio
Networks
Core &
Packet
Networks
Mobile
Terminals
Fixed Telephony
Networks
This is an example of GSM, 3G and
4G networks.
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 7
Example of Mobile Radio Network
Test Environment
Mobile
simulator
GSM
Radio
Equipment
Mobile
Rack
BSC
SGSN
Air
Interface
Internet
Core Network
Real
Mobile
Test
Automation
Fixed
phone
Radio
Equipment
GGSN
WCDMA
Protocol
Analyzer(s)
RNC
SGSN
Core Network
Testers
FTP, mail,
web servers
Load
Generator(s)
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 8
Global ICT center (GIC)
*
Montreal
Stockholm
Linköping
Links to:
GIC video
Hans Vestberg video
* ICT = Information and Communications Technology
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 9
GIC & Engineering Environment
~25.000
R&D users
Engineering Environment
Activity
Test
Design
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 10
Global ICT
Centers
IT & Test
Environment
Operations
Test Automation
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 11
Magnitude and scale…
› In the past: a test case were run a few times per year…
› Today (in Agile and Continuous Integration): a test case (tc) is run
every week, every day, every hour (and by several teams)!
– Example: 1tc x 52weeks x 7days x 24hours = ~8700 times/year
› Assume we have 1000tc and 20 teams in a PDU:
- ~8700 x 1000 x 20 = ~17Million tc runs/year
› Assume we have 25 PDU’s:
- ~17M x 25 = ~425Million test case runs/year
› So Test Automation is mandatory for software development
in Ericsson!
PDU = Product Development Unit
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 12
Demands on TA
Test
Automation
Area
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 13
overview
Strategy
Test Libraries
Commitment
Test Automation
Support
Product
Development
Principles
Competence
Tool Set
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 14
Architecture
The principles
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 15
1. We have a Test Automation Strategy
that we follow
Rationale (Why?)
•Provides the big picture
•Set direction
•Guide us on:
-what we want to achieve, what to automate, goals
-measurements (KPI’s etc), business aspects
Success criteria (When?)
•Strategy is followed
•Continuously update and review our strategy
•Supports all aspects (test design, test execution, test evaluation)
Practical implications (How?)
•Local TA Strategies
•The Ericsson TA Strategy
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 16
Strategy
Test Automation Strategy Background
Co-operation
Common tools/frameworks
Component Based
Architecture for tools
Integrated Tools
Aligned ways-of-working
Goal
Test
Plan
Test
Design
Test
Implementation
and preparation
Test
Execution
Test
Evaluation
Shorter lead time
Test
Plan
Test
Design
Test
Implementation
and preparation
Test
Execution
Several tools/frameworks
Automation level:
High
Medium
Challenges:
Test
Evaluation
Different ways-of-working
Many interfaces
No common tools
architecture
Low
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 17
Lack of co-operation
Tools not integrated
2. We have continuous commitment
and engagement for Test Automation
Rationale (Why?)
•Key enabler for CI
•Reduce the risk for errors
•Increase the speed of Software delivery
•Test Automation is an investment
Success criteria (When?)
•Implement Continuous Integration (and Continuous Delivery)
•Dedicated resources
•Continuous learning
practical implications (How?)
•Money and resources
•Management and leaders should promote, recognize and support TA
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 18
Commitment
3. Our Test Automation regime
supports product development
ways-of-working
Rationale (Why?)
•Support our current way-of-working (e.g. Agile, CI)
•Test Environment used efficiently
Success criteria (When?)
•Feedback on product quality
•Short feedback loops
practical implications (How?)
•Synchronized with product development cycle
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 19
Product Development
Continuous Integration
Continuous Integration is a software development
practice where members of a team integrate their
work frequently, usually each person integrates at
least daily - leading to multiple integrations per day.
Each integration is verified by an automated build
(including test) to detect integration errors as
quickly as possible.
Martin Fowler
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 20
4. We are continuously increasing SW
development competence and skills
for Test Automation
Rationale (Why?)
•Produce more in shorter time
•Continuously improve within test
•TA can do repetitive, error prone tests and free up time for other
tests, like exploratory tests
Success criteria (When?)
•High quality, re-usable and maintainable code
•Automated test design (e.g. Model Based Testing)
•Automated test evaluation (and log analysis)
practical implications (How?)
•Educate and encourage people to learn SW development
•SW design guidelines
•Use e.g. pair-programming
•Explore automatic test design and test case generation
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 21
Competence
Example from
Professional service
› Goal: Support to new Test Automation users
› On site training
– Basic courses
– Advanced courses
› Face-2-Face, workshops
–Follow up, plan next steps, discuss issues and progress
report/demo for management.
› Online co-work (phone & video conferences)
› Involvement in programming (after on-site support)
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 22
Test Evaluation Automatic Log processing
› Automatic log collection and processing should be
used whenever it is possible.
A) Log Collection
B) Log Processing
Logs
Filters
Protocol
Analyzers
Collect logs
for storage
Database
File
system
FAIL
Log Viewers
Terminals
PASS
servers/
clients
Database File system
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 23
User
5. We develop a flexible Test
Automation architecture that
supports reusability
Rationale (Why?)
•Flexibility and speed
•Support several test levels, re-use components
•Add new tools/interfaces fast and easily
Success criteria (When?)
•Components developed, verified and released independently
•Re-use source code, binaries and other artifacts across products
•Extensible and flexible
practical implications (How?)
•Developed and improved iteratively
•Decoupled modules with open interfaces
•High standalone testability, e.g. able to test, deploy and release standalone
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 24
Architecture
Architecture
Test Automation
Users
Automated
Test Code
Development
Test Automation
Core
Test Environment
Test Automation
Support Services
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 25
Test Automation
Reference Architecture
Test Automation
Support Services
Components like Booking,
Scheduling
and Log Viewers.
Principles of Test Automation, LIU 2015-05-06
Tool
Interfaces
SUT
Interface
An environment that is used
to develop the automated test
Booking often an IDE like
code/cases,
Eclipse.
Tools
Test Execution
System
Scheduling
Equipment
Interfaces
Automated
Test Code
Development
Test Execution Client
Equipment
Support Services
Continuous
Integration
Engine
Testers
Teams
Node
Interfaces
IDE
Test Code
Repository
Test Automation Users
Nodes
Test Code Development
System Under Test
Test Environment
| © Ericsson AB 2015 | Page 26
Test Automation
Users
A user of Test Automation, could be a
tester or a CI engine.
Support Services
Test Automation
Core
This is the “engine” in Test
Automation where the
automated test cases are
executed and interfaces to
the test equipment.
Log/Result
Database
Statistics
Test Environment
The Test Environment consists of
SUT (System Under Test) and all
other tools (like instruments etc).
Test Automation
Reference Architecture
Test Execution Client
Tool
Interfaces
Tools
SUT
Interface
Booking
Support Services
Test Execution
System
Scheduling
Equipment
Interfaces
Support Services
Equipment
IDE
Continuous
Integration
Engine
Testers
Teams
Node
Interfaces
Test Code
Repository
Test Automation Users
Nodes
Test Code Development
System Under Test
Test Environment
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 27
Log/Result
Database
Statistics
Inner source development
Test Execution Client
Tool
Interfaces
Tools
SUT
Interface
Booking
Support Services
Test Execution
System
Scheduling
Equipment
Interfaces
Support Services
Equipment
IDE
Continuous
Integration
Engine
Testers
Teams
Node
Interfaces
Test Code
Repository
Test Automation Users
Nodes
Test Code Development
Small inner source communities
System Under Test
Test Environment
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 28
Log/Result
Database
Statistics
6. We constantly strive to use the
best available Test Automation tool
set
Rationale (Why?)
•Avoid technology and partner lock-in
•Reduce cost to adopt and replace tools
•Support tools applicable for many, still catering local needs
Success criteria (When?)
•Interfacing tools (like CI, Test Management, Test Equipment)
•Easy-to-learn, easy-to-use and reliable
•Scalable, used in several layers of testing (e.g. Unit, Component, System test)
practical implications (How?)
•Reduce dependencies to specific tools and tool vendors
•Integrate tools via open APIs
•Managed as a product with (tool manager, community, requirement and trouble report
handling, etc)
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 29
Tool set
Ericsson Tools / Frameworks
› JCAT (Java Common Auto Test)
 JCAT is an automation test framework which can support Unit Test, Component Test,
Function Test and up-to System Test.
 Built on-top of industry standard open source products, for example: TestNG and JUnit
› Titan (TTCN-3)
 Titan is a TTCN-3 compilation and execution environment with an Eclipse-based IDE.
 Titan is easy to learn and scalable to support several layers of testing
 Open source product: https://projects.eclipse.org/projects/tools.titan
› TEOF (Test Environment Orchestration Framework)
 New product suite for the new Global ICT Centers
 Focused on Automatic provisioning of dynamic and flexible Test Environments, SW
Load & Configuring, Booking & Scheduling.
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 30
What is TTCN-3?
› Standard test language (ETSI)
› Specifically designed for automated testing
› Used in several domains: telecom, automotive, avionics,
model based testing tools, test equipment vendors, HW
vendors, system integrators, standardization
See more at http://www.ttcn-3.org,
https://www.youtube.com/watch?feature=player_detailpage&v=nPfPC5inM_k
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 31
Main TTCN-3 features
› Easy to learn, easy to use
› Easy concurrency handling
› Test case verdict calculation
(Pass/Fail)
› Message matching
– Wide range of wildcards
– Incoming messages matched
against templates
– Can be specified on bit level
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 32
open source Titan
› In 2014 Ericsson submitted Titan to Open Source:
https://projects.eclipse.org/projects/tools.titan
› Why?
– Facilitate use of TTCN-3 in universities, research and industry
– Extend the community, receive feedback and contributions to the
language and to the tool
› What?
– A complete toolset with a selection of protocol adaptations
– Second biggest open source contribution of Ericsson ever
(after Erlang/OTP)
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 33
7. Our test libraries (testware)
supports rapid development of
Automated test cases
Rationale (Why?)
•High quality reusable test libraries
•Get new people up to speed faster
•Test libraries (testware) is an asset
Success criteria (When?)
•Uses normal SW practices:
-develop test libraries (testware) that are easy to enhance and maintain
-use the libraries in all layers of test (unit, component, system)
practical implications (How?)
•Version control
•Code review and automated checks
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 34
test Libraries
Test Automation
Reference Architecture
Test Execution Client
Tool
Interfaces
Tools
SUT
Interface
Booking
Support Services
Test Execution
System
Scheduling
Equipment
Interfaces
Support Services
Equipment
IDE
Continuous
Integration
Engine
Testers
Teams
Node
Interfaces
Test Code
Repository
Test Automation Users
Nodes
Test Code Development
System Under Test
Test Environment
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 35
Log/Result
Database
Statistics
Test Equipment Interfaces
Titan Test Ports (TTCN-3)
Node
Interfaces
Equipment
Interfaces
Tool
Interfaces
Nodes
Equipment
Tools
SUT
Interface
JCAT - Java Common Libraries
Several interfaces
must be handled:
* MML (Ericsson internal)
* SSH, Telnet
* Corba
* Tcl
NTAF is just one example on how to
System Under Test
* Java
to write automated test code/cases!
* Moshell (Ericsson internal)
Test Environment
* GPIB
* RMI
* SNMP
* SMTP
* HTTP
* and many more…
NTAF
make it easier
Current members:
Brocade, Cisco, Ericsson,
Google, Juniper, Spirent, iol,
Verizon, Netscout and
growing…
NTAF = Network Test
Automation Forum
http://www.ntaforum.org/
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 36
Ericsson
Test Automation principles
1. We have a Test Automation Strategy that we follow
2. We have continuous commitment and engagement for Test Automation
3. Our Test Automation regime supports product development ways-of-working
4. We are continuously increasing SW development competence and skills for
Test Automation
5. We develop a flexible Test Automation architecture that supports reusability
6. We constantly strive to use the best available Test Automation tool set
7. Our test libraries (testware) supports rapid development of Automated test
cases
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 37
Challenges today and
beyond…
 Continuous Integration & Continuous Deployment!
 Software Test & Development competence in writing
automated tests!
 Advanced automated selection of test cases and test
suites!
 Automatic Test Evaluation (log analysis…)!
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 38
Sum up
› Test Automation is one key enabler for
Continuous Integration!
› Test Automation is much more than tools!
› Test Automation should be managed as
any other SW development!
Principles of Test Automation, LIU 2015-05-06 | © Ericsson AB 2015 | Page 39