Economics of automation in
functional testing of network
service platforms
Supervisor: Prof. Heikki Hämmäinen
Instructor: M.Sc. Reima Kaitajärvi
Mikko Lehtonen, Nokia Networks
Contents
•
•
•
•
•
•
•
•
Background
Research Problem and Research Methods
Testing, overview
Test automation
Cost efficiency model
Flexi Platform, overview
Results of the case study
Conclusions and further study
Background
• Test automation is an intriguing concept,
because it includes a promise of more
comfortable testing.
• The business interests in automation are
clear, e.g. to reduce the testing effort,
enable earlier release for publication and
save testing costs
Research Problems
and Research methods
– How test automation can be used to improve
testing in FlexiServer’s subcomponent Basic
IP Services?
– How to analyze costs and benefits of test
automation in FlexiServer’s subcomponent
Basic IP Services?
Methods: A literature study was performed to
gain the required know-how of the problem
domain. Case-study was performed to solve
the exact research problems.
Testing, overview
• The main goal of testing is to verify the
quality of the product
• Testing is consuming about 50% of
development resources and effort in many
software projects
• As the amount of source code increases
significantly, the testing costs seem to
increase even more need to cut costs
Test Automation (TA)
• One way to enhance the cost efficiency of
testing
• Problem:
– Test automation is not always an appropriate
solution
– The cost efficiency of TA is difficult to quantify
due to the intangible nature of many variables
(e.g. quality, improved professionalism)
Solution: the cost efficiency model
The model connects three perspectives:
• A procedure for choosing the most cost
effectively automated test areas (cost
efficiency calculations)
• defines ways to analyze the quality of
automated testing
• provides guidelines for implementation of
the testware
Cost efficiency calculations
• Formulas concentrate on tangible factors
i.e. financial values
• Intangible values must be considered
separately
• Two possible formulas:
CostEfficiency(t )  Costa  Costm
NPV  C0  t 1 Ct  (1  r ) t
T
Calculations (cont.)
• The most difficult thing is to chooce the correct cost
factors for a given situation (formulas quite simple)
• Principles on how to calculate the additional costs of
manual and automated testing
Ca (t)  (additiona l fixed costs of automated testing times (t/Useful Life) ) 
(variable costs of creating automated tests)
 (variable costs of creating manual tests) 
 (variable costs of maintainin g automated tests during time t)
Cm (t)  (additiona l fixed costs of manual testing times (t/Useful Life)) 
 (variable costs of running manual tests during time t)
(variable costs of running automated tests during time t)
FlexiServer arhitechture
• The BIPS implemented mostly in Linux + some in Nokia extensions
(e.g. SCTP stack)
• The BIPS implements protocols that enable the communication
between network elements (IP, TCP, UDP…)
cost efficiency
The results
4,5
4
3,5
3
2,5
2
1,5
1
0,5
0
BIPS_estimate
BIPS_actual_1
BIPS_averige
5
6
7
8
9 10 11 12 13 14 15 16
time (months)
Reasons for the differences:
1) Implementation took more time than estimated
2) Differences in manual test time (actual_1>average)
Results (cont.)
• The average manual testing time is used (probably more realistic)
• The figure on right side is calculated using discounted future costs
(NPV)
2
1,8
1,6
1,4
1,2
1
0,8
0,6
0,4
0,2
0
100
50
saved working hours
cost efficiency
150
0
1
2
3
-50
-100
-150
-200
1
2
3
time (years)
4
5
4
5
NPV
efficiency
time (years)
Summary result of BIPS
• The automation of BIPS was a quite small
scale project (2 person months)
– Obtained financial savings also small,
– intangible benefits (e.g. improved
professionalism) support financial savings
– The results are dependent from the test
environment
• Some of the tests of BIPS have to be still
performed manually.
Main conclusions
• The main results:
– the automation of BIPS
– the information of the economy of automation in that environment
– and the toolkit for analysing the cost efficiency of automation in a
defined project
• Some abstracted results:
• The general environment (organization,SUT,test processes,test
environment) affects to the economy of TA
• The suitable candidates for automation are tests or supporting
activities that are executed often, have a long lifetime and take long
time to perform manually.
• The choice what to automate first is essential, because the
implementation of TA takes that much time that not all the test can
be automated in the normal schedule of a test organization.
Further study
• The effects of automation to the quality of
testing (and product!) and test
thoroughness
• How the possible saved testing time is
consumed?
Questions?
Thank you for your interest!