Test process analysis of
Gateway GPRS Support Node
Author:
Jorma Tuominen
Supervisor:
Nokia Networks
Professor Sven-Gustav Häggman
Agenda
• Introduction
•
•
•
GGSN in GPRS and 3G packet core network
GGSN in Nokia Intelligent Content Delivery system
Software development process and testing phases
• Research problem
• Research method
• Results
• Conclusions
17.1.2006
Jorma Tuominen
Test process analysis of Gateway GPRS Support Node
2 (15)
GPRS and 3G packet core network
Mobile ISP
News
Trading
Sports
Games
WAP
MMS
BTS
NMS
BSC
CG
LIG
2G SGSN
FW
IP
backbone
BS
RNC
GGSN
3G SGSN
FW
BG
Inter-PLMN
network
17.1.2006
Intranet
services
Internet
Jorma Tuominen
Test process analysis of Gateway GPRS Support Node
Corporate
customer
3 (15)
Nokia Intelligent Content Delivery System
ICD
LDAP
CG
GTP’
user data
GGSN
OSC
NSM
RADIUS or
DIAMETER
LDAP
user data
TA
user data
CA
user data
+ RADIUS
17.1.2006
Jorma Tuominen
Test process analysis of Gateway GPRS Support Node
4 (15)
SW development process and testing
phases
Customer/Business
Requirements
Customer
Acceptance
Network/System
Requirements
System Verification
SyVe
System Integration
System Program
Product Program
Product
Requirements
System Testing
ST
Product Architecture
Design
Functional Testing
FT
Functional
Specifications
Product Integration
PI
Detailed Design &
Specifications
Module Testing
MT
Implementation
17.1.2006
Jorma Tuominen
Test process analysis of Gateway GPRS Support Node
5 (15)
Research problem
Testing requires tradeoffs between cost, time and
quality.
How available testing time and resources can be used
as effectively as possible to find and correct the
critical defects from the product before it is released?
17.1.2006
Jorma Tuominen
Test process analysis of Gateway GPRS Support Node
6 (15)
Research method
• Participating in GGSN system testing
• Literature study of GPRS and 3G packet core network
elements and protocols
• Literature study of software process and software
testing
• Collecting and analyzing testing metrics
• Defect analysis
• Risk analysis
• Planning risk based testing
17.1.2006
Jorma Tuominen
Test process analysis of Gateway GPRS Support Node
7 (15)
Testing metrics
• Testing effort metrics
•
Distribution of working hours by test phases
• Defect metrics
•
•
Number of discovered defects per week
Percentage of “correction not needed” defect reports
• Test case metrics
•
•
Number of test cases
Percentage of not relevant test cases
• Testing effectiveness metrics
•
•
•
17.1.2006
Hours needed to discover one defect
Number of test cases needed to discover one defect
Defect Removal Effectiveness
Jorma Tuominen
Test process analysis of Gateway GPRS Support Node
8 (15)
50%
Testing metrics
examples
45%
40%
35%
30%
Rel1
Rel2
Rel3
25%
• Distribution of
working hours by
test phases
(lessons learned)
20%
15%
10%
5%
0%
PI
FT
ST
18
16
E3
E3.5
14
12
• Number of
discovered defects
per week
(real-time metrics)
PI
10
FT
8
ST
E4
Cu
6
E2
4
2
0
1
17.1.2006
Jorma Tuominen
3
5
7
9
11
13
15
17
Test process analysis of Gateway GPRS Support Node
19
21
23
9 (15)
25
27
29
Defect analysis
• Search for the error prone functionality areas
•
•
•
Defects discovered in the earlier product releases
Defects discovered in the earlier testing phases
Defects discovered by the customers
• Can be done after product release or during testing
• Pareto 80-20 phenomena
•
•
17.1.2006
Many software phenomena follow a Pareto distribution:
80% of contribution comes from 20% of the contributors
[Boehm 1989]
Examples:
• 20% of the software modules contribute 80% of the
errors.
• 20% of the errors cause 80% of the down time
Jorma Tuominen
Test process analysis of Gateway GPRS Support Node
10 (15)
Risk analysis
• System requirements must be prioritised
• What is the likelihood that this feature will fail to
operate?
• What would be the impact on the user if this feature
will fail to operate?
• The likelihood of failure depends on the following:
•
•
•
error proneness of the requirement
newness of the requirement
complexity of the requirement
• Likelihood of failure: 1 = very high, ... 5 = very low
• Impact on the user: 1 = severe, ... 5 = negligible
• Risk priority =
(Likelihood of failure) x (Impact on the user)
17.1.2006
Jorma Tuominen
Test process analysis of Gateway GPRS Support Node
11 (15)
Risk based testing
Risk
100%
Risk based testing
Traditional testing
50%
17.1.2006
Jorma Tuominen
E
E
2
1
Testing effort
Test process analysis of Gateway GPRS Support Node
12 (15)
Results
• Requirements have been divided into three groups:
•
High, medium and low risk requirements
• Test case titles will be planned for each requirement
• Test cases will be designed and executed as follows:
•
•
•
High risk requirements:
Medium risk requirements:
Low risk requirements:
100% of test cases
70% of test cases
30 % of test cases
• Test design and execution effort can be reduced about
30%.
• Effect on the quality of the product can not be
estimated yet, because the product is not yet
released.
17.1.2006
Jorma Tuominen
Test process analysis of Gateway GPRS Support Node
13 (15)
Conclusions and future work
• The availability of the real-time testing metrics should
be improved.
• Co-operation between the product development, the
testing and the customer support should be
strengthened.
• The most important future work is to harmonize the
usage of requirement management database, test
case database and defect tracking database.
17.1.2006
Jorma Tuominen
Test process analysis of Gateway GPRS Support Node
14 (15)
Thank You!
Questions?
17.1.2006
Jorma Tuominen
Test process analysis of Gateway GPRS Support Node
15 (15)
Additional slides
17.1.2006
Jorma Tuominen
Test process analysis of Gateway GPRS Support Node
16 (15)
Cost of defects
Customer/Business
Requirements
Customer
Acceptance
Phase where defect is
discovered
Requirements
Network/System
Requirements
System Verification
SyVe
1
Design
3-6
Coding
10
Development testing
15-40
System Program
Acceptance testing
30-70
Product Program
Production
System Integration
Product
Requirements
System Testing
ST
Product Architecture
Design
Functional Testing
FT
Functional
Specifications
Product Integration
PI
Detailed Design &
Specifications
Module Testing
MT
Implementation
17.1.2006
Relative cost to
correct a
defect
Jorma Tuominen
40-1000
Phase where defect is
discovered
Relative cost to
correct a
defect
Definition
1
High-level design
2
Low-level design
5
Code
10
Unit test
15
Integration test
22
System test
50
Post delivery
100+
Test process analysis of Gateway GPRS Support Node
17 (15)
Early involvement of the testers
• Important aspects of early involvement of the testers
are: “Testers need a solid understanding of the
product so they can devise better and more complete
test plans, designs, procedures, and cases. Early testteam involvement can eliminate confusion about
functional behavior later in the project lifecycle. In
addition, early involvement allows the test team to
learn over time which aspects of the application are
the most critical to the end user and which are the
highest-risk elements. This knowledge enables testers
to focus on the most important parts of the application
first, avoiding over-testing rarely used areas and
under-testing the more important ones.” [Dustin 2003].
17.1.2006
Jorma Tuominen
Test process analysis of Gateway GPRS Support Node
18 (15)
Thoughts about testing
• “Program testing can be used to show the presence of
bugs, but never to show their absence!”
[Dijkstra 1969].
• Testing completion criteria:
Risk of stopping testing = Risk of continuing testing
• “In most cases ‘what’ you test in a system is much
more important than ‘how much’ you test”
[Craig 2002].
• “Prioritise tests so that, when ever you stop testing,
you have done the best testing in the time available”
[ISEB 2003].
• No risk  No test
17.1.2006
Jorma Tuominen
Test process analysis of Gateway GPRS Support Node
19 (15)
References
•
Myers G. The art of software testing, USA, John Wiley & Sons, ISBN 0-471-04328-1, 1979,
177 p.
•
Hetzel B. Complete Guide to Software Testing, USA, John Wiley & Sons, ISBN 0-471-565679, 1988, 284 p.
•
Craig R., Jaskiel S. Systematic Software Testing, USA, Artech House Publishers, ISBN 158053-508-9, 2002, 536 p.
•
Dijkstra E. W. Structured programming. Proceedings of the Second NATO Conference on
Software Engineering Techniques, Rome Italy, NATO, 1969, pp. 84-88.
•
Dustin E. Effective Software Testing, USA, Addison-Wesley, ISBN 0-201-79429-2, 2003, 271
p.
•
Bender R. Requirements Based Testing presentation. Proceedings of StarEast 2005,
Software testing analysis & review conference, USA Orlando, Bender RBT Inc. 2005, 232 p.
•
Beizer B. Black-Box Testing: Techniques for Functional Testing of Software and Systems,
USA, John Wiley & Sons, ISBN 0-471-12094-4, 1995, 294 p.
•
Koomen T., Pol M. Test Process Improvement: A practical step-by-step guide to structured
testing, Great Britain, Addison-Wesley, ISBN 0-201-59624-5, 1999, 218 p.
•
Hutcheson M. Software Testing Fundamentals: Methods and Metrics, USA, John Wiley &
Sons, ISBN 0-471-43020-X, 2003, 408 p.
•
Boehm B. W. Software Risk Management, USA, IEEE Press, ISBN 0-8186-8906-4, 1989,
495 p.
•
Grove Consultants, ISEB testing foundation course material, Great Britain, Grove
Consultants, 2003, 98 p.
17.1.2006
Jorma Tuominen
Test process analysis of Gateway GPRS Support Node
20 (15)