University of Southern California
Center for Systems and Software Engineering
Value-Based Software Test Prioritization
University of Southern California
Qi Li, Barry Boehm
{qli1, boehm}@usc.edu
November 4, 2011
University of Southern California
Center for Systems and Software Engineering
Outline
•
•
•
•
6/27/2016
Research Motivation
Research Method
Case Studies
Conclusion
COCOMO Forum 2011
2
University of Southern California
Center for Systems and Software Engineering
Research Motivation
• Value-neutral SE methods are increasingly risky [Boehm, 2003]
– Every requirement, use case, object, test case, and defect is equally important
– “Earned Value” Systems don’t track business value
– System value-domain problems are the chief sources of software project failures
• Testing & Inspection resources are expensive and scarce
– 30%-50%, even higher for high reliability projects [Ramler, 2005]
– Time-to-market [Boehm, Huang, 2005]
• Empirical Findings [Bullock 2000, Boehm & Basili 2001 ]
–
–
–
–
–
About 20 percent of the features provide 80 percent of business value
About 80 percent of the defects come from 20 percent of the modules
About 80 percent of avoidable rework comes from 20 percent of the defects
About 90 percent of the downtime comes from, at most, 10 percent of the defects
…
6/27/2016
COCOMO Forum 2011
3
University of Southern California
Center for Systems and Software Engineering
Outline
•
•
•
•
6/27/2016
Research Motivation
Research Method
Case Studies
Conclusion
COCOMO Forum 2011
4
University of Southern California
Center for Systems and Software Engineering
Fundamental Theory
• Value-based Software Engineering 4+1 theorem [Boehm, 2005]
Dependency
Theory
How do dependencies
affect value realization?
Utility Theory
What values are important?
How is success assured?
How important are the
values?
Theory W:
SCS Win-Win
How to adapt to change and
control value realization?
How do values determine
decision choices?
Control Theory
6/27/2016
Decision Theory
COCOMO Forum 2011
5
University of Southern California
Center for Systems and Software Engineering
Value-Based Software Test Prioritization
What to be prioritized?
•Testing items: Testing Scenarios, Testing Features, Test Cases
How to prioritize?
•Value-Based (Business Importance, Risk, Cost)
•Dependency Aware
How to Measure?
•Average Percentage of Business Importance Earned (APBIE)
University of Southern California
Center for Systems and Software Engineering
Research Method: Value-Based
• Risk Exposure (RE)
– Where Size (Loss) is the risk impact size of loss if the
outcome is unsatisfactory, Pro (Loss) is the probability
of an unsatisfactory outcome
• Risk Reduction Leverage (RRL)
– Where REbefore is the RE before initiating the risk
reduction effort and REafter is the RE afterwards.
– RRL is a measure of the cost-benefit ratio of
performing a candidate risk reduction or defect
removal activity
6/27/2016
COCOMO Forum 2011
7
University of Southern California
Center for Systems and Software Engineering
Research Method: Value-Based
• Value-Based Prioritization Drivers:
– Business Case Analysis
– Business Value
– Stakeholder
Prioritization
– Impact of Defect
– Size of
Loss
Defect Criticality
Risk Exposure
– Experience Base
6/27/2016
– Defect-prone
Components,
Performers
COCOMO Forum 2011
– Probability
of Loss
8
University of Southern California
Center for Systems and Software Engineering
Research Method: Value-Based
• Objects are to be ranked by how well they can reduce risk
exposure
• Combining with their relative option costs
• =>Priority Trigger:
• This proposed strategy enables them to be prioritized in
terms of Risk Reduction Leverage (RRL) or ROI
• Supposed to improve the lifecycle cost-effectiveness of
defect removal techniques
6/27/2016
COCOMO Forum 2011
9
University of Southern California
Center for Systems and Software Engineering
Research Method: Dependency Aware
Value of software product to organization
Natural speech input
Tertairy application
functions
Animated displays
Secondary application functions
User amenities
Main application functions
Operating System
Investment
Basic application functions
Data management system
High-payoff
Diminishing returns
Cost of software product [Boehm, 1981]
6/27/2016
COCOMO Forum 2011
10
University of Southern California
Center for Systems and Software Engineering
Research Method: Dependency Aware
• Dependency:
– Example: dependencies
among test cases to be
executed
– Solution: Prioritization
Algorithm (greedy alg)
• Select the one with the
highest RRL
• Check dependency
6/27/2016
COCOMO Forum 2011
9->3->9->5->9->4->7
11
University of Southern California
Center for Systems and Software Engineering
Research Method: Metrics
• Testing Cost Effectiveness
– Average Percentage of Business Importance Earned (APBIE)
6/27/2016
COCOMO Forum 2011
12
University of Southern California
Center for Systems and Software Engineering
Outline
•
•
•
•
6/27/2016
Research Motivation
Research Method
Case Studies
Conclusion
COCOMO Forum 2011
13
University of Southern California
Center for Systems and Software Engineering
Case Studies
• Exercise Test Prioritization based on Risk
Reduction Level (RRL)
 software testing scenarios to be walked through
 software features to be tested
 software test cases to be executed
6/27/2016
COCOMO Forum 2011
14
University of Southern California
Center for Systems and Software Engineering
Case Studies
Study 1: Prioritize testing scenarios to be walked through
[Li TR1 2011]
• Case Study:
– Galorath Inc. (2011 Summer)
– Project: Installation Process Automation
– Challenge: Testing all scenarios is impossible under limited
testing resources (69 scenarios)
6/27/2016
COCOMO Forum 2011
15
University of Southern California
Center for Systems and Software Engineering
Case Studies
Study 1: Prioritize testing scenarios to be walked through
• Case Study Results:
100.00%
90.00%
Value-based 100.00%
93.83%
90.12%
95.06%
83.95%
87.65%
77.78%
80.00%
Value-neutral
70.00%
74.07%
58.02%
61.73%
60.00%
58.02%
51.85%
50.00%
PBIE-1
45.68%
40.00%
Value-inverse
(worst case)
39.51%
30.00%
– Value-based prioritization
can improve the costeffectiveness of testing
PBIE-2
PBIE-3
35.80%
25.93%
20.00%
22.22%
16.05%
10.00%
9.88%
4.94%
6.17%
0.00%
8
10
6/27/2016
12
14
16Stop 18
Testing20
22
24
26
28
APBIE-1
70.99%
APBIE-2
10.08%
APBIE-3
32.10%
30
COCOMO Forum 2011
16
University of Southern California
Center for Systems and Software Engineering
Case Studies
Study 2: Prioritize software features to be tested
[Li ICSP 2009]
• Case Study:
– Institute of Software, Chinese Academy of Sciences
(2008 Spring)
– System testing of Version3.1 SoftPM that covers 9
features
6/27/2016
COCOMO Forum 2011
17
University of Southern California
Center for Systems and Software Engineering
Case Studies
Study 2: Prioritize software features to be tested
• Case Study Results
99.9%
92.5%96.2%
87.6%
81.4%
75.3%
69.1%
69.1%
59.2%
100.0%
Value-based
90.0%
80.0%
PBIE
70.0%
60.0%
50.0%
40.0%
30.8%
30.0%
20.0%
10.0%
3.7% 7.4%
0.0%
1
2
3
40.7%
30.9%
24.7%
18.5% Value-inverse
12.3%
(worst case)
4
5
6
7
8
– Value-based prioritization
can improve the costeffectiveness of testing
9
Features
6/27/2016
APBIE-1
76.9%
APBIE-2
34.1%
COCOMO Forum 2011
18
University of Southern California
Center for Systems and Software Engineering
Case Studies
Study 3: Prioritize software test cases to be executed
[Li TR2 2011]
• Experiment
– USC-CSCI 577b (2011 Spring)
– 5 Real Client Course Projects
– Acceptance testing phase
6/27/2016
COCOMO Forum 2011
19
University of Southern California
Center for Systems and Software Engineering
Case Studies
Study 4: Prioritize software test cases to be executed
• Experiment Results (Quantitative)
APBIE-1
81.9%
100.0%
Value-based
90.0%
80.0%
PBIE
70.0%
– Project 1 as an example,
– Value-based prioritization
can improve the costeffectiveness of testing
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
APBIE-2
60.00%
APBIE-3
50.38%
100.0%
90.0%
80.0%
PBIE
70.0%
Value-based
60.0%
50.0%
40.0%
Value-neutral
30.0%
20.0%
10.0%
0.0%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
Test Case Order
6/27/2016
COCOMO Forum 2011
20
University of Southern California
Center for Systems and Software Engineering
Case Studies
Study 3: Prioritize software test cases to be executed
• Experiment Results (Observance)
Defect ID in
Bugzilla
Severity
Priority
Test Case
ID
BI
FP
#4444
Critical
Resolve
Immediately
TC-04-01
VH
0.7
#4445
Major
Normal Queue
TC-04-03
H
0.7
#4460
Major
Normal Queue
TC-05-10
H
0.7
#4461
Major
Resolve
Immediately
TC-18-01
H
0.7
– Defects with higher Priority and Severity are reported earlier and
resolved earlier
6/27/2016
COCOMO Forum 2011
21
University of Southern California
Center for Systems and Software Engineering
Case Studies
Study 4: Prioritize software test cases to be executed
• Experiment Results (Qualitative)
“Before doing the prioritization, I had a vague idea of which test cases are
important to clients. But after going through the Value-Based testing, I had a
better picture as to which ones are of critical importance to the client.”
“I prioritized test cases mainly based on the sequence of the system work
flow, which is performing test cases with lower dependencies at first before
using value-based testing. I like the value-based process because it can
save time by letting me focus on more valuable test cases or risky ones.
Therefore, it improves testing efficiency.”
“Value-based testing is very useful in complex systems with hundreds or
thousands of test-cases. However in 577 it should not be difficult to run
every test-case in every test iteration, making the prioritization less useful.
The impact of value-based testing and automated test management on
software quality is entirely dependent on the complexity of the project. If
complete test coverage is possible in the time given, the benefit of VBST to
software quality is minimal.”
6/27/2016
COCOMO Forum 2011
22
University of Southern California
Center for Systems and Software Engineering
Outline
•
•
•
•
6/27/2016
Research Motivation
Research Method
Case Studies
Conclusion
COCOMO Forum 2011
23
University of Southern California
Center for Systems and Software Engineering
Conclusion
• Propose a Real “Earned Value” System to Track Business Value
of Testing and Measure Testing Efficiency in terms of APBIE
• Propose a Systematic Strategy for Value-based, Dependency
Aware Test Processes
• Apply This Strategy to Series of Empirical Studies with different
granularities of Prioritizations
• Elaborate Decision Criteria of Testing Priorities Per Project
Contexts, Which are Helpful for Real Industry Practices
6/27/2016
COCOMO Forum 2011
24
University of Southern California
Center for Systems and Software Engineering
Backup Slides
6/27/2016
COCOMO Forum 2011
25
University of Southern California
Center for Systems and Software Engineering
Tight Dependency
3
5
9
4
9
9->3->9->5->9->4->7
9
7
University of Southern California
Center for Systems and Software Engineering
Tight Dependency
5
5
5
Path 1
Path 1->Path 2
1
2
9
Path 2
University of Southern California
Center for Systems and Software Engineering
Study 1. Value-based Testing:
Prioritize Testing Scenarios
6/27/2016
COCOMO Forum 2011
28
University of Southern California
Center for Systems and Software Engineering
Local Installation
Virtual Machine working on the host on the
same row
Host Machine
FU
RP
C
TP (RRL)
Winxp (x32)
5
0
1
0
Win7 (x64)
5
0.3
1
1.5
Vista (x32)
Vista (x64)
Win7 (x32)
3
1
3
0.7
0.7
0.5
2
5
5
1.05
0.14
0.3
FU
RP
C
TP
(RRL)
Vista (x32)
Winxp (x32)
3
5
0.9
0.3
2
2
1.35
0.75
Win7 (x32)
Vista(x32)
Winxp (x32)
5
3
1
0.5
0.9
0.9
2
2
2
1.25
1.35
0.45
Server Installation
VM
Win 7
(x64)
6/27/2016
WinServer
2003x32
WinServer
2008x64
WinServer
2008x32
FU
RP
C
TP
(RRL)
5
0.3
2
0.75
5
0.5
2
1.25
3
0.3
2
0.45
COCOMO Forum 2011
Product
Product Importance
SEER-MFG
2
SEER-H
2
SEER-SEM
3
29
University of Southern California
Center for Systems and Software Engineering
Study 2. Value-based Testing:
Prioritize Testing Features
6/27/2016
COCOMO Forum 2011
30
University of Southern California
Center for Systems and Software Engineering
Business Importance (BI)
•Use Karl Wieger’s Method to get the relative BI for each feature
•BIi= WBenefit* Benifiti +WPenalty* Penaltyi
35.00%
30.00%
30.86%
28.40%
25.00%
20.00%
15.00%
9.88%
10.00%
6.17% 6.17%
6.17%
4.94%
3.70%
5.00%
3.70%
0.00%
F1
F2
F3
F4
F5
F6
F7
F8
Business Importance
6/27/2016
COCOMO Forum 2011
31
F9
University of Southern California
Center for Systems and Software Engineering
Quality Risks & Weights Allocation
Analytical Hierarchy Process (AHP)
6/27/2016
COCOMO Forum 2011
32
University of Southern California
Center for Systems and Software Engineering
Quality Risks Probability
n
Pi 
6/27/2016
R
j 1
i, j
*Wj
9
COCOMO Forum 2011
33
University of Southern California
Center for Systems and Software Engineering
Testing Cost
25.00%
21.43%
20.00%
14.29%
15.00%
11.90%11.90%
11.90%
9.52%
10.00%
5.00%
7.14% 7.14%
4.76%
0.00%
F1
F2
F3
F4
F5
F6
F7
F8
F9
Cost
6/27/2016
COCOMO Forum 2011
34
University of Southern California
Center for Systems and Software Engineering
Put the three together: Value Priority
0.90
0.81
0.80
0.70
0.63
0.60
0.50
0.40
0.30
0.20
0.10
0.09
0.07
0.05
0.03
F3
F4
F5
0.14
0.04
0.02
F8
F9
0.00
F1
6/27/2016
F2
F6
COCOMO Forum 2011
F7
35
Priority
University of Southern California
Center for Systems and Software Engineering
Study 3. Value-based Testing:
Prioritize Test Cases
6/27/2016
COCOMO Forum 2011
36
University of Southern California
Center for Systems and Software Engineering
Step1: Dependency Analysis
Start
TC-01-01(3 ,VH )
TC-03-01 (3 , VH )
TC-04-01 (5 , VH )
TC-05-01
( 4,N )
TC-11-01 (4 , VL )
6/27/2016
TC-03-02 ( 1,VL )
TC-04-02 (4, VL )
TC-05-02
( 4,VL )
TC-13-01 (4 , VL )
TC-05-03
( 4,VL )
TC-03-03 (1 ,VL )
TC-04-03 (4 , VL )
TC-05-05
( 4,VL )
TC-16-03 (2 , VL )
TC-01-02 (2 ,VL )
TC-03-04 (1,VL)
TC-14-01 (3 , VL )
TC-05-07
( 4,VL )
TC-05-08
( 4,VL )
TC-02-01 (3 ,VL )
TC-02-02 (2,VL)
TC-15-01 (2 ,VL )
TC-16-01 (2 , VL )
TC-05-10
( 4,VL )
TC-12-01
(4 ,VL )
TC-16-02
(2 , VL)
TC-18-01
( 4, VL)
TC-19-01 (4 , VL)
COCOMO Forum 2011
37
University of Southern California
Center for Systems and Software Engineering
Step 2:Rate Business Importance
VH:5
H:4
N:3
L:2
VL:1
6/27/2016
This test case is used to test the functionality that will bring the Very High
benefit for the client, without passing it, the functionality won’t run
This test case is used to test the functionality that will bring the Very High
benefit for the client, without passing it, the functionality can still run
This test case is used to test the functionality that will bring the High benefit for
the client, without passing, the functionality won’t run
This test case is used to test the functionality that will bring the High benefit for
the client, without passing it, the functionality can still run
This test case is used to test the functionality that will bring the Normal benefit
for the client, without passing it, the functionality won’t run
This test case is used to test the functionality that will bring the Normal benefit
for the client, without passing it, the functionality can still run
This test case is used to test the functionality that will bring the Low benefit for
the client, without passing it, the functionality won’t run
This test case is used to test the functionality that will bring the Low benefit for
the client, without passing it, the functionality can still run
This test case is used to test the functionality that will bring the Very Low benefit
for the client, without passing it, the functionality won’t run
COCOMO Forum 2011
38
University of Southern California
Center for Systems and Software Engineering
Step 3: Rate Criticality
Block most (70%-100%) of the test cases, AND most of those
blocked test cases have High Business Importance or above
Block most (70%-100%) of the test cases, OR most of those
H:4
blocked test cases have High Business Importance or above
Block some (40%-70%) of the test cases, AND most of those
N:3
blocked test cases have Normal Business Importance
Block a few (0%-40%) of the test cases, OR most of those
L:2
blocked test cases have Normal Business Importance or below
VL:1 Won’t block any other test cases
VH:5
6/27/2016
COCOMO Forum 2011
39
University of Southern California
Center for Systems and Software Engineering
Step 4: Rate Fail Probability
Experience
Did the test case fail before? --People tend to repeat
previous mistakes, so does software. From pervious
observations, e.g. unit test, performance at CCD, or
informal random testing, the test case failed before tends
to fail again
Is the test case new? --The test case that hasn’t not been
tested before has a higher probability to fail
Change Impact Does any recent code change (delete/modify/add) have
impact on some features? --if so, the test cases for these
features have a higher probability to fail
Personnel
Are the people responsible for this feature qualified? -- If
not, the test case for this feature tends to fail
Complexity
Does the feature have some complex algorithm/ IO
functions? --If so, the test case for this feature have a
higher probability to fail
Dependencies Does this test cases have a lot of connections (either
depend on or to be depended on) with other test case? -If so, this test case have a higher probability to fail
6/27/2016
COCOMO Forum 2011
40
University of Southern California
Center for Systems and Software Engineering
Research Proposal: Dependency Aware
• Prioritization Algorithm
– Value First:
• Test the one with the highest Testing Value. If several test cases’
Test Values are the same, test the one with the highest Criticality
– Dependency Second:
• If the test case selected from the first step is not “Ready-to-Test”,
which means at least one of the test cases in its Dependencies Set
is “Not-Tested-Yet”. In such situation, prioritize the “Not-TestedYet” test cases according to “Value First” in this Dependencies Set
and start to test until all test cases in the Dependencies Set are
“Passed”. Then the test case with the highest value is “Ready-toTest”
– Update the prioritization:
6/27/2016
• After one round, update the Fail Probability based on updated
observation from previous testing rounds
COCOMO Forum 2011
41
University of Southern California
Center for Systems and Software Engineering
Research Proposal: Dependency Aware
Pick the one with the
highest Test Value (if the
same, choose the one
with higher Criticality)
Exclude the “Passed”
one for prioritization
N
<<- -In the Whole Test
Case Set- ->> Y
<<In the Dependencies Set>>
N
Have dependencies?
Start to test
<<Ready-to-Test>>
Failed?
<<Report for Resolution>>
Y
N
Resovled?
All dependencies
passed?
6/27/2016
Y
<<Ready-to-Test>>
COCOMO Forum 2011
N
Exclude the “Failed” one
and the others “NA” that
depends on it for
prioritization
42
University of Southern California
Center for Systems and Software Engineering
Future Research Plan
• Tool Support for Value-based Test Prioritization
6/27/2016
COCOMO Forum 2011
43