Tests prioritization
Tor Stålhane
How to prioritize
There are several ways to prioritize tests. We will
however, focus on risk based prioritization.
This lecture will cover
• An introduction to risk assessment
• How to use risk as a prioritization mechanism
for tests
• A small example
Risk-Based Testing
Risk-based testing is not a new concept. Most
companies that develop or buy software think
risk, albeit often in an unstructured,
undocumented manner.
Only companies with a tradition for using risk
analysis will use systematic methods for
handling risk through analysis and testing.
Risk and stakeholders
Risk has only meaning as a relationship between
a system and its environment. Thus, what is a
risk and how important it is, will vary from
stakeholder to stakeholder.
While the probability of an event is a system
characteristic, the consequences will vary.
Thus, we need to identify all stakeholders before
we start to discuss and analyze risk.
Stakeholders - 1
We have two main groups of stakeholders, each
with their own concerns if a risk is allowed to
become a problem, e.g.:
• Customers: lose money, either directly – e.g.
an expensive failure – or indirectly – losing
business.
• The company that develops the system: lose
business – e.g. lose marked shares.
Stakeholders - 2
All stakeholders must be involved in the risk
assessment process. They will have different
areas of expertise and experience. Thus, the
methods we use must be simple to:
• Use – no long training period needed.
• Understand – people don’t have confidence in
something they don’t understand.
Risk identification
We start with the system’s use cases. Prepare
the use case diagram for the function to be
analyzed. Each participant should familiarize
himself with the use case diagram.
Start with a warm-up exercise, for instance going
through results from previous risk
identification processes. The warm-up
exercise will help us to clear up
misunderstandings and agree on a common
process
Use case – high level example
Review
Treatment
Plan
Review
Drug Data
Doctor
Review
Documents
Review
Diagnoses
Order
Tests
Lab
Send
Test
Results
Use case – detailed example
Update existing
schedule
Create new
schedule
Re(schedule) train
conflicting schedules
Control center
operator
<<extends>>
Change the schedule
of a train
<<extends>>
Change the schedule
of a track maintenance activity
Risk identification from use cases
For each function we need to consider:
• How can this function fail? Identifies failure
modes and what part of the code will
contribute to this failure mode.
• What is the probability that there are defects
in this part of the code?
• Which consequences can this failure have for
the stakeholders?
• Document results in a consequence table.
• Assess the severity of each failure mode.
Consequence Table
Subsystem:
Function
What the
user wants
to achieve
Failure mode description
How the function may
fail
Consequences
Code involved
System parts
involved.
How likely is it
that they will
cause the failure
mode
User
Cust.
Dev.
Risk assessment - 1
Even though risk assessment is a subjective
activity it is not about throwing out any
number that you like.
To be useful, a risk assessment must be
• Based on relevant experience.
• Anchored in real world data.
• The result of a documented and agreed-upon
process.
12
Risk assessment - 2
Risk assessment uses the participants’
experience and knowledge to answer
questions such as
• Can this really happen; e.g. has it happened
before?
• Can we describe a possible cause consequence chain for the event?
• How bad can it get?
• How often has this happened in the past?
13
How to make an assessment
We will look at the two main methods for risk
assessment:
• Qualitative risk assessment based on
– The probability / consequence matrix
– The GALE (Globally At Least Equivalent) method
• Quantitative risk assessment based on the
CORAS model
Qualitative assessment
We can assess consequences, probabilities and
benefits qualitatively in two ways. We can use:
• Categories – e.g. High, Medium and Low
• Numbers – e.g. values from 1 to 10. Note that
this does not make the assessment
quantitative – it is just another way to
document the assessments.
15
Categories – 1
When using categories, it is important to give a
short description for what each category
implies. E.g. it is not enough to say “High
consequences”. We must relate it to
something already known, e.g.
• Project size
• Company turn-over
• Company profit
16
Categories – 2
Two simple examples:
• Consequences: we will use the category
“High” if the consequence will gravely
endanger the profitability of the project.
• Probability: we will use the category “Low” if
the event can occur but only in extreme cases.
17
Consequences and probability - 1
Consequences
Probability
H
M
L
H
M
L
H
H
M
H
M
L
M
L
L
18
Consequences and probability - 2
The multiplication table is used to rank the risks.
It can not tell us how large they are.
In the general case, we should only use
resources on risks that are above a certain,
predefined level.
19
The GALE method
The GALE method is a method for making
decision about introducing or not introducing
a change in e.g. a process or construction.
We will, however, only use the scoring scheme
for risk assessment.
The scoring scheme focuses on deviations from
current average. This is reasonable, given that
it is mainly concerned with comparing status
quo to a new situation.
20
The GALE risk index
The GALE risk index is computed based on our
assessment of an incident’s
• Frequency score – how often will the event
occur. The event here is a defect that has not
been removed.
• Probability score – what is the probability that
the event will cause a problem
• Severity score – how serious is the problem.
The risk index I = FE + PE + S
Frequency score for event
Frequency
class
Occurrences per project
FE
Very frequent
200
Every project
6
Frequent
100
Every few projects
5
Probable
40
Every 10th project
4
Occasional
10
Every 100th project
3
Remote
1
A few times in the company’s
lifetime
2
Improbable
0.2
One or two times during the
company’s lifetime
1
Incredible
0.01
Once in the company’s
lifetime
0
22
Probability score for problem
Classification
Interpretation
PE
Probable
It is probable that this event, if it
occurs, will cause a problem
3
Occasional
The event, if it occurs, will
occasionally cause a problem
2
Remote
There is a remote chance that this
event, if it occurs, will cause a
problem
It is improbable that this event, if it
occurs, will cause a problem
Improbable
1
0
23
Severity score for event
Severity
class
Severe
Average
Minor
Interpretation
The portion of occurring problems that
have serious consequences is much
larger than average
The portion of occurring problems that
have serious consequences is similar
to our average
The portion of occurring problems that
have serious consequences is much
lower than average
S
2
1
0
24
The CORAS model – 1
CORAS was developed as a framework for
assessment of security risks.
What should concern us here, however, is how
they related the qualitative risk categories, not
to absolute values, but to the company’s turnover.
25
The CORAS model – 2
Quantitative risks and opportunities give us real
values. The usefulness of this is, however,
limited since it is
• Difficult to find real values for all risks.
• Not obvious how we can compare qualitative
and quantitative risks.
When we use the CORAS tables it is important
to remember that developers, customers and
users will have different values – e.g. different
company incomes.
26
The CORAS consequence table
Consequence values
Category
Measured
related to
income
Measured
loss due to
impact on
business
Insignificant
Minor
Moderate
Major
Catastrophic
0.0 – 0.1%
0.1 – 1.0%
1 – 5%
5 – 10%
10 – 100%
Lost profits
Reduce the
resources of one
or more
departments
Loss of a couple
of customers
Close down
departments or
business
sectors
No impact on
business.
Minor delays
Out of
business
27
The CORAS frequency table - 1
As we will see on the next slide, CORAS allows
us to interpret frequency in two ways:
• The number of unwanted incidents per year –
e.g. the number of times a function will fail.
• The failing portion of demands – e.g. number
of service demands to a system.
28
The CORAS frequency table - 2
Frequency values
Category
Rare
Unlikely
Possible
Likely
Almost
certain
Number of
unwanted
incidents per
Year
1/100
1/100 – 1/50
1/50 - 1
1 - 12
> 12
Number of
unwanted
incidents per
demand
1/1000
(1/500)
1/50
(1/25)
1/1
Each five
times the
system is
used
Each tenth
time the
system is
used
Every
second
time the
system is
used
Interpretation
of number of
demands
Unwanted
incident
never
occurs
Each
thousand
time the
system is
used
29
An alternative consequence table
We have been introducing risk based testing in a
Norwegian company that develop software
that is used in health care. This company have
made a table including:
• Service receiver – the patient
• Service provider – the hospital
• Developing company – e.g. the software
developers
Consequence Levels
Consequence
level
High – 3
Medium – 2
Low – 1
Service receiver
Service provider
One person killed
or several
persons seriously
injured
Several persons
killed or many
seriously injured
Bad press
Lose customer
One person
seriously injured
User looks bad to
his superior(s) or
large sums lost
Dissatisfied
customers or
users
Minor irritant,
small amount of
money lost or
wasted
-
Minor irritant
Developing
company
Earlier test experiences – Worry list
Testing is a sampling process. Thus, if we find a
lot of defects in a component or sub-system,
we should conclude that this component has
many defects.
The conclusion will not necessarily be the same
if we conclude based on an inspection.
The Worry List
Consequences
High (3)
Number
of errors
registered
High
10 Medium
3–9
Low
0-2
Medium
(2)
Low (1)
Testing and resources
We will always have limited resources for
testing. This is made worse by the fact that
testing often starts late in the project
We make best use of the available resources by
trying to minimize the total system-related
risk.
Risk-Based Testing – 1
Risk-based testing has the following steps
• Identify how the product interact with its
environment. This is necessary to understand
failure consequences
• Identify and rank risks – probability and
consequence. If this is a white box or grey box
test we should also identify possible causesevent chains to understand the failure
mechanisms.
Risk-Based Testing – 2
• Define tests that can be used to ensure that
the code defect probability is low.
– Black box test – apply e.g. random testing or
domain testing.
– White box or grey box test – make sure all
identified cause-event chains are exercised.
• Run the tests – highest risk first
ATM example – 1
ATM example – 2
Subsystem: ATM transaction
Function
Failure mode description
Consequences
Code
involved
User
Cust
Dev.
Withdrawal
Withdraw more than allowed
Wrong amount registered
Wrong account
W-1, M-1
Acc-1, M-1
Acc-2
L
H
L
M
H
H
H
H
H
Deposit
Wrong amount registered
Wrong account
M-1, V-1
Acc-2
H
H
H
H
H
H
Inquiry
Wrong account
Wrong value returned
Acc-2, V-1
V-1, M-1
M
M
L
M
M
M
ATM example – 3
Frequency
class
Occurrences per project
FE
Very
Frequent
200
Every project
Frequent
100
Every few projects
5
Probable
40
Every 10th project
4
Occasional
10
Every 100th project
3
Remote
1
A few times in the
company’s lifetime
2
Improbable
0.2
One or two times during
The company’s lifetime
1
Incredible
0.01
Once in the company’s
lifetime
0
Function
Deposit – wrong
amount registered
Inquiry – wrong
account
6
Classification
Interpretation
PE
Probable
It is probable that this event, if it
occurs, will cause a problem
3
Occasional
The event, if it occurs, will
occasionally cause a problem
2
Remote
There is a remote chance that this
event, if it occurs, will cause a
problem
1
It is improbable that this event, if it
occurs, will cause a problem
0
Improbable
Components
S
FE
PE
I
M-1, V-1
2
4
3
9
Acc-2, V-1
1
3
2
6