Path selection criteria
Tor Stålhane & ‘Wande Daramola
Why path selection criteria
Doing white box testing (Control flow testing,
data flow testing, coverage testing) using
the test-all-paths strategy can be a tedious
and expensive affair. The strategies
discussed here are alternative ways to
reduce the number of paths to be tested.
As with all white box tests, it should only be
used for small chunks of code – less than
say 200 lines.
Data flow testing -1
• Data flow testing is a powerful tool to
detect improper use of data values due to
coding errors.
main() {
int x;
if (x==42){ ...}
}
Data flow testing -2
• Variables that contain data values have a defined life
cycle. They are created, they are used, and they are
killed (destroyed) - Scope
{
// begin outer block
int x;
// x is defined as an integer within this outer block
…;
// x can be accessed here
{
// begin inner block
int y;
// y is defined within this inner block
...;
// both x and y can be accessed here
}
// y is automatically destroyed at the end of this block
...;
…; // x can still be accessed, but y is gone
}
// x is automatically destroyed
Static data flow testing
• Variables can be used
– in computation
– in conditionals
• Possibilities for the first occurrence of a variable
through a program path
– ~d the variable does not exist, then it is defined (d)
– ~u the variable does not exist, then it is used (u)
– ~k the variable does not exist, then it is killed or
destroyed (k)
define, use, kill (duk) – 1
We define three usages of a variable:
• d – define the variable
• u – use the variable
• k – kill the variable.
A large part of those who use this approach
will only use define and use – du.
Based on the usages we can define a set of
patterns potential problems.
duk – 2
We have the following nine patterns:
•
•
•
•
•
•
•
•
•
dd: define and then define again – error
dk: define and then kill – error
ku: kill and then used – error
kk: kill and then kill again – error
du: define and then use – OK
kd: kill and then redefine – OK
ud: use and then redefine – OK
uk: use and then kill – OK
uu: use and then use – OK
Example: Static data flow testing
For each variable within the
module we will examine
define-use-kill patterns along
the control flow paths
Example cont’d:
Consider variable x as we traverse the left and then the right path
~define correct, the normal case
define-define suspicious, perhaps
a programming error
define-use correct, the normal
case
ddu
du
duk examples (x) – 1
Define x
Use x
Define x
Use x
ddu
Define x
Define x
Use x
Use x
du
Example Cont’d: Consider variable y
~use
use-define
define-use
use-kill
major blunder
acceptable
correct, the normal case
acceptable
uduk
udk
duk examples (y)- 2
Use y
Use y
Define y
Define y
Use y
Use y
Kill y
Kill y
uduk
udk
Example Cont’d:
Consider variable z
~kill programming error
kill-use
major blunder
use-use correct, the normal
case
use-define acceptable
kill-kill
probably a
programming error
kill-define
acceptable
define-use correct, the normal
case
kuuud
kkduud
duk examples (z) - 3
Kill z
Kill z
Kill z
Define z
Use z
Use z
Kill z
Define z
Use z
Use z
Use z
Use z
Define z
Define z
kuuud
kkduud
Dynamic data flow testing
Test strategy – 1
Based on the three usages we can define a
total of seven testing strategies. We will
have a quick look at each
• All definitions (AD): test cases cover each
definition of each variable for at least one
use of the variable.
• All predicate-uses (APU): there is at least
one path of each definition to p-use of the
variable
Test strategy – 2
• All computational uses (ACU): there is at
least one path of each variable to each cuse of the variable
• All p-use/some c-use (APU+C): there is at
least one path of each variable to each cuse of the variable. If there are any
variable definitions that are not covered
then cover a c-use
Test strategy – 3
• All c-uses/some p-uses (ACU+P): there is
at least one path of each variable to each
c-use of the variable. If there are any
variable definitions that are not covered
then cover a p-use.
• All uses (AU): there is at least one path of
each variable to each c-use and each puse of the variable.
Test strategy – 4
• All du paths (ADUP): test cases cover
every simple sub-path from each variable
definition to every p-use and c-use of that
variable.
Note that the “kill” usage is not included in
any of the test strategies.
Application of test strategies – 1
All definitions
All c-use
Define x
Kill z
c-use x
Define z
Define y
p-use z
Define x
p-use y
Kill z
p-use y
Kill z
Define x
c-use x
c-use z
All p-use
Define x
c-use x
c-use z
Kill z
c-use x
Define z
Define y
p-use z
c-use
c-use z
c-use
c-use z
Kill y
Define z
Kill y
Define z
Application of test strategies – 2
ACU
APU+C
Define x
p-use y
Kill z
Define x
c-use x
c-use z
Kill z
c-use x
Define z
Define y
p-use z
c-use
c-use z
Kill y
Define z
Relationship between strategies - 1
All paths
All du-paths
All uses
All p/some c
All c/some p
All p-uses
All c-uses
All defs
Branch
The higher up in the hierarchy, the better is the
test strategy
Statement
Relationship between strategies - 2
To make the relationships clear:
The main testing method here is pathtesting.
However, since full path testing often will
require us to execute a large number of
paths, the duk-strategy helps us to reduce
the number of paths necessary.
The diagram on the preceding slide helps us
to make the necessary decisions.
Acknowledgement
The material on the duk patterns and testing
strategies are taken from a presentation
made by L. Williams at the North Carolina
State University.
Available at: http://agile.csc.ncsu.edu/testing/DataFlowTesting.pdf
Further Reading:
An Introduction to data flow testing – Janvi Badlaney et al., 2006
Available at: ftp://ftp.ncsu.edu/pub/tech/2006/TR-2006-22.pdf
Use of coverage measures
Tor Stålhane
Model – 1
We will use the following notation:
• c: a coverage measure
• r(c): reliability
• 1 – r(c): failure rate
• r(c) = 1 – k*exp(-b*c)
Thus, we also have that
ln[1 – r(c)] = ln(k) – b*c
Model – 2
The equation ln[1 – r(c)] = ln(k) – b*c is of
the same type as Y = α*X + β.
We can thus use linear regression to
estimate the parameters k and b by doing
as follows:
1.Use linear regression to estimate α and β
2.We then have
– k = exp(α)
– b=-β
Coverage measures considered
We have studied the following coverage
measures:
• Statement coverage:
percentage of statements executed.
• Branch coverage:
percentage of branches executed
• LCSAJ
Linear Code Sequence And Jump
Statement coverage
Scatterplot of -ln(F5) vs Statment
13
12
-ln(F5)
11
10
9
8
7
0,3
0,4
0,5
0,6
0,7
Statment
0,8
0,9
1,0
Graph summary
Scatterplot of -ln(F5) vs Statment; Branch; LCSAJ
0,0
Statment
0,2
0,4
0,6
0,8
Branch
12
10
-ln(F5)
8
LCSAJ
12
10
8
0,0
0,2
0,4
0,6
0,8
Equation summary
Statements:
-ln(F) = 6.5 + 6.4 Cstatement, R2(adj) = 85.3
Branches:
-ln(F) = 7.5 + 6.2 Cbranches, R2(adj) = 82.6
LCSAJ
-ln(F) = 6.5 + 6.4 CLCSAJ, R2(adj) = 77.8
Usage patterns – 1
Not all parts of the code are used equally
often. When it comes to reliability, we will
get the greatest effect if we have a high
coverage for the code that is used most
often.
This also explains why companies or user
groups disagrees so much when
discussing the reliability of a software
product.
Usage patterns – 2
input
domain
X
X
X
X
X
Input
space A
X
X
X
X
Corrected
X
Usage patterns – 3
As long as we do not change our input
space – usage pattern – we will
experience no further errors.
New user groups with new ways to use the
system will experience new errors.
Usage patterns – 4
input
domain
Input
space B
X
X
X
X
X
X
Input
space A
Extended model – 1
We will use the following notation:
• c: coverage measure
• r(c): reliability
• 1 – r(c): failure rate
• r(c) = 1 – k*exp(-a*p*c)
• p: the strength of the relationship between
c and r. p will depend the coupling
between coverage and faults.
• a: scaling constant
Extended model – 2
Residual
unreliability
R(C)
1
Large p
Small p
1-k
0.0
1.0
C
Extended model - comments
The following relation holds:
ln[1 – r(c)] = ln(k) – a*p*c
• Strong coupling between coverage and
faults will increase the effect of test
coverage on the reliability.
• Weak coupling will create a residual gap
for reliability that cannot be fixed by more
testing, only by increasing the coupling
factor p – thus changing the usage
pattern.
Bishop’s coverage model – 1
Bishop’s model for predicting remaining
errors is different from the models we have
looked at earlier. It has a
• Simpler relationship between number of
remaining errors and coverage
• More complex relationship between
number of tests and achieved coverage
Bishop’s coverage model – 2
We will use f = P(executed code fails). Thus,
the number of observed errors will depend
on three factors
• Whether the code
– Is executed – C
– Fails during execution – f
• Coupling between coverage and faults - p
N0 – N(n) = F(f, C(n, p))
C(n) = 1 – 1/(1 + knp)
Bishop’s coverage model – 3
Based on the assumptions and expression
previously presented , we find that
N 0  N (n)
1  C (n)
 1 p
N0
f  [1  C (n)](1  f p )
If we use the expression on the previous
slide to eliminate C(n) we get
N 0  N ( n)
1
 1
N0
k (nf ) p  1
A limit result
It is possible to show that the following
relation holds under a rather wide set of
conditions:
e
MTTFt 
t
Nˆ 0
The initial number of defects – N0 – must be
estimated e.g. based on experience from
earlier projects as number of defects per
KLOC.
An example from telecom