Outline of the Lecture
• White box testing
(Glass box testing, Open box testing, Clear box testing,
Structural testing)
Software Testing
– Control flow testing
– Data flow testing
Lecture Notes 2 (of 4)
• Regression testing
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
1
White box testing
•
•
•
•
•
CUGS, SE, Mariam Kamkar, IDA, LiU
2
White-box Testing Techniques
• Definition: a strategy in which testing is based on the internal paths,
structure, and implementation of the software under test (SUT)
• Applicability: all levels of system development (path testing!)
logical decision
loops
internal data structure
paths
...
–
–
–
–
CUGS, SE, Mariam Kamkar, IDA, LiU
Unit
Integration
System
Acceptance
• Disadvantages: 1) number of execution paths may be so large; 2) test
cases may not detect data sensitivity; 3) assumes that control flow is
correct (nonexistent paths!); 4) tester must have programming skills.
• Advantages: tester can be sure that every path have been identified
and tested.
Coverage!!
January 2007
January 2007
3
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
4
Control Flow Graphs
Process blocks
Decision Point
Definition: Given a program written in an
imperative programming language, its
program graph is a directed graph in which
nodes are statement fragments, and edges
represent flow of control (a complete
statement is a “default” statement
fragment).
Junction Point
Sequence
If
While
January 2007
Until
CUGS, SE, Mariam Kamkar, IDA, LiU
Case
5
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
6
1
Levels of Coverage
(test coverage metrics)
1.
2.
3.
4.
5.
6.
7.
Statement Coverage
January 2007
y >= 0
test case-1(yes):
input:
expected result:
actual result:
7
January 2007
y=?
?
?
CUGS, SE, Mariam Kamkar, IDA, LiU
begin
y >= 0
begin
Begin
if ( y >= 0)
then y = 0;
abs = y;
end;
yes
y=0
y >= 0
test case-1(yes):
input: y = 0
expected result: 0
actual result: 0
y=0
0
0
CUGS, SE, Mariam Kamkar, IDA, LiU
9
January 2007
abs = y
CUGS, SE, Mariam Kamkar, IDA, LiU
10
begin
Begin
if ( y >= 0)
then y = 0;
abs = y;
end;
January 2007
test case-2 (no) :
input: y = ?
expected result: ?
actual result: ?
Condition Coverage
Branch Coverage
test case-1(yes):
input: y = 0
expected result: 0
actual result: 0
yes
y=0
no
abs = y
January 2007
8
Branch Coverage
Statement Coverage
test case- (yes):
input:
expected result:
actual result:
yes
y=0
abs = y
CUGS, SE, Mariam Kamkar, IDA, LiU
Begin
if ( y >= 0)
then y = 0;
abs = y;
end;
begin
Begin
if ( y >= 0)
then y = 0;
abs = y;
end;
Statement (Line) coverage
Decision (Branch) coverage
Condition coverage
Decision/Condition coverage
Multiple Condition coverage
Loop coverage
Path coverage
y >= 0
no
yes
y=0
abs = y
test case-2 (no) :
input: y = -5
expected result: 5
actual result: -5
CUGS, SE, Mariam Kamkar, IDA, LiU
Begin
if ( x < 10 && y > 20) {
z = foo (x,y); else z =fie (x,y);
}
end;
test case-1(T,F):
input: x = ?, y = ?
expected result: ?
actual result: ?
11
January 2007
no
z=fie (x,y)
x<10
&&
y>20
yes
z=foo (x,y)
test case-2 (F;T):
input: x = ?, y = ?
expected result: ?
actual result: ?
CUGS, SE, Mariam Kamkar, IDA, LiU
12
2
Condition Coverage
Begin
if ( x < 10 && y > 20) {
z = foo (x,y); else z =fie (x,y);
}
end;
test case- (T,F):
input: x = -4, y = 12
expected result: ?
actual result: ?
January 2007
Decision/Condition Coverage
x<10
&&
y>20
no
yes
z=fie (x,y)
z=foo (x,y)
test case-2 (F,T):
input: x = 12, y = 30
expected result: ?
actual result: ?
test case-1 (T,T,yes):
input: x = ?, y = ?
expected result: ?
actual result: ?
CUGS, SE, Mariam Kamkar, IDA, LiU
13
test case-1 (T,T,yes):
input: x = -4, y = 30
expected result: ?
actual result: ?
January 2007
x<10
&&
y>20
no
yes
z=fie (x,y)
z=foo (x,y)
yes
z=fie (x,y)
z=foo (x,y)
test case-2 (F,F,no):
input: x = ?, y = ?
expected result: ?
actual result: ?
CUGS, SE, Mariam Kamkar, IDA, LiU
Begin
if ( x < 10 && y > 20) {
z = foo (x,y); else z =fie (x,y);
}
end;
14
no
yes
x < 10
no
y > 20
yes
x<? y>?
----------------------------------test-case-1: t
t
test-case:2
t
f
test-case-3: f
t
test-case-4
f
f
test case-2 (F,F,no):
input: x = 12, y = 12
expected result: ?
actual result: ?
CUGS, SE, Mariam Kamkar, IDA, LiU
15
Multiple Condition Coverage
Begin
if ( x < 10 && y > 20) {
z = foo (x,y); else z =fie (x,y);
}
end;
January 2007
x<10
&&
y>20
no
Multiple Condition Coverage
Decision/Condition Coverage
Begin
if ( x < 10 && y > 20) {
z = foo (x,y); else z =fie (x,y);
}
end;
Begin
if ( x < 10 && y > 20) {
z = foo (x,y); else z =fie (x,y);
}
end;
January 2007
first
A
B
no
z=fie (x,y)
z=foo (x,y)
CUGS, SE, Mariam Kamkar, IDA, LiU
16
Loop Coverage
first
first
first
A
A
A
yes
x < 10
no
y > 20
B
B
B
C
C
C
C
D
D
D
D
last
last
last
last
yes
x < 10 y > 20
----------------------------------test-case-1: t
t
test-case:2
t
f
test-case-3: f
t
test-case-4
f
f
January 2007
z=fie (x,y)
CUGS, SE, Mariam Kamkar, IDA, LiU
z=foo (x,y)
Simple
17
January 2007
Concatenated
Nested
CUGS, SE, Mariam Kamkar, IDA, LiU
Knotted (horrible)/
Unstructured
18
3
Loop Coverage
Loop Coverage
• Simple loops :
– Skip the loop entirely
– Only one pass through the loop
– Two passes through the loop
– m passes through the loop when m < n
m: small number representing a typical loop value
n: maximum number of allowable passes through the loop
– n-1, n, n+1 passes through the loop
• Concatenated loops :
– If each of the loops is independent of the other, use
Simple loops approach
– If the loops are concatenated and dependent ( counter
for loop 1 is used as the initial value for loop 2) then
use Nested loop approach.
• Nested loops:
– Start at the innermost loop, set all other loops to minimum value.
– Conduct Simple loop test for innermost loop. Add other tests for
out-of-range or excluded values.
– Continue until all loops have been tested.
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
19
• Knotted (horrible) /unstructured loops:
– Whenever possible, this class of loops should be
redesigned to reflect the use of the structured
programming constructs.
January 2007
Path with loops
CUGS, SE, Mariam Kamkar, IDA, LiU
Path with loops
a
a
d
b
20
d
c
b
a
e
c
a
e
?
?
c,b,d
e
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
e
21
January 2007
22
Cyclomatic Complexity is a software metric that
provides a quantitative measure of the logical
complexity of a program. When used in context of
the basis path testing method, the value computed
for cyclomatic complexity defines the number of
independent paths in the basis set of a program
and provides us with an upper bound for the
number of tests that must be conducted to ensure
that all statements have been executed at least
once.
• All possible execution paths
• Question: How do we know how many
paths to look for?
• Answer: The computation of cyclomatic
complexity
CUGS, SE, Mariam Kamkar, IDA, LiU
CUGS, SE, Mariam Kamkar, IDA, LiU
Cyclomatic Complexity
Path Coverage
January 2007
d
23
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
24
4
Computation of cyclomatic complexity
A
1. V(G) = E – N + 2
Cyclomatic complexity has a foundation in graph theory
and is computed in the following ways:
1.
F
E
H
G
CUGS, SE, Mariam Kamkar, IDA, LiU
P=?
V(G) = ?
R
25
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
26
Independent Paths
1. V(G) = E – N + 2
C
B
E = 24
N = 19
V(G) = 24 – 19 + 2 = 7
D
F
E
H
J
I
2.
An independent path is any path through
the program that introduces at least one new
set of processing statements or a new
condition. When stated in terms of a flow
graph, an independent path must move
along at least one edge that has not been
traversed before the path is defined.
V(G) = P + 1
L
K
N
M
V(G) = P + 1
S
A
G
P
O
Q
V(G) = P + 1
P: number of binary decision
2.
L
N
M
J
I
K
Cyclomatic complexity V(G), for a flow graph, G, with
only binary decisions, is defined as:
January 2007
E=?
N=?
V(G) = ?
D
Cyclomatic complexity V(G), for a flow graph, G, is
defined as:
V(G) = E – N + 2
E: number of edges
N: number of nodes
2.
C
B
P
O
P=6
V(G) = 6 +1 = 7
R
Q
S
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
27
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
28
Determine a basis set of linearly independent paths
McCabe’s baseline method
Basis Path Testing
• Derive the control flow graph from the software module.
1.
• Compute the graph’s Cyclomatic Complexity of the
resultant flow graph.
2.
• Determine a basis set of linearly independent paths.
3.
• Create a test case for each basis path.
4.
• Execute these tests.
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
5.
29
Pick a “baseline” path. This path should be a “normal
case” program execution. McCabe advises: choose a
path with as many decision as possible.
To choose the next path, change the outcome of the first
decision along the baseline path while keeping the
maximum number of other decisions the same as the
baseline path.
To generate the third path, begin again with the baseline
but vary the second decision rather than the first.
Repeat the 3 for other paths until all decision along
baseline path have been flipped.
Now proceed to the second path, flipping its decisions,
one by one until the basis path set is completed.
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
30
5
Path-1
1:a decision
Path-2
A
A
C
B
E
H
G
J
I
H
N
M
N
M
R
D
R
H
G
Path-1
K
N
M
R
Q
CUGS, SE, Mariam Kamkar, IDA, LiU
D
H
L
N
R
Q
S
H
G
R
J
R
F
H
G
L
N
M
35
R
S
ACDFILPRS
ACDEGKMQS
January 2007
P
O
Q
S
ACDFJLORS
J
I
K
P
O
Q
S
CUGS, SE, Mariam Kamkar, IDA, LiU
D
I
N
M
C
E
6:e decision L
K
P
O
Q
ACDEGKMQS
January 2007
J
L
N
M
A
B
F
E
I
K
P
O
H
G
F
34
Path-7
C
B
D
5:e decision
E
J
I
K
M
F
ABDEGKNQS
A
Path-2
D
5:e decision
E
R
Q
CUGS, SE, Mariam Kamkar, IDA, LiU
C
B
P
O
S
January 2007
A
C
L
S
ABDEGKMQS
J
I
N
M
R
Q
33
Path-6
A
B
M
H
G
K
P
O
S
January 2007
J
L
N
F
E
I
4:e decision K
ABDEHKMQS
Path-2
H
R
Q
ABDEGKMQS
D
F
G
P
O
S
G
J
L
C
B
C
E
I
N
M
A
D
K
P
O
H
G
L
32
Path-5
A
B
F
E
J
I
S
ABDFILORS
D
E
R
Q
CUGS, SE, Mariam Kamkar, IDA, LiU
3:e decision
F
P
O
ACDEGKMQS
C
B
L
N
M
J
I
K
P
O
January 2007
A
C
B
31
Path-4
A
M
H
G
L
Q
ACDEGKMQS
CUGS, SE, Mariam Kamkar, IDA, LiU
Path-1
J
I
N
F
E
S
S
ABDEGKMQS
January 2007
H
R
Q
D
K
P
O
S
D
F
G
L
K
P
O
Q
J
I
C
B
E
F
G
6:e decision L
4:e decision K
2:e decision
E
A
C
B
D
5:decision F
Path-3
A
C
B
2:a decision D
3:e decision
Path-1
1:a decision
CUGS, SE, Mariam Kamkar, IDA, LiU
36
6
C
B
• Basis path testing calls for the creation of a test
case for each of these paths.
1. ABDEGKMQS
D
2. ACDEGKMQS
F
E
H
G
J
I
N
P
O
R
Q
• Note that multiple sets of basis paths can be
created that are not necessarily unique. Each set,
however, has the property that a set of test cases
based on it will execute every statement and every
branch.
5. ABDEGKNQS
6. ACDFJLORS
7. ACDFILPRS
S
January 2007
• This set of test cases will guarantee both statement
and branch coverage.
3. ABDFILORS
4. ABDEHKMQS
L
K
M
Observation
Set of basis paths:
A
CUGS, SE, Mariam Kamkar, IDA, LiU
37
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
38
Structured programming constructs
Guidelines
• Program with high cyclomatic complexity require more testing.
• Of the organizations that use the cyclomatic complexity metric, most
set some guideline for maximum acceptable complexity; V(G) =10 is a
common choice.
• What happens if a unit has higher complexity?
Sequence
Pre-test Loop
Post-test Loop
– Either simplify the unit
– Or plan to do more testing
• In general, when a program is well structured (i.e., composed solely of
the structured programming constructs), it can be reduced to a graph
with one path.
If-Then
• If the unit is well structured, its essential complexity is 1
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
39
January 2007
If-Then-Else
CUGS, SE, Mariam Kamkar, IDA, LiU
Case
40
Condensing with respect to the structured programming constructs (cont.)
Condensing with respect to the structured programming constructs
Triangle Program
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
41
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
42
7
Condensing with respect to the structured programming constructs (cont.)
Violations of structured programming
Branching into a loop
Branching out of a loop
Branching into a decision
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
43
January 2007
Branching out of a decision
CUGS, SE, Mariam Kamkar, IDA, LiU
44
Applicability and Limitation
Data Flow Testing
• Control flow testing is the cornerstone of unit testing. It
should be used for all modules of code that cannot be
tested sufficiently through reviews and inspections.
• Its limitation are that the tester must have sufficient
programming skill to understand the code and its control
flow.
• Control flow testing can be very time consuming because
of all modules and basic paths that comprise a system.
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
45
Define/Reference Anomalies
• Early data flow analyses often centered on a
set of faults that are known as define/reference
anomalies.
– A variable that is defined but never used
(referenced)
– A variable that is used but never defined
– A variable that is defined twice before it is used
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
47
Data flow testing focuses on the points at
which variables receive values and the
points at which these values are used (or
referenced). It detects improper use of data
values due to coding errors.
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
46
• dd: defined and defined again – not invalid but
suspicious
• du: defined and used – perfectly correct
• dk: defined and then killed – not invalid but
probably a programming error
• ud: used and defined – acceptable
• uu: used and used again – acceptable
• uk: used and killed – acceptable
• kd: killed and defined – acceptable
• ku: killed and used – a serious defect
• kk: killed and killed – probably a programming
error.
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
48
8
Definitions
Data flow Graph G(P) = (N, E)
Definition: Given a program (P) written in
an imperative programming language, its
program graph (G) is a directed graph in
which nodes (N) are statement fragments,
and edges (E) represent flow of control. In
addition it details the definition, use and
destruction of each of the module’s
variable.
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
49
• DEF(v, n): node n in G(P) is a defining node of
variable v in V, iff the value of variable v is
defined at the statement fragment corresponding to
node n.
• USE(v, n): node n in G(P) is a usage node of
variable v in V, iff the value of variable v is used at
the statement fragment corresponding to node n.
– P-use, C-use: a usage node USE(v, n) is a predicate use
(P-use) iff statement n is a predicate statement;
otherwise, USE(v, n) is computation use (C-use).
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
50
Data Flow Graphs
Definitions (cont.)
define x
• du-path: a definition-use path (du-path) with
respect to variable v is a path in PATHS(P) such
that, for some v in V, there are defined and usage
nodes DEF(v, m) and USE(v, n) such that m and n
are initial and final nodes of the path.
define x
• ~d: the variable does not exist, then it is defined
• ~u: the variable does not exist, then it is used
• ~k: the variable does not exist, then it is killed
use y
kill z
kill z
use x
define z
define x
use x
use z
define x
use x
use x
define y
use z
Variable x:
• dc-path: a definition-clear path with respect to a
variable v is a du-path with initial and final node
such that no other node in the path is defining
node of v.
use y
use z
kill y
define z
~define: correct
define-define: suspicious,
programming error
define-use: correct
Control flow graph annotated with define-use-kill information for x, y, z
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
Variable y
51
Variable z
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
Variable y
use y
52
Variable z
Variable y:
~use: major blunder
use-define: acceptable
define-use: correct
use-kill: acceptable
define-kill: probable programming error
kill z
kill z
define z
use z
define y
use z
use y
kill y
Variable z
~kill: programming error
kill-use: major blunder
use-use: correct
use-define: acceptable
kill-kill: probable
programming error
kill-define: acceptable
define-use: correct
Total: 6 problem!
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
53
use z
define z
Total: 6 problem!
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
54
9
Hierarchy of data flow coverage metrics
Slice-Based Testing
All-Paths
All-DU-Paths
Program slice S(V, n): given a program P and a
set V of variables in P, a slice on the variable set V
at statement n is the set of all statements in P that
contribute to the values of variables in V.
All-Uses
All C-Uses/some P-Uses
All P-Uses/some C-Uses
All-Defs
January 2007
All-P-Uses
All-Edges
Branch
All-Nodes
Statement
CUGS, SE, Mariam Kamkar, IDA, LiU
55
Program slice on variable i
1
2
3
s = 0;
i = 1;
while (i <= n)
{
s + = i;
i ++
}
print (s);
print (i);
print (n);
4
5
6
7
8
January 2007
2
3
5
7
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
56
• Program slices give us very precise ways to
describe parts of a program that we would like to
test.
i = 1;
while (i <= n)
{
i ++
}
print (i);
CUGS, SE, Mariam Kamkar, IDA, LiU
The idea of slices is to separate a program into
components that have some useful (functional)
meaning.
• Obs: good testing practices lead to better
programming practices:
– developing programs in terms of compilable slices,
– coding and testing one slice
– coding and testing other slices and merging them into a
solid program.
57
January 2007
Applicability and Limitations
CUGS, SE, Mariam Kamkar, IDA, LiU
58
Regression testing
(data-flow testing)
• It should be used for all modules of code that
cannot be tested sufficiently through reviews and
inspections.
• Regression testing is the activity that helps to
ensure that changes (due to testing or for other
reasons) do not introduce unintended behavior or
additional errors.
• Tester must have sufficient programming skill
• Regression testing may be conducted:
– manually, by re-executing a subset of all test cases
– using automated capture/playback tools
• Can be very time consuming
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
59
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
60
10
Regression test
• A regression test is a test applied to a new version or
release to verify that it still performs the same functions in
the same manner as an old version or release.
• The regression test suite (the subset of tests to be executed)
contains three different classes of test cases:
– A representative sample of tests that will exercise all software
functions.
– Additional tests that focus on software functions that are likely to
be affected by the change.
– Tests that focus on the software components that have been
changed.
Obs: it is impractical and inefficient to re-execute every test for every
program function once a change has been occurred.
January 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
61
11