CITS 5501: Logic and Path Coverage
CITS5501
Software Quality and Testing
Logic and Path Coverage
From Introduction to Software Testing,
2nd ed, Ammann & Offutt
Ch. 3 : Logic Coverage
Four Structures for
Modeling Software
Graphs
Logic
Input Space
Syntax
Applied to
Applied
to
Specs
Source
Applied
to
FSMs
Source
DNF
Source
Specs
Design
Introduction to Software Testing (Ch
3)
Models
Integ
Use cases
© Ammann & Offutt
Input
2
1
CITS 5501: Logic and Path Coverage
Covering Logic Expressions
(3.1)
• Logic expressions show up in many situations
• Covering logic expressions is required by the US Federal
Aviation Administration for safety critical software
• Logical expressions can come from many sources
– Decisions in programs
– FSMs and statecharts
– Requirements
• Tests are intended to choose some subset of the total number
of truth assignments to the expressions
Introduction to Software Testing (Ch 3)
© Ammann & Offutt
3
Logic Predicates and Clauses
• A predicate is an expression that evaluates to a boolean
value
• Predicates can contain
– boolean variables
– non-boolean variables that contain >, <, ==, >=, <=, !=
– boolean function calls
• Internal structure is created by logical operators
– ¬ – the negation operator
–  – the and operator
–  – the or operator
–  – the implication operator
–  – the exclusive or operator
–  – the equivalence operator
• A clause is a predicate with no logical operators
Introduction to Software Testing (Ch 3)
© Ammann & Offutt
4
2
CITS 5501: Logic and Path Coverage
Examples
• (a < b)  f (z)  D  (m >= n*o)
• Four clauses:
–
–
–
–
(a < b) – relational expression
f (z) – boolean-valued function
D – boolean variable
(m >= n*o) – relational expression
• Most predicates have few clauses
– It would be nice to quantify that claim!
• Sources of predicates
–
–
–
–
–
Decisions in programs
Guards in finite state machines
Decisions in UML activity graphs
Requirements, both formal and informal
SQL queries
Introduction to Software Testing (Ch 3)
© Ammann & Offutt
5
Testing and Covering Predicates
(3.2)
• We use predicates in testing as follows :
– Developing a model of the software as one or more predicates
– Requiring tests to satisfy some combination of clauses
• Abbreviations:
–
–
–
–
–
P is the set of predicates
p is a single predicate in P (that is, 𝑝 ∈ 𝑃)
C is the set of clauses in P
Cp is the set of clauses in predicate p
c is a single clause in C
Introduction to Software Testing (Ch 3)
© Ammann & Offutt
6
3
CITS 5501: Logic and Path Coverage
Testing and Covering Predicates
(3.2)
• Example: Suppose that we have two predicates:
- predicate p1 is: (a < b)  f (z)
- predicate p2 is: y > f (10).
There are 3 clauses: Clause c1 is (a < b), clause c2 is f (z),
and clause c3 is y > f (10).
Then
•
•
•
•
𝑃 = 𝑝1 , 𝑝2 ,
𝐶 = 𝑐1 , 𝑐2 , 𝑐3 ,
𝐶𝑝1 = 𝑐1 , 𝑐2 ,
𝐶𝑝2 = 𝑐3
Introduction to Software Testing (Ch 3)
© Ammann & Offutt
7
Predicate and Clause Coverage
• The first (and simplest) two criteria require that each
predicate and each clause be evaluated to both true and false
Predicate Coverage (PC) : For each p in P, TR contains two
requirements: p evaluates to true, and p evaluates to false.
• When predicates come from conditions on edges, this is equivalent
to edge coverage
• PC does not evaluate all the clauses, so …
Clause Coverage (CC) : For each c in C, TR contains two
requirements: c evaluates to true, and c evaluates to false.
Introduction to Software Testing (Ch 3)
© Ammann & Offutt
8
4
CITS 5501: Logic and Path Coverage
Predicate Coverage Example
(a < b)  D  (m >= n*o)
predicate coverage
Predicate = true
a = 5, b = 10, D = true, m = 1, n = 1, o = 1
= (5 < 10)  true  (1 >= 1*1)
= true  true  true
= true
Recall that
^ has a higher
precedence
than v.
Predicate = false
a = 10, b = 5, D = false, m = 1, n = 1, o = 1
= (10 < 5)  false  (1 >= 1*1)
= false  false  true
= false
Introduction to Software Testing (Ch 3)
© Ammann & Offutt
9
Clause Coverage Example
((a < b)  D)  (m >= n*o)
Clause coverage
(a < b) = true
(a < b) = false
D = true D = false
a = 5, b = 10
a = 10, b = 5
D = true
m >= n*o = true
D = false
m >= n*o = false
m = 1, n = 1, o = 1 m = 1, n = 2, o = 2
false cases
Two tests
true cases
1) a = 5, b = 10, D = true, m = 1, n = 1, o = 1
2) a = 10, b = 5, D = false, m = 1, n = 2, o = 2
Introduction to Software Testing (Ch 3)
© Ammann & Offutt
10
5
CITS 5501: Logic and Path Coverage
Problems with PC and CC
• PC does not fully exercise all the clauses,
especially in the presence of short circuit
evaluation
• CC does not always ensure PC
– That is, we can satisfy CC without causing the predicate
to be both true and false
– This is definitely not what we want !
• The simplest solution is to test all combinations …
Introduction to Software Testing (Ch 3)
© Ammann & Offutt
11
Combinatorial Coverage
• CoC requires every possible combination
• Sometimes called Multiple Condition Coverage
Combinatorial Coverage (CoC) : For each 𝑝 ∈ 𝑃, TR has test
requirements for the clauses in 𝐶𝑝 to evaluate to each possible
combination of truth values.
a<b D
1 T
2 T
3 T
4 T
5
F
6
F
7
F
8
F
Introduction to Software Testing (Ch 3)
T
T
F
F
T
T
F
F
m >= n*o
T
F
T
F
T
F
T
F
((a < b)  D)  (m >= n*o)
T
F
T
F
T
F
F
F
© Ammann & Offutt
12
6
CITS 5501: Logic and Path Coverage
Combinatorial Coverage
• This is simple, neat, clean, and comprehensive …
• But quite expensive!
• 2N tests, where N is the number of clauses
– Impractical for predicates with more than 3 or 4 clauses
• The literature has lots of suggestions – some confusing
• The general idea is simple:
Test each clause independently from the other clauses
• Getting the details right is hard
• What exactly does “independently” mean ?
• The book presents this idea as “making clauses active” …
Introduction to Software Testing (Ch 3)
© Ammann & Offutt
13
Active Clauses
• Clause coverage has a weakness : The values do not always
make a difference
• Consider the first test for clause coverage, which caused
each clause to be true:
– (5 < 10)  true  (1 >= 1*1)
• Since 5 < 10 is true, only the first clause counts !
• To really test the results of a clause, the clause should be the
determining factor in the value of the predicate
Determination : Given a major clause ci in predicate p, we say that
ci determines p if the minor clauses 𝑐𝑗 ∈ 𝑝, where
𝑗 ≠ 𝑖, have values so that changing the truth value
of ci changes the value of p
• This is considered to make the clause active
Introduction to Software Testing (Ch 3)
© Ammann & Offutt
14
7
CITS 5501: Logic and Path Coverage
Determining Predicates
p=A B
p=A B
If B = true, p is always true.
If B = false, p is always false.
A determines p if B = false .
If B = true, A determines p.
B determines p if A = false.
If A = true, B determines p.
• Goal : Find tests for each clause when the clause
determines the value of the predicate
• This is formalized in several criteria that have
subtle, but very important, differences
Introduction to Software Testing (Ch 3)
© Ammann & Offutt
15
General Active Clause Coverage (GACC)
GACC :
For each predicate 𝑝 ∈ 𝑃 and each major clause 𝑐𝑖
of the predicate 𝑝, choose minor clauses 𝑐𝑗 , where
𝑗 ≠ 𝑖, so that 𝑐𝑖 determines 𝑝. TR has two
requirements for each 𝑐𝑖 : 𝑐𝑖 evaluates to true and
𝑐𝑖 evaluates to false. The values chosen for the
minor clauses 𝑐𝑗 do not need to be the same when
𝑐𝑖 is true as when 𝑐𝑖 is false.
Restricted Active Clause Coverage (RACC) : Same as
above except that the values chosen for the minor clauses 𝒄𝒋 must be
the same when 𝒄𝒊 is true as when 𝒄𝒊 is false.
Question: Does GACC subsume PC? Consider the predicate
𝑝 = 𝐴 ↔ 𝐵. How many tests do you have using GACC?
Introduction to Software Testing (Ch 3)
© Ammann & Offutt
16
8
CITS 5501: Logic and Path Coverage
Correlated Active Clause Coverage (CACC)
CACC :
In general, N
clauses result in
N+1 tests.
For each predicate 𝑝 ∈ 𝑃 and each major clause 𝑐𝑖
of the predicate 𝑝, choose minor clauses 𝑐𝑗 , where
𝑗 ≠ 𝑖, so that 𝑐𝑖 determines 𝑝. TR has two
requirements for each 𝑐𝑖 : 𝑐𝑖 evaluates to true and
𝑐𝑖 evaluates to false. The values chosen for the
minor clauses 𝑐𝑗 must cause p to be true for one
value of the major clause 𝑐𝑖 and false for the other.
Example: Consider 𝑝 = 𝐴 ↔ 𝐵. Then
•
Consider A to be the major clause. We have A determines p iff B = A. This generates
two tests:
•
Test 1: {A = true, B = true}. This gives p = true.
•
Test 2: {A = false, B = false}. However, since this also gives p = true, the
condition above (in yellow) requires B to be set to true. Thus Test 2 becomes
{A = false, B = true}. This gives p = false.
•
Next, consider B to be the major clause. Then we have Tests 3 and 4: {B=true,
A=true} and {B=false, A=true}. Since Test 3 is the same as Test 1, it is eliminated.
Thus, there are 3 tests in total using the CACC criterion.
Introduction to Software Testing (Ch 3)
© Ammann & Offutt
17
General and Restricted Inactive Clause Coverage
Properties of ICC:
– Unlike ACC, the notion of correlation is not relevant to ICC
• Since 𝑐𝑖 does not determine p, it cannot correlate with p
– Predicate coverage is always guaranteed
General Inactive Clause Coverage (GICC) : For each p ∈ P and each major
clause 𝑐𝑖 ∈ 𝐶𝑝 , choose minor clauses 𝑐𝑗 , where 𝑗 ≠ 𝑖 , so that ci does not
determine p. The TR has four requirements for ci : (1) ci evaluates to true with p
true, (2) ci evaluates to false with p true, (3) ci evaluates to true with p false, and
(4) ci evaluates to false with p false. The values chosen for the minor clauses 𝑐𝑗
may vary among the four cases.
Restricted Inactive Clause Coverage (RICC) : Same as above except for the
following: The values chosen for the minor clauses 𝑐𝑗 must be the same in
cases (1) and (2), and the values chosen for the minor clause 𝑐𝑗 must also be
the same in cases (3) and (4).
Introduction to Software Testing (Ch 3)
© Ammann & Offutt
18
9
CITS 5501: Logic and Path Coverage
General and Restricted Inactive Clause Coverage
An example for RICC: Suppose that we have the predicate
𝑝 = (𝐴 ∨ 𝐵) ∧ 𝐶. Consider
• 𝐴 to be the major clause, B and C to be the minor clauses
A
•
•
•
•
B
C
p
Case (1) True
True
True
True
Case (2) False
True
True
True
Case (3) True
Anything
False
False
Case (4) False
Anything
False
False
B to be the major clause, A and C to be the minor clauses
C to be the major clause, A and B to be the minor clauses
Merge all the tests together to form the test set.
In this example, by setting ‘Anything’ to False, the 4 tests are
sufficient since we have both clause coverage and predicate
coverage (there are N=3 clauses and we have N+1=4 tests).
Introduction to Software Testing (Ch 3)
© Ammann & Offutt
19
Subsumption Relations among Logic Coverage
Combinatorial
Clause Coverage
CoC
Restricted Inactive
Clause Coverage
RICC
Restricted Active
Clause Coverage
RACC
Correlated Active
Clause Coverage
CACC
General Inactive
Clause Coverage
GICC
General Active
Clause Coverage
GACC
Clause
Coverage
CC
Introduction to Software Testing (Ch 3)
© Ammann & Offutt
Predicate
Coverage
PC
20
10
CITS 5501: Logic and Path Coverage
Making Clauses Determine a
Predicate
• Finding values for minor clauses cj is easy for simple
predicates
• But how to find values for more complicated predicates ?
• Definitional approach:
– pc=true is predicate p with every occurrence of c replaced by true
– pc=false is predicate p with every occurrence of c replaced by false
• To find values for the minor clauses, connect pc=true and
pc=false with exclusive OR :
pc = pc=true  pc=false
• After solving, pc describes exactly the values needed for c
to determine p
Introduction to Software Testing (Ch 3)
A determines p
iff b is false
© Ammann & Offutt
Examples
p=ab
pa = pa=true  pa=false
= (true  b) XOR (false  b)
= true XOR b
=¬b
21
A determines p
iff b is true
p=ab
pa = pa=true  pa=false
= (true  b)  (false  b)
= b  false
=b
p = a  (b  c)
pa = pa=true  pa=false
= (true  (b  c))  (false  (b  c))
= true  (b  c)
= ¬ (b  c)
=¬ b¬c
A determines p iff
b^c is false (i.e., b
or c is false)
• “NOT b  NOT c” means either b or c can be false
• RACC requires the same choice for both values of a, CACC does not
Introduction to Software Testing (Ch 3)
© Ammann & Offutt
22
11
CITS 5501: Logic and Path Coverage
Repeated Variables
• The definitions in this chapter yield the same tests
no matter how the predicate is expressed
• (a  b)  (c  b) == (a  c)  b
• (a  b)  (b  c)  (a  c)
– Only has 8 possible tests, not 64
• Use the simplest form of the predicate, and ignore
contradictory truth table assignments
Introduction to Software Testing (Ch 3)
© Ammann & Offutt
23
A More Subtle Example
p = ( a  b )  ( a  ¬ b)
pa = pa=true  pa=false
= ((true  b)  (true  ¬ b))  ((false  b)  (false  ¬ b))
= (b  ¬ b)  false
= true  false
= true
p = ( a  b )  ( a  ¬ b)
pb = pb=true  pb=false
= ((a  true)  (a  ¬ true))  ((a  false)  (a  ¬ false))
= (a  false)  (false  a)
=aa
= false
• a always determines the value of this predicate
• b never determines the value – b is irrelevant !
Introduction to Software Testing (Ch 3)
© Ammann & Offutt
24
12
CITS 5501: Logic and Path Coverage
Infeasible Test Requirements
• Consider the predicate:
(a > b  b > c)  c > a
• (a > b) = true, (b > c) = true, (c > a) = true is infeasible
• As with graph-based criteria, infeasible test requirements
have to be recognized and ignored
• Recognizing infeasible test requirements is hard, and in
general, undecidable
• Software testing is inexact – engineering, not science
Introduction to Software Testing (Ch 3)
© Ammann & Offutt
25
Logic Expressions from Source
• Predicates are derived from decision statements in
programs
• In programs, most predicates have less than four
clauses
– Wise programmers actively strive to keep predicates simple
• When a predicate only has one clause, CoC, ACC, ICC,
and CC all collapse to predicate coverage (PC)
• Applying logic criteria to program source is hard
because of reachability and controllability:
– Reachability : Before applying the criteria on a predicate at a particular statement, we have to
get to that statement
– Controllability : We have to find input values that indirectly assign values to the variables in the
predicates
– Variables in the predicates that are not inputs to the program are called internal variables
Introduction to Software Testing (Ch 3)
© Ammann & Offutt
26
13
CITS 5501: Logic and Path Coverage
Specifications in Software
• Specifications can be formal or informal
– Formal specs are usually expressed mathematically
– Informal specs are usually expressed in natural language
• Lots of formal languages and informal styles are available
• Most specification languages include explicit logical expressions, so
it is very easy to apply logic coverage criteria
• Implicit logical expressions in natural-language specifications
should be re-written as explicit logical expressions as part of test
design
– You will often find mistakes
• One of the most common is preconditions …
Introduction to Software Testing (Ch 3)
© Ammann & Offutt
27
Preconditions
• Programmers often include preconditions for their methods
• The preconditions are often expressed in comments in
method headers
• Preconditions can be in javadoc, “requires”, “pre”, …
Example – Saving addresses
// name must not be empty
// state must be valid
// zip must be 5 numeric digits
// street must not be empty
// city must not be empty
Conjunctive
Normal
Form
Rewriting to logical expression
name != “”  state in stateList  zip >= 00000  zip <= 99999 
street != “”  city != “”
Introduction to Software Testing (Ch 3)
© Ammann & Offutt
28
14
CITS 5501: Logic and Path Coverage
Logic Coverage of Finite State Machines
• FSMs are graphs
– nodes represent states
– edges represent transitions among states
• Transitions often have logical expressions as
guards or triggers
• As we said:
Find a logical expression and cover it
Introduction to Software Testing (Ch 3)
© Ammann & Offutt
29
Example—Subway Train
secondPlatform = right
Left Doors
Open
secondPlatform= left
¬ emergencyStop  ¬ overrideOpen 
doorsClear
(all three transitions)
trainSpeed = 0  platform=left 
(inStation  (emergencyStop 
overrideOpen))
Introduction to Software Testing (Ch 3)
All Doors
Open
All Doors
Closed
© Ammann & Offutt
Right Doors
Open
trainSpeed = 0  platform=right
 (inStation  (emergencyStop 
overrideOpen))
30
15
CITS 5501: Logic and Path Coverage
Determination of the Predicate
trainSpeed = 0  platform=left  (inStation  (emergencyStop  overrideOpen))
trainSpeed = 0 : platform = left  (inStation  (emergencyStop  overrideOpen))
platform = left : trainSpeed = 0  (inStation  (emergencyStop  overrideOpen))
inStation : trainSpeed = 0  platform = left  (¬ emergencyStop  ¬ overrideOpen)
emergencyStop : trainSpeed = 0  platform = left  (¬ inStation  overrideOpen)
overrideOpen : trainSpeed = 0  platform = left  (¬ inStation  emergencyStop)
Introduction to Software Testing (Ch 3)
© Ammann & Offutt
31
Introduction to Software Testing
(2nd edition)
Chapter 6.1, 6.2
Overview Graph Coverage Criteria
Paul Ammann & Jeff Offutt
http://www.cs.gmu.edu/~offutt/softwar
etest/
First version, 23 September 2013
16
CITS 5501: Logic and Path Coverage
Ch. 06 : Graph Coverage
Four Structures for
Modeling Software
Graphs
Logic
Input Space
Syntax
Applied to
Applied
to
Specs
Source
Applied
to
FSMs
Source
DNF
Source
Specs
Design
Integ
Use cases
Introduction to Software
Testing, Edition 2 (Ch 06)
Models
© Ammann & Offutt
Input
33
Covering Graphs (6.1)
• Graphs are the most commonly used structure
for testing
• Graphs can come from many sources
–
–
–
–
Control flow graphs
Design structure
FSMs and statecharts
Use cases
• Tests usually are intended to “cover” the
graph in some way
Introduction to Software
Testing, Edition 2 (Ch 06)
© Ammann & Offutt
34
17
CITS 5501: Logic and Path Coverage
Definition of a Graph
• A set N of nodes, N is not empty
• A set N0 of initial nodes, N0 is not empty
• A set Nf of final nodes, Nf is not empty
• A set E of edges, each edge from one node to
another
– ( ni , nj ), i is predecessor, j is successor
Introduction to Software
Testing, Edition 2 (Ch 06)
© Ammann & Offutt
35
Three Example Graphs
1
1
2
3
4
4
2
5
8
3
6
9
1
7
10
2
Not a
valid
graph
3
4
N0 = { 1}
N0 = { 1, 2, 3 }
N0 = { }
Nf = { 4 }
Nf = { 8, 9, 10 }
Nf = { 4 }
Introduction to Software
Testing, Edition 2 (Ch 06)
© Ammann & Offutt
36
18
CITS 5501: Logic and Path Coverage
Paths in Graphs
• Path : A sequence of nodes – [n1, n2, …, nM]
– Each pair of nodes is an edge
• Length : The number of edges
– A single node is a path of length 0
• Subpath : A subsequence of nodes in p is a subpath of p
• Reach (n) : Subgraph that can be reached from n
1
2
3
A Few Paths
[ 1, 4, 8 ]
4
5
6
7
[ 2, 5, 9, 6, 2 ]
[ 3, 7, 10 ]
8
9
Reach (1) = { 1, 4, 5, 8,
9, 6, 2, 10 }
Reach ({1, 3}) = G
Reach([3,7]) = {3, 7, 10}
10
Introduction to Software
Testing, Edition 2 (Ch 06)
© Ammann & Offutt
37
Test Paths and SESEs
• Test Path : A path that starts at an initial node and
ends at a final node
• Test paths represent execution of test cases
– Some test paths can be executed by many tests
– Some test paths cannot be executed by any tests
• SESE graphs : All test paths start at a single node
and end at another node
– Single-entry, single-exit
– N0 and Nf have exactly one node
2
1
4
3
Introduction to Software
Testing, Edition 2 (Ch 06)
5
7
Double-diamond graph
Four test paths
[1, 2, 4, 5, 7]
[1, 2, 4, 6, 7]
[1, 3, 4, 5, 7]
[1, 3, 4, 6, 7]
6
© Ammann & Offutt
38
19
CITS 5501: Logic and Path Coverage
Visiting and Touring
• Visit : A test path p visits node n if n is in p
A test path p visits edge e if e is in p
• Tour : A test path p tours subpath q if q is a subpath of p
Path [ 1, 2, 4, 5, 7 ]
Visits nodes 1, 2, 4, 5, 7
Visits edges (1, 2), (2, 4), (4, 5), (5, 7)
Tours subpaths [1, 2, 4], [2, 4, 5], [4, 5, 7], [1, 2, 4, 5], [2, 4, 5, 7]
Introduction to Software
Testing, Edition 2 (Ch 06)
© Ammann & Offutt
39
Tests and Test Paths
• path (t) : The test path executed by test t
• path (T) : The set of test paths executed by the set of
tests T
• Each test executes one and only one test path
• A location in a graph (node or edge) can be reached
from another location if there is a sequence of edges
from the first location to the second
– Syntactic reach : A subpath exists in the graph
– Semantic reach : A test exists that can execute that
subpath
Introduction to Software
Testing, Edition 2 (Ch 06)
© Ammann & Offutt
40
20
CITS 5501: Logic and Path Coverage
test 1
Testsmany-to-one
and Test Paths
Test
Path
test 2
test 3
Deterministic software–a test always executes the same test path
test 1
many-to-many
Test Path 1
test 2
Test Path 2
test 3
Test Path 3
Non-deterministic software–a test can execute different test paths
Introduction to Software
Testing, Edition 2 (Ch 06)
© Ammann & Offutt
41
Testing and Covering Graphs (6.2)
• We use graphs in testing as follows :
– Developing a model of the software as a graph
– Requiring tests to visit or tour specific sets of nodes, edges or subpaths
• Test Requirements (TR) : Describe properties of test paths
• Test Criterion : Rules that define test requirements
• Satisfaction : Given a set TR of test requirements for a criterion C, a set
of tests T satisfies C on a graph if and only if for every test requirement tr
in TR, there is a test path in path(T) that meets the test requirement tr
• Structural Coverage Criteria : Defined on a graph just in terms of
nodes and edges
• Data Flow Coverage Criteria : Requires a graph to be annotated
with references to variables
Introduction to Software
Testing, Edition 2 (Ch 06)
© Ammann & Offutt
42
21
CITS 5501: Logic and Path Coverage
Node and Edge Coverage
• The first (and simplest) two criteria require that each node
and edge in a graph be executed
Node Coverage (NC) : Test set T satisfies node coverage on
graph G iff for every syntactically reachable node n in N, there is
some path p in path(T) such that p visits n.
• This statement is a bit cumbersome, so we abbreviate it in terms of
the set of test requirements
Node Coverage (NC) :TR contains each reachable node in G.
Introduction to Software
Testing, Edition 2 (Ch 06)
© Ammann & Offutt
43
Structural Coverage Example
Node Coverage
TR = { 1, 2, 3, 4, 5, 6, 7 }
Test Paths: [ 1, 2, 3, 4, 7 ] [ 1, 2, 3, 5, 6, 5, 7 ]
1
Edge Coverage
TR = { (1,2), (1, 3), (2, 3), (3, 4), (3, 5), (4, 7), (5, 6), (5, 7), (6, 5) }
Test Paths: [ 1, 2, 3, 4, 7 ] [1, 3, 5, 6, 5, 7 ]
2
3
4
5
7
6
Introduction to Software
Testing, Edition 2 (Ch 06)
Edge-Pair Coverage
TR = {[1,2,3], [1,3,4], [1,3,5], [2,3,4], [2,3,5], [3,4,7],
[3,5,6], [3,5,7], [5,6,5], [6,5,6], [6,5,7] }
Test Paths: [ 1, 2, 3, 4, 7 ] [ 1, 2, 3, 5, 7 ] [ 1, 3, 4, 7 ]
[ 1, 3, 5, 6, 5, 6, 5, 7 ]
Complete Path Coverage (CPC)
Test Paths: [ 1, 2, 3, 4, 7 ] [ 1, 2, 3, 5, 7 ] [ 1, 2, 3, 5, 6, 5, 6 ] [ 1, 2, 3, 5,
6, 5, 6, 5, 7 ] [ 1, 2, 3, 5, 6, 5, 6, 5, 6, 5, 7 ] …
© Ammann & Offutt
44
22
CITS 5501: Logic and Path Coverage
Loops in Graphs
• If a graph contains a loop, it has an infinite
number of paths
• Thus, CPC is not feasible
• SPC is not satisfactory because the results are
subjective and vary with the tester
• Attempts to “deal with” loops:
–
–
–
–
1970s : Execute cycles once ([4, 5, 4] in previous example, informal)
1980s : Execute each loop, exactly once (formalized)
1990s : Execute loops 0 times, once, more than once (informal description)
2000s : Prime paths
Introduction to Software
Testing, Edition 2 (Ch 06)
© Ammann & Offutt
45
Simple Paths and Prime Paths
• Simple Path : A path from node ni to nj is simple if no
node appears more than once, except possibly the
first and last nodes are the same
– No internal loops
– A loop is a simple path
• Prime Path : A simple path that does not appear as a
proper subpath of any other simple path
Simple Paths : [1,2,4,1], [1,3,4,1], [2,4,1,2], [2,4,1,3], [3,4,1,2],
[3,4,1,3], [4,1,2,4], [4,1,3,4], [1,2,4], [1,3,4], [2,4,1], [3,4,1], [4,1,2],
[4,1,3], [1,2], [1,3], [2,4], [3,4], [4,1], [1], [2], [3], [4]
1
2
3
4
Introduction to Software
Testing, Edition 2 (Ch 06)
Prime Paths : [2,4,1,2], [2,4,1,3], [1,3,4,1], [1,2,4,1], [3,4,1,2],
[4,1,3,4], [4,1,2,4], [3,4,1,3]
© Ammann & Offutt
46
23
CITS 5501: Logic and Path Coverage
Prime Path Coverage
• A simple, elegant and finite criterion that requires
loops to be executed as well as skipped
Prime Path Coverage (PPC) :TR contains each prime path in G.
• Will tour all paths of length 0, 1, …
• That is, it subsumes node and edge coverage
• PPC does NOT subsume EPC
• If a node n has an edge to itself, EPC will require [n, n, m]
• [n, n, m] is not prime
Introduction to Software
Testing, Edition 2 (Ch 06)
© Ammann & Offutt
47
Round Trips
• Round-Trip Path : A prime path that starts and ends at
the same node
Simple Round Trip Coverage (SRTC) : TR contains at least one
round-trip path for each reachable node in G that begins and
ends a round-trip path.
Complete Round Trip Coverage (CRTC) : TR contains all roundtrip paths for each reachable node in G.
• These criteria omit nodes and edges that are not in round trips
• That is, they do not subsume edge-pair, edge, or node coverage
Introduction to Software
Testing, Edition 2 (Ch 06)
© Ammann & Offutt
48
24
CITS 5501: Logic and Path Coverage
Prime Path Example
• The previous example has 38 simple paths
• Only nine prime paths
1
2
3
4
5
6
Prime Paths
[1, 2, 3, 4, 7]
[1, 2, 3, 5, 7]
[1, 2, 3, 5, 6]
[1, 3, 4, 7]
[1, 3, 5, 7]
[1, 3, 5, 6]
[6, 5, 7]
[6, 5, 6]
[5, 6, 5]
7
Introduction to Software
Testing, Edition 2 (Ch 06)
Execute loop 0
times
Execute loop
once
Execute loop more
than once
© Ammann & Offutt
49
Touring, Sidetrips and Detours
• Prime paths do not have internal loops … test paths might
• Tour : A test path p tours subpath q if q is a subpath of p
• Tour With Sidetrips : A test path p tours subpath q with sidetrips iff
every edge in q is also in p in the same order
• The tour can include a sidetrip, as long as it comes back to the same node
• Tour With Detours : A test path p tours subpath q with detours iff
every node in q is also in p in the same order
• The tour can include a detour from node ni, as long as it comes back to the prime
path at a successor of ni
Introduction to Software
Testing, Edition 2 (Ch 06)
© Ammann & Offutt
50
25
CITS 5501: Logic and Path Coverage
Overview
• A common application of graph criteria is to program
source
• Graph : Usually the control flow graph (CFG)
• Node coverage : Execute every statement
• Edge coverage : Execute every branch
• Loops : Looping structures such as for loops, while loops,
etc.
• Data flow coverage : Augment the CFG
– defs are statements that assign values to variables
– uses are statements that use variables
Introduction to Software
Testing, Edition 2 (Ch 6)
© Ammann & Offutt
51
Control Flow Graphs
• A CFG models all executions of a method by describing
control structures
• Nodes : Statements or sequences of statements (basic
blocks)
• Edges : Transfers of control
• Basic Block : A sequence of statements such that if the first
statement is executed, all statements will be (no branches)
• CFGs are sometimes annotated with extra information
– branch predicates
– defs
– uses
• Rules for translating statements into graphs …
Introduction to Software
Testing, Edition 2 (Ch 6)
© Ammann & Offutt
52
26
CITS 5501: Logic and Path Coverage
CFG : The if Statement
if (x < y)
{
y = 0;
x = x + 1;
}
else
{
x = y;
}
1
x<y
y=0
x=x+1
x >= y
2
3
x=y
4
1
if (x < y)
{
y = 0;
x = x + 1;
}
x<y
y=0
x=x+1
x >= y
2
3
Introduction to Software
Testing, Edition 2 (Ch 6)
© Ammann & Offutt
53
CFG : The if-Return Statement
if (x < y)
{
return;
}
print (x);
return;
1
x<y
return
x >= y
2
3
print (x)
return
No edge from node 2 to 3.
The return nodes must be distinct.
Introduction to Software
Testing, Edition 2 (Ch 6)
© Ammann & Offutt
54
27
CITS 5501: Logic and Path Coverage
Loops
• Loops require “extra” nodes to be added
• Nodes that do not represent statements or basic
blocks
Introduction to Software
Testing, Edition 2 (Ch 6)
© Ammann & Offutt
55
CFG : while and for Loops
x = 0;
while (x < y)
{
y = f (x, y);
x = x + 1;
}
x=0
1
dummy node
2
x<y
x >= y
3
4
implicitly initializes
x=0
loop
y =f(x,y)
x=x+1
2
for (x = 0; x < y; x++)
{
y = f (x, y);
}
y = f (x, y)
x<y
x >= y
3
5
4
Introduction to Software
Testing, Edition 2 (Ch 6)
1
implicitly increments
loop
© Ammann & Offutt
x=x+1
56
28
CITS 5501: Logic and Path Coverage
CFG : do Loop, break and continue
x = 0;
do
{
y = f (x, y);
x = x + 1;
} while (x < y);
println (y)
x=0
1
2
x >= y
y = f (x, y)
x = x+1
x<y
x = 0;
while (x < y)
{
y = f (x, y);
if (y == 0)
{
break;
} else if y < 0)
{
y = y*2;
continue;
}
x = x + 1;
}
print (y);
1
x=0
2
3
y =f(x,y)
y == 0
4
break
5
y<0
6
7
3
y = y*2
continue
x=x+1
8
Introduction to Software
Testing, Edition 2 (Ch 6)
© Ammann & Offutt
57
CFG : The case (switch) Structure
read ( c) ;
switch ( c )
{
case ‘N’:
y = 25;
break;
case ‘Y’:
y = 50;
break;
default:
y = 0;
break;
}
print (y);
Introduction to Software
Testing, Edition 2 (Ch 6)
1
c == ‘N’
y = 25;
break;
2
read ( c );
c == ‘Y’ default
3
4
y = 50;
break;
y = 0;
break;
5
print (y);
© Ammann & Offutt
58
29
CITS 5501: Logic and Path Coverage
Example Control Flow – Stats
public static void computeStats (int [ ] numbers)
{
int length = numbers.length;
double med, var, sd, mean, sum, varsum;
sum = 0;
for (int i = 0; i < length; i++)
{
sum += numbers [ i ];
}
med = numbers [ length / 2];
mean = sum / (double) length;
varsum = 0;
for (int i = 0; i < length; i++)
{
varsum = varsum + ((numbers [ I ] - mean) * (numbers [ I ] - mean));
}
var = varsum / ( length - 1.0 );
sd = Math.sqrt ( var );
System.out.println ("length:
" + length);
System.out.println ("mean:
" + mean);
System.out.println ("median:
" + med);
System.out.println ("variance:
" + var);
System.out.println ("standard deviation: " + sd);
} to Software
Introduction
Testing, Edition 2 (Ch 6)
© Ammann & Offutt
© Ammann & Offutt
59
Control Flow Graph for Stats
public static void computeStats (int [ ] numbers)
{
int length = numbers.length;
double med, var, sd, mean, sum, varsum;
sum = 0;
for (int i = 0; i < length; i++)
{
sum += numbers [ i ];
}
med = numbers [ length / 2];
mean = sum / (double) length;
1
2
i=0
3
i >= length
varsum = 0;
i < length
for (int i = 0; i < length; i++)
4
i++
{
varsum = varsum + ((numbers [ I ] - mean) * (numbers [ I ] - mean));
}
var = varsum / ( length - 1.0 );
sd = Math.sqrt ( var );
System.out.println ("length:
" + length);
System.out.println ("mean:
" + mean);
System.out.println ("median:
" + med);
System.out.println ("variance:
" + var);
System.out.println ("standard deviation: " + sd);
} to Software
Introduction
Testing, Edition 2 (Ch 6)
© Ammann & Offutt
© Ammann & Offutt
5
i=0
6
i < length
i >= length
7
i++
8
60
30
CITS 5501: Logic and Path Coverage
Control Flow TRs and Test Paths—EC
1
Edge Coverage
TR
A. [ 1, 2 ]
B. [ 2, 3 ]
C. [ 3, 4 ]
D. [ 3, 5 ]
E. [ 4, 3 ]
F. [ 5, 6 ]
G. [ 6, 7 ]
H. [ 6, 8 ]
I. [ 7, 6 ]
2
3
4
5
6
7
Test Path
[ 1, 2, 3, 4, 3, 5, 6, 7, 6, 8 ]
8
Introduction to Software
Testing, Edition 2 (Ch 6)
© Ammann & Offutt
61
Control Flow TRs and Test Paths—EPC
Edge-Pair Coverage
1
2
3
4
5
6
7
Introduction to Software
Testing, Edition 2 (Ch 6)
8
TR
A. [ 1, 2, 3 ]
B. [ 2, 3, 4 ]
C. [ 2, 3, 5 ]
D. [ 3, 4, 3 ]
E. [ 3, 5, 6 ]
F. [ 4, 3, 5 ]
G. [ 5, 6, 7 ]
H. [ 5, 6, 8 ]
I. [ 6, 7, 6 ]
J. [ 7, 6, 8 ]
K. [ 4, 3, 4 ]
L. [ 7, 6, 7 ]
Test Paths
i. [ 1, 2, 3, 4, 3, 5, 6, 7, 6, 8 ]
ii. [ 1, 2, 3, 5, 6, 8 ]
iii. [ 1, 2, 3, 4, 3, 4, 3, 5, 6, 7,
6, 7, 6, 8 ]
© Ammann & Offutt
TP
TRs toured
sidetrips
i
A, B, D, E, F, G, I, J
C, H
ii
A, C, E, H
iii
A, B, D, E, F, G, I, J, K,
L
C, H
62
31
CITS 5501: Logic and Path Coverage
Control Flow TRs and Test Paths—PPC
Prime Path Coverage
1
2
3
4
5
6
7
Introduction to Software
Testing, Edition 2 (Ch 6)
TR
A. [ 3, 4, 3 ]
B. [ 4, 3, 4 ]
C. [ 7, 6, 7 ]
D. [ 7, 6, 8 ]
E. [ 6, 7, 6 ]
F. [ 1, 2, 3, 4 ]
G. [ 4, 3, 5, 6, 7 ]
H. [ 4, 3, 5, 6, 8 ]
I. [ 1, 2, 3, 5, 6, 7 ]
J. [ 1, 2, 3, 5, 6, 8 ]
Test Paths
i. [ 1, 2, 3, 4, 3, 5, 6, 7, 6, 8 ]
ii. [ 1, 2, 3, 4, 3, 4, 3,
5, 6, 7, 6, 7, 6, 8 ]
iii. [ 1, 2, 3, 4, 3, 5, 6, 8 ]
iv. [ 1, 2, 3, 5, 6, 7, 6, 8 ]
v. [ 1, 2, 3, 5, 6, 8 ]
8
© Ammann & Offutt
TP
TRs toured
sidetrips
i
A, D, E, F, G
H, I, J
ii
A, B, C, D, E, F, G,
H, I, J
iii
A, F, H
J
iv
D, E, F, I
J
v
J
63
32