Software Testing
Logic Coverage
Logic Coverage
Four Structures for
Modeling Software
Graphs
Logic
Input Space
Syntax
Applied to
Applied
to
Source
Specs
Source
Specs
Design
Introduction to Software Testing (Ch 3)
Applied
to
FSMs
DNF
Source
Models
Integ
Use cases
© Ammann & Offutt
Input
2
Logic Predicates
• A predicate is an expression that evaluates to a boolean
value
• Predicates can contain
– boolean variables
– non-boolean variables that contain >, <, ==, >=, <=, !=
– boolean function calls
• Internal structure is created by logical operators
– ¬ – the negation operator
–  – the and operator
–  – the or operator
–  – the implication operator
–  – the exclusive or operator
–  – the equivalence operator
3
Predicate Coverage
• The first (and simplest) two criteria require that each
predicate and each clause be evaluated to both true and
false
Predicate Coverage (PC) : For each p in P, TR
contains two requirements: p evaluates to true, and p
evaluates to false.
• When predicates come from conditions on edges, this is
equivalent to edge coverage
4
Examples
(a < b)  f (z)  D  (m >= n*o)
• Four clauses:
– (a < b) – relational expression
– f (z) – Boolean-valued function
– D – Boolean variable
– (m >= n*o) – relational expression
• Sources of predicates
– Decisions in programs
– Guards in finite state machines
– Decisions in UML activity graphs
– Requirements, both formal and informal
– SQL queries
5
Shortcut for Predicates in Conjunctive
Normal Form
• Conjunctive clauses are connected only by the and operator
– ABC…
• Each major clause is made active by making all other clauses
true
• ACC tests are “all true” and then a “diagonal” of false values:
1
2
3
4
A
T
F
T
T
B
T
T
F
T
.
.
.
…
C
T
T
T
F
…
.
.
.
6
Shortcut for Predicates in Disjunctive
Normal Form
• Disjunctive clauses are connected only by the or operator
– ABC…
• Each major clause is made active by making all other clauses
false
• ACC tests are “all false” and then a “diagonal” of true values:
1
2
3
4
A
F
T
F
F
B
F
F
T
F
.
.
.
…
C
F
F
F
T
…
.
.
.
7
Predicate Coverage Example
((a < b)  D)  (m >= n*o)
predicate coverage
Predicate = true
a = 5, b = 10, D = true, m = 1, n = 1, o = 1
= (5 < 10)  true  (1 >= 1*1)
= true  true  TRUE
= true
Predicate = false
a = 10, b = 5, D = false, m = 1, n = 1, o = 1
= (10 < 5)  false  (1 >= 1*1)
= false  false  TRUE
= false
8
Clause Coverage Example
((a < b)  D)  (m >= n*o)
Clause coverage
(a < b) = true
(a < b) = false
D = true D = false
a = 5, b = 10
a = 10, b = 5
D = true
m >= n*o = true
D = false
m >= n*o = false
m = 1, n = 1, o = 1 m = 1, n = 2, o = 2
false cases
Two tests
true cases
1) a = 5, b = 10, D = true, m = 1, n = 1, o = 1
2) a = 10, b = 5, D = false, m = 1, n = 2, o = 2
9
Combinatorial Coverage
• CoC requires every possible combination
• Sometimes called Multiple Condition Coverage
Combinatorial Coverage (CoC) : For each p in P, TR has test
requirements for the clauses in Cp to evaluate to each possible
combination of truth values.
a<b D
1
2
3
4
5
6
7
8
T
T
T
T
F
F
F
F
T
T
F
F
T
T
F
F
m >= n*o
T
F
T
F
T
F
T
F
((a < b)  D)  (m >= n*o)
T
F
T
F
T
F
F
F
10
Combinatorial Coverage
• This is simple, neat, clean, and comprehensive …
• But quite expensive!
• 2N tests, where N is the number of clauses
– Impractical for predicates with more than 3 or 4 clauses
• The literature has lots of suggestions – some confusing
• The general idea is simple:
Test each clause independently from the other clauses
Determining Predicates
P=A B
P=A B
if B = true, p is always true.
if B = false, p is always false.
so if B = false, A determines p.
so if B = true, A determines p.
if A = false, B determines p.
if A = true, B determines p.
• Goal : Find tests for each clause when the
clause determines the value of the
predicate
• This is formalized in several criteria that
have subtle, but very important,
differences
12
Active Clause Coverage
Active Clause Coverage (ACC) : For each p in P and each
major clause ci in Cp, choose minor clauses cj, j != i, so that ci
determines p. TR has two requirements for each ci : ci
evaluates to true and ci evaluates to false.
p=ab
1) a = true, b = false
a is major clause
2) a = false, b = false
b is major clause
3) a = false, b = true
4) a = false, b = false
Duplicate
• Ambiguity : Do the minor clauses have to have the same values
when the major clause is true and false?
13
A More Subtle Example
p = ( a  b )  ( a  ¬ b)
pa = pa=true  pa=false
= ((true  b)  (true  ¬ b))  ((false  b)  (false  ¬ b))
= (b  ¬ b)  false
= true  false
= true
p = ( a  b )  ( a  ¬ b)
pb = pb=true  pb=false
= ((a  true)  (a  ¬ true))  ((a  false)  (a  ¬ false))
= (a  false)  (false  a)
=aa
= false
• a always determines the value of this predicate
• b never determines the value – b is irrelevant !
14
Condition Coverage (cont’d)
if A or B then
s1
else
s2
end_if_then_el
se
A
B
Branch
test 1
T
F
true
test 2
F
F
false
Compound Condition Coverage
(cont’d)
Combinations of condition
values: TT, TF, FT, FF
input(X,Y)
if (Y<=0) or (X=0) then
Y := -Y
end_if
Thermostat (pg 1 of 2)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
// Jeff Offutt--October 2010
// Programmable Thermostat
import java.io.*;
class thermostat
{
private Heater myHeater;
// Decide whether to turn the heater on, and for how long.
public boolean turnHeaterOn (
int curTemp, /* Current temperature reading */
int thresholdDiff, /* Temp difference until we turn heater on */
Minutes timeSinceLastRun, /* Time since heater stopped */
Minutes minLag, /* How long I need to wait */
Time timeOfDay, /* current time (Hours and minutes) */
Day dayOfWeek, /* Monday, Tuesday, ... */
Settings programmedSettings [], /* User's program, by day */
boolean Override, /* Has user overridden the program */
int overTemp /* OverridingTemp */
)
17
Thermostat (pg 2 of 2)
19 {
20 int desiredTemp;
21 // getPeriod() translates time into Morning, Day, Evening, Night
22 desiredTemp = programmedSettings [dayOfWeek].getDesiredTemp
23
(getPeriod [TimeOfDay]);
24 if (((curTemp < desiredTemp - thresholdDiff) ||
25
(Override && curTemp < overTemp - thresholdDiff)) &&
26
timeSinceLastRun.greaterThan (minLag))
27 { // Turn on the heater
28
// How long? Assume 1 minute per degree (Fahrenheit)
29
int timeNeeded = curTemp - desiredTemp;
30
if (Override)
31
timeNeeded = curTemp - overTemp;
32
myHeater.setRunTime (timeNeeded);
33
return (true);
33 }
34 else
35
return (false);
36 } // End turnHeaterOn
37 } // End class
18
Two Thermostat Predicates
24-26: (((curTemp < desiredTemp - thresholdDiff) ||
(Override && curTemp < overTemp - thresholdDiff)) &&
timeSinceLastRun.greaterThan (minLag))
30: (Override)
Simplify
a : curTemp < desiredTemp - thresholdDiff
b : Override
c : curTemp < overTemp - thresholdDiff
d : timeSinceLastRun.greaterThan (minLag)
24-26: (a || (b && c)) && d
30:
b
19
Predicate Coverage (true)
(a || (b && c)) && d
a : true
b : true
c : true
d : true
curTemp < desiredTemp – thresholdDiff : true
Override : true
curTemp < overTemp – thresholdDiff : true
timeSinceLastRun.greaterThan (minLag) : true
programmedSettings [Monday].setDesiredTemp (Morning, 69)
// dayOfWeek = Monday
// timeOfDay = 8:00
curTemp = 63
63 < 69 – 5
Override : true
overTemp = 70
63 < 70 – 5
timeSinceLastRun.setValue (12)
minLag = 10
20
Predicate Coverage (false)
(a || (b && c)) && d
a : false
b : false
c : false
d : false
curTemp < desiredTemp – thresholdDiff : false
Override : false
curTemp < overTemp – thresholdDiff : false
timeSinceLastRun.greaterThan (minLag) : false
programmedSettings [Monday].setDesiredTemp (Morning, 69)
// dayOfWeek = Monday
// timeOfDay = 8:00
curTemp = 66
66 < 69 – 5
Override : false
overTemp = 70
66 < 70 – 5
timeSinceLastRun.setValue (8)
minLag = 10
21
Correlated Active Clause Coverage (1 of 5)
Pa = ((a || (b && c)) && d)  ((a || (b && c)) && d)
((T || (b && c)) && d)  ((F || (b && c)) && d)
(T && d)  ((b && c) && d)
d  ((b && c) && d)
!(b && c) && d
( !b || !c ) && d
22
Correlated Active Clause Coverage (2 of 5)
a:
b:
c:
d:
(a || (b && c)) && d
T t
f
t
F t
f
t
f T
t
t
f F
t
t
f t
T
t
f t
F
f
t
t
t
T
t t
t
F
(row 3)
(row 13)
(row 9)
(row 13)
(row 9)
(row 11)
(row 1)
(row 2)
duplicates
Six tests needed for CACC on Thermostat
23
Correlated Active Clause Coverage (3 of 5)
curTemp desiredTemp thresholdDiff
curTemp < desiredTemp - thresholdDiff
63
69
5
!(curTemp < desiredTemp - thresholdDiff) 66
69
5
desiredTemp = programmedSettings [Monday].setDesiredTemp (Morning, 69)
dayOfWeek = Monday
timeOfDay = 8:00
Override
Override T
!Override F
These values then need
to be placed into calls to
turnHeaterOn() to satisfy
the 6 tests for CACC
curTemp overTemp thresholdDiff
curTemp < overTemp - thresholdDiff
63
70
5
!(curTemp < overTemp - thresholdDiff)
66
70
5
timeSinceLastRun minLag
timeSinceLastRun.greaterThan (minLag)
12
10
!(timeSinceLastRun.greaterThan (minLag))
8
10
24
Correlated Active Clause Coverage (4 of 5)
desiredTemp = programmedSettings [Monday].setDesiredTemp (Morning, 69)
1. T t f t
a = T : curTemp = 63; c = f : curTemp = 66
turnHeaterOn ( 63/66, 5, 12, 10, 8:00, Monday, programmedSettings, true, 70 )
2. F t f t
turnHeaterOn ( 66, 5, 12, 10, 8:00, Monday, programmedSettings, true, 70 )
3. f T t t
a = f : curTemp = 66; c = t : curTemp = 63
turnHeaterOn ( 63/66, 5, 12, 10, 8:00, Monday, programmedSettings, true, 70 )
4. f t F f
turnHeaterOn ( 66, 5, 8, 10, 8:00, Monday, programmedSettings, true, 70 )
5. t t t T
turnHeaterOn ( 63, 5, 12, 10, 8:00, Monday, programmedSettings, true, 70 )
6. t t t F
turnHeaterOn ( 63, 5, 8, 10, 8:00, Monday, programmedSettings, true, 70 )
25
Correlated Active Clause Coverage (5 of 5)
• curTemp is fixed by the solution to clause a
• thresholdDiff is also fixed by the solution to clause a
• So we choose different values for overtemp ...
1. T t f t
turnHeaterOn ( 63, 5, 12, 10, 8:00, Monday, programmedSettings, true, 62 )
3. f T t t
turnHeaterOn ( 66, 5, 12, 10, 8:00, Monday, programmedSettings, true, 66 )
26
Program Transformation Issues
if ((a && b) || c) {
S1;
}
else {
S2;
}
Transform (2)?
d = a && b;
e = d || c;
if (e) {
S1;
}
else {
S2;
}
Transform (1)?
if (a) {
if (b)
S1;
else {
if (c)
S1;
else
S2;
}
}
else {
if (c)
S1;
else
S2;
}
27
Problems with Transformed Programs
• Maintenance is certainly harder
with Transform (1)
– Not recommended!
• Coverage on Transform (1)
– PC on transform does not imply
CACC on original
– CACC on original does not imply
PC on transform
• Coverage on Transform (2)
– Structure used by logic criteria is
“lost”
– Hence CACC on transform 2 only
requires 3 tests
– Note: Mutation analysis (Chapter
5) addresses this problem
a b c (ab)c
T T T
T
T
T
T
F
F
F
F
T
F
F
T
T
F
F
F
T
F
T
F
T
F
T
T
F
T
F
T
F
CACC
PC
CACC (2)
X
X
X
X
X
X
X
X
X
X
X
X
• Bottom Line: Logic coverage
criteria are there to help you!
28
float exam, course, average; char trailer[20];
cin>>exam;
cin>>course;
if ( (exam<0.0) || (exam>100.0) || (course<0.0) || (course>100.00) )
cout<<"Marks out of range - program terminating";
else {
if ( (exam<50.0) || (course<50.0))
cout<<"Fail";
else if (exam < 60)
cout<<"Pass - Grade C";
else if ( ((exam+course)/2.0) >= 70.0) {
average = (exam+course)/2.0;
printf("Pass - Grade A with an average mark of %5.2f\n",
average);
cout<<"Pass - Grade A with an average mark of "<< average
<<endl;}
else
cout<<"Pass - Grade B"; }
29