Coverage Criteria
Drawn mostly from
Ammann&Offutt and Pezze&Yooung
IEEE definition of V&V
• Validation: The process of evaluating software at
the end of software development to ensure
compliance with intended usage
• Verification: The process of determining whether
the products of a given phase of the software
development process fulfill the requirements
established during the previous phase
2
Static vs dynamic testing
• Static testing : Testing without executing the
program
• Include software inspections and some forms of
analyses
• Very effective at finding certain kinds of problems –
especially “potential” faults, that is, problems that could
lead to faults when the program is modified
• Dynamic testing : Testing by executing the program
with real inputs
3
Testing vs debugging
• Testing : Finding inputs that cause the software to
fail
• Debugging : The process of finding a fault given a
failure
4
A test case
• Test case values: The values that directly satisfy one
test requirement
• Expected results: The result that will be produced
when executing the test if the program satisfies it
intended behavior
5
Top-down vs bottom up testing
• Top-down testing : Test the main procedure, then
go down through procedures it calls, and so on
• Bottom-up testing : Test the leaves in the tree
(procedures that make no calls), and move up to
the root.
• Each procedure is not tested until all of its children have
been tested
6
Test requirement
• Test requirements: Specific things (software
artifacts) that must be satisfied or covered during
testing
• Non-software example: test a bag of jelly beans
• Come up with ways to test
• Suppose the following flavors: Lemon (yellow), pistachio
(green), cantaloupe (orange), pear (white), tangerine
(orange), apricot (yellow)
• One requirement: test each flavor (six test
requirements)
7
Test criterion
• Test criterion: A collection of rules and a process
that define test requirements
Flavor criterion: TR = {flavor=lemon, flavor=pistachio,
flavor=cantaloupe, flavor=pear, flavor=tangerine,
flavor=apricot}
8
Test coverage
• Given a set of test requirements TR for coverage
criterion C, a test set T satisfies C coverage if and
only if for every test requirement tr in TR, there is
at least one test t in T such that t satisfies tr
• A test case with 12 beans: 3 lemons, 1 pistachio, 2
cantaloupe, 1 pear, 1 tangerine, 4 apricot
• OK to satisfy a test requirement with more than
one test
9
Coverage level
• Given a set of test requirements TR and a test set T,
the coverage level is simply the ratio of the number
of test requirements satisfied by T to the size of TR
• Why? Sometime test requirements may be
infeasible
• Example: suppose tangerine jelly beans are rare
and some bags may not contain any
• Flavor criteria cannot be 100% satisfied
• Maximum coverage level: 5/6 or 83%
10
Criteria vs subsumption
• Criteria subsumption : A test criterion C1 subsumes
C2 if and only if every set of test cases that satisfies
criterion C1 also satisfies C2
• Must be true for every set of test cases
• Example: color criteria for the jelly bean: {yellow,
green, orange, white}
• If we satisfy flavor criteria, we’ll satisfy color criteria
• Example : If a test set has covered every branch in a
program (satisfied the branch criterion), then the
test set is guaranteed to also have covered every
statement
11
Coverage is not size
• Coverage does not depend on the number of test
cases
• T0 , T1 : T1 >coverage T0 T1 <cardinality T0
• T1 , T2 : T2 =coverage T1 T2 >cardinality T1
• Small test cases make failure diagnosis easier
• A failing test case in T2 gives more information for
fault localization than a failing test case in T1
12
Question 1
• Suppose that coverage criterion C1 subsumes
coverage criterion C2. Further suppose that test set
T1 satisfies C1 on program P and test set T2
satisfies C2, also on P.
• Does T1 necessarily satisfy C2? Explain.
13
Question 2
• Suppose that coverage criterion C1 subsumes
coverage criterion C2. Further suppose that test set
T1 satisfies C1 on program P and test set T2
satisfies C2, also on P.
• Does T1 necessarily satisfy C2? Explain.
Yes. This follows directly from the definition of
subsumption.
14
Question 2
• Suppose that coverage criterion C1 subsumes
coverage criterion C2. Further suppose that test set
T1 satisfies C1 on program P and test set T2
satisfies C2, also on P.
• Does T2 necessarily satisfy C1? Explain.
15
Question 2
• Suppose that coverage criterion C1 subsumes
coverage criterion C2. Further suppose that test set
T1 satisfies C1 on program P and test set T2
satisfies C2, also on P.
• Does T2 necessarily satisfy C1? Explain.
• No. There is no reason to expect test requirements
generated by C1 to be satisfied by T2.
16
Question 3
• Suppose that coverage criterion C1 subsumes
coverage criterion C2. Further suppose that test set
T1 satisfies C1 on program P and test set T2
satisfies C2, also on P.
• If P contains a fault, and T2 reveals the fault, T1
does not necessarily also reveal the fault. Explain.
17
Question 3
• Suppose that coverage criterion C1 subsumes
coverage criterion C2. Further suppose that test set
T1 satisfies C1 on program P and test set T2
satisfies C2, also on P.
• If P contains a fault, and T2 reveals the fault, T1
does not necessarily also reveal the fault. Explain.
No. This is the hard question. Testers often think that test sets
for strong criteria are at least as good at finding faults as test
sets for weaker criteria. But subsumption is about criteria, not
about test sets. In particular, there is no requirement that test
set T2 be a subset of test set T1. So, it could happen that T2
contains that one test that reveals the fault, and T1 doesn't.
18
Statements vs branches
• Traversing all edges of a graph causes all nodes to
be visited
• So test suites that satisfy the branch adequacy criterion
for a program P also satisfy the statement adequacy
criterion for the same program
• The converse is not true: A statement-adequate (or
node-adequate) test suite may not be branchadequate (edge-adequate)
19
“All branches” can still miss
conditions
• Sample fault: missing operator (negation)
digit_high == -1 || digit_low == -1
• Branch adequacy criterion can be satisfied by
varying only digit_low
• The faulty sub-expression might never determine the
result
• We might never really test the faulty condition, even
though we tested both outcomes of the branch
20
Condition testing
• Branch coverage exposes faults in how a
computation has been decomposed into cases
• Intuitively attractive: check the programmer’s case
analysis
• But only roughly: groups cases with the same outcome
• Condition coverage considers case analysis in more
detail
• Also individual conditions in a compound Boolean
expression
• e.g., both parts of digit_high == 1 || digit_low == -1
21
Basic conditions vs branches
• Basic condition adequacy criterion can be satisfied
without satisfying branch coverage
• Branch and basic condition are not comparable
(neither implies the other)
22
Covering branches and conditions
• Branch and condition adequacy: cover all conditions and all
decisions
• Compound condition adequacy
• Cover all possible evaluations of compound conditions
• Cover all branches of a decision tree
digit_high == -1
true
digit_low == 1
true
TRUE
false
FALSE
false
FALSE
Ch 12, slide 23
Compound conditions: Exponential
complexity
(((a || b) && c) || d) && e
Test
a
b
c
d
e
(1)
T
—
T
—
T
(2)
F
T
T
—
T
(3)
T
—
F
T
T
(4)
F
T
F
T
T
(5)
F
F
—
T
T
(6)
T
—
T
—
F
(7)
F
T
T
—
F
(8)
T
—
F
T
F
(9)
F
T
F
T
F
(10)
F
F
—
T
F
(11)
T
—
F
F
—
(12)
F
T
F
F
—
(13)
F
F
—
F
—
Case
Short-circuit evaluation often reduces this to a more manageable
number, but not always
24
Multiple (compound) condition
coverage
• The multiple condition coverage of T is computed
as Cc/(Ce -Ci) , where Cc is the number of
combinations covered, Ci is the number of
infeasible simple combinations, and Ce is the total
number of combinations in the program
• Potentially a large number of test cases
25
Modified condition/decision (MC/DC)
• Motivation: Effectively test important combinations
of conditions, without exponential blowup in test
suite size
• “Important” combinations means: Each basic condition
shown to independently affect the outcome of each
decision
• Compound condition as a whole evaluates to true for
one and false for the other
26
MC/DC: linear complexity
• N+1 test cases for N basic conditions
(((a || b) && c) || d) && e
Test
Case
(1)
(2)
(3)
(6)
(11)
(13)
a
b
c
d
e
outcome
true
false
true
true
true
false
-true
---false
true
true
false
true
false
--
--true
-false
false
true
true
true
false
---
true
true
true
false
false
false
• Underlined values independently affect the output of the decision
• Required by the RTCA/DO-178B standard
27
Comments on MC/DC
• MC/DC is
• Basic condition coverage (C)
• Branch coverage (DC)
• Plus one additional condition (M): every condition must
independently affect the decision’s output
• It is subsumed by compound conditions and subsumes
all other criteria discussed so far
• Stronger than statement and branch coverage
• A good balance of thoroughness and test size (and
therefore widely used)
• Federal Aviation Administration’s requirement that
test suites be MC/DC adequate
28
Paths? (Beyond individual branches)
• Should we explore sequences of branches (paths) in the
control flow?
• Many more paths than branches
• A pragmatic compromise will be needed
Ch 12, slide 29
Path adequacy
• Decision and condition adequacy criteria consider
individual program decisions
• Path testing focuses consider combinations of
decisions along paths
• Adequacy criterion: each path must be executed at
least once
• Coverage:
# executed paths
# paths
(c) 2007 Mauro Pezzè &
Michal Young
Ch 12, slide 30
Practical path coverage criteria
• The number of paths in a program with loops is
unbounded
• the simple criterion is usually impossible to satisfy
• For a feasible criterion: Partition infinite set of
paths into a finite number of classes
• Useful criteria can be obtained by limiting
• the number of traversals of loops
• the length of the paths to be traversed
• the dependencies among selected paths
(c) 2007 Mauro Pezzè &
Michal Young
Ch 12, slide 31
Cyclomatic adequacy
• Cyclomatic number: number of independent paths in the
CFG
• If e = #edges, n = #nodes, c = #connected components of a
graph, it is:
• e - n + c for an arbitrary graph
• e - n + 2 for a CFG
• Cyclomatic coverage counts the number of independent
paths that have been exercised, relative to cyclomatic
complexity
(c) 2007 Mauro Pezzè &
Michal Young
Ch 12, slide 32