Introduction to Software
Testing
(2nd edition)
Chapter 5
Criteria-Based Test Design
Paul Ammann & Jeff Offutt
http://www.cs.gmu.edu/~offutt/softwaretest/
Second version, 17 September 2012
Changing Notions of Testing

Old view focused on testing at each software
development phase as being very different from other
phases
– Unit, module, integration, system …

New view is in terms of structures and criteria
– Graphs, logical expressions, syntax, input space

Test design is largely the same at each phase
– Creating the model is different
– Choosing values and automating the tests is different
Introduction to Software Testing, Edition 2 (Ch 5)
© Ammann & Offutt
2
New : Test Coverage Criteria
A tester’s job is simple : Define a model of the
software, then find ways
to cover it
 Test Requirements : Specific things that must be satisfied
or covered during testing
 Test
Criterion : A collection of rules and a process that
define test requirements
Testing researchers have defined dozens of
criteria, but they are all really just a few criteria
on four types of structures …
Introduction to Software Testing, Edition 2 (Ch 5)
© Ammann & Offutt
3
Criteria Based on Structures
Structures : Four ways to model software
1. Graphs
2. Logical Expressions
3. Input Domain
Characterization
4. Syntactic Structures
Introduction to Software Testing, Edition 2 (Ch 5)
(not X or not Y) and A and B
A: {0, 1, >1}
B: {600, 700, 800}
C: {swe, cs, isa, infs}
if (x > y)
z = x - y;
else
z = 2 * x;
© Ammann & Offutt
4
Old View : Black and White Boxes
Black-box testing : Derive tests from external
descriptions of the software, including specifications,
requirements, and design
 White-box testing : Derive tests from the source code
internals of the software, specifically including branches,
individual conditions, and statements
 Model-based testing : Derive tests from a model of the
software (such as a UML diagram)

MDTD makes these distinctions less important.
The more general question is:
from what abstraction level do we derive tests?
Introduction to Software Testing, Edition 2 (Ch 5)
© Ammann & Offutt
5
Source of Structures

These structures can be extracted from lots of software
artifacts
– Graphs can be extracted from UML use cases, finite state
machines, source code, …
– Logical expressions can be extracted from decisions in program
source, guards on transitions, conditionals in use cases, …

This is not the same as “model-based testing,” which derives
tests from a model that describes some aspects of the
system under test
– The model usually describes part of the behavior
– The source is usually not considered a model
Introduction to Software Testing, Edition 2 (Ch 5)
© Ammann & Offutt
6
1. Graph Coverage – Structural
2
1
6
5
3
This graph may represent
• statements & branches
• methods & calls
• components & signals
4
7
Node
(Statement)
Edge
Path(Branch)
Cover
every
node
Cover
every
Cover everyedge
path
• 12567
• 12567
• 12567
• 1343567
• 1343567
• 1257
• 1357
• 13567
• 1357
• states and transitions
• 1343567
• 134357 …
Introduction to Software Testing, Edition 2 (Ch 5)
© Ammann & Offutt
7
1. Graph Coverage – Data Flow
def = {m}
def = {a , m}
use = {y}
2
def = {x, y}
6
use = {x}
use = {a}
1
5
use = {x} def = {a}
use = {a}
use = {m}
7
Defs
& Uses Pairs
All
Defs
All Uses
• (x, 1, (1,2)), (x, 1, (1,3))
Every
once
Everydef
defused
“reaches”
3
• (y, 1, 4), (y, 1, 6)
This graph contains:
• defs: nodes & edges
where variables get values
• uses: nodes & edges
where values are accessed
• every
1, 2, 5,use
6, 7
4
• (a, 2, (5,6)), (a, 2, (5,7)),
• •1,
2,2,5,5,76,
(a,
3,1,
(5,6)),
(a,73, (5,7)),
def = {m}
75,4,7 7), (m, 6,
• •1,1,2,
3,2,
4,5,3,
• (m,
7),
(m,
7)
use = {y}
• 1, 3, 5, 6, 7
• 1, 3, 5, 7
• 1, 3, 4, 3, 5,7
Introduction to Software Testing, Edition 2 (Ch 5)
© Ammann & Offutt
8
1. Graph - FSM Example
Memory Seats in a Lexus ES 300
Guard (safety constraint)
Trigger (input)
[Ignition = off] | Button2
Driver 2
Configuration
Driver 1
Configuration
[Ignition = off] | Button1
[Ignition = on] | seatBack ()
(to Modified)
Ignition = off
New
Configuration
Driver 2
[Ignition = on] | seatBottom ()
[Ignition = on] | lumbar ()
[Ignition = on] | sideMirrors ()
Ignition = off
[Ignition = on] | Reset AND Button2
New
Configuration
Driver 1
Modified
Configuration
[Ignition = on] | Reset AND Button1
Introduction to Software Testing, Edition 2 (Ch 5)
© Ammann & Offutt
9
2. Logical Expressions
( (a > b) or G ) and (x < y)
Transitions
Program Decision Statements
Logical
Expressions
Software Specifications
Introduction to Software Testing, Edition 2 (Ch 5)
© Ammann & Offutt
10
2. Logical Expressions
( (a > b) or G ) and (x < y)

Predicate Coverage : Each predicate must be true and
false
– ( (a>b) or G ) and (x < y) = True, False

Clause Coverage : Each clause must be true and false
– (a > b) = True, False
– G = True, False
– (x < y) = True, False

Combinatorial Coverage : Various combinations of
clauses
– Active Clause Coverage: Each clause must determine the
predicate’s result
Introduction to Software Testing, Edition 2 (Ch 5)
© Ammann & Offutt
11
2. Logic – Active Clause Coverage
( (a > b) or G ) and (x < y)
With these
values for G and
(x<y), (a>b)
determines the
value of the
predicate
Introduction to Software Testing, Edition 2 (Ch 5)
1
T
F
T
2
F
F
T
3
F
T
T
4
F
F
T
5
T
T
T
6
T
T
F
© Ammann & Offutt
duplicate
12
3. Input Domain Characterization

Describe the input domain of the software
– Identify inputs, parameters, or other categorization
– Partition each input into finite sets of representative values
– Choose combinations of values

System level
– Number of students
– Level of course
– Major

{ 0, 1, >1 }
{ 600, 700, 800 }
{ swe, cs, isa, infs }
Unit level
– Parameters
F (int X, int Y)
– Possible values
X: { <0, 0, 1, 2, >2 }, Y : { 10, 20, 30 }
– Tests
F (-5, 10), F (0, 20), F (1, 30), F (2, 10), F (5, 20)
Introduction to Software Testing, Edition 2 (Ch 5)
© Ammann & Offutt
13
4. Syntactic Structures


Based on a grammar, or other syntactic definition
Primary example is mutation testing
1.
2.
3.
4.

Induce small changes to the program: mutants
Find tests that cause mutant programs to fail: killing mutants
Failure is defined as different output from the original program
Check the output of useful tests on the original program
Example program and mutants
if (x > y)
z = x - y;
else
z = 2 * x;
Introduction to Software Testing, Edition 2 (Ch 5)
if (x > y)
if (x >= y)
z = x - y;
 z = x + y;
 z = x – m;
else
z = 2 * x;
© Ammann & Offutt
14
Coverage Overview
Four Structures for
Modeling Software
Graphs
Logic
Input Space
Syntax
Applied to
Applied
to
Source
Specs
Source
Specs
Design
Introduction to Software Testing, Edition 2 (Ch 5)
Applied
to
FSMs
DNF
Source
Models
Integ
Use cases
© Ammann & Offutt
Input
15
Coverage
Given a set of test requirements TR for coverage
criterion C, a test set T satisfies C coverage if and only
if for every test requirement tr in TR, there is at least
one test t in T such that t satisfies tr

Infeasible test requirements : test requirements that
cannot be satisfied
– No test case values exist that meet the test requirements
– Dead code
– Detection of infeasible test requirements is formally undecidable for most
test criteria

Thus, 100% coverage is impossible in practice
Introduction to Software Testing, Edition 2 (Ch 5)
© Ammann & Offutt
16
Two Ways to Use Test Criteria
1. Directly generate test values to satisfy the criterion
– often assumed by the research community
– most obvious way to use criteria
– very hard without automated tools
2. Generate test values externally and measure against the
criterion usually favored by industry
– sometimes misleading
– if tests do not reach 100% coverage, what does that
mean?
Test criteria are sometimes called
metrics
Introduction to Software Testing, Edition 2 (Ch 5)
© Ammann & Offutt
17
Generators and Recognizers
Generator : A procedure that automatically generates
values to satisfy a criterion
 Recognizer : A procedure that decides whether a given
set of test values satisfies a criterion

Both problems are provably undecidable for most criteria
 It is possible to recognize whether test cases satisfy a
criterion far more often than it is possible to generate
tests that satisfy the criterion


Coverage analysis tools are quite plentiful
Introduction to Software Testing, Edition 2 (Ch 5)
© Ammann & Offutt
18
Comparing Criteria with
Subsumption

Criteria Subsumption : A test criterion C1 subsumes C2 if
and only if every set of test cases that satisfies criterion
C1 also satisfies C2

Must be true for every set of test cases

Example : If a test set has covered every branch in a
program (satisfied the branch criterion), then the test set
is guaranteed to also have covered every statement
Introduction to Software Testing, Edition 2 (Ch 5)
© Ammann & Offutt
19
Test Coverage Criteria
Traditional software testing is expensive and laborintensive
 Formal coverage criteria are used to decide which test
inputs to use
 More likely that the tester will find problems
 Greater assurance that the software is of high quality and
reliability
 A goal or stopping rule for testing
 Criteria makes testing more efficient and effective

How do we start applying these ideas in practice?
Introduction to Software Testing, Edition 2 (Ch 5)
© Ammann & Offutt
20
How to Improve Testing ?
Testers need more and better software tools
 Testers need to adopt practices and techniques that lead
to more efficient and effective testing

– More education
– Different management organizational strategies

Testing / QA teams need more technical expertise
– Developer expertise has been increasing dramatically

Testing / QA teams need to specialize more
– This same trend happened for development in the 1990s
Introduction to Software Testing, Edition 2 (Ch 5)
© Ammann & Offutt
21
Four Roadblocks to Adoption
1.
Lack of test education
Microsoft and Google say half their engineers are testers, programmers test half the time
Number of UG CS programs in US that require testing ?
0
Number of MS CS programs in US that require testing ?
0
Number of UG testing classes in the US ?
2.
Necessity to change process
3.
Usability of tools
~30
Adoption of many test techniques and tools require changes in development process
This is very expensive for most software companies
Many testing tools require the user to know the underlying theory to use them
Do we need to know how an internal combustion engine works to drive ?
Do we need to understand parsing and code generation to use a compiler ?
4.
Weak and ineffective tools
Most test tools don’t do much – but most users do not realize they could be better
Few tools solve the key technical problem – generating test values automatically
Introduction to Software Testing, Edition 2 (Ch 5)
© Ammann & Offutt
22
Needs From Researchers
1.
Isolate : Invent processes and techniques that isolate the
theory from most test practitioners
2. Disguise : Discover engineering techniques, standards
and frameworks that disguise the theory
3. Embed : Theoretical ideas in tools
4. Experiment : Demonstrate economic value of criteriabased testing and ATDG (ROI)
–
–
5.
Which criteria should be used and when ?
When does the extra effort pay off ?
Integrate high-end testing with development
Introduction to Software Testing, Edition 2 (Ch 5)
© Ammann & Offutt
23
Needs From Educators
1.
Disguise theory from engineers in classes
2.
Omit theory when it is not needed
3.
Restructure curriculum to teach more than test design
and theory
– Test automation
– Test evaluation
– Human-based testing
– Test-driven development
Introduction to Software Testing, Edition 2 (Ch 5)
© Ammann & Offutt
24
Changes in Practice
1.
Reorganize test and QA teams to make effective use of
individual abilities
–
2.
Retrain test and QA teams
–
–
3.
Use a process like MDTD
Learn more testing concepts
Encourage researchers to embed and isolate
–
4.
One math-head can support many testers
We are very responsive to research grants
Get involved in curricular design efforts through
industrial advisory boards
Introduction to Software Testing, Edition 2 (Ch 5)
© Ammann & Offutt
25
Future of Software Testing
1. Increased specialization in testing teams will lead to
more efficient and effective testing
2. Testing and QA teams will have more technical expertise
3. Developers will have more knowledge about testing and
motivation to test better
4. Agile processes put testing first—putting pressure on
both testers and developers to test better
5. Testing and security are starting to merge
6. We will develop new ways to test connections within
software-based systems
Introduction to Software Testing, Edition 2 (Ch 5)
© Ammann & Offutt
26
Advantages of CriteriaBased Test Design

Criteria maximize the “bang for the buck”
– Fewer tests that are more effective at finding faults


Comprehensive test set with minimal overlap
Traceability from software artifacts to tests
– The “why” for each test is answered
– Built-in support for regression testing
A “stopping rule” for testing—advance knowledge of how
many tests are needed
 Natural to automate

Introduction to Software Testing, Edition 2 (Ch 5)
© Ammann & Offutt
27
Some Open Questions

Which criteria work best on embedded, highly reliable
software?
– Which software structure to use?

How can we best automate this testing with robust tools?
– Deriving the software structure
– Constructing the test requirements
– Creating values from test requirements
– Creating full test scripts
– Solution to the “mapping problem”
Empirical validation
 Technology transition
 Application to new domains

Introduction to Software Testing, Edition 2 (Ch 5)
© Ammann & Offutt
28
Criteria Summary
• Many companies still use “monkey testing”
• A human sits at the keyboard, wiggles the mouse and
bangs the keyboard
• No automation
• Minimal training required
• Some companies automate human-designed tests
• But companies that use both automation and criteriabased testing
Save money
Find more faults
Build better software
Introduction to Software Testing, Edition 2 (Ch 5)
© Ammann & Offutt
29
Summary of Part 1’s New Ideas

Why do we test – to reduce the risk of using the software
– faults, failures, the RIP model
– Test process maturity levels – level 4 is a mental discipline that
improves the quality of the software

Model-Driven Test Design
– Four types of test activities – test design, automation, execution
and evaluation

Test Automation
– Testability, observability and controllability, test automation
frameworks

Criteria-based test design
– Four structures – test requirements and criteria
Earlier and better testing can empower the
test manager
Introduction to Software Testing, Edition 2 (Ch 5)
© Ammann & Offutt
30