Introduction to Software Testing
Chapter 1
Introduction
Paul Ammann & Jeff Offutt
www.introsoftwaretesting.com
What is testing?
Test inputs
Software
under test
Test outputs
Acceptability test

Yes
Outputs
OK?
Debug
2
What is testing?
No
Test outputs
OK?
Bug in
SUT?
No
Bug in
Acceptability
test?
No
Bug in
specification?
No
Yes
Yes
Yes
Yes
Bug in OS,
compiler,
libraries,
hardware
3
Types of Test Activities – Summary
1.
Test design
Design test values to satisfy coverage criteria or
other engineering goal
Requires technical knowledge of discrete math,
programming and testing
2.
Test automation Embed test values into executable scripts
Requires knowledge of scripting
3.
Test execution
Run tests on the software and record the results
Requires very little knowledge
4.
Test evaluation
Evaluate results of testing, report to developers
Requires domain knowledge


These four general test activities are quite different
It is a poor use of resources to use people inappropriately
But most test organizations use the same people for
ALL FOUR activities !!
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com
© Ammann & Offutt
4
Important Terms
Validation & Verification (IEEE)

Validation : The process of evaluating software at the end of
software development to ensure compliance with intended
usage

Verification : The process of determining whether the products
of a given phase of the software development process fulfill the
requirements established during the previous phase
IV&V stands for “independent verification and validation”
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com
© Ammann & Offutt
5
Static and Dynamic Testing

Static Testing : Testing without executing the program

Dynamic Testing : Testing by executing the program with real
inputs
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com
© Ammann & Offutt
6
Software Faults, Errors & Failures

Software Fault : A static defect in the software

Software Error : An incorrect internal state that is the
manifestation of some fault

Software Failure : External, incorrect behavior with respect to
the requirements or other description of the expected behavior
Faults in software are design mistakes and will always exist
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com
© Ammann & Offutt
7
Testing & Debugging

Testing : Finding inputs that cause the software to fail

Debugging : The process of finding a fault given a failure
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com
© Ammann & Offutt
8
Fault & Failure Model
Three conditions necessary for a failure to be observed
1. Reachability : The location or locations in the program that
contain the fault must be reached
2. Infection : The state of the program must be incorrect
3. Propagation : The infected state must propagate to cause some
output of the program to be incorrect
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com
© Ammann & Offutt
9
Test Case

Test Case Values : The values that directly satisfy one test
requirement

Expected Results : The result that will be produced when
executing the test if the program satisfies its intended behavior
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com
© Ammann & Offutt
10
White-box and Black-box Testing

Black-box testing : Deriving tests from external descriptions of
the software, including specifications, requirements, and design

White-box testing : Deriving tests from the source code internals
of the software, specifically including branches, individual
conditions, and statements
This view is really out of date.
The more general question is: from what level of abstraction
to we derive tests?
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com
© Ammann & Offutt
11
Changing Notions of Testing

Old view of testing is of testing at specific
software development phases
– Unit, module, integration, system …

New view is in terms of structures and criteria
– Graphs, logical expressions, syntax, input space
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com
© Ammann & Offutt
12
Old : Testing at Different Levels

Acceptance testing: Is
the software acceptable
to the user?

System testing: Test the
overall functionality of
the system

Integration testing:
Test how modules
interact with each
other

Module testing: Test
each class, file, module
or component

Unit testing: Test each
unit (method)
individually
main Class P
Class A
Class B
method mA1()
method mB1()
method mA2()
method mB2()
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com
© Ammann & Offutt
13
New : Test Coverage Criteria
A tester’s job is simple : Define a model of the
software, then find ways
to cover it
 Test
Requirements : Specific things that must be satisfied or
covered during testing
 Test
Criterion : A collection of rules and a process that define
test requirements
Testing researchers have defined dozens of criteria, but they
are all really just a few criteria on four types of structures …
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com
© Ammann & Offutt
14
New : Criteria Based on Structures
Structures : Four ways to model software
1. Graphs
2. Logical Expressions
3. Input Domain
Characterization
4. Syntactic Structures
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com
(not X or not Y) and A and B
A: {0, 1, >1}
B: {600, 700, 800}
C: {swe, cs, isa, infs}
if (x > y)
z = x - y;
else
z = 2 * x;
© Ammann & Offutt
15
1. Graph Coverage – Structural
2
1
6
5
Node
Edge
(Branch)
Path(Statement)
Cover
Cover
every
edge
Coverevery
everynode
path
3
This graph may represent
• statements & branches
• methods & calls
7
4
•••12567
12567
12567
•••1343567
1343567
1257
•• 1357
13567
• components & signals
• 1357
• states and transitions
• 1343567
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com
© Ammann & Offutt
• 134357 …
16
1. Graph Coverage – Data Flow
def = {m}
def = {a , m}
use = {y}
2
def = {x, y}
6
use = {x}
use = {a}
1
5
use = {x} def = {a}
use = {a}
use = {m}
7
Defs & Uses Pairs
3
• (x, 1, (1,2)), (x, 1, (1,3))
• (y, 1, 4), (y, 1, 6)
This graph contains:
• defs: nodes & edges where
variables get values
4
def = {m}
use = {y}
• (a, 2, (5,6)), (a, 2, (5,7)), (a,
3, (5,6)), (a, 3, (5,7)),
• (m, 2, 7), (m, 4, 7), (m, 6, 7)
• uses: nodes & edges where
values are accessed
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com
© Ammann & Offutt
17
1. Graph - FSM Example
Memory Seats in a Lexus ES 300
Guard (safety constraint)
Trigger (input)
[Ignition = off] | Button2
Driver 2
Configuration
Driver 1
Configuration
[Ignition = off] | Button1
[Ignition = on] | seatBack ()
(to Modified)
Ignition = off
New
Configuration
Driver 2
[Ignition = on] | seatBottom ()
[Ignition = on] | lumbar ()
[Ignition = on] | sideMirrors ()
Ignition = off
[Ignition = on] | Reset AND Button2
New
Configuration
Driver 1
Modified
Configuration
[Ignition = on] | Reset AND Button1
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com
© Ammann & Offutt
18
2. Logical Expressions
( (a > b) or G ) and (x < y)
Transitions
Logical
Program Decision Statements
Expressions
Software Specifications
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com
© Ammann & Offutt
19
2. Logical Expressions
( (a > b) or G ) and (x < y)

Predicate Coverage : Each predicate must be true and false
– ( (a>b) or G ) and (x < y) = True, False

Clause Coverage : Each clause must be true and false
– (a > b) = True, False
– G = True, False
– (x < y) = True, False

Combinatorial Coverage : Various combinations of clauses
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com
© Ammann & Offutt
20
2. Logic – Active Clause Coverage
( (a > b) or G ) and (x < y)
With these values
for G and (x<y),
(a>b) determines
the value of the
predicate
1
T
F
T
2
F
F
T
3
F
T
T
4
F
F
T
5
T
T
T
6
T
T
F
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com
© Ammann & Offutt
duplicate
21
3. Input Domain Characterization

Describe the input domain of the software
– Identify inputs, parameters, or other categorization
– Partition each input into finite sets of representative values
– Choose combinations of values

System level
– Number of students
– Level of course
– Major

{ 0, 1, >1 }
{ 600, 700, 800 }
{ swe, cs, isa, infs }
Unit level
– Parameters
F (int X, int Y)
– Possible values
X: { <0, 0, 1, 2, >2 }, Y : { 10, 20, 30 }
– Tests
• F (-5, 10), F (0, 20), F (1, 30), F (2, 10), F (5, 20)
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com
© Ammann & Offutt
22
4. Syntactic Structures


Based on a grammar, or other syntactic definition
Primary example is mutation testing
1.
2.
3.
4.

Induce small changes to the program: mutants
Find tests that cause the mutant programs to fail: killing mutants
Failure is defined as different output from the original program
Check the output of useful tests on the original program
Example program and mutants
if (x > y)
z = x - y;
else
z = 2 * x;
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com
if (x > y)
if (x >= y)
z = x - y;
 z = x + y;
 z = x – m;
else
z = 2 * x;
© Ammann & Offutt
23
Source of Structures

These structures can be extracted from lots of software artifacts
– Graphs can be extracted from UML use cases, finite state machines,
source code, …
– Logical expressions can be extracted from decisions in program source,
guards on transitions, conditionals in use cases, …
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com
© Ammann & Offutt
24
Coverage Overview
Four Structures for
Modeling Software
Graphs
Logic
Input Space
Syntax
Applied to
Applied
to
Source
Specs
Source
Specs
Design
Applied
to
FSMs
DNF
Source
Integ
Use cases
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com
Models
© Ammann & Offutt
Input
25
Coverage
Given a set of test requirements TR for coverage criterion
C, a test set T satisfies C coverage if and only if for every
test requirement tr in TR, there is at least one test t in T
such that t satisfies tr

Infeasible test requirements : test requirements that cannot be
satisfied
– No test case values exist that meet the test requirements
– Detection of infeasible test requirements is formally undecidable for most test
criteria

Thus, 100% coverage is impossible in practice
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com
© Ammann & Offutt
26
Generators and Recognizers




Generator : A procedure that automatically generates values to
satisfy a criterion
Recognizer : A procedure that decides whether a given set of
test values satisfies a criterion
It is possible to recognize whether test cases satisfy a criterion
far more often than it is possible to generate tests that satisfy the
criterion
Coverage analysis tools are quite plentiful
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com
© Ammann & Offutt
27
Comparing Criteria with Subsumption

Criteria Subsumption : A test criterion C1 subsumes C2 if and
only if every set of test cases that satisfies criterion C1 also
satisfies C2

Must be true for every set of test cases
Example : If a test set has covered every branch in a program
(satisfied the branch criterion), then the test set is guaranteed to
also have covered every statement

Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com
© Ammann & Offutt
28
Introduction to Software Testing
Chapter 2.1, 2.2
Overview Graph Coverage Criteria
Paul Ammann & Jeff Offutt
www.introsoftwaretesting.com
Three Example Graphs
0
0
1
2
3
3
1
4
7
2
5
8
0
6
9
1
Not a
valid
graph
3
N0 = { 0 }
N0 = { 0, 1, 2 }
N0 = { }
Nf = { 3 }
Nf = { 7, 8, 9 }
Nf = { 3 }
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
2
30
Paths in Graphs

Path : A sequence of nodes – [n1, n2, …, nM]
– Each pair of nodes is an edge

Length : The number of edges
– A single node is a path of length 0


Subpath : A subsequence of nodes in p is a subpath of p
Reach (n) : Subgraph that can be reached from n
0
1
2
Paths
[ 0, 3, 7 ]
3
4
5
6
[ 1, 4, 8, 5, 1 ]
[ 2, 6, 9 ]
7
8
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
Reach (0) = { 0, 3, 4,
7, 8, 5, 1, 9 }
Reach ({0, 2}) = G
Reach([2,6]) = {2, 6,
9}
9
© Ammann & Offutt
31
Test Paths


Test Path : A path that starts at an initial node and ends at a
final node
Test paths represent execution of test cases
1
0
4
3
2
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
6
5
Double-diamond graph
Four test paths
[ 0, 1, 3, 4, 6 ]
[ 0, 1, 3, 5, 6 ]
[ 0, 2, 3, 4, 6 ]
[ 0, 2, 3, 5, 6 ]
© Ammann & Offutt
32
Visiting and Touring


Visit : A test path p visits node n if n is in p
A test path p visits edge e if e is in p
Tour : A test path p tours subpath q if q is a subpath of p
Path [ 0, 1, 3, 4, 6 ]
Visits nodes 0, 1, 3, 4, 6
Visits edges (0, 1), (1, 3), (3, 4), (4, 6)
Tours subpaths (0, 1, 3), (1, 3, 4), (3, 4, 6), (0, 1, 3, 4), (1, 3, 4, 6)
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
33
Tests and Test Paths

path (t) : The test path executed by test t

path (T) : The set of test paths executed by the set of tests T

Each test executes one and only one test path
A location in a graph (node or edge) can be reached from
another location if there is a sequence of edges from the first
location to the second

– Syntactic reach : A subpath exists in the graph
– Semantic reach : A test exists that can execute that subpath
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
34
Tests and Test Paths
test 1
many-to-one
Test
Path
test 2
test 3
Deterministic software – a test always executes the same test path
test 1
many-to-many
Test Path 1
test 2
Test Path 2
test 3
Test Path 3
Non-deterministic software – a test can execute different test paths
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
35
Testing and Covering Graphs
(2.2)

We use graphs in testing as follows :
– Developing a model of the software as a graph
– Requiring tests to visit or tour specific sets of nodes, edges or subpaths
• Test Requirements (TR) : Describe properties of test paths
• Test Criterion : Rules that define test requirements
• Satisfaction : Given a set TR of test requirements for a graph
criterion C, a test set T satisfies C on graph G if and only if for
every test requirement tr in TR, there is a test path p in path(T)
such that p meets tr.
• Structural Coverage Criteria : Defined on a graph just in terms
of nodes and edges
• Data Flow Coverage Criteria : Requires a graph to be annotated
with references to variables
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
36
Node and Edge Coverage

The first (and simplest) two criteria require that each node and
edge in a graph be executed
Node Coverage (NC) : Test set T satisfies node coverage on
graph G iff for every syntactically reachable node n in N,
there is some path p in path(T) such that p visits n.
• This statement is a bit cumbersome, so we abbreviate it in terms
of the set of test requirements
Node Coverage (NC) : TR contains each reachable node in G.
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
37
Node and Edge Coverage

Edge coverage is slightly stronger than node coverage
Edge Coverage (EC) : TR contains each reachable path of
length up to 1, inclusive, in G.
• The “length up to 1” allows for graphs with one node and no
edges
• NC and EC are only different when there is an edge and another
subpath between a pair of nodes (as in an “if-else” statement)
0
1
2
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
Node Coverage : TR = { 0, 1, 2 }
Test Path = [ 0, 1, 2 ]
Edge Coverage : TR = { (0,1), (0, 2), (1, 2) }
Test Paths = [ 0, 1, 2 ]
[ 0, 2 ]
© Ammann & Offutt
38
Covering Multiple Edges

Edge-pair coverage requires pairs of edges, or subpaths of
length 2
Edge-Pair Coverage (EPC) : TR contains each reachable path
of length up to 2, inclusive, in G.
• The “length up to 2” is used to include graphs that have less
than 2 edges
• The logical extension is to require all paths …
Complete Path Coverage (CPC) : TR contains all paths in G.
• Unfortunately, this is impossible if the graph has a loop, so a
weak compromise is to make the tester decide which paths:
Specified Path Coverage (SPC) : TR contains a set S of test
paths, where S is supplied as a parameter.
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
39
Structural Coverage Example
Node Coverage
TR = { 0, 1, 2, 3, 4, 5, 6 }
Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 5, 4, 6 ]
0
Edge Coverage
TR = { (0,1), (0,2), (1,2), (2,3), (2,4), (3,6), (4,5), (4,6), (5,4) }
Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 2, 4, 5, 4, 6 ]
1
Edge-Pair Coverage
TR = { [0,1,2], [0,2,3], [0,2,4], [1,2,3], [1,2,4], [2,3,6],
[2,4,5], [2,4,6], [4,5,4], [5,4,5], [5,4,6] }
Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 6 ] [ 0, 2, 3, 6 ]
[ 0, 2, 4, 5, 4, 5, 4, 6 ]
2
3
4
5
6
Complete Path Coverage
Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 6 ] [ 0, 1, 2, 4, 5, 4, 6 ]
[ 0, 1, 2, 4, 5, 4, 5, 4, 6 ] [ 0, 1, 2, 4, 5, 4, 5, 4, 5, 4, 6 ] …
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
40
Loops in Graphs

If a graph contains a loop, it has an infinite number of paths

Thus, CPC is not feasible

SPC is not satisfactory because the results are subjective and
vary with the tester

Attempts to “deal with” loops:
–
–
–
–
1970s : Execute cycles once ([4, 5, 4] in previous example, informal)
1980s : Execute each loop, exactly once (formalized)
1990s : Execute loops 0 times, once, more than once (informal description)
2000s : Prime paths
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
41
Simple Paths and Prime Paths

Simple Path : A path from node ni to nj is simple if no node
appears more than once, except possibly the first and last nodes
are the same
– No internal loops
– Includes all other subpaths
– A loop is a simple path

Prime Path : A simple path that does not appear as a proper
subpath of any other simple path
0
1
2
Simple Paths : [ 0, 1, 3, 0 ], [ 0, 2, 3, 0], [ 1, 3, 0, 1 ],
[ 2, 3, 0, 2 ], [ 3, 0, 1, 3 ], [ 3, 0, 2, 3 ], [ 1, 3, 0, 2 ],
[ 2, 3, 0, 1 ], [ 0, 1, 3 ], [ 0, 2, 3 ], [ 1, 3, 0 ], [ 2, 3, 0 ],
[ 3, 0, 1 ], [3, 0, 2 ], [ 0, 1], [ 0, 2 ], [ 1, 3 ], [ 2, 3 ], [ 3, 0 ],
[0], [1], [2], [3]
3
Prime Paths : [ 0, 1, 3, 0 ], [ 0, 2, 3, 0], [ 1, 3, 0, 1 ],
[ 2, 3, 0, 2 ], [ 3, 0, 1, 3 ], [ 3, 0, 2, 3 ], [ 1, 3, 0, 2 ],
© Ammann & Offutt
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
[ 2, 3, 0, 1 ]
42
Prime Path Coverage

A simple, elegant and finite criterion that requires loops to be
executed as well as skipped
Prime Path Coverage (PPC) : TR contains each prime path in G.
• Will tour all paths of length 0, 1, …
• That is, it subsumes node, edge, and edge-pair coverage
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
43
Touring, Sidetrips and Detours

Prime paths do not have internal loops … test paths might
• Tour : A test path p tours subpath q if q is a subpath of p
• Tour With Sidetrips : A test path p tours subpath q with sidetrips
iff every edge in q is also in p in the same order
• The tour can include a sidetrip, as long as it comes back to the same node
• Tour With Detours : A test path p tours subpath q with detours iff
every node in q is also in p in the same order
• The tour can include a detour from node ni, as long as it comes back to
the prime path at a successor of ni
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
44
Sidetrips and Detours Example
1
0
2
3
2
1
Touring without
sidetrips or
detours
1
0
1
5
4
p = [0, 1, 2, 4, 5 ]
q = [1, 2, 4 ]
3
2
5
6
2
3
Touring with a
sidetrip
5
4
4
3
1
0
4
2
5
2
1
5
4
3
Touring with a
detour
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
3
4
© Ammann & Offutt
45
Infeasible Test Requirements

An infeasible test requirement cannot be satisfied
– Unreachable statement (dead code)
– A subpath that can only be executed if a contradiction occurs (X > 0 and X < 0)
• Most test criteria have some infeasible test requirements
• It is usually undecidable whether all test requirements are
feasible
• When sidetrips are not allowed, many structural criteria have
more infeasible test requirements
• However, always allowing sidetrips weakens the test criteria
Practical recommendation – Best Effort Touring
– Satisfy as many test requirements as possible without sidetrips
– Allow sidetrips to try to satisfy unsatisfied test requirements
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
46
Simple & Prime Path Example
Simple
paths
0
1
2
3
Len 0
[0]
[1]
[2]
[3]
[4]
[5]
[6] !
Len 1
[0, 1]
[0, 2]
[1, 2]
[2, 3]
[2, 4]
[3, 6] !
[4, 6] !
[4, 5]
[5, 4]
4
5
6
‘!’ means path
Len
2
Len 3
terminates
[0, 1, 2]
[0, 1,‘*’
2, means
3]
path
[0, 2, 3]
[0, 1, 2, 4]cycles
[0, 2, 4]
[0, 2, 3, 6] !
[1, 2, 3]
[0, 2, 4, 6] !
[1, 2, 4]
[0, 2, 4, 5] !
[2, 3, 6] !
[1, 2, 3, 6] !
[2, 4, 6] !
[1, 2, 4, 5] !
[2, 4, 5] !
[1, 2, 4, 6] !
[4, 5, 4] *
[5, 4, 6] !
[5, 4, 5] *
Len 4
[0, 1, 2, 3, 6] !
[0, 1, 2, 4, 6] !
[0, 1, 2, 4, 5] !
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
Prime Paths
© Ammann & Offutt
47
Data Flow Criteria
Goal: Try to ensure that values are computed and used correctly




Definition (def) : A location where a value for a variable is
stored into memory
Use : A location where a variable’s value is accessed
def (n) or def (e) : The set of variables that are defined by node n
or edge e
use (n) or use (e) : The set of variables that are used by node n
or edge e
X = 42
0
Defs: def (0) = {X}
Z = X*2
4
1
3
def (4) = {Z}
6
2
5
Z = X-8
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
def (5) = {Z}
Uses: use (4) = {X}
use (5) = {X}
© Ammann & Offutt
48
DU Pairs and DU Paths





DU pair : A pair of locations (li, lj) such that a variable v is
defined at li and used at lj
Def-clear : A path from li to lj is def-clear with respect to variable
v if v is not given another value on any of the nodes or edges in
the path
du-path : A simple subpath that is def-clear with respect to v
from a def of v to a use of v
du (ni, nj, v) – the set of du-paths from ni to nj
du (ni, v) – the set of du-paths that start at ni
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
49
Data Flow Test Criteria
• First, we make sure every def reaches a use
All-defs coverage (ADC) : For each set of du-paths S = du (n,
v), TR contains at least one path d in S.
• Then we make sure that every def reaches all possible uses
All-uses coverage (AUC) : For each set of du-paths to uses S =
du (ni, nj, v), TR contains at least one path d in S.
• Finally, we cover all the paths between defs and uses
All-du-paths coverage (ADUPC) : For each set S = du (ni, nj,
v), TR contains every path d in S.
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
50
Data Flow Testing Example
Z = X*2
1
X = 42
0
4
3
6
2
5
Z = X-8
All-defs for X
All-uses for X
All-du-paths for X
[ 0, 1, 3, 4 ]
[ 0, 1, 3, 4 ]
[ 0, 1, 3, 4 ]
[ 0, 1, 3, 5 ]
[ 0, 2, 3, 4 ]
[ 0, 1, 3, 5 ]
[ 0, 2, 3, 5 ]
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
51
Introduction to Software Testing
Chapter 2.3
Graph Coverage for Source Code
Paul Ammann & Jeff Offutt
www.introsoftwaretesting.com
Overview
The most common application of graph criteria is to
program source
 Graph : Usually the control flow graph (CFG)
 Node coverage : Execute every statement
 Edge coverage : Execute every branch
 Loops : Looping structures such as for loops, while
loops, etc.
 Data flow coverage : Augment the CFG

– defs are statements that assign values to variables
– uses are statements that use variables
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
53
Control Flow Graphs





A CFG models all executions of a method by describing control
structures
Nodes : Statements or sequences of statements (basic blocks)
Edges : Transfers of control
Basic Block : A sequence of statements such that if the first
statement is executed, all statements will be (no branches)
CFGs are sometimes annotated with extra information
– branch predicates
– defs
– uses

Rules for translating statements into graphs …
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
54
CFG : The if Statement
if (x < y)
{
y = 0;
x = x + 1;
}
else
{
x = y;
}
1
x<y
y=0
x=x+1
x >= y
2
3
x=y
4
if (x < y)
{
y = 0;
x = x + 1;
}
1
x<y
y=0
x=x+1
2
x >= y
3
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
55
CFG : The if-Return Statement
if (x < y)
{
return;
}
print (x);
return;
1
x<y
return
2
x >= y
3
print (x)
return
No edge from node 2 to 3.
The return nodes must be distinct.
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
56
Loops

Loops require “extra” nodes to be added

Nodes that do not represent statements or basic blocks
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
57
CFG : while and for Loops
x = 0;
while (x < y)
{
y = f (x, y);
x = x + 1;
}
x=0
1
dummy node
2
x<y
x >= y
3
4
implicitly
initializes loop
x=0
y =f(x,y)
x=x+1
2
for (x = 0; x < y; x++)
{
y = f (x, y);
}
y = f (x, y)
implicitly
increments loop
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
1
© Ammann & Offutt
x<y
x >= y
3
5
4
x=x+1
58
Example Control Flow – Stats
public static void computeStats (int [ ] numbers)
{
int length = numbers.length;
double med, var, sd, mean, sum, varsum;
sum = 0;
for (int i = 0; i < length; i++)
{
sum += numbers [ i ];
}
med = numbers [ length / 2 ];
mean = sum / (double) length;
varsum = 0;
for (int i = 0; i < length; i++)
{
varsum = varsum + ((numbers [ I ] - mean) * (numbers [ I ] - mean));
}
var = varsum / ( length - 1.0 );
sd = Math.sqrt ( var );
System.out.println
System.out.println
System.out.println
System.out.println
System.out.println
("length:
" + length);
("mean:
" + mean);
("median:
" + med);
("variance:
" + var);
("standard deviation: " + sd);
}
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
59
Control Flow Graph for Stats
public static void computeStats (int [ ] numbers)
{
int length = numbers.length;
double med, var, sd, mean, sum, varsum;
sum = 0;
for (int i = 0; i < length; i++)
{
sum += numbers [ i ];
}
med = numbers [ length / 2 ];
mean = sum / (double) length;
1
2
3
i=0
i >= length
varsum = 0;
i < length
for (int i = 0; i < length; i++)
i++ 4
{
varsum = varsum + ((numbers [ I ] - mean) * (numbers [ I ] - mean));
}
var = varsum / ( length - 1.0 );
sd = Math.sqrt ( var );
System.out.println
System.out.println
System.out.println
System.out.println
System.out.println
("length:
" + length);
("mean:
" + mean);
("median:
" + med);
("variance:
" + var);
("standard deviation: " + sd);
}
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
5
i=0
6
i < length
i >= length
7
8
i++
60
Control Flow TRs and Test Paths – EC
1
Edge Coverage
2
TR
Test Path
A. [ 1, 2 ] [ 1, 2, 3, 4, 3, 5, 6, 7, 6, 8 ]
B. [ 2, 3 ]
C. [ 3, 4 ]
D. [ 3, 5 ]
E. [ 4, 3 ]
F. [ 5, 6 ]
G. [ 6, 7 ]
H. [ 6, 8 ]
I. [ 7, 6 ]
3
4
5
6
7
8
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
61
Control Flow TRs and Test Paths – EPC
1
Edge-Pair Coverage
2
3
4
5
6
7
8
TR
A. [ 1, 2, 3 ]
B. [ 2, 3, 4 ]
C. [ 2, 3, 5 ]
D. [ 3, 4, 3 ]
E. [ 3, 5, 6 ]
F. [ 4, 3, 5 ]
G. [ 5, 6, 7 ]
H. [ 5, 6, 8 ]
I. [ 6, 7, 6 ]
J. [ 7, 6, 8 ]
K. [ 4, 3, 4 ]
L. [ 7, 6, 7 ]
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
Test Paths
i. [ 1, 2, 3, 4, 3, 5, 6, 7, 6, 8 ]
ii. [ 1, 2, 3, 5, 6, 8 ]
iii. [ 1, 2, 3, 4, 3, 4, 3, 5, 6, 7,
6, 7, 6, 8 ]
TP
TRs toured
sidetrips
i
A, B, D, E, F, G, I, J
C, H
ii
A, C, E, H
iii
A, B, D, E, F, G, I,
J, K, L
© Ammann & Offutt
C, H
62
Control Flow TRs and Test Paths – PPC
Prime Path Coverage
1
2
3
4
5
6
7
TR
A. [ 3, 4, 3 ]
B. [ 4, 3, 4 ]
C. [ 7, 6, 7 ]
D. [ 7, 6, 8 ]
E. [ 6, 7, 6 ]
F. [ 1, 2, 3, 4 ]
G. [ 4, 3, 5, 6, 7 ]
H. [ 4, 3, 5, 6, 8 ]
I. [ 1, 2, 3, 5, 6, 7 ]
J. [ 1, 2, 3, 5, 6, 8 ]
8
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
Test Paths
i. [ 1, 2, 3, 4, 3, 5, 6, 7, 6, 8 ]
ii. [ 1, 2, 3, 4, 3, 4, 3,
5, 6, 7, 6, 7, 6, 8 ]
iii. [ 1, 2, 3, 4, 3, 5, 6, 8 ]
iv. [ 1, 2, 3, 5, 6, 7, 6, 8 ]
v. [ 1, 2, 3, 5, 6, 8 ]
TP
TRs toured
sidetrips
i
A, D, E, F, G
H, I, J
ii
A, B, C, D, E, F, G,
H, I, J
iii
A, F, H
J
iv
D, E, F, I
J
v
J
© Ammann & Offutt
63
Data Flow Coverage for Source

def : a location where a value is stored into memory
–
–
–
–

use : a location where variable’s value is accessed
–
–
–
–
–

x appears on the left side of an assignment (x = 44;)
x is an actual parameter in a call and the method changes its value
x is a formal parameter of a method (implicit def when method starts)
x is an input to a program
x appears on the right side of an assignment
x appears in a conditional test
x is an actual parameter to a method
x is an output of the program
x is an output of a method in a return statement
If a def and a use appear on the same node, then it is only a DUpair if the def occurs after the use and the node is in a loop
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
64
Example Data Flow – Stats
public static void computeStats (int [ ] numbers)
{
int length = numbers.length;
double med, var, sd, mean, sum, varsum;
sum = 0;
for (int i = 0; i < length; i++)
{
sum += numbers [ i ];
}
mean = sum / (double) length;
med = numbers [ length / 2 ];
varsum = 0;
for (int i = 0; i < length; i++)
{
varsum = varsum + ((numbers [ i ] - mean) * (numbers [ i ] - mean));
}
var = varsum / ( length - 1.0 );
sd = Math.sqrt ( var );
System.out.println
System.out.println
System.out.println
System.out.println
System.out.println
("length:
" + length);
("mean:
" + mean);
("median:
" + med);
("variance:
" + var);
("standard deviation: " + sd);
}
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
65
Control Flow Graph for Stats
1
( numbers )
sum = 0
length = numbers.length
2
i=0
3
i >= length
i < length
4
5
sum += numbers [ i ]
i++
mean = sum / (double) length;
med = numbers [ length / 2 ]
varsum = 0
i=0
6
i >= length
i < length
7
varsum = …
i++
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
8
var = varsum / ( length - 1.0 )
sd = Math.sqrt ( var )
print (length, mean, med, var, sd)
© Ammann & Offutt
66
CFG for Stats – With Defs & Uses
1
def (1) = { numbers, sum, length }
2
def (2) = { i }
3
use (3, 5) = { i, length }
use (3, 4) = { i, length }
4
5
def (5) = { mean, med, varsum, i }
use (5) = { numbers, length, sum }
def (4) = { sum, i }
use (4) = { sum, numbers, i }
6
use (6, 8) = { i, length }
use (6, 7) = { i, length }
7
def (7) = { varsum, i }
use (7) = { varsum, numbers, i, mean }
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
8
def (8) = { var, sd }
use (8) = { varsum, length, mean,
med, var, sd }
© Ammann & Offutt
67
Defs and Uses Tables for Stats
Node
1
2
3
4
5
Def
Edge
Use
{ numbers, sum,
length }
{i}
{ sum, i }
{ mean, med,
varsum, i }
(1, 2)
(2, 3)
{ numbers, i, sum }
{ numbers, length, sum }
8
{ varsum, i }
{ var, sd }
(3, 4)
(4, 3)
{ i, length }
(3, 5)
{ i, length }
(5, 6)
6
7
Use
{ varsum, numbers, i,
mean }
{ varsum, length, var,
mean, med, var, sd }
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
(6, 7)
{ i, length }
(7, 6)
(6, 8)
{ i, length }
68
DU Pairs for Stats
DU Pairs defs come before uses, do
not count as DU pairs
variable
numbers (1, 4) (1, 5) (1, 7)
length
(1, 5) (1, 8) (1, (3,4)) (1, (3,5)) (1, (6,7)) (1, (6,8))
med
var
sd
mean
sum
varsum
i
(5, 8)
(8, 8)
defs after use in loop,
these are valid DU pairs
(8, 8)
(5, 7) (5, 8)
No def-clear path …
(1, 4) (1, 5) (4, 4) (4, 5)
different scope for i
(5, 7) (5, 8) (7, 7) (7, 8)
(2, 4) (2, (3,4)) (2, (3,5)) (2, 7) (2, (6,7)) (2, (6,8))
(4, 4) (4, (3,4)) (4, (3,5)) (4, 7) (4, (6,7)) (4, (6,8))
(5, 7) (5, (6,7)) (5, (6,8))
(7, 7) (7, (6,7)) (7, (6,8))
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
69
DU Paths for Stats – No Duplicates
There are 38 DU paths for Stats, but only 12 unique
[ 1, 2, 3, 4 ]
[ 1, 2, 3, 5 ]
[ 1, 2, 3, 5, 6, 7 ]
[ 1, 2, 3, 5, 6, 8 ]
[ 2, 3, 4 ]
[ 2, 3, 5 ]
[ 4, 3, 4 ]
[ 4, 3, 5 ]
[ 5, 6, 7 ]
[ 5, 6, 8 ]
[ 7, 6, 7 ]
[ 7, 6, 8 ]
5 expect a loop not to be “entered”
5 require at least one iteration of a loop
2 require at least two iteration of a loop
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
70
Test Cases and Test Paths
Test Case : numbers = (44) ; length = 1
Test Path : [ 1, 2, 3, 4, 3, 5, 6, 7, 6, 8 ]
Additional DU Paths covered (no sidetrips)
[ 1, 2, 3, 4 ] [ 2, 3, 4 ] [ 4, 3, 5 ] [ 5, 6, 7 ] [ 7, 6, 8 ]
The five stars
that require at least one iteration of a loop
Test Case : numbers = (2, 10, 15) ; length = 3
Test Path : [ 1, 2, 3, 4, 3, 4, 3, 4, 3, 5, 6, 7, 6, 7, 6, 7, 6, 8 ]
DU Paths covered (no sidetrips)
[ 4, 3, 4 ] [ 7, 6, 7 ]
The two stars
that require at least two iterations of a loop
Other DU paths require arrays with length 0 to skip loops
But the method fails with divide by zero on the statement …
mean = sum / (double) length;
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
A fault was
found
© Ammann & Offutt
71
Graph Coverage Criteria Subsumption
Complete Path
Coverage
CPC
All-DU-Paths
Coverage
ADUP
All-uses
Coverage
AUC
All-defs
Coverage
ADC
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
Prime Path
Coverage
PPC
Edge-Pair
Coverage
EPC
Edge
Coverage
EC
Node
Coverage
NC
© Ammann & Offutt
72
Introduction to Software Testing
Chapter 2.4
Graph Coverage for Design Elements
Paul Ammann & Jeff Offutt
www.introsoftwaretesting.com
Structural Graph Coverage for DE



Call Graph: The most common graph for structural design
testing
Nodes : Units (in Java – methods)
Edges : Calls to units
Node coverage : call every unit at least
once (method coverage)
A
B
C
E
D
F
Edge coverage : execute every call at
least once (call coverage)
Example call graph
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
74
Data Flow at the Design Level

Data flow couplings among units and classes are more
complicated than control flow couplings
– When values are passed, they “change names”
– Many different ways to share data
– Finding defs and uses can be difficult – finding which uses a def can reach
is very difficult (in unit test defs and uses were in the same unit)
• Caller : A unit that invokes another unit
• Callee : The unit that is called
• Callsite : Statement or node where the call appears
• Actual parameter : Variable in the caller
• Formal parameter : Variable in the callee
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
75
Example Call Site
interface
A

B (x)

end A
B (Y)

end B



Caller
Actual
Parameter
Callee
Formal
Parameter
Applying data flow criteria to def-use pairs between units is too
expensive
Too many possibilities
But this is integration testing, and we really only care about the
interface …
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
76
Inter-procedural DU Pairs

If we focus on the interface, then we just need to consider the last
definitions of variables before calls and returns and first uses
inside units and after calls

Last-def : The set of nodes that define a variable x and has a defclear path from the node through a callsite to a use in the other
unit
– Can be from caller to callee (parameter or shared variable) or from callee
to caller as a return value

First-use : The set of nodes that have uses of a variable y and for
which there is a def-clear and use-clear path from the callsite to
the nodes
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
77
Coupling DU-path
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
78
Example Inter-procedural DU Pairs
Caller
F
X = 14

y = G (x)

DU pair
print (y)
last-def
x=5
10
callsite
2
first-use
Callee
G (a) print (a)

DU
pair b = 42

return (b)
1
x=4
3 x=3
4
B (int y)
B (x)
11 Z = y
12 T = y
13 print (y)
first-use
Last Defs
2, 3
last-def
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
First Uses
11, 12
© Ammann & Offutt
79
Example – Quadratic
1 // Program to compute the quadratic root for
two numbers
2 import java.lang.Math;
3
4 class Quadratic
5{
6 private static float Root1, Root2;
7
8 public static void main (String[] argv)
9 {
10 int X, Y, Z;
11 boolean ok;
12 int controlFlag = Integer.parseInt (argv[0]);
13 if (controlFlag == 1)
14 {
15
X = Integer.parseInt (argv[1]);
16
Y = Integer.parseInt (argv[2]);
17
Z = Integer.parseInt (argv[3]);
18 }
19 else
20 {
21
22
23
24
X = 10;
Y = 9;
Z = 12;
}
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
25
ok = Root (X, Y, Z);
26
if (ok)
27
System.out.println
28
(“Quadratic: ” + Root1 + Root2);
29
else
30
System.out.println (“No Solution.”);
31 }
32
33 // Three positive integers, finds quadratic root
34 private static boolean Root (int A, int B, int C)
35 {
36
float D;
37
boolean Result;
38
D = (float) Math.pow ((double)B,
39
40
41
(double2-4.0)*A*C );
if (D < 0.0)
{
Result = false;
42
return (Result);
43
}
44
Root1 = (float) ((-B + Math.sqrt(D))/(2.0*A));
45
Root2 = (float) ((-B – Math.sqrt(D))/(2.0*A));
46
Result = true;
47
return (Result);
48 } / /End method Root
49
50 ©} Ammann
// End
class Quadratic
80
& Offutt
last-defs
1 // Program to compute the quadratic root for two numbers
2 import java.lang.Math;
3
4 class Quadratic
5{
6 private static float Root1, Root2;
shared variables
7
8 public static void main (String[] argv)
9 {
10 int X, Y, Z;
11 boolean ok;
12 int controlFlag = Integer.parseInt (argv[0]);
13 if (controlFlag == 1)
14 {
15
X = Integer.parseInt (argv[1]);
16
Y = Integer.parseInt (argv[2]);
17
Z = Integer.parseInt (argv[3]);
18 }
19 else
20 {
21
22
23
24
X = 10;
Y = 9;
Z = 12;
}
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
81
first-use
first-use
last-def
last-defs
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
ok = Root (X, Y, Z);
if (ok)
System.out.println
(“Quadratic: ” + Root1 + Root2);
else
System.out.println (“No Solution.”);
}
// Three positive integers, finds the quadratic root
private static boolean Root (int A, int B, int C)
{
float D;
boolean Result;
D = (float) Math.pow ((double)B, (double2-4.0)*A*C);
if (D < 0.0)
{
Result = false;
42
return (Result);
43
}
44
Root1 = (float) ((-B + Math.sqrt(D)) / (2.0*A));
45
Root2 = (float) ((-B – Math.sqrt(D)) / (2.0*A));
46
Result = true;
47
return (Result);
48 } / /End method Root
49
50 } // End class Quadratic
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
82
Quadratic – Coupling DU-pairs
Pairs of locations: method name, variable name, statement
(main (), X, 15) – (Root (), A, 38)
(main (), Y, 16) – (Root (), B, 38)
(main (), Z, 17) – (Root (), C, 38)
(main (), X, 21) – (Root (), A, 38)
(main (), Y, 22) – (Root (), B, 38)
(main (), Z, 23) – (Root (), C, 38)
(Root (), Root1, 44) – (main (), Root1, 28)
(Root (), Root2, 45) – (main (), Root2, 28)
(Root (), Result, 41) – ( main (), ok, 26 )
(Root (), Result, 46) – ( main (), ok, 26 )
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
© Ammann & Offutt
83
Introduction to Software Testing
Chapter 5.1
Syntax-based Testing
Paul Ammann & Jeff Offutt
www.introsoftwaretesting.com
Mutation testing






The original and most widely known application of syntax-based
testing is to modify programs
Operators modify a ground string (program under test) to
create mutant programs
Mutant programs must compile correctly (valid strings)
Mutants are not tests, but used to find tests
Once mutants are defined, tests must be found to cause mutants
to fail when executed
This is called “killing mutants”
Introduction to Software Testing (Ch 5), www.introsoftwaretesting.com
© Ammann & Offutt
85
Killing Mutants
Given a mutant m  M for a ground string program P
and a test t, t is said to kill m if and only if the output
of t on P is different from the output of t on m.



If mutation operators are designed well, the resulting tests will
be very powerful
Different operators must be defined for different programming
languages and goals
Testers can keep adding tests until all mutants have been killed
–
–
–
–
Dead mutant : A test case has killed it
Stillborn mutant : Syntactically illegal
Trivial mutant : Almost every test can kill it
Equivalent mutant : No test can kill it (equivalent to original program)
Introduction to Software Testing (Ch 5), www.introsoftwaretesting.com
© Ammann & Offutt
86
Program-based Grammars
Original Method
int Min (int A, int B)
{
int minVal;
minVal = A;
if (B < A)
{
minVal = B;
}
return (minVal);
} // end Min
6 mutants
Each represents a
separate program
Introduction to Software Testing (Ch 5), www.introsoftwaretesting.com
With Embedded Mutants
int Min (int A, int B)
Replace one variable
{
with another
int minVal;
minVal = A;
Changes operator
∆ 1 minVal = B;
if (B < A)
Immediate runtime
∆ 2 if (B > A)
failure … if reached
∆ 3 if (B < minVal)
{
Immediate runtime
minVal = B;
failure if B==0 else
∆4
Bomb ();
does nothing
∆5
minVal = A;
∆6
minVal = failOnZero (B);
}
return (minVal);
} // end Min
© Ammann & Offutt
87
Syntax-Based Coverage Criteria
Mutation Coverage (MC) : For each m  M, TR contains
exactly one requirement, to kill m.
• The RIP model from chapter 1:
• Reachability : The test causes the faulty statement to be
reached (in mutation – the mutated statement)
• Infection : The test causes the faulty statement to result in an
incorrect state
• Propagation : The incorrect state propagates to incorrect
output
• The RIP model leads to two variants of mutation coverage …
Introduction to Software Testing (Ch 5), www.introsoftwaretesting.com
© Ammann & Offutt
88
Syntax-Based Coverage Criteria
• 1) Strongly Killing Mutants:
Given a mutant m  M for a program P and a test t, t is said to
strongly kill m if and only if the output of t on P is different
from the output of t on m
• 2) Weakly Killing Mutants:
Given a mutant m  M that modifies a location l in a program
P, and a test t, t is said to weakly kill m if and only if the state of
the execution of P on t is different from the state of the
execution of m immediately on t after l
• Weakly killing satisfies reachability and infection, but not
propagation
Introduction to Software Testing (Ch 5), www.introsoftwaretesting.com
© Ammann & Offutt
89
Weak Mutation
Weak Mutation Coverage (WMC) : For each m  M, TR
contains exactly one requirement, to weakly kill m.
• Some mutants can be killed under weak mutation but not
under strong mutation (no propagation)
Introduction to Software Testing (Ch 5), www.introsoftwaretesting.com
© Ammann & Offutt
90
Mutation Example

Mutant 1 in the Min( ) example is:
minVal = A;
∆ 1 minVal = B;
if (B < A)
minVal = B;
• The complete test specification to kill mutant 1:
•
•
•
•
Reachability : true // Always get to that statement
Infection : A ≠ B
Propagation: (B < A) = false // Skip the next assignment
Full Test Specification : true  (A ≠ B)  ((B < A) = false)
≡ (A ≠ B)  (B ≥A)
≡ (B > A)
• (A = 5, B = 7) will kill mutant 1.
Introduction to Software Testing (Ch 5), www.introsoftwaretesting.com
© Ammann & Offutt
91
Equivalent Mutation Example

Mutant 3 in the Min() example is equivalent:
minVal = A;
if (B < A)
∆ 3 if (B < minVal)
• The infection condition is “(B < A) != (B < minVal)”
• However, the previous statement was “minVal = A”
– Substituing, we get: “(B < A) != (B < A)”
• Thus no input can kill this mutant
Introduction to Software Testing (Ch 5), www.introsoftwaretesting.com
© Ammann & Offutt
92
Strong Versus Weak Mutation
1 boolean isEven (int X)
2 {
Reachability : X < 0
3
if (X < 0)
4
X = 0 - X;
Infection : X != 0
∆4
X = 0;
(X = -6) will kill mutant
5
if (float) (X/2) == ((float) X) / 2.0
4 under weak mutation
6
return (true);
7
else
Propagation :
8
return (false);
((float) ((0-X)/2) == ((float) 0-X) / 2.0)
9
}
!= ((float) (0/2) == ((float) 0) / 2.0)
That is, X is not even …
Thus (X = -6) does not kill the mutant
under strong mutation
Introduction to Software Testing (Ch 5), www.introsoftwaretesting.com
© Ammann & Offutt
93
Designing Mutation Operators


At the method level, mutation operators for different
programming languages are similar
Mutation operators do one of two things:
– Mimic typical programmer mistakes ( incorrect variable name )
– Encourage common test heuristics ( cause expressions to be 0 )

Researchers design lots of operators, then experimentally select
the most useful
Effective Mutation Operators
If tests that are created specifically to kill mutants created
by a collection of mutation operators O = {o1, o2, …} also
kill mutants created by all remaining mutation operators
with very high probability, then O defines an effective set of
mutation operators
Introduction to Software Testing (Ch 5), www.introsoftwaretesting.com
© Ammann & Offutt
94
Software security
Michael Hicks
University of Maryland
Online course (www.coursera.org)
What is computer security?

Most developers and operators are concerned with correctness:
achieving desired behavior
– A working banking web site, word processor, blog, …

Security is concerned with preventing undesired behavior
– Considers an enemy/opponent/hacker/adversary who is actively and
maliciously trying to circumvent any protective measures you put in place
– Kinds of undesired behavior:
• Stealing information (confidentiality)
• Modifying information or functionality (integrity)
• Denying access (availability)
96
Defects and Vulnerabilities

Many breaches begin by exploiting a vulnerability
– This is a security-relevant software defect that can be exploited to effect an
undesired behavior

A software defect is present when the software behaves incorrectly,
i.e., it fails to meet its requirements

Defects occur in the software’s design and its implementation
– A flaw is a defect in the design
– A bug is a defect in the implementation
97
Considering Correctness


All software is buggy. So what?
A normal user never sees most bugs, or works around them
– Most (post-deployment) bugs due to rare feature interactions or failure to
handle edge cases

Assessment: Would be too expensive to fix every bug before
deploying
– So companies only fix the ones most likely to affect normal users
98
Considering Security


Key difference: An adversary is not a normal user
The adversary will actively attempt to find defects in rare feature
interactions and edge cases
– For a typical user, (accidentally) finding a bug will result in a crash, which he
will now try to avoid
– An adversary will work to find a bug and exploit it to achieve his goals
99
To ensure security, we must
eliminate bugs and design
flaws, and/or
make them harder to exploit
100