Mutation Testing
(ch 5)
Course Software Testing & Verification
2013/14
Wishnu Prasetya
Negative Test
• Recall: to test a program outside its normal behavior
range.
• Idea: mutate inputs.
• For inputs described by BNF we can mutate the
grammar instead (5.5.2), e.g. replace/delete/
duplicate terminals and non-terminals in a
production rule.
• Or mutate on-the-fly as we build derivation trees (so
that we can control how many times a mutated rule
is applied).
2
Example
NLpostcode  Street Area
NLpostcode  Area Space Street
Area  FirstDigit Digit Digit
Area  FirstDigit Digit Digit Digit
Street  Letter Letter
FirstDigit  1 | 2 ...
Area  FirstDigit Digit Digit Digit Digit
Digit  0 | 1 | 2 ...
Letter  a | b | c ... | A | B | C ...
Space   | “ “ Space
3
Some coverage concepts for grammarbased mutants generation
• (Def 5.46) Mutation operator (e.g. nonterminal deletion); (Def 5.47) mutant
• (C 5.33) Mutation Operator Coverage (MOC) :
for every mutation operator TR contains one
mutant produced via this operator.
• (C5.34) Mutation Production Cov: for every
production rule and every mutation operator
that can be applied to the rule, TR contains
one mutant via that rule and mutation.
4
Using mutation as “coverage” (5.2)
• We can “mutate” a program to “fake” a mistake; but
the mutant must be a valid program.
• (D 5.45) Ground : the original.
• (D 5.48) Let P be a ground. A test t of P kills a mutant
P’ of P if t’s output of both differ.
• We can use mutants to measure the adequacy of our
test-set: it is good if it can kill all those mutants.
• Obviously, the strength of this scheme depends on
your choice of the mutation operators.
5
Example
Fundamental premise : If a program contains a fault, there will
usually be a set of mutants that can only be killed by test cases
that also detect that fault.
6
(mutations generated by the tool Mothra)
We can also do:
mutation-driven testing
Fundamental premise : If a program contains a fault, there will
usually be a set of mutants that can only be killed by test cases
that also detect that fault.
not enough mutants percentage killed
Program P
construct
test set T
run T on
P
run T on
mutants
Mutants of P
a test-case fails
Fix P
This is a simpler variation of the scheme in Fig. 5.2. The
one above does not presume automated generation of
test-cases.
Things to note (see next slide) :
• reachability of a mutation
• infection
• propagation
7
How to kill a mutant?
P(x) { if (x<0) x++ ;
return x }
P(x) { if (x0) x++ ;
return x }
test() { r = P(1) ; assert (r==2) }
test() { r = P(0) ; assert (r==0) }
• (reachability) you need a test-case whose execution passes
the mutation.
• You need a test-case that actually observes the effect of the
mutation. Note there is a difference in:
– (infection) “incorrect” state, right after the mutation.
– (propagation) the infection propagates to incorrect P’s
output. In principle, a test-case can only kill if there is
propagation.
8
Another example
1 boolean isEven (int X) {
2
if (X < 0)
3
X=0–X;
∆3
X=0;
4
if ((double) (X/2) == ((double) X) / 2.0)
5
return true ;
6
else return false ;
7 }
test1() { assert (isEven(-2)==true) }
9
Issues with killing
• No reachability  usually can easily be fixed.
• You have propagation, but used oracle is only partial
(is not strong enough).  fix the oracle.
• You have infection, but no propagation  as in the
isEven example  find another test-case.
• What if we weaken the definition of “kill” ? We can
at least see what that gives us.
10
Strong and weak kill
• Let t be a test on a ground P, and P’ is a mutant. Let l
be the location of the mutation in P’.
• (D 5.49) t strongly kills P’ if its verdict on P’ and P are
different. In other words, if it fails on P’.
• ( D 5.50) t weakly kills P’ if the states right after l
when executing t on P’ and P are different.
P(x) { if (x<0) x++ ;
return x }
P(x) { if (x0) x++ ;
return x }
test() { r = P(1) ; assert (r0) }
does not even weakly kill
test() { r = P(0) ; assert (r0) }
does not kill, but still weakly kill
11
Mutation-based coverage criteria
• (C5.32) Given a set M of mutants, the TR is simply M: you
have to kill every mutant in M.
• C5.35 if you use “strong killing”, C5.36 for “weak killing”.
• OA remark that in practice strong vs weak do not make much
difference.
• The strength depends on the choice of mutation operators.
See 5.2.2 and 5.3.2.
• Typical problem in mutation-based testing: it generates lots of
mutants, roughly O(#dataobjs*#refs)  make testing very
computation intensive!
• O(#refs) using selective mutation (next slide)
12
Mutation operators
• This depends on the used programming language.
• We can start with some set of mutation operators,
then see if we can minimize it.
• (D5.51) Given a set O of mutation operators; a subset
O1 is effective if test-cases designed specifically to kill
mutants produced by O1 also kill mutants from O/
O1.
• There have been plenty of research in this direction.
For mutations over imperative constructs, see 5.2.2.
Mutations on OO aspects, see 5.3.2.
13
Examples of mutops on imperative
constructs (5.2.2)
• ABS (Absolute Value Insertion) Modify each arithmetic
(sub)expression by using functions abs(), negAbs(), and
failOnZero().
• AOR (Arithmetic Operator Replacement) Replace each
occurrence of arithmetic operators +,－,*,／, % by each
other; and in addition, by leftOp, and rightOp.
• SVR (Scalar Variable Replacement) Replace each variable
reference by another (type compatible and in-scope) variable.
• BSR (Bomb Statement Replacement) Replace each statement
by Bomb().
• More... see book.
14
Addressing integration issues (5.3.2)
• Faults in component integration are caused by a
mismatch of assumptions; e.g. callee thought:
– a list was sorted, caller did not do that
– all fields were initialized, caller only initialized some
– thought values are in kilometers, caller use miles
• Integration mutation focuses on mutating the
connections between components
– Sometimes called “interface mutation”
– Both caller and callee methods are considered
15
Four types of integration mutations
• Change a calling method by modifying the call (e.g.
deleting the call; replacing the returned obj by a
type-compatible obj).
• Change a calling method by modifying values that are
sent to a called method.
• Change a called method by modifying values that
enter and leave a method
– Includes parameters as well as variables from higher
scopes (class level, package, public, etc.)
16
How about OO?
These are typical OO features; also where typical mistakes are made
• Encapsulation (hiding details).
class
subclass
private
+
protected
+
+
public
+
+
internal (C#)
+
no modifier Java
+
no modifier C#
+
protected internal (C#)
+
package
assembly
world
+
+
+
+
+
+
+
17
Typical OO aspects
• Inheritance: method overriding, variables shadowing,
constructors overriding works differently
• Polymorphism
– a required object of type T can be instantiated by a
subclass at the runtime
– overloading
• In addition to object level members we also have
class level members (the ‘static’).
• 5.3.3 lists 20 mutops targeting the above aspects.
18
Examples of inheritance mutops
• OMD (Overriding Method Deletion) Each entire
declaration of an overriding method is deleted.
• OMM (Overridden Method Moving) Each call to a
super.method is moved to the first and last position,
and up and down one statement.
override m() {
x ++ ;
y = y/x ;
base.m() ; // super
}
19
Examples of inheritance mutops
HVD – Hiding Variable Deletion
HVI – Hiding Variable Insertion
point
int x;
int y;
point
int x;
int y;
colorpoint
int x;
1 // int x;
int y;
2 // int y;
colorpoint
1 int x;
2 int y;
20
Examples of inheritance mutops
point
colorpoint
ATC – Actual Type Change
DTC – Declared Type Change
point p;
p = PointFactory();
 p = new colorpoint ();
point p;
 colorpoint p;
p = PointFactory();
21
Examples of overloading mutops
AOC – Argument Order Change
point
point3D
void set (int x, int y, char c);
void set (char a, int x, int y);
point3D p ;
p.set (1, 2, ‘t’) ;
 p.set (‘t’, 1, 2) ;
ANC – Argument Number
Change
point3D
void set (int x, int y, int z);
void set (int x, int y);
void set (int z);
point3D p;
p.set (1, 2, 3);
 p.set (2, 3);
 p.set (3);
22
Examples of “other” mutops
TKD – This Keyword Deletion
point
…
void set (int x, int y)
{
this.x = x;
1 x = x;
this.y = y;
2 y = y;
}
…
VID –Variable
Initialization Deletion
point
int x = 5;
 int x;
…
DCD – Default Constructor
Delete
point
point() { … }
 // point() { … }
…
23
Ok... but how useful is this !?
• Mutation is widely considered the strongest test criterion; but
also the most expensive!
• This because the TRs of ‘other ‘criteria can be encoded by a
mutation (assuming you are completely free to define your
mutops), at least with respect to weak killing (other criteria
impose local requirements, like weak mutation)
– Node coverage, edge coverage, clause coverage
• But.. e.g. correlated active clause coverage  cannot be
expressed in terms of mutation killing, as the TR is expressed
in terms of pairs of tests
(unless we also define a concept of killing a pair of mutants)
24