Automating the Generation of
Mutation Tests
Mike Papadakis and Nicos Malevris
Department of Informatics
Athens University of Economics and Business
1
Test Data Generation Approaches
 Symbolic execution
 Select set of paths
 Produce a system of algebraic constraints
 Solve and generate test cases
 Challenges (Symbolic execution)
 Infeasible paths
 Complex-unhandled expressions
 Availability of source code
2
Test Data Generation Approaches
 Search based approaches
 Definition of program input
 Dynamic program execution
 Fitness function
o (efficiency and effectiveness)
 Challenges
 Handling of dynamic program inputs
 Require a high number of executions
 Handling of specific cases (e.g. flag problem)
 Effective fitness function
3
Test Data Generation Approaches
 Dynamic Symbolic Execution
 Simultaneously perform actual and symbolic
execution
o Simplify the process based on concrete program
execution
 Produce a system of algebraic constraints
 Solve and generate test cases
 Challenges
 Infeasible paths
 Scalability issues
4
Killing Mutants
 In order to kill a mutant, tests must
Stage 1: Reach
the mutant
Stage 2: infect
the program state
Stage 3:
propagate the
infected state
5
Killing Mutants
 Reach the mutant (Reachability condition)
 Infect program state (necessity condition)
 Program execution must cause a discrepancy in the
program state
 E.g. Orig: a > b, Mut = a ≥ b → a > b ≠ a ≥ b
o a == b where Orig = false, Mut = true
 Propagate (sufficiency condition)
 Heuristic approaches
o explore path space, measure of mutant’s impact
 Joint satisfaction (Reach && Infect && Propagate)
6
Killing Mutants (path selection)
 Symbolic Execution
 Reach
o Use paths from program input to mutant
 Infect
o Use necessity constraints to infect program state
o Original Statement ≠ Mutated Statement
 Propagate
o Explore the path space (from mutant to program output)
 Path selection Strategy
7
Killing Mutants (path selection)
 Enhanced Test Model (Enhanced Control Flow
Graph)


Represent mutants in the model
Guide the selection of paths and the produced
constraints based on the Shortest path strategy
8
Killing Mutants (path selection)
 Shortest Path Strategy, the k-value
 An efficient path selection strategy. (eliminates
the effects of infeasible paths)
 Select shortest paths from program input (S) to
the mutant statement (m)
   (S , x)  xm
(k )
m
(k )
 Heuristic approach to weak mutation
 Strong mutation can be tackled incrementally

(k )
m
  (S , x)  xm  mx  ( x, F )
(r )
(t )
9
Killing Mutants Dynamic approaches
 Dynamic approaches
 Gain the required information from actual program
execution
 Require high number of program execution
 Mutation Testing
 Need to unify the runtime information of the original
and mutant programs
 Need to include mutant neccesity and sufficiency
conditions into program structure
 Need for efficiency (high cost)
10
Mutant Schemata
 Mutant schemata-Parameterized mutants
 Technique for efficiently producing mutants
 Embeds all mutants into one meta-mutant program
 Use of global parameter to specify the mutants
 Example
original
meta-mutant
a > b + c → RelationalGT(a, ArithmeticPlus(M(b), M(c)))
M(x)
Arithmetic(x, y)
Relational(x, y)
mutates variable x, e.g. abs(x)
mutates arithmetic operator, e.g. x - y
mutates relational operator, e.g. x ≥ y
11
Mutant Schemata (approach I)
 Weak mutation
 Include the mutant evaluation inside schematic
functions
o Original expression ≠ Mutant expression
o Original statement ≠ Mutant statement
 After evaluation continue with the original execution
(schematic function returns the original expression)
 Able to execute all mutants with one execution run
 Embeds all mutant’s killable condition into
program structure (mutant evaluation → branches)
12
Mutant Schemata
 Proposed approach (Mutants to branches)
 Mutants M[1]…M[n] on node N.
Node
N-1
Mutant
Evaluation
Branches
Node
N_M[1]
Node
N-1
Node
N_M[2]
Node
N
...
Node
N
Mutant is killed
Mutant is Alive
Node
N_M[n-1]
Node
N+1
Node
N+2
Node
N+1
Node
N+2
Node
N_M[n]
13
Mutant Schemata (approach II)
 Strong mutation
 In Same lines as Weak
o Include the mutant evaluation inside schematic functions
 After evaluation continue with the mutant execution
(schematic function returns the mutant expression)
 Incremental approach, from weak to strong
 Requires multiple execution runs
 Record original and mutant’s execution paths
 Evaluate strongly killed mutants
14
Killing Mutants (DSE)
 Use the shortest path heuristic
 Negate the condition that will lead closer to mutant
node (reach the mutant)
 Weakly kill the mutant
 Fulfill the mutant necessity condition
 Make the mutant evaluation Branch true
 Negate the produced conditions after the mutant
 Start from the mutant program
 Explores the mutant program path space
15
DSE-Example Scenario
Target
Mutant
16
DSE-Example Scenario
Target
Mutant
17
DSE-Example Scenario
Target
Mutant
18
DSE-Example Scenario
Target
Mutant
19
DSE-Example Scenario
Target
Mutant
Infect Mutant
Mutant is
alive
20
DSE-Example Scenario
Target
Mutant
Propagate
Mutant is
alive
21
DSE-Example Scenario
Target
Mutant
Mutant is
alive
22
DSE-Example Scenario
Target
Mutant
Mutant
Killed
23
DSE-Necessity condition
Example: a + b > k → a + abs(b) > k
RelationalGT(ArithmeticPlus(a, Abs(b)), k)
Abs(x) → if (x<0) //mutant necessity condition
a + b > k ≠ a + abs(b) > k //Statement level condition
Test: a=15, b=2, k=0
Conditions: b≥0 && a+b>k && …
Negates: b≥0 → b < 0
Produces: a=15, b=-10, k=0
24
DSE-Necessity condition
Example: a + b > k → a + abs(b) > k
RelationalGT(ArithmeticPlus(a, Abs(b)), k)
Abs(x) → if (x<0) //mutant necessity condition
a + b > k ≠ a + abs(b) > k //Statement level condition
Test: a=15, b=-10, k=0
Conditions: b<0 && a+b>k && a+abs(b)>k && …
Negates: a+abs(b)>k → b<0 && a+b>k && a+abs(b)≤k →
Infeasible
Negates: a+b>k → b<0 && a+b≤k
Produces: a=-15, b=-10, k=0
25
DSE-Necessity condition
Example: a + b > k → a + abs(b) > k
RelationalGT(ArithmeticPlus(a, Abs(b)), k)
Abs(x) → if (x<0) //mutant necessity condition
a + b > k ≠ a + abs(b) > k //Statement level condition
Test: a=-15, b=-10, k=0
Conditions: b<0 && a+b≤k && a+abs(b)≤k && …
Negates: a+abs(b)>k → b<0 && a+b≤k && a+abs(b)>k
Produces: a=5, b=-10, k=0
Mutant infected: Orig: 5-10>0 (false), Mut: 5+10>0 (true)
26
Killing Mutants (Search Based)
 Weak-Strong mutation
 Measure the closeness of reaching a mutant
 Measure branch distance of mutant branches
o Closeness of weakly killing the targeted mutant (mutant
necessity condition)
 Use simplified necessity fitness for improved
performance
 Sufficiency condition can be approximated by
exploring the path space or based on mutants
impact
27
Fitness function
 Approach level
 Closeness of executing a targeted mutant
 Calculated by the number of control dependent
nodes missed
 Branch Distance
 Closeness of flipping a specific branch
 Mutation Distance
 Closeness of weakly killing the targeted mutant
𝑓𝑖𝑡𝑛𝑒𝑠𝑠 = 2 ∗ 𝑎𝑝𝑝𝑟𝑜𝑎𝑐ℎ 𝑙𝑒𝑣𝑒𝑙
+ 𝑛𝑜𝑟𝑚𝑖𝑙𝑖𝑧𝑒𝑑 𝐵𝑟𝑎𝑛𝑐ℎ 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒
+ 𝑛𝑜𝑟𝑚𝑖𝑙𝑖𝑧𝑒𝑑 𝑀𝑢𝑡𝑎𝑡𝑖𝑜𝑛 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒
28
Mutant Fitness (example)
Operator Original expression
Relational
a>b
Arithmetic
a+b
Absolute
a
a && b
Logical
a || b
Mutant Fitness
a >= b: abs(a-b)
a < b: k
a <= b: 0
a != b: abs(a-b+k)
a == b: abs(a-b)
true: abs(a-b)
false: abs(a-b+k)
a - b:k
a % b:k
a * b:k
a:k
a / b:k
b:k
abs(a):abs(a+k)
-abs (a):abs(a)
0:abs(a)
a||b:min[Tfit(a)+Ffit(b), b:Ffit(a)+Tfit(b)
Ffit(a)+Tfit(b)]
true:min [Ffit(a), Ffit(b)]
a:Tfit(a)+Ffit(b)
false:Tfit(a)+Tfit(b)
a&&b:min[Tfit(a)+Ffit( b:Tfit(a)+Ffit(b)
b), Ffit(a)+Tfit(b)]
true:Ffit(a)+Ffit(b)
a:Ffit(a)+Tfit(b)
false:min[Tfit(a), Tfit(b)]
29
Path explosion problem
Mutants
 Original program
 8 program paths
Mutants
 Mutant program
 8 program paths per mutant
Mutants
 Schematic program
 Mutant evaluation decisions
 8 * 2number of mutants paths
Mutants
 Introduction of fake paths
 Same of them may be useful
for higher order mutation
30
Handling of path explosion
 Path selection strategy, (symbolic execution)
 Use weights on the Enhanced Control Flow Graph
 Shortest paths include only the original paths
 Dynamic Symbolic Execution
 Use the mutant parameters as local values
 Dynamic production of constraints eliminates the fake
paths. Paths are produced dynamically
 Search based approaches
 Approach level contains only the control depended
nodes
o Ignores the path space
31
Case Studies
 Dynamic Symbolic Execution (Strong mutation)
 Program units: Tritype, Remainder
 Siemens Programs: Schedule, Tcas, Replace
 ABS, AOR and ROR operators
 Search based (Weak mutation results)
 Hill climbing approach (AVM)
 Maximum 10 attempts per mutant
 Program units: Tritype, Triangle, Remainder,
Calendar
 ABS, AOR, ROR and LCR operators
32
DSE Study-Results
DSE Study-Results
DSE Study-Results
DSE Study-Results
DSE Study-Results
DSE Study-Results
 Strong mutation results
 Replace 86.5%, Tcas 100%, Schedule 91% (compared with
the accompanied test suite),
 Remainder 97.5 and Tritype 96.8% (all killable mutants)
½ Iterations
Program
All Iterations
Killed
No. Mut.
Killed
Solver Calls
Solver Calls
Mutants
Executions Mutants
No. Mut.
Executions
Produced
Tests
Tritype
214
1012
500
216
1588
514
90
Remainder
234
2749
435
235
5498
741
268
Replace
514
14740
19400
520
30238
80471
8927
Tcas
136
1733
3729
137
3705
4720
422
Schedule
94
714
293
94
949
303
301
Search Based Study-Results
39
Search Based Study-Results
40
Search Based Study-Results
41
Search Based Study-Results
42
Search Based Study-Results
 Weak mutation results
Test
Objects
No. of Killed Mutants
Mutation Score
Time
(Sec)
Random
Metallaxis
Random
Metallaxis
Tritype
168
245
66.4%
96.8%
42
Triangle
246
275
89.5%
100%
70
Remainder
238
242
98.3%
100%
402
Cal
237
252
93.3%
99.2%
44
43
Conclusion
 Mutation based test case generation
 Lack of attempts
 First steps using dynamic, state of the art
techniques
 Dynamic approaches can be adopted to
perform mutation

Search based approaches
o

Mutation distance leads to improved effectiveness
Dynamic Symbolic Execution
o
Quite effective
44
Conclusion
 Test data generation approaches and tools
can be effectively extended to mutation
 Practical Technique (schemata)
 Future directions
 Efficient
handling of infeasible paths
 Handling of the “flag” problem
 Efficient handling of equivalent mutants
 Hybrid approaches
45
Thank you for your attention…
Questions ?
Contact
Mike Papadakis
Nicos Malevris
mpapad@aueb.gr
ngm@aueb.gr
46
References
 Mike Papadakis and Nicos Malevris. "Automatic Mutation Test Case
Generation Via Dynamic Symbolic Execution", in 21st International
Symposium on Software Reliability Engineering (ISSRE'10), San Jose,
California, USA, November 2010.
 Mike Papadakis and Nicos Malevris. “Metallaxis an Automated Framework
for Weak Mutation", Technical Report,
http://pages.cs.aueb.gr/~mpapad/TR/MetallaxisTR.pdf.
 Mike Papadakis, Nicos Malevris and Maria Kallia. "Towards Automating
the Generation of Mutation Tests", in Proceedings of the 5th International
Workshop on Automation of Software Test (AST'10), Cape Town, South
Africa, May 2010, pp. 111-118.
 Mike Papadakis and Nicos Malevris. “An Effective Path Selection Strategy
for Mutation Testing", in Proceedings of the 16th Asia-Pacific Software
Engineering Conference (APSEC'09), Penang, Malaysia, December 2009,
pp. 422-429.
47