Using Control Flow Analysis to
Improve the Effectiveness
of Incremental Mutation Testing
IWPSE 2015
Luke Bajada, Mark Micallef, and Christian Colombo
Project
GOMTA
Software development
System
System
Mutation Testing
Mutated
System
+
*
<
>
…




/
≤
≥
Mutant
Mutation Testing
Mutant
Mutation Testing
Mutant
Mutation Testing
Mutant
Performance issues
 Every line of code can be modified in many ways!
 Each resultant mutant has to go through the test suite!
Incremental Mutation Testing
 Systems are created in increments
 Only mutate the new parts or those which changed
Increment
System
System’
Incremental Mutation Testing
 Systems are created in increments
 Only mutate the new parts or those which changed
 Identify methods which changed (syntactically)
 Identify methods which are new
 Mutate only those
Incremental Mutation Testing
 Systems are created in increments
 Only mutate the new parts or those which changed
 Identify methods which changed (syntactically)
 Identify methods which are new
 Mutate only those
 Around 10 times speedup over traditional mutation testing!
Effects of changes are not clear cut
How does a method affect others?
 Affects methods calling it
 Affects global data it modifies
 Affects data it passes on to other methods
Effects of changes are not clear cut
How does a method affect others?
 Affects methods calling it
 Affects global data it modifies
 Affects data it passes on to other methods
Algorithm to improve
Incremental Mutation Testing
1. Identify changed methods
2. Generate call graph
3. Identify methods calling changed methods (up to some
number of levels)
4. Mutate all identified methods
Algorithm to improve
Incremental Mutation Testing
1. Identify changed methods
2. Generate call graph
3. Identify methods calling changed methods (up to some
number of levels)
4. Mutate all identified methods
Algorithm to improve
Incremental Mutation Testing
1. Identify changed methods
2. Generate call graph
3. Identify methods calling changed methods (up to some
number of levels)
4. Mutate all identified methods
Algorithm to improve
Incremental Mutation Testing
1. Identify changed methods
2. Generate call graph
3. Identify methods calling changed methods (up to some
number of levels)
4. Mutate all identified methods
Algorithm to improve
Incremental Mutation Testing
1. Identify changed methods
2. Generate call graph
3. Identify methods calling changed methods (up to some
number of levels)
4. Mutate all identified methods
Expected gain over
Incremental Mutation Testing
 More mutants are generated, possibly identifying more
deficiencies in the test suite
Expected cost
Incremental
Mutation Testing
(Changed
methods)
Incremental +
Control flow
(Affected
methods)
Expected cost
Incremental
Mutation Testing
(Changed
methods)
Incremental +
Control flow
(Affected
methods)
Traditional
Mutation Testing
(All methods)
Trade off
Saved mutant generation/testing
Call graph generation/analysis
Case Studies
 Apache Commons
 Mature project (13 years old)
 5 KLOC
 97% code coverage
 Industrial system
 Still under development (1 year old)
 10 KLOC
 60% code coverage
Experiment
 Try different number of changes
 Day (worth of changes)
 Week
 Month
 Try different depth levels
 Until no more methods are identified
Gains - # of mutants
Gains - # of mutants
Shallow call graph
Gains - # of mutants
Changes spread
across call graph
Changes focused
on particular
branches of call
graph
Gains - # of mutants
Changes spread
across call graph
Also depends on how
interconnected the call
graph branches are
Changes focused
on particular
branches of call
graph
Costs – Slowdown on incremental
mutation testing
(seconds)
(seconds)
Costs – Compared to traditional
mutation testing
Costs – Compared to traditional
mutation testing
The more the
changes are
spread, the smaller
the benefit
The more focused
the changes, the
bigger the gain
Little extra cost
when going into
more depth
Costs – Compared to traditional
mutation testing
Conclusions
 More mutations are identified (over incremental)
 In general improved performance (over traditional)
 Day better than week/month
 Might not be worth the hassle if
 Changes are widespread
 Test suite does not take long to execute
vs. time to generate call graph
Limitations
We ignore these, e.g.,
 Runtime-resolved dependencies
J2EE XML configurations
Limitations
 Runtime-resolved dependencies
 Useful depth estimation
There might be no
cheaper way to do this
than is already done
Limitations
 Runtime-resolved dependencies
 Useful depth estimation
 Changes demographics
Taking into account how
the changes are spread
across the call graph it
might be possible to
estimate feasibility
Limitations
 Runtime-resolved dependencies
 Useful depth estimation
 Changes demographics
 Incremental generation of call graph
Call graph is currently
generated for each
increment
Future Work
 Address the identified limitations
 Analyse how useful the extra generated mutants are in
identifying “extra” deficiencies in test suite
 More industrial case studies
 Perform data dependency analysis
Questions?
What about the mutation score?
 If starting from mutation score of 100% (in previous
iteration) then expect issues to be focused around
changes
 I.e. Lower score over traditional
 Otherwise we cannot really say anything