David Pryor
Mutation-Based Testing
 Same basic goal as Code Coverage
 Evaluate the tests
 Determine “how much” code exercised
 Mutation testing goes beyond checking which lines of code
were executed
 Goal: Distinguish statements that are simply executed from
those that are fully tested
 History
 Theory published in 1971 by a student

Computationally Infeasible
 Technological advances: 90’s – present
 Still not mainstream, but gaining popularity
2
Basic Procedure
 Requirements
 Complete working program
 Tests written and passing
 Make a small change(mutation)
to the source code of the program
 Run the tests
 If a test fails:
 The test “killed” this mutant
 If all tests still pass:
 Redundant/Unnecessary code
 Tests are incomplete
3
Example test functions
 This test achieves 100% code coverage
 testAbs1 will never fail
 Even if abs(-3) == -3
 Mutation testing can detect this
 testAbs1 will initially pass
 It will still pass on mutated code
 It failed to “kill” the mutant
 The test is inadequate
4
Example test functions
 testAbs2 passes initially
 On the mutated function, it fails
 abs(-5) != 5
 Test is supposed to fail
 It shows the test is robust enough to
catch errors such as this mutation
 It killed the mutant
5
Mutation Operators
 Types of mutations to be applied
 Types of “changes” to make to the code
 Defined by the testing framework
 Chosen by the user
 Goals
 Introduce errors
 Simulate common coding bugs
 Ensure the testing of all possible circumstances
 Traditional Mutation Operators
 Simple
 Common
 Included in most mutation frameworks
6
Dropped Statement
 Removes a single statement from
the program
 Unnecessary code
 Needs to be selective
 Many possible mutants
7
Arithmetic Operator Replacement
 Swaps arithmetic operators
 +, -, *, /, %
 Some frameworks have set
translations
 + always becomes *
 Others allow any/random
swaps
8
Boolean Relation Replacement
 Swaps boolean relations
 ==, !=, <, <=, >, >=
 Tightly constrained form
 Only mutates to/from similar relations
 < to <=
 > to >=
 == to !=
9
Boolean Expression Replacement
 Replaces an entire expression with
true or false
 Unnecessary Code
 Code paths that aren’t sufficiently
tested
10
Variable Replacement
 Replaces a variable reference with
a different variable reference
 The new variable must be
accessible and defined in the
same scope
 Not trying to create compiler
errors
 False positives
 Unnecessary/Duplicate variables
11
Non-Traditional Operators
 Lots of Operators out there
 Bit operation / Binary operators
 Shift/Rotate replacement
 Increments / Decrements
 Invert Negatives
 Should be able to define your own for customized
testing
 Ideally:
 Minimize false positives
 Reasonable number of created mutants
12
Replace inline constants
 Replaces an inline
constant
 Numeric or string
 Non-Deterministic
 Tests for responses
more than behavior
 Requires many tests
that may not be
helpful
 Used in Heckle(Ruby)
13
Object-Oriented Operators
 Encapsulation
 Changing access modifiers
 Inheritance
 Hiding variables
 Method overriding
 Parent / super actions
 Polymorphism
 Lots of operators here
 Basic Idea: change something between the parent and
the child in the usage of an object
14
Concurrency Operators
 Modify sleep/wait calls
 Mutual exclusion and semaphores
 Change boundaries of the critical section
 Change synchronization details
 Switch concurrent objects
 Others
 Goal is the same
 Evaluate the adequacy of the tests
15
Computation Time
 The biggest problem/roadblock with Mutation testing
 Theory has existed for 40+ years
 Computationally infeasible for use in industry for a long
time
 Frameworks can automate almost everything, but:
 Every mutant has to be run against the entire test suite
 Many mutants and a large test suite cause immense
testing times
16
Computation Time Estimation
 T = Time to run test suite against the code base
 M = # of Mutation Operators in use (3-20+)
 N = # of Mutants per Operator (depends on code base size)
 Mutation testing causes time to increase from T to T*M*N
 Minimum time increase of a factor of 30
 For small code base and few operators
 Total time increases very quickly
 Only time needed to run tests, not compilation
 If T = 1 minute, T*M*N can become hours or days
 If T = 1 hour, T*M*N can become weeks or longer
17
Addressing Computation Time
 Need to spend less time on mutation testing
 Variety of methods that fall into three categories
 Do Less
 Test fewer mutants and mutation operators
 Need to be careful

Fewer may result in poor tests slipping through
 Do Faster
 Increase the speed of execution
 Do Smarter
 Eliminate mutants and mutation operators that do not
provide meaningful results
18
Source code or Byte code?
 Can perform mutations on the source code itself, but:
 Large code bases result in lots of slow disk reads
 Have to compile EVERY mutant
 Instead, compile the original source once
 Mutate the compiled byte code
 Much faster
 Can be difficult to back-trace the byte code to the
source code to show the mutants that were created
19
Weak vs. Strong Testing
 Two conditions for killing a mutant
 Testing data should cause a different
program state for the mutant than for
the original

For example, a test results in:
 valid = false
 done = true
 The difference in state should result
in a difference in output and be
checked by the test
In this case: the test should check ‘result’
 Weak Testing: Only satisfy the first condition
 Strong Testing: Both conditions

20
Weak vs. Strong Testing
 Weak assures that the tests cause a difference
 Not assured that they check the difference
 Not as thorough
 Strong is ideal
 Computationally expensive
 Must always run to the end
 Weak can stop as soon as it detects a difference in state
21
Incremental Analysis
 Currently experimental
 Most useful for long-term projects with large code
bases, which use mutation testing over and over
 Basic Idea: save state and results of tests and code
 Only re-run those tests and mutants for which relevant
code has changed
 Decide which to skip based on changes made

Not perfected yet – can be “tricked” by odd behavior
22
Selective Mutation
 Goal: Eliminate some mutants or operators that are not necessary
 Mutants
 Remove duplicates caused by multiple operators
 Remove “likely” duplicates – those that will probably cause
duplicate results

Some amount of error here
 Mutation Operators
 Some pairs of operators might produce many of the same
mutants
 An operator might produce a subset of mutants from a
different operator
 Detection can be difficult
23
Coverage Based Test Selection
 Typical mutant only affects a single statement / line
 Typical test suite has many tests that do not execute
this line
 No need to run these tests on the mutant
 Only run tests that exercise the mutated code
 Optimize the running order of tests
24
Other Problems and Design Considerations
Equivalent Mutants
 This mutant is functionally equivalent
to the original
 No test that calls this code could ever
distinguish the two
 “Some mutants can’t be killed”
 Can sometimes be detected
automatically and filtered out
 Not all can be detected
 Requires human effort to determine if
equivalent
 Not always an easy task
25
Other Problems and Design Considerations
Mutant Infinite Loops
 Some mutations can cause infinite
loops in the mutants
 Statement deletion removed the only
way out of this loops
 Solution: Time the un-mutated code
 If the mutant takes significantly longer
than this time


Probably an infinite loop
Timeout after the un-mutated time, plus
some padding
26
Other Problems and Design Considerations
Complex Bugs
 Mutation only makes small changes
 What if the test cases miss a large, complex error?
 Mutation doesn’t create complex mutants
 Coupling Effect
 Hypothesis – Tests that detect simple errors are sensitive
enough to detect more complex errors
 Supported by empirical data
 Testing for “simple” errors helps to find the more
complex ones
27
Fuzz Testing / Fuzzing
 Completely unrelated to Mutation testing
 Often confused with Mutation
 Involves generating random data to use as input to a
program
 Test security/vulnerability
 See if the program crashes
 Fuzzing – modifies input
 Checks program behavior
 Mutation – modifies source code
 Checks test case results
 Deterministic
28
Tools and Environments
 Java
 MuJava
 Bacterio
 Javalanche
 Jumble
 PIT
 Jester
 C/C++
 Insure++
 Fortran
 Mothra
 PHP
 Mutagenesis
 Ruby
 Heckle
 Mutant
 C#
 Nester
29
Using Mutation Testing in Industry
 Use Mutation from the beginning of a project
 Don’t use it with “dangerous” methods
 Look for high quality tools/environments
 Speed optimizations
 Reporting/Coverage information
 Configurable Operators
30
Benefits of Mutation testing
 Evaluation of tests / test suite
 More than code coverage: are the tests adequate?
 Mutation Score: % of non-equivalent mutants killed
 Evaluation of code
 Find unreachable or redundant code
 Find bugs that were hidden through inadequate tests
 Future: Automatic Test Generation
 Create dummy tests, use Mutation to revise
 Repeat until all or most non-equivalent mutants killed
 Still experimental, but promising
31
Questions
32
References
 Alexander, R. T., & Bieman, J. M. (2002). Mutation of java objects. IEEE Int. Symp.




Software Reliability Engineering, Retrieved from
http://www.cs.colostate.edu/~bieman/Pubs/AlexanderBiemanGhoshJiISSRE02.pdf
Offutt, A. J. (n.d.). A practical system for mutation testing: Help for the common
programmer. Retrieved from http://cs.gmu.edu/~offutt/rsrch/papers/practical.pdf
Offutt, A. J., & Untch, R. H. (n.d.). Mutation 2000: Uniting the orthogonal.
Retrieved from http://cs.gmu.edu/~offutt/rsrch/papers/mut00.pdf
Ma, Y. S., Kwon, Y. R., & Offutt, A. J. (n.d.). Mujava: An automated class mutation
system. Retrieved from http://www.cs.gmu.edu/~offutt/rsrch/papers/mujava.pdf
Bradbury, J. S., Cordy, J. R., & Dingel, J. (n.d.). Mutation operators for concurrent
java(j2se 5.0). Retrieved from
http://www.irisa.fr/manifestations/2006/Mutation2006/papers/14_Final_version.pdf
 Pit mutation testing. (n.d.). Retrieved from http://pitest.org/
33