Semantic Mutation Testing
John A. Clark, Haitao Dan, and Robert M. Hierons
Andreas Voraberger
13.01.2014
Andreas Voraberger
AK Softwaretechnologie 1 WS 2013/14
1
Content
• Introduction
• Traditional Mutation Testing
•
•
•
•
The idea
Mutation count
Possibilities
Advantages / Disadvantages
• Semantic Mutation Testing
•
•
•
•
•
•
•
•
•
The idea
Implementing semantic mutation
Advantages
Semantic Error Model
Common misunderstandings
Refinement
Translation
Example (Cruise-control system)
Conclusion
Andreas Voraberger
AK Softwaretechnologie 1 WS 2013/14
2
Introduction
• Testing is important but expensive
• 50% of development budget
• Test automation
• More efficient/effective
• Mutation Testing
• Distinguishing between descriptions (N)
• Semantic Mutation Testing
• Distinguishing between semantics (L)
Andreas Voraberger
AK Softwaretechnologie 1 WS 2013/14
3
Introduction
• Code defined by the pair: (N, L)
• N: the description (source code)
• L: the semantics of the language
Andreas Voraberger
AK Softwaretechnologie 1 WS 2013/14
4
Traditional Mutation Testing
Andreas Voraberger
AK Softwaretechnologie 1 WS 2013/14
5
The idea
• Mutants are classes of faults
• Produce mutants (N’)
• Making changes to the program
• Apply mutation operators
• {+, -}
• {<=, <}
• {delete parts}
• (N,L)  (N‘,L)
• Test Cases should distinguish N/N‘ (mutant killed)
Andreas Voraberger
AK Softwaretechnologie 1 WS 2013/14
6
Mutation count
• Single mutation operator creates large number of mutations
• Even in small programs
• Only use “first-order mutants“
• Produced by application of one operator
• Hypothesis: all “first-order mutants“ kill most “higher-order mutants“
• Equivalent mutant: not kill able
Andreas Voraberger
AK Softwaretechnologie 1 WS 2013/14
7
Possibilities
1. Judge test data against the mutants created
2. Produce test data to kill all mutants
• A test set that is good in distinguishing N from N’ is good at finding
faults
Andreas Voraberger
AK Softwaretechnologie 1 WS 2013/14
8
Advantages
• Target particular classes of faults
• Optimal for small slips or typos
• Simulate other test criteria
• Mutants lead to errors  provide code coverage
Andreas Voraberger
AK Softwaretechnologie 1 WS 2013/14
9
Disadvantages
• High number of mutants, even with
• Only “first-order mutants“
• Small programs
• Solution: “selective mutation“
• Equivalent mutants
• Leads to manual effort
• Increases costs
• Doesn’t aim on semantic mistakes
Andreas Voraberger
AK Softwaretechnologie 1 WS 2013/14
10
Semantic Mutation Testing (SMT)
Overview
Andreas Voraberger
AK Softwaretechnologie 1 WS 2013/14
11
The idea
• Produce mutants (L’)
• (N,L)  (N,L‘)
• Discover misunderstandings related to semantic mistakes
• Requires description language with a semantics that allows
manipulation
Andreas Voraberger
AK Softwaretechnologie 1 WS 2013/14
12
Implementing semantic mutation
• Parameterisable system
• Mutate through changing the parameters
• Express semantics that can be manipulated
• A set of rewrite rules
• Simulate a mutation of the semantics
• Changes to the syntax throughout the description
Andreas Voraberger
AK Softwaretechnologie 1 WS 2013/14
13
Advantages
• Lead to far fewer mutants
• Fewer equivalent mutants
• Tests translation, refinement, migration, porting code
• Eliminates misunderstandings
Andreas Voraberger
AK Softwaretechnologie 1 WS 2013/14
14
Semantic Mutation Testing (SMT)
Scenarios
Andreas Voraberger
AK Softwaretechnologie 1 WS 2013/14
15
Semantic Error Model
• Change of description in development process
• Abstract  Concrete
1.
2.
3.
4.
5.
Requirements
Specification
Design
Code
Machine code
• Misunderstandings in every transformation
Andreas Voraberger
AK Softwaretechnologie 1 WS 2013/14
16
Semantic Error Model
Andreas Voraberger
AK Softwaretechnologie 1 WS 2013/14
17
Semantic Error Model
• Refinement
• Transformation to different abstraction level
• Translation
• Transformation on the same abstraction level
 Basis for misunderstandings
 Solution: SMT
Andreas Voraberger
AK Softwaretechnologie 1 WS 2013/14
18
Common misunderstandings
• Find set of semantic misunderstandings
• For different languages
• Based on studies or experience
• Implement tests, finding them
• Set of operators should reflect the environment
• Different mistakes depending on programmer skills
Andreas Voraberger
AK Softwaretechnologie 1 WS 2013/14
19
Refinement
• Semantic mistakes between abstraction levels
• Problems on copying syntax
• Maybe different semantics
• Examples
•
•
•
•
•
Truncation rules
Precedence rules
Datatypes
Floating point precision
Binary representation
Andreas Voraberger
AK Softwaretechnologie 1 WS 2013/14
20
Migration / Porting (Translation)
• Migration / Porting to a different language
• Maybe different semantics
• Use a semantic test case generation tool
• Suits of semantic mutation operators for common combinations
• Point to common failures
• Example: C to Z
• Division of negative numbers
• Branching structures
• floating-point comparison
Andreas Voraberger
AK Softwaretechnologie 1 WS 2013/14
21
Semantic Mutation Testing (SMT)
Example
Andreas Voraberger
AK Softwaretechnologie 1 WS 2013/14
22
Example
• Shows
• cruise-control system
• Not only slips have to be tested
• Developers can make semantic mistakes
Andreas Voraberger
AK Softwaretechnologie 1 WS 2013/14
23
Statechart for a cruise-control system
Andreas Voraberger
AK Softwaretechnologie 1 WS 2013/14
24
Cruise-control system (Case 1)
• State: no_vehicle_in_front
• Events:
• brake
• level=increase
• What happens?
Andreas Voraberger
AK Softwaretechnologie 1 WS 2013/14
25
Cruise-control system (Case 2)
• State: no_vehicle_in_front
• Events:
• Vehicle detected
• level=increase
• What happens?
Andreas Voraberger
AK Softwaretechnologie 1 WS 2013/14
26
Conclusion
• Semantic Testing is necessary
• SMT is a working practice
• Captures a different failure type than MT
• Focuses on misunderstandings
• Normal MT still necessary
Andreas Voraberger
AK Softwaretechnologie 1 WS 2013/14
27
Quelle
• John A. Clark, Haitao Dan, and Robert M. Hierons. Semantic mutation
testing. In Third International Conference on Software Testing,
Verification, and Validation Workshops, pages 100-109. IEEE, 2010
Andreas Voraberger
AK Softwaretechnologie 1 WS 2013/14
28