An Analysis of OO Mutation Operators
Jingyu Hu, Nan Li, and Jeff Offutt
Presented by
Nan Li
03/24/2011

4/13/2020
Overview
• Motivation
• Experiment
• Equivalence Conditions
• Results
• Analysis
• Findings and Recommendations
• Threats to Validity
• Conclusions
© Nan Li, 2011 2

OO Mutation Operators
• Modify OO language features
– Inheritance
– Polymorphism
– Dynamic binding
– Access control
– Type conversion
• Developed around
10 years ago
• Implemented in several mutation tools
– 28 OO mutation operators in muJava
Are these the right operators?
© Nan Li, 2011 4/13/2020 3

Our Empirical Questions
1. Can we use fewer mutation operators?
1.1. Do some operators create mutants that are really easy to kill ?
1.2. Can we eliminate some overlapping operators?
2. Can we create a “ selective set
” of mutation operators?
3. Can we avoid creating some equivalent mutants?
4/13/2020 © Nan Li, 2011 4

Empirical Study Summary
• Choose classes to study (subjects)
• Generate all
OO mutants
- muJava
• Document numbers of mutants of each type
• Create tests to kill as many mutants as possible
• Determine equivalent mutants by hand
• Run all tests on all mutants
• Measure “ strength
” of mutation operators
4/13/2020 © Nan Li, 2011 5

Previous Similar Study
Number of classes
Ma, Harrold, and Kwon
ICSE 2006
11
Number of mutants 679
Our paper
38
3965
Number of mutation operators
Goal
19 28
Can tests that kill all traditional mutants also kill
OO mutants?
How hard is it to kill mutants created from each operator?
4/13/2020 © Nan Li, 2011 6

Experimental Subjects
• 38 classes from 5 programs
• Two programs from Deitel’s Java textbook
• One classroom assignment
• One program from org.joda.time
• One program from javassist
4/13/2020 © Nan Li, 2011 7

Statistics About Classes
100
90
80
70
60
10%
> 500
26%
201-
500
50
40
30
20
32%
101-
200
10
0
32%
< 100
# Lines
8%
3
29%
2
42%
1
5%
> 20
21%
11-
20
29%
2-10
3%
> 100
18%
51-
100
47%
11-
50
21% 45%
0
Inheritance
Depth
1
#
Constructors
32%
1-10
#
Declared
Methods
10%
> 50
16%
> 10
13%
> 1
21%
> 10
34%
21-
50
32%
6-10
29%
6-10
61%
2-5
18%
1-5
32%
1-20
32%
2-5
24%
0
#
Inherited
Methods
21%
1
#
Declared
Variables
87%
0
#
Inherited
Variables
29%
1
32% 10%
0
#
Overloading
Methods
0
# Overriding
Methods
4/13/2020 © Nan Li, 2011 8

Equivalence Conditions
• Detection
: Recognizing that a mutant is equivalent after it is created
– Undecidable problem
– Several useful heuristics
• Avoidance
: Do not create a mutant that will be equivalent
4/13/2020
An equivalence condition gives a rule that can be used to avoid creating a mutant that cannot be killed
© Nan Li, 2011 9

Equivalence Condition Summary
• We determined equivalent mutants by hand
• 3965 mutants
• 492 equivalent mutants ( 12.4% )
• Designed new equivalence conditions for 5 operators
• If included in muJava, these equivalence conditions would have avoided 269 equivalent mutants
– Leaving only 223 ( 5.6% )
• Example
– JSD (Static modifier deletion) : deletes the static keyword, thus instead of one copy of a variable being shared among all objects, each object gets a unique copy
– Equivalence condition
: If a primitive variable is also defined as final, the variable cannot be changed, thus whether it is static does not matter and mutants cannot be killed
4/13/2020 © Nan Li, 2011 10

Killing Mutants
• Procedure
:
– Create tests to kill all mutants in each class
– Eliminate duplicate (ineffective) tests
– Run each test against each mutant
• 575 tests
• 3398 mutants are killed
• 492 equivalent mutants
• 75 mutants were not killed
• Overall mutation score is
0.98
4/13/2020 © Nan Li, 2011 11

Threats to Validity
• External threat
: subjects may not be representative
• Internal threats
:
– Only one test suite was generated for each class
– Equivalent mutants were determined by hand
– Possible flaws in muJava
– Order in which we generated tests for killing mutants
4/13/2020 © Nan Li, 2011 12

4/13/2020
1500
1400
1200
1000
800
600
400
200
0
0
%
Number of Tests That Killed Each
Mutant
1
%
-
5
%
6
%
-
1
0
%
1
1
%
-
1
5
%
6
1
%
-
2
0
%
1
2
%
-
3
0
%
1
3
%
-
4
0
%
1
4
%
-
5
0
%
1
5
%
-
6
0
%
Percentage of tests
1
6
%
-
7
0
%
1
7
%
-
8
0
%
1
8
%
-
9
0
%
1
9
%
-
1
0
0
%
© Nan Li, 2011 13

Mutation Operator Strength
• An operator is strong if mutants generated for that operator are killed by relatively few tests
• Mutation Operator Strength (
MOS ):
#minTests / (#muts - #notKilled)
– #minTest : minimal number of tests needed to kill all mutants of that type in that class
–
#muts : number of mutants that are created from that operator
–
#notKilled : number of equivalent mutants and number of mutants our tests were not able to kill
4/13/2020 © Nan Li, 2011 14

Strength of Mutation Operators
1
.8
.6
.4
.2
0
EAM EMM EOA JDC JID JSD JSI JTD JTI IHD IHI IOD IOP IOR IPC
Mutation Operators
ISD ISI OAC OMD OMR PCC PCD PCI PMD PNC PRV
4/13/2020 © Nan Li, 2011 15

Findings and Recommendations
• OAC (Arguments of Overloading Method Call Change) operator only has strength 0.011
– 934
OAC mutants – 23.6% of all mutants
• PCI (Type Cast Operator Insertion) operator only has strength 0.079
– 1282
PCI mutants – 32.3% of all mutants
• Test that killed all EAM (Accessor Method Change) mutants also killed most EMM (Modifier Method Change) mutants
– Conversely, tests that killed all
EMM mutants killed most EAM mutants
Recommendations
(1) Eliminate OAC and PCI
(2) Use EAM or EMM but not both
4/13/2020 © Nan Li, 2011 16

Findings and Recommendations
• 104 class-level mutants per class
• 12.4% of the mutants are equivalent
– would be
5.6% if the new equivalence conditions are used
• Most statement-level mutants are killed by far more tests than most class-level mutants
We cannot find a selective set of
OO mutation operators
© Nan Li, 2011 4/13/2020 17

Empirical Questions & Answers
1. Can we use fewer mutation operators?
1.1. Do some operators create mutants that are really easy to kill ?
1.2. Can we eliminate some overlapping operators?
2. Can we create a “ selective set
” of mutation operators?
Yes
Yes
Yes
No
3. Can we avoid creating some equivalent mutants?
Yes
4/13/2020 © Nan Li, 2011 18

Conclusions
• Most comprehensive study of killing class-level mutants
• How many equivalent class-level mutants can be expected
• Detailed data on killing mutants
– 3398 mutants were killed by 575 tests
• Equivalence conditions for EAM, JSD, JSI (Static Modifier
Insertion), PCI, and PPD (Parameter Variable Declaration with
Parent Class Type)
– Equivalence conditions would eliminate 269 equivalent mutants
– Bring equivalent mutants down from 12.4% to 5.6%
• Ran every test against every mutant to measure how hard it is to kill individual mutants, and mutation operators
• Recommend eliminating the mutation operators OAC, PCI
• Recommend using
EAM xor EMM
4/13/2020 © Nan Li, 2011 19

4/13/2020
Contact
Nan Li
Email: nli1@gmu.edu
Homepage: http://cs.gmu.edu/~nli1/
MuJava: http://cs.gmu.edu/~offutt/mujava/
© Nan Li, 2011 20