International Journal of Engineering Trends and Technology (IJETT) – Volume 21 Number 10 – March 2015
A Survey and Analysis of Mutation Testing
Bharti kumari1, Ms. Susheela hooda2
1
2
M.Tech Student, Department of CSE, B.S.Anangpuria Institute of Technology and Management, Alampur, India
Assistant professor, Department of CSE, B.S.Anangpuria Institute of Technology and Management, Alampur, India
Abstract - The purpose of software testing is to verify
and validate the software application or program so that the user
& project can achieve the expected requirement. In Mutation
Testing, we change certain statements of the source code and
check if the test cases are able to find the errors. Although
mutation testing is a high effective technique for evaluating the
quality of the test data but it has also some problems. in this
paper, we study about the mutation testing , the problems arises
during this testing and the techniques to solve these problems.
III. Undetected mutants: In some cases, mutants that is
functionally equivalent, although syntactically different to the
original program always produces the same output & thus no
test cases will be able to kill are called undetected or live
mutants.
Keywords- Mutation testing, mutants, cost reduction
techniques.
I.
INTRODUCTION
Mutation testing was proposed in 1970 & its tool was
implemented by Timothy Budd in 1980 in his research
work[1]. According to Budd ―Mutation testing is a fault based
testing technique in which we seek the errors in the program
and find the errors‖.
Mutation testing is one of the software testing technique
which is proposed by Hamlet[2].It is a fault based testing
technique [3,4,5].Mutations are based on operators are called
mutant operators[6,7].
In Fig. A Mutants are generated by making changes in to the
original program & many test cases are generate.
We produce mutants by introducing changes in to the program
and test cases are used to distinguish the original program
from its mutants. the process begins by systematically seeding
faults into the program. these faults can be some predefined
operators which is called mutation operators. eg. Replacing +
with -, or replacing > with >=.
The faulty version of program which is called mutant will be
slightly differs from the original program. The purpose is to
run test cases generated on the mutants to see if any test case
distinguishes the faults introduces earlier into the code.
Fig. A. Mutation testing process.
Mutation score :
Where total number of mutants = number of killed mutants +
number of live mutants.
I.
Killed mutants: If mutant produce different output
than original program will be killed by test cases.
The value of mutation score always lies between 0 and 1. The
highest value of mutation score indicates the effectiveness of
test suites.
II.
Equivalent mutants: If mutant produce same output
than original program will be classified as either equivalent or
live mutants.
Effectiveness of test cases : It can be calculated by multiply
the mutation score (M) and the ratio of average of number of
ISSN: 2231-5381
http://www.ijettjournal.org
Page 179
International Journal of Engineering Trends and Technology (IJETT) – Volume 21 Number 10 – March 2015
test cases that kill mutants (K) divided by the total no of test
c)
cases (T).
Running test :- we have to run all the test cases of a program
on each live mutant [4][10]. This is very expensive.
1.2 Human effort :
a)
Average number of test cases: Average number of test cases
that killed mutants divided by number of dead mutants.
Human oracle problem: In human oracle problem , output of
original program will be checked with each test case . It s not
the problem unique to mutation testing. In all form of testing,
once a set of input has been arrised at, there remains a
problem of checking output . But mutation testing is so
demanding & this can lead to an increase in the number of test
cases that is why increase the oracle cost. This oracle cost is
most expensive part of overall testing .
III. LITERATURE WORK :
Mutation technique is computationally expensive techniques.
for the purpose of of making this technique a practical testing
technique, many cost reduction techniques has been proposed.
II.
PROBLEMS OF MUTATION TESTING :
1.1 High computational cost : Mutation testing is very
computationally expensive .This hinders mutation testing
from being used on everyday basis practical situations [9] [10]
.
Computational complexity of mutation testing is O(n2).
Where n=number of operations[8].
The key bottlenecks of mutation testing are large amount of
generated mutants, equivalent mutant problem and running
test, as discussed below a) Large amount of generated mutants:- A program can
have a fault in many possible places & with only one inserted
semantic fault we will have one mutant . thus, a large number
of mutants will be generated in the mutant generation phase of
mutation testing.
e.g , a simple program with a simple sentence K*L can be
mutated in many difference ways : K-L , K+L , K/L , K+L++ ,
-K+L , K+0 , 0+L etc. This problems leads to a very high
execution cost.
According to Offutt and untch[13] .cost reduction techniques
are divided into three types that are ‗do fewer‘,‘do
faster‘,and‘do smarter‘. In this paper , we divides these
techniques in to two types :
Mutant reduction techniques : Mutant reduction technique
reduces the number of mutants, by which computational cost
also reduces . we reduces the number of mutants in such a
manner that there is no significant lose of test effectiveness.
For example :
M= given set of mutants.
T= a set of test data.
MST(M)= mutation score of test set T applied to mutants M.
In mutant reduction problem we find the subset of mutants
Mʹ from M. where MST(Mʹ)≈MST(M). In this paper , we
discuss three mutant reduction techniques A. Mutant sampling :
b) Equivalent mutant problem :- Many operators can
produce equivalent
mutants which behaves same as the
original program. Equivalent
mutants can not be killed by the test cases & detection of these
type of mutants also involves the addition human effort [11].
Furthermore , the generation & execution of tests on
equivalent . Mutants are waste of computational time [12] .
ISSN: 2231-5381
Acree[14] and Budd[15] was first proposed mutant sampling
approach in 1980. It is a simple approach that randomly
choose a small number of subset of mutants (X%) from the
entire set of mutants. In Wong & Mathur[16], an experiment
using a random selection rate X% from 10% to 40% in steps
of 5%. The results suggested that random selection of 10% of
mutants is only 16% less effective than a full set of mutants.
This study implies that mutant sampling is valid with a X%
value higher than 10%.
http://www.ijettjournal.org
Page 180
International Journal of Engineering Trends and Technology (IJETT) – Volume 21 Number 10 – March 2015
B. Mutant clustering:
In Hussain‘s master thesis [17] mutants are no selected
randomly but I this subset of mutants are selected using
clustering algorithm. Mutant clustering generate all first order
e)
mutants. A clustering algorithm is then applied to classify the
f)
first order mutants in to different clusters based on the killable
g)
test cases. Each mutant in the same cluster is killed by similar
set of test cases. only a small number of mutants are selected
from each cluster to be used in mutation testing & remaining
are discarded.
a) C. Selective mutation :
b)
c) In selective mutation all set of mutation operator are
find that generate a subset of all possible mutants. Mathur
analysed the 22 mutation operators and observed that just a
few operators result in the generation of large mutants. In [18]
suggested the exclusion of the two most prodigious operators :
svr & asr. This idea was implemented as ―2 selective
mutation‖.Offutt el. al[19] have also extended Mathur &
Wong‘s work by omitting four mutation operators ―4 selective
mutation‖ & omitting six mutation operators ―6 selctive
mutation‖.
I.
Execution cost reduction techniques: The
computational cost can also be reduced by reduce the
execution cost. In this selection we introduce two techniques
to optimize the execution process a)
b) D. Strong, weak & firm mutation :
c)
d) DeMillo et al.[20] proposed strong mutation . strong
mutation is referred as traditional mutation testing. In this, a
program P, a mutant m of program P will be killed only if
mutant m gives the difference output from the original
program P
Hargon & Mathur[21] gives a theoretical proof of weak
mutation. By which it shows that under certain conditions test
set generated by weak mutation can also be expected to be as
effective as strong mutation.
In weak mutation, instead of checking mutants after the
execution of the entire program , the mutants need only to be
checked immediately after the execution point of mutant or
mutated components. The advantage of weak mutation is that
each mutants does not require a complete execution process,
once the mutated component is executed we can check for
survival.
ISSN: 2231-5381
Woodward & Halewood[22] proposed firm mutation in 1988.
In firm mutation, disadvantage of strong and weak mutation is
removed. It lies between the weak mutation & the strong
mutation.
E. Run-time optimization technique :
The interpreter based technique is one of the optimization
technique used in first generation of mutation testing
tools[23]. ie the result of a mutant is interpreted from its
source code directly. mutant optimization is sufficient and
efficient for small mutant program.
To reduce the cost of interpretation, compiler-based technique
were
subsequently
introduced[24].in
compiler-based
technique, the mutant program is compiled in to an executable
program, then each compiled mutant is executed by a number
of test cases. the limitation of this technique is extra overhead
cost.
The mutant schemata approach is designed to reduce the
overhead cost of traditional compiler-based [25].instead of
compiling each mutant separately, the mutant schemata
technique generates a meta program. This meta program is
need to compile once time to test each mutant. So the cost is
calculated once time compilation technique. The new
approach is introduced as a byte code translation technique
which is proposed by the Ma et. Al.[26].
IV.
CONCLUSION & FUTURE WORK :
In this paper, we introduce about the mutation testing, the
process of mutation testing to find the live mutants and killed
mutants .in this paper we also calculate the mutation score ,
effectiveness of test cases and average number of test cases.
Mutation testing is computationally expensive technique. This
paper introduce two main problems of mutation testing that is
high computational cost and human effort . we also discuss
techniques to solve these problems. Our future work is to
overcome the problems arises during mutation testing .
V.
REFERENCES
[1]. E.J Weyuker and T.J Ostrand, ―Theories of Program
Testing and the Application of Revealing Subdomains,‖ IEEE
Transaction Software Engineering., vol. SE-6, 1980.
[2]. R.G Hamlet, (1977), ―Testing programs with the AID of a
Compiler‖, IEEE Transactions on Software engineering, 1977.
http://www.ijettjournal.org
Page 181
International Journal of Engineering Trends and Technology (IJETT) – Volume 21 Number 10 – March 2015
[3]. R. DeMillo, R. Lipton and F Sayward,(1978), ―Hints on
Test Data Selection: Help for the Practicing Programmer,‖
Computer, 11(4): 34-41: April, 1978.
[4]. Y. Jia and M. Harman, ―An Analysis and Survey of the
Development of Mutation Testing‖, CREST Center, King‘s
College, London, Tech. Rep. TR-09-06, 2009.
[5]. A.J. Offut., ―Investigations of the Software Testing
Coupling Effect‖, ACM Transactions on Software engineering
Methodology 1(1):3-18 January 1992.
[6]. Antonia Estero-Botaro Palomo-Lozano and Inmaculada
Medina Bulo, ―Quantitative Evaluation of Mutation Operators
for WS-BPEL Compositions‖ Department of Computer
Languages and Systems, University of C? adiz, Spain.
[7]. M.Woodward,― Errors in Algebaric Specification and an
Experimental Mutaion Testing Tool‖ Software Engineering
Journal, pages 211-224, July 1993.
[8] Mathur A. P. and Wong W E, ―An Empirical Comparison
of Mutation and Data Flow Based Test Adequacy Criteria‖,
1993 Purdue University, West Lafayette, Indiana, Technique
Report.
[9] Offutt A. J, Rothermel G, Zaph C, ―An Experimental
Evaluation of Selective Mutation‖,‘ ICSE '93 Proceedings of
the
15th
international
conference
on
Software
Engineering‘,1993, California, USA
[10] Jefferson A, Offutt A J, and Untch R J, ―Mutation 2000:
Uniting the Orthogonal‖, ‗Mutation Testing for the new
century‘, 2001, Massachusetts, USA
[11]L. Madeyski, W. Orzeszyna, R. Torkar, and M. Józala:
Overcoming the Equivalent Mutant Problem: A Systematic
Literature Review and a Comparative Experiment of Second
Order Mutation. IEEE Transactions on Software Engineering,
accepted in 2013. http://dx.doi.org/10.1109/TSE.2013.44
[12] Grun B, Schuler D and Zeller A, ―The Impact of
Equivalent Mutants‖, ‗ICSTW '09 Proceedings of the IEEE
International Conference on Software Testing, Verification,
and Validation Workshops‘, 2009, California, USA
[13] A. J. Offutt and R. H. Untch, ―Mutation 2000: Uniting
the Orthogonal,‖in Proceedings of the 1st Workshop on
Mutation Analysis (MUTATION’00), published in book form,
as Mutation Testing for the New Century. San Jose, California,
6-7 October 2001, pp. 34–44.
[14] A. T. Acree, ―On Mutation,‖ PhD Thesis, Georgia
Institute of Technology,Atlanta, Georgia, 1980.
ISSN: 2231-5381
[15] T. A. Budd, ―Mutation Analysis of Program Test Data,‖
PhD Thesis,Yale University, New Haven, Connecticut, 1980.
[16] A. P. Mathur and W. E. Wong, ―An Empirical
Comparison of Mutation and Data Flow Based Test Adequacy
Criteria,‖ Purdue University, West Lafayette, Indiana,
Technique Report, 1993.
[17] S. Hussain, ―Mutation Clustering,‖ Masters Thesis,
King‘s College London, UK, 2008.
[18] A. P. Mathur, ―Performance, Effectiveness, and
Reliability Issues in Software Testing,‖ in Proceedings of the
5th International Computer Software and Applications
Conference (COMPSAC’79), Tokyo, Japan, 11-13 September
1991, pp. 604–605.
[19] A. J. Offutt, G. Rothermel, and C. Zapf, ―An
Experimental Evaluation of Selective Mutation,‖ in
Proceedings of the 15th International Conference on Software
Engineering (ICSE’93). Baltimore, Maryland: IEEE
Computer Society Press, May 1993, pp. 100–107.
[20]. R. DeMillo, R. Lipton and F Sayward,(1978), ―Hints on
Test Data Selection: Help for the Practicing Programmer,‖
Computer, 11(4): 34-41: April, 1978.
[21] J. R. Horgan and A. P. Mathur, ―Weak Mutation is
Probably Strong Mutation,‖ Purdue University, West
Lafayette, Indiana, Technical Report SERC-TR-83-P, 1990.
[22]. M.R Woodward and K. Halewood, (1988), ―From Weak
to Strong, Dead or Alive? An analysis of some mutation
testing issues‖, Workshop on Software Testing, Verification
and Analysis, pages 152-158, Banff Alberta, July 1988. IEEE
Computer Society Press.
(23)K. N. King and A. J. Offutt. A Fortran language system
for mutation-based software testing. Software—Practice and
Experience, 21(7):685–718, July 1991
(24) M. E. Delamaro and J. C. Maldonado. Proteum - A tool
for the assessment of test adequacy for C programs: User‘s
guide. Technical report, Jan. 20 1996
(25)R. H. Untch, A. J. Offutt, and M. J. Harrold. Mutation
analysis using mutant schemata. In International Symposium
on Software Testing and Analysis, pages 139–148, 1993.
(26)Y.S. Ma., A.J. Offutt and Y.R. kwon, ―MuJava: An
automated Class
Mutation System‖ Software Testing, Verification
and
Reliability, vol. 15, no. 2, pp. 97-133, June 2005.
http://www.ijettjournal.org
Page 182