Feature
Mutate
40
my software
Computer systems run the world and are found in fridges to hospitals. Every
application needs testing, which is expensive and time-consuming. Dr Mark
Micallef and Dr Christian Colombo from the PEST research group (Faculty of
ICT, University of Malta) tells THINK about a new technique which could make
testing easier and more consistent. Illustrations by NO MAD.
Internet banking applications or
focus for us is known as mutation testing.
In order to understand the technique, it helps
play games on a tablet? Then you
to imagine passengers going through an airport
are a software system user. Several
security scan. Airports are equipped with security
companies are trying to compete
scanners, a doorway-like device that scans for
for your attention to buy or use their software.
forbidden items such as weapons or dangerous
To convince us, they need to supply a high
chemicals. Now imagine that you are Head of
quality product or we will simply download their
Security at an airport and have just bought a
competitor’s application.
scanner. You naturally want to make sure it works
Testing is essential to ensure quality. This
properly. How would you do this? Probably by
process is very challenging, time consuming,
walking through the scanner carrying various
expensive, and provides inconsistent results
hidden items. If the machine does not raise the
depending on circumstances such as who tests
alarm, you would be on a hotline to the supplier.
the product or how much time is allocated to it.
Software is tested in a similar way. Software
Our research team (the Processing Engineering,
systems are usually developed stepwise.
Security and Testing [PEST] research group) is
Developers release a version of an app or program
trying to solve these problems by making it easier
to customers, who start using it, whilst the
and quicker to judge whether or not testing is being
developers continue to develop the next system
done correctly within an organisation. One area of
version. These arrive on your device as updates.
Feature
D
o you browse the Internet, use
41
42
Feature
The problem with updates is that
precise differences between
they can break a system. To prevent
the system on day one and
this issue, software engineers develop
two. Our system then sees
automated test suites, which are
the key parts of the
computer programs that scan software
system that need
and find hidden problems—like an
mutation
airport scanner would. Instead of
testing
scanning passengers for forbidden
and only
items, a test suite scans a software
analyses
system for faults. When a developer
those.
makes a change to a system, she tests
When we
it using the automated test suite and
is deemed good enough to release to
customers. Theoretically, this process
works. In practice, test suites are not
perfect and tend to miss errors.
To make better test suites our PEST
research group has turned towards
mutation testing. This technique tests
The idea
behind
our solution is
deceivingly simple.
the test suite. Going back to the airport
it reduced testing
time by 88% and 91%.
Achieving this goal took
us three years and the
technique’s development
was challenging. We
needed to mathematically prove
that checking the test suite at every
stage of development was at least
scanner scenario, much like we asked a
system version every two weeks.
as good as testing it at the very end.
passenger to walk through the scanner
Because it is so time consuming it has
We also needed a quick and accurate
multiple times, each time concealing a
never been adopted by industry; we
system to analyse which parts of a
different forbidden item, with mutation
are trying to help them bring it
system are most vulnerable to change.
testing we purposely inject a fault
on board.
Our solution was to determine the
into a system and then check whether
The idea behind our solution is
importance of a system part by how it
the test suite detects the fault. The
deceivingly simple. As already stated,
communicates with another part and
faulty version of the system is called
software is developed incrementally
uses data—a good compromise. Our
a mutant, hence the technique’s
over time. On a project’s first day,
system is now ready for the industry
name. We do this thousands of times,
there is no system and no test suite.
and we have named it Incremental
injecting all sorts of faults which
At the end of the day, there might be
Mutation Testing because it analyses a
programmers are likely to make and
a few dozen lines of code to build the
test suite as it is being built.
each time we see whether the test
program. A mutation test run at this
We now want to prove that it
suite is good enough to catch that
point would only take a few minutes.
can work in an industrial setting. We
fault. If not, we update the test suite to
Possible mutants are few because there
want to collaborate with industrial
make it better.
is so little to work on. The test suite
partners to run our system. We hope
Mutation testing takes time. A
would finish in two to five seconds. At
this will prove that our system can
typical test suite takes ten minutes,
the end of the second day, because of
make companies more efficient and
a modest mutation test run of 5,000
more code the mutation test should
cost-effective to develop higher quality
mutants would take 50,000 minutes,
take longer. Our technique is cleverer
software. We want them to build
or over a month of continuous analysis.
because it does not need to analyse all
better systems for everyone’s laptops,
A development team usually releases a
the code every time but only sees the
tablets, and phones.
Feature
if no failures occur, then the system
ran our system
43