Regression testing
Tor Stållhane
What is regression testing – 1
Regression testing is testing done to check
that a system update does not reintroduce errors that have been corrected
earlier.
All – or almost all – regression tests aim at
checking
• Functionality – black box tests.
• Architecture – grey box tests
What is regression testing – 2
Since they are supposed to test all
functionality and all previously done
changes, regression tests are usually
large.
Thus, regression testing needs automatic
• Execution – no human intervention
• Checking. Leaving the checking to
developers will not work.
Automating regression tests – 1
We face the same challenge when doing
automating regression test as we face
when doing automatic test checking in
general:
Which parts of the output should be
checked against the oracle?
This question gets more important as we
need to have more version of the same
test due to system variability.
Automating regression tests – 2
Simple but annoying – and some times
expensive – problems are e.g.
• Use of date in the test output
• Changes in number of blanks or line shifts
• Other format changes
• Changes in lead texts
Automating regression tests – 3
A simple solution seems to be to use
assertions. This is not a good alternative
for the following reasons: The assertion
will have to be
• Inside the system all the time – extra
space and execution time
• Only in the test version – one extra version
per variation to maintain.
Automating regression tests – 4
The answer to the question on the previous
slide is to parameterize the test results.
Instead of using a complete output file as
the oracle we use:
• A tool to extract the relevant info from the
output file
• Compare the extracted info with the info
stored in the oracle
Automating regression tests – 5
Test data
Build results
for version x
Build tests
for version x
Run tests
for version x
Compare
Verdict
Run all regression tests
Since a regression test is large, it is always a need
to identify which parts of the regression tests
need to be run after a change – e.g. an error
correction.
Thus, traceability – which components are used to
realize which functionality or requirement – is
important info for two reasons:
• Save execution time. Fast and cheap hardware
is, however, steadily reducing this need.
• Know which tests to change when we change
functionality.
Improving the regression test
Once we have a regression test, it is
important to update it each time we
• Fix a bug
• Add, change or remove functionality
• Change platform
If we create variants of the system, we also
need to create parallel variants of the
regression test suite.
Bug fixing – 1
When somebody reports a bug, the first
activity is to reproduce it – preferably with
the simplest input sequence possible.
When the bug is reproduced, we need to
make a new entry in the regression test.
This entry will be one of the following
• The original reported input sequence
• The simplest possible input sequence
Bug fixing – 2
• The original reported input sequence
– Pro: is the real thing and will tell us something
about how the system is used in real life.
– Con: can be large, complex and difficult to
understand
• The simplest possible input sequence
– Pro: is (usually) simple to understand
– Con: is an artificial input sequence and has no
info beside being able to reproduce a reported
bug