Workshop on Distributed Software
Management for e-Science
Day 2 - Software Design
Lecture 6
Software Testing Basics
Davy Virdee
EPCC
Email:
D.Virdee@epcc.ed.ac.uk
Lecture Overview
4What is Software Testing?
4Why test software?
4When should we test?
4How to test software - techniques
4Summary
2
1
What is software testing?
Software testing includes the processes that :
4Find out how our software behaves
4Give us confidence that it does what we designed
it to do
4Establish the quality of the product
4In two words :
– Quality
– Quantification
3
Why Test? - Quantify
4Ultimately, testing is needed to quantify whether
your final software satisfies the design goals…
intersection
sluggish but functional
final item
good
final
item
perfect final item
clever final item but which
doesn’t satisfy basic
functional goals
hopeless final item
de
il
ta
merit
4
2
Why Test? - Qualify
4…and to make sure we have a quality product
A quality product is one that meets the end users
expectations
4End users opinions important
5
But…
4We should not use testing to
– Find bugs in badly made software
– As a point scoring exercise against a development
team
– As something “we have to do because you always
have to test software”
4Testing should not be a symptom of bad software
design and/or programming
4after all, we don’t build bad bridges…..
6
3
Bad design = failure!
Tacoma Bridge : testing the design is important!
7
What testing is not
4Poor reasons given for testing:
– We’re not very confident it does what it should
– So and So isn’t a very good programmer
– Our client has strict acceptance criteria
4And worst of all… a bug hunt!
– We rushed the design and development, so we’re
testing to find all the bad bugs before someone else
does!
4Testing is something we do all the time
– Test early, test often!
8
4
A Word on Test Failures
4Some programmers are reluctant to test because
tests can fail
4But a software test failure is no reason for tears
– a test failure gives you advance warning of a problem
that would otherwise lurk hidden
– early knowledge gives you the time to react calmly
– only the bugs you haven’t found yet are causes for
anxiety
4As mentioned in Problem Anticipation, the sooner
you find a problem, the less it will cost you
4Rejoice in the savings you’ve just made!
9
Types of Testing
4Different types of testing are appropriate at
different phases of the project
Code & Design
Early Design
no design
last design
(before coding)
final software
10
5
Early Design
4 Testing the quality of documents
– requirements document
– functional specification
– system design document
4 Seen already
4 Looking for deficiencies in the big 3 design criteria
– detail, intersection and merit
4 Start testing and designing tests at the beginning of your
project.
– Test early, test often
11
Early Design Quality
Definition
4Our design / specs tell us what we require
4We also need a quality definition
4A formal definition that lets us measure how good
our product is at all stages of the project
4Needs to be established at the start
4Examples
– “Windows look and feel”
– No user induced crashes
– Results +/- 1 % of published values
12
6
Code / Product Testing
4Testing the quality of your code / product
4Looking for deficiencies in intersection and merit
– have maximum detail once you’re at code level
4Low intersection
– code problems that prevent the software satisfying the basic
functional design goals
– crashes
• e.g. calling a function with a NULL argument, divide by zero,
unhandled exceptions
– data or state corruption
• e.g. an Aeroplane instance with a NULL wings variable
• may cause incorrect functionality or a crash
13
Code / Product Testing
4Low intersection cont.
– incorrect functionality
• e.g. matrix inversion gives the wrong answer
– missing functionality
• can’t test for functionality problems without a functional spec
• otherwise you don’t know what expected behaviour to compare
against the observed
– memory leaks
• memory allocated but not released
• system may run out of memory and grind to a halt
14
7
Code Testing
4Low merit
4Seen already
– ugly, sloppy, badly-commented code
– inefficient code
– interdependent code
15
Test Granularity
4Unit test
– testing of a small piece of code e.g. an ADT, class, module
– almost always the responsibility of the code author
4Integration test
– testing of several units combined into a sub-system
– often the responsibility of the code author(s)
4System test
– testing of the whole product –Stress/Volume testing, manual
– might (should?) be the responsibility of an external tester
4Acceptance test
– testing of the whole product by the customer / recipient
16
8
Difficulties of Testing
4Difficulties
– exhaustive test on all inputs is prohibitively expensive
– have to pick some ‘representative’ sample data
– awkward to measure the success of certain tests
• e.g. “when you click button X, panel Y should go a nice shade of
turquoise increasing in brightness towards the top right”
– test attitudes
• programmers don’t want to find bugs
• they like to believe they’ve got it right already
– testing is sometimes dull
• may involve repetitive manual work
• try to automate it
– when projects overrun, testing is the first thing to get squeezed
17
Useful Test Techniques
4Unit tests
– specially-written test code to exercise your ‘real’ code
4invariant checks on data during normal operation
– can run checks on arguments and results
4Systems tests
4Manual testing
4Targeted Code reviews / inspections (Day 3)
– Should not be used as a primary method
18
9
Unit Testing
4Unit Testing
4Possibly the most useful testing?
4Allows developers to test their own code
– Quick
– Defects easy to fix
4Quantifies product from the “ground –up”
4High degree of coverage
4Re-usability
4Gives confidence
19
Unit Test Details
4A unit test should exercise the module’s code
4It should call functions and compare actual results
against expected results
4Generate a test error if they don’t match
– if (stackTop(s) <> ‘c’) then testError(“Top element not correct in stack”)
4Count up the test errors
4Distinguish between test errors and ‘normal errors’
– a unit test may be designed to trigger a ‘normal error’
• e.g. deliberately trying to pop an element from an empty stack
– it’s then a test error if the normal error doesn’t occur!
20
10
Unit Test Approaches
4Black box testing
– exercise the code with no regard to its structure
– quick and easy to write the test
– may not get good code coverage
4White box testing
– have a ‘peek inside’ first
– designing a test having seen how the code has been written
– arrange the choice of function calls and inputs so that:
• a particular if...then...else branch gets executed
• a particular boundary condition gets exercised
– can expose more bugs, but takes longer to design the test
21
How to write a unit test
4 What do we need?
4 Design says we need a function adder :
int adder(int, int);
4 which takes two integers as arguments, and returns the sum
as an integer
4 We write the test first
22
11
How to write a unit test
4So the test looks something like :
public class TestAdder{
int first =1;
int second =2;
int sum;
sum = adder(first,second);
if ( sum != 3)
print “unit test failed for adder”;
}
23
How to write a unit test
4Now we write the code :
int adder(int first, int second)
{
return (first + second);
}
4Write your test first, code later
– Encourages developers to test
– “Extreme Programming”
24
12
Code Organisation
4Real code is in the red module directories
4Test code is in the blue test/ subdirectories
4simulator/
– infra/
• test/
– database/
• test/
– engine/
• test/
– product/
infra.lib
infra.exe
database.lib
database.exe
engine.lib
engine.exe
product.exe
25
Code Organisation
4 Makes unit testing much easier
– separate test executable for each module
– easier to identify source of an error
4 Makes module dependencies explicit
4 Needs some makefile expertise (Day 1)
4 Or for those with PCs, Microsoft Developer Studio is good
for setting this up
4 Junit
– excellent unit test framework for Java
26
13
When to unit test?
4All the time
4Unit tests should be quick, short and easy to run
4Test whenever you make a change
– But don’t forget to write a new test for any new functionality
– Get into the habit of testing often
4Batch runs of unit tests
– regression
– smoke tests
4Test early, test often!
27
Unit testing -cons
4Developers do it.
– Most developers find testing dull
4Constructive vs. destructive testing
– “Lets prove it does what it should”
– “Lets try and break it!”
– Both needed
4Bad unit tests will give a false sense of quality
– Tests need updating when new functionality is added
– A test that only tests 10% of module is a bad test
– Over-complex tests = bad code?
28
14
Invariant Checks
4Programs often rely on relationships between data
4Preservation of those relationships is something
that can be tested
– invariant checking
4In scientific programs, the science itself can yield
useful invariants
– e.g. conservation of energy, angular momentum
29
Invariant Check Example
4Consider a Rectangle ADT
– suppose it stores (x1, y1), (x2, y2) and area
– it should always be the case that area = abs((x2 - x1) * (y2 - y1))
– the ADT’s code can check that this invariant is maintained
FUNCTION rectCheckArea(r)
! Return false if area is wrong.
...
END FUNCTION rectCheckArea
SUBROUTINE rectDoubleSize(r)
! Double the size of the rectangle.
…
IF (.NOT. rectCheckArea(r)) THEN
WRITE(*, *) ‘Area wrong in rectDoubleSize’
END IF
END SUBROUTINE rectDoubleSize
30
15
Invariant Checks
4Unlike unit test code, invariant test code requires
additions to the body of ‘real’ code
4Keeping such tests ‘on’ in the release version may
slow it down unnecessarily
– or lead to embarrassing messages being printed
– Customer finding “Help, I am overwhelmed” in a release
4Can use conditional compilation to toggle them
– the assert function (in C) can be useful for this
– ‘debug’ and ‘release’ modes
– see Day 1
31
System Tests
4Test the entire system in its current state
4Performed at project milestones
– systems tests designed before coding takes place
– staged development
4Will include batch runs of unit test
– regression testing
4Will include stress / volume testing
– Trying to push your system to the limits
– Destructive testing
4Manual testing
– GUI Button pushing
32
16
Manual Testing
4Checking integrity of UIs
– Can be dull and labourious
– Very important from user’s perspective
4Destructive testing
– Trying to break software
– Can cause friction between testers and developers
4Will form part of acceptance criteria
– Check to make sure user can’t crash the system
– e.g. double-click top-right causes crash
33
Acceptance Testing
4Customer defined set of criteria
– Write a test to ensure each is met
4They should be included as part of your quality
definition
– So should not come as a surprise!
34
17
Maintaining Quality
4 Run systems tests often
– This includes batch runs of unit test suites
4 Recording test result metrics
– Identify which areas of the system are weak
– Gives confidence new functionality does not break software
4 Do not let tests or data rust!
– DO scrap old tests and write new ones
– If your testing finds a bug, turn this into a regression test
4 Use Revision Control
– Have a ‘bug-fix’ stream
4 DO re-define your quality definition
– Quality Assurance
35
Summary
4The Two Qs
– Quantify
– Quality
4Identify your quality goals at the start
4Design tests when you design the product
4Use unit testing
– Write tests, then write code
4Make testing part of every process
– Test early, test often
36
18
Useful Links
4General Testing
www.testing.com
www.sqatester.com
4Unit Testing
www.junit.org
www.extremeprogramming.org
37
19