Workshop on Distributed Software Management for e-Science Day 2 - Software Design Lecture 6 Software Testing Basics Davy Virdee EPCC Email: D.Virdee@epcc.ed.ac.uk Lecture Overview 4What is Software Testing? 4Why test software? 4When should we test? 4How to test software - techniques 4Summary 2 1 What is software testing? Software testing includes the processes that : 4Find out how our software behaves 4Give us confidence that it does what we designed it to do 4Establish the quality of the product 4In two words : – Quality – Quantification 3 Why Test? - Quantify 4Ultimately, testing is needed to quantify whether your final software satisfies the design goals… intersection sluggish but functional final item good final item perfect final item clever final item but which doesn’t satisfy basic functional goals hopeless final item de il ta merit 4 2 Why Test? - Qualify 4…and to make sure we have a quality product A quality product is one that meets the end users expectations 4End users opinions important 5 But… 4We should not use testing to – Find bugs in badly made software – As a point scoring exercise against a development team – As something “we have to do because you always have to test software” 4Testing should not be a symptom of bad software design and/or programming 4after all, we don’t build bad bridges….. 6 3 Bad design = failure! Tacoma Bridge : testing the design is important! 7 What testing is not 4Poor reasons given for testing: – We’re not very confident it does what it should – So and So isn’t a very good programmer – Our client has strict acceptance criteria 4And worst of all… a bug hunt! – We rushed the design and development, so we’re testing to find all the bad bugs before someone else does! 4Testing is something we do all the time – Test early, test often! 8 4 A Word on Test Failures 4Some programmers are reluctant to test because tests can fail 4But a software test failure is no reason for tears – a test failure gives you advance warning of a problem that would otherwise lurk hidden – early knowledge gives you the time to react calmly – only the bugs you haven’t found yet are causes for anxiety 4As mentioned in Problem Anticipation, the sooner you find a problem, the less it will cost you 4Rejoice in the savings you’ve just made! 9 Types of Testing 4Different types of testing are appropriate at different phases of the project Code & Design Early Design no design last design (before coding) final software 10 5 Early Design 4 Testing the quality of documents – requirements document – functional specification – system design document 4 Seen already 4 Looking for deficiencies in the big 3 design criteria – detail, intersection and merit 4 Start testing and designing tests at the beginning of your project. – Test early, test often 11 Early Design Quality Definition 4Our design / specs tell us what we require 4We also need a quality definition 4A formal definition that lets us measure how good our product is at all stages of the project 4Needs to be established at the start 4Examples – “Windows look and feel” – No user induced crashes – Results +/- 1 % of published values 12 6 Code / Product Testing 4Testing the quality of your code / product 4Looking for deficiencies in intersection and merit – have maximum detail once you’re at code level 4Low intersection – code problems that prevent the software satisfying the basic functional design goals – crashes • e.g. calling a function with a NULL argument, divide by zero, unhandled exceptions – data or state corruption • e.g. an Aeroplane instance with a NULL wings variable • may cause incorrect functionality or a crash 13 Code / Product Testing 4Low intersection cont. – incorrect functionality • e.g. matrix inversion gives the wrong answer – missing functionality • can’t test for functionality problems without a functional spec • otherwise you don’t know what expected behaviour to compare against the observed – memory leaks • memory allocated but not released • system may run out of memory and grind to a halt 14 7 Code Testing 4Low merit 4Seen already – ugly, sloppy, badly-commented code – inefficient code – interdependent code 15 Test Granularity 4Unit test – testing of a small piece of code e.g. an ADT, class, module – almost always the responsibility of the code author 4Integration test – testing of several units combined into a sub-system – often the responsibility of the code author(s) 4System test – testing of the whole product –Stress/Volume testing, manual – might (should?) be the responsibility of an external tester 4Acceptance test – testing of the whole product by the customer / recipient 16 8 Difficulties of Testing 4Difficulties – exhaustive test on all inputs is prohibitively expensive – have to pick some ‘representative’ sample data – awkward to measure the success of certain tests • e.g. “when you click button X, panel Y should go a nice shade of turquoise increasing in brightness towards the top right” – test attitudes • programmers don’t want to find bugs • they like to believe they’ve got it right already – testing is sometimes dull • may involve repetitive manual work • try to automate it – when projects overrun, testing is the first thing to get squeezed 17 Useful Test Techniques 4Unit tests – specially-written test code to exercise your ‘real’ code 4invariant checks on data during normal operation – can run checks on arguments and results 4Systems tests 4Manual testing 4Targeted Code reviews / inspections (Day 3) – Should not be used as a primary method 18 9 Unit Testing 4Unit Testing 4Possibly the most useful testing? 4Allows developers to test their own code – Quick – Defects easy to fix 4Quantifies product from the “ground –up” 4High degree of coverage 4Re-usability 4Gives confidence 19 Unit Test Details 4A unit test should exercise the module’s code 4It should call functions and compare actual results against expected results 4Generate a test error if they don’t match – if (stackTop(s) <> ‘c’) then testError(“Top element not correct in stack”) 4Count up the test errors 4Distinguish between test errors and ‘normal errors’ – a unit test may be designed to trigger a ‘normal error’ • e.g. deliberately trying to pop an element from an empty stack – it’s then a test error if the normal error doesn’t occur! 20 10 Unit Test Approaches 4Black box testing – exercise the code with no regard to its structure – quick and easy to write the test – may not get good code coverage 4White box testing – have a ‘peek inside’ first – designing a test having seen how the code has been written – arrange the choice of function calls and inputs so that: • a particular if...then...else branch gets executed • a particular boundary condition gets exercised – can expose more bugs, but takes longer to design the test 21 How to write a unit test 4 What do we need? 4 Design says we need a function adder : int adder(int, int); 4 which takes two integers as arguments, and returns the sum as an integer 4 We write the test first 22 11 How to write a unit test 4So the test looks something like : public class TestAdder{ int first =1; int second =2; int sum; sum = adder(first,second); if ( sum != 3) print “unit test failed for adder”; } 23 How to write a unit test 4Now we write the code : int adder(int first, int second) { return (first + second); } 4Write your test first, code later – Encourages developers to test – “Extreme Programming” 24 12 Code Organisation 4Real code is in the red module directories 4Test code is in the blue test/ subdirectories 4simulator/ – infra/ • test/ – database/ • test/ – engine/ • test/ – product/ infra.lib infra.exe database.lib database.exe engine.lib engine.exe product.exe 25 Code Organisation 4 Makes unit testing much easier – separate test executable for each module – easier to identify source of an error 4 Makes module dependencies explicit 4 Needs some makefile expertise (Day 1) 4 Or for those with PCs, Microsoft Developer Studio is good for setting this up 4 Junit – excellent unit test framework for Java 26 13 When to unit test? 4All the time 4Unit tests should be quick, short and easy to run 4Test whenever you make a change – But don’t forget to write a new test for any new functionality – Get into the habit of testing often 4Batch runs of unit tests – regression – smoke tests 4Test early, test often! 27 Unit testing -cons 4Developers do it. – Most developers find testing dull 4Constructive vs. destructive testing – “Lets prove it does what it should” – “Lets try and break it!” – Both needed 4Bad unit tests will give a false sense of quality – Tests need updating when new functionality is added – A test that only tests 10% of module is a bad test – Over-complex tests = bad code? 28 14 Invariant Checks 4Programs often rely on relationships between data 4Preservation of those relationships is something that can be tested – invariant checking 4In scientific programs, the science itself can yield useful invariants – e.g. conservation of energy, angular momentum 29 Invariant Check Example 4Consider a Rectangle ADT – suppose it stores (x1, y1), (x2, y2) and area – it should always be the case that area = abs((x2 - x1) * (y2 - y1)) – the ADT’s code can check that this invariant is maintained FUNCTION rectCheckArea(r) ! Return false if area is wrong. ... END FUNCTION rectCheckArea SUBROUTINE rectDoubleSize(r) ! Double the size of the rectangle. … IF (.NOT. rectCheckArea(r)) THEN WRITE(*, *) ‘Area wrong in rectDoubleSize’ END IF END SUBROUTINE rectDoubleSize 30 15 Invariant Checks 4Unlike unit test code, invariant test code requires additions to the body of ‘real’ code 4Keeping such tests ‘on’ in the release version may slow it down unnecessarily – or lead to embarrassing messages being printed – Customer finding “Help, I am overwhelmed” in a release 4Can use conditional compilation to toggle them – the assert function (in C) can be useful for this – ‘debug’ and ‘release’ modes – see Day 1 31 System Tests 4Test the entire system in its current state 4Performed at project milestones – systems tests designed before coding takes place – staged development 4Will include batch runs of unit test – regression testing 4Will include stress / volume testing – Trying to push your system to the limits – Destructive testing 4Manual testing – GUI Button pushing 32 16 Manual Testing 4Checking integrity of UIs – Can be dull and labourious – Very important from user’s perspective 4Destructive testing – Trying to break software – Can cause friction between testers and developers 4Will form part of acceptance criteria – Check to make sure user can’t crash the system – e.g. double-click top-right causes crash 33 Acceptance Testing 4Customer defined set of criteria – Write a test to ensure each is met 4They should be included as part of your quality definition – So should not come as a surprise! 34 17 Maintaining Quality 4 Run systems tests often – This includes batch runs of unit test suites 4 Recording test result metrics – Identify which areas of the system are weak – Gives confidence new functionality does not break software 4 Do not let tests or data rust! – DO scrap old tests and write new ones – If your testing finds a bug, turn this into a regression test 4 Use Revision Control – Have a ‘bug-fix’ stream 4 DO re-define your quality definition – Quality Assurance 35 Summary 4The Two Qs – Quantify – Quality 4Identify your quality goals at the start 4Design tests when you design the product 4Use unit testing – Write tests, then write code 4Make testing part of every process – Test early, test often 36 18 Useful Links 4General Testing www.testing.com www.sqatester.com 4Unit Testing www.junit.org www.extremeprogramming.org 37 19