Testing
Today’s Topics
•
•
•
•
•
1
Why Testing?
Basic Definitions
Kinds of Testing
Test-driven Development
Code Reviews (not testing)
Why Testing?
• Ideally, we’d prove that the code was correct using formal
mathematics
• This is extremely difficult even for trivially small programs
• Not practical in most cases
• Not even for mission critical or safety code
2
Why Testing?
•
•
•
•
Near ideal: use symbolic or abstract model testing
Prove that model is correct
Automatically extract mathematical model from code
Prove properties of model for all possible cases
• In practice, can work well for simple properties
• E.g., “System will not crash in this way”
• Doesn’t work well for complex systems / data structures
3
Why Testing?
• We can actually run code to see if it works
• This is software testing
• Always necessary, even when you can prove correctness
• Proof seldom tied to actual code
• Testing is NOT a lost resort; it is critical part of sw dev
4
Why Testing?
• Testing all too often an afterthought
• Test before faults become too expensive to fix or pressing
deadlines
• The more code written before fault is detected, the more code
needs to be fixed…
• Better to detect fault(s) on a small prototype or “finished”
product?
5
Waterfall
Requirements analysis
Prototyping
Design
Implementation
Testing
Operation
6
Spiral
Draft a menu of
program designs
Analyze risk &
prototype
Draft a menu of
architecture designs
Draft a menu of
requirements
Establish
architecture
Plan
7
Analyze risk &
prototype
Establish
requirements
Plan
Operation
Analyze risk &
prototype
Testing
Implementation
Establish
program design
Agile
Do “spike” to evaluate & control risk
Customer provides “stories”
(short requirement snippets)
Prioritize
stories and plan
Operation
Write/run/modify
unit tests
Implement
System and acceptance tests
8
Testing Saves Lives and Money
• NIST Report: “The Economic Impact of Inadequate
Infrastructure for Software Testing”
• Inadequate software testing costs US $22 - $59 billion annually
• Better approaches could cut this is half
9
Testing Saves Lives and Money
• Major failures:
•
•
•
•
Ariane 5 explosion
Mars Polar Lander
Intel Pentium FDIV bug
THERAC-25 radiation machine: killed 5
Ariane 5- exception-handling
bug forced self destruct on
maiden flight (64-bit to 16-bit
conversion: about $370 mill lost)
10
Today’s Topics
•
•
•
•
•
11
Why Testing?
Basic Definitions
Kinds of Testing
Test-driven Development
Code Reviews (not testing)
Definitions: Testing
•
•
•
•
•
12
Running a program to find faults (bugs)
… in code
… in specs
… in documentation
… in tests
Definitions: Test Case, Test Suite, Tester
• Test case: one execution of the program that may find a bug
• Test suite: a set of program executions, grouped together
• Made of up test cases
• Tester: program that generates tests
13
Definitions: Coverage
• Measures or metrics
• Abstraction of “what a test suite tests”
• Common measures:
•
•
•
•
14
Statement coverage
Decision coverage
Path coverage
Mutation coverage
Definitions: Coverage
• Statement coverage:
• aka: line or basic block coverage
• Which statements execute in test suite
• Decision coverage:
• Which boolean expressions in control structures evaluate to true and
false during suite execution.
15
Definitions: Coverage
• Path coverage:
• Which paths through a program’s control flow graph are taken in the test
suite
• Mutation coverage:
• Ability to detect random variations in the code
16
Today’s Topics
•
•
•
•
•
17
Why Testing?
Basic Definitions
Kinds of Testing
Test-driven Development
Code Reviews (not testing)
Kinds of Testing
•
•
•
•
•
•
18
White box
Black box
Unit
Integration
System
Acceptance
White Box Testing
• Opens up the box
• aka glass box, clear box, structural testing
• Use source code to design test cases
19
Black Box Testing
• Testing does NOT look at source code or internal structures
• Sends program stream of inputs, observes outputs
• Abstracts away the internals
• Useful perspective for system and integration testing
• Sometimes you don’t have access to the source code
20
Testing Stages
• Unit testing: first phase; done by module develops
• Integration testing: combines unit tested modules, tests
interactions
• System testing: test whole program to make sure reqs met
• Acceptance testing: tests by users to see if system meets reqs
21
Unit Testing
•
•
•
•
First phase; mostly done by module developers
Typically earliest type of testing
Unit could be as small as single function or method
Often relies on “stubs” to represent other modules or
incomplete code
• Tools to support unit tests available for most popular
languages, e.g., Junit ( http://junit.org )
?
22
Integration Testing
•
•
•
•
Combines unit-tested modules and tests interactions
Relies on completed units
After unit testing, before system testing
Test cases focus on interfaces between components and
assemblies of multiple components
• Often more formal than unit testing
• More detailed test plan(s)
?
?
?
23
System Testing
•
•
•
•
Tests whole program; make sure reqs met
After integration testing
Focuses on “breaking the system”
Finds defects in completed product, not just component
interactions
• Checks quality of requirements as well as system
• Often includes stress testing: beyond bounds of well-defined
behavior
24
System Testing: Functional Testing
• Developer tests program from “user’s” perspective
• Different mindset than unit testing
• Analogy: home inspection vs. walk-through
• inspector => software engineer; looking for “hard facts”
• buyer => system user; looking for functionality
25
Acceptance Testing
• Tests by user to see if system meets use requirements
• Form of black box testing
• Done by end-users to determine if system really meets needs
• May revise requirements
• May find bugs in the system
26
Today’s Topics
•
•
•
•
•
27
Why Testing?
Basic Definitions
Kinds of Testing
Test-driven Development
Code Reviews (not testing)
Test Drive Development
•
•
•
•
Write test cases before code
Make sure code is tested as early as possible
Idea arises from Extreme Programming
Often used in agile development
• Write automated test cases first
• Then write code to satisfy tests
28
Test Driven Development
• How to add a feature:
1.
2.
3.
4.
29
Add a test case that fails (but succeeds with new feature)
Run all tests, make sure only new test fails
Write code to implement new feature
Rerun all tests, make sure new test succeeds (no others fail)
Test Driven Development Cycle
30
Test Driven Development Benefits
• Results in lots of useful test cases
• A very large regression set
• Forces attention to controllable and observable sys behavior
• Only write code as needed to pass tests
• May get good coverage of paths through program
• Reduces temptation to tailor tests to idiosyncratic behaviors
• Testing is a first-class activity in this kind of development
31
Problems with Test Driven Development
• Needs institutional support
• Difficult to integrate with waterfall process
• Management may wonder why time is spent writing tests, not code
• Many test cases may create false confidence
• If developers have written all tests, there may be blind spots due
32
Exhaustive vs. Representative Testing
• Can we test everything? (exhaustive)
• Probably not! Permutations / combinations are exponential!
• Need to be thoughtful about creating representative tests
• Create tests at boundary conditions
• Want to “represent” the range of input values
33
Today’s Topics
•
•
•
•
•
34
Why Testing?
Basic Definitions
Kinds of Testing
Test-driven Development
Code Reviews (not testing)
Code Reviews
• Code Review ≠ Testing
• Method for finding bugs and determining code quality
• Code walkthrough: developer leads review team through code
• Informal
• Code inspection: review team checks against concerns
• Prepared offline (in many cases)
• Led by a team moderator
35
Code Reviews
• Code inspections: found to be one of the most effective
practices for finding faults
• Some experiments show removal of 67-85% of defects via inspections
• Some consider XP’s pair programming as a kind of “code
review” process
• Can review/walkthrough requirements and design docs, not
just code
36
Next Steps
• Homework 5 due Friday!!
• Vision paper due 11/21!!
• Midterm due 11/25!!
37