Anders Fröberg
anders.froberg@liu.se
TDDD80
STORAGE AND TESTING
1
Agenda: Test
Unit testing vs Traditional Testing
Debugging and Refactoring
Deployment
(Test Driven Development (TDD))
(Acceptance Test Driven Development)
2
Software Faults, Errors &
Failures
Software Fault : A static defect in the software
Software Failure : External, incorrect behavior
with respect to the requirements or other
description of the expected behavior
Software Error : An incorrect internal state that
is the manifestation of some fault
3
Fault and Failure Example
A patient gives a doctor a list of symptoms
Failures
The doctor tries to diagnose the root cause, the
ailment
Fault
The doctor may look for anomalous internal
conditions (high blood pressure, irregular heartbeat,
bacteria in the blood stream)
Errors
4
The Term Bug
BUG
Bug is used informally
Sometimes speakers mean fault, sometimes
error, sometimes failure … often the speaker
doesn’t know what it means !
This class will try to use words that have
precise, defined, and unambiguous meanings
“It has been just so in all of my inventions.
The first step is an intuition, and comes with
a burst, then difficulties arise—this thing
gives out and [it is] then that 'Bugs'—as such
little faults and difficulties are called—show
themselves and months of intense watching,
study and labor are requisite. . .” – Thomas
Edison
5
“an analyzing process
must equally have been
performed in order to
furnish the Analytical
Engine with the
necessary operative data;
and that herein may also
lie a possible source of
error. Granted that the
actual mechanism is
unerring in its processes,
the cards may give it
wrong orders. ” – Ada,
Countess Lovelace (notes
on Babbage’s Analytical
Engine)
Mars Polar
Lander crash
site?
Spectacular Software Failures
Intel’s Pentium FDIV fault : Public relations
nightmare
NASA’s Mars lander: September 1999,
crashed due to a units integration fault
THERAC-25 radiation machine : Poor testing
of safety-critical software can cost lives : 3
patients were killed
Ariane 5 explosion : Very expensive
We need our software to be dependable
Testing is one way to assess dependability
6
Ariane 5:
exception-handling
bug : forced self
destruct on maiden
flight (64-bit to 16-bit
conversion: about
370 million $ lost)
THERAC-25 design
Northeast Blackout of 2003
508 generating units and 256
power plants shut down
Affected 10 million people in
Ontario, Canada
Affected 40 million people in 8
US states
Financial losses of
$6 Billion USD
The alarm system in the energy management system failed due to a software error and operators
were not informed of the power overload in the system
7
Costly Software Failures
NIST report, “The Economic Impacts of
Inadequate Infrastructure for Software Testing”
(2002)
Inadequate software testing costs the US alone between
$22 and $59 billion annually
Better approaches could cut this amount in half
Huge losses due to web application failures
Financial services : $6.5 million per hour (just in USA!)
Credit card sales applications : $2.4 million per hour (in
USA)
In Dec 2006, amazon.com’s BOGO offer
turned into a double discount
2007 : Symantec says that most security
vulnerabilities are due to faulty software
8
9
Enhetstest (Unit Testing)
Unit Testing vs Traditional Testing
Benefits of Unit Testing
10
Traditional Testing
Test the system as a whole
Individual components rarely
tested
Errors go undetected
Isolation of errors difficult to
track down
11
Traditional Testing Strategies
Print Statements
Use of Debugger
Debugger Expressions
Test Scripts
12
Unit Testing
Each part tested individually
All components tested at least once
Errors picked up earlier
Scope is smaller, easier to fix errors
13
Unit Testing Ideals
Isolatable
Repeatable
Automatable
Easy to Write
14
Why Unit Test?
Faster Debugging
Faster Development
Better Design
Excellent Regression Tool
Reduce Future Cost
15
Concept of Unit Testing
Static Unit Testing
•
•
•
Dynamic Unit Testing
•
•
•
•
•
16
Code is examined over all possible behaviors that might
arise during run time
Code of each unit is validated against requirements of
the unit by reviewing the code
A program unit is actually executed and its outcomes are
observed
One observe some representative program behavior,
and reach conclusion about the quality of the system
Static unit testing is not an alternative to
dynamic unit testing
Static and Dynamic analysis are
complementary in nature
Static Unit Testing (Code Review)
Step 1: Readiness
Criteria
Completeness
Minimal functionality
Readability
Complexity
Requirements and design
documents
Roles
Moderator
Author
Presenter
Record keeper
Reviewers
Observer
Step 2: Preparation
List of questions
Potential Change Request (CR)
Suggested improvement
opportunities
17
Dynamic Unit Testing
The environment of a unit is
emulated and tested in
isolation
The caller unit is known as
test driver
A test driver is a program that
invokes the unit under test (UUT)
It provides input data to unit
under test and report the test
result
The emulation of the units
called by the UUT are called
stubs
It is a dummy program
Dynamic unit test environment
18
The test driver and the stubs
are together called
scaffolding
The low-level design
document provides guidance
for selection of input test
data
Unit Testing in Programming
19
Unit Testing in Programming
20
JUnit (www.junit.org)
Java-based unit testing
framework
Elegantly simple
Easy to write unit tests
Easy to manage unit tests
Open source = Free!
21
Mature Framework
De facto java standard
Ant integration
Generic testing framework
Key Concepts in (J)Unit
Test interface
Assert
TestCase
Assert
<Test>
run(TestResult)
TestCase
setUp()
tearDown()
TestSuite
run(TestResult)
assertTrue
assertEquals
fail
TestSuite
TestDecorator/TestSetup
Failures vs Errors
22
(J)Unit is Easy
import unittest
class TestStringMethods(unittest.TestCase):
def test_upper(self):
self.assertEqual('foo'.upper(), 'FOO')
def test_isupper(self):
self.assertTrue('FOO'.isupper())
self.assertFalse('Foo'.isupper())
def test_split(self):
s = 'hello world'
self.assertEqual(s.split(), ['hello', 'world'])
# check that s.split fails when the separator is not a string
with self.assertRaises(TypeError):
s.split(2)
if __name__ == '__main__':
unittest.main()
23
Writing a Unit Test
1.
2.
3.
4.
5.
6.
7.
24
DEMO
Create a class to hold the unit tests
Initialise objects (setUp() method)
(State assertions – preconditions)*
Call operations on the objects that are being
unit tested
State assertions/failures expected
Clean up (tearDown() method)
Execute the unit test
Code Coverage
25
DEMO