Software Testing
Testing a ballpoint pen
No issue is meaningful unless it can be put to
the test of decisive verification.
C.S. Lewis, 1934
Oktober 2004
PUM, Mariam Kamkar, IDA, LiU
1
Goal: develop software to meet its intended use!
But: human beings make mistake!
bridge
automobile
television
Customer
2
• Software must be verified in much the same
spirit. In this lecture, however, we shall
learn that verifying software is perhaps
more difficult than verifying other
engineering products.
word processor
PUM, Mariam Kamkar, IDA, LiU
PUM, Mariam Kamkar, IDA, LiU
• Verifying bridge = verifying design,
construction, process,…
⇒ Product of any engineering activity must be verified
against its requirements throughout its development.
Maj 2004
Maj 2004
3
We shall try to clarify why this is so.
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
4
Developer
Outline
Requirements definition
Requirements specification
•
•
•
•
•
•
•
Functional requirements
Nonfunctional requirements
Some notations
Integration testing
Component/Unit/Module/Basic testing
Function testing
Performance testing
Acceptance testing
Installation testing
• Real life examples
Design Specification
Code = System
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
5
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
6
1
Error, Fault, Failure
Debugging vs Testing
• Debugging: to find the bug
• Testing: to demonstrate the existence of a
fault
Can lead to
Can lead to
– fault identification
– fault correction / removal
Human error
Fault
Failure
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
7
Maj 2004
Types of Faults
•
•
•
Algorithmic: division by zero
Computation & Precision: order of op
Documentation: doc - code
Stress/Overload: data-str size ( dimensions of
tables, size of buffers)
Capacity/Boundary: x devices, y parallel tasks, z
interrupts
Timing/Coordination: real-time systems
Throughout/Performance: speed in req
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
•
•
•
9
Recovery: power failure
Hardware & System Software: modem
Standards & Procedure: organizational standard;
difficult for programmers to follow each other
Maj 2004
Unit & Integration Testing
PUM, Mariam Kamkar, IDA, LiU
10
Classes of Integration Testing
Objective: to ensure that code implemented
the design properly.
•
•
•
•
Top-down
Bottom-up
Big bang
Sandwich
Design Specification
Code = System
Maj 2004
8
Types of Faults
(dep. on org. IBM, HP)
•
•
•
•
PUM, Mariam Kamkar, IDA, LiU
PUM, Mariam Kamkar, IDA, LiU
11
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
12
2
Top-down
Components
driver
Boundary conditions
independent paths
interface
...
Component
to be
tested
stub
stub
Maj 2004
Test
cases
PUM, Mariam Kamkar, IDA, LiU
13
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
14
Top-down
A
B
E
Maj 2004
C
F
D
G
PUM, Mariam Kamkar, IDA, LiU
15
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
16
Bottom-up
A
B
E
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
17
Maj 2004
C
F
D
G
PUM, Mariam Kamkar, IDA, LiU
18
3
Bottom-up
Maj 2004
Big-bang
PUM, Mariam Kamkar, IDA, LiU
19
Maj 2004
Sandwich
Component code
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
21
Bottom-up
Big-bang
Sandwich
Time to a
basic working
program
Early
Late
Late
Early
Driver needed
No
Yes
Yes
In part
Stubs needed
Yes
No
Yes
In part
Maj 2004
Component code
PUM, Mariam Kamkar, IDA, LiU
22
Unit Testing
Tested components
•
•
•
•
Integration
test
Unit
test
20
Top-down
Design Specification
Unit
test
PUM, Mariam Kamkar, IDA, LiU
Tested components
Code Inspections
Code Walkthroughs
Open box testing
Black box testing
Integrated modules
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
23
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
24
4
Two Types of Oracles
Input
Failure?
Test
Object
Output
Oracle
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
25
• Human: an expert that can examine an input
and its associated output and determine
whether the program delivered the correct
output for this particular input.
• Automated: a system capable of performing
the above task.
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
26
Balls and Urn
• Testing can be viewed as selecting different colored balls
from an urn where:
– Black ball = input on which program fails.
– White ball = input on which program succeeds.
• Only when testing is exhaustive is there an “empty” urn.
A correct program
A program that always fails
Urn (program)
Balls (inputs)
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
A typical program
27
Maj 2004
some classical programming errors
•
•
•
•
•
•
•
•
overview
preparation
inspection
rework
follow-up
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
28
Inspection (cont)
Inspection
•
•
•
•
•
PUM, Mariam Kamkar, IDA, LiU
29
Use of un-initialized variables
Jumps into loops
Non-terminating loops
Incompatible assignments
Array indexes out of bounds
Off-by-one errors
Improper storage allocation or de-allocation
Mismatches between actual and formal parameters in
procedure calls
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
30
5
Walkthroughs
Discovery activity
design, code, chapter of user’s guide,…
• presenter
• coordinator
• secretary
• maintenance oracle
• standards bearer
• user representative
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
Faults found per
thousand lines of code
Requirements review
Design review
Code inspection
Integration test
Acceptance test
31
Maj 2004
Black box / Closed box testing
2.5
5.0
10.0
3.0
2.0
PUM, Mariam Kamkar, IDA, LiU
32
Black box testing
• incorrect or missing functions
• interface errors
• performance error
• Equivalence partitioning
• Boundary value analysis
• Exhaustive testing
input
output
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
33
Equivalence partitioning
Invalid inputs
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
34
Specification: the program accepts four to
eight inputs which are 5 digit integers
greater than 10000.
Valid inputs
outputs
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
35
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
36
6
Guidelines
Boundary value analysis
If an input condition specifies
• A range: one valid and two invalid
equivalence classes.
• A specific value: one valid and two invalid
equivalence classes.
• A member of a set: one valid and one
invalid equivalence classes.
• A boolean: one valid and one invalid class.
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
Less than 10000
37
Maj 2004
Between 10000 and 99999
More than 99999
PUM, Mariam Kamkar, IDA, LiU
38
Exhaustive testing
Glass box testing!
White box testing!
Open box testing!
Clear box testing!
• Definition: testing with every member of
the input value space.
• Input value space: the set of all possible
input values to the program.
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
39
Maj 2004
logical decision
loops
internal data structure
paths
...
Begin
if ( y >= 0)
then y = 0;
abs = y;
end;
test case-1 (yes):
input:
expected result:
actual result:
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
40
Statement Coverage
Glass box testing
•
•
•
•
•
PUM, Mariam Kamkar, IDA, LiU
41
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
begin
y >= 0
yes
y=0
abs = y
y=?
?
?
42
7
Branch Coverage
Begin
if ( y >= 0)
then y = 0;
abs = y;
end;
if ( x < 10 && y > 20) {
begin
y >= 0
no
yes
y=0
abs = y
test case-1(yes):
input: y = 0
expected result: 0
actual result: 0
Maj 2004
Begin
z = foo (x,y); else z =fie (x,y);
}
end;
test case-1 (yes):
input: x = ?, y = ?
expected result: ?
actual result: ?
test case-2 (no):
input: y = ?
expected result: ?
actual result: ?
PUM, Mariam Kamkar, IDA, LiU
43
Maj 2004
x<? y>?
----------------------------------test-case-1: t
t
test-case:2
t
f
test-case-3: f
t
test-case-4
f
f
PUM, Mariam Kamkar, IDA, LiU
45
no
z=fie (x,y)
z=foo (x,y)
test case-2 (no):
input: x = ?, y = ?
expected result: ?
actual result: ?
44
yes
x <> 0
z = z-x
z = sin(x)
yes
(n, y) x = ?, z = ?
(y, n) x = ?, z = ?
no
z > 10
z=0
z=z/x
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
Path with loops
(n, n) x = ?, z = ?
(n, y) x = ?, z = ?
(y, n) x = ?, z = ?
(y, y) x = ?, z = ?
46
Path with loops
a
a
d
d
c
b
e
c
e
a
?
?
PUM, Mariam Kamkar, IDA, LiU
a
c,b,d
e
Maj 2004
yes
Path Coverage
Begin
if ( x < 10 && y > 20) {
no
yes
x < 10
z = foo (x,y); else z =fie (x,y);
no
y > 20
}
yes
end;
z=fie (x,y)
z=foo (x,y)
b
x<10
&&
y>20
PUM, Mariam Kamkar, IDA, LiU
Condition - Branch Coverage
Maj 2004
no
d
e
47
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
48
8
Data Flow Testing
Data Flow testing
s = 0;
i = 1;
s = 1;
while (i <= n)
{
DEF(S) = {x | statement S contains a definition of variable x}
USE(S) = {x | statement S contains a use of variable x}
DEF-USE-Chain (du chain) = [x, S, S’]
...
s + = i;
i ++
S1:
i = 1;
S2:
while (i <= n)
Maj 2004
du: def-use
dk: def-kill
?
}
print (s);
print (i);
print (n);
PUM, Mariam Kamkar, IDA, LiU
49
Maj 2004
50
Relative strengths of test strategies (B. Beizer 1990)
Program Slicing
s = 0;
i = 1;
while (i <= n)
{
s + = i;
i ++
}
print (s);
print (i);
print (n);
PUM, Mariam Kamkar, IDA, LiU
All paths
All definition-use paths
i = 1;
while (i <= n)
{
All uses
i ++
All predicate/
Some computational uses
All computational/
Some predicate uses
}
print (i);
All computational uses
All predicate uses
Branch
All definition
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
51
PUM, Mariam Kamkar, IDA, LiU
Customer
Function testing
Performance testing
Acceptance testing
Installation testing
Developer
Requirements definition
Requirements specification
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
52
Objective: to ensure that the system does what
the customer wants it to do.
Outline
•
•
•
•
Maj 2004
Statement
53
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
Functional requirements
Nonfunctional requirements
54
9
Component code
Unit
test
Function
test
Tested components
Integrated modules
Acceptance
test
Customer requirements spec.
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
55
Function testing
Performance
test
Installation
test
System
In
Use!
User environment
PUM, Mariam Kamkar, IDA, LiU
56
Cause-Effect
(test case generation from req.)
(testing one function at a time)
functional requirements
• have a high probability of detecting a fault
• use a test team independent of the designers and
programmers
• know the expected actions and output
• test both valid and invalid input
• never modify the system just to make testing
easier
• have stopping criteria
Maj 2004
Maj 2004
Other software requirements
Verified validated
software
Integration
test
Functioning systems
Tested components
Accepted system
Unit
test
Integrated modules
Component code
System functional requirements
Design Specification
PUM, Mariam Kamkar, IDA, LiU
Causes
C1: command is credit
C2: command is debit
C3: account number is
valid
C4: transaction amount is
valid
57
Maj 2004
Effects
E1: print “invalid command”
E2: print “invalid account
number”
E3: print “debit amount not valid ”
E4: debit account print
E5: credit account print
PUM, Mariam Kamkar, IDA, LiU
58
Performance testing
nonfunctional requirements
C2
and
C3
•
•
•
•
•
•
•
E3
not
C4
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
59
Security
Accuracy
Speed
Recovery
Stress test
Volume test
…
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
60
10
Acceptance testing
customers, users need
Installation testing
users site
• Benchmark test: a set of special test cases
• Pilot test: everyday working
Acceptance test at developers site
Æ installation test at users site,
otherwise may not be needed!!
– Alpha test: at the developer’s site, controlled
environment
– Beta test: at one or more customer site.
• Parallel test: new system in parallel with
previous one
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
61
Maj 2004
Test Planing
•
•
•
•
•
•
PUM, Mariam Kamkar, IDA, LiU
62
Automated Testing Tools
Establishing test objectives
Designing test cases
Writing test cases
Testing test cases
Executing tests
Evaluating test results
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
• Code Analysis tools
– Static, Dynamic
• Test execution tools
– Capture-and-Replay
– Stubs & Drivers
• Test case generator
63
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
64
Termination Problem
How decide when to stop testing
Scaffolding
Oracle
• The main problem for managers!
What can be automated?
• Termination takes place when
• resources (time & budget) are over
• found the seeded faults
• some coverage is reached
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
Test case
generation
65
Maj 2004
Termination
PUM, Mariam Kamkar, IDA, LiU
66
11
Real life examples
Real life examples
• First U.S. space mission to Venus failed.
(reason: missing comma in a Fortran do loop)
• December 1995: AA, Boeing 575, mountain crash
in Colombia, 159 killed. Incorrect one-letter
computer command (Cali, Bogota 132 miles in
opposite direction, have same coordinate code)
• June 1996: Ariane-5 space rocket, self-destruction,
$500 million.
(reason: reuse of software from Ariane-4 without
recommended testing).
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
67
• Australia: Man jailed because of computer glitch.
He was jailed for traffic fine although he had
actually paid it for 5 years ago.
• Dallas Prisoner released due to program design
flaw: He was temporary transferred from one prison
to another (witness). Computer gave him
“temporary assignment”.
Maj 2004
Goals of software testing: Historical Evolution
PUM, Mariam Kamkar, IDA, LiU
68
And …
Years
Prevent software faults
Measure SQA test objectives
1980
Find faults 1981: Deutsch. Software project V and V
1979: Myers, “Art of software testing”
1970
cost
complexity
# applic.
Testing can show the presence, but never the
absence of errors in software.
E. Dijkstra, 1969
Establish confidence
1972 June, First formal conference software testing, university of
North Carolina, Bill Hetzel.
1960
1957, Charles Baker distinguished debugging from testing
1950
Not distinguished from debugging
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
69
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
71
Maj 2004
PUM, Mariam Kamkar, IDA, LiU
70
12