Software Testing
No issue is meaningful unless it can be put to
the test of decisive verification.
C.S. Lewis, 1934
2001-10-10
PUM-I, Mariam Kamkar, IDA
1
Real life examples
• First U.S. space mission to Venus failed.
(reason: missing comma in a Fortran do loop)
• December 1995: AA, Boeing 575, mountain crash
in Colombia, 159 killed. Incorrect one-letter
computer command (Cali, Bogota 132 miles in
opposite direction, have same coordinate code)
• June 1996: Ariane-5 space rocket, self-destruction,
$500 million.
(reason: reuse of software from Ariane-4 without
recommended testing).
2001-10-10
PUM-I, Mariam Kamkar, IDA
2
1
Real life examples
• Australia: Man jailed because of computer glitch.
He was jailed for traffic fine although he had
actually paid it for 5 years ago.
• Dallas Prisoner released due to program design
flaw: He was temporary transferred from one prison
to another (witness). Computer gave him
“temporary assignment”.
2001-10-10
PUM-I, Mariam Kamkar, IDA
3
Why Testing and Analysis?
• Software is never correct no matter which
developing technique is used
• Any software must be verified.
• Software testing and analysis are
– important to control the quality of the product
(and of the process)
– very (often too) expensive
– difficult and stimulating
2001-10-10
PUM-I, Mariam Kamkar, IDA
4
2
Statement Coverage
begin
Begin
if ( y >= 0)
then y = 0;
abs = y;
end;
test case-1:
y >= 0
yes
y=0
abs = y
input: y = 0
expected result: 0
actual result: ?
2001-10-10
PUM-I, Mariam Kamkar, IDA
5
Branch Coverage
Begin
if ( y >= 0)
then y = 0;
abs = y;
end;
begin
y >= 0
no
y=0
abs = y
test case-1:
input: y = 0
expected result: 0
actual result: ?
2001-10-10
yes
test case-2:
input: y = -5
expected result: 5
actual result: ?
PUM-I, Mariam Kamkar, IDA
6
3
Begin
no
if ( x < 10 && y > 20) {
z = foo (x,y); else z =fie (x,y);
}
end;
test case-1:
input: x = -4, y = 30
expected result: ...
actual result: ?
2001-10-10
z=fie (x,y)
x<10
&&
y>20
yes
z=foo (x,y)
test case-2:
input: x = 12, y = 12
expected result: ...
actual result: ?
PUM-I, Mariam Kamkar, IDA
7
Condition / Branch Coverage
Begin
no
yes
if ( x < 10 && y > 20) {
x < 10
z = foo (x,y); else z =fie (x,y);
no
y > 20
}
yes
end;
z=fie (x,y)
z=foo (x,y)
x<10 y>20
----------------------------------test-case-1: t
t
test-case:2
t
f
test-case-3: f
t
test-case-4
f
f
2001-10-10
PUM-I, Mariam Kamkar, IDA
8
4
Path Coverage
no
yes
x <> 0
z = z-x
x = 0, z = 12
x = 2, z = 6
z = sin(x)
yes
no
z > 10
z=0
x = 0, z = 7
x = 0, z = 13
x = 1, z = 5
x = 2, z = 15
z=z/x
2001-10-10
PUM-I, Mariam Kamkar, IDA
9
Path with loops
a
d
b
c
e
a
c,b,d
d
e
2001-10-10
PUM-I, Mariam Kamkar, IDA
10
5
Input
Failure?
P
r
o
g
r
a
m
Output
Oracle
2001-10-10
PUM-I, Mariam Kamkar, IDA
11
Glass box testing!
White box testing!
Structural testing!
2001-10-10
PUM-I, Mariam Kamkar, IDA
12
6
Glass box testing
•
•
•
•
•
logical decision
loops
internal data structure
paths
...
2001-10-10
PUM-I, Mariam Kamkar, IDA
13
Error, Fault, Failure
Can lead to
Can lead to
Human error
Fault
Failure
2001-10-10
PUM-I, Mariam Kamkar, IDA
14
7
Black box testing!
Functional testing!
2001-10-10
PUM-I, Mariam Kamkar, IDA
15
Black box testing
• incorrect or missing functions
• interface errors
• performance error
input
output
2001-10-10
PUM-I, Mariam Kamkar, IDA
16
8
Black box testing
•
•
•
•
•
Equivalence partitioning
Boundary value analysis
Decision Table
Cause-Effect
Exhaustive testing
2001-10-10
PUM-I, Mariam Kamkar, IDA
17
Equivalence partitioning
Invalid inputs
Valid inputs
outputs
2001-10-10
PUM-I, Mariam Kamkar, IDA
18
9
Specification: the program accepts four to
eight inputs which are 5 digit integers
greater than 10000.
input values
Less than 10000
Between 10000 and 99999
More than 99999
Number of input values
Less than 4
2001-10-10
Between 4 and 8
More than 8
PUM-I, Mariam Kamkar, IDA
19
Guidelines
If an input condition specifies
• a range: one valid and two invalid
equivalence classes.
• A specific value: one valid and two invalid
equivalence classes.
• A member of a set: one valid and one
invalid equivalence classes.
• A boolean: one valid and one invalid class.
2001-10-10
PUM-I, Mariam Kamkar, IDA
20
10
Boundary value analysis
10000
9999
Less than 10000
2001-10-10
99999
10001
99998
Between 10000 and 99999
100000
More than 99999
PUM-I, Mariam Kamkar, IDA
21
Cause-Effect
#AUSES
%FFECTS
C1: command is credeit
C2: command is debit
C3: account number is
valid
C4: transaction amount is
valid
E1: print “invalid command”
E2: print “invalid account
2001-10-10
number”
E3: print “debit amount not valid ”
E4: debit account print
E5: credit account print
PUM-I, Mariam Kamkar, IDA
22
11
C2
and
C3
E3
not
C4
2001-10-10
PUM-I, Mariam Kamkar, IDA
23
Data Flow Testing
DEF(S) = {x | statement S contains a definition of variable x}
USE(S) = {x | statement S contains a use of variable x}
S1 :
i = 1;
DEFS (S1) = {i}
(d1-i)
S2 :
while (i <= n)
USE(S2) = {i, n}
(u1-i, u2-n)
definition-use chain (du chain) = [x, S, S’]
du-1: [i, S1, S2]
2001-10-10
(d1-i), (u1-i)
PUM-I, Mariam Kamkar, IDA
24
12
Data Flow testing
s = 0;
i = 1;
while (i <= n)
{
s + = i;
i ++
}
print (s);
print (i);
print (n);
(d1-s)
(d2-i)
(u1-i, u2-n)
2001-10-10
PUM-I, Mariam Kamkar, IDA
du-s
(u3-s, u4-i, d3-s)
(u5-i, d4-i)
du-s
(u6-s)
(u7-i)
(u8-n)
25
Data Flow Testing
s = 0;
i = 1;
while (i <= n)
{
s + = i;
i ++
}
print (s);
print (i);
print (n);
2001-10-10
(d1-s)
(d2-i)
(u1-i, u2-n)
dd: def-def
dk: def-kill
du: def-use
...
(u3-s, u4-i, d3-s)
(u5-i, d4-i)
(u6-s)
(u7-i)
(u8-n)
PUM-I, Mariam Kamkar, IDA
26
13
Program Slicing
s = 0;
i = 1;
while (i <= n)
{
s + = i;
i ++
}
print (s);
print (i);
print (n);
2001-10-10
i = 1;
while (i <= n)
{
i ++
}
print (i);
PUM-I, Mariam Kamkar, IDA
27
Static Software Testing
• Inspection (design, code)
–
–
–
–
–
overview
preparation
inspection
rework
follow-up
2001-10-10
PUM-I, Mariam Kamkar, IDA
28
14
• Walkthroughs (design, code, chapter of
user’s guide,…)
–
–
–
–
–
–
presenter
coordinator
secretary
maintenance oracle
standards bearer
user representative
2001-10-10
PUM-I, Mariam Kamkar, IDA
29
Levels of Software Testing
•
•
•
•
Component
Integration
System
Acceptance
2001-10-10
PUM-I, Mariam Kamkar, IDA
30
15
Components
driver
Component
to be
tested
stub
2001-10-10
stub
Boundary conditions
independent paths
interface
...
Test
cases
PUM-I, Mariam Kamkar, IDA
31
Classes of Integration Testing
•
•
•
•
•
Bottom-up
Top-down
Big bang
Sandwich
Regression
2001-10-10
PUM-I, Mariam Kamkar, IDA
32
16
A
B
E
C
F
2001-10-10
D
G
PUM-I, Mariam Kamkar, IDA
33
Bottom-up
ABCDEFG
BEF
E
2001-10-10
C
F
DG
G
PUM-I, Mariam Kamkar, IDA
34
17
Top-down
A
ABCD
ABCDEFG
2001-10-10
PUM-I, Mariam Kamkar, IDA
35
System Testing
•
•
•
•
Recovery testing (fault tolerant)
Security testing
Stress testing (volume, resources,…)
Performance testing (real-time, embedded
system)
2001-10-10
PUM-I, Mariam Kamkar, IDA
36
18
Acceptance Testing
• Alpha test: at the developer’s site,
controlled environment
• Beta test: at one or more customer site.
2001-10-10
PUM-I, Mariam Kamkar, IDA
37
Termination Problem
How decide when to stop testing
• The main problem for managers!
• Termination takes place when
• resources (time & budget) are over
• found the seeded faults
• some coverage is reached
2001-10-10
PUM-I, Mariam Kamkar, IDA
38
19
Scaffolding
Oracle
What can be automated?
Test case
generation
2001-10-10
Termination
PUM-I, Mariam Kamkar, IDA
39
Testing object-oriented systems
Easier
Harder
inheritance
modularity
Small methods
reuse
Interface
identified early
2001-10-10
encapsulation
Quicker development
polymorphism
Dynamic binding
Complex interfaces
More integration
PUM-I, Mariam Kamkar, IDA
40
20
Goals of software testing: Historical Evolution
Years
Prevent software faults
Measure SQA test objectives
1980
Find faults 1981: Deutsch. Software project V and V
1979: Myers, “Art of software testing”
1970
cost
complexity
# applic.
Establish confidence
1972 June, First formal conference software testing, university of
North Carolina, Bill Hetzel.
1960
1957, Charles Baker distinguished debugging from testing
1950
2001-10-10
Not distinguished from debugging
PUM-I, Mariam Kamkar, IDA
41
And …
Testing can show the presence, but never the
absence of errors in software.
E. Dijkstra, 1969
2001-10-10
PUM-I, Mariam Kamkar, IDA
42
21