Software Testing
Objective
• Component/Unit/Module/Basic testing
• Integration testing
No issue is meaningful unless it can be put to
the test of decisive verification.
C.S. Lewis, 1934
PUM, Mariam Kamkar, IDA
1
Outline
Component/Unit/Module/Basic testing
Integration testing
Function testing
Performance testing
Acceptance testing
Installation testing
10/22/2002
PUM, Mariam Kamkar, IDA
Component code
•
•
•
•
•
•
10/22/2002
Component code
10/22/2002
Objective: to ensure that code implemented
the design properly.
2
PUM, Mariam Kamkar, IDA
3
Design Specification
Unit
test
Tested components
Integration
test
Unit
test
Tested components
Integrated modules
10/22/2002
PUM, Mariam Kamkar, IDA
4
1
Error, Fault, Failure
Types of Faults
•
•
•
•
•
•
•
Can lead to
Can lead to
Human error
Fault
Algorithmic
Computation & Precision
Documentation
Stress/Overload
Capacity/Boundary
Timing/Coordination
Throughout/Performance
• Recovery
• Hardware & System
Software
• Standards & Procedure
Failure
10/22/2002
PUM, Mariam Kamkar, IDA
5
10/22/2002
Debugging vs Testing
•
•
•
•
– fault identification
– fault correction / removal
PUM, Mariam Kamkar, IDA
7
Unit Testing
• Debugging: to find the bug
• Testing: to demonstrate the existence of a
fault
10/22/2002
PUM, Mariam Kamkar, IDA
6
Code Inspections
Code Walkthroughs
Open box testing
Black box testing
10/22/2002
PUM, Mariam Kamkar, IDA
8
2
Walkthroughs
Input
Failure?
design, code, chapter of user’s guide,…
• presenter
• coordinator
• secretary
• maintenance oracle
• standards bearer
• user representative
Test
Object
Output
Oracle
10/22/2002
PUM, Mariam Kamkar, IDA
9
Inspection
•
•
•
•
•
PUM, Mariam Kamkar, IDA
Discovery activity
overview
preparation
inspection
rework
follow-up
10/22/2002
10/22/2002
Faults found per thousand
lines of code
Requirements review
Design review
Code inspection
Integration test
Acceptance test
PUM, Mariam Kamkar, IDA
10
10/22/2002
11
PUM, Mariam Kamkar, IDA
2.5
5.0
10.0
3.0
2.0
12
3
Black box / Closed box testing
Equivalence partitioning
• incorrect or missing functions
• interface errors
• performance error
Valid inputs
Invalid inputs
input
outputs
output
10/22/2002
PUM, Mariam Kamkar, IDA
13
Black box testing
•
•
•
•
10/22/2002
PUM, Mariam Kamkar, IDA
15
Specification: the program accepts four to
eight inputs which are 5 digit integers
greater than 10000.
Equivalence partitioning
Boundary value analysis
Decision Table
Cause-Effect
input values
• Exhaustive testing
Less than 10000
Between 10000 and 99999
More than 99999
Number of input values
Less than 4
10/22/2002
PUM, Mariam Kamkar, IDA
14
10/22/2002
Between 4 and 8
More than 8
PUM, Mariam Kamkar, IDA
16
4
Guidelines
If an input condition specifies
• A range: one valid and two invalid
equivalence classes.
• A specific value: one valid and two invalid
equivalence classes.
• A member of a set: one valid and one
invalid equivalence classes.
• A boolean: one valid and one invalid class.
10/22/2002
PUM, Mariam Kamkar, IDA
Glass box testing!
White box testing!
Open box testing!
Clear box testing!
17
10/22/2002
Boundary value analysis
10000
9999
Less than 10000
10/22/2002
99998
Between 10000 and 99999
PUM, Mariam Kamkar, IDA
19
Glass box testing
•
•
•
•
•
99999
10001
PUM, Mariam Kamkar, IDA
100000
logical decision
loops
internal data structure
paths
...
More than 99999
18
10/22/2002
PUM, Mariam Kamkar, IDA
20
5
Begin
Statement Coverage
Begin
if ( y >= 0)
then y = 0;
abs = y;
end;
test case-1 (yes):
y >= 0
yes
y=0
abs = y
PUM, Mariam Kamkar, IDA
21
begin
y >= 0
no
10/22/2002
yes
y=0
abs = y
test case-1(yes):
input: y = 0
expected result: 0
actual result: ?
test case-2 (no):
input: y = -5
expected result: 5
actual result: ?
PUM, Mariam Kamkar, IDA
10/22/2002
z=fie (x,y)
x<10
&&
y>20
yes
z=foo (x,y)
test case-2 (no):
input: x = 12, y = 12
expected result: ...
actual result: ?
PUM, Mariam Kamkar, IDA
23
Condition / Branch Coverage
Branch Coverage
Begin
if ( y >= 0)
then y = 0;
abs = y;
end;
z = foo (x,y); else z =fie (x,y);
}
end;
test case-1 (yes):
input: x = -4, y = 30
expected result: ...
actual result: ?
input: y = 0
expected result: 0
actual result: ?
10/22/2002
no
if ( x < 10 && y > 20) {
begin
22
Begin
if ( x < 10 && y > 20) {
no
yes
x < 10
z = foo (x,y); else z =fie (x,y);
no
y > 20
}
yes
end;
z=fie (x,y)
z=foo (x,y)
x<10 y>20
----------------------------------test-case-1: t
t
test-case:2
t
f
test-case-3: f
t
test-case-4
f
f
10/22/2002
PUM, Mariam Kamkar, IDA
24
6
Path Coverage
no
yes
x <> 0
z = z-x
Data Flow Testing
x = 0, z = 12 (n,y)
x = 2, z = 6 (y,n)
z = sin(x)
yes
no
z > 10
z=0
x = 0, z = 7
x = 0, z = 13
x = 1, z = 5
x = 2, z = 15
z=z/x
10/22/2002
(n,n)
(n,y)
(y,n)
(y,y)
PUM, Mariam Kamkar, IDA
25
DEF(S) = {x | statement S contains a definition of variable x}
USE(S) = {x | statement S contains a use of variable x}
S1:
i = 1;
DEFS (S1) = {i}
(d1-i)
S2:
while (i <= n)
USE(S2) = {i, n}
(u1-i, u2-n)
definition-use chain (du chain) = [x, S, S’]
du-1: [i, S1, S2]
10/22/2002
Path with loops
d
c
e
a
c,b,d
d
e
10/22/2002
PUM, Mariam Kamkar, IDA
PUM, Mariam Kamkar, IDA
27
Data Flow testing
a
b
(d1-i), (u1-i)
26
s = 0;
i = 1;
while (i <= n)
{
s + = i;
i ++
}
print (s);
print (i);
print (n);
(d1-s)
(d2-i)
(u1-i, u2-n)
10/22/2002
PUM, Mariam Kamkar, IDA
du-s
(u3-s, u4-i, d3-s)
(u5-i, d4-i)
du-s
(u6-s)
(u7-i)
(u8-n)
28
7
Relative strengths of test strategies (B. Beizer 1990)
Data Flow Testing
s = 0;
i = 1;
while (i <= n)
{
s + = i;
i ++
}
print (s);
print (i);
print (n);
10/22/2002
(d1-s)
(d2-i)
(u1-i, u2-n)
All paths
dd: def-def
dk: def-kill
du: def-use
...
All definition-use paths
All uses
All predicate uses
All computational uses
(u6-s)
(u7-i)
(u8-n)
Branch
All definition
PUM, Mariam Kamkar, IDA
29
10/22/2002
Program Slicing
s = 0;
i = 1;
while (i <= n)
{
s + = i;
i ++
}
print (s);
print (i);
print (n);
10/22/2002
All predicate/
Some computational uses
All computational/
Some predicate uses
(u3-s, u4-i, d3-s)
(u5-i, d4-i)
Statement
PUM, Mariam Kamkar, IDA
31
Classes of Integration Testing
i = 1;
while (i <= n)
{
•
•
•
•
i ++
}
Bottom-up
Top-down
Big bang
Sandwich
print (i);
PUM, Mariam Kamkar, IDA
30
10/22/2002
PUM, Mariam Kamkar, IDA
32
8
Bottom-up
Components
driver
Tets ABCDEFG
Boundary conditions
independent paths
interface
...
Component
to be
tested
Test BEF
Test E
stub
stub
10/22/2002
Test C
Test F
Tets DG
Tets G
Test
cases
PUM, Mariam Kamkar, IDA
33
10/22/2002
PUM, Mariam Kamkar, IDA
35
Top-down
A
B
C
Test A
Test ABCD
D
Test ABCDEFG
E
10/22/2002
F
G
PUM, Mariam Kamkar, IDA
34
10/22/2002
PUM, Mariam Kamkar, IDA
36
9
Big-bang
Test A
Test Planing
Test B
•
•
•
•
•
•
Test C
Test D
Test ABCDEFG
Test E
Test F
Test G
10/22/2002
PUM, Mariam Kamkar, IDA
37
Establishing test objectives
Designing test cases
Writing test cases
Testing test cases
Executing tests
Evaluating test results
10/22/2002
PUM, Mariam Kamkar, IDA
39
Sandwich
Test A
Automated Testing Tools
• Code Analysis tools
Test B, E, F
– Static, Dynamic
Test D, G
• Test execution tools
Test ABCDEFG
– Capture-and-Replay
– Stubs & Drivers
Test E
Test F
• Test case generator
Test G
10/22/2002
PUM, Mariam Kamkar, IDA
38
10/22/2002
PUM, Mariam Kamkar, IDA
40
10
Termination Problem
How decide when to stop testing
Outline
• The main problem for managers!
•
•
•
•
•
•
• Termination takes place when
• resources (time & budget) are over
• found the seeded faults
• some coverage is reached
10/22/2002
PUM, Mariam Kamkar, IDA
41
Component/Unit/Module/Basic testing
Integration testing
Function testing
Performance testing
Acceptance testing
Installation testing
10/22/2002
PUM, Mariam Kamkar, IDA
43
Objective
•
•
•
•
Scaffolding
Oracle
What can be automated?
Test case
generation
Termination
Function testing
Performance testing
Acceptance testing
Installation testing
Objective: to ensure that the system does what
the customer wants it to do.
10/22/2002
PUM, Mariam Kamkar, IDA
42
10/22/2002
PUM, Mariam Kamkar, IDA
44
11
Tested components
Integration
test
Unit
test
Tested components
Integrated modules
10/22/2002
PUM, Mariam Kamkar, IDA
Functioning systems
Function
test
Acceptance
test
Customer requirements spec.
10/22/2002
• have a high probability of detecting a fault
• Use a test team independent of the designers and
programmers
• know the expected actions and output
• test both valid and invalid input
• never modify the system just to make testing
easier
• have stopping criteria
10/22/2002
PUM, Mariam Kamkar, IDA
47
Other software requirements
Accepted system
Integrated modules
System functional requirements
45
Performance
test
Installation
test
Verified validated
software
Component code
Component code
Function testing
functional requirements
Design Specification
Unit
test
System
In
Use!
Cause-Effect
(test case generation from req.)
Causes
C1: command is credit
C2: command is debit
C3: account number is
valid
C4: transaction amount is
valid
Effects
E1: print “invalid command”
E2: print “invalid account
number”
E3: print “debit amount not valid ”
E4: debit account print
E5: credit account print
User environment
PUM, Mariam Kamkar, IDA
46
10/22/2002
PUM, Mariam Kamkar, IDA
48
12
Acceptance testing
customers, users need
C2
and
C3
• Benchmark test: a set of special test cases
• Pilot test: everyday working
E3
– Alpha test: at the developer’s site, controlled
environment
– Beta test: at one or more customer site.
not
C4
• Parallel test: new system in parallel with
previous one
10/22/2002
PUM, Mariam Kamkar, IDA
49
10/22/2002
Performance testing
nonfunctional requirements
•
•
•
•
•
•
•
51
Installation testing
users site
Security
Accuracy
Speed
Recovery
Stress test
Volume test
…
10/22/2002
PUM, Mariam Kamkar, IDA
Acceptance test at developers site
Æ installation test at users site,
otherwise may not be needed!!
PUM, Mariam Kamkar, IDA
50
10/22/2002
PUM, Mariam Kamkar, IDA
52
13
Goals of software testing: Historical Evolution
Real life examples
Years
• First U.S. space mission to Venus failed.
(reason: missing comma in a Fortran do loop)
• December 1995: AA, Boeing 575, mountain crash
in Colombia, 159 killed. Incorrect one-letter
computer command (Cali, Bogota 132 miles in
opposite direction, have same coordinate code)
• June 1996: Ariane-5 space rocket, self-destruction,
$500 million.
(reason: reuse of software from Ariane-4 without
recommended testing).
10/22/2002
PUM, Mariam Kamkar, IDA
53
Prevent software faults
Measure SQA test objectives
1980
1979: Myers, “Art of software testing”
1970
1972 June, First formal conference software testing, university of
North Carolina, Bill Hetzel.
1960
1957, Charles Baker distinguished debugging from testing
1950
10/22/2002
Not distinguished from debugging
PUM, Mariam Kamkar, IDA
55
And …
• Australia: Man jailed because of computer glitch.
He was jailed for traffic fine although he had
actually paid it for 5 years ago.
• Dallas Prisoner released due to program design
flaw: He was temporary transferred from one prison
to another (witness). Computer gave him
“temporary assignment”.
PUM, Mariam Kamkar, IDA
cost
complexity
# applic.
Establish confidence
Real life examples
10/22/2002
Find faults 1981: Deutsch. Software project V and V
54
Testing can show the presence, but never the
absence of errors in software.
E. Dijkstra, 1969
10/22/2002
PUM, Mariam Kamkar, IDA
56
14
10/22/2002
PUM, Mariam Kamkar, IDA
57
15