Outline of the Lecture
• Integration Testing
–
–
–
–
Software Testing
• System Testing
• Acceptance Testing
• Distribution of Faults in a large Industrial Software System
(ISSTA 2002)
Lecture Notes 3 (of 4)
CUGS, SE, Mariam Kamkar, IDA, LiU
1
Unit & Integration Testing
January, 2007
Component code
Objective: to ensure that code implemented
the design properly.
Design Specification
Code = System
CUGS, SE, Mariam Kamkar, IDA, LiU
January, 2007
Component code
January, 2007
Top-down
Bottom-up
Big-bang
Sandwich
CUGS, SE, Mariam Kamkar, IDA, LiU
2
Design Specification
Unit
test
Tested components
Integration
test
Unit
test
Tested components
Integrated modules
3
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
4
5
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
6
Components
driver
Component
to be
tested
stub
January, 2007
stub
CUGS, SE, Mariam Kamkar, IDA, LiU
Boundary conditions
independent paths
interface
...
Test
cases
1
Top-down
Stubs for four components
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
7
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
8
A
B
E
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
9
January, 2007
A
B
E
C
F
C
F
D
G
CUGS, SE, Mariam Kamkar, IDA, LiU
10
A
D
Top-down
B
G
E
C
F
D
Top-down
G
Test A
Test ABCD
Test ABCDG
Test ABCDEFG
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
11
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
12
2
Modified Top-down
Modified Top-down
Test B
Test A
Test C
Test D
Test
A, B, C,
E
F
G
Test
A, B, C, D,
E, F, G
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
13
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
14
Bottom-up
Driver for five components
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
15
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
16
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
17
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
18
3
A
B
E
C
F
Bottom-up
D
G
A
B
E
C
D
F
January, 2007
G
CUGS, SE, Mariam Kamkar, IDA, LiU
19
January, 2007
A
B
E
C
F
CUGS, SE, Mariam Kamkar, IDA, LiU
20
A
Bottom-up
D
B
G
E
C
F
Big-bang
D
G
Test A
Test E
Test F
Test B
Tets G
Test C
Test BEF
Test C
Tets DG
Test D
Test ABCDEFG
Test E
Tets ABCDEFG
Test F
Test G
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
21
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
22
A
B
E
C
F
D
A
B
E
January, 2007
C
F
CUGS, SE, Mariam Kamkar, IDA, LiU
Sandwich
G
Target level B,C,D
D
G
23
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
24
4
A
B
E
C
F
A
Sandwich
D
B
G
E
Test E
C
F
Modified Sandwich
D
G
Target level B,C,D
Test B, E, F
Test F
Test ABCDEFG
Test G
Test D, G
Test A
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
25
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
26
A
B
E
C
F
Modified Sandwich
D
Comparison of Integration Strategies
G
Test E
Top-down
Modified
Top-Down
Integration
Early
Early
Early
Time to
basic
working
program
Early
Early
Driver
needed
No
Stubs
needed
Yes
Test B
Test B, E, F
Bottom-up Big-bang
Sandwich
Modified
Sandwich
Late
Early
Early
Late
Late
Early
Early
Yes
Yes
Yes
Yes
Yes
Yes
No
Yes
Yes
Yes
Test F
Test D
Test ABCDEFG
Test G
Test D, G
Test A
Test C
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
27
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
28
System Testing
• System Testing Steps
– Function testing / Thread testing
– Performance testing
– Acceptance testing
– Installation testing
Objective: to ensure that the system does what
the customer wants it to do.
Customer
• Test Automation
• Termination Problem
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
Developer
Requirements definition
Requirements specification
29
January, 2007
Functional requirements
Nonfunctional requirements
CUGS, SE, Mariam Kamkar, IDA, LiU
30
5
Unit
test
Function
test
Tested components
Integrated modules
Acceptance
test
Customer requirements spec.
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
31
Function testing/Thread testing
January, 2007
Performance
test
Verified validated
software
Integration
test
Functioning systems
Tested components
Other software requirements
Accepted system
Unit
test
Integrated modules
Component code
Component code
System functional requirements
Design Specification
Installation
test
System
In
Use!
User environment
CUGS, SE, Mariam Kamkar, IDA, LiU
32
A function test checks that the integrated system
performs its function as specified in the
requirement
(testing one function at a time)
functional requirements
• Threads:
–
–
–
–
–
A scenario of normal usage
A stimulus/response pair
Behavior that results from a sequence of system-level input
An interleaved sequence of port input and output events
A sequence of atomic system functions (ASF)
• Guidelines
• ASF: an atomic system function is an action that is
observable at the system level in terms of port input and
output events
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
33
Cause-and-Effect-Graph
(test case generation from req.)
– use a test team independent of the designers and
programmers
– know the expected actions and output
– test both valid and invalid input
– never modify the system just to make testing easier
– have stopping criteria
January, 2007
a
• causes: inputs
• effects: outputs and transformations
• causes-and-effect graph:
b
c
Identity: if a then b
b
And: if (a and b) then c
a
– boolean graph reflecting causes and effects
relationships
– is a formal language into which a natural language
specification is translated
b
c
CUGS, SE, Mariam Kamkar, IDA, LiU
34
Basic cause-effect graph symbols
a
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
35
January, 2007
d
a
b
Identity: if (not a) then b
Or: if (a or b or c) then d
CUGS, SE, Mariam Kamkar, IDA, LiU
36
6
Sample cause-effect graph
Specification: the character in column 1 must be an “A” or a “B”. The
character in column 2 must be a digit. In this situation, the file update is
made. If the first character is incorrect, message X12 is issued. If the second
character is not a digit, message X13 is issued.
1
Intermediate node
E2
11
Causes
Effects
C1: character in column 1 is “A”
C2: character in column 1 is “B”
C3: character in column 2 a digit
E1: update made
E2: message X12 is issued
E3:message X13 is issued
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
37
Constraint symbols
a
a
I
I cause-constraint:
at least one of a, b and c
must always be true
E
b
January, 2007
a
3
E3
CUGS, SE, Mariam Kamkar, IDA, LiU
b
1
E
a
a
E1
38
Sample cause-effect graph with “exclusive” constraint
c
E cause-constraint: at most one of
a or b can be true
2
11: Intermediate node
E2
11
2
E1
3
E3
M
R
O
O cause-constraint: one, and only one,
of a and b must be true
January, 2007
b
b
b
R cause-constraint:
for a to be true,
b must be true
M effect-constraint:
If effect a to is true,
Effect b is forced to be false
CUGS, SE, Mariam Kamkar, IDA, LiU
39
January, 2007
Test 1
Test 2
Test 3
Test 4
1
0
0
X
Cause 2
0
1
0
X
Cause 3
1
1
X
0
Effect E1
1
1
0
0
Effect E2
0
0
1
0
Effect E3
0
0
0
1
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
40
Performance Testing
nonfunctional requirements
Decision table for cause-and effect graph
Cause 1
CUGS, SE, Mariam Kamkar, IDA, LiU
•
•
•
•
•
•
•
41
Stress tests
Volume tests
Configuration tests
Compatibility tests
Regression tests
Security tests
Timing tests
January, 2007
•
•
•
•
•
•
Environment tests
Quality tests
Recovery tests
Maintenance tests
Documentation tests
Human factors tests /
usability tests
CUGS, SE, Mariam Kamkar, IDA, LiU
42
7
Acceptance Testing
customers, users need
Installation Testing
users site
• Benchmark test: a set of special test cases
Acceptance test at developers site
Æ installation test at users site,
otherwise may not be needed!!
• Pilot test: everyday working
– Alpha test: at the developer’s site, controlled
environment
– Beta test: at one or more customer site.
• Parallel test: new system in parallel with previous
one
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
43
Test Automation
• Automating parts of the testing process can provide
long-term benefits to organization, such as:
– reducing the amount of time it takes to execute a suite of
tests
– reducing the tester’s involvement in executing tests
– facilitating regression testing
– allowing for the simulation of hundreds of users
– avoiding human mistakes by having tools control
repetitive and tedious tasks
• Test automation refers to two key testing activities:
– Executing the tests
– Evaluating the output
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
44
Automated Testing Tools
• Code Analysis tools
– Static, Dynamic
• Test execution tools
– Capture-and-Replay
– Stubs & Drivers
– Comparators
• Test case generator
45
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
46
Termination Problem
How decide when to stop testing
• The main problem for managers!
• Termination takes place when
• resources (time & budget) are over
• found the seeded faults
• some coverage is reached
January, 2007
CUGS, SE, Mariam Kamkar, IDA, LiU
47
8