Course Notes Set 10:
Testing Strategies
Computer Science and Software Engineering
Auburn University
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-0
Strategic Approach to
Testing
• Testing begins at the unit level and
works toward integrating the
entire system
• Various techniques for testing are
appropriate at different times
• Conducted by the developer and
by independent test groups
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-1
Testing Strategies
Requirements
Specification
System
Testing
Preliminary
Design
Integration
Testing
Detailed
Design
Unit Testing
Coding
[Adapted from Software Testing A Craftman’s Approach, by Jorgensen, CRC Press, 1995]
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-2
A Testing Strategy
System engineering
Requirements
Design
Code
Unit Test
Integration Test
Validation Test
System Test
[Adapted from Software Engineering 4th Ed, by Pressman, McGraw-Hill, 1997]
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-3
Integration Testing
• After individual components have
passed unit testing, they are
merged together to form
subsystems and ultimately one
complete system.
• Integration testing is the process
of exercising this “hierarchically
accumulating” system.
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-4
Integration Testing
• We will (normally) view the system as a
hierarchy of components.
– Call graph
– Structure chart
– Design tree
• Integration testing can begin at the top
of this hierarchy and work downward,
or it can begin at the bottom of the
hierarchy and work upwards.
• It can also employ a combination of
these two approaches.
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-5
Example Component Hierarchy
A
B
E
C
F
D
G
[Figure and associated examples adapted from Pfleeger 2001]
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-6
Integration Testing Strategies
• Big-bang integration
• Bottom-up Integration
• Top-down Integration
• Sandwich Integration
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-7
Big-bang Integration
• All components are tested in isolation.
• Then, the entire system is integrated in
one step and testing occurs at the top
level.
• Often used (perhaps wrongly),
particularly for small systems.
• Does not scale.
• Difficult or impossible to isolate faults.
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-8
Big-bang Integration
Test
A
Test
B
Test
A,B,C,
D,E,F,
G
Test
C
A
Test
D
B
Test
E
Auburn University
Computer Science and Software Engineering
Test
F
E
C
F
D
G
COMP 6710 Course Notes Slide 10-9
Bottom-up Integration
• Test each unit at the bottom of the
hierarchy first.
• Then, test the components that call the
previously tested ones (one layer up in
the hierarchy).
• Repeat until all components have been
tested.
• Component drivers are used to do the
testing.
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-10
Bottom-up Integration
Test
E
Test
B,E,F
Test
F
Test
G
Test
C
Test
A,B,C,
D,E,F,
G
Test
D,G
A
B
E
Auburn University
Computer Science and Software Engineering
C
F
D
G
COMP 6710 Course Notes Slide 10-11
Bottom-up Integration
• The manner in which the software was designed will
influence the appropriateness of bottom-up integration.
• While it is normally appropriate for object-oriented
systems, bottom-up integration has disadvantages for
functionally-decomposed systems:
– Top-level components are usually the most important, but
the last to be tested.
– The upper levels are more general while the lower levels
are more specific. Thus, by testing from the bottom up the
discover of major faults can be delayed.
– Top-level faults are more likely to reflect design errors,
which should obviously be discovered as soon as possible
and are likely to have wide-ranging consequences.
– In timing-based systems, the timing control is usually in
the top-level components.
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-12
Top-down Integration
• The top-level component is tested in
isolation.
• Then, all the components called by the
one just tested are combined and
tested as a subsytem.
• This is repeated until all components
have been integrated and tested.
• Stubs are used to fill in for components
that are called but are not yet included
in the testing.
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-13
Top-down Integration
Test
A
Test
A,B,
C,D
Test
A,B,C,
D,E,F,
G
A
B
E
Auburn University
Computer Science and Software Engineering
C
F
D
G
COMP 6710 Course Notes Slide 10-14
Top-down Integration
• Again, the design of the system influences the
appropriateness of the integration strategy.
• Top-down integration is obviously well-suited
to systems that have been created through
top-down design.
– When major system functions are localized to
components, top-down integration allows the testing
to isolate one function at a time and follow its control
flow from the highest levels of abstraction to the
lowest levels.
– Also, design problems show up earlier rather than
later.
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-15
Top-down Integration
• A major disadvantage is the need of stubs.
– Writing stubs can be complex since they must function
under the same conditions as their real counterpart.
– The correctness of the stub will influence the validity of the
test.
– A large number of stubs could be required, particularly
when there are a large number of general-purpose
components in the lowest layer.
• Another criticism is the lack of individual testing on
interior components.
– To address this concern, a modified top-down integration
strategy can be used. Instead of incorporating an entire
layer at once, each component in a given layer is tested
individually before the integration of that layer occurs.
– This introduces another problem, however: Now both
stubs and component drivers are needed.
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-16
Modified Top-down Integration
Test
B
Test
A
Test
C
Test
E
Test
A,B,
C,D
Test
D
Test
F
Test
A,B,C,
D,E,F,
G
Test
G
A
B
E
Auburn University
Computer Science and Software Engineering
C
F
D
G
COMP 6710 Course Notes Slide 10-17
Sandwich Integration
• Top-down and bottom-up can be combined
into what Myers calls “sandwich integration.”
• The system is viewed as being composed of
three major levels: the target layer in the
middle, the layers above the target, and the
layers below the target.
• A top-down approach is used for the top level
while a bottom-up approach is used for the
bottom level. Testing converges on the target
level.
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-18
Sandwich Integration
Test
E
Test
B,E,F
Test
F
Test
D,G
Test
A,B,C,
D,E,F,
G
Test
G
A
B
Test
A
E
Auburn University
Computer Science and Software Engineering
C
F
D
G
COMP 6710 Course Notes Slide 10-19
Measures for Integration Testing
• Recall v(G) is an upper bound on the
number of independent/basis paths in a
source module
• Similarly, we would like to limit the
number of subtrees in a structure chart
or call graph
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-20
Subtrees in Architecture vs.
Paths in Units
• A call graph (or equivalent) architectural representation
corresponds to a design tree representation, just as the
source code for a unit corresponds to a flowgraph.
• Executing the design tree means it is entered at the
root, modules in the subtrees are executed, and it
eventually exits at the root.
• Just as the program can have a finite (if it halts), but
overwhelming, number of paths, a design tree can have
an inordinately large number of subtrees as a result of
selection and iteration.
• We need a measure for design trees that is the analog
of the basis set of independent paths for units.
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-21
Design Tree : Complexity of 1
1
2
5
10
3
6
7
4
9
8
12
11
13
14
[Adapted from McCabe and Butler, “Design Complexity Measurement and Testing,” CACM 32(12)]
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-22
Design Tree : Complexity > 1
1
2
5
10
3
6
7
4
9
8
12
11
13
14
[Adapted from McCabe and Butler, “Design Complexity Measurement and Testing,” CACM 32(12)]
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-23
Design Tree Notation
M
M
B
A
M
B
A
B
A
Possible Paths:
Possible Paths:
Possible Paths:
Neither
A
Neither
A
B
A
B
B
AB
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-24
Subtrees vs. Paths
M’s Flowgraph
Design Tree C
E
M
A
B
A
B
X
[Adapted from McCabe and Butler, “Design Complexity Measurement and Testing,” CACM 32(12)]
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-25
Flowgraph Information
• Flowgraph symbols
– A black dot is a call to a subordinate module
– A white dot is a sequential statement (or a collection
of sequential statements)
• Rules for reduction
– Sequential black dot : may not be reduced
– Sequential white dot : a sequential node may be
reduced to a single edge
– Repetitive white dots : a logical repetition without a
black dot can be reduced to a single node
– Conditional white dots : a logical decision with two
paths without a black dot may be reduced to one
path
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-26
Reduction Rules
1. Sequential Black Dot
2. Sequential White Dot
3. Repetitive White Dot
4. Conditional or Looping White Dot Decisions
or
[Adapted from McCabe and Butler, “Design Complexity Measurement and Testing,” CACM 32(12)]
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-27
Example Reduction
1
1
1
3
2
3
2
3
2
5
4
4
4
7
6
6
6
8
9
8
9
8
9
[Adapted from McCabe and Butler, “Design Complexity Measurement and Testing,” CACM 32(12)]
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-28
Example Reduction
1
1
1
3
1
3
3
4
4
4
8
8
8
3
4
6
9
9
9
9
[Adapted from McCabe and Butler, “Design Complexity Measurement and Testing,” CACM 32(12)]
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-29
Architectural Design
Measures
• Number of subtrees
– The set of all subtrees is not particularly useful, but a basis set
would be.
• Module Design Complexity : iv(G)
– The cyclomatic complexity of the reduced flowgraph of the module
• Design Complexity: S0
– S0 of a module M is
S0 =
iv(Gj)
j
D
where D is the set of descendants of M unioned with M
– Note: If a module is called several times, it is added only once
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-30
Design Complexity Example
S0=11
iv=3
S0=3
S0=4
S0=1
iv=2
iv=2
iv=1
S0=1
S0=1
S0=1
iv=1
iv=1
iv=1
[Adapted from McCabe and Butler, “Design Complexity Measurement and Testing,” CACM 32(12)]
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-31
Design Complexity Example
M
S0=9
iv=2
A
B
S0=6
iv=2
S0=1
iv=1
S0(A) = iv(A) + iv(C) + iv(D) + iv(E)
C
S0=4
iv=2
D
S0=1
iv=1
E
S0=1
iv=1
[Adapted from McCabe and Butler, “Design Complexity Measurement and Testing,” CACM 32(12)]
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-32
Architectural Design
Measures
• Integration Complexity : S1
– Measure of the number of integration
tests required
– S1 = S0 - n + 1
• S0 is the design complexity
• n is the number of modules
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-33
Integration Complexity
S1=5
M
S1=5
S0=9
N
iv=3
S0=9
iv=3
A
S0=5
B
iv=1
S
iv=3
S0=5
T
iv=1
iv=3
C
iv=1
D
iv=1
U
iv=1
V
iv=1
[Adapted from McCabe and Butler, “Design Complexity Measurement and Testing,” CACM 32(12)]
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-34
Integrated Properties of M
and N
M
S0=18
iv=3
A
S0=5
B
iv=3
C
iv=1
Integration Point
S0=10
iv=1
D
iv=1
N
S0=9
iv=3
S
S0=5
T
iv=3
U
Auburn University
Computer Science and Software Engineering
iv=1
V
S0=1
iv=1
iv=1
COMP 6710 Course Notes Slide 10-35
Integration Testing
• Module integration testing
– Scope is a module and its immediate
subordinates
– Testing Steps
• Apply reduction rules to the module
• Cyclomatic complexity of the subalgorithm is the
module design complexity of the original
algorithm. This determines the number of
required tests.
• The baseline method applied to the subalgorithm
yields the design subtrees and the module
integration tests
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-36
Integration Testing
• Design integration testing
– Derived from integration complexity, which quantifies a basis set
of integration tests
– Testing steps
•
•
•
•
Calculate iv and S0 for each module
Calculate S1 for the top module (number of basis subtrees required)
Build a path matrix (S1 x n) to establish the basis set of subtrees
Identify and label each predicate in the design tree and place those
labels above each column of the path matrix corresponding to the
module it influences
• Apply the baseline method to the design to complete the matrix (1 :
the module is executed; 0 : the module is not executed)
• Identify the subtrees in the matrix and the conditions which derive the
subtrees
• Build corresponding test cases for each subtree
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-37
Design Integration Example
S0=8
M
iv=2
P1
A
S0=3
S0=4
B
iv=1
iv=2
P2
C
S0=1
iv=1
D
S0=1
E
iv=1
S0=1
iv=1
S1 = S 0 - n + 1
P1 : condition W = X
=8-6+1
P2 : condition Y = Z
=3
[Adapted from McCabe and Butler, “Design Complexity Measurement and Testing,” CACM 32(12)]
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-38
Integration Path Test Matrix
P1
P2
Path
M
A
B
C
D
E
Test Condition
Expected Execution
Baseline
1
1
1
1
1
1
W=X and Y=Z
Invoke A & E
Subtree2
1
0
1
0
1
1
WX and Y=Z
Invoke E, not A
Subtree3
1
1
1
1
1
0
W=X and YZ
Invoke A, not E
Relative
Frequency
3
2
3
2
3
2
[Adapted from McCabe and Butler, “Design Complexity Measurement and Testing,” CACM 32(12)]
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-39
Integration Path Test Matrix
P1
P2
Path
M
A
B
C
D
E
Baseline
1
0
1
0
1
1
Subtree2
1
1
1
1
1
1
Subtree3
1
0
1
0
1
0
Relative Frequency
3
1
3
1
3
2
[Adapted from McCabe and Butler, “Design Complexity Measurement and Testing,” CACM 32(12)]
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-40
Example
with ModuleA, ModuleB;
use ModuleA, ModuleB;
procedure Main is
begin
S1;
while CM loop
ProcA;
ProcB;
end loop;
end Main;
package body ModuleC is
procedure ProcC is
begin
S1;
if CC then
S2;
else
S3;
end if;
end ProcC;
begin
null;
end ModuleC;
with ModuleC;
use ModuleC;
package body ModuleA is
procedure ProcA is
begin
S1;
if CA then
S1;
else
ProcC;
end if;
end ProcA;
begin
null;
end ModuleA;
with ModuleC;
use ModuleC;
package body ModuleB is
procedure ProcB is
begin
S1;
if CB then
ProcC;
else
S2;
end if;
if CB2 then
S3;
end if;
end ProcB;
begin
null;
end ModuleB;
What is an appropriate number of integration test cases
and what are those cases?
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-41
Example
Main
ProcA
ProcB
ProcC
[Adapted from Watson and McCabe, “Structured Testing: A Testing Methodology Using the Cyclomatic Complexity Metric,” NIST 500-235, 1996]
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-42
System Testing
• The primary objective of unit and
integration testing is to ensure that the
design has been implemented properly;
that is, that the programmers wrote
code to do what the designers intended.
(Verification)
• The primary objective of system testing
is very different: We want to ensure
that the system does what the
customer wants it to do. (Validation)
[Some notes adapted from Pfleeger 2001]
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-43
System Testing
• Steps in system testing
–
–
–
–
Function Testing
Performance Testing
Acceptance Testing
Installation Testing
Function
Test
Integrated
Modules
Performance
Test
Functioning
System
Auburn University
Computer Science and Software Engineering
Acceptance
Test
Verified, Validated
Software
Installation
Test
Accepted
System
Delivered
System
COMP 6710 Course Notes Slide 10-44
Function Testing
• Checks that an integrated system
performs its functions as specified in
the requirements.
• Common functional testing techniques
(cause-effect graphs, boundary value
analysis, etc.) used here.
• View the entire system as a black box.
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-45
Performance Testing
• Compares the behavior of the functionally
verified system to nonfunctional requirements.
• System performance is measured against the
performance objectives set by the customer
and expressed as nonfunctional requirements.
• This may involve hardware engineers.
• Since this stage and the previous constitute a
complete review of requirements, the software
is now considered validated.
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-46
Types of Performance Tests
•
•
•
•
•
•
•
•
•
•
Stress tests
Configuration tests
Legacy Regression tests
Security tests
Timing tests
Environmental tests
Quality tests
Recovery tests
Documentation tests
Usability tests
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-47
Acceptance Testing
• Customer now leads testing and defines
the cases to test.
• The purpose of acceptance testing is to
allow the customer and users to
determine if the system that was built
actually meets their needs and
expectations.
• Many times, the customer
representative involved in requirements
gathering will specify the acceptance
tests.
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-48
Types of Acceptance Tests
• Benchmark tests
– Subset of users operate the system under a set of
predefined test cases.
• Pilot tests
– Subset of users operate the system under normal or
“everyday” situations.
– Alpha testing if done at developer’s site
– Beta testing if done at customer’s site
• Parallel tests
– New system operates in parallel with the previous
version. Users gradually transition to the new
system.
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-49
Installation Testing
• Last stage of testing
• May not be needed if acceptance testing
was performed at the customer’s site.
• The system is installed in the
environment in which it will be used,
and we verify that it works in the field
as it did when tested previously.
Auburn University
Computer Science and Software Engineering
COMP 6710 Course Notes Slide 10-50