Software Testing A Software Life-cycle Model January 2008
advertisement
A Software Life-cycle Model
Which part will we talk about today?
2
Mariam Kamkar
Maintenance
marka@ida.liu.se
Validate Requirements, Verify Specification
Acceptance Test
Requirements
(Release testing)
Software Testing
Verify System Design
System Design
System Testing
(Architecture,
High-level Design)
(Integration testing of modules)
Software Engineering
Module Design
January 2008
(Program Design,
Detailed Design)
Verify Module Design
Module Testing
(Integration testing of units)
Verify Implementation
Implementation
of Units (classes, procedures,
functions)
Mariam Kamkar
Department of Computer and Information Science
Linköping University, Sweden
marka@ida.liu.se
Project Management, Software Quality Assurance (SQA), Supporting Tools, Education
Part I
Introduction,
Testing Process
Agenda - What will you learn today?
Unit testing
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
3
4
Mariam Kamkar
marka@ida.liu.se
Part I
Part II
Introduction, Testing Process
Unit Testing:
Black-box, White-box testing
Mariam Kamkar
marka@ida.liu.se
Part I
Introduction, Testing Process
Part III
Part IV
Module Testing
(Integration testing)
System Testing
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
5
Triangle program (simple version)
Part I
Introduction,
Testing Process
Mariam Kamkar
marka@ida.liu.se
The program accepts three integers, a, b, and c as input. The three values are
interpreted as representing the lengths of sides of a triangle. The program
prints a message that states whether the triangle is scalene (oregelbunden),
isosceles (likbent) or equilateral (liksidig).
On a sheet of paper, write a set of test cases (i.e., specific sets of data) that
you feel would adequately test this program.
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Part III
Module Testing
(Integration testing)
Part IV
System Testing
6
Testing a ballpoint pen
triangle problem is the most widely used example in software testing literature.
Test case
a
b
c
Expected output
__________________________________________________________________
1
3
3
4
isosceles (likbent)
2
?
?
?
?
…
Part II
Unit Testing:
Black-box, White-box
Mariam Kamkar
marka@ida.liu.se
Does the pen write in the right color,
with the right line thickness?
Is the logo on the pen according to
company standards?
Is it safe to chew on the pen?
Does the click-mechanism still work
after 100 000 clicks?
Does it still write after a car has run
over it?
What is expected from this pen?
Intended use!!
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
1
7
Mariam Kamkar
marka@ida.liu.se
Customer
8
Error, Fault, Failure
Mariam Kamkar
marka@ida.liu.se
Developer
Can lead to
Functional requirements
Nonfunctional requirements
Requirements definition
Requirements specification
Can lead to
Human error (Mistake, Bug)
Fault (Defect, Bug)
Part II
Unit Testing:
Black-box, White-box
Failure
Design Specification
Code = System
Part I
Introduction,
Testing Process
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Types of Faults
9
Mariam Kamkar
marka@ida.liu.se
(dep. on org. IBM, HP)
Part II
Unit Testing:
Black-box, White-box
Part I
Introduction,
Testing Process
Part III
Module Testing
(Integration testing)
Part IV
System Testing
10
A Testing Life Cycle
Mariam Kamkar
marka@ida.liu.se
Error
Error
Algorithmic: division by zero
Computation & Precision: order of op
Documentation: doc - code
Stress/Overload: data-str size ( dimensions of tables, size of buffers)
Capacity/Boundary: x devices, y parallel tasks, z interrupts
Timing/Coordination: real-time systems
Throughout/Performance: speed in req.
Recovery: power failure
Hardware & System Software: modem
Standards & Procedure: organizational standard; difficult for
programmers to follow each other.
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Black-box: a strategy in which
testing is based on
requirements and specifications.
Fix
Error
Fault
Design
Fault
Coding
Incident
Fault
Putting Bugs IN
Development phases
Part I
Introduction,
Testing Process
11
Mariam Kamkar
marka@ida.liu.se
Fault
Resolutio
n
Fault
Isolation
Error
Part IV
System Testing
Types (strategy) of testing
Requirements
Specification
Fault
Classification
Testing
Getting Bugs OUT
Finding Bugs
Testing phase
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
12
Contents of a Test Case
Mariam Kamkar
marka@ida.liu.se
Black-box testing
Specification
Program
"Boilerplate": author, date, purpose, test case ID
Pre-conditions (including environment)
White-box: a strategy in which
testing is based on internal
paths, structure, and
implementation.
Inputs
Test cases
Expected Outputs
Gray-box: peek into the “box”
just long enough to understand
how it has been implemented.
White-box testing
Specification
Program
Observed Outputs
Pass/Fail
Test cases
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
2
13
Mariam Kamkar
marka@ida.liu.se
14
Unit & Integration Testing
Mariam Kamkar
marka@ida.liu.se
Objective: to ensure that code implemented the design
properly.
Part II
Unit Testing:
Black-box, White-box testing
Design Specification
Code = System
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
15
16
Component code
Mariam Kamkar
marka@ida.liu.se
Mariam Kamkar
marka@ida.liu.se
Design Specification
Failure?
Input
Unit
test
Tested components
Test
Object
Component code
Integration
test
Unit
test
Tested components
Output
Oracle
Integrated modules
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Part I
Introduction,
Testing Process
17
Two Types of Oracles
Mariam Kamkar
marka@ida.liu.se
Human: an expert that can examine an input and its
associated output and determine whether the
program delivered the correct output for this
particular input.
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
18
Unit Testing
Mariam Kamkar
marka@ida.liu.se
Black-box Testing
White-box Testing
Automated: a system capable of performing the
above task.
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
3
19
Black-box Testing
1.
2.
3.
4.
Mariam Kamkar
marka@ida.liu.se
20
Black-box / Closed-box Testing
Mariam Kamkar
marka@ida.liu.se
incorrect or missing functions
interface errors
performance error
Exhaustive testing
Equivalence class testing (Equivalence Partitioning)
Boundary value analysis
Decision table testing
input
output
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Part I
Introduction,
Testing Process
21
Block-Box Testing Techniques
Mariam Kamkar
marka@ida.liu.se
Definition: a strategy in which testing is based on requirements and
specifications.
Applicability: all levels of system development
y Unit
y Integration
y System
y Acceptance
Disadvantages: never be sure of how much of the system under test
(SUT) has been tested.
Advantages: directs tester to choose subsets to tests that are both
efficient and effective in finding defects.
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part III
Module Testing
(Integration testing)
Part IV
System Testing
22
1. Exhaustive testing
Mariam Kamkar
marka@ida.liu.se
Definition: testing with every member of the input
value space.
Input value space: the set of all possible input values
to the program.
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part IV
System Testing
Part I
Introduction,
Testing Process
23
2. Equivalence Class Testing
Mariam Kamkar
marka@ida.liu.se
Equivalence Class (EC) testing is a technique used to reduce
the number of test cases to a manageable level while still
maintaining reasonable test coverage.
Each EC consists of a set of data that is treated the same by the
module or that should produce the same result. Any data value
within a class is equivalent, in terms of testing, to any other
value.
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Identifying the Equivalence Classes
24
Mariam Kamkar
marka@ida.liu.se
Taking each input condition (usually a sentence or phrase in the
specification) and partitioning it into two or more groups:
y Input condition
• range of values x: 1-50
y Valid equivalence class
• ?<x<?
y Invalid equivalence classes
• x<?
• x>?
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
4
25
Guidelines
1.
2.
3.
4.
5.
Mariam Kamkar
marka@ida.liu.se
If an input condition specifies a range of values; identify one valid EC
and two invalid EC.
If an input condition specifies the number (e.g., one through 6
owners can be listed for the automobile); identify one valid EC and
two invalid EC (- no owners; - more than 6 owners).
If an input condition specifies a set of input values and there is
reason to believe that each is handled differently by the program;
identify a valid EC for each and one invalid EC.
If an input condition specifies a “must be” situation (e.g., first
character of the identifier must be a letter); identify one valid EC (it is
a letter) and one invalid EC (it is not a letter)
If there is any reason to believe that elements in an EC are not
handled in an identical manner by the program, split the equivalence
class into smaller equivalence classes.
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Mariam Kamkar
marka@ida.liu.se
Most suited to systems in which much of the input data takes on values
within ranges or within sets.
It makes the assumption that data in the same EC is, in fact, processed
in the same way by the system. The simplest way to validate this
assumption is to ask the programmer about their implementation.
EC testing is equally applicable at the unit, integration, system, and
acceptance test levels. All it requires are inputs or outputs that can be
partitioned based on the system’s requirements.
Mariam Kamkar
marka@ida.liu.se
1.
Assign a unique number to each EC.
2.
Until all valid ECs have been covered by test cases, write a new test
case covering as many of the uncovered valid ECs as possible.
3.
Until all invalid ECs have been covered by test cases, write a test
case that cover one, and only one, of the uncovered invalid ECs.
Part I
Introduction,
Testing Process
27
Applicability and Limitations
26
Identifying the Test Cases
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
28
Equivalence partitioning
Invalid inputs
Mariam Kamkar
marka@ida.liu.se
Valid inputs
outputs
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Part I
Introduction,
Testing Process
29
Mariam Kamkar
marka@ida.liu.se
Specification: the program accepts four to eight
inputs which are 5 digit integers greater than 10000.
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
30
3. Boundary Value Testing
Mariam Kamkar
marka@ida.liu.se
Boundary value testing focuses on the boundaries
simply because that is where so many defects hide.
The defects can be in the requirements or in the
code.
The most efficient way of finding such defects, either
in the requirements or the code, is through inspection
(Software Inspection, Gilb and Graham’s book).
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
5
31
Technique
Mariam Kamkar
marka@ida.liu.se
1.
Identify the ECs.
2.
Identify the boundaries of each EC.
3.
Create test cases for each boundary value by
choosing one point on the boundary, one point just
below the boundary, and one point just above the
boundary.
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Less than 10000
Part IV
System Testing
Part I
Introduction,
Testing Process
33
Applicability and Limitations
Mariam Kamkar
marka@ida.liu.se
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Mariam Kamkar
marka@ida.liu.se
Between 10000 and 99999
Part II
Unit Testing:
Black-box, White-box
More than 99999
Part III
Module Testing
(Integration testing)
Part IV
System Testing
34
4. Decision Table Testing
Mariam Kamkar
marka@ida.liu.se
Decision tables are an excellent tool to capture certain kinds of
system requirements and to document internal system design.
They are used to record complex business rules that a system
must implement.
Boundary value testing is equally applicable at the
unit, integration, system, and acceptance test levels.
All it requires are inputs that can be partitioned and
boundaries that can be identified based on the
system’s requirements.
Part I
Introduction,
Testing Process
32
Boundary value analysis
In addition, they can serve as a guide to creating test cases.
Part IV
System Testing
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
35
Technique
36
Mariam Kamkar
marka@ida.liu.se
Mariam Kamkar
marka@ida.liu.se
Rule 1
A decision table
with ”don’t care” entry
Rule 2
Rules
3,4
Rule 5
Rule 6
Rules
7,8
C1
T
T
T
F
F
F
C2
T
T
F
T
T
F
C3
T
F
_
T
F
_
Conditions
A1
X
X
X
Condition-1
A2
X
Condition-2
A3
X
X
The general format of a decision table:
Rule 1
Rule 2
…
Rule P
…
X
X
A4
X
Condition-m
Actions
Action-1
• _ : “don’t care” entry. The don’t care entry has two major interpretations: the condition is
irrelevant, or the condition does not apply. Sometimes the “n/a” symbol for this latter
interpretation..
Action-2
…
Action-n
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
6
37
Technique (cont.)
Mariam Kamkar
marka@ida.liu.se
Test Case 2
…
Mariam Kamkar
marka@ida.liu.se
The program accepts three integers, a, b, and c as input. The
three values are interpreted as representing the lengths of sides
of a triangle. The integers a, b, and c must satisfy the following
conditions:
A decision table converted to a test case table:
Test Case 1
38
Triangle program (new conditions)
Test Case P
Inputs
Condition-1
C1:
C2:
C3:
C4:
C5:
C6:
Condition-2
…
Condition-m
Expected
Results
Action-1
1 <= a <= 200
1 <= b <= 200
1 <= c <= 200
a<b+c
b<a+c
c<a+b
Action-2
…
Action-n
Part II
Unit Testing:
Black-box, White-box
Part I
Introduction,
Testing Process
Part III
Module Testing
(Integration testing)
Part IV
System Testing
39
Test Cases for the Triangle Problem
1
2
3
4
5
6
7
8
9
10
11
C1: a < b + c ?
Case
ID
Part I
Introduction,
Testing Process
Mariam Kamkar
marka@ida.liu.se
a
b
c
DT3
C4: a = b?
DT4
C5: a = c?
DT5
C6: b = c?
DT6
DT8
A3: Isosceles
DT9
A4: Equilateral
DT
10
A5: Impossible
40
Mariam Kamkar
marka@ida.liu.se
Decision table testing can be used whenever the
system must implement complex business rules
when these rules can be represented as a
combination of conditions and when these conditions
have discrete actions associated with them.
DT2
C3: c < a + b ?
DT7
Part IV
System Testing
Expected output
C2: b < a + c ?
A2: Scalene
Part III
Module Testing
(Integration testing)
Applicability and Limitations
DT1
A1:
Not a triangle
Part II
Unit Testing:
Black-box, White-box
DT
11
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
White-box Testing
41
Mariam Kamkar
marka@ida.liu.se
(Glass box testing, Open box testing, Clear box testing, Structural testing)
1.
2.
Control flow testing
Data flow testing
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part I
Introduction,
Testing Process
Part IV
System Testing
Part III
Module Testing
(Integration testing)
Part IV
System Testing
42
1. Control flow testing
Mariam Kamkar
marka@ida.liu.se
Definition: a strategy in which testing is based on the internal paths, structure,
and implementation of the software under test (SUT)
Applicability: all levels of system development (path testing!)
y
y
y
y
Part III
Module Testing
(Integration testing)
Part II
Unit Testing:
Black-box, White-box
Unit
Integration
System
Acceptance
Disadvantages: 1) number of execution paths may be so large; 2) test cases
may not detect data sensitivity; 3) assumes that control flow is correct
(nonexistent paths!); 4) tester must have programming skills.
Advantages: tester can be sure that every path have been identified and
tested.
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
7
43
Control Flow Graphs
Process blocks
44
Mariam Kamkar
marka@ida.liu.se
Decision Point
Mariam Kamkar
marka@ida.liu.se
Definition: Given a program written in an imperative
programming language, its program graph is a
directed graph in which nodes are statement
fragments, and edges represent flow of control (a
complete statement is a “default” statement
fragment).
Junction Point
Sequence
If
While
Part I
Introduction,
Testing Process
Case
Until
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Code Coverage
(test coverage metrics)
Part I
Introduction,
Testing Process
45
Mariam Kamkar
marka@ida.liu.se
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Part I
Introduction,
Testing Process
What is Wrong with Line Coverage
47
Mariam Kamkar
marka@ida.liu.se
Steve Cornett (Bullseye testing technology)
Software developers and testers commonly use line coverage
because of its simplicity and availability in object code
instrumentation technology.
Of all the structural coverage criteria, line coverage is the
weakest, indicating the fewest number of test cases.
Bugs can easily occur in the cases that line coverage cannot
see.
The most significant shortcoming of line coverage is that it
fails to measure whether you test simple if statements with a
false decision outcome. Experts generally recommend to only
use line coverage if nothing else is available. Any other
measure is better.
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
46
Mariam Kamkar
marka@ida.liu.se
begin
y >= 0
no
yes
y=0
abs = y
test case-1 (yes):
input:
expected result:
actual result:
Part II
Unit Testing:
Black-box, White-box
Part IV
System Testing
Statement Coverage
Begin
if ( y >= 0)
then y = 0;
abs = y;
end;
Levels of Coverage:
Statement/Line/Basic block/Segment Coverage
Decision (Branch) Coverage
Condition Coverage
Decision/Condition Coverage
Path Coverage
Part I
Introduction,
Testing Process
Part III
Module Testing
(Integration testing)
y=?
?
?
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
48
Branch Coverage
Mariam Kamkar
marka@ida.liu.se
begin
Begin
if ( y >= 0)
then y = 0;
abs = y;
end;
y >= 0
no
yes
y=0
abs = y
test case-1 (yes):
input: y = 0
expected result: 0
actual result: 0
Part I
Introduction,
Testing Process
test case-2 (no):
input: y = ?
expected result: ?
actual result: ?
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
8
50
Condition Coverage
Mariam Kamkar
marka@ida.liu.se
Begin
if ( x < 10 && y > 20) {
z = foo (x,y); else z =fie (x,y);
}
end;
… more conditions?
x<10
&&
y>20
no
z=fie (x,y)
test case-1 (T,F):
input: x = ?, y = ?
expected result: ?
actual result: ?
Part I
Introduction,
Testing Process
51
Decision/Condition Coverage
Mariam Kamkar
marka@ida.liu.se
yes
z=foo (x,y)
test case-2 (F,T):
input: x = ?, y = ?
expected result: ?
actual result: ?
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
52
Path Coverage
Mariam Kamkar
marka@ida.liu.se
A path is a sequence of branches, or conditions.
Begin
if ( x < 10 && y > 20) {
z = foo (x,y); else z =fie (x,y);
}
end;
x<10
&&
y>20
no
yes
A path corresponds to a test case, or a set of inputs.
In code coverage testing, branches have more importance than
the blocks they connect.
z=fie (x,y)
z=foo (x,y)
Bugs are often sensitive to branches and conditions.
test case-1 (T,T,yes):
input: x = ?, y = ?
expected result: ?
actual result: ?
Part I
Introduction,
Testing Process
test case-2 (F,F,no):
input: x = ?, y = ?
expected result: ?
actual result: ?
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Part I
Introduction,
Testing Process
53
Path with loops
Mariam Kamkar
marka@ida.liu.se
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
54
Path Coverage (cont.)
Mariam Kamkar
marka@ida.liu.se
All possible execution paths
a
Question: How do we know how many paths to look
for?
d
b
c
Answer: The computation of cyclomatic complexity
a
e
c,b,d
d
e
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
9
55
Computation of cyclomatic complexity
Mariam Kamkar
marka@ida.liu.se
Examples of Graphs and calculation of McCabe’s
Complexity Metric
56
Mariam Kamkar
marka@ida.liu.se
Cyclomatic complexity has a foundation in graph theory and is
computed in the following ways:
1.
E= , N= , P=
V=
Cyclomatic complexity V(G), for a flow graph, G, is defined as:
E= , N= , P=
V=
V(G) = E – N + 2P
E: number of edges
N: number of nodes
P: number of disconnected parts of the graph
2.
E= , N= , P=
V=
E= , N= , P=
V=
Cyclomatic complexity V(G), for a flow graph, G, with only binary
decisions, is defined as:
V(G) = b + 1
b: number of binary decision
E=, N= , P=
V=
Part II
Unit Testing:
Black-box, White-box
Part I
Introduction,
Testing Process
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Part I
Introduction,
Testing Process
57
Mariam Kamkar
marka@ida.liu.se
A
1. V(G) = E – N + 2P
C
B
F
E
H
G
Mariam Kamkar
marka@ida.liu.se
2.
Control flow testing can be very time consuming because of all
modules and basic paths that comprise a system.
V(G) = b + 1
b=?
N
M
58
Applicability and Limitation
Its limitation are that the tester must have sufficient
programming skill to understand the code and its control flow.
L
K
Part IV
System Testing
V(G) = ?
J
I
Part III
Module Testing
(Integration testing)
Control flow testing is the cornerstone of unit testing. It should
be used for all modules of code that cannot be tested
sufficiently through reviews and inspections.
E=?
N=?
P=
D
Part II
Unit Testing:
Black-box, White-box
P
O
V(G) = ?
R
Q
S
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
59
2. Data Flow Testing
Mariam Kamkar
marka@ida.liu.se
Data flow testing focuses on the points at which
variables receive values and the points at which
these values are used (or referenced). It detects
improper use of data values due to coding errors.
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part I
Introduction,
Testing Process
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
60
Define/Reference Anomalies
Mariam Kamkar
marka@ida.liu.se
Early data flow analyses often centered on a set of
faults that are known as define/reference anomalies.
y A variable that is defined but never used (referenced)
y A variable that is used but never defined
y A variable that is defined twice before it is used
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
10
61
Mariam Kamkar
marka@ida.liu.se
dd: defined and defined again – not invalid but suspicious
du: defined and used – perfectly correct
dk: defined and then killed – not invalid but probably a
programming error
ud: used and defined – acceptable
uu: used and used again – acceptable
uk: used and killed – acceptable
kd: killed and defined – acceptable
ku: killed and used – a serious defect
kk: killed and killed – probably a programming error.
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
define x
Part I
Introduction,
Testing Process
63
Mariam Kamkar
marka@ida.liu.se
• ~d: the variable does not exist, then it is defined
• ~u: the variable does not exist, then it is used
• ~k: the variable does not exist, then it is killed
Mariam Kamkar
marka@ida.liu.se
du-path: a definition-use path (du-path) with respect
to variable v is a path in PATHS(P) such that, for
some v in V, there are defined and usage nodes
DEF(v, m) and USE(v, n) such that m and n are initial
and final nodes of the path.
Part IV
System Testing
Data Flow Graphs
62
Definitions
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
64
Hierarchy of data flow coverage metrics
define x
Mariam Kamkar
marka@ida.liu.se
All-Paths
All-DU-Paths
use y
kill z
define x
use x
use z
kill z
use x
define z
define x
use x
All-Uses
use x
All C-Uses/some P-Uses
All P-Uses/some C-Uses
define y
use z
All-Defs
use y
use z
kill y
define z
Part I
Introduction,
Testing Process
All-Edges
Branch
All-Nodes
Statement
Control flow graph annotated with define-use-kill
information for x, y, z
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Part I
Introduction,
Testing Process
Applicability and Limitations
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
65
66
Mariam Kamkar
marka@ida.liu.se
(data-flow testing)
Mariam Kamkar
marka@ida.liu.se
Part III
Module Testing
(Integration testing)
It should be used for all modules of code that cannot be tested
sufficiently through reviews and inspections.
Tester must have sufficient programming skill
Can be very time consuming
Part I
Introduction,
Testing Process
All-P-Uses
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
11
67
Integration Testing
1.
2.
3.
4.
Mariam Kamkar
marka@ida.liu.se
Design Specification
Code = System
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
69
Component code
Part II
Unit Testing:
Black-box, White-box
Part I
Introduction,
Testing Process
Mariam Kamkar
marka@ida.liu.se
Part III
Module Testing
(Integration testing)
Part IV
System Testing
70
Components
Mariam Kamkar
marka@ida.liu.se
Design Specification
Unit
test
driver
Tested components
Integration
test
Component code
Mariam Kamkar
marka@ida.liu.se
Objective: to ensure that code implemented the design
properly.
Top-down
Bottom-up
Big-bang
Sandwich
Part I
Introduction,
Testing Process
68
Unit & Integration Testing
Unit
test
Boundary conditions
independent paths
interface
...
Component
to be
tested
Tested components
stub
stub
Test
cases
Integrated modules
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Part II
Unit Testing:
Black-box, White-box
Part I
Introduction,
Testing Process
71
Mariam Kamkar
marka@ida.liu.se
Part III
Module Testing
(Integration testing)
Part IV
System Testing
72
1. Top-down
Mariam Kamkar
marka@ida.liu.se
A
B
E
C
F
D
G
A
B
E
Part I
Introduction,
Testing Process
C
F
Part II
Unit Testing:
Black-box, White-box
D
G
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
12
73
2. Bottom-up
Mariam Kamkar
marka@ida.liu.se
A
B
E
C
F
Mariam Kamkar
marka@ida.liu.se
A
D
B
G
Part II
Unit Testing:
Black-box, White-box
Part I
Introduction,
Testing Process
74
3. Big-bang
E
Part III
Module Testing
(Integration testing)
Part IV
System Testing
C
F
Part I
Introduction,
Testing Process
D
G
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
75
4. Sandwich
76
Mariam Kamkar
marka@ida.liu.se
Mariam Kamkar
marka@ida.liu.se
A
B
E
C
F
D
G
Target level B,C,D
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Part IV
System Testing
Part I
Introduction,
Testing Process
77
Mariam Kamkar
marka@ida.liu.se
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
78
System Testing
Mariam Kamkar
marka@ida.liu.se
Objective: to ensure that the system does what the
customer wants it to do.
System Testing Steps
y Function testing / Thread testing
y Performance testing
y Acceptance testing
y Installation testing
Customer
Developer
Test Automation
Termination Problem
Functional requirements
Nonfunctional requirements
Requirements definition
Requirements specification
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
13
79
80
Mariam Kamkar
marka@ida.liu.se
Mariam Kamkar
marka@ida.liu.se
Component code
Integration
test
Unit
test
Tested components
Function
test
Acceptance
test
Integrated modules
Customer requirements spec.
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Part I
Introduction,
Testing Process
81
Function testing/Thread testing
Mariam Kamkar
marka@ida.liu.se
Part II
Unit Testing:
Black-box, White-box
Performance
test
Installation
test
Verified validated
software
Tested components
Functioning systems
Integrated modules
Unit
test
Other software requirements
Accepted system
Component code
System functional requirements
Design Specification
System
In
Use!
User environment
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Cause-and-Effect-Graph
(test case generation from req.)
82
Mariam Kamkar
marka@ida.liu.se
(testing one function at a time)
functional requirements
causes: inputs
effects: outputs and transformations
causes-and-effect graph:
A function test checks that the integrated system performs its function as
specified in the requirement
Guidelines
y
y
y
y
y
y boolean graph reflecting causes and effects relationships
y is a formal language into which a natural language specification
is translated
use a test team independent of the designers and programmers
know the expected actions and output
test both valid and invalid input
never modify the system just to make testing easier
have stopping criteria
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Part I
Introduction,
Testing Process
84
Mariam Kamkar
marka@ida.liu.se
Specification: the character in column 1 must be an “A” or a “B”. The character in
column 2 must be a digit. In this situation, the file update is made. If the first
character is incorrect, message X12 is issued. If the second character is not a
digit, message X13 is issued.
c
Identity: if a then b
Part IV
System Testing
Mariam Kamkar
marka@ida.liu.se
a
b
Part III
Module Testing
(Integration testing)
83
Basic cause-effect graph symbols
a
Part II
Unit Testing:
Black-box, White-box
b
And: if (a and b) then c
a
b
c
Part I
Introduction,
Testing Process
a
d
Causes
Effects
C1: character in column 1 is “A”
C2: character in column 1 is “B”
C3: character in column 2 a digit
E1: update made
E2: message X12 is issued
E3:message X13 is issued
b
Identity: if (not a) then b
Or: if (a or b or c) then d
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
14
85
Sample cause-effect graph
1
Intermediate node
Mariam Kamkar
marka@ida.liu.se
Decision table for cause-and effect graph
E2
11
2
Part II
Unit Testing:
Black-box, White-box
Stress tests
Volume tests
Configuration tests
Compatibility tests
Regression tests
Security tests
Timing tests
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Test 3
Test 4
Cause 1
1
0
0
X
Cause 2
0
1
0
X
Cause 3
1
1
X
0
Effect E1
1
1
0
0
Effect E2
0
0
1
0
Effect E3
0
0
0
1
E3
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Performance Testing
nonfunctional requirements
Test 2
E1
3
Part I
Introduction,
Testing Process
Test 1
86
Mariam Kamkar
marka@ida.liu.se
Part I
Introduction,
Testing Process
87
Mariam Kamkar
marka@ida.liu.se
Environment tests
Quality tests
Recovery tests
Maintenance tests
Documentation tests
Human factors tests /
usability tests
Part III
Module Testing
(Integration testing)
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Acceptance Testing
customers, users need
88
Mariam Kamkar
marka@ida.liu.se
Benchmark test: a set of special test cases
Pilot test: everyday working
y Alpha test: at the developer’s site, controlled environment
y Beta test: at one or more customer site.
Parallel test: new system in parallel with previous one
Part IV
System Testing
Installation Testing
users site
Part II
Unit Testing:
Black-box, White-box
Part I
Introduction,
Testing Process
89
Mariam Kamkar
marka@ida.liu.se
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
90
Test Automation
Mariam Kamkar
marka@ida.liu.se
Automating parts of the testing process can provide long-term
benefits to organization, such as:
Acceptance test at developers site
Æ installation test at users site,
otherwise may not be needed!!
y
y
y
y
y
reducing the amount of time it takes to execute a suite of tests
reducing the tester’s involvement in executing tests
facilitating regression testing
allowing for the simulation of hundreds of users
avoiding human mistakes by having tools control repetitive and
tedious tasks
Test automation refers to two key testing activities:
y Executing the tests
y Evaluating the output
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
15
91
Automated Testing Tools
Mariam Kamkar
marka@ida.liu.se
Code Analysis tools
Termination Problem
How decide when to stop testing
92
Mariam Kamkar
marka@ida.liu.se
The main problem for managers!
y Static, Dynamic
Termination takes place when
Test execution tools
• resources (time & budget) are over
• found the seeded faults
• some coverage is reached
y Capture-and-Replay
y Stubs & Drivers
y Comparators
Test case generator
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Summary - What have we learned today? (1/2)
Part I
Introduction,
Testing Process
93
Mariam Kamkar
marka@ida.liu.se
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Summary - What have we learned today? (2/2)
94
Mariam Kamkar
marka@ida.liu.se
Part I: Introduction, Testing process
Part III: Module Testing (Integration Testing)
y Top-down
y Bottom-up
y Big-bang
y Sandwich
Part II: Unit Testing:
y Black-box Testing
1. Exhaustive testing
2. Equivalence class testing (Equivalence Partitioning)
3. Boundary value analysis
4. Decision table testing
y
Part IV: System Testing
White-box Testing
•
Control Flow Testing
1. Statement/Line/Basic block/Segment Coverage
2. Decision (Branch) Coverage
3. Condition Coverage
4. Decision/Condition Coverage
5. Path Coverage
•
Data Flow Testing
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
95
Mariam Kamkar
marka@ida.liu.se
Part I
Introduction,
Testing Process
Part II
Unit Testing:
Black-box, White-box
Part III
Module Testing
(Integration testing)
Part IV
System Testing
16
