Unit Testing
CS 4311
Frank Tsui, Orland Karam, and Barbara Bernal, Essential of Software
Engineering, 3rd edition, Jones & Bartett Learning. Sections 10.3.
Hans Van Vliet, Software Engineering, Principles and Practice, 3rd edition,
John Wiley & Sons, 2008. Chapter 13.
1
Hierarchy of Testing
Testing
Ad Hoc
Program Testing
Unit Testing
Integration Testing
Black Box
Top Down
Equivalence
State Transition
Acceptance
Testing
Function
Properties
Benchmark
Performance
Pilot
Reliability
Alpha
Bottom Up
Boundary
Decision Table
System Testing
Big Bang
Sandwich
Availability
Beta
Security
Use Case
Usability
Domain Analysis
Documentation
Portability
White Box
Control Flow
Data Flow
Capacity
2
Outline
Basic concepts
 Black box testing
 White box testing

3
Basic Concepts

What is unit testing?
 Test


focusing on the smallest units of code, such as
Functions, procedures, subroutines, subprograms
Methods, classes
 Component
tested in isolation from the rest of the
system and in a controlled environment:


Uses appropriately chosen input data
Uses component-level design description as guide
 Often
the target of testing frameworks such as JUnit
4
Basics

Why unit testing?
 Foundation
for other testing like integration and
system testing

What to test?
 Data
transformations across the unit are tested
 Data structures are tested to ensure data integrity

When and who?
 Frequently
done (at least partially) during code
development by code developers
5
Unit Test Procedures
Driver
Module to Be
Tested
Results
Test
cases
Stub
Stub
6
Two Different Techniques

Black box



White box



Based on specification
Inner structure of test object is not considered
Based on source code
Inner structure of test object is the basis of test case selection
Often complementary


Effectiveness of black box is similar to white box, but the
mistakes found are different (Hetzel 1976, Myers 1978)
Use in combinations
7
Outline
Basic concepts
 Black box testing

 Basic
concepts
 Equivalence classes
…

White box testing
8
What Is Black Box Testing?



Unit (code, module) seen as a black box
No access to the internal or logical structure
Determine if given input produces expected
output
Input
Output
9
Black Box Testing



Test set is derived from specifications or requirements
Goal is to cover the input space
Lots of approaches to describing input space:






Equivalence classes
Boundary value analysis
Decision tables
State transitions
Use cases
...
10
Advantage and Disadvantage

(Dis)Advantages
 It
does not require access to the internal logic of a
component
 However, in most real-world applications, impossible
to test all possible inputs

Need to define an efficient strategy to limit
number of test cases
11
General Process






Analyze specifications or requirements
Select valid and invalid inputs (i.e., positive and
negative tests)
Determine expected outputs
Construct tests
Run tests
Compare actual outputs to expected outputs
12
Outline
Basic concepts
Black box testing

Basic concepts
 Equivalence classes
 Boundary values
…

White box testing
13
Motivation


Assume you test a sign function, int sign(int x),
whose result is defined as:
1 if x > 0
0 if x = 0
-1 otherwise (i.e., x < 0)
One way to reduce the number of test cases is:
 to
partition the input into three subsets
{ x | x > 0}, { x | x = 0}, { x | x < 0}
 to pick one from each subset, e.g., 10, 0, and -10.
14
Basic Strategy of Equivalence Classes

Partition the input into equivalence classes
 This
is the tricky part.
 It’s an equivalence class if:




Every test using one element of the class tests the same
thing that any other element of the class tests
If an error is found with one element, it should be found
with any element
If an error is not found with some element, it is not found by
any element
Test a subset from each class
15
Exercise
Consider a factorial function, fact(n):
if n < 0 or n >= 200, error
if 0  n  20, exact value
if 20 < n < 200, approximate value within 0.1%.
Q: What equivalence classes can you see?
16
Simple Example
Suppose you are building an airline reservation system.



A traveler can be a child, an adult, or a senior.
The price depends on the type of traveler.
The seat reservation does not depend on the type of traveler.
Q: How many test cases can you identify for the
reservation component and the billing component?
17
Heuristics for Finding Equivalence Classes

Identify restrictions for inputs and outputs in the
specification.
18
Heuristics for Finding Equivalence Classes


Identify restrictions for inputs and outputs in the
specification.
If there is a continuous numerical domain, create one
valid and two or three invalid classes (above, below,
and NaN).
19
Heuristics for Finding Equivalence Classes



Identify restrictions for inputs and outputs in the
specification.
If there is a continuous numerical domain, create one
valid and two or three invalid classes (above, below,
and NaN).
If a number of values is required, create one valid and
one or more invalid classes.
20
Heuristics for Finding Equivalence Classes




Identify restrictions for inputs and outputs in the
specification.
If there is a continuous numerical domain, create one
valid and two or three invalid classes (above, below,
and NaN).
If a number of values is required, create one valid and
one or more invalid classes.
If a set of values is specified where each may be
treated differently, create a class for each element of
the set and one more for elements outside the set.
21
Heuristics for Finding Equivalence Classes





Identify restrictions for inputs and outputs in the
specification.
If there is a continuous numerical domain, create one
valid and two or three invalid classes (above, below,
and NaN).
If a number of values is required, create one valid and
one or more invalid classes.
If a set of values is specified where each may be
treated differently, create a class for each element of
the set and one more for elements outside the set.
If there is a condition, create two classes, one
satisfying and one not satisfying the condition.
22
Exercise
Define test cases for a program that reserves a golf tee time.
 The standard green fee is $65 on weekdays (Monday-Friday) and
$80 on weekend (Saturday and Sunday).
 However, an El Paso resident pays a reduced green fee of $45 and
$60 on weekdays and weekend, respectively.
 A senior (of age 60+) pays only $40 and $50 on weekdays and
weekend, respectively.
 A junior (of age <17) pays only $20 and $30 on weekdays and
weekend, respectively.
Q: How many equivalence classes? Define them.
Q: Define one test case per equivalence class.
23
Outline
Basic concepts
 Black box testing

 Basic concepts
 Equivalence classes
 Boundary values
 Decision tables
…

White box testing
24
Boundary Values

Observations
 Programs
that fail at interior elements of an
equivalence class usually fail at the boundaries too.
 Programmers often make errors on boundary
conditions (e.g., branch/loop condition x <= 10
instead of x < 10).

Test the boundaries
 if
it should work for 1-99, test 0, 1, 99, 100.
 if it works for A-Z, try @, A, Z, [, a, and z

The hard part is identifying boundaries.
25
Hints

If a domain is a restricted set, check the boundaries.
e.g., D=[1,10], test 0, 1, 10, 11





It may be possible to test the boundaries of outputs, also.
For ordered sets, check the first and last elements.
For complex data structures, the empty list, full lists, the
zero array, and the null pointer should be tested.
Extremely large data sets should be tested.
Check for off-by-one errors.
26
More Hints

Some boundaries are not obvious and may
depend on the implementation (use gray box
testing if needed)
 Numeric
limits (e.g., test 255 and 256 for 8-bit values)
 Implementation limits (e.g., max array size)
27
Example

Consider a simple voltage stabilizer with an input range
of 190-240 volts 50-60 hertz.
28
Example


Consider a simple voltage stabilizer with an input range
of 190-240 volts 50-60 hertz.
Voltage boundaries





189 volts (below the lowest boundary)
190 volts (lowest boundary)
240 volts (highest boundary)
241 volts (above the highest boundary)
Cycles (hertz) boundaries




49 hertz (below)
50 hertz (lowest)
60 hertz (highest)
61 hertz (above)
29
Example


Consider a simple voltage stabilizer with an input range
of 190-240 volts 50-60 hertz.
Voltage boundaries





189 volts (below the lowest boundary)
190 volts (lowest boundary)
240 volts (highest boundary)
How to combine?
241 volts (above the highest boundary)
Cycles (hertz) boundaries




49 hertz (below)
50 hertz (lowest)
60 hertz (highest)
61 hertz (above)
Exhaustive,
Pairwise (later),
One extremes with
others fixed.
30
Exercise


Determine the boundary values for US Postal Service
ZIP codes (5 digits such as 79912 or 5 digits hyphen 4
digits such as 79912-1818).
Determine the boundary values for a 15-character last
name entry.
Hints: Two kinds of boundaries---size boundary and value
boundary.
31
Outline
Basic concepts
 Black box testing

 Basic concepts
 Equivalence classes
 Boundary values
 Decision tables
 Pairwise testing
 State transitions
 Use cases

White box testing
32
Decision Tables



Construct a table (to help organize the testing)
Identify each rule or condition in the system that
depends on some input
For each input to one of these rules, list the
combinations of inputs and the expected results
33
Example
Theater ticket prices are discounted for senior
citizens and students.
Test Case
1
2
3
C1
Student?
T
T
F
C2
Senior?
T
F
T
Result
Discount?
T
T
T
4
F
F
F
34
Exercise
Construct a decision table for a program that reserves a golf tee time.
 The standard green fee is $65 on weekdays (Monday-Friday) and
$80 on weekend (Saturday and Sunday).
 However, an El Paso resident pays a reduced green fee of $45 and
$60 on weekdays and weekend, respectively.
 A senior (of age 60+) pays only $40 and $50 on weekdays and
weekend, respectively.
 A junior (of age <17) pays only $20 and $30 on weekdays and
weekend, respectively.
35
Pairwise Testing: Motivation
Consider a website that must:




operate correctly with different browsers: IE 5, IE 6, and IE 7;
Mozilla 1.1; Opera 7; FireFox 2, 3, and 4, and Chrome.
work using RealPlayer, MediaPlayer, or no plug-in.
run under Windows ME, NT, 2000, XP, and Vista, 7.0, and 8.0
accept pages from IIS, Apache and WebLogic running on
Windows NT, 2000, and Linux servers.
How many different configurations are there?
36
Motivation
Consider a website that must:




operate correctly with different browsers: IE 5, IE 6, and IE 7;
Mozilla 1.1; Opera 7; FireFox 2, 3, and 4, and Chrome.
work using RealPlayer, MediaPlayer, or no plug-in.
run under Windows ME, NT, 2000, XP, and Vista, 7.0, and 8.0
accept pages from IIS, Apache and WebLogic running on
Windows NT, 2000, and Linux servers.
How many different configurations are there?
A: 9 browsers x 3 plug-ins x 7 OS’s x 3 web servers x 3
server OS’s = 1701 combinations
37
Motivation
Many such examples (i.e., finite sets of discrete values).
A bank is ready to test a data processing system



Customer types: Gold, Platinum, Business, Non profits
Account types: Checking, Savings, Mortgages, Consumer loans,
Commercial loans
States (with different rules): CA, NV, UT, ID, AZ, NM
How many different configurations are there?
38
Approaches?
When given a large number of combinations:
 Give up and don’t test
 Test all combinations: miss targets, delay product
launch, and go out of business
 Choose one or two cases
 Choose a few that the programmers already ran
 Choose the tests that are easy to create
 List all combinations and choose first few
 List all combinations and randomly choose some
 “Magically” choose a small subset with high probability of
revealing defects
39
Pairwise Testing


Combinatorial testing technique in which every pair of
input parameters of software is tested.
Reasonable cost-benefit compromise





Much faster than exhaustive testing
More effective than less exhaustive methods that fail to exercise
all possible pairs of input parameters
Why? Majority of software failures are caused by a single input
parameter or a combination of two input parameters.
Each pair of input parameter values should be captured
at least by one test case.
However, finding the least number of test cases is an
NP-complete problem.
40
Example




Consider software that takes three input parameters, say
x, y, and z.
If each parameter can have three different values, then
there will be 27 different pairs: (x1, y1), (x1, y2), …, (y3,
z3).
A test case (x1, y3, z2), for example, captures three of
these 27 pairs: (x1, y3), (x1, z2), and (y3, z3).
By selecting test cases judiciously, all pairs of input
parameters can be exercised with a minimum number of
test cases; e.g., a set of 9 test cases can capture all 27
pairs of three parameters, each with three different
values.
41
State Transitions


Use state transitions (e.g., UML state machine
diagrams) to derive test inputs
Cover a state transition diagram, e.g.,
 Visit
each state at least once
 Trigger each event at least once
 Exercise each transition at least once
 Exercise each path at least once*
* Might not be possible.
42
Example and Exercise
Customer makes a reservation and has a limited time to
pay. May cancel at any time.
paid
ordered/
start timer
printed
Paid
Ticketed
canceled
Reserved
timer
expired
canceled/
refund
Cancelled
delivered
canceled/
refund
Used
Unpaid
Groups: How many tests to visit each state, trigger each event,
exercise each transition, and exercise each path?
43
Use Cases



Use the use cases to specify test cases
Use case specifies both normal, alternate, and
exceptional operations
However, use cases may not have sufficient
details, esp. for unit testing
44
Outline
Basic concepts
Black box testing
 White box testing

 Control
flow graph (CFG)
 Statement coverage
…
45
White Box Testing


Test set is derived from structure of (source) code
Also known as:



Glass box testing
Structural testing
Often use a graph called a control flow graph (CFG)


To represent code structure
To cover it (CFG), e.g., all statements
Input
Output
46
Control Flow Graphs

Programs are made of three kinds of statements:



Sequence (i.e., series of statements)
Condition (i.e., if statement)
Iteration (i.e., while, for, repeat statements)
47
Control Flow Graphs

Programs are made of three kinds of statements:




Sequence (i.e., series of statements)
Condition (i.e., if statement)
Iteration (i.e., while, for, repeat statements)
CFG: visual representation of flow of control


Node represents a sequence of statements with single entry
and single exit
Edge represents transfer of control from one node to another
48
Control Flow Graph (CFG)
Sequence
If-then-else
If-then
Iterative
49
Control Flow Graph (CFG)
Join
Sequence
If-then-else
Join
If-then
Iterative
When drawing CFG, ensure that there is
one exit: include the join node if needed
50
Example
1
2
1.
2.
3.
4.
5.
6.
7.
8.
9.
read (result);
read (x,k)
while result < 0 {
ptr  false
if x > k then
ptr  true
xx+1
result  result + 1 }
print result
3
4
9
5
6
7
8
51
Example
1
1,2
2
1.
2.
3.
4.
5.
6.
7.
8.
9.
read (result);
read (x,k)
while result < 0 {
ptr  false
if x > k then
ptr  true
xx+1
result  result + 1 }
print result
3
3
4
9
9
4,5
5
6
6
7
8
Join
7,8
52
Exercise: Draw CFGs
Example 1
1. if (a < b) then
2.
while (a < n)
3.
a  a + 1;
4. else
5.
while (b < n)
6.
b  b + 1;
Example 2
1. read (a, b);
2. if (a  0 && b  0) then {
3. c  a + b;
4. if c > 10 then
5.
c  max;
6. else c  min; }
7. else print ‘Done’;
Example 3
1. read (a, b);
2. if (a  0 || b  0) then {
3. c  a + b;
4. while( c < 100)
5.
c  a + b; }
6. c  a * b;
53
Outline
Basic concepts
 Black box testing
 White box testing

 Control flow graph (CFG)
 Statement coverage
 Branch coverage
 Condition coverage
 Path coverage
…
54
Coverage

Coverage-based testing
Adequacy of testing in terms of the coverage of the product to be
tested, e.g., the percentage of statements executed or the
percentage of functional requirements tested.

Control-flow coverage





Statement
Branch
Condition
Path
Data-flow coverage

Def-use
55
Statement Coverage


Every statement gets executed at least once
Every node in the CFG gets visited at least
once, also known as all-nodes coverage
56
Example
Q: Number of paths needed for statement coverage?
1
6
1
6
2
7
2
7
3
8
3
8
4
9
4
9
5
5
57
Branch Coverage



Every decision is made true and false.
Every edge in a CFG of the program gets traversed at
least once.
Also known as



Decision coverage
All edges coverage
Basis path coverage

Branch is finer than statement.

C1 is finer than C2 if

true
false
T1  C1  (T2  C2  T2  T1)
58
Example
Q: Number of paths needed for branch coverage?
1
6
1
6
2
7
2
7
3
8
3
8
4
9
4
9
5
5
59
Condition Coverage


Make each Boolean sub-expression (called a
condition) of a decision evaluated both to true
and false
Example:
if (x > 0 || y > 0) { … } else { … }
x > 0, x  0
 C2: y > 0, y  0
 C1:

Q: How many tests?
60
Many Variances of Condition Coverage

Condition coverage is not finer than branch
coverage.
 There
are pathological cases where you can achieve
condition and not branch. E.g., if (x > 0 || y > 0) { … }

(x = 10, y = 0), (x = 0, y = 10): condition but not branch
 Under
most circumstances, achieving condition
achieves branch

Many variances, e.g.,
 Multiple
condition
 Condition/decision
 Modified condition/decision (MC/DC)
61
CFG for Condition Coverage


One way to determine the number of paths is to break the
compound conditional into atomic conditionals
Suppose you were writing the CFG for the assembly language
implementation of the control construct
if (A AND B) then
C
endif
(short circuit eval)
LD A
BZ :endif
LD B
BZ :endif
JSR C
:endif nop
(no short circuit eval)
LD A
; in general, lots of
LAND B
; code for A and B
BZ :endif
JSR C
:endif nop
62
AND Condition
1
1. read (a, b);
2. if (a == 0 && b == 0) then {
3. c  a + b; }
4. else c  a * b;
paths:
1, 2A,2B,3, J
1, 2A, 2B, 4, J
1, 2A, 4, J
2A
true
true
false
2B
4
false
3
Join
63
OR Condition
1
1. read (a,b);
2. if (a == 0 || b == 0) then {
3. c  a + b;
4. while( c < 100)
5.
c  a + b;}
2A
true
false
3
2B
true
false
Paths:
1, 2A, 3, 4, 5, 4 … J
1, 2A, 3, 4, J
1, 2A, 2B, 3, 4, 5, 4, … J
1, 2A, 2B, J
4
5
Join
64
Outline
Basic concepts
 Black box testing
 White box testing

 Control flow graph (CFG)
 Statement coverage
 Branch coverage
 Condition coverage
 Path coverage
 Def-use coverage
65
Path Coverage



Every distinct path through code is executed at
least once.
All-paths coverage similar to exhaustive testing
A path is a sequence of:
 Statements
 Branches
 Edges
66
Example
1. read (x)
2. read (z)
3. if x  0 then {
4. y  x * z;
5. x  z; }
6. else print ‘Invalid’;
7. if z > 1 then
8. print z;
9. else print ‘Invalid’;
Test Paths:
1, 2, 3, 4, 5, J1, 7, 8, J2
1, 2, 3, 4, 5, J1, 7, 9, J2
1, 2, 3, 6, J1, 7, 8, J2
1, 2, 3, 6, J1, 7, 9, J2
1,2,3
4,5
6
Join1
7
8
9
Join2
67
Linearly Independent Paths


It is not feasible to calculate the total number of paths.
It is feasible to calculate the number of linearly
independent paths:



A complete path which, disregarding back tracking (such as
loops), has an unique set of decisions in a program.
Any path through the program that introduces at least one new
edge that is not included in any other linearly independent paths.
The number of linearly independent paths is the number
of end-to-end paths required to touch every path
segment at least once.
68
McCabe’s Cyclomatic Complexity




Software metric for the logical complexity of a program
Defines the number of independent paths in the basis
set of a program (basis set: maximal linearly
independent set of paths).
Provides the upper bound for the number of tests that
must be conducted to ensure that all statements have
been executed at least once.
For edges (E) and nodes (N), cyclomatic complexity
number V is:
V=E–N+2
69
Example: Complexity of CFG
1
3,4
1
2
5
3,4
6
Join
5
Join
3
N=1
E=0
V=E-N+2
=0-1+2
=1
N=3
E=2
V=E-N+2
=2–3+2
=1
N=4
E=4
V=E-N+2
=4-4+2
=2
N=3
E=3
V=E-N+2
=3-3+2
=2
70
Example
1
2,3
6
V = 11 – 9 + 2 = 4
4,5
8
7
Independent paths:
1-11
1-2-3-4-5-10-1-11
1-2-3-6-8-9-10-1…-11
1-2-3-6-7-9-10-1…-11
9
10
11
71
Exercise: Cyclomatic Complexity
1,2
1,2
3
1, 2
9
2
6
Join
3,4
1
4,5
5
3
6
5
7
6
3
4
5
Join
Join
6
7,8
Join
Join
72
Linearly Independent Path Coverage
Cyclomatic-number criterion: construct a test
set such that all linearly independent paths
are covered.
1. read (x)
2. read (z)
3. if x  0 then {
4. y  x * z;
5. x  z; }
6. else print ‘Invalid’;
7. if z > 1 then
8. print z;
9. else print ‘Invalid’;
Cyclomatic complexity: 3 (= 9 – 8 + 2)
Test Paths:
1, 2, 3, 4, 5, J1, 7, 8, J2
1, 2, 3, 4, 5, J1, 7, 9, J2
1, 2, 3, 6, J1, 7, 8, J2,
1,2,3
4,5
6
Join1
7
8
9
Join2
73
Outline
Basic concepts
 Black box testing
 White box testing

 Control flow graph (CFG)
 Statement coverage
 Branch coverage
 Condition coverage
 Path coverage
 Def-use coverage
74
Data Flow Coverage


Consider how variables are treated along the various
paths, a.k.a. data flow analysis, e.g., definitions and
uses.
Many variations






All-defs-uses
All-defs
All-uses
All-C-uses/Some-P-uses
All-P-uses/Some-C-uses
All-P-uses
* P-uses are predicate uses, e.g., in conditions; all others are C-uses.
75
Def-Use Coverage

Def-use coverage: every path
from every definition of every
variable to every use of that
definition is exercised in some
test.
1. read (x);
2. read (z);
3. if x  0 then {
4. y  x * z;
5. x  z; }
6. else print ‘Invalid’;
7. if y > 1 then
8. print y;
9. else print ‘Invalid’;
1,2,3
U: x, z
D: y, x
4,5
D: x, z
U: x
6
U: none
Join
7
U: y
8
U: y
9
U: none
Join
Test Path: 1, 2, 3, 4, 5, 7, 8, J
76
Strength of Coverage
Statement
Arrows point from
weaker to stronger
coverage.
Branch
Condition
Def-Use
Path
Stronger coverage
requires more test
cases.
77
Limitation of Paths

What they don’t tell you:
 Timing
errors
 Unanticipated error conditions
 User interface inconsistency (or anything else)
 Configuration errors
 Capacity errors
78