Software Testing
Topics

The testing process
 unit
testing
 integration and system testing
 acceptance testing


Test case planning and tracking
Testing types
 black
box vs. white box testing
 coverage testing
 regression testing
 stress testing

Test case prioritization
CMSC 345, Version 4/12
2
Testing
Goal is to discover defects
 The only validation technique for nonfunctional requirements
 Successful test: causes a program to
behave in an anomalous way
 Testing shows the presence, not the
absence, of defects.

CMSC 345, Version 4/12
3
Testing Process Overview
Integration and
System Testing
Unit Testing
Acceptance
Testing
Subsystem1
unit1
unit
unit
unit
unit2
Subsystem2
unit3
unitn
unit
Final System
Final System
unit
Subsystemm
unit
unit
CMSC 345, Version 4/12
unit
unit
4
The Testing Process Elaborated

Unit (component) testing

Testing of individual program components (e.g., functions, methods, or
classes)
 Usually the responsibility of the component’s developer
unit
 Must be structured
 White box testing

Integration and system testing






Testing of groups of components integrated to create a subsystem or
the entire system
Sometimes the responsibility of an independent testing team Subsystem
unit
Tests are based on system specification (black box testing)
unit
Is iterative (keeps building)
May include alpha and beta testing
Acceptance testing


Run in the presence of customer or by customer
Used to validate all system requirements
CMSC 345, Version 4/12
Final System
$$$$!
5
Unit (Component) Testing
You may do this on your own, but always …
 have a plan
 keep records
 automate!
 JUnit
(for Java) or other tools
 even #ifdef - #endif pairs (C/C++)
 main( ) method (Java)
CMSC 345, Version 4/12
6
Integration Testing Approaches –
Procedural Programming

Top-down testing



Bottom-up testing



Start with low-level components and integrate from bottom up
Uses drivers
Sandwich testing


Start with high-level components and integrate from top down
Uses stubs
Combination of top-down and bottom-up
Thread testing

Tests one piece of functionality at a time
CMSC 345, Version 4/12
7
Top-down Integration & Testing
Level
Being
Tested
0
1
Stub
Stub
Stub: 1) says “I’ve been called!”,
2) confirms receipt of inputs, if
any, and 3) returns hard-coded
outputs, if necessary
Stub
n
CMSC 345, Version 4/12
8
Top-down Integration & Testing
Level
Stub: 1) says “I’ve been called!”,
2) confirms receipt of inputs, if
any, and 3) returns hard-coded
outputs, if necessary
Tested
0
Being
Tested
1
Stub
Being
Tested
Stub
Being
Tested
Stub
Stub
n
CMSC 345, Version 4/12
9
Top-down Integration & Testing
Level
Tested
0
1
Tested
Tested
n
Stub: 1) says “I’ve been called!”,
2) confirms receipt of inputs, if
any, and 3) returns hard-coded
outputs, if necessary
Being
Tested
CMSC 345, Version 4/12
Tested
Tested
Being
Tested
Tested
Tested
Being
Tested
Tested
Being
Tested
10
Bottom-up Integration & Testing
Level
Driver: 1) Calls functions, 2)
passes in inputs, if any, and 3)
confirms receipt of outputs, if
any
0
1
Driver
n
Being
Tested
CMSC 345, Version 4/12
Driver
Being
Tested
Being
Tested
Being
Tested
11
Bottom-up Integration & Testing
Level
Driver: 1) Calls functions, 2)
passes in inputs, if any, and 3)
confirms receipt of outputs, if
any
0
1
Being
Tested
n
Tested
CMSC 345, Version 4/12
Being
Tested
Tested
Being
Tested
Tested
Being
Tested
Tested
12
Bottom-up Integration & Testing
Level
Driver: 1) Calls functions, 2)
passes in inputs, if any, and 3)
confirms receipt of outputs, if
any
0
1
Driver
Tested
n
Tested
CMSC 345, Version 4/12
Driver
Tested
Tested
Driver
Tested
Tested
Tested
Tested
13
Sandwich Integration & Testing
Uses both stubs and drivers
Level
0
1
n
CMSC 345, Version 4/12
14
Thread Integration & Testing
Level
May be done top-down,
bottom-up, or sandwich
0
1
n
CMSC 345, Version 4/12
15
Exercise (on your own!)
What are some of the benefits associated with
each of the following integration and testing
methods?

Top-down

Bottom-up

Sandwich

Thread
CMSC 345, Version 4/12
16
Integration Testing Approaches –
Object-oriented Programming

Use based




Cluster




Start with classes that have no dependencies
Move “out” to next “level” of classes, and so on
Similar to bottom-up for procedural programming
Test clusters of associated classes
Integrate the clusters
May be combined with use based testing (above)
Scenario


Based on functional specifications (possibly expressed as use
cases)
Similar to thread testing for procedural programming
CMSC 345, Version 4/12
17
System Testing

Alpha
 Internal
to development organization, but not
the developers themselves

Beta
 External

to development organization
Acceptance
 Performed
in the presence of or possibly by
the customer
 Determines final “acceptance” by the
customer
CMSC 345, Version 4/12
18
Test Case Planning and Tracking
Test
#
ID
Description
Natural language
description
CMSC 345, Version 4/12
Input
Values
Specific
values (if
possible)
Expected
Output
Actual
Output
Specific
values (if
possible)
Specific
values (if
possible)
P/F
Criteria
Comments
How to
Pass, Fail, other
determine
notes
Pass or
Fail
19
A Sample Testing Form
Test
#
Description
1
Tests that a string
will be displayed
centered and in
12 point
CMSC 345, Version 4/12
Input
Values
font = 12
alignment =
“C”
string =
“CMSC 345”
Expected
Output
“CMSC 345” is
displayed and
centered in 12
point
Before Testing
Actual
Output
P/F
Criteria
Actual =
Expected
Comments
20
A Sample Testing Form
Test
#
Description
1
Tests that a string
will be displayed
centered and in
12 point
CMSC 345, Version 4/12
Input
Values
font = 12
alignment =
“C”
string =
“CMSC 345”
Expected
Output
“CMSC 345” is
displayed and
centered in 12
point
After Testing
Actual
Output
“CMSC 345 is
displayed and
centered in 12
point
P/F
Criteria
Comments
Actual = Pass
Expected
21
A Sample Testing Form
Test
#
Description
1
Tests that a string
will be displayed
centered and in
12 point
2
Tests that the
largest integer in
the list is located
CMSC 345, Version 4/12
Input
Values
font = 12
alignment =
“C”
string =
“CMSC 345”
intList = 5,
-2,100,16,
-51,12
listSize = 6
Expected
Output
“CMSC 345” is
displayed and
centered in 12
point
100
Before Testing
Actual
P/F
Output
Criteria
Comments
“CMSC 345” is Actual =
Pass
displayed and Expected
centered in 12
point
Actual =
Expected
22
A Sample Testing Form
Test
#
Description
1
Tests that a string
will be displayed
centered and in
12 point
2
Tests that the
largest integer in
the list is located
CMSC 345, Version 4/12
Input
Values
font = 12
alignment =
“C”
string =
“CMSC 345”
intList = 5,
-2,100,16,
-51,12
listSize = 6
After Testing
Expected
Output
“CMSC 345” is
displayed and
centered in 12
point
Actual
Output
“CMSC 345 is
displayed and
centered in 12
point
P/F
Criteria
Comments
Actual =
Pass
Expected
100
-51
Actual =
Fail
Expected
23
White Box Testing
input
code code code code code
code code code code code
code code code code code
code code code code code
output
Sometimes called structural testing
 Can “see inside” the code unit
 Objectives:

 Exercise
unique algorithms
 Exercise all program statements at least once

Typically tests small program units such
as functions or class methods
CMSC 345, Version 4/12
24
Coverage Testing







A technique for white box testing
Ensures that each statement is executed at least
once
Use a program flow graph to derive paths
Find minimum paths needed to “cover” all
statements (there are tools to do this)
Drive the code through all of these paths
All branches are executed, but not all
combinations of branches.
Some paths may be impossible to test.
CMSC 345, Version 4/12
25
#include <iostream>
using namespace std;
void triangle (int, int, int);
triangle.cpp
// parameters are sides of the triangle
int main()
{
int s1, s2, s3;
char repeat;
cout << "This program will accept the sides of a triangle\n"
<< " and tell you the type of triangle.\n" << endl;
do
{
cout << "Enter the 3 sides: " ;
cin >> s1 >> s2 >> s3;
triangle(s1, s2, s3);
cout << "Want to do another? (Y or N) ";
cin >> repeat;
cout << endl << endl;
} while (repeat =='Y' || repeat == 'y');
cout <<"Program terminated" << endl << endl;
cout << "Enter any character to continue" << endl;
cin >> repeat;
return 0;
}
void triangle (int side1, int side2, int side3)
{
// Error checking
/***********************************/
// Check sides for validity
if (side1 <= 0 || side2 <= 0)
cout << "Illegal value for side" << endl << endl;
// Check Triangle inequality
if (side1+side2 <= side3 || side2+side3 <= side1)
cout << "Triangle inequality violation" << endl << endl;
/***********************************/
Pick Version
triangle type
CMSC //345,
4/12
if (side1 == side2 && side2 == side3)
// 1
// 2
// 3
// 4
// 5
26
triangle.exe Control Graph
1a
1b
2
2
3a
3
Baseline path
3b
4
1
4
5
5a
5b
6
7b
7
9
7c
9a
9b
7a
11
Vertices = 18
Edges = 28
Regions = 12
6
8
8
10
10
12
11
12
V(g) = 28 - 18 + 2
= 11 + 1
= 12
CMSC Northrup
345, Version
4/122009
D. Cheslock,
Grumman,
(e-n+2)
(p+1)
27
A Set Of Basis Paths For triangle.cpp
1)
1a - 1b - 3a - 3b - 5a - 7a - 7b -7c -9a - 9b - 11 - 12
sample data: 3, 5, 7 (baseline path)
2)
1a - 2 - 3a - 3b - 5a - 7a - 7b -7c -9a - 9b - 11 - 12
sample data: -3, 7, 3
3)
1a - 1b - 2 - 3a - 3b - 5a - 7a - 7b -7c - 9a - 9b - 11 - 12
sample data: can’t perform this test
4)
1a - 1b - 3a - 4 - 5a - 7a - 7b -7c - 9a - 9b - 11 - 12
sample data: 2, 3, 7
5)
1a - 1b - 3a - 3b - 4 - 5a - 7a - 7b -7c - 9a - 9b - 11 - 12
sample data: 7, 3, 2
6)
1a - 1b - 3a - 3b - 5a - 5b - 6 - 12
sample data: 5, 5, 5
7)
1a - 1b - 3a - 3b - 5a - 5b - 7a - 7b -7c - 9a - 9b - 11 - 12
sample data: can’t perform this test
8)
1a - 1b - 3a - 3b - 5a - 5b - 7a - 8 - 12
sample data: 4, 4, 2
9)
1a - 1b - 3a - 3b - 5a - 7a - 7b - 8 - 12
sample data: 2, 4, 4
10)
1a - 1b - 3a - 3b - 5a - 7a - 7b -7c - 8 - 12
sample data: 4, 2, 4
11)
1a - 1b - 3a - 3b - 5a - 7a - 7b -7c - 9a - 10 - 12
sample data: 3, 4, 5
12)
1a - 1b - 3a - 3b - 5a - 7a - 7b -7c - 9a - 9b - 10 - 12
CMSC 345, sample
Version 4/12
data: 3, 5, 4
282009
D. Cheslock, Northrup Grumman,
Metrics – Cyclomatic Complexity
The minimum number of tests needed to execute
all statements
CC = number_edges – number_nodes + 2
OR
CC = number_decisions + 1
OR
the number of regions
CMSC 345, Version 4/12
29
Black Box Testing
input
output
Cannot “see inside” code unit
 Test cases are:

 based
on the system specification and/or
interfaces (preconditions, postconditions,
invariants, and parameters)

Use equivalence partitions when
conducting black box testing
CMSC 345, Version 4/12
30
Equivalence Partitions and Boundary Testing
Equivalence Partition
 A group of input values that will produce
equivalent behavior
 Example
 Square
root function
the set of all numbers >= zero
 the set of all negative numbers


Test cases should be chosen from each
partition, especially at the boundaries.
CMSC 345, Version 4/12
31
Equivalence Partitions and Boundary Testing
Exercise
 A system accepts 4 to 10 inputs that are 5digit integers >= 10,000
 Partition the system inputs into groups
(partitions) that should cause equivalent
behavior. Include both valid and invalid
inputs.
Ian Somerville, Software Engineering,
6th edition
CMSC 345, Version 4/12
32
The six partitions with
corresponding boundary values
3
4
Less than 4
7
11
10
Between 4 and 10
More than 10
Number of input values
9999
10000
Less than 10000
50000
100000
99999
Between 10000 and 99999
More than 99999
Input values
Ian Somerville, Software Engineering,
6th edition
CMSC 345, Version 4/12
33
Regression Testing
Code A
Code A
Code C
Code B
Code B
Code D
Code A
Code C
Test 1
Test 1
Test 2
Test 2
Test 1
Test 3
Test 2
Test 4
Test 3
Test 5
Test 4
Test 6
Test 5
Test 7
… and so on
Test 8
CMSC 345, Version 4/12
34
Stress Testing


Example: A child’s car seat
Exercises the system beyond its maximum
design load. Examples:
 Exceed
string lengths
 Store/manipulate more data than in specification
 Load system with more users than in specification


Often causes defects to come to light
Systems should not fail catastrophically.
 Stress
testing checks for unacceptable loss of service
or data.
CMSC 345, Version 4/12
35
Test Case Prioritization
Why?
 To locate defects quickly to:

begin debugging ASAP
 debug and deliver in increments ASAP


To locate the majority of errors quickly
Other
How?
 Can prioritize according to:





module or functional dependencies
module size
module complexity
module suspected to be the most error-prone
most “important” module or functionality
 other
CMSC 345, Version 4/12
36
References
Somerville, I., Software Engineering, 6th ed
 SourceForge.net, JUnit,
http://junit.sourceforge.net, accessed
11/6/07
 JUnit.org, http://www.junit.org, accessed
11/6/07

CMSC 345, Version 4/12
37