ESE Module 6-2
Software Testing Methods
Everyone who creates computer programs performs
some testing, but a significantly smaller number of
people explicitly design tests using test case design
methods. Why? Because the majority of software
developers know relatively little about test case
design.
There are two broad categories into which we
can place the majority of test case design methods.
The first--white-box testing--is predicated on a knowledge of the internal processing details of a given
module. We design test cases using the program logic
as a guide.
The second category-black-box methods-encompasses a set of test case design methods that
focus on user requirements (often driven by tests associated with inputs and outputs of a system).
Some people argue that these categories are
overly simplistic; and they may be right. Black-box
testing can be used to exercise an individual module
(rather than an entire program or system) and whitebox tests can be adapted for use in designing tests for
an entire system. However, the categories help us to
organize our thinking as we consider test case design;
and for that reason, we'll use them in this ESE module.
hood of finding the most errors, with a minimum amount
of time and effort.
Over the past decade, a rich variety of test case design
methods has evolved for software. These methods provide
the developer with a systematic approach to testing. More
importantly, methods provide a mechanism that can help
to ensure the completeness of tests and provide the highest
likelihood for uncovering errors in software.
Any engineered product (and most other things) can
be tested in one of two ways: (1) knowing the specified
function that a product has been designed to perform, tests
can be conducted to demonstrate each function is fully
operational; (2) knowing the internal workings of a product, tests can be conducted to ensure that all gears mesh,
that is, internal operation performs according to specification and all internal components have been adequately
exercised. The first test approach is called black-box testing
and the second, white- box testing.
When computer software is considered, black- box
testing alludes to tests that are conducted at the software
interface. Although they are designed to uncover errors,
black-box tests are used to demonstrate that software functions are operational; that input is properly accepted and
output is correctly produced; and that the integrity of
external information (e.g., data files) is maintained. A
black-box test examines some aspect of the fundamental
Test Case Design
system, with little regard for the internal logical structure
of the software.
The design of test cases is a notion that has not perWhite-box testing of software is predicated on close
meated the software development community. Most
examination
of procedural detail. Logical paths through
people continue to create test cases using an ad
the
software
are
tested by providing test cases that exercise
hoc, intuitive approach. Although this works in its own
specific
sets
of
conditions
and/or loops. The "status of the
fashion, testing is rarely as effective as the tester
program"
may
be
examined
at various points to determine
would like.
if
the
expected
or
asserted
status
corresponds to the actual
Test case design demands a systematic
status.
approach to testing. Rather than simply "creating a
At first glance, it would seem that very thorough
few test cases, " the software engineer uses characterwhite-box
testing would lead to "100 percent correct proistics of the program logic, the input domain, and
grams."
All
we need do is define all logical paths, develop
even the program output to guide the design of test
test
cases
to
exercise them, and evaluate results; that is,
cases. The result is a set of tests that has a higher probgenerate
test
cases to exercise program logic exhaustively.
ability of finding errors.
Unfortunately, exhaustive testing presents certain logistical
problems. For even small programs, the number of possible logical paths can be very large. The procedural design
for a 100-line Pascal program, with a single loop that may
be executed no more than 20 times and four conditions
The following excerpt has been adapted from
nested in the loop, could have as many as 1014 possible
Software Engineering: A Practitioner's Approach and
considers some of the basic issues associated with test paths that may be executed!
To put this number in perspective, we assume that a
case design.
magic test processor ("magic" because no such processor
The design of tests for software and other engineered prodexists) has been developed for exhaustive testing. The
ucts can be as challenging as the initial design of the prodprocessor can develop a test case, execute it, and evaluate
uct itself. Yet for reasons that we have already discussed,
the results in one millisecond. Working 24 hours a days,
software engineers often treat testing as an afterthought,
365 days a year, the processor would work for 3,170 years
developing test cases that may "feel right" but have little
to test the program. This would, undeniably, cause havoc
assurance of being complete. Recalling the objectives of
in most development schedules. Exhaustive testing is
testing, we must design tests that have the highest likeliimpossible for large software systems.
White-box testing should not, however, be dismissed
Readings
6-2.2 ·· Essential Software Engineering
as impractical. A limited number of important logical
paths can be selected and exercised. Important data structures can be probed for validity. The attributes of both
black-box and white-box testing can be combined to provide an approach that validates the software interface and
selectively assures that the internal workings of the software are correct.
White-Box Testing
White-box testing is a category of test case design methods
that uses the control structure of the procedural design to
derive test cases. Using white- box testing methods, the
software engineer can derive test cases that (1) guarantee
that all independent paths within a module have been exercised at least once; (2) exercise all logical decisions on their
true and false sides; (3) execute all loops at their boundaries and within their operational bounds, and (4) exercise
internal data structures to assure their validity.
A reasonable question might be posed at this juncture:
"Why spend time and energy worrying about (and testing)
logical minutiae when we might better expend effort assuring that program requirements have been met?" Stated
another way, Why don't we spend all our energies on
black-box tests? The answer lies in the nature of software
defects:
Readings
The following excerpt has been adapted from
Software Engineering: A Practitioner's Approach and
presents an overview of loop testing, one of the whitebox testing methods presented in the preceding
video segment.
Loops are the cornerstone for the vast majority of all algorithms implemented in software. And yet, we often pay
them little heed while conducting software testing.
Loop testing is a white box testing technique that
focuses exclusively on the validity of loop constructs. Four
different classes of loops can be defined: simple loops, concatenated loops,nested loops and unstructured loops [see
SAPE].
Simple Loops. The following set of tests should be applied
to simple loops, where n is the maximum number of allowable passes through the loop.
1. Skip the loop entirely.
2. Only one pass through the loop.
3. Two passes through the loop.
4. m passes through the loop where m < n
5. n-l, n, n+l passes through the loop
Nested Loops. If we were to extend the test approach for
simple
loops to nested loops, the number of possible tests
 Logic errors and incorrect assumptions are inversely proportional to the probability that a program path will be executed. would grow geometrically as the level of nesting increases.
This would result in an impractical number of tests. Beizer
Errors tend to creep into our work when we design and
suggests an approach that will help to reduce the number
implement functions, conditions or controls that are outof
tests:
side the mainstream. Everyday processing tends to be well
understood (and well scrutinized) while special case pro1. Start at the innermost loop. Set all other loops to minicessing tends to fall into the cracks.
mum
values.
 We often believe that a logical path is not likely to be
2.
Conduct
simple loop tests for the inner most loop
executed when, infact, it may be executed on a regular basis.
while
holding
the outer loops at their minimum iteration
The logical flow of a program is sometimes counterintuitive,
parameter (e.g., loop counter) values. Add other tests for
meaning our unconscious assumptions about flow of control
out-of-range or excluded values.
and data may lead us to make design errors that are uncov3.
Work outward, conducting tests for the next loop,but
ered only after path testing commences.
keeping
all other outer loops at minimum values and other
 Typographical errors are random. When a program is
nested
loops
to “typical" values.
translated into programming language source code, it is
4.
Continue
until you have tested all loops.
likely that some typing errors will occur. Many will be
uncovered by syntax checking mechanisms, but others will
Concatenated Loops. Concatenated loops can be tested
go undetected until testing begins. It is as likely that a typo
using the approach defined above for simple loops, if each
will exist on an obscure logical path as on a mainstream
loop is independent of the other. However, if two loops are
path.
concatenated and the loop counter for loop 1 is used as the
initial value for loop 2,then the loops are not independent.
Each of these reasons provides an argument for conducting
When
the loops are not independent, the approach applied
white-box tests. Black-box testing, no matter how thorto
nested
loops is recommended.
ough, may miss the kinds of errors noted above. As Beizer
[1] has stated: "Bugs lurk in corners and congregate at
boundaries." White-box testing is far more likely to uncover them.
[1] Beizer, B., Software Testing Techniques, 2nd edition,
Van Nostrand-Reinhold, 1990.
Software Testing Methods · · 6-2.3
Unstructured Loops. Whenever possible, this class of loops
should be redesigned to reflect the use of the structured
programming constructs. O
Exercise 6-3,
White-Box Testing
In the video segment that you just viewed, Dr.
Pressman discussed basic path testing. Review
the following
during later stages of testing. Because black-box testing
purposely disregards control structure, attention is focused
on the information domain. Tests are designed to answer
the following questions:
 How is functional validity tested?
 What classes of input will make good test cases?
 Is the system particularly sensitive to certain input values?
 How are the boundaries of a data class isolated?
 What data rates and data volume can the system tolerate?
 What effect will specific combinations of data have on
system operation?
By applying black-box techniques, we derive a set of test
cases that satisfies the following criteria: (1) test cases that
reduce, by a count that is greater than one, the number of
additional test cases that must be designed to achieve reasonable testing, and (2) test cases that tell us something
about the presence or absence of classes of errors, rather
than an error associated only with the specific test at hand.
Equivalence Partitioning
Number each processing box and each condition
diamond. Assume that each decision diamond represents
a simple Boolean condition.
1. What is the cyclomatic complexity of the module?
2. Identify a set of basis paths for the module.
3. How many additional tests would you recommend to fully exercise the simple loop shown in the
procedural logic for the flowchart?
Readings
The following excerpt, adapted from Software
Engineering: A Practitioner's Approach, presents an
overview of black-box testing and two of the most
common black-box testing methods.
Black-box testing methods focus on the functional requirements of the software. That is, black-box testing enables the
software engineer to derive sets of input conditions that
will fully exercise all functional requirements for a program. Black-box testing is not an alternative to white-box
techniques. Rather, it is a complementary approach that is
likely to uncover a different class of errors than white-box
methods.
Black-box testing attempts to find errors in the following categories: (1) incorrect or missing functions; (2) interface errors; (3) errors in data structures or external database
access; (4) performance errors; and (5) initialization and termination errors.
Unlike white-box testing, which is performed early in
the testing process, black-box testing tends to be applied
Equivalence partitioning is a black- box testing method that
divides the input domain of a program into classes of data
from which test cases can be derived. An ideal test case
single-handedly uncovers a class of errors (e.g., incorrect
processing of all character data) that might otherwise
require many cases to be executed before the general error
is observed. Equivalence partitioning strives to define a
test case that uncovers classes of errors, thereby reducing
the total number of test cases that must be developed.
Test case design for equivalence partitioning is based
on an evaluation of equivalence classes for an input condition. An equilanence class represents a set of valid or invalid
states for input conditions. Typically, an input condition
can be a specific numeric value, a range of values, a set of
related values or a Boolean condition (yes or no).
Equivalence classes may be defined according to the following guidelines:
1. If an input condition specifies a range, one valid and
two invalid equivalence classes are defined.
2. If an input condition requires a specific value, one
valid and two invalid equivalence classes are defined.
3. If an input condition specifies a member of a set, one
valid and one invalid equivalence classes are defined.
4. If an input condition is Boolean, one valid and one
invalid class are defined.
As an example, consider data maintained as part of an
automated banking application. The user can dial the bank
using his or her personal computer, provide a 6-digit password, and follow with a series of keyword commands that
triggers various banking functions. The software supplied
for the banking application accepts data in these forms:
6-2.4 ·· Essential Software Engineering
area code -- blank or 3-digit number.
prefix -- 3-digit number not beginning with 0 or 1.
suffix -- 4-digit number.
password -- 6-digit alphanumeric value.
commands -- "check," "deposit," "bill pay," etc.
The input conditions associated with each data element for the banking application can be specified as:
area code: input condition, Boolean
the area code may or may not be present;
input condition, range--values defined between 200
and 999, with specific exceptions (e.g., no values >
905) and requirements (e.g., all area codes have a 0 or 1
as a second digit).
prefix: input condition, range--specified
value > 200 with no 0 digits.
suffix: input condition, value-4-digit length.
password: input condition, Boolean
a password may or may not be present;
input condition, value—6-characder string.
command: input condition, set--containing commands
noted above.
Applying the guidelines for the derivation of equivalence
classes, test cases for each input domain data item could be
developed and executed. Test cases are selected so that the
largest number of attributes of an equivalence class are
exercised at once.
Boundary Value Analysis
For reasons that are not completely clear, a greater number
of errors tends to occur at the boundaries of the input
domain than in the center. It is for this reason that boundary
value analysis (BVA) has been developed as a testing technique. Boundary value analysis leads to a selection of test
cases that exercise bounding values.
Boundary value analysis is a test case design technique
that complements equivalence partitioning. Rather than
selecting any element of an equivalence class, BVA leads to
the selection of test cases at the edges of the class. Rather
than focusing solely on input conditions, BVA derives test
cases from the output domain as well.
Guidelines for BVA are similar in many respects to
those provided for equivalence partitioning:
1. If an input condition specifies a range bounded by values a and b, test cases should be designed with values a
and b, and just above and just below a and b, respectively.
2. If an input condition specifies a number of values, test
cases should be developed to exercise the minimum and
maximum numbers. Values just above and below minimum and maximum are also tested.
3. Apply guidelines 1 and 2 to output conditions. For
example, assume that a temperature vs, pressure table is
required as output from an engineering analysis program.
Test cases should be designed to create an output report
that produces the maximum (and minimum) allowable
number of table entries.
4. If internal program data structures have prescribed
boundaries (e.g., an array has a defined limit of 100
entries), be certain to design a test case to exercise the data
structure at its boundary.
Most software engineers intuitively perform BVA to some
degree. By applying the guidelines noted above, boundary
testing will be more complete, and thereby have a higher
likelihood for error detection.
To fully understand test case design methods (whitebox and/or black-box methods), you'll need to do
some additional reading. A good place to start is with
a general purpose software engineering textbook
(e.g. Software Engineering: A Practitioner's
Approach). You should then move on to one of the
many good books dedicated to software testing.
Exercise 6-4
Black-Box Testing
Consider a program that reads in integer values A, B,
and C. These values represent the side dimensions of
a triangle. The output of the program is TYPE, the
triangle type. TYPE takes on the following values:
TYPE = NT, no triangle is input; EQ, an equilateral
triangle is input; IS, an isosceles triangle is input: SC, a scalene triangle is input; ER, erroneous data is input
1. Build a table containing the values A, B, C, and
TYPE, and Purpose. The Purpose column contains a
description of the purpose of the test.
2. Define as many independent test cases as you
can to fully test the program described above. An
independent test case is one that has a purpose that
is different from any other test case. For example, (A,
B, C) = (1,1,1) would test recognition of equilateral triangles. (2,2,2) would not be an independent test,
because it has the same purpose: but (0,0,0) would
be independent, because it has a different purpose.
3. Grade yourself based on the following ranges:
1 - 6 tests:
7 - 9 tests:
think harder!
OK, but a bit more effort would be
worthwhile
10 - 13 tests: good
14+ tests: very good
Software Testing Methods · ·6-2.5
Post-Test, Module 6-2
This post-test has been designed to help you assess
the degree to which you've absorbed the information
presented in this ESE module.
7. The minimum number of tests required to thoroughly exercise a simple loop is:
a. 1
b. 3
c. 5
d. 7
e. 9
Software testing Methods
1. The objective of testing is to uncover errors. What
is the criteria and constraint?
a. in a complete manner with a minimum of
cost and effort
b, in a complete manner with customer
oversight
c. with 100%coverage and low cost
d. with 95+% coverage and low cost
2. At a minimum, white-box test case design requires
that you have:
a. source code
b. an operational program
c. a detailed procedural design
d. the program architecture
3. An independent program path is:
a. one that executes statements that have not
been executed before
b. one that tests a condition on its true and false
sides
c. one that passes around a loop
d. one that exercises independent input
4. Basis path testing uses cyclomatic complexity to
identify:
a. the number of conditions in the program
b. the basis set of independent paths
c. the basis data
d. a basis from which all errors in a module
can be uncovered
5. If a compound condition is encountered in procedural logic:
a. there is no effect on the value of cyclomatic
complexity
b. cyclomatic complexity value increases
c. cyclomatic complexity decreases
d. compound conditions should not be used
6. Concatenated loops are:
a. nested one inside the other
b. connected via a value in the first that controls
looping in the second
c. never to be used, because they are error
prone
d. always have a higher cyclomatic complexity
than two simple loops
8. Equivalence partitioning is a black-box testing
method that:
a. looks for equivalent data values in the
program
b. looks for classes of output
c. focuses on output errors
d. defines classes of input
9. An equivalence class succeeds if it:
a. uncovers an error that is representative of an
entire class of input errors
b. finds no errors are found in the input class
c. optimizes class performance
d. none of the above
10. Boundary value analysis is closely related to:
a. basis path testing
b. equivalence partitioning
c, loop testing
d. stress testing
11. The intent of black-box testing is to uncover errors:
a. without using information about internal
processing details
b. without using information about software
requirements
c. without running many tests
d. none of the above
12. Assume that a module has 200 lines of C code
that contains 6 if-then-else statements and 3 nested
loops. The best way to test this module would be to
use:
a. basis path testing
b. loop testing
c. black-box testing
d. all of the above
e. none of the above
Copyright 1995 R.S. Pressman & Associates, Inc.
No part of this material may be electronically copied,
transmitted, or posted without the express written
permission of R.S. Pressman & Associates, Inc.
These materials have been posted on the Local Web
Server of the Department of Computer Science with
permission of R.S. Pressman & Associates, Inc. and are
for use only by students registered for DCS/235 at
Queen Mary and Westfield College.