CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Chapter 14
Software Testing Techniques
Highlights of Software Testing Techniques
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Review - What is Testing?
- Testing is the process of exercising/examining a program with
the intent of finding errors prior to delivery to the end user.
- Errors may belong to a variety of classes (functional,
behaviour, performance).
- Among others, the objectives of testing are to:
> find uncovered errors,
> demonstrate conformance to requirements, and
> get indication of quality and reliability.
- A testing strategy is a plan that outlines detailed testing
activities (steps, test case design, test execution, effort, time,
resources), and it results in Test Specification document.
Note: 30% - 40% of development effort is spent on testing!
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Testing Principles
Among others, some basic testing principles include:
- Test cases should be traceable to requirements.
- Design of test cases should starts as early as completing
software requirements and before testing starts.
- Testing should begin “in the small” and progress toward testing
“in the large”.
- Exhaustive testing is not possible. Test selective execution
paths.
- Testing should be conducted by independent team.
Pareto Principle: 80% of all uncovered errors during testing is
likely be traceable to 20% of all program components.
See section 5.5.2, page 114.
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Software Testability
As testing become integral part of development, software is
developed with “testability” in mind! (how easy to test the
program)
Testability results from the design (good design leads to better
testability)
Testability attributes (next slides) are mapped back to the quality
of design and implementation.
When testing is performed by an independent tester, the tester
must learn the system, try to “break” it, and keep the testing
quality-driven.
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Testability Attributes (1)
Characteristics that lead to a testable software:
Operability: (the program operates cleanly)
- no or few known bugs
- no serious bugs to prevent execution of test cases
- ability to conduct simultaneous testing on components
Observability: (the results of each test case are readily observed)
- distinct output for each input
- ability to see system state and variables
- ability to easily identify incorrect outputs
- ability of detect internal errors and report them
- access to source code
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Testability Attributes (2)
Controllability: (the degree to which testing can be automated
and optimized)
- ability to generate all possible outputs (via inputs)
- all code is executable using combination of inputs
- ability to control SW/HW states and variables
- consistent input/output format
- tests may be automated and re-produced
Decomposeability: (component targeted testing)
- modular design of the software
- ability to test modules independently
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Testability Attributes (3)
Simplicity: (reduce complex architecture and logic to simplify
testing)
- functional simplicity (specific functions of requirements)
- structural simplicity (modularized architecture)
- code simplicity (following coding and documentation
standards)
Stability: (few changes are required during testing)
- changes are infrequent, controlled, don’t invalidate test cases
- the program can recover from failures
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Testability Attributes (4)
Understandability: (good knowledge of the design)
- understanding component design
- understanding of component hierarchy
- understanding of design changes
- accessibility to technical documents
> readily available
> well organized
> detailed and accurate content
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Test Case Attributes
What is a “good” test?
- It has high probability of finding errors. This requires good
understanding of the software.
- It is not redundant. Each test case must have a well-defined
and unique purpose and objective.
- It is neither too simple nor too complex to avoid error masking
in case of combining test case together.
- It should be the “best of breed” (select the best from among
similar test cases)
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Test Case Design
"Bugs lurk in corners and congregate
at boundaries ...“ Boris Beizer
OBJECTIVE:
to uncover errors
CRITERIA:
in a complete manner
CONSTRAINT:
with a minimum of effort and time
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Exhaustive Testing is not Possible!
- Two nested loops
- Each loop runs 20 iterations
- Four if-then-else constructs are
defined inside the inner loop
- There are about 1014 possible
executable paths!
- If we execute one test per
millisecond, it would take 3,170
years to test this program!!
See page 392, Info Box
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Testing Selected Paths
Selected path
Due to large number of
possible paths, select limited
number of “important” logical
paths for testing!
(e.g., Basic Path Testing)
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Testing Methods
black-box
methods
white-box
methods
Methods
Strategies
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
White-Box Testing (1)
The goal is to ensure that all
statements and conditions have been
executed at least once
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
White-Box Testing (2)
With white-box testing, test
- all independent paths within the module/component
- all decision structures for their true and false values
- all loop structures for their boundaries and operations
- all data structures (arrays, sets, maps, stacks, etc…)
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Why White-Box Testing?
- Logical errors and incorrect assumptions are inversely
proportional to a path's execution probability.
- It is often believed that a logical path is not likely to be
executed when it may be executed on regular basis!
- It is likely that untested paths will contain some typographical
errors.
- Others?
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Basic Path Testing (1)
Basic path method is white-box techniques that enables the
designer to derive a logical complexity measure (number of
independent paths) that can be used for defining basic set of
execution paths.
Steps:
-
Construct flow graph (nodes and edges)
Determine the cyclomatic complexity value
Identify the basic set of execution paths
Design test cases for basic set paths
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Flow (Program) Graph
1
1
2
2
3
4
3
4
5
7
8
6
5
6
7
8
9
9
10
10
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Cyclomatic Complexity
1
Cyclomatic complexity V(G) is a metric that
provides quantitative measure of the logical
complexity of code (that is, number of
independent paths).
2
4
R
1
3
5
R3
7
8
9
10
R
2
R4
Computed as:
6
1. V(G) = E - N + 2
2. V(G) = P + 1
3. V(G) = # of regions on the graph
where E is # of edges
N is # of nodes
P is # of predicate nodes (red nodes)
(for combined conditions, see page 395 figure)
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Basic Set of Paths
1
Independent path: Is a path that includes
a new edge(s) that have not need visited
by other paths.
2
3
4
6
5
7
8
9
10
Since V(G) = 4, there are 4 independent
paths (called Basic Set):
Path 1:
Path 2:
Path 3:
Path 4:
1,2,3,6,7,8,9,10
1,2,3,5,7,8,9,10
1,2,4,8,9,10
1,2,4,8,9,2, ... ,9,10
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Design Test Cases
Each test case must force the execution of the intended path.
Note: A path my not be tested stand-alone. It must be
tested as part of another path(s) (e.g., loop paths)
See complete example on page 397, figure 14.4, Procedure
average.
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Basic Paths - Notes
You don't need a flow chart,
but the picture will help when
you trace program paths.
1
2
3
4
5
6
7
8
9
10
Count each simple logical test,
compound tests count as 2 or
more nodes.
Basis path testing should be
applied to critical modules.
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Graph Matrices
• A graph matrix is a square matrix whose size (i.e., number of
rows and columns) is equal to the number of nodes on a flow
graph (see page 399, figure 14.6)
• Each row and column corresponds to an identified node, and
matrix entries correspond to connections (an edge) between
nodes.
• Adding a link weight to each matrix entry, the graph matrix can
be used to calculate useful info such as
–
–
–
–
Probability of execution
Required memory
Processing time
Other resources
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Cyclomatic Complexity - Note
A number of industry studies have indicated that the higher
the cyclomatic complexity V(G) of a module, the higher the
probability of errors in that module.
modules
V(G)
modules in this range
are more error prone
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Other Testing Techniques (concepts)
Other White-Box based testing techniques include:
1. Condition Testing: Focus on testing logical conditions.
2. Data Flow Testing: Focus on testing variables.
3. Loop Testing: Focus on testing loops.
Note: these techniques are used to test “suspect” modules
and not necessarily the entire software.
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Condition Testing Concept
Conditions are potential places for errors. Condition testing is
designing test cases to detect errors in conditional structures.
That is,
-
Arithmetic expressions
Relational operators
Boolean operator (with compound conditions)
Boolean variables (with simple conditions)
Formation of parenthesis
A condition may require several test cases for complete
coverage.
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Data Flow Testing Concept
Data flow testing is designing test cases based on the definition
and use of a variable. The selected path includes the module
that defines the targeted variable and all modules that make
use of that variable.
Data flow testing may be used with paths that include nested
loops and if-structures.
The difficulty with this approach is measuring test coverage and
selecting test paths.
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Loop Testing Concept
Loop testing is designing test cases to validate a loop structure
and it boundaries.
A loop structure can be
- simple
- nested
- concatenated
- unstructured
Each structure requires different test cases.
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Loop Structures
Simple
Loop
Nested
Loops
Concatenated
Loops
Unstructured
Loops
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Testing Simple Loops
Typical strategy to test a simple loop that iterates n times:
1.
2.
3.
4.
5.
Skip the loop entirely
Test only one pass through the loop
Test two passes through the loop
Test m passes through the loop m < n
Test iterations n-1, n, and n+1
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Testing Nested Loops
Typical strategy to test nested loop:
1. Set all outer loops to their minimum iteration parameter
values.
2. Test the the innermost for min+1, typical, max-1, and max
values, while holding the outer loops at their minimum values.
3. Move out one loop and test it as in step 2, holding all other
inner loops at typical values and outer loops at minimum
values. Continue this step until the outermost loop has been
tested.
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Testing Concatenated Loops
If the loops are independent of one another then test them as
simple loop, otherwise, test them as nested loops.
Two loops are dependent is when the loop counter value of first
loop is used to initialize the second loop.
Avoid unstructured loop! They can be re-designed using other
loop structures.
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Black-Box (behavior) Testing (1)
requirements
output
input
events
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Black-Box (behavior) Testing (2)
Black-Box testing focuses on testing functional requirements of
program components.
Test cases should be designed to fully exercise the outlined
functions of the component.
It complements White-Box testing and uncovers:
- Incorrect/missing functions
- Interface errors
- Module initialization and termination errors
- External data errors
- Performance errors
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Black-Box (behavior) Testing (3)
Black-Box testing is applied in later stages of the testing process.
Test cases are designed to answer these questions:
•
•
•
•
•
•
•
How is functional validity tested?
How are system behavior and performance tested?
What classes of input will make good test cases?
Is the system particularly sensitive to certain input values?
How are the boundaries of a data class isolated?
What data rates and data volume can the system tolerate?
What effect will specific combinations of data have on system
operation?
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Black-Box Testing Techniques (1)
Black-Box based testing techniques include:
1. Graph-Based methods: Focus on testing relationship between
modules. Different graphs my be utilized (Transition Flow,
Finite State, Data Flow, Execution Time)
2. Equivalence Partitioning: Focus on dividing input data into
equivalence classes from which test cases are derived to test
different classes of errors.
(classes may include user queries, mouse click,
prompts, output format, inputs, etc…)
(valid and invalid values (or ranges of values) for each
class are defined)
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Black-Box Testing Techniques (2)
3. Boundary Value Analysis: Extends Equivalence partitioning
and focus on testing values at the edges of each partitioning
(ranges of valid and invalid values)
4. Comparison Testing: Focus on testing software systems with
redundancy. (multiple instances of the program)
5. Orthogonal Testing: Focus on minimizing number of test cases
by designing test cases with maximum coverage. (“testing one
item at a time” vs. “multiple items at a time”) (see figure 14.10)
See textbook for details.
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
OO Testing
By nature, OO design results in layered subsystems of
collaborating classes
OO testing focuses on uncovering errors in class collaborations.
OO testing starts “in the small” (testing classes and their
collaborations) toward testing “in the large” (testing
interfacing among subsystems)
OO testing is derived by “class state” changes (sequence of
operations to test class states)
Encapsulation can be an obstacle for testing! How?
Inheritance can be an obstacle for testing! How?
White-box techniques can be applied to class operations!
or shouldn't be?
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
OO Testing Methods (1)
1. Fault-based testing:
- The tester looks for plausible faults (part of the implementation
that may result in defects). To determine whether these faults
exist, test cases are designed to exercise the design or code.
- Test cases focus on class methods.
2. Class testing and the class hierarchy:
- Inheritance complicates testing since testing derived classes
does not preclude the need to test derived classes.
- Test cases for a base class may be used to test a derived
class, but expect different results.
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
OO Testing Methods (2)
3. Scenario-based testing:
- It focuses on user (actor) interaction (what the user does
not what the system does)
- It uncovers interaction errors among subsystems
- Its test cases are more complex to design (than other methods)
4. Surface and deep structure testing:
- Surface structure testing is black-box testing (focus on
what the user see in (and can do with) the system)
- Deep structure testing is white-box testing (focus on the
technical (analysis and design) details of the system, such
as interactions, behavior, and dependencies)
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Class-Level Testing Methods (1)
1. Random testing:
- identify operations applicable to a class.
- define constraints on their use.
- identify a minimum test sequence (sequence of operation)
that defines the minimum life history of the class (object).
- generate a variety of random (but valid) test sequences to
exercise other (more complex) class instance life histories.
See class “Account” testing sequences page 415.
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Class-Level Testing Methods (2)
2. Partition testing:
- It is similar to equivalence partitioning for conventional
software.
- It reduces the number of test cases required to test a class.
- State-based partitioning groups and test operations based on
their ability to change the state of the class.
- Attribute-based partitioning groups and tests operations based
on the attributes that they use in the class.
- Category-based partitioning groups and tests operations
based on the generic function each operation performs.
See class “Account” testing sequences page 416.
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Inter-Class Testing Methods (1)
1. Inter-Class testing: (multi-class integration testing)
- For each class, use the list of class operations to generate a
series of random test sequences.
- Use the operations to send messages to other classes.
- For each generated message, determine the collaborator class
and the corresponding operator in that class.
- For each operation in the called class, determine the messages
that it transmits.
- For each of the messages, determine the next level of
operations that are invoked and incorporate these into the test
sequence.
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Inter-Class Testing Methods (2)
2. Behavior-based integration testing
- Test sequences are derived based on the class behavior and
the classes it collaborates with.
- The tests should achieve all state coverage (for examples,
the operation sequences should cause the “Account” class
to make transition through all allowable states)
(see figure 14.12 for Account class states)
- In some cases, multiple state diagrams may be need to
derive test sequences.
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Specialized Testing
Black and white box testing may be specialized for different
environments, architectures and applications.
See textbook for testing guideline for (page 420):
1.
2.
3.
4.
GUI development environment
Client/Server architecture
Documentation and help application
Real-Time system
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Testing Patterns
Test patterns may be developed to define generic templates for
designing test cases, similar to analysis and design patterns.
Many testing pattern have nee proposed in the literature. See
textbook, page 424, for three example patterns:
Pair testing: It describes a technique in which two testers work together
to design and execute a series of tests for unit, integration, or
validation testing activities.
Separate test interface: It describes how to create a test interface that
can be used to describe specific tests for classes local to a
component.
Scenario testing: It describes a technique for exercising the software
from the user’s point of view. A failure at this level indicates that the
software has failed to meet a user visible requirement.
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Suggested Problems
Consider working the following problems from chapter 14,
page 427:
3, 9, 10, 11, and 12.
No submission is required for practice assignments. Work it
for yourself!
CS 3610: Software Engineering – Spring 2009
Dr. Hisham Haddad – CSIS Dept.
Last Slide
End of chapter 14