CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Chapter 14
CS 8532: Advanced Software Engineering
Dr. Hisham Haddad
Class
will
start
momentarily.
Please Stand By …
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Software Testing Techniques
Discussion of Software
Testing Techniques
Chapter 14
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Review - What is Testing?
- Testing is the process of exercising/examining a program with
the intent of finding errors prior to delivery to the end user.
- Errors may belong to a variety of classes (functional,
behaviour, performance).
- Among others, the objectives of testing are to:
>
>
>
>
find uncovered errors,
demonstrate conformance to requirements,
get insight on performance, and
get indication of quality and reliability.
- A testing strategy is a plan that outlines detailed testing
activities (steps, test case design, test execution, effort, time,
resources), and it results in Test Specification document.
Note: 30% - 40% of development effort is spent on testing!
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Testing Principles
Among others, some basic testing principles include:
- Test cases should be traceable to requirements.
- Design of test cases should starts as early as completing software
requirements and before testing starts.
- Testing should begin “in the small” and progress toward testing “in the
large”.
- Exhaustive testing is not possible. Test selective execution paths.
- Testing should be conducted by independent team.
Pareto Principle: 80% of all uncovered errors during testing is
likely be traceable to 20% of all program components.
See section 5.5.2, page 114.
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Software Testability
As testing become integral part of development, software is
developed with “testability” in mind! (i.e., how easy to test the
program) (evolving mind set)
Testability results from the design (good design leads to better
testability)
Testability attributes (next slides) are mapped back to the quality
of design and implementation.
When testing is performed by an independent tester, the tester
must learn the system, try to “break” it, and keep the testing
quality-driven.
Q: what attributes make software testable?
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Testability Attributes (1)
Characteristics that lead to a testable software:
Operability: (the program operates cleanly)
- no or few known bugs
- no serious bugs to prevent execution of test cases
- ability to conduct simultaneous testing on components
Observability: (the results of each test case are readily observed)
- distinct output for each input
- ability to see system state and variables
- ability to easily identify incorrect outputs
- ability of detect internal errors and report them
- access to source code
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Testability Attributes (2)
Controllability: (the degree to which testing can be automated
and optimized)
- ability to generate all possible outputs (via inputs)
- all code is executable using combination of inputs
- ability to control SW/HW states and variables
- consistent input/output format
- tests may be automated and re-produced
Decomposeability: (component targeted testing)
- modular design of the software
- ability to test modules independently
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Testability Attributes (3)
Simplicity: (reduce complex architecture and logic to simplify
testing)
- functional simplicity (specific functions of requirements)
- structural simplicity (modularized architecture)
- code simplicity (following coding and documentation
standards)
Stability: (few changes are required during testing)
- changes are infrequent, controlled, don’t invalidate test cases
- the program can recover from failures
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Testability Attributes (4)
Understandability: (good knowledge of the design)
- understanding component design
- understanding of component hierarchy
- understanding of design changes
- accessibility to technical documents
> readily available
> well organized
> detailed and accurate content
Q: how about the test cases themselves?
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Test Case Attributes
What is a “good” test?
- It has high probability of finding errors. This requires good
understanding of the software.
- It is not redundant. Each test case must have a well-defined
and unique purpose and objective.
- It is neither too simple nor too complex to avoid error masking
in case of combining test case together.
- It should be the “best of breed” (select the best from among
similar test cases)
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Test Case Design
"Bugs lurk in corners and congregate
at boundaries ...“ Boris Beizer
OBJECTIVE:
to uncover errors
CRITERIA:
in a complete manner
CONSTRAINT:
with a minimum of effort and time
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Exhaustive Testing is not Possible!
- Two nested loops
- Each loop runs 20 iterations
- Four if-then-else constructs are
defined inside the inner loop
- There are about 1014 possible
executable paths!
- If we execute one test per
millisecond, it would take 3,170
years to test this program!!
See page 392, Info Box
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Testing Selected Paths
Selected path
Due to large number of
possible paths, select limited
number of “important” logical
paths for testing!
(e.g., Basic Path Testing)
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Testing Methods
black-box
methods
white-box
methods
Methods
Strategies
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
White-Box Testing (1)
The goal is to ensure that all
statements and conditions have been
executed at least once
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
White-Box Testing (2)
With white-box testing, test
- all independent paths within the module/component
- all decision structures for their true and false values
- all loop structures for their boundaries and operations
- all data structures (arrays, sets, maps, stacks, etc…)
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Why White-Box Testing?
- Logical errors and incorrect assumptions are inversely
proportional to a path's execution probability.
- It is often believed that a logical path is not likely to be
executed when it may be executed on regular basis!
- It is likely that untested paths will contain some typographical
errors.
- Others?
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Basic Path Testing (1)
Basic path method is white-box techniques that enables the
designer to derive a logical complexity measure (number of
independent paths) that can be used for defining basic set of
execution paths.
Steps:
-
Construct flow graph (nodes and edges)
Determine the cyclomatic complexity value
Identify the basic set of execution paths
Design test cases for basic set paths
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Flow (Program) Graph
1
1
2
2
3
4
3
4
5
7
8
6
5
6
7
8
9
9
10
10
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Cyclomatic Complexity
1
Cyclomatic complexity V(G) is a metric that
provides quantitative measure of the logical
complexity of code (that is, number of
independent paths).
2
4
R
1
3
5
R3
7
8
9
10
R
2
R4
Computed as:
6
1. V(G) = E - N + 2
2. V(G) = P + 1
3. V(G) = # of regions on the graph
where E is # of edges
N is # of nodes
P is # of predicate nodes (red nodes)
(for combined conditions, see page 395 figure)
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Basic Set of Paths
1
Independent path: Is a path that includes
a new edge(s) that have not need visited
by other paths.
2
3
4
6
5
7
8
9
10
Since V(G) = 4, there are 4 independent
paths (called Basic Set):
Path 1:
Path 2:
Path 3:
Path 4:
1,2,3,6,7,8,9,10
1,2,3,5,7,8,9,10
1,2,4,8,9,10
1,2,4,8,9,2, ... ,9,10
What is next?
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Design Test Cases
Each test case must force the execution of the intended path.
The goal is to ensure that each statement is tested at least once.
Note: A path my not be tested stand-alone. It must be
tested as part of another path(s) (e.g., loop paths)
See complete example on page 397, figure 14.4, Procedure
average.
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Basic Paths - Notes
You don't need a flow chart,
but the picture will help when
you trace program paths.
1
2
3
4
5
6
7
8
9
10
Count each simple logical test,
compound tests count as 2 or
more nodes.
Basis path testing should be
applied to critical modules.
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Cyclomatic Complexity - Note
A number of industry studies have indicated that the higher
the cyclomatic complexity V(G) of a module, the higher the
probability of errors in that module.
modules
V(G)
modules in this range
are more error prone
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Graph Matrices
• A graph matrix is a square matrix whose size (i.e., number of
rows and columns) is equal to the number of nodes on a flow
graph (see page 399, figure 14.6)
• Each row and column corresponds to an identified node, and
matrix entries correspond to connections (an edge) between
nodes.
• Adding a link weight to each matrix entry, the graph matrix can
be used to calculate useful info such as
–
–
–
–
Probability of execution
Required memory
Processing time
Other resources
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Other Testing Techniques (concepts)
Other White-Box based testing techniques include:
1. Condition Testing: Focus on testing logical conditions
2. Data Flow Testing: Focus on testing variables
3. Loop Testing: Focus on testing loops
Note: these techniques are used to test “suspect” modules
and not necessarily the entire software.
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Condition Testing Concept
Conditions are potential places for errors. Condition testing is
designing test cases to detect errors in conditional structures.
That is,
-
Arithmetic expressions
Relational operators
Operator precedence
Boolean operator (with compound conditions)
Boolean variables (with simple conditions)
Formation of parenthesis
A condition may require several test cases for complete
coverage.
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Data Flow Testing Concept
Data flow testing is designing test cases based on the definition
and use of a variable (called DU testing). The selected path
includes the module that defines the targeted variable and all
modules that make use of that variable.
Data flow testing may be used with paths that include nested
loops and if-structures.
The difficulty with this approach is measuring test coverage and
selecting test paths.
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Loop Testing Concept
Loop testing is designing test cases to validate a loop structure
and it boundaries.
A loop structure can be
- simple
- nested
- concatenated
- unstructured
Each structure requires different test cases.
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Loop Structures
Simple
Loop
Nested
Loops
Concatenated
Loops
Unstructured
Loops
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Testing Simple Loops
Typical strategy to test a simple loop that iterates n times:
1.
2.
3.
4.
5.
Skip the loop entirely
Test only one pass through the loop
Test two passes through the loop
Test m passes through the loop m < n
Test iterations n-1, n, and n+1
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Testing Nested Loops
Typical strategy to test nested loop:
1. Set all outer loops to their minimum iteration parameter
values.
2. Test the the innermost for min+1, typical, max-1, and max
values, while holding the outer loops at their minimum values.
3. Move out one loop and test it as in step 2, holding all other
inner loops at typical values and outer loops at minimum
values. Continue this step until the outermost loop has been
tested.
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Testing Concatenated Loops
If the loops are independent of one another then test them as
simple loop, otherwise, test them as nested loops.
Two loops are dependent is when the loop counter value of first
loop is used to initialize the second loop.
Avoid unstructured loop! They can be re-designed using other
loop structures.
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Black-Box (behavior) Testing (1)
requirements
output
input
events
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Black-Box (behavior) Testing (2)
Black-Box testing focuses on testing functional requirements of
program components.
Test cases should be designed to fully exercise the outlined
functions of the component.
It complements White-Box testing and uncovers:
- Incorrect/missing functions
- Interface errors
- Module initialization and termination errors
- External data errors
- Performance errors
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Black-Box (behavior) Testing (3)
Black-Box testing is applied in later stages of the testing process.
Test cases are designed to answer these questions:
•
•
•
•
•
•
•
How is functional validity tested?
How are system behavior and performance tested?
What classes of input will make good test cases?
Is the system particularly sensitive to certain input values?
How are the boundaries of a data class isolated?
What data rates and data volume can the system tolerate?
What effect will specific combinations of data have on system
operation?
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Black-Box Testing Techniques (1)
Black-Box based testing techniques include:
1. Graph-Based methods: Focus on testing relationship between
modules. Different graphs my be utilized (Transition Flow,
Finite State, Data Flow, Execution Time)
2. Equivalence Partitioning: Focus on dividing input data into
equivalence classes from which test cases are derived to test
different classes of errors.
(classes may include user queries, mouse click,
prompts, output format, inputs, etc…)
(valid and invalid values (or ranges of values) for each
class are defined)
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Black-Box Testing Techniques (2)
3. Boundary Value Analysis: Extends Equivalence partitioning
and focus on testing values at the edges of each partitioning
(ranges of valid and invalid values)
4. Comparison Testing: Focus on testing software systems with
redundancy. (multiple instances of the program)
5. Orthogonal Testing: Focus on minimizing number of test cases
by designing test cases with maximum coverage. (“testing one
item/parameter at a time” vs. “multiple items at a time”) (see
figure 14.10)
See textbook for details.
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
OO Testing
By nature, OO design results in layered subsystems of
collaborating classes
OO testing focuses on uncovering errors in class collaborations.
OO testing starts “in the small” (testing classes and their
collaborations) toward testing “in the large” (testing
interfacing among subsystems)
OO testing is derived by “class state” changes (sequence of
operations to test class states)
Encapsulation can be an obstacle for testing! How?
Inheritance can be an obstacle for testing! How?
White-box techniques can be applied to class operations!
or shouldn't be?
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
OO Testing Methods (1)
1. Fault-based testing:
- The tester looks for plausible faults (part of
the implementation
that may result in defects). To determine
whether these faults
exist, test cases are designed to exercise
the design or code.
- Test cases focus on class methods.
2. Class hierarchy testing:
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
OO Testing Methods (2)
3. Scenario-based testing:
- It focuses on user (actor) interaction (what the user does
not what the system does)
- It uncovers interaction errors among subsystems
- Its test cases are more complex to design (than other methods)
4. Surface and deep structure testing:
- Surface structure testing is black-box testing (focus on
what the user see in (and can do with) the system)
- Deep structure testing is white-box testing (focus on the
technical (analysis and design) details of the system, such
as interactions, behavior, and dependencies)
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Class-Level Testing Methods (2)
1. Random testing at class-level:
- identify operations applicable to a class.
- define constraints on their use.
- identify a minimum test sequence
(random sequence of
operation) that defines the minimum life
history of the class
(object).
- generate a variety of random (but valid) test
sequences to
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Class-Level Testing Methods (2)
2. Partition testing (grouping of class operations):
- It is similar to equivalence partitioning for conventional
software.
- It reduces the number of test cases required to test a class.
- State-based partitioning groups and test operations based on
their ability to change the state of the class.
- Attribute-based partitioning groups and tests operations based
on the attributes that they use in the class.
- Category-based partitioning groups and tests operations
based on the generic function each operation performs.
See class “Account” testing sequences page 416.
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Inter-Class Testing Methods (1)
1. Inter-Class testing: (multi-class integration testing)
- For each class, use the list of class operations to generate a
series of random test sequences.
- Use the operations to send messages to other classes.
- For each generated message, determine the collaborator class
and the corresponding operator in that class.
- For each operation in the called class, determine the messages
that it transmits.
- For each of the messages, determine the next level of
operations that are invoked and incorporate these into the test
sequence.
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Inter-Class Testing Methods (2)
2. Behavior-based integration testing (state diagram based)
- Test sequences are derived based on the class behavior and
the classes it collaborates with.
- The tests should achieve all state coverage (for examples,
the operation sequences should cause the “Account” class
to make transition through all allowable states)
(see figure 14.12 for Account class states)
- In some cases, multiple state diagrams may be need to
derive test sequences.
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Specialized Testing
Black and white box testing may be specialized for different
environments, architectures and applications.
See textbook for testing guideline for (page 420):
1.
2.
3.
4.
GUI development environment
Client/Server architecture
Documentation and help application
Real-Time system
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Testing Patterns
Test patterns may be developed to define generic templates for
designing test cases, similar to analysis and design patterns.
Many testing pattern have been proposed in the literature. See
textbook, page 424, for three example patterns:
CS 8532: Adv. Software Eng. – Spring 2009
Dr. Hisham Haddad
Suggested Problems
Consider working the following problems from chapter 14,
page 427:
3, 9, 10, 11, and 12.
No submission is required for practice assignments. Work it
for yourself!