A Review of Software Testing
- P David Coward
Reprinted: Information and Software
Technology; Vol. 30, No. 3 April 1988
Software Engineering: The
Development Process, Vol 1,
Chapter 7
Presented By: Andrew Diemer
Software Engineering II – EEL 6883
Aim of paper
 No guarantee that software meets
functional requirements
 Introduces software testing
techniques
Needs

Software is to be correct
–
What does this mean
 It often means the program
matches the specifications.

Problem with specification
– Specification could be wrong
Needs

If this happens then the
correctness is measured by the
software meeting the user
requirements
Needs

Testing
–
Why test
 Tests may have not been
adequate enough

Asses the performance of the
tasks
Terminology

Verification –vs- Validation

Verification
– Ensures correctness from phase
to phase of the software life cycle
process
– Formal proofs of correctness
Terminology

Validation
– Checks software against
requirements
 Executes software with test data

Author uses testing and checking
instead of verification and validation
Categories of Testing

Two categories of testing:
– Functional
– Non-functional

Functional Testing
Functional
– Addresses to see if the program
obtains the correct output
 It is normally used when testing a
modified or new program

Functional Testing
Regression Testing
– Tests following modification
– Tests to see if the unchanging
functions have indeed changed
Non-functional Requirements
 Style
 Documentation standards
 Response times
 Legal obligations
Situation testing
 Two situations testing can fall
under:
– Testing which finds faults in the
software
– Testing which does NOT find
faults in the software

Situation testing
Finding faults
– Destructive process
 more probing

Not finding faults
– miss inherent faults
 too gentle



Questions
How much testing is needed?
Confidence in testing?
Ignore faults?
– Which ones are important?

Are there more faults?
 What is the purpose of this
testing?
Testing Strategies

Functional -vs- Structural
 Static -vs- Dynamic analysis
Strategy starting points

Specification
– It makes known the required
functions
– Asses to see if they are provided
– Functional testing
Strategy starting points

Software
– Tests the structure of the system
– Structural testing
– Functions are included into the
system but are NOT required
– Example: accessing a database
that has not been asked by the user
Functional testing

Identify the functions which the
software is expected to perform
 Creating test data that will check
to see if these functions are
performed by the software
 Does NOT matter how the
program performs these functions
Functional testing

Rules may be applied to uncover
the functions
 Functional testing methods of
formal documentation that includes
descriptions of faults that correlate
to each part of the design and the
design features themselves
Functional testing

Isolation of these particular
properties of each function should
take place
 Fault class associations
 Black box approach
 Testers have an understanding of
what the output should be
Functional testing

Oracle
– An expert on what the outcome of
a program will be for a particular
test
When might the oracle
approach not work?

– Simulation testing
 Only provides a “range of values”
Structural testing

Testing is based on the detailed
design rather than the functions
required by the program
Structural testing

Two approaches for this testing
– First and most common is to
execute the program with test
cases
– Second is symbolic execution
 Functions of the program are
compared to the required functions
for congruency
Structural testing

May require single path or
percentage testing
 Research has been conducted to
find out what the minimum amount
of testing would be to ensure a
degree of reliability
Structural testing

Measure of reliability
– All statements should be
executed at least once
– All branches should be executed
at least once
– All linear code sequence and
jumps in the program should be
executed at least once
Structural testing

Measure of reliability (cont.)
– Best approach would be the
exhaustive approach in which every
path is tested
Structural testing

Problems with the exhaustive
approach
– Extensive number of paths
– Multiple combinations constitutes
multiple conditions
– Infeasible paths
 Contradictions of predicates at
conditional statements
Structural testing

Path issues
– There is a path for a loop not
executing, executing once, and
executing multiple of times
– Control loops determine the
number of paths
Structural testing

Path issues
– Known as the “level-i” path or
island code
– Island code
 A series of lines of code, following
a program termination, which is not
the destination of a transfer control
from somewhere else in the
program
Structural testing

Path issues
– When does island code occur?
 When failing to delete redundant
code after maintenance
Static analysis

Does NOT involve execution of
software with data but involves the
use of constraints on the input and
output data sets mathematically on
software components
 Examples of static analysis would
be program proving and symbolic
execution
Static analysis

Symbolic execution
– Use symbolic values for variables
instead of numeric or string values
Dynamic analysis

Relies on program statements
which make calls to analysis
routines
 They record the frequency of
execution of elements of the
program
Dynamic analysis

Act as a liaison between
functional and structural testing
 Test cases monitored
dynamically, then structurally tested
to see what idle code is left by
previous tests
 Shows functions the program
should perform
Classification of Techniques

There are three classifications:
 Static – Structural
–
–
–
–
Symbolic execution
Partition analysis
Program proving
Anomaly analysis
Classification of Techniques

Dynamic - Functional
–
–
–
–
Domain testing
Random testing
Adaptive perturbation
Cause-effect graphing
Classification of Techniques

Dynamic - Structural
– Domain and computational
testing
– Automatic test data generation
– Mutation analysis
Classification of Techniques

Dynamic - Structural
– Domain and computational
testing
– Automatic test data generation
– Mutation analysis
Symbolic execution

Non traditional approach
– traditional is the execution
requires that a selection of paths
through a program is exercised by a
set of test classes

Produces a set of expressions,
one per output variable
Symbolic execution

Usually a program executes using
input data values with the output
resulting in a set of actual values
 Use of flow-graphs
– Each branch contains decision
points (directed graph)
– Branch predictions are produced
Symbolic execution

Use of top down approach
 During the top down traversal, the
input variable is given a symbol in
place of an actual value
 A problem is in the iterations
 As mentioned before, no
executing loop, executing once, and
then executing multiple times.
Partition analysis

Uses symbolic execution to find
sub domains of the input domain
 Path conditions are used to find
them
 Input domains that cannot be
allocated to a sub domain infer a
fault
Partition analysis

Specifications need to be written
at a higher lever
Program proving

At the beginning and end of the
procedure, place a mathematical
method assertions
 Similar to symbolic execution
 Neither of them execute actual
data and both examine source code
Program proving

Tries to come up with a proof that
encompasses all possible iterations
 Program proving steps:
– Construct a program
– Examine the program and insert
mathematical assertions at the
beginning and end of procedures
Program proving

Program proving steps (cont):
– Determine whether the code
between each pair of start and end
assertions will achieve the end
assertion given the start assertion
– If the code reaches the end
assertion then the block has been
proven
Program proving

DeMillo says that proofs can only
be stated as acceptable and not
correct
 His acceptance is determined by
a gathering of people who cannot
find fault with the proof
Program proving

The larger the audience, the more
confidence in the software
 Total correctness means loops
will terminate
Anomaly analysis

Two levels of anomalies:
– made by the compiler (language
syntax)
– problems that are not necessarily
wrong by the programming
language
Anomaly analysis

Some anomalies are:
–
–
–
–
–
Unexecutable code
Array boundaries
Initializing variables wrong
Unused labels and variables
Traversing in and out of loops
Anomaly analysis

Produce flow-graphs
 Determine infeasible paths
 Some use data-flow analysis
– Where the input values turn into
intermediate, which then turn into
output values
Anomaly analysis

Some data-flow anomalies:
– Assigning values to variables
which are not used
– Using variables that have not
been assigned previously to a value
– Re-assigning of a variable without
making use of a previous variable
– Indicates possible faults
Domain testing

Based upon informal
classifications of the requirements
 Test cases executed and
compared against the expected to
determine whether faults have been
detected
Random testing

Produces data without reference
to code or specification
 Random number generation is
used
 Main problem is there is no
complete coverage guarantee
Random testing

Key is to examine small subsets
 If followed it will give you a high
branch coverage success rate
Adaptive perturbation testing

Concerns with assessing the
effectiveness of sets of test cases
 Used to generate further test
cases for effectiveness
Adaptive perturbation testing

Optimization is reached when
routines find the best value to
replace the discarded value so the
number of assertions is maximized
 Process is replaced until the
violated assertions are maximized to
the limit
Cause-effect graphing

Power comes from the input
combinations used by logic (AND,
OR, NOT, etc)
Cause-effect graphing

Five steps:
– Divide into workable pieces
– Identify cause and effect
– Represent a graph to link cause
and effect semantics
Cause-effect graphing

Five steps (cont):
– Annotate to show impossible
combinations and effects
– Convert the graph into a limitedentity decision table

It helps identify small test cases
Domain and computational testing

Based upon selecting test cases
 Assignment statements are used
for deciding computational paths
Domain and computational testing

Paths considered:
– Path computation
 Set of algebraic expressions, one
for each output variable, in terms of
input variables and constraints
– Path condition
 A joining of constraints of a path
Domain and computational testing

Paths considered (cont):
– Path domain
 Set of input values that satisfy the
path condition
– Empty path
 Infeasible paths and cannot
execute

A path that follows errors is a
computation error
Automatic test data generation

Courage comes with covering
metrics
 Contradictory paths are infeasible
 Needs detailed specification to
achieve this testing
 Formal specifications may provide
fundamental help
Mutation analysis

Concerns the quality of sets of
test data
 Uses the program to test the test
data
 This means the original and
mutant program are tested using the
same test data
Mutation analysis

The two outputs are compared
 If the mutant output is different
from the original output, then the
mutant is of little value
 If the two outputs are the same,
then the problem is there has been
a change
Mutation analysis

The mutant is then said to be live
 Ratios are then taken (dead/alive)
 High ratio of live mutants equals
poor test data
 If this happens then more tests
need to be run until the ratio goes
down
Conclusion

I thought this paper was thorough
 This paper gave good examples
(compartmentalized)
 I thought this paper was a little
out of date