CS4723
Software Validation
and Quality Assurance
Lecture 02
Overview of Software Testing
Approach to remove bugs

Testing



Limitations

Impossible to cover all cases

Test oracles (what is expected)
Static checking


2
Feed input to software and run it to see whether its behavior is
as expected
Identify specific problems (e.g., memory leak) in the software by
scanning the code or all possible paths
Limitations

Limited problem types

False positives
Approach to remove bugs

Formal Proof



3
Formally prove that the program implements the
specification
Limitations

Difficult to have a formal specification

The proof cost a lot of human efforts
Inspection

Manually review the code to detect faults

Limitations:

Hard to evaluate

Sometime hard to get progress
Answer is testing, why?

“50% of my employees are testers, and the rest
spends 50% of their time testing”
---- Bill Gates, 1995

More reliable than inspection, relatively cheap


Actually in the old days, when testing is expensive,
inspection was the major answer
You get what you pay (linear rewards)

Compared to other 3 approaches

4
Inspection, static checking, formal proof
Testing: Concepts
5

Test case

Test oracle

Test suite

Test script

Test driver

Test coverage
Testing: Concepts

Test case



Include:

Input values, sometimes fed in different steps

Expected outputs
Test oracle

6
An execution of the software with a given list of input
values
The expected outputs of software by feeding in a list of
input values

A part of test cases

Hardest problem in auto-testing: test oracle problem
Testing: Concepts: Example
7
Testing: Concepts

Test suite


A collection of test cases
Usually these test cases share similar pre-conditions and
configuration

Usually can be run together in sequence

Different test suites for different purposes


Test Script

8
Smoke test, Certain platforms, Certain feature,
performance, …
A script to run a sequence of test cases or a test suite
automatically
Testing: Concepts

Test Driver



9
A software framework that can load a collection of test
cases or a test suite
It can usually handle the configuration and comparison
between expected outputs and actual outputs
Test Coverage

A measurement to evaluate how well the testing is done

The measure can be based on multiple elements

Code

Input combinations

Specifications
Granularity of Testing: V-model
10
Granularity of testing

Unit / Module Testing


Integration Testing


Test the system as a whole, by developers on test cases
Acceptance Testing

11
Test the interaction between modules
System Testing


Test of a single module
Validate the system against user requirements, by
customers with no formal test cases
Stage of software Testing

Development-time testing


Before-release testing


System testing, Acceptance Testing
User testing

12
Unit testing, Integration Testing
Actual usage -> field bugs & patches
Types of testing by
how they are designed


13
White box testing

The tester knows everything about the implementation

They knows where the bugs are more probably be

They can exercise paths in the code
Black box testing

The tester are just like normal users

They just try to cover input space and corner cases
Black Box Testing: General Guidelines

Divide value range and cover each part

Cover boundary values

Try to reach all error messages

Try to trigger potential exceptions

Feed invalid inputs



14
wrong formats, too long, too short, empty, …
Try combinations of all above
Repeat same and use different inputs for many
times if the input is a sequence
Black Box Testing Techniques

15
Boundary value testing

Boundary value analysis

Robustness testing

Worst case testing

Equivalence class testing

Decision table based testing
Boundary Value Analysis



16
Errors tend to occur near the extreme values of
an input variables
Boundary value analysis focuses on the boundary of
the input space to identity test cases
Boundary value analysis selects input variable
values at their

Minimum

Just above the minimum

A nominal value

Just below the maximum

Maximum
Example of Boundary Value Analysis

Assume a program accepting two inputs y1 and y2,
such that a < y1< b and c < y2 < d
y2
d
c
..
.. ..
.
a
17
..
b
y1
Single Fault Assumption for
Boundary Value Analysis

Boundary value analysis is also augmented by the
single fault assumption principle
“Failures occur rarely as the result of the
simultaneous occurrence of two (or more) faults”

In this respect, boundary value analysis test cases
can be obtained by holding the values of all but one
variable at their nominal values, and letting that
variable assume its extreme values
18
Generalization of Boundary
Value Analysis

The basic boundary value analysis can be
generalized in two ways:

By the number of variables - (4n +1) test cases for
n variables

19
By the kinds of ranges of variables : map to order

Strings

Sequences

Complex Data Structures, e. g., trees
Application Scenario of
Boundary Value Analysis

Several independent variables that represent
bounded physical quantities

No consideration of the function of the program,
nor of the semantic meaning of the variables

Good news: We can distinguish between physical
and other types of variables
20
Robustness Testing



21
A simple extension of boundary value analysis
In addition to the five boundary value analysis
values of variables, we add values slightly greater
that the maximum (max+) and a value slightly less
than the minimum (min-)
The main value of robustness testing is to force
attention on exception handling
Example of Robustness Testing
y2
d
c
..
... ..
..
a
22
…
b
y1
UTSA CS3773
Worst Case Testing




No single fault assumption: error happens when
more than one variable has an extreme value
Considering n inputs in boundary analysis, we take
the Cartesian product of the five values for 1, 2, 3,
… n variables
We can have 5n test cases for n input variables
The more interactions on inputs -> more on worse
case testing

23
Input partitions: Length & Width vs. Length & price
Example of Worst Case Testing
y2
d
c
....
....
..
a
24
..
..
.
....
....
..
b
y1
Equivalence Class Testing


25
Divide the value range of an input to a number of
subsets

Subsets are disjoint

The union of the subset if the value range
Values in one subset does not make difference for
the software concerned

Water temp in a car: <32, 32 – 212, >=212

Normal colors vs. Metallic colors
Example of Equivalence Class Testing
26
Equivalence Class Testing

The use of equivalence class testing has two
motivations:

Sense of complete testing


Avoid redundancy


The disjointness assures a form of non-redundancy
Note


27
The entire set is represented provides a form of
completeness
Also check boundaries
Combinations of inputs also follows the rule: more
interaction -> more combinations
Equivalent Class for non-numeric inputs

Feature extraction

For string and structure inputs

Split the possible value set with a certain feature

Example:
String passwd => {contains space}, {no space}

It is possible to extract multiple features from one input

Example:
String name => {capitalized first letter}, {not}
=> {contains space}, {not}
=> {length >10}, {2-10}, {1}, {0}
One test case may cover multiple features
28
Decision Table

Make it easy to observe that all possible
conditions are accounted for

29
Decision tables can be used for:

Specifying complex program logic

Generating test cases with oracles
Example of Decision Table
Conditions
Printer does not print
Y
Y
Y
Y
N
N N
N
A red light is flashing
Y
Y
N
N
Y
Y N
N
Printer is unrecognized
Y
N
Y
N
Y
N Y
N
X
X
Check the power cable
Actions
Check the printer-computer cable
X
X
Ensure printer software is installed
X
X
Check/replace ink
X
Check for paper jam
30
X
Printer Troubleshooting
X
X
X
X
X
Decision Table Usage

The use of the decision-table model is applicable
when:

Specification is given or can be converted to a
decision table

The order in which the predicates are evaluated
does not affect the interpretation of resulting
action

Note:

31
Decision table needs not cover all combinations
White Box Testing: General Guidelines

Try to cover all branches


Test more on complex modules

32
Study the relationship between input value and
branch logic
Measure complexities of modules by code size,
number of branches and loops, number of calls
and recursions
White Box Testing: Techniques
33

More difficult than black box testing

Seldom done manually

Automatic support

Symbolic execution

Complexity measurement and Defect prediction
Review: Test overview




Test is the practical choice: the best
affordable approach
Concepts: test case, test oracle, test suite,
test driver, test script, test coverage
Granularity: unit, integration, system,
acceptance
Type by design principle: black-box, white-box


34
Black-box-testing: boundary, equivalence, decision
table
White-box-testing: branch coverage, complexity