Testing
• Testing is a critical process within any software
development and maintenance life cycle model.
• Testing is becoming increasingly important to the
software development industry because of its
implications on the quality of the produced software
and the reputation of the producing company.
• Testing activities include testing specifications,
designs and implementations.
• Testing implementations is often referred to as
software testing, or simply testing.
Test coverage
• Only exhaustive testing can guarantee full coverage.
• Exhaustive testing is impractical since it may take years to
complete even for testing small size programs.
• Consider testing the software when a first name text field of
20 characters maximum is filled by a user. Exhaustive testing
requires that we test the software for all possible correct and
incorrect field input values.
– Assume 80 possible correct and incorrect characters, the number of
possible combinations for a valid or invalid first name would exceed
8020. Using a computer that takes, say, 50 nanoseconds to process one
combination, we would need about 1011 years to complete the testing
for all possible combinations. It is indeed impractical!
The reality
• With the absence of exhaustive testing, there are no
formal guarantees that software is error-free.
• In fact, it may happen in practice that even when all
errors have been detected during testing, they may
not all be fixed due to the lack of time.
• Unfixed errors may be fixed in future releases of the
software.
• The difficulty in software testing is the inability to
quantify the errors existing in software, and hence
predict how many errors are left undiscovered.
Test coverage experiment
• Random testing experiment: errors of different levels of
complexity were inserted in the software under test
• All errors except for one were discovered progressively.
• No indication on how much more testing is needed
• Finding errors is an asymptotic nonlinear function of time.
Dynamic and static testing
• Dynamic testing
– Test the software by executing it and applying test
inputs
– Observe the output and make a judgment to take a
verdict
• Static testing
– Check code without executing it
– Automatic static testing: Code as an input to a static analysis
tool
– Non-automatic static testing: review, walkthrough, inspection
Test case
• Test plan includes a test suite
• A test suite is a set of organized test cases
– Organize around use cases for example
• Template to describe a test case
– Id, name, reference, priority, preconditions,
preamble, input, expected output, observed
output, verdict and remarks
• Test cases are obtained using known testing
techniques and strategies
Test case design vs execution
• Test case is developed first – verdict and remarks are
empty
• Test case is executed when the actual testing is
performed – the verdict is obtained
• Typically test cases are developed (designed) before
even writing any code
• Acceptance test cases in the acceptance test plan (ATP)
can be developed after requirements specification
phase
• Unit and integration test plans can be developed after
the design is completed and prior to coding
Test Case (TC) Example
TC#: S-221
TC Description: This test is to check the response
to an invalid input selection.
TC Precondition: Go to screen
Flight_Management
TC Input: Enter <F7>
TC Expected Output: Error message: “Invalid
input, Enter “1” to “4” or ESC”
OOP Development
15
White-box, black-box and grey-box
testing techniques
• White-box: module or unit testing
• Black-box: module testing and system testing
Black-box unit/module testing
techniques
•
•
•
•
•
Boundary value analysis
Equivalence class testing
Decision table / decision tree based testing
State based testing
Robustness testing
Boundary value analysis
• BVA: test inputs obtained from the boundaries
– Serious errors hide at the boundaries.
• For example, simple boundary values for an input of integer
data type would include -INT_MAX, -INT_MAX + 1, -1, 0, +1,
INT_MAX - 1 and INT_MAX.
• For complex and composite data types, boundary values
become more difficult to generate and may require more
ingenuity to obtain them.
• Moreover, if interface specification includes more than one
input, different test cases considering possible feasible
combinations of BVs for each input must be considered
BVA Example
• Suppose you are given an executable code including
a function called digitCount whose functional
interface specification states that digitCount takes an
integer n as an input parameter, and returns an
integer representing the number of digits in n.
• Test cases whose test inputs are selected based on
boundary value analysis are partially shown in Table
8.3.
• Assume that for a 32-bit integer, INT_MAX is
+2147483647.
Testing function digitCount based
on boundary value analysis
Example 8.2 – Testing search function
Equivalence class testing
• Partitioning input domain into disjoint subsets or
partitions
• Partition includes related values
• Consider one test input from each partition
• Equivalence relation among partition elements
–
–
–
–
–
–
‘having the same number of digits’
‘all positive integers’
‘all even numbers’
‘all prime numbers’
‘all powe or 2 numbers’
…..
Decision table based testing
• Decision table: each column denoting a rule becomes
the basis of a test case
– Conditions produce inputs
– Actions produce expected outputs
– In table 4.1: input variables are age, history and
speed, and output variables are penalty and
points
Decision tree-based testing
• Each distinct path from the root node to a leaf node
is the basis of a test case
• If along the path, a variable input value does not
exist, the test input for that variable can be
assigned any valid value.
• For example, in fig 4.23, for the test case
corresponding to the leftmost path, the speed value
can be any illegal speed
State-based testing
• An implemented finite state machine conforms to its
specification, if the observed output sequence
matches the expected output according to the
specification
• Test sequence must cover all states transitions
• Used to test do black box testing of systems
– GUI, protocols, web page connectivity, …
• Transition tour: a sequence covering all transitions
and returning back to the initial state
– Can only cover output errors
Black box-based robustness testing
• Test a system using one or preferably more black box
based testing method
• Test for incorrect and illegal inputs also
• In equivalence class testing, select one
representative bad input from every partition of the
bad input values
• For FSM based systems, apply unexpected inputs at
each state – the system should survive
• For boundary value testing, test for illegal values
beyond the boundaries – the system should survive
White-box testing techniques
•
•
•
•
Given the source code of the module to test
Control flow based testing techniques
Data flow based testing techniques
Path testing
Control flow based testing
• Given source code of a module, transform it to
a control flowgraph
• Nodes and edges – decision nodes, joining
nodes
Example: digitCount
1 int digitCount(int n)
2{
3 int count = 0;
4 while (n != 0)
5 { n = n / 10;
6
count++;
7 }
8 return count;
9}
White-box testing
• Statement coverage
• Graph based
– Branch coverage
– Condition coverage
– Path coverage
OOP Development
50
Example: Statement Coverage
1 int maxPositive(int n1, int n2) {
2 if (n1<=1 && n2<=1)
3
n1=1;
4 if (n1<=n2)
5
n1=n2;
6
return n1; }
TC Inputs: n1=n2=1
OOP Development
51
Example: Branch Coverage
1 int maxPositive(int n1, int n2) {
2 if (n1<=1 && n2<=1)
3
n1=1;
4 if (n1<=n2)
5
n1=n2;
3
6
5
return n1; }
1,2
4
6
TC#1 Inputs: n1=n2=1
TC#2 Inputs: n1=2, n2=0
OOP Development
52
Example: Condition Coverage
1 int maxPositive(int n1, int n2) {
2 if (n1<=1 && n2<=1)
3
n1=1;
4 if (n1<=n2)
5
n1=n2;
3
6
5
return n1; }
TC#1 Inputs: n1=n2=1
TC#2 Inputs: n1=2, n2=0
TC#3 Inputs: n1=0, n2=2
OOP Development
1,2
4
6
53
Example: Path Coverage
1 int maxPositive(int n1, int n2) {
2 if (n1<=1 && n2<=1)
3
n1=1;
4 if (n1<=n2)
5
n1=n2;
3
6
5
return n1; }
TC#1 Inputs: n1=n2=1
TC#2 Inputs: n1=2, n2=0
TC#3 Inputs: n1=0, n2=2
TC#4 Inputs: n1=n2=0
OOP Development
1,2
4
6
54
Path expression
• Representing in a generic way an execution
sequence
• Can represent the complete program
• Use operators:
+ for choice
. For a concatenation
{} * for zero or more repetition
{}+ for one or more repetition
() for grouping
Path expression
• Useful to study properties of test sequences
• Annotate the transitions with desirable
property then analyze the resulting expression
• For example: assign a probability to each
transition – allows you to evaluate the
probability of each path; helps you decide on
testing the most probable paths
Integration testing
• After testing all modules individually using
both white and black box testing techniques
• Top-down integration
• Bottom-up integration
• Sandwich / hybrid integration
Top down integration
• Start with top level modules and integrate left to
rigth
• Good for testing GUI heavy modules
• Increase of confidence to see GUI at the beginning
• Less time to test low level utility modules – high
reusability modules
• Variation: integrate i/o modules to get real input and
outputs as early as possible
• Need stubs – fake modules to represent called
modules
Bottom up integration
•
•
•
•
Start from low level modules
Good testing of low level utility modules
Can see the GUI late in the process
Require driver modules – to call the integrated
modules
Sandwich integration testing
• Two teams – one starting bottom up and the other
starting top down
• They meet in the middle
• Good testing of both high level GUI modules and low
level utility modules
• Requires coordination and good management to
ensure middle level modules also get a good share of
the testing time
System Testing
• Performed in terms of Inputs and outputs of the
system.
• Performed on the targeted platform.
• Goals:
– Reveal bugs that are represented only at system level.
– Demonstrate that the system implements all required
capabilities.
– Answer the question: Can we ship it yet?
OOP Development
64
Regression Testing
• Ensuring that modifications have not caused
unintended effects.
• Retesting is required
OOP Development
65