Introduction to Software
Testing
Why Do We Test Software?
Dr. Pedro Mejia Alvarez
Testing in the 21st Century

Software defines behavior
– network routers, finance, switching networks, other infrastructure

Today’s software market :
Industry is going
– is much bigger
through a revolution in
– is more competitive
what testing means to
– has more users
the success of software
products
 Embedded Control Applications
–
–
–
–
–

airplanes, air traffic control
spaceships
watches
ovens
remote controllers
– PDAs
– memory seats
– DVD players
– garage door openers
– cell phones
Agile processes put increased pressure on testers
– Programmers must unit test – with no training or education!
– Tests are key to functional requirements – but who builds those tests ?
Introduction to Software Testing,
2
Software is a Skin that
Surrounds Our Civilization
Quote due to Dr. Mark Harman
Introduction to Software Testing,
3
Software Faults, Errors & Failures

Software Fault : A static defect in the software

Software Error : An incorrect internal state that is the
manifestation of some fault

Software Failure : External, incorrect behavior with
respect to the requirements or other description of the
expected behavior
Faults in software are equivalent to design mistakes in
hardware.
Software does not degrade.
Introduction to Software Testing,
4
Fault and Failure Example

A patient gives a doctor a list of symptoms
– Failures

The doctor tries to diagnose the root cause, the ailment
– Fault

The doctor may look for anomalous internal conditions
(high blood pressure, irregular heartbeat, bacteria in the
blood stream)
– Errors
Most medical problems result from external attacks
(bacteria, viruses) or physical degradation as we age.
They were there at the beginning and do not “appear”
when a part wears out.
Introduction to Software Testing,
5
A Concrete Example
Fault: Should start
searching at 0, not 1
public static int numZero (int [ ] arr)
Test 1
{ // Effects: If arr is null throw NullPointerException
[ 2, 7, 0 ]
// else return the number of occurrences of 0 in arr
Expected: 1
int count = 0;
Actual: 1
for (int i = 1; i < arr.length; i++)
{
Error: i is 1, not 0, on
Test 2
if (arr [ i ] == 0)
the first iteration
[ 0, 2, 7 ]
{
Failure: none
Expected: 1
count++;
Actual: 0
}
}
Error: i is 1, not 0
return count;
Error propagates to the variable count
}
Failure: count is 0 at the return statement
Introduction to Software Testing,
6
The Term Bug



Bug is used informally
Sometimes speakers mean fault, sometimes error, sometimes failure
… often the speaker doesn’t know what it means !
This class will try to use words that have precise, defined, and
unambiguous meanings
“an analyzing process
BUG
“It has been just so in all of my inventions. The
first step is an intuition, and comes with a burst,
then difficulties arise—this thing gives out and
[it is] then that 'Bugs'—as such little faults and
difficulties are called—show themselves and
months of intense watching, study and labor are
requisite. . .” – Thomas Edison
Introduction to Software Testing,
must equally have been
performed in order to
furnish the Analytical
Engine with the necessary
operative data; and that
herein may also lie a
possible source of error.
Granted that the actual
mechanism is unerring in
its processes, the cards
may give it wrong orders.
” – Ada, Countess Lovelace
(notes on Babbage’s
Analytical Engine)
7
Spectacular Software Failures

NASA’s Mars lander: September 1999, crashed
due to a units integration fault
Mars Polar
THERAC-25 design
Lander crash
site?



THERAC-25 radiation machine : Poor testing of
safety-critical software can cost lives : 3 patients Ariane 5:
were killed
exception-handling
Ariane 5 explosion : Very expensive
bug : forced self
Intel’s Pentium FDIV fault : Public relations
nightmare
destruct on maiden
flight (64-bit to 16-bit
conversion: about
370 million $ lost)
We need our software to be dependable
Testing is one way to assess dependability
Introduction to Software Testing,
8
Northeast Blackout of 2003
508 generating
units and 256
power plants shut
down
Affected 10 million
people in Ontario,
Canada
Affected 40 million
people in 8 US
states
Financial losses of
$6 Billion USD
The alarm system in the energy management system failed
due to a software error and operators were not informed of
the power overload in the system
Introduction to Software Testing,
9
Costly Software Failures

NIST report, “The Economic Impacts of Inadequate
Infrastructure for Software Testing” (2002)
– Inadequate software testing costs the US alone between $22 and
$59 billion annually
– Better approaches could cut this amount in half

Huge losses due to web application failures
– Financial services : $6.5 million per hour (just in USA!)
– Credit card sales applications : $2.4 million per hour (in USA)
In Dec 2006, amazon.com’s BOGO offer turned into a
double discount
 2007 : Symantec says that most security vulnerabilities are
due to faulty software
World-wide monetary loss due to poor software is staggering

Introduction to Software Testing,
10
Testing in the 21st Century
More safety critical, real-time software
 Embedded software is ubiquitous … check your pockets
 Enterprise applications means bigger programs, more
users
 Paradoxically, free software increases our expectations !
 Security is now all about software faults

– Secure software is reliable software

The web offers a new deployment platform
– Very competitive and very available to more users
– Web apps are distributed
– Web apps must be highly reliable
Industry desperately needs our inventions !
Introduction to Software Testing,
11
What Does This Mean?
Software testing is getting more
important
What are we trying to do when we test ?
What are our goals ?
Introduction to Software Testing,
12
Testing Goals Based on Test
Process Maturity
 Level 0 : There’s no difference between testing and
debugging
 Level 1 : The purpose of testing is to show correctness
 Level 2 : The purpose of testing is to show that the
software doesn’t work
 Level 3 : The purpose of testing is not to prove anything
specific, but to reduce the risk of using the software
 Level 4 : Testing is a mental discipline that helps all IT
professionals develop higher quality software
Introduction to Software Testing,
13
Level 0 Thinking
 Testing
is the same as debugging
 Does
not distinguish between incorrect behavior
and mistakes in the program
 Does
safe
not help develop software that is reliable or
This is what we teach undergraduate CS majors
Introduction to Software Testing,
14
Level 1 Thinking
 Purpose
is to show correctness
 Correctness is impossible to achieve
 What do we know if no failures?
– Good software or bad tests?
 Test
engineers have no:
– Strict goal
– Real stopping rule
– Formal test technique
– Test managers are powerless
This is what hardware engineers often expect
Introduction to Software Testing,
15
Level 2 Thinking
 Purpose
is to show failures
 Looking
for failures is a negative activity
 Puts
testers and developers into an adversarial
relationship
 What
if there are no failures?
This describes most software companies.
How can we move to a team approach ??
Introduction to Software Testing,
16
Level 3 Thinking
 Testing
can only show the presence of failures
 Whenever
we use software, we incur some risk
 Risk
may be small and consequences unimportant
 Risk
may be great and consequences catastrophic
 Testers
and developers cooperate to reduce risk
This describes a few “enlightened” software companies
Introduction to Software Testing,
17
Level 4 Thinking
A mental discipline that increases quality

Testing is only one way to increase quality

Test engineers can become technical leaders of the project

Primary responsibility to measure and improve software
quality

Their expertise should help the developers
This is the way “traditional” engineering works
Introduction to Software Testing,
18
Where Are You?
Are you at level 0, 1, or 2 ?
Is your organization at work at level
0, 1, or 2 ?
Or 3?
We hope to teach you to become
“change agents” in your workplace …
Advocates for level 4 thinking
Introduction to Software Testing,
19
Tactical Goals : Why Each Test ?
If you don’t know why you’re conducting
each test, it won’t be very helpful
 Written
test objectives and requirements must be
documented
 What are your planned coverage levels?
 How much testing is enough?
objective – spend the budget … test
until the ship-date …
 Common
– Sometimes called the “date criterion”
Introduction to Software Testing,
20
Here! Test This!
My first “professional” job
Big
software– big
program
MicroSteff
software system
for the mac
V.1.5.1
Jan/2007
Jan/2011
MF2-HD
1.44 MB
DataLife
Verdatim
A stack of computer printouts—and no documentation
Introduction to Software Testing,
21
Why Each Test ?
If you don’t start planning for each test when the
functional requirements are formed, you’ll never
know why you’re conducting the test
 1980:
“The software shall be easily maintainable”
 Threshold
 What
reliability requirements?
fact does each test try to verify?
 Requirements
Introduction to Software Testing,
definition teams need testers!
22
Cost of Not Testing
Poor Program Managers might say:
“Testing is too expensive.”
 Testing
is the most time consuming and
expensive part of software development
 Not testing is even more expensive
 If we have too little testing effort early, the cost
of testing increases
 Planning for testing after development is
prohibitively expensive
Introduction to Software Testing,
23
Cost of Late Testing
60
Assume $1000 unit cost, per fault, 100 faults
50
40
30
20
10
Fault origin (%)
Fault detection (%)
Unit cost (X)
0
Software Engineering Institute; Carnegie Mellon University; Handbook CMU/SEI-96-HB-002
Introduction to Software Testing,
24
Summary: Why Do We Test
Software ?
A tester’s goal is to eliminate faults
as early as possible
• Improve quality
• Reduce cost
• Preserve customer satisfaction
Introduction to Software Testing,
25
Software Testing
Testing is the process of exercising
a program with the specific intent of
finding errors prior to delivery to the
end user.
26
What Testing
Shows
errors
requirements conformance
performance
an indication
of quality
27
Strategic Approach





To perform effective testing, you should conduct effective
technical reviews. By doing this, many errors will be eliminated
before testing commences.
Testing begins at the component level and works "outward"
toward the integration of the entire computer-based system.
Different testing techniques are appropriate for different
software engineering approaches and at different points in
time.
Testing is conducted by the developer of the software and (for
large projects) an independent test group.
Testing and debugging are different activities, but debugging
must be accommodated in any testing strategy.
28
V&V
Verification refers to the set of tasks that ensure
that software correctly implements a specific
function.
 Validation refers to a different set of tasks that
ensure that the software that has been built is
traceable to customer requirements. Boehm
[Boe81] states this another way:

– Verification: "Are we building the product right?"
– Validation: "Are we building the right product?"
29
Who Tests the
Software?
developer
independent tester
Understands the system
but, will test "gently"
Must learn about the system,
but, will attempt to break it
and, is driven by "delivery"
and, is driven by quality
30
Testing Strategy
System engineering
Analysis modeling
Design modeling
Code generation Unit test
Integration test
Validation test
System test
31
Testing Strategy
We begin by ‘testing-in-the-small’ and move toward
‘testing-in-the-large’
 For conventional software

– The module (component) is our initial focus
– Integration of modules follows

For OO software
– our focus when “testing in the small” changes from an individual
module (the conventional view) to an OO class that
encompasses attributes and operations and implies
communication and collaboration
32
Strategic Issues








Specify product requirements in a quantifiable manner long before testing
commences.
State testing objectives explicitly.
Understand the users of the software and develop a profile for each user
category.
Develop a testing plan that emphasizes “rapid cycle testing.”
Build “robust” software that is designed to test itself
Use effective technical reviews as a filter prior to testing
Conduct technical reviews to assess the test strategy and test cases
themselves.
Develop a continuous improvement approach for the testing process.
33
Unit Testing
module
to be
tested
results
software
engineer
test cases
34
Unit Testing
module
to be
tested
interface
local data structures
boundary conditions
independent paths
error handling paths
test cases
35
Unit Test Environment
driver
interface
local data structures
boundary conditions
Module
independent paths
error handling paths
stub
stub
test cases
RESULTS
36
Integration Testing
Strategies
Options:
• the “big bang” approach
• an incremental construction strategy
37
Top Down
Integration
A
B
F
top module is tested with
stubs
G
stubs are replaced one at
a time, "depth first"
C
as new modules are integrated,
some subset of tests is re-run
D
E
38
Bottom-Up
Integration
A
B
F
G
drivers are replaced one at a
time, "depth first"
C
worker modules are grouped into
builds and integrated
D
E
cluster
39
Regression Testing




Regression testing is the re-execution of some subset of tests
that have already been conducted to ensure that changes
have not propagated unintended side effects
Whenever software is corrected, some aspect of the software
configuration (the program, its documentation, or the data that
support it) is changed.
Regression testing helps to ensure that changes (due to
testing or for other reasons) do not introduce unintended
behavior or additional errors.
Regression testing may be conducted manually, by reexecuting a subset of all test cases or using automated
capture/playback tools.
40
Object-Oriented Testing
begins by evaluating the correctness and consistency of
the analysis and design models
 testing strategy changes

– the concept of the ‘unit’ broadens due to encapsulation
– integration focuses on classes and their execution across a
‘thread’ or in the context of a usage scenario
– validation uses conventional black box methods

test case design draws on conventional methods, but also
encompasses special features
41
OO Testing Strategy


class testing is the equivalent of unit testing
– operations within the class are tested
– the state behavior of the class is examined
integration applied three different strategies
– thread-based testing—integrates the set of classes
required to respond to one input or event
– use-based testing—integrates the set of classes
required to respond to one use case
– cluster testing—integrates the set of classes required
to demonstrate one collaboration
42
WebApp Testing - I
The content model for the WebApp is reviewed to
uncover errors.
 The interface model is reviewed to ensure that all use
cases can be accommodated.
 The design model for the WebApp is reviewed to
uncover navigation errors.
 The user interface is tested to uncover errors in
presentation and/or navigation mechanics.
 Each functional component is unit tested.

43
WebApp Testing - II





Navigation throughout the architecture is tested.
The WebApp is implemented in a variety of different environmental
configurations and is tested for compatibility with each configuration.
Security tests are conducted in an attempt to exploit vulnerabilities in the
WebApp or within its environment.
Performance tests are conducted.
The WebApp is tested by a controlled and monitored population of endusers. The results of their interaction with the system are evaluated for
content and navigation errors, usability concerns, compatibility concerns,
and WebApp reliability and performance.
44
High Order Testing







Validation testing
– Focus is on software requirements
System testing
– Focus is on system integration
Alpha/Beta testing
– Focus is on customer usage
Recovery testing
– forces the software to fail in a variety of ways and verifies that recovery is properly
performed
Security testing
– verifies that protection mechanisms built into a system will, in fact, protect it from
improper penetration
Stress testing
– executes a system in a manner that demands resources in abnormal quantity,
frequency, or volume
Performance Testing
– test the run-time performance of software within the context of an integrated system
45
Debugging: A Diagnostic
Process
46
The Debugging Process
47
Debugging Effort
time required
to diagnose the
symptom and
determine the
cause
time required
to correct the error
and conduct
regression tests
48
Symptoms & Causes
symptom and cause may be
geographically separated
symptom may disappear when
another problem is fixed
cause may be due to a
combination of non-errors
cause may be due to a system
or compiler error
symptom
cause
cause may be due to
assumptions that everyone
believes
symptom may be intermittent
49
Consequences of Bugs
infectious
damage
catastrophic
extreme
serious
disturbing
mild
annoying
Bug Type
Bug Categories: function-related bugs,
system-related bugs, data bugs, coding bugs,
design bugs, documentation bugs, standards
violations, etc.
50
Debugging
Techniques
brute force / testing
backtracking
induction
deduction
51
Correcting the Error




Is the cause of the bug reproduced in another part of the program?
In many situations, a program defect is caused by an erroneous
pattern of logic that may be reproduced elsewhere.
What "next bug" might be introduced by the fix I'm about to make?
Before the correction is made, the source code (or, better, the
design) should be evaluated to assess coupling of logic and data
structures.
What could we have done to prevent this bug in the first place? This
question is the first step toward establishing a statistical software
quality assurance approach. If you correct the process as well as the
product, the bug will be removed from the current program and may
be eliminated from all future programs.
What if the bug persist and I cannot correct it ? Can I do Fault
Tolerance ?
52
Final Thoughts
Think -- before you act to correct
 Use tools to gain additional insight
 If you’re at an impasse, get help from someone else
 Once you correct the bug, use regression testing to
uncover any side effects

53