CS 217
Software Verification and
Validation
Summer 2014
Instructor: Dong Si
http://www.cs.odu.edu/~dsi
Self introduction

Dong SI


PhD candidate in Computer Science
MS in Computer Science & Information System Engineering
BS in Electronic Information Science

dsi@stratford.edu

Course Policy
Please read the syllabus carefully and sign it;
 You will keep a copy of the signed syllabus.

Missing 2 classes = Fail;
 One hour late = missing a class.

Please be respectful of your classmates and instructor by
minimizing distractions during class;
 Talk to me after the class.

3
Classroom Etiquette (Dr. Bicey)

“No Children are allowed in classes.”

“No food or drink is allowed in the computer labs.”
4
Grading

Quizzes 30%

Class attendance & discussion 20%

Midterm exam 20%

Final exam 30%
5
Expectation
6
Temporary Course Schedule
Week 1
Introduction to Software Testing
Week 2
Basic Logics & Code Coverage
Week 3
Logic Testing & Input Space Partitioning
Week 4
Black Box Testing
Week 5
Midterm Exam
Week 6
White Box Testing & Syntax Testing
Week 7
Testing Levels & Types & Process
Week 8
Automated testing Tools & Challenges in Testing
Week 9
Review Week
Week 10
Final Exam
Dates and topics are subject to change during the semester,
Please check the course website for frequent updates:
http://www.cs.odu.edu/~dsi/teaching_stuff/CIS217/
Meeting schedule
9:00 – 9:30
 9:30 – 9:40
 9:40 – 10:45
 10:45 – 11:00
 11:00 – 12:05
 12:05 – 12:30

Quiz on readings
Review of last class
Lectures
Break
Lectures
Reading assignments
We will learn…

What is (software) testing?

What are the testing main principles?

Why, what, how (can be tested)?

What is the current / future of testing?
Introduction to Software
Testing
(2nd edition)
Chapter 1
Why Do We Test Software?
Paul Ammann & Jeff Offutt
http://www.cs.gmu.edu/~offutt/softwaretest/
What is (software) testing?
Testing?

A myriad of definitions
– Checking whether or not “something” conforms to
requirements
– Executing a program with the aim to find bugs (defects in the
program)
– Checking whether or not the program behaves expectedly
12
Softeware Testing – definition

The process consisting of all life cycle activities,
concerned with planning, preparation and evaluation
of software products and related work products to
determine:
– that they satisfy specified requirements,
– to demonstrate that they are fit for purpose and
– to detect defects
13
Validation & Verification
Validation : Have we built the right software?
 i.e., do the requirements satisfy the customer?
 (This is dynamic process for checking and testing the real
product. Software validation always involves with executing the
code)

Verification : Have we built the software right?
 i.e., does it implement the requirements?
 This is static method for verifying design, code. Software
verification is human based checking of documents and files

Introduction to Software Testing, Edition 2 (Ch 1)
© Ammann & Offutt
14
It is possible to test… everything

Requirements can be defined for every single outcome (artifact) in
software development

Everything can be tested:
– Analysis documentation (must describe customer’s needs, must be
sufficient for further development)
– Design documentation (must conform to analysis, must be efficient and
effective)
– Installation, maintenance guides (must be sufficient)
– The software (code) itself (stable, functional)
– The same software running on different configurations, different
platforms (stable, functional)
Introduction to Software Testing, Edition 2 (Ch 1)
© Ammann & Offutt
15
Testing in the 21st Century

Software defines behavior
– network routers, finance, switching networks, other infrastructure

Today’s software market :
Industry is going
– is much bigger
through a revolution in
– is more competitive
what testing means to
– has more users
the success of software
products
 Embedded Control Applications
–
–
–
–
–

airplanes, air traffic control
spaceships
watches
ovens
remote controllers
– PDAs
– memory seats
– DVD players
– garage door openers
– cell phones
Agile processes put increased pressure on testers
– Programmers must unit test – with no training or education!
– Tests are key to functional requirements – but who builds those tests ?
Introduction to Software Testing, Edition 2 (Ch 1)
© Ammann & Offutt
16
Software is a Skin that
Surrounds Our Civilization
Quote due to Dr. Mark Harman
Introduction to Software Testing, Edition 2 (Ch 1)
© Ammann & Offutt
17
COMPUTER BUG?
What is a computer bug?

In 1947 Harvard University was operating a room-sized computer
called the Mark II. – made of vacuum tubes

A moth flew into the computer and was killed by the high
voltage. Operators traced an error in the Mark II and
taped the bug to log book.
Hence, the first
computer bug!
I am not making this up
:-)
19
The Term Bug

Bug is used informally

Sometimes speakers mean fault, sometimes error, sometimes
failure, Incident, problem, Inconsistency … often the speaker
doesn’t know what it means !

This class will try to use words that have precise, defined, and
unambiguous meanings
BUG
Introduction to Software Testing, Edition 2 (Ch 1)
© Ammann & Offutt
20
Software Faults, Errors & Failures

Software Fault : A static defect in the software

Software Failure : External, incorrect behavior with
respect to the requirements or other description of the
expected behavior

Software Error : An incorrect internal state that is the
manifestation/expression of some fault
Faults in software are equivalent to design mistakes in
hardware.
Software does not degrade.
Introduction to Software Testing, Edition 2 (Ch 1)
© Ammann & Offutt
21
Fault and Failure Example

The doctor tries to diagnose the root cause, the disease
– Fault

A patient gives a doctor a list of symptoms
– Failures

The doctor may look for anomalous internal conditions
(high blood pressure, irregular heartbeat, bacteria in the
blood stream)
– Errors
Most medical problems result from external attacks
(bacteria, viruses) or physical degradation as we age.
They were there at the beginning and do not “appear”
when a part wears out.
Introduction to Software Testing, Edition 2 (Ch 1)
© Ammann & Offutt
22
A Concrete Example
Fault: Should start
searching at 0, not 1
public static int numZero (int [ ] arr)
Test 1
{ // Effects: If arr is null throw NullPointerException
[ 2, 7, 0 ]
// else return the number of occurrences of 0 in arr
Expected: 1
int count = 0;
Actual: 1
for (int i = 1; i < arr.length; i++)
{
Error: i is 1, not 0, on
Test 2
if (arr [ i ] == 0)
the first iteration
[ 0, 2, 7 ]
{
Failure: none
Expected: 1
count++;
Actual: 0
}
}
Error: i is 1, not 0
return count;
Error propagates to the variable count
}
Failure: count is 0 at the return statement
Introduction to Software Testing, Edition 2 (Ch 1)
© Ammann & Offutt
23
Defective Software

We develop programs that contain defects
– How many? What kind?

Hard to predict the future, however…
it is highly likely, that the software we (including you!) will
develop in the future will not be significantly better.
Sources of Problems

Requirements Definition: Erroneous, incomplete, inconsistent
requirements.

Design: Fundamental design flaws in the software.

Implementation: Mistakes in chip fabrication, wiring,
programming faults, malicious code.

Support Systems: Poor programming languages, faulty compilers
and debuggers, misleading development tools.
Sources of Problems (Cont’d)

Inadequate Testing of Software: Incomplete testing,
poor verification, mistakes in debugging.

Evolution: Sloppy redevelopment or maintenance,
introduction of new flaws in attempts to fix old flaws,
incremental escalation to inordinate complexity.
Fault in Real Code

Project Mercury’s FORTRAN code had the following fault:
“DO I=1.10” instead of ... “DO I=1,10”

The fault was discovered in an analysis of why the
software did not seem to generate results that were
sufficiently accurate.

The erroneous 1.10 would cause the loop to be executed
exactly ONLY once!
Military Aviation Simulations

In simulation, an F-16 program fault caused the virtual
plane to flip over whenever it crossed the equator, as a
result of a missing minus sign to indicate south latitude.
+
-
Real World Lessons learned
from poor testing
Spectacular Software Failures

NASA’s Mars lander: September 1999, crashed
due to a units integration fault
Mars Polar
THERAC-25 design
Lander crash
site?



THERAC-25 radiation machine : Poor testing of
safety-critical software can cost lives : 3 patients Ariane 5:
were killed
exception-handling
Ariane 5 explosion : Very expensive
bug : forced self
Intel’s Pentium FDIV fault : Public relations
nightmare
destruct on maiden
flight (64-bit to 16-bit
conversion: about
370 million $ lost)
We need our software to be dependable
Testing is one way to assess dependability
Introduction to Software Testing, Edition 2 (Ch 1)
© Ammann & Offutt
30
Northeast Blackout of 2003
508 generating
units and 256
power plants shut
down
Affected 10 million
people in Ontario,
Canada
Affected 40 million
people in 8 US
states
Financial losses of
$6 Billion USD
The alarm system in the energy management system failed
due to a software error and operators were not informed of
the power overload in the system
Introduction to Software Testing, Edition 2 (Ch 1)
© Ammann & Offutt
31
Airbus 319 Safety Critical
Software Control
Loss of autopilot
Loss of most flight deck lighting and intercom
Loss of both the commander’s and the co-pilot’s
primary flight and navigation displays !
Introduction to Software Testing (Ch 1)
© Ammann & Offutt
32
Costly Software Failures

NIST report, “The Economic Impacts of Inadequate
Infrastructure for Software Testing” (2002)
– Inadequate software testing costs the US alone between $22 and
$59 billion annually
– Better approaches could cut this amount in half

Huge losses due to web application failures
– Financial services : $6.5 million per hour (just in USA!)
– Credit card sales applications : $2.4 million per hour (in USA)
In Dec 2006, amazon.com’s BOGO offer turned into a
double discount
 2007 : Symantec says that most security vulnerabilities are
due to faulty software
World-wide monetary loss due to poor software is staggering

Introduction to Software Testing, Edition 2 (Ch 1)
© Ammann & Offutt
33
Testing in the 21st Century
More safety critical, real-time software
 Embedded software is common … check your pockets
 Enterprise applications means bigger programs, more
users
 Security is now all about software faults

– Secure software is reliable software

The web offers a new deployment platform
– Very competitive and very available to more users
– Web apps are distributed
– Web apps must be highly reliable
Industry desperately needs our inventions !
Introduction to Software Testing, Edition 2 (Ch 1)
© Ammann & Offutt
34
Discussion …

Have you heard of other software defects?
– In the media?
– From personal experience?

Does this embarrass you as a future software engineer?
What Does This Mean?
Software testing is getting more
important
What are we trying to do when we test ?
What are our goals ?
Introduction to Software Testing, Edition 2 (Ch 1)
© Ammann & Offutt
36
Testing Goals
Testing Goals Based on Test
Process Maturity
 Level 0 : There’s no difference between testing and
debugging
 Level 1 : The purpose of testing is to show correctness
 Level 2 : The purpose of testing is to show that the
software doesn’t work
 Level 3 : The purpose of testing is not to prove anything
specific, but to reduce the risk of using the software
 Level 4 : Testing is a mental discipline that helps all IT
professionals develop higher quality software
Introduction to Software Testing, Edition 2 (Ch 1)
© Ammann & Offutt
38
Level 0 Thinking
 Testing
is the same as debugging
 Does
not distinguish between incorrect behavior
and mistakes in the program
 Does
safe
not help develop software that is reliable or
Introduction to Software Testing, Edition 2 (Ch 1)
© Ammann & Offutt
39
Level 1 Thinking
 Purpose
is to show correctness
 Correctness is impossible to achieve
 What do we know if no failures?
– Good software or bad tests?
 Test
engineers have no:
– Strict goal
– Real stopping rule
– Formal test technique
– Test managers are powerless
This is what hardware engineers often expect
Introduction to Software Testing, Edition 2 (Ch 1)
© Ammann & Offutt
40
Level 2 Thinking
 Purpose
is to show failures
 Looking
for failures is a negative activity
 Puts
testers and developers into an adversarial
relationship
 What
if there are no failures?
This describes most software companies.
How can we move to a team approach ??
Introduction to Software Testing, Edition 2 (Ch 1)
© Ammann & Offutt
41
Level 3 Thinking
 Testing
can only show the presence of faults
 Whenever
we use software, we incur some risk
 Risk
may be small and consequences unimportant
 Risk
may be great and consequences catastrophic
 Testers
and developers cooperate to reduce risk
This describes a few “enlightened” software companies
Introduction to Software Testing, Edition 2 (Ch 1)
© Ammann & Offutt
42
Level 4 Thinking
A mental discipline that increases quality

Testing is only one way to increase quality

Test engineers can become technical leaders of the project

Primary responsibility to measure and improve software
quality

Their expertise should help the developers
This is the way “traditional” engineering works
Introduction to Software Testing, Edition 2 (Ch 1)
© Ammann & Offutt
43
Where Are You?
Are you at level 0, 1, or 2 ?
Is your organization at work at level
0, 1, or 2 ?
Or 3?
We hope to teach you to become
“change agents” in your workplace …
Advocates for level 4 thinking
Introduction to Software Testing, Edition 2 (Ch 1)
© Ammann & Offutt
44
A tester “serves” many clients
Project manager
 Software developer
 Technical writer
 Techincal support
 Marketing
 Management
 End users

45
More about tester’s role

Since a tester focuses to defects, his clients can focus to success
–
–
–
–
–
–
Find important defects fast
Evaluate the product tested
Use certain methods and tools
Educate your clients about testing
Educate yourself
Do your best to make your clients happy, but face the fact and let
others know – all bugs can not be found!
46
Discussion…

Your experiences on interacting with testers/clients/users

Based your experience, describe
– Information You -> Tester
– Information Tester -> You
– Any problems?
47
Tactical Goals : Why Each Test ?
If you don’t know why you’re conducting
each test, it won’t be very helpful
 Written
test objectives and requirements must be
documented
 What are your planned coverage levels?
 How much testing is enough?
objective – spend the budget … test
until the ship-date …
 Common
– Sometimes called the “date criterion”
Introduction to Software Testing, Edition 2 (Ch 1)
© Ammann & Offutt
48
Cost of Not Testing
Poor Program Managers might say:
“Testing is too expensive.”
 Testing
is the most time consuming and
expensive part of software development
 Not testing is even more expensive
 If we have too little testing effort early, the cost
of testing increases
 Planning for testing after development is
prohibitively expensive
Introduction to Software Testing, Edition 2 (Ch 1)
© Ammann & Offutt
49
Cost of Late Testing
60
Assume $1000 unit cost, per fault, 100 faults
50
40
Fault origin (%)
30
Fault detection (%)
20
Unit cost (X)
10
0
Software Engineering Institute; Carnegie Mellon University; Handbook CMU/SEI-96-HB-002
Introduction to Software Testing, Edition 2 (Ch 1)
© Ammann & Offutt
50
Summary: Why Do We Test
Software ?
A tester’s goal is to eliminate faults
as early as possible
• Improve quality
• Reduce cost
• Preserve customer satisfaction
Introduction to Software Testing, Edition 2 (Ch 1)
© Ammann & Offutt
51
Software Quality Factors
Functionality (exterior quality, user point of view)
- Correctness, reliability, usability, and integrity

Engineering (interior quality, designer point of view)
- Efficiency, testability, documentation, structure

Adaptability (future qualities, collaborator point of view)
- Flexibility, reusability, maintainability

52
Testing main principles
Testing Principles (1)

Testing can demonstrate only the presence of defects and
not their absence
– Testing can show that defects are present, but cannot prove that there
are no defects. Testing reduces the probability of undiscovered defects
remaining in the software but, even if no defects are found, it is not a
proof of correctness.

Exhaustive testing is impossible
– Exhaustive testing (all combinations of inputs and preconditions) is not
feasible except for trivial cases. Instead of exhaustive testing, risk
analysis and priorities should be used to focus testing efforts.
Testing Principles (2)

Early testing is important
– Testing activities should start as early as possible in the software
or system development life cycle and should be focused on
defined objectives.

Defects are clustering
– A small number of modules contain most of the defects
discovered during pre-release testing, or are responsible for the
most operational failures.
Testing Principles (3)

Testing is context dependent
– Testing is done differently in different contexts. For example, military
software is tested differently from an business site.
Types of testing
Types of Testing
Several (overlapping) classifications of testing
 Who is testing?
 What is tested?
 How it is tested?

Quality Attributes of ISO

Functional testing
– Everything concerning activities performed with the software

Non-functional testing
–
–
–
–
–
–
Reliability
Recoverability
Usability
Efficiency -> Performance, Load, Stress testing
Portability, Installability,
Scalability
Who is testing?
Alpha and Beta testing (developer)
 Acceptance testing (user)
 Subject-matter expert testing (expert)
…

How it is tested?










Static vs dynamic testing
Manual vs automated testing
Black box vs white box testing
Regression testing
Smoke testing
Exploratory testing
Guerilla testing
Paired testing
Ad-hoc testing
…
What data is used?
Equivalence partitions (input and output)
 Coverage criteria
 Random testing
 Statistical testing
 Function testing
 Menu tour

Granularity of the system
Module/unit testing
 Integration testing
 System testing

Software Testing Process
V&V Targets
Code & Implementation
Unit test
Software Design
Integration
test
System
test
System engineering
64
Unit Test (Component Level Test)

Individual components are tested independently to ensure
their quality. The focus is to uncover errors in design and
implementation, including
The focus is to uncover errors in:
- data structure in a component
- program logic and program structure in a component
- component interface
- functions and operations of a component
White-box
input
Black-box
interface
interface
input
Internal logic, data, structure
Operations and
Functions with I/O
output
output
operation
65
Integration Testing

A group of dependent components are tested together to
ensure their the quality of their integration unit.
- Design and construction of software architecture
- Integrated functions or operations at sub-system level
- Interfaces and interactions between them
- Resource integration and/or environment integration
interface
interface
input
Component #1
Component #2
Operations and
Functions with I/O
Operations and
Functions with I/O
output
operation
operation
66
System Testing

The system software is tested as a whole. It verifies all
components mesh properly to make sure that all system
functions and performance are achieved in the target
environment.
The focus areas are:
- System functions and performance
- System reliability and recoverability (recovery test)
- System installation (installation test)
- System behavior in the special conditions (stress and load test)
- System user operations (acceptance test/alpha test)
- Hardware and software integration and collaboration
- Integration of external software and the system
When a system is to be marketed as a software product, a testing process called beta testing is often used.
User
interface
System
User
External interfaces
(Operations &
Functions
& Behavior)
67
Software Development Lifecycles
Code and Fix
 Waterfall
 Spiral

Useful websites

Wiki - https://en.wikipedia.org/wiki/Software_testing

Software Testing Fundamentals http://softwaretestingfundamentals.com/

Software Testing Mentor http://www.softwaretestingmentor.com/

Computer Fundamentals Tutorial http://www.tutorialspoint.com/computer_fundamentals/
69
Next class we will discuss
Basic computational logics
- Logic in computer science
- Basic Logical Operators
- Beginning Mathematics/Basic Logic

Testing coverage
- Logic coverage: Chapter 3 of text book
- Code coverage

70