2014 REU Program at ECU
Software Testing - Foundations,
Tools, and Applications
Lecture 1
May 27, 2014
Introduction to Software Testing
Dr. Sergiy Vilkomir
1/42
Dr. Sergiy Vilkomir
Instructor’s office: S & T Building, C-111
Email: vilkomirs@ecu.edu
http://www.cs.ecu.edu/reu/index.html
http://core.ecu.edu/vilkomirs
http://core.ecu.edu/STRG/
2/42
Software life cycle
•
•
•
•
•
•
•
concept
requirements
design
implementation (coding)
testing
testing
installation
operation and maintenance
Is testing important ?
3/42
Software Errors
Software errors can be found and eliminated
during software testing
“Software bugs, or errors, are so prevalent and so
detrimental that they cost the U.S. economy an
estimated $59.5 billion annually, or about 0.6 percent of
the gross domestic product, according to a newly
released study commissioned by the Department of
Commerce's National Institute of Standards and
Technology (NIST)”.
(NIST News, June 42, 2002 )
4/42
Software Errors
Buggy Software May Have Crashed Mars
Polar Lander (March 2000)
• The software problem crashed the Mars Polar Lander into the Red
Planet’s frozen ground
• The most likely cause of the lander’s failure, investigators decided,
was that a spurious sensor signal associated with the craft’s legs
falsely indicated that the craft had touched down when in fact it was
some 130-feet (40 meters) above the surface.
• This caused the descent engines to shut down prematurely and the
lander to free fall out of the martian sky.
• The error was traced to a single bad line of software code.
5/42
Causes of software defects
Software Bugs with Fatal
Consequences
• 1985 - 1986 few patients were
killed in a hospital in USA
because the medicine dose was
calculated wrongly (software
bugs in Therac-25 treatment
machine lead to radiation
overdoses, approximately 100
times the intended dose).
6/42
7/42
Causes of software defects
• In1996 the Ariane 5
rocket of the European
space agency was
destroyed because
they used the software
from Ariane 4 rocket
(was self-destroyed 40
seconds after takeoff;
more than $370 million
was lost ).
8/42
Causes of software defects
Software Bugs with Fatal
Consequences
• In 1962 NASA lost their 80
Million Dollar "Mariner 1"
because of a missing
"hyphen" in the program
code.
9/42
Causes of software defects
Software Bugs with Fatal
Consequences
During the Gulf War in February,
1991, a Patriot missile system
operating in Dhahran, Saudi
Arabia, failed to track and
intercept an incoming Scud. The
Iraqi missile impacted into an
army barracks, killing 42 U.S.
soldiers and injuring another 98.
The cause of the missile system failing to defend against the incoming
Scud was traced back to a bug in Patriot’s radar and tracking software.
The bug occurs in the calculation of the next location of the incoming
target. That modified software reached Dhahran the day after the attack.
10/42
Computers and Software Errors
• On Tuesday, June 3, 1980, at 1:26 a.m., the display system at the
command post of the Strategic Air Command (SAC) near Omaha,
Nebraska, indicated that two submarine-launched ballistic missiles
(SLBMs) were headed toward the United States.
• The SAC duty controller directed all alert crews to move to their B52 bombers and to start their engines
• Land-based missile crews were put on a higher state of alert, and
battle-control aircraft prepared for flight
• Fortunately, there were a number of factors which made those
involved in the assessment doubt that an actual attack was
underway
• The cause of these incidents was eventually traced to the failure of a
single integrated circuit chip in a computer which was part of a
communication system
11/42
Causes of software defects
Software Bugs with Fatal Consequences
• Software bugs in a Soviet early-warning monitoring
system nearly brought on nuclear war in 1983.
• The software was supposed to filter out false missile
detections caused by Soviet satellites picking up sunlight
reflections off cloud-tops, but failed to do so.
• Disaster was averted when a Soviet commander, based
on what he said was a '...funny feeling in my gut',
decided the apparent missile attack was a false alarm.
The filtering software code was rewritten.
12/42
Software Errors
Investigation found that the satellite in question had picked up the sun's reflection
off the cloud tops and somehow interpreted that as a missile launch.
13/42
CNN Money - August 9, 2012
Computer glitch that set
fire to $440 million of
Knight Capital Group's funds
In less than an hour, Knight Capital's computers executed a
series of automatic orders that were supposed to be
spread out over a period of days. Millions of shares
changed hands. The resulting loss, which was nearly
four times the company's 2011 profit, crippled the firm
and brought it to the edge of bankruptcy.
14/42
Top Software Failures Of 2011
(from http://www.businesscomputingworld.co.uk)
1. Financial services giant fined $25 million for hiding
software glitch that cost investors $217 million
2. Computer system bugs cause Asian banking
facilities’ downtime
3. Cash machine bug benefits customers by giving
them extra money
4. 22 people wrongly arrested in Australia due to
failures in new NZ $54.5 million courts computer
system
5. 42,420 cars recalled after airbag-related software
glitch
15/42
Top Software Failures Of 2011
(from http://www.businesscomputingworld.co.uk)
Conclusions
• Software failures are costing companies and consumers
large amounts of money
• The worst software failures have damaged reputations,
impacted negatively on financials and caused stress to
users.
• Each of the top 2011 software failure examples could
easily have been avoided through an effective quality
management and proper software testing
16/42
• Systems where failure means: death, injury, destruction,
chaos, severe financial loss, etc.
• Such systems often are safety-critical systems or
mission-critical systems
Student project
17/54
18/54
•
•
•
•
•
•
•
Nuclear power plant systems
Military systems
Space systems
Avionic software
Medical software
Large telecommunication systems
...
19/54
Software testing
• Software errors can be found during
software testing
• Testing can help but it is not for free
• The cost of software testing could be
around 40% of the overall costs of software
development
20/42
Terminology
Validation
Verification
Functional testing
Regression testing
Unit testing
Failure
Fault
Error
Software testing
Oracle
Input domain
V&V
21/42
Terminology
• Unfortunately, SE terminology in textbooks
is often contradictory
• Two different textbooks could provide two
different definitions for the same term
• What to do in this situation?
Solution: use terminology from standards
22/42
IEEE (Institute of Electrical and
Electronics Engineers) standard
IEEE Std 610.12 1990
IEEE Standard Glossary
of Software Engineering
Terminology
23/42
New standard
• IEEE Std 610.12 1990 has not been updated for many years
• Some terms are obsolete
• Instead: ISO/IEC/IEEE 24765:2010 “Systems and
software engineering – Vocabulary”
24/42
New International Software Testing
Standard
“Software and systems
engineering — Software testing”
• ISO/IEC/IEEE 29119-1: Concepts &
Definitions (published September 2013)
• ISO/IEC/IEEE 29119-2: Test Processes
(published September 2013)
• ISO/IEC/IEEE 29119-3: Test
Documentation (published September
2013)
• ISO/IEC/IEEE 29119-4: Test Techniques
(draft)
• ISO/IEC/IEEE 29119-5: Keyword Driven
Testing (draft)
25/42
From IEEE Std 610.12 1990
testing - the process of operating a system
or component under specified conditions,
observing or recording the results, and
making an evaluation of some aspect of
the system or component
Contrast with: Static Analysis, Inspection
26/42
From IEEE Std 610.12 1990
• Static Analysis - the process of evaluating a system or
component based on its form, structure, content, or
documentation.
• Inspection - a static analysis technique that relies on
visual examination of development products to detect
errors, violations of development standards, and other
problems.
27/42
From IEEE Std 610.12 1990
• Fault - an incorrect step, process, or data
definition in a computer program. Note: In
common usage, the terms "error" and
"bug" are used to express this meaning.
• Failure - the inability of a system or
component to perform its required
functions within specified performance
requirements. The result of the fault.
28/42
From IEEE Std 610.12 1990
• test case - a set of test inputs, execution
conditions, and expected results
developed for a particular objective, such
as to exercise a particular program path or
to verify compliance with a specific
requirement.
29/42
• Input domain – the set of all possible
inputs to a program
• test oracle - a source to determine
expected results to compare with the
actual result of the software under test. An
oracle may be the existing system (for a
benchmark), a user manual, or an
individual’s specialized knowledge, but
should not be the code
30/42
Y= a X + b – it is very easy to determine expected results
expected results ?
31/42
Goal of software testing
Increasing our confidence on
the quality and reliability of the program
Test case that did not find an error
- successful
Test case that discovered a new error - successful
Increasing the quality and reliability
of the program after removing the error
32
Goal of software testing
Dual Goal
• To find undiscovered errors AND
• to get confidence on the quality and
reliability of the program
33
Lifecycle phase in which testing
takes place
34
© Aditya P. Mathur 2005
From IEEE Std 610.12 1990
•
black-box testing -Testing that ignores the
internal mechanism of a system or component
and focuses solely on the outputs generated in
response to selected inputs and execution
conditions.
Syn:. functional testing
Contrast with: structural testing.
• white-box testing -Testing that takes into
account the internal mechanism of a system or
component. Types include branch testing, path
testing, statement testing.
Syn: structural testing, glass-box testing
Contrast with: functional testing
35
Is testing always done in the same way?
Types of software
Program A
Program B
Program C
Program D
Personal software
Company’s internal software
Commercial software
Safety-critical software
Consequences of
defects
nothing serious
internal problems
financial losses,
damaged reputation
loss of human life
36/42
• Testing is done differently in different contexts.
– Type of software
– Recourses (constraints): human recourses,
financial recourses, time
– Regulatory requirements
• Conclusion: testing is context dependent
37/42
Causes of software defects
Human
mistake
(error)
Defect (bug, fault)
in software
Failure
during
software
execution
38/42
How much testing is enough?
Exhaustive testing is impossible
Testing everything (all combinations of inputs and
preconditions) is not feasible
Example:
• The exam has 30 questions and four
possible answers for each question.
• What is a number of all possible
combinations of answers?
• If we can test 1000 combinations per
second, how long will we test all
combinations? (An hour? A day? A month?)
Total number of combinations: 444 …. 4 = 430  1.000.000.000.000.000.000
30 times
1 year  42.000.000 sec. We need  42.000.000 years for testing
39/42
How much testing is enough?
Not possible to say in general
Depends on each specific situation
• Pressures on a project include time and budget
• Risk assessment to decide how much testing to do.
Important:
• prioritization (do the most important tests first)
• setting criteria when to stop testing
• Take into account technical and business risk
• Take into account project constraints.
40/42
Testing principles
Testing shows presence of defects
Testing can show that defects are present, but cannot
prove that there are no defects. Testing reduces the
probability of undiscovered defects remaining in the
software but, even if no defects are found, it is not a
proof of correctness
“Program testing can be used to show the presence of
bugs, but never to show their absence!”
1970, famous quote of ?
41/42
Testing principles
Testing shows presence of defects
“Program testing can be used to show the presence of
bugs, but never to show their absence!”
1970, famous quote of Dijkstra
Edsger Dijkstra (1930 – 2002)
was a Dutch computer scientist.
Winner of the 1972 Turing
Award; creator of "structured
programming."
42/42