Introduction to Software Testing
Chapter 9.3
Challenges in Testing Software
Test Criteria and the Future of Testing
Paul Ammann & Jeff Offutt
http://www.cs.gmu.edu/~offutt/softwaretest/
Chapter 9 Outline
1. Testing for Emergent Properties:
Safety and Security
2. Software Testability
3. Test Criteria and the Future of
Software Testing
Testing
Introduction to Software Testing (Ch 9.2)
© Ammann & Offutt
2
Testing in the 1980s and 1990s
• Software engineering was very different
– Low quality software was normal and expected
– The cost of building better software outweighed the benefits
• Software was smaller
– Most software was bundled, shrink-wrapped, contracted
– Industry became dominated by one monopolistic vendor
• About a dozen books
– A few broad survey books, one or two on testing, design, requirements, etc
• In the early 1980s, one semester course was enough to learn most
of the knowledge in software engineering
• Safety critical software was a tiny part of the industry
Testing, especially high-end testing,
simply was not very important
Introduction to Software Testing (Ch 9.2)
© Ammann & Offutt
3
Cost of Late Testing
Planning &
Deploy to
Design Development Testing UAT
Requirements
Production
Fault
Origination 10%
Fault
Discovery
40%
6%
50%
13%
20%
20%
36%
5%
Requirements Design Development Functional System
Production
Test Planning Review Unit Testing Testing
Testing
Cost per Fault
1X
1X
1X
5X
10X
50X
Current State
$6K
$13K
$20K
$101K
$363K
$252K
Source – Software Engineering Institute; Carnegie Mellon University; Handbook CMU/SEI-96-HB-002; page 56-58
Introduction to Software Testing (Ch 9.2)
© Ammann & Offutt
4
Testing in the 21st Century
• The field has dramatically changed
• Today’s software market :
–
–
–
–
is much bigger
is more competitive
has more users
used in more places
• The web offers a new deployment platform
– Very competitive and very available to more users
• Enterprise applications means bigger programs and more users
• Embedded software is now ubiquitous … check your pockets !
• Paradoxically, free software increases our expectations !
• Security has now become essential …
Introduction to Software Testing (Ch 9.2)
© Ammann & Offutt
5
Problems Leading to Security Vulnerabilities
• 1980 : Security was mostly about math
– Cryptography
• 1990 : Security was mostly about protecting the database
• 2000 : Security was mostly about bullet-proofing the network
• Today : Most security vulnerabilities are a result of faults in the
software
Software testing is fast becoming essential to an
essential problem in the software industry
Software Security
Introduction to Software Testing (Ch 9.2)
© Ammann & Offutt
6
Testing Research
• 1980s : Criteria and algorithms for unit testing
• 1990s : Test criteria for other technologies and “levels”
• 2000s : More automation for test criteria and more technologies
• 2010s : Practical solutions to industry problems
A major point of this book is that we have enough criteria
• Test criteria are uniform across different software artifacts
– A graph is a graph is a graph …
– 36 criteria on 4 structures
• So are we done with testing research ?
– Can I retire now …
Introduction to Software Testing (Ch 9.2)
© Ammann & Offutt
7
Testing Research – Are We Done ?
No !
• We need more work on how to test modern technologies
– The ideas in chapter 7 are still developing
– OO, web, GUIs, real-time, embedded, …
– How to create the structures and how to adapt the criteria ?
• Which criteria to use when ?
– Cost / benefit tradeoffs among criteria are not known
• Which structure should be used when ?
• Technology transition to industry
– How best to automate the testing research ideas ?
– How to insert new testing techniques into the development process ?
– How to effectively and efficiently balance research theory with practical
needs ?
Introduction to Software Testing (Ch 9.2)
© Ammann & Offutt
8
Testing Research – Going Forward
• Automatic test data generation is one of the hardest problems
– But also the core of testing – getting values to satisfy criteria
– Research started in the 1970s – major advances in the 1980s, 1990s and
2000s
– Still very little help from industry-quality tools
• Testability, reliability, and other related areas are wide open
• We know a lot about testing new software – much less about
regression testing or testing evolving software
– Which happens to account for most of the effort in industry …
• Research in software testing is increasing
– Currently lots of funding in Europe and Asia – from both government
and industry sources
– New, well attended venues (eg ICST)
Introduction to Software Testing (Ch 9.2)
© Ammann & Offutt
9
Future of Software Testing
1. Increased specialization in testing teams will lead to
more efficient and effective testing
2. Testing and QA teams will have more technical
expertise
3. Developers will have more knowledge about testing
and motivation to test better
Agile processes puts testing first—putting pressure
on both testers and developers to test better
5. Testing and security are starting to merge
4.
Introduction to Software Testing (Ch 9.2)
© Ammann & Offutt
10
Summary
• With more attention from
– Industry
– Research
– Education …
We have entered a golden age for software
testing
Finally we have the knowledge, resources,
and motivation to make testing a
technical discipline
Introduction to Software Testing (Ch 9.2)
© Ammann & Offutt
11