1/12/09
Class 01 - 2009-01-12
Prof. Matt Rutherford
(c) 2009 Matthew J. Rutherford
New professor in the CS department
Study topics at the intersection of software engineering and distributed systems
Have 12+ years experience as a software engineer, technical lead, etc… mostly at small companies and startups
I’m an engineer not a mathematician
Class 01 - 2009-01-12 (c) 2009 Matthew J. Rutherford
1
Brief intro:
Name and Job description
Testing experience?
Goals for this course?
Favorite book?
(Mine is anything by Neal Stephenson: Cryptonomicon, The Big
U, …)
Class 01 - 2009-01-12 (c) 2009 Matthew J. Rutherford
Brief Introduction
Administrative / Logistics
Chapter 1: Introduction to Software Testing
Section 2.1: Graph Coverage Criteria
Class 01 - 2009-01-12 (c) 2009 Matthew J. Rutherford
1/12/09
2
“Software testing consists of the dynamic verification of the behavior of a program on a finite set of test cases, suitably selected from the usually infinite executions domain, against the expected behavior.” - SWEBOK
Start with the Software Under Test (SUT)
Execute the software using specific “inputs”
Look at the “outputs” to determine if the software did the “right thing”
Class 01 - 2009-01-12 (c) 2009 Matthew J. Rutherford
1/12/09
Class 01 - 2009-01-12 (c) 2009 Matthew J. Rutherford
3
Class 01 - 2009-01-12 x
3
6
5
3
3 y
6
Result
3
3
3
Precise Imprecise
==3 ✔ >0 ✔
==3 ✔
==1 ✗
>0
>0
✔
✔
(c) 2009 Matthew J. Rutherford
Input spaces are vast, exhaustive testing is impossible
Testing proves nothing!
Move from math to engineering:
How to reduce the probability of releasing faulty software within constraints imposed by budget?
Class 01 - 2009-01-12 (c) 2009 Matthew J. Rutherford
1/12/09
4
Which input values should be used?
How many?
When is a set of test cases “adequate”
Class 01 - 2009-01-12 (c) 2009 Matthew J. Rutherford
A stopping condition on the test-case creation activity
Help measure progress
E.g.,:
white-box: 100% branch coverage
black-box: all combinations of input space partitions
Class 01 - 2009-01-12 (c) 2009 Matthew J. Rutherford
1/12/09
5
Is primarily about test adequacy criteria
Instantiating a test adequacy criterion on an applicable software artifact gives a finite set of “test requirements”
The requirements create a TODO list for testing
Class 01 - 2009-01-12 (c) 2009 Matthew J. Rutherford
The relationship between criteria and bug-finding effectiveness is not really clear
It is likely that the effectiveness of criteria varies by system
Selecting the “right” criteria to use for your system is not possible
Take Home Message:
Use your brain when testing: add test cases that you think will be valuable, even if they don’t improve the coverage numbers!
Class 01 - 2009-01-12 (c) 2009 Matthew J. Rutherford
1/12/09
6
I love to study testing techniques:
Very hands-on and empirical
Interested in techniques to help reduce the risk of using adequacy criteria
Clear engineering tradeoff:
Resources versus quality
I love to use testing techniques:
Unit testing my own code
System-level testing
Testing other people’s code
Class 01 - 2009-01-12 (c) 2009 Matthew J. Rutherford
I have, unfortunately, delivered a lot of code without testing it
Schedule pressure
Budget constraints
…
Have to be realistic:
There is always a bigger picture than the purely technical in a project
Systems need to be delivered to be used
Class 01 - 2009-01-12 (c) 2009 Matthew J. Rutherford
1/12/09
7
10 classes, including today, 5pm – 7:50pm
Mix of lecture, discussion and working problems
Help me out with breaks, etc.
Office hours
Before class (starting at 4pm?)
After class (8pm -?)
Any other time over the phone (appointments work best)
Sound reasonable?
Class 01 - 2009-01-12 (c) 2009 Matthew J. Rutherford
http://mjrutherford.org/teaching/2009/winter/COMP3705
Slides will be available for download
Homework / reading assignments for each class
Can you guys access this easily from here?
Class 01 - 2009-01-12 (c) 2009 Matthew J. Rutherford
1/12/09
8
I’ll bring hard copies to class each week
They will also be available for download
(Probably at about 3:30pm on Mondays)
(c) 2009 Matthew J. Rutherford Class 01 - 2009-01-12
Introduction to Software Testing by Paul Ammann and
Jeff Offutt
Great new book! http://cs.gmu.edu/~offutt/softwaretest/
Class 01 - 2009-01-12 (c) 2009 Matthew J. Rutherford
1/12/09
9
Homework: 50%
The lowest 1 grade out of 8 will be dropped
Will morph into project milestones near the end of the quarter
Midterm Exam: 25%
February 16 th – Parts I and II of the textbook
Final Exam: 25%
March 16 th - Cumulative
Class 01 - 2009-01-12 (c) 2009 Matthew J. Rutherford
Please do the readings before class
Homework is due at the start of class each week, will not be accepted late unless you make prior arrangements
Please DO discuss the subject matter and homework problems outside of class. Please DO NOT copy.
Class 01 - 2009-01-12 (c) 2009 Matthew J. Rutherford
1/12/09
10
Class 01 - 2009-01-12 (c) 2009 Matthew J. Rutherford
www.introsoftwaretesting.com
1/12/09
11
In the real-world, testing is the principle post-design activity
Restricting early testing usually increases cost
Extensive hardware-software integration requires more testing
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 23
Written test objectives and requirements are rare
What are your planned coverage levels?
How much testing is enough?
Common objective – spend the budget …
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 24
1/12/09
12
Not testing is even more expensive
Planning for testing after development is prohibitively expensive
A test station for circuit boards costs half a million dollars …
Software test tools cost less than $10,000 !!!
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 25
They’re teaching a new way of plowing over at the Grange tonight - you going?
Naw - I already don’t plow as good as I know how...
1/12/09
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 26
13
1.
Types of test activities
2.
Software testing terms
3.
Changing notions of testing
– test coverage criteria
– criteria based on structures
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt
Testing can be broken up into four general types of activities
1.
Test Design
2.
Test Automation
3.
Test Execution
4.
Test Evaluation
Each type of activity requires different skills, background knowledge, education and training
No reasonable software development organization uses the same people for requirements, design, implementation, integration and configuration control
27
28 Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt
1/12/09
14
This is the most technical job in software testing
Requires knowledge of :
– Discrete math
– Programming
– Testing
Requires much of a traditional CS degree
This is intellectually stimulating, rewarding, and challenging
Test design is analogous to software architecture on the development side
Using people who are not qualified to design tests is a sure way to get ineffective tests
29 Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt
This is slightly less technical
Requires knowledge of programming
– Fairly straightforward programming – small pieces and simple algorithms
Requires very little theory
Very boring for test designers
Programming is out of reach for many domain experts
Who is responsible for determining and embedding the expected outputs ?
– Test designers may not always know the expected outputs
– Test evaluators need to get involved early to help with this
30 Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt
1/12/09
15
This is easy – and trivial if the tests are well automated
Requires basic computer skills
– Interns
– Employees with no technical background
Asking qualified test designers to execute tests is a sure way to convince them to look for a development job
If, for example, GUI tests are not well automated, this requires a lot of manual labor
Test executors have to be very careful and meticulous with bookkeeping
31 Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt
This is much harder than it may seem
Requires knowledge of :
– Domain
– Testing
Usually requires almost no traditional CS
– A background in the domain of the software is essential
– An empirical background is very helpful (biology, psychology, …)
– A logic background is very helpful (law, philosophy, math, …)
This is intellectually stimulating, rewarding, and challenging
– But not to typical CS majors – they want to solve problems and build things
Domain experts must be free to add new tests
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 32
1/12/09
16
1. Test design Design test values to satisfy coverage criteria or other engineering goal
Requires technical knowledge of discrete math, programming and testing
2. Test automation Embed test values into executable scripts
Requires knowledge of scripting
3. Test execution Run tests on the software and record the results
Requires very little knowledge
4. Test evaluation Evaluate results of testing, report to developers
Requires domain knowledge
These four general test activities are quite different
It is a poor use of resources to use people inappropriately
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 33
1/12/09
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 34
17
Like any field, software testing comes with a large number of specialized terms that have particular meanings in this context
Some of the following terms are standardized, some are used consistently throughout the literature and the industry, but some vary by author, topic, or test organization
The definitions here are intended to be the most commonly used
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt
IEEE
Validation : The process of evaluating software at the end of software development to ensure compliance with intended usage
Verification : The process of determining whether the products of a given phase of the software development process fulfill the requirements established during the previous phase
IV&V stands for “ independent verification and validation ”
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 36
35
1/12/09
18
Test Engineer : An IT professional who is in charge of one or more technical test activities
– designing test inputs
– producing test values
– running test scripts
– analyzing results
– reporting results to developers and managers
Test Manager : In charge of one or more test engineers
– sets test policies and processes
– interacts with other managers on the project
– otherwise helps the engineers do their work
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 37
Test
Manager design
Test
Designs
Test
Engineer instantiate
Executable
Tests
Test
Engineer
P Computer execute
Output Evaluate
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 38
1/12/09
19
Static Testing : Testing without executing the program
– This include software inspections and some forms of analyses
– Very effective at finding certain kinds of problems – especially “potential” faults, that is, problems that could lead to faults when the program is modified
Dynamic Testing : Testing by executing the program with real inputs
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt
Software Fault : A static defect in the software
Software Error : An incorrect internal state that is the manifestation of some fault
Software Failure : External, incorrect behavior with respect to the requirements or other description of the expected behavior
39
40 Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt
1/12/09
20
Testing : Finding inputs that cause the software to fail
Debugging : The process of finding a fault given a failure
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt
41
1/12/09
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 42
21
Test Case Values : The values that directly satisfy one test requirement
Expected Results : The result that will be produced when executing the test if the program satisfies it intended behavior
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt
Software Observability : How easy it is to observe the behavior of a program in terms of its outputs, effects on the environment and other hardware and software components
– Software that affects hardware devices, databases, or remote files have low observability
Software Controllability : How easy it is to provide a program with the needed inputs, in terms of values, operations, and behaviors
– Easy to control software with inputs from keyboards
– Inputs from hardware sensors or distributed software is harder
– Data abstraction reduces controllability and observability
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 44
43
1/12/09
22
Prefix Values : Any inputs necessary to put the software into the appropriate state to receive the test case values
Postfix Values : Any inputs that need to be sent to the software after the test case values
Two types of postfix values
1.
Verification Values : Values necessary to see the results of the test case values
2.
Exit Commands : Values needed to terminate the program or otherwise return it to a stable state
Executable Test Script : A test case that is prepared in a form to be executed automatically on the test software and produce a report
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 45
Top-Down Testing : Test the main procedure, then go down through procedures it calls, and so on
Bottom-Up Testing : Test the leaves in the tree (procedures that make no calls), and move up to the root.
– Each procedure is not tested until all of its children have been tested
1/12/09
46 Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt
23
Black-box testing : Deriving tests from external descriptions of the software, including specifications, requirements, and design
White-box testing : Deriving tests from the source code internals of the software, specifically including branches, individual conditions, and statements
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt
Old view of testing is of testing at specific software development phases
– Unit, module, integration, system …
New view is in terms of structures and criteria
– Graphs, logical expressions, syntax, input space
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 48
47
1/12/09
24
Class A method mA1() method mA2() main Class P
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com
Class B method mB1()
Acceptance testing: Is the software acceptable to the user?
System testing: Test the overall functionality of the system
Integration testing: Test how modules interact with each other method mB2()
© Ammann & Offutt
Module testing: Test each class, file, module or component
Unit testing: Test each unit (method) individually
49
Tailored to:
– a particular software artifact
• code, design, specifications
– a particular phase of the lifecycle
• requirements, specification, design, implementation
This viewpoint obscures underlying similarities
Graphs do not characterize all testing techniques well
Four abstract models suffice …
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 50
1/12/09
25
A tester’s job is simple : Define a model of the software, then find ways to cover it g
Test Requirements : Specific things that must be satisfied or covered during testing g
Test Criterion : A collection of rules and a process that define test requirements
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt
Structures : Four ways to model software
1.
Graphs
2.
Logical Expressions
3.
Input Domain
Characterization
4.
Syntactic Structures
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com
(not X or not Y) and A and B
A: {0, 1, >1}
B: {600, 700, 800}
C: {swe, cs, isa, infs} if (x > y)
z = x - y; else
z = 2 * x;
© Ammann & Offutt 52
51
1/12/09
26
2
1
3
This graph may represent
• statements & branches
• methods & calls
• components & signals
• states and transitions
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com
6
5 7
4
Path
Cover every path
• • • 12567
• • • 1257
• • 1357
• 1357
© Ammann & Offutt
• 1343567
• 134357 …
53
def = {a , m}
2 def = {x, y}
1 use = {x} use = {x} def = {a}
3 def = {m} use = {y}
6
5 use = {a} use = {a} use = {m}
7
This graph contains:
• defs: nodes & edges where variables get values
• uses: nodes & edges where values are accessed
4 def = {m} use = {y}
•
All Uses
Every def “reaches” every use
• •
• 1, 2, 5, 6, 7
•
3, (5,6)), (a, 3, (5,7)),
•
•
•
•
•
1, 2, 5, 7
1, 3, 5, 6, 7
• 1, 3, 5, 7
• 1, 3, 4, 3, 5,7
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 54
1/12/09
27
Guard (safety constraint) Trigger (input)
[Ignition = off] | Button2
(
Driver 1
Configuration to Modified )
Ignition = off
Driver 2
Configuration
[Ignition = off] | Button1
New
Configuration
Driver 2
Ignition = off
[Ignition = on] | seatBack ()
[Ignition = on] | seatBottom ()
[Ignition = on] | lumbar ()
[Ignition = on] | sideMirrors ()
[Ignition = on] | Reset AND Button2
New
Configuration
Driver 1
Modified
Configuration
[Ignition = on] | Reset AND Button1
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 55
( (a > b) or G ) and (x < y)
Transitions
Program Decision Statements
Software Specifications
Logical
Expressions
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 56
1/12/09
28
( (a > b) or G ) and (x < y)
Predicate Coverage : Each predicate must be true and false
– ( (a>b) or G ) and (x < y) = True, False
Clause Coverage : Each clause must be true and false
– (a > b) = True, False
– G = True, False
– (x < y) = True, False
Combinatorial Coverage : Various combinations of clauses
– Active Clause Coverage : Each clause must determine the predicate’s result
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 57
Describe the input domain of the software
– Identify inputs, parameters, or other categorization
– Partition each input into finite sets of representative values
– Choose combinations of values
System level
– Number of students { 0, 1, >1 }
– Level of course { 600, 700, 800 }
– Major { swe, cs, isa, infs }
Unit level
– Parameters F (int X, int Y)
– Possible values X : { <0, 0, 1, 2, >2 }, Y : { 10, 20, 30 }
– Tests
• F (-5, 10), F (0, 20), F (1, 30), F (2, 10), F (5, 20)
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 58
1/12/09
29
Based on a grammar, or other syntactic definition
Primary example is mutation testing
1.
Induce small changes to the program: mutants
2.
Find tests that cause the mutant programs to fail: killing mutants
3.
Failure is defined as different output from the original program
4.
Check the output of useful tests on the original program
Example program and mutants if (x > y)
z = x - y; else
z = 2 * x; if (x > y)
Δ if (x >= y)
z = x - y;
Δ z = x + y;
Δ z = x – m; else
z = 2 * x;
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 59
These structures can be extracted from lots of software artifacts
– Graphs can be extracted from UML use cases, finite state machines, source code, …
– Logical expressions can be extracted from decisions in program source, guards on transitions, conditionals in use cases, …
Model-based testing derives tests from a model that describes some aspects of the system under test
– The model usually describes part of the behavior
– The source is usually not considered a model
60 Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt
1/12/09
30
Applied to
Applied to
Applied to
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt
61
Infeasible test requirements : test requirements that cannot be satisfied
– No test case values exist that meet the test requirements
– Dead code
– Detection of infeasible test requirements is formally undecidable for most test criteria
Thus, 100% coverage is impossible in practice
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 62
1/12/09
31
1.
Directly generate test values to satisfy the criterion often assumed by the research community most obvious way to use criteria very hard without automated tools
2.
Generate test values externally and measure against the criterion usually favored by industry
– sometimes misleading
– if tests do not reach 100% coverage, what does that mean?
Test criteria are sometimes called metrics
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 63
Generator : A procedure that automatically generates values to satisfy a criterion
Recognizer : A procedure that decides whether a given set of test values satisfies a criterion
Both problems are provably undecidable for most criteria
It is possible to recognize whether test cases satisfy a criterion far more often than it is possible to generate tests that satisfy the criterion
Coverage analysis tools are quite plentiful
64 Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt
1/12/09
32
Criteria Subsumption : A test criterion C1 subsumes C2 if and only if every set of test cases that satisfies criterion C1 also satisfies C2
Must be true for every set of test cases
Example : If a test set has covered every branch in a program
(satisfied the branch criterion), then the test set is guaranteed to also have covered every statement
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt
Traditional software testing is expensive and labor-intensive
Formal coverage criteria are used to decide which test inputs to use
More likely that the tester will find problems
Greater assurance that the software is of high quality and reliability
A goal or stopping rule for testing
Criteria makes testing more efficient and effective
65
66 Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt
1/12/09
33
1/12/09
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt
Level 0 : There’s no difference between testing and debugging
Level 1 : The purpose of testing is to show correctness
Level 2 : The purpose of testing is to show that the software doesn’t work
Level 3 : The purpose of testing is not to prove anything specific, but to reduce the risk of using the software
Level 4 : Testing is a mental discipline that helps all IT professionals develop higher quality software
68 Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt
67
34
Testing is the same as debugging
Does not distinguish between incorrect behavior and mistakes in the program
Does not help develop software that is reliable or safe
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt
Purpose is to show correctness
Correctness is impossible to achieve
What do we know if no failures?
– Good software or bad tests?
Test engineers have no:
– Strict goal
– Real stopping rule
– Formal test technique
– Test managers are powerless
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt
69
70
1/12/09
35
Purpose is to show failures
Looking for failures is a negative activity
Puts testers and developers into an adversarial relationship
What if there are no failures?
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt
Testing can only show the presence of failures
Whenever we use software, we incur some risk
Risk may be small and consequences unimportant
Risk may be great and the consequences catastrophic
Testers and developers work together to reduce risk
71
72 Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt
1/12/09
36
A mental discipline that increases quality
Testing is only one way to increase quality
Test engineers can become technical leaders of the project
Primary responsibility to measure and improve software quality
Their expertise should help the developers
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt
More testing saves money
– Planning for testing saves lots of money
Testing is no longer an “art form”
– Engineers have a tool box of test criteria
When testers become engineers, the product gets better
– The developers get better
73
74 Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt
1/12/09
37
Which criteria work best on embedded, highly reliable software?
– Which software structure to use?
How can we best automate this testing with robust tools?
– Deriving the software structure
– Constructing the test requirements
– Creating values from test requirements
– Creating full test scripts
– Solution to the “mapping problem”
Empirical validation
Technology transition
Application to new domains
75 Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt
Section 1.2, Problem #3 (a), (b), (c)
1/12/09
Class 01 - 2009-01-12 (c) 2009 Matthew J. Rutherford
38
1/12/09
www.introsoftwaretesting.com
Applied to
Applied to
Applied to
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 78
39
•
Graphs are the most commonly used structure for testing
•
Graphs can come from many sources
– Control flow graphs
– Design structure
– FSMs and statecharts
– Use cases
•
Tests usually are intended to “cover” the graph in some way
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt
•
A set N of nodes, N is not empty
•
A set N
0
of initial nodes, N
0
is not empty
•
A set N f
of final nodes, N f
is not empty
•
A set E of edges, each edge from one node to another
– ( n i
, n j
), i is predecessor, j is successor
79
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 80
1/12/09
40
0
0 1 2 0
1 2 3 4 5
3 7 8 9
N
N
0 f
= { 0 }
= { 3 }
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
N
0
= { 0, 1, 2 }
N f
= { 7, 8, 9 }
© Ammann & Offutt
6 1 2
3
N
0
= { }
N f
= { 3 }
81
•
•
•
•
Path : A sequence of nodes – [n
1
, n
2
, …, n
M
]
– Each pair of nodes is an edge
Length : The number of edges
– A single node is a path of length 0
Subpath : A subsequence of nodes in p is a subpath of p
Reach ( n ) : Subgraph that can be reached from n
0 1 2
3 4 5 6
Paths
[ 0, 3, 7 ]
[ 1, 4, 8, 5, 1 ]
[ 2, 6, 9 ]
Reach (0) = { 0, 3, 4, 7,
8, 5, 1, 9 }
Reach ({0, 2}) = G
Reach([2,6]) = {2, 6, 9}
7 8
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
9
© Ammann & Offutt 82
1/12/09
41
•
•
Test Path : A path that starts at an initial node and ends at a final node
Test paths represent execution of test cases
– Some test paths can be executed by many tests
– Some test paths cannot be executed by any tests
•
SESE graphs : All test paths start at a single node and end at another node
– Single-entry, single-exit
– N0 and Nf have exactly one node
0
1
2
3
4
5
6
Double-diamond graph
Four test paths
[ 0, 1, 3, 4, 6 ]
[ 0, 1, 3, 5, 6 ]
[ 0, 2, 3, 4, 6 ]
[ 0, 2, 3, 5, 6 ]
© Ammann & Offutt Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com 83
•
Visit : A test path p visits node n if n is in p
A test path p visits edge e if e is in p
•
Tour : A test path p tours subpath q if q is a subpath of p
Path [ 0, 1, 3, 4, 6 ]
Visits nodes 0, 1, 3, 4, 6
Visits edges (0, 1), (1, 3), (3, 4), (4, 6)
Tours subpaths (0, 1, 3), (1, 3, 4), (3, 4, 6), (0, 1, 3, 4), (1, 3, 4, 6)
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 84
1/12/09
42
• path ( t ) : The test path executed by test t
• path ( T ) : The set of test paths executed by the set of tests T
•
•
Each test executes one and only one test path
A location in a graph (node or edge) can be reached from another location if there is a sequence of edges from the first location to the second
– Syntactic reach : A subpath exists in the graph
– Semantic reach : A test exists that can execute that subpath
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 85
test 1 many-to-one test 2
Test
Path test 3
Deterministic software – a test always executes the same test path many-to-many test 1 Test Path 1 test 2 Test Path 2 test 3 Test Path 3
Non-deterministic software – a test can execute different test paths
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 86
1/12/09
43
•
We use graphs in testing as follows :
– Developing a model of the software as a graph
– Requiring tests to visit or tour specific sets of nodes, edges or subpaths
•
•
•
Test Requirements (TR) : Describe properties of test paths
Test Criterion : Rules that define test requirements
Satisfaction : Given a set TR of test requirements for a criterion C, a set of tests T satisfies C on a graph if and only if for every test requirement in TR, there is a test path in path(T) that meets the test requirement tr
•
•
Structural Coverage Criteria : Defined on a graph just in terms of nodes and edges
Data Flow Coverage Criteria : Requires a graph to be annotated with references to variables
© Ammann & Offutt 87 Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
•
The first (and simplest) two criteria require that each node and edge in a graph be executed
•
This statement is a bit cumbersome, so we abbreviate it in terms of the set of test requirements
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 88
1/12/09
44
•
Edge coverage is slightly stronger than node coverage
•
The “length up to 1” allows for graphs with one node and no edges
•
NC and EC are only different when there is an edge and another subpath between a pair of nodes (as in an “ if-else ” statement)
0
Node Coverage : TR = { 0, 1, 2 }
Test Path = [ 0, 1, 2 ]
1
2
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
Edge Coverage : TR = { (0,1), (0, 2), (1, 2) }
Test Paths = [ 0, 1, 2 ]
[ 0, 2 ]
© Ammann & Offutt 89
•
A graph with only one node will not have any edges
0
•
It may be boring, but formally, Edge Coverage needs to require
Node Coverage on this graph
•
Otherwise, Edge Coverage will not subsume Node Coverage
– So we define “length up to 1” instead of simply “length 1”
•
We have the same issue with graphs that only have one edge – for Edge Pair Coverage …
0
1
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 90
1/12/09
45
•
Edge-pair coverage requires pairs of edges, or subpaths of length 2
•
The “length up to 2” is used to include graphs that have less than 2 edges
•
The logical extension is to require all paths …
•
Unfortunately, this is impossible if the graph has a loop, so a weak compromise is to make the tester decide which paths:
© Ammann & Offutt Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com 91
1
3
2
6
0
Node Coverage
TR = { 0, 1, 2, 3, 4, 5, 6 }
Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 5, 4, 6 ]
4
Edge Coverage
TR = { (0,1), (0,2), (1,2), (2,3), (2,4), (3,6), (4,5), (4,6), (5,4) }
Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 2, 4, 5, 4, 6 ]
Edge-Pair Coverage
TR = { [0,1,2], [0,2,3], [0,2,4], [1,2,3], [1,2,4], [2,3,6],
[2,4,5], [2,4,6], [4,5,4], [5,4,5], [5,4,6] }
Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 6 ] [ 0, 2, 3, 6 ]
[ 0, 2, 4, 5, 4, 5, 4, 6 ]
5
Complete Path Coverage
Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 6 ] [ 0, 1, 2, 4, 5, 4, 6 ] [ 0, 1, 2,
4, 5, 4, 5, 4, 6 ] [ 0, 1, 2, 4, 5, 4, 5, 4, 5, 4, 6 ] …
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 92
1/12/09
46
•
If a graph contains a loop, it has an infinite number of paths
•
Thus, CPC is not feasible
•
SPC is not satisfactory because the results are subjective and vary with the tester
•
Attempts to “deal with” loops:
– 1970s : Execute cycles once ([4, 5, 4] in previous example, informal)
– 1980s : Execute each loop, exactly once (formalized)
– 1990s : Execute loops 0 times, once, more than once (informal description)
– 2000s : Prime paths
© Ammann & Offutt Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com 93
•
•
Simple Path : A path from node ni to nj is simple if no node appears more than once, except possibly the first and last nodes are the same
– No internal loops
– Includes all other subpaths
– A loop is a simple path
Prime Path : A simple path that does not appear as a proper subpath of any other simple path
1
0
2
Simple Paths : [ 0, 1, 3, 0 ], [ 0, 2, 3, 0], [ 1, 3, 0, 1 ],
[ 2, 3, 0, 2 ], [ 3, 0, 1, 3 ], [ 3, 0, 2, 3 ], [ 1, 3, 0, 2 ],
[ 2, 3, 0, 1 ], [ 0, 1, 3 ], [ 0, 2, 3 ], [ 1, 3, 0 ], [ 2, 3, 0 ],
[ 3, 0, 1 ], [3, 0, 2 ], [ 0, 1], [ 0, 2 ], [ 1, 3 ], [ 2, 3 ], [ 3, 0 ], [0],
[1], [2], [3]
3
Prime Paths : [ 0, 1, 3, 0 ], [ 0, 2, 3, 0], [ 1, 3, 0, 1 ],
[ 2, 3, 0, 2 ], [ 3, 0, 1, 3 ], [ 3, 0, 2, 3 ], [ 1, 3, 0, 2 ],
[ 2, 3, 0, 1 ]
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 94
1/12/09
47
•
A simple, elegant and finite criterion that requires loops to be executed as well as skipped
•
•
Will tour all paths of length 0, 1, …
That is, it subsumes node, edge, and edge-pair coverage
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt
•
Round-Trip Path : A prime path that starts and ends at the same node
95
1/12/09
•
•
These criteria omit nodes and edges that are not in round trips
That is, they do not subsume edge-pair, edge, or node coverage
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 96
48
•
•
The previous example has 38 simple paths
Only nine prime paths
0
1
3
2
6
4
5
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
Prime Paths
[ 0, 1, 2, 3, 6 ]
[ 0, 1, 2, 4, 5 ]
[ 0, 1, 2, 4, 6 ]
[ 0, 2, 3, 6 ]
[ 0, 2, 4, 5]
[ 0, 2, 4, 6 ]
[ 5, 4, 6 ]
[ 4, 5, 4 ]
[ 5, 4, 5 ]
© Ammann & Offutt
Execute loop 0 times
Execute loop once
Execute loop more than once
97
•
Prime paths do not have internal loops … test paths might
•
Tour : A test path p tours subpath q if q is a subpath of p
•
Tour With Sidetrips : A test path p tours subpath q with sidetrips iff every edge in q is also in p in the same order
•
The tour can include a sidetrip, as long as it comes back to the same node
•
Tour With Detours : A test path p tours subpath q with detours iff every node in q is also in p in the same order
•
The tour can include a detour from node ni , as long as it comes back to the prime path at a successor of ni
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 98
1/12/09
49
1 2 3 4
0 1 2 4 5
Touring without sidetrips or detours
3
1 6
0
Touring with a sidetrip
0
1
Touring with a detour
1
1
2
3
2
4
5
3
2
2
3
4
3
4
4
5
5
5
© Ammann & Offutt Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com 99
•
An infeasible test requirement cannot be satisfied
– Unreachable statement (dead code)
– A subpath that can only be executed if a contradiction occurs ( X > 0 and X < 0 )
•
•
•
•
Most test criteria have some infeasible test requirements
It is usually undecidable whether all test requirements are feasible
When sidetrips are not allowed, many structural criteria have more infeasible test requirements
However, always allowing sidetrips weakens the test criteria
Practical recommendation – Best Effort Touring
– Satisfy as many test requirements as possible without sidetrips
– Allow sidetrips to try to satisfy unsatisfied test requirements
Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com © Ammann & Offutt 100
1/12/09
50
1
3
0
2
6
4
Simple paths
Len 0
[0]
[1]
[2]
[3]
[4]
[5]
[6] !
‘!’ means path terminates
Len 1
[0, 1]
[0, 2]
[1, 2]
[2, 3]
[2, 4]
[3, 6] !
[4, 6] !
[4, 5]
[5, 4]
5
Len 4
[0, 1, 2, 3, 6] !
[0, 1, 2, 4, 6] !
[0, 1, 2, 4, 5] !
Len 2
[0, 1, 2]
[0, 2, 3]
[0, 2, 4]
[1, 2, 3]
[1, 2, 4]
[2, 3, 6] !
[2, 4, 6] !
[2, 4, 5] !
[4, 5, 4] *
[5, 4, 6] !
[5, 4, 5] *
Len 3
[0, 1, 2, 3]
[0, 1, 2, 4]
[0, 2, 3, 6] !
[0, 2, 4, 6] !
[0, 2, 4, 5] !
[1, 2, 3, 6] !
[1, 2, 4, 5] !
[1, 2, 4, 6] !
‘*’ means path cycles
Prime Paths
© Ammann & Offutt 101 Introduction to Software Testing (Ch 2), www.introsoftwaretesting.com
Section 2.2.1, Exercise #4
(b) List test paths that achieve node coverage, but not edge coverage
(c) List test paths that achieve edge coverage, but not edge-pair coverage
(d) List test paths that achieve edge-pair coverage
Class 01 - 2009-01-12 (c) 2009 Matthew J. Rutherford
1/12/09
51
Read Chapters 1 and 2
Homework #1
Due January 19 th at the start of class
Section 1.1 Exercise #2
Section 2.1 Exercise #1, #2
Class 01 - 2009-01-12 (c) 2009 Matthew J. Rutherford
1/12/09
52