Lecture 03
Software Testing Terminology
Limitation of Software Testing
• Testing can only show presence of failures, not their absence
• Fundamental, theoretical limitation
• Finding ALL failures is undecidable
1
July 24, 2016
Turing machine…
Limitations of Software Testing
• A “successful” or “effective” test- one that finds an error
• Example of Beizer’s level 2 thinking
2
July 24, 2016
Software Testing Terminology
•
•
•
•
•
•
•
•
•
validation
verification
IV&V
software fault
software error
software failure
testing
test failure
debugging
3
July 24, 2016
•
•
•
•
•
•
•
reachability
infection
propagation
RIP model
test case values
expected results
software
observability
• software
controllability
•
•
•
•
•
prefix value
postfix values
test case
test set
test script
validation
• Process of evaluating software at the end of development
process to ensure compliance with intended usage
• Usually depends on domain knowledge
• Slightly more weight than verification
4
July 24, 2016
verification
• Process of determining whether products of a given phase of
development fulfill the requirements
• Usually a technical activity
• Uses knowledge of software artifacts, requirements, and
specifications
• Slightly less weight than validation
5
July 24, 2016
IV&V- independent verification and validation
• “independent”- evaluation done by non-developers
• Teams within project
• Sometimes within same company
• Sometimes an entirely different entity
• Usually not started until software is complete
• Often done by people with application domain expertise
(rather than software development)
6
July 24, 2016
software fault, error, and failure
• fault- A static defect in the software
• error- An incorrect internal state that is the manifestation of
some fault
• failure- external, incorrect behavior with respect to the
requirements or expected behavior
7
July 24, 2016
Example of fault, error, and failure
•
•
•
•
Patient and doctor
Patient enters doctor’s office with symptoms (failures)
Doctor must discover the root cause (fault)
To aid diagnosis, doctor may order tests
• Anomalous internal conditions -> errors
8
July 24, 2016
Software Errors are Design Mistakes
• Errors do not appear spontaneously
• Errors exist as a a result of a human decision
• “We can make software development foolproof, but we
cannot, and should not attempt to, make it damn-foolproof”
9
July 24, 2016
Code Example
//count number of zeros in x
int numZero( int *x, int length )
{
int count = 0;
for(int j = 1; j < length; j++){
if( x[j] == 0 ) count++;
}
return count;
}
10
July 24, 2016
• fault- loop starts at 1, not 0
numZero( [2,7,0] ,3) => 1
CORRECT!!
numZero( [0,7,2] ,3) => 0
INCORRECT!!
• both cases: fault executed
• both cases result in error
• only second case => failure
More Generally…
If the required sequence of states is { s0, s1, s2, … }
and the actual sequence of states is { s0, s2, s3, … },
then s2 is the error
11
July 24, 2016
Software Testing Terminology
•
•
•
•
•
•
•
•
•
validation
verification
IV&V
software fault
software error
software failure
testing
test failure
debugging
12
July 24, 2016
•
•
•
•
•
•
•
reachability
infection
propagation
RIP model
test case values
expected results
software
observability
• software
controllability
•
•
•
•
•
prefix value
postfix values
test case
test set
test script
testing, test failure, debugging
• testing- evaluating software by observing its execution
• test failure- execution that results in a failure
• debugging- the process of finding the fault given a failure
13
July 24, 2016
Central Issue in Software Testing…
• For a given fault, not all inputs will “trigger” the fault into
creating incorrect output (failure)
• Often very difficult to relate a failure to the associated fault
• Leads of fault / failure model (or RIP model)
14
July 24, 2016
Reachability, Infection, Propagation (RIP)
• Three conditions must be present for failure to be observed:
1. reachability- the fault location(s) in the program must be
reached
2. infection- after executing the location, the program state
must be incorrect
3. propagation- the infected state must propagate to cause
some incorrect program output
15
July 24, 2016
RIP Model
• reachability, infection, propagation important for:
• Coverage criteria
• Automatic test data generation
16
July 24, 2016
test case values
• Input values necessary to complete some execution of the
software under test
• Depends on software being tested
• E.g., Web applications vs. real-time embedded systems
• Test case values (alone) are not enough
• Must also test: commands, method calls, user inputs
17
July 24, 2016
expected results
• The result that will be produced when executing the test if and
only if the program satisfies the intended behavior
18
July 24, 2016
Two Common Problems with Software Testing
• How to provide the “right” values
• How to observe details of the software’s behavior
• TOPIC OF THIS COURSE!!
19
July 24, 2016
Software Testing Terminology
•
•
•
•
•
•
•
•
•
validation
verification
IV&V
software fault
software error
software failure
testing
test failure
debugging
20
July 24, 2016
•
•
•
•
•
•
•
reachability
infection
propagation
RIP model
test case values
expected results
software
observability
• software
controllability
•
•
•
•
•
prefix value
postfix values
test case
test set
test script
Software Observability and Controllability
• observability- how “easy” it is to observe software’s behavior
in terms of outputs, effects on environment, and other
hardware or software components
• controllability- how “easy” it is to provide a program with the
needed inputs, in terms of values, operations, and behaviors
21
July 24, 2016
Example of Observability and Controllability
• E.g., embedded software
• Does not often produce output for human consumption
• … but DOES affect the behavior of some hardware
• observability is LOW
• Input from hardware sensors is difficult to control
• Can be simulated, but not the same (e.g., auto-pilot controls)
• controllability is LOW
22
July 24, 2016
prefix values
• Any inputs necessary to put the software into an appropriate
state to receive test case values
23
July 24, 2016
postfix values
• Any necessary input values to be sent after test case values
• verification values- necessary to see results of test case values
• exit commands- values needed to terminate program or return
to a stable state
24
July 24, 2016
test case
• Contains necessary items to complete execution and
evaluation of software under test
• Composed of:
•
•
•
•
test case values
expected results
prefix values
postfix values
25
July 24, 2016
test set
• A set of test cases
• Coverage is a property of a set of test cases
26
July 24, 2016
e.g., branch coverage
test script
• A test case prepared in a form to be executed automatically on
the test software and produce a report
27
July 24, 2016
Next Steps
• Homework_01
• Question 3 - a, b, and f (page 16)
28
July 24, 2016