Verification Methods
CS2110: SW Development Methods
•Reading: Study Section 1.2.5 in the MSD text
•These slides supplement the textbook
– These touch on other important methods,
but focus on execution-based testing
(1)
Some Definitions
• Verification: showing that something meets
specifications
– We verify “against” something
– Important: if you don’t have requirements
specifications documented, then you can’t verify!
– If behavior is not defined, then what counts as
incorrect behavior?
• Verification can happen on any kind of software workproduct
– designs, code, etc.
(2)
Verification Techniques
At least four approaches to verifying code quality:
1. Execution-based Testing (running tests)
2. Inspections
– People examine the work product by eye
3. Static Analysis by a tool
– For C programs, an old tool called “lint”
– For Java programs, some nice tools integrate
with Eclipse: CheckStyle and PMD
4. Formal Correctness Proofs
– Use mathematics to demonstrate the desired
property
(3)
Execution-based Testing
Some Basic Definitions:
• Fault: What we think of as a bug
– Something in the code that’s wrong
• Failure: Observed incorrect behavior of a system
resulting from a fault
• Error: Mistake made by a developer
A possible definition of execution-based testing:
– Process of inferring certain behavioral properties
of a product based, in part, on results of executing
it in a known environment with selected inputs.
(4)
Famous and Important Quote
• Program testing can effectively show the
presence of bugs but is hopeless for showing
their absence.
– Edsger Dijkstra
– Famous computer scientist
– Turing Award winner
– Wrote "Go To Statement
Considered Harmful“ (1968)
– Shortest-path algorithm
(5)
Types of Testing
• Unit testing
– An individual class or function is tested thoroughly
– Done as part of coding process
– Done by the programmer him/herself
– Usually by a test harness (or driver) or using a
test-class
• Integration testing
– Multiple parts of a system are combined and
tested to see if they work together
• Acceptance testing
– The customer is involved. (Alpha or beta testing)
(6)
Approaches to Unit Testing
• Two approaches
– Test only based on inputs, outputs (i.e. the
specification) without looking at the code
– Test based on the logic of the code (must study it)
• The 1st of these is called black-box testing
• The 2nd of these is glass-box or white-box testing
• Some principles of glass-box testing
– Make sure every line (or “case”) gets executed
– Make sure loops get done zero times, >zero times
– Etc.
(7)
Limitations of Glass-Box Testing
• Why not fully exercise our code?
• Logic of a piece of code defines a set of possible
execution paths, “flow-paths,” through the function
– Think of a flow-chart
– The inputs control which path is taken
• A good set of test data might make sure every
possible path is followed
– This tests every possible behavior
• Problem: for even small programs with loops and
conditionals, the total number of paths becomes too
large too quickly
– See MSD text, p. 24-25
(8)
Black-Box Testing
• This means testing based on inputs to the codemodule without basing tests on code’s internals
– You can’t see inside a black-box
– Note: MSD’s definition on p. 26 isn’t the usual
one
• Choose test cases for common problems
– Boundary values, boundary conditions
– Off-by-one errors
– Incorrect inputs
• Important: You have to know in advance what the
expected result of any test should be!
(9)
Lab on Monday/Tuesday
• The JUnit (v. 3) framework is a tool to help people generate and
format unit tests in a systematic way.
• Stand-alone and also public class CoffeeMakerTest
built into Eclipse
extends TestCase {
private CoffeeMaker cm;
• Example JUnit test
private Recipe recipe1;
– Class contains all
tests
public void setUp() {
cm = new CoffeeMaker();
– setUp() performs
preconditions
recipe1 = new Recipe();
recipe1.setName("Coffee");
– “test…” is an
recipe1.setPrice(50);
individual test case
recipe1.setAmtCoffee(6);
– “assert” checks a
recipe1.setAmtMilk(1);
program condition
}
public void testAddRecipe() {
assertTrue(cm.addRecipe(recipe1));
}
}
(10)
(11)
When we test a class’ method to see if its
loop goes to size-1 and not size, is this
clearly an example of:
1.
2.
3.
4.
5.
A black box test
A glass-box test
A unit test
Both 1 and 3
Both 2 and 3
(12)
Planning Your Tests
• How do we plan a “test case”?
Define three parts first:
– A purpose (why are we doing this particular test?)
– An input
– An expected result
• We execute the test to see if observed output
matched the expected result
– It’s really just like a scientific experiment
• When testing, have a plan (at least in your head)
– Don’t test blindly or randomly
(13)
Planning Your Tests
Test ID
Description
Expected
Results
1
Player 1 rolls dice
and moves.
Player 1
moves on the
board.
2
Player 2 rolls dice
and moves.
Player 2
moves on the
board.
Actual
Results
• Poorly specified test cases…
(14)
Planning Your Tests
Test ID
Description
Expected
Results
1
Precondition: Game is in
test mode, GameBoard is
loaded, and game begins.
Number of Players: 2
Player 1 dice roll: 3
Player 1
moves to
square
Blue 3.
2
Precondition: TestID 1
completed successfully.
Player 2 dice roll: 3
Player 2
moves to
square
Blue 3.
Actual
Results
• Why are these test cases better than the previous
ones?
(15)
Unit Testing: Boundary values
• Often programs misbehave because we make
mistakes dealing with the “extremes”
• Examples:
– The last item or first item in a collection
– The start or end of a range of values
• Equivalent classes of test cases
– Don’t have test cases or data that test the same
situation (e.g. the middle of a range)
• Boundary or extreme conditions
– An empty list, an empty file
– One item in a list or file
(16)
Unit Testing: Off-by-one errors
•
•
•
•
Related to boundary conditions
Using the nth value instead of n-1
Using 0 instead of 1
Executing a loop body once when it the body should
not be executed at all
– Wait – is this one a black-box testing method?
Why or why not?
• Etc.
(17)
Unit Testing: Invalid Inputs
• Rely on pre-conditions defined for your functions
– They specify what the agreement is about what
you are responsible for when its time to handle
mis-use
• If a function is mis-used in this way
– Return an error condition if the interface provides
a way to do this
– Otherwise, halt program! Better than producing
unexpected results
• Use assertions (more later!)
(18)
Example: Program with Student and Course
classes
• Homework 1 this term
– Define class Student
• methods include:
– Define class Course
• methods include:
• Focus on defining:
– Black-box test-cases for the method
boolean drop(Course c) in class Student
• Let’s make a list of possible test-cases for this
method!
(19)
Example of Full Test-Case Description
• Test Case #1: Drop last course in list with >1 item
– Purpose: Drop Course c for Student s where that
course is the last course stored in the Student
object's courses list. This tests the boundary
condition of last item in list for Student.drop(c).
– Input: Student s is enrolled in three courses, and
the course to be dropped is the last of these
stored in the list.
– Expected output: The Student s is unchanged
except that only two courses are stored for that
student. Also, the list of students stored for that
course is altered by having that student removed.
(20)
Other Test Cases for this Program
• There are a lot!
• Boundary conditions?
– for Student.drop(): end of list, start of list
– same for Course.drop()
– Question: affected by whether you used ArrayList
instead of Java arrays?
– Note: also test “typical” case, item in middle of list
– Empty lists: Course.cancel(), Student.dropAll()
• Off-by-one errors?
– Really included in boundary condition testing here
(21)
Other Test Cases for this Program
• Invalid inputs?
– See the specifications!
– Test return values
• For boolean, test both success and failure
• Remember: make it fail!
• Test of equals() methods?
– If you get this wrong, trouble!
– Easy to test. (Easy to forget to.)
(22)
Good or Bad Test?
Student s adds a course
and the course appears
in the schedule
1.
2.
Good Test?
Poor Test?
(23)
Good or Bad Test?
Student s currently is in
the system and currently
has no courses.
Student s adds a course
number 394 and it appears
at the end of the schedule.
1.
2.
Good Test?
Poor Test?
(24)
Good or Bad Test?
Student s has 4 courses,
drops 3, adds 1. Check if the
student is then deleted
from the system.
1.
2.
Good Test?
Poor Test?
(25)
Extra Slides
(26)
#2: Inspections: We Might See These Later
• Many different “styles” or approaches
– Walk-throughs, Inspections
• Advantages:
– You can inspect all of a product but usually it is
impossible to test all of a product.
– Applicable to all work products (e.g. code,
designs, requirements documents, etc.)
– Fairly easy: training, application, follow-up
• Disadvantages:
– Staff time is expensive
– Works best if follows a rigorous process
(27)