Testing
MSO 08/09, WP
Why testing ?

Obvious.

Still, how many of you actually test your assignment before
submitting them?

How do you know if you have tested enough??

How much testing really cost?
Seemingly a lot, since there are plenty of software testing
companies in NL; they seem to make good business!
2
Some numbers

US financial sector suffers 1.5 billion USD (2002) due to
inadequate testing (US Nat. Inst. of Standards and Tech.)

Testing cost 30 – 70% of total development cost! (various
studies)

Effort to find a bug (Ellims et all 2004, JTest)
at system testing 12 – 20 hrs (600 Eur per bug!)
at unit testing 0 – 3 hrs (90 Eur per bug)
3
Why is it so hard ?

Size
At the system level you can be dealing with millions LOCs!

OO adds a multitude of other complications.

Concurrency
4
Plan
Dennis et al only discuss testing very globally 







5
Unit test, integration test, system test
OO issues, concurrency issues
Testing with JUnit
Adequacy of your tests
Assignment-7
Specification-based testing
Regression testing
Taxonomy of testing

According to the "scope" :
 Unit testing
 Integration testing
 System testing

According to ... "other things"  :
 Functional testing
 GUI testing
 Acceptance testing
 Performance testing
 Regression testing
6
Unit Testing

Testing each "program unit" in isolation.

Manageable size; finding bugs are easier at this level.

Widely supported by xUnit tools.

Automated tools e.g. T2.

We'll take a look at Junit.
7
Unit Testing

What is "unit" ? Options :
 method
 class
 package

Typically people take method as "unit"
But you should be aware that in OO methods can influence
each other through their side effects on objects' states.
This implies that there are methods that should not be tested
in isolation.
OO issues complicates testing

Intra-class side effect makes it difficult to test each method in
isolation:
class Subscription {
int price ;
int n = 12 ;
public Subscription(p) { price = p }
public int pricePerMonth() { return price/n }
public cancel() { price=0; n=0 }
}
9
OO issues

We fix Subscription and find no more error. So, now we're sure that every
client can safely call "pricePerMonth".
class client {
m(Subscription s) { s.cancel() ; return s.pricePerMonth() }

Watch out!! Inheritance adds another complication 
class EvilSubscription extends Subscription {
public int pricePerMonth() { return 0/0 }
...
This implies that whenever you override a method, you need to test if you still
respect the spec of the original method (Lipkov substitution).
10
Integration test

Test if a set of "units" (that logically belong together)
functions expectedly.

But what is a "set of units" ?  Class or package

Class level testing. Given a class C, check that
Various sequences of calls to C's methods and updates to C's
exposed fields do not violate C's class invariant and the
specs of each method.

11
Lots of work to do by hand; use automated tool (T2).
Integration test

Package level testing.
SubsctiptionPackage
Customer
Subscription
Service
price
...
classinv price 0
Potential problem: if there are fields in Subscription which are open
to package-level access, but their values are constrained by a class
invariant.
Implying you have to test that every method in the package repects the class
invariant of every class in the package.
12
Concurrency

Concurrent threads multiply the number of possible
scenarios to consider:
thread 1 : { price++ ; price++ }
thread 2 : { price = 0 }


Will this satisfy :
0  price  2 ?
This implies you need lots of tests to cover all scenarios 
expensive!!
But even that is not possible without special tools:
 We can't control JVM to do a specific scheduling.
 This also implies that you can't replay an error in a
concurrent setup!
And multi-cores are coming!
(shudder)
13
The famous V model of testing
Image is taken from Wikipedia.
14
V model of testing
System
Requirement
acceptance
testing
System
Specifications
Package/class
level
specifications
Implementation
This approach
assumes a water-fall
SDLC model.
15
system testing
integration
testing
unit testing
But more generally
phases can run parallel.
System test

System testing tests if a software as a whole functions as
expected.

Problem: at this level a software often has to be interacted
through its (complex) GUI.

Testing through GUI is HARD to write and hard to automate!!

Alternative: split system testing to
 Functional system testing
 Separate GUI testing
16
Functional System Testing

You need a well defined interface (set of methods) through
which we can directly access a software functionality
(without going through the GUI).
We will test through this interface.
GUI
Test Interface
Business Logic

17
Ideally this interface is prepared all the way from your design
phase.
Test Plan

For a large application you (obviously) need a test plan.
Because it involves lots of activities (and huge investment).

18
IEEE 829, test plan is a document:
1.
describing scope, approach, resources, and schedules of
testing activities.
2.
identifying test items, features to test, testing tasks, who
will do the tasks, risks, and contingency plans.
Test Plan

IEEE 829 







Oh well, a test plan essentially:



19
c) Approach
d) Test items
e) Features to be tested; features not to be tested
g) Pass/fail criteria
h) Suspension and resumption criteria
i) Test deliverables
Defines the scope of the testing
Explains how we do the testing
Estimates the resources needed
(Unit) Testing with Junit
C
Ctest
report
Junit
Java
public class Subscription {
private int price ;
private int length ;
...
public double pricePerMonth() {
double r = (double) price / (double) length ;
return r ;
}
}
20
Testing with Junit
C
Ctest
import org.junit.* ;
import static org.junit.Assert.* ;
report
Junit
Java
public class SubscriptionTest {
@Test
public void test_if_pricePerMonth_returns_Euro() {
Subscription S = new Subscription(200,2) ;
assertTrue(S.pricePerMonth() == 1.0) ;
}
@Test
public void test_if_pricePerMonth_rounds_up() {
Subscription S3 = new Subscription(200,3) ;
assertTrue(S3.pricePerMonth() == (double) 0.67) ;
}
}
Read the Tutorial, linked from Opdracht-7.
21
JUnit with Eclipse
22
JUnit with Eclipse
23
Have we tested enough ??

In principle, no way to know that.

Still, we can use "coverage" as indication:
The precentage of "parts" of your code that are executed
during your tests.

Keep in mind: 100% coverage does not imply absence of
bugs!!
But low coverage is obviously not good.
24
Various concepts of "coverage"

Very abstract:
 Class coverage (have we tested all classes?)
 Method coverage (have we tested all methods?)

Code level:
 Line coverage
 Branch coverage
 Path coverage
25
Line, branch, and path coverage
discount(Subscription s) {
if (s.period > 12) d = 0.05 else d=0
if (s.price > 1000) d += 0.05
return d
}


Line coverage: percentage of lines executed by your tests.
E.g. a test (1x) with
s.period = 0

26
s.price = 0
will give you 100% line coverage.
Very weak. But in practice is used most 
Line, branch, and path coverage
s.period > 12
discount(Subscription s) {
if (s.period > 12) d = 0.05 else d=0
if (s.price > 1000) d += 0.05
return d
}
s.price > 1000
But you still miss the "red" case!!


Branch coverage: percentage of branches (of "if", "while"
etc) executed by your tests.
E.g. the tests (2x)
s1.price=0
s2.price=2000
s1.period=0
s2.period=2000
gives 100% branch coverage.
27
Line, branch, and path coverage
s.period > 12
discount(Subscription s) {
if (s.period > 12) d = 0.05 else d=0
if (s.price > 1000) d += 0.05
return d
}
s.price > 1000

Path coverage: percentage of paths in the "control flow
graph" that your tests pass through.

There are 4 paths. You will need 4 tests to cover them all.
Much stronger.
Complication with loop & recursion  take e.g. "prime
paths".


28
Measuring Coverge in Eclipse
Read the Tutorial, linked from Opdracht-7.
29
Assignment 7 : Foo Subscription System
30
Foo architecture in more detail
31
Foo architecture in more detail
32
Discount

Abstract class DiscountStrategy
 description()
 calcDiscount(customer, activeDiscountStrategies)

Two concrete implementations:
 Discount_5top
 Discount_10pack

TASK 1: test these two classes (unit testing)
 Verify against the specifications (of the above two methods)
 100% (emma) coverage!
33
Problem with traditional tests
@Test
public void test_if_pricePerMonth_returns_Euro() {
Subscription S = new Subscription(200,2) ;
assertTrue(S.pricePerMonth() == 1.0) ;
}
@Test
public void test_if_pricePerMonth_rounds_up() {
Subscription S3 = new Subscription(200,3) ;
assertTrue(S3.pricePerMonth() == (double) 0.67) ;
}


34
We need concrete data as inputs and oracles  need
manual effort.
This is very fragile; if we change the functionality a bit, you'll
have to re-calculate them.
Alternative: specification-based testing

We'll write specifications in-code (in Java).
Alternative: use JML or OCL.
You'll need additional tools.
Your need to train your programmers in them.

TASK 2 : (optional, 1.5 pt)



Write specifications for ApplicationLogic, and do
specification-based testing.
We'll limit to the methods addCustomer and
removeCustomer of ApplicationLogic.
35
Architecture
ApplicationLogic_SPEC
ApplicationLogic_SPEC_test
addCustomer
removeCustomer
classinv
app
int addCustomer(String name, String email) {
... // pre-condition
ApplicationLogic
int result = app.addCustomer(name,email) ;
.... // post-condition
return result
}
36
Example of in-code specification : spec-1
Suppose this is the spec of addCustomer:
"Name and email should not be null nor empty (pre-cond). If name does
not occur in the system, then add the customer, return +1. Else don't add,
return -1."
int addCustomer(String name, String email) {
assertTrue(name != null && email != null && ! name.equals("") && ... )
boolean occur = false ;
for (String n : getCustNames())
if (n.equals(name)) { occur = true ; break }
int result = app.addCustomer(name,email) ;
if (occur) assertTrue(result == 1) else assertTrue(result == -1) ;
return result
37
}
Example of in-code specification : spec-2
Suppose we want to impose this class invariant on the application logic:
"The value of the variable NextCustId should be non-negative."
ApplicationLogic_SPEC
addCustomer
removeCustomer
classinv
boolean classinv() {
assertTrue(app.nextCustID  0) ;
return true ;
app
ApplicationLogic
38
}
Examples of the tests
import org.junit.*;
import static org.junit.Assert.*;
public class ApplicationLogic_test {
@Test
public void test1() {
ApplicationLogic.reset()
ApplicationLogic_SPEC applogic = new ApplicationLogic_SPEC()
assertTrue (applogic.classinv())
applogic.addCustomer("Pinky","Pinky@world.com")
assertTrue (applogic.classinv())
applogic.addCustomer("Pinky","Pinky@world.com")
assertTrue (applogic.classinv())
}
}
39
Advatanges of spec-based testing
public void test1() {
...
assertTrue (applogic.classinv())
applogic.addCustomer("Pinky","Pinky@world.com")
assertTrue (applogic.classinv())
applogic.addCustomer("Pinky","Pinky@world.com")
assertTrue (applogic.classinv())
}




40
No (fragile) oracle!
But we still have to manually write the test sequences and inputs. Can be
automated!  e.g. with T2
You must combine this with pair-programming.
Disadvantage:
 writing specs takes time
 Incure maintenace cost.
Regression test







41
The re-testing of a software that has been modified.
Is a serious problem for large software  may take days!
Between versions, the changes are usually very limited (to
some modules).
However, the impact may be global due to interactions
between modules.
This forces you to test the entire software. That is, you need
a test set that will deliver full coverage over your entire
software.
But if we just re-execute the whole set of tests collected so
far, this will take too long!!
Challenge: how do we identify a good minimum set?