Some Testing Techniques
(enhanced 6 + 4)
Course Software Testing & Verification
2013/14
Wishnu Prasetya
Plan
•
•
•
•
Unit testing
 tool
Mocking (Sec. 6.2.1)  enhanced
Black box testing  Input Partitioning (Ch 4).
Regression (6.1)  enhanced
2
Unit Testing
• Making sure that the units are correct.
• Insufficient unit testing can be costly:
debugging an error at the system-test level is
much more costly than at the unit level (some
reports suggest more than 10x).
• Don’t confuse the concept of “unit testing”
and “unit testing tool”; the later can often also
be used to facilitate integration and system
testing.
3
But what is a “unit” ?
• In principle, you decide. “Units” are just something
you can compose to build something bigger.
Possibilities: function, method, or class.
• But be aware that different types of units may have
types of interactions and complexity, thus requiring
different testing approach
– a function’s behavior depends only on its parameters; does
not do any side effect.
– procedure depends-on/affects params and global
variables
– method: static vars, params, instance vars
4
– class: is a collection of interacting methods
Unit testing in C#
• You need at least Visual Studio Professional; and for
code coverage feedback you need at least Premium.
• Check these tutorials/docs (Visual Studio 2010):
–
–
–
–
Walkthrough: Creating and Running Unit Tests
Walkthrough: Run Tests and View Code Coverage
Walkthrough: Using the Command-line Test Utility
API Reference for Testing Tools for Visual Studio, in
particular Microsoft.VisualStudio.TestTools.UnitTesting,
containg classes like
• Assert, CollectionAssert, ...
• In this lecture we will just go through the concepts
5
The structure of a “test project”
class Thermometer
private double val
private double scale
private double offset
public Thermometer(double s, double o)
public double value()
public double warmUp(double v)
public double coolDown(double v)
A test project is a just a project in your
solution that contains your test-classes.
6
The structure of a “test project”
• A solution may contain multiple projects; it may thus
contain multiple test projects.
• A test project is used to group related test classes.
• You decide what “related” means; e.g. you may want
to put all test-cases for the class Thermometer in its
own test project.
• A test class is used to group related test method.
• A test method does the actual testing work. It may
encode a single test-case, or multiple test-cases. You
decide.
7
Test Class and Test Method
[TestClass()]
public class ThermometerTest {
private TestContext testContextInstance;
//[ClassInitialize()]
//public static void MyClassInitialize(...) ...
//[ClassCleanup()]
//public static void MyClassCleanup() ...
//[TestInitialize()]
//public void MyTestInitialize() ...
//[TestCleanup()]
//public void MyTestCleanup() ...
[TestMethod()]
public void valueTest1() ...
[TestMethod()]
public void valueTest2() ....
public void valueTest1() {
target = new Thermometer(1,0);
double expected = - 273.15 ;
double actual = target.value();
Assert.AreEqual(expected, actual);
}
Be careful when comparing floating
numbers, you may have to take
imprecision into account, e.g. use this
instead:
AreEqual(expected,actual,delta,”...”)
}
8
Positive and Negative Test
• Positive test: test the program on its normal
parameters’ range.
• But can we afford to assume that the program
is always called in its normal range?
Else do Negative test: test that the program
beyond its normal range.
• E.g. when unit testing a method, to test if it
throws the right kind of exceptions.
9
Test Oracle
some patterns  will return later
public void valueTest1() {
target = new Thermometer(1,273.15);
double expected = - 273.15 ;
double actual = target.value();
Assert.AreEqual(expected, actual);
}
An oracle specifies your
expectation on the program’s
responses.
• Full: Assert.IsTrue( T == -273.15)
• Partial: Assert.IsTrue( T >= -273.15)
• Property-based : Assert.isTrue(safe(T))
More costly to maintain,
e.g. if you change the
intended behavior of the
program.
Usually partial, but allow re-use the predicate “safe” in
other test-cases.
10
Discussion: propose test cases
in particular the oracles...
reverse(a) {
N = a.length
if (N  1) return
for (int i=0; i< N/2 ; i++)
swap(a,i, N-1-i)
}
incomeTax(i) {
if (i18218) return 0.023 * i
t = 419
if (i32738)
return t + 0.138 * (i – 18218)
t += 1568
if (i54367)
return t + 0.42 * (i – 32738)
}
Property-based testing fits nicely for reverse, but not for incomeTax; for the latter we’ll have
to fall back to conrete-value oracles, which unfortunately tend to be more costly to
maintain.
11
Discussion: the Oracle Problem (6.5)
• Every test-case needs an oracle; how to construct it!?
 always a big problem!
• Using concrete values as oracles is often powerful,
but potentially expensive to maintain.
• Using “properties” on the other hand has the
problem that often it is hard to write a complete yet
simple property capturing correctness. E.g. how to
express an oracle for a sorting function?
• Alternatively we can fall back to redundancy-based
testing.
12
Inspecting Test Result
13
Inspecting Coverage
14
Finding the source of an error: use a
debugger!
•
•
•
Add break points; execution is stopped at
every BP.
You can proceed to the next BP, or execute
one step at a time: step-into, step-over,
step-out.
VisualStudio uses IntelliTrace  logging 
you can even inspect previous BPs.
15
The Debug class
• Debug.Print(“therm. created”)
• Debug.Assert(t.scale() > 0) to check for
properties that should hold in your program.
• Will be removed if you build with debug off
(for release).
• Check the doc of System.Diagnostics.Debug
16
(Unit) Testing a class
• Many classes have methods that interact with
each other (e.g. as in Stack). How to test these
interactions?
• How to specify and test these interactions?
Options:
– class invariant
– Abstract Data Type (ADT)
– Finite State Machine (FSM)
17
Specifying with class invariant
• Regardless the interaction with other
methods, each method of a class C has to
keep the state of its target object consistent.
• Express this with a class invariant, e.g.
– this.T >= -273.15
– this.saldo >= 0
– forall x in persons, typeOf(x) is Employee
• Class invariant cannot express a constraint
over the interaction itself.
18
Formulate and test the class invariant
Stack<T>
private Object[] content
private int top
public Stack()
push(T x)
T pop()
bool classinv() {
return
0top
&& top<content.length
}
}
Example test-cases, check
class-inv after :
1. call to the constructor
2. constructor ; push(x)
3. constructor ; push(x) ; push()
4. constructor ; push(x) ; pop()
5. some random sequence of
push and pop
19
Specifying a class as an ADT
• An Abstract Data Type (ADT) is a model of a
(stateful) data structure. The data structure is
modeled abstractly by only describing a set of
operations (without exposing the actual
state).
• The semantic is described in terms of “logical
properties” (also called the ADT’s axioms) over
those operations.
20
Example : stack
Stack<T>
bool isEmpty()
push(T x)
T pop()
Stack axioms :
•For all x,s :
s.push(x) ; y = s.pop ;
assert (y==x )
• For all x and s :
s.push(x) ;
assert ( s.isEmpty())
Depending of the offered
operations, it may be hard/not
possible to get a complete
axiomatization.
• For all x and s.isEmpty() :
s.push(x) ; s.pop
assert (s.isEmpty())
21
Test each ADT’s axiom
For all x,s :
s.push(x) ; y = s.pop ;
assert (y==x )
For example, three test cases :
1. empty s
2. non-empty s
3. s already contains x
22
Specifying a class with a finite state machine (FSM)
(2.5.1, 2.5.2)
File:
open()
write()
close()
FSM can also come from your
UML models.
• good when the sequence with which the methods of the class is
called matters
• For example, test that after every valid sequence the class invariant
hold.
• relevant concepts to apply: node coverage, edge coverage, edge23
pair coverage.
In a larger project....
• You want to test your class Heater ; it uses
Thermometer which is not ready yet!
• We can opt to use a mock Thermometer. A mock of a
program P:
– has the same interface as P
– only implement a very small subset of P’s behavior
– fully under your control
• Analogously we have the concept of mock object.
• Make mocks yourself e.g. exploiting inheritance, or use
a mocking tool.
24
Mocking with Moq
class Heater
double limit
bool active
public check() {
if (thermometer.value() >= limit)
active = false
}
thermometer
interface IThermometer
double value()
double warmUp(double v)
test1() {
Heater heater = new Heater()
var mock = new Mock<IThermometer>()
mock.Setup(t => t.value()).Returns(-275.15)
heater.thermometer = mock.object
heater.limit = 303.0
heater.check()
Assert.IsFalse(heater.active)
}
25
Mocking with Moq
(google it for more info!)
var mock = new Mock<IThermometer>()
mock.Setup(t => t.value()).Returns(303.00001)
mock.Setup(t => t.warmUp(0)).Returns(0)
mock.Setup(t => t.warmUp(It.IsInRange <double>(-10, 10, Range.Inclusive))
.Returns(0)
mock.Setup(t => t.warmUp (It.IsAny<double>()))
.Returns((double s) => s + 273.15)
Many more mock-functionalities in Moq; but in general mocking can be tedious.
E.g. what to do when your Heater wants to call warmUp in an iteration?
26
Beyond unit testing, what if we don’t
have access to the source code ?
• Or you deliberately want to abstract away
• Or, you do have access, but you don’t have a
tool to instrument the code
• (Def 1.26) White box testing : common at the
unit-testing level
• (Def 1.25) Black box testing: common at the
system-testing level. Approaches :
– Model-based testing
– Partition-based testing
27
Model-based testing
TextEditor:
new(fname)
type(c)
save()
• We already seen that  use an FSM as a model of
the system you test
• Such an FSM specifies :
– sequences which should be valid and invalid
– these tell you which sequences to test
– furthermore, coverage can be defined over the FSM
28
Model-based testing
size() = 0
TextEditor:
new(...)
save()
type(c)
type(c)
size() > 0
• Predicates can be defined on the FSM’s states to specify
properties that should hold; but keep in mind that in black
box testing you cannot just inspect objects state at will.
• Useful concepts : mutators and inspectors.
• Mutators : methods with side effect  arrows in the FSM.
• Inspectors : methods with no side effect  used to express
your state predicates,
29
Partitioning the inputs
save(String fname, Object o)
fname : (A) existing file
(B) non-existing file
o : (P) null
(Q) non-null serializable
(R) non-null non-serializable
• Based on “your best understanding” of save’s semantic.
• Terminology: characteristic, block. The domain of a
characteristic is divided into disjoint blocks; the union of
these blocks must cover the entire domain of the
characteristic.
• Assumption : values of the same block are “equivalent”
30
So, what input values to choose?
save(String fname, Object o)
fname : (A) existing file
(B) non-existing file
o : (P) null
(Q) non-null serializable
(R) non-null non-serializable
• (C4.23, ALL) All combinations must be tested.
|T| = (i: 0i<k: Bi) ; does not scale up.
• (C4.24, EACH CHOICE) Each block must be tested.
|T| = (max i: 0i<k: Bi) ; usually too weak.
31
t-wise coverage
A
B
P
Q
X
Y
T : (A,P,X) ,
(A,Q,Y) , ... more?
• (C4.25, pair-wise coverage). Each pair of blocks (from
different characteristics) must be tested.
• (C4.26, t-wise coverage). Generalization of pair-wise.
• Obviously stronger than EACH CHOICE, and still
scalable.
• Problem: we just blindly combine; no semantical
awareness.
32
Adding a bit of semantic
A
B
P
Q
X
Y
Z
Example: t0 = (A,P,X), generates these
additional test requirements :
(B,P,X)
(A,Q,X)
(A,P,Y) (A,P,Z)
• (C4.27, Base Choice Coverage, BCC) Decide a single
base test t0. Make more tests by each time removing
one block from t0, and forming combinations with all
remaining blocks (of the same characteristics).
|T| = 1 + (i : 0i<k : Bi - 1)
33
Or, more bits of semantics
A
B
P
Q
X
Y
Z
Example, base tests = (A,P,X), (A,P,Y)
(A,P,X) generates
(B,P,X)
(A,Q,X)
(A,P,Z)
(A,P,Y) generates
(B,P,Y)
(A,Q,Y)
(A,P,Z)
• (C4.28, Multiple Base Choices). For each
characteristic we decide at least one base block.
Then decide a set of base tests; each only include
base blocks. For each base test, generate more tests
by each time removing one base block, and forming
combinations with remaining non-base blocks.
34
|T| at most M + (i : 0i<k : M*(Bi - mi))
Example-2, MBCC
A
B
C
P
Q
R
(B,P,X)
(C,P,X)
(A,R,X)
(A,R,Y)
(B,Q,Y)
(C,Q,Y)
X
Y
Z
Bold : base blocks
Chosen base tests = (A,P,X), (A,Q,Y)
These produce these additional test
requirements:
(A,P,Z)
(A,Q,Z)
• base-blocks are not cross-combined
except as in the base tests.
• non-base blocks are not crosscombined with each other.
• (C4.28, Multiple Base Choices). For each characteristic we decide at least
one base block. Then decide a set of base tests; each only include base
blocks. For each base test, generate more tests by each time removing one
base block, and forming combinations with remaining non-base blocks. 35
Constraints, to exclude non-sensical
cases
• Example:
– combo (A,P,Y) is not allowed.
– if P is selected, then X must also be selected.
• Solvable: pair-wise coverage + (A,P,Y) is not allowed.
• Can be unsolvable, e.g. pair-wise coverage + (A,P) is
not allowed.
• General problem: given a coverage criterion C and a
set of constraints, find a test set T satisfying both.
• In general the problem is not trivial to solve.
36
Overview of partition-based coverage
ALL
t-Wise
Multiple Base
Choice Coverage
Pair-Wise
Base Choice
Coverage
EACH
CHOICE
37
Regression Test
• To test that a new modification in your program does
not break old functionalities. To be efficient, people
typically reuse existing test sets.
• Usually applied for system-testing, where the
problem is considered as more urgent. Challenge:
very time consuming (hours/days!).
• There are also research to apply it continuously at
the unit level, as you edit your program; see it as
extended type-checking. Result was positive!
(Saff & Enrst, An experimental evaluation of continuous testing during
development. ISSTA 2004)
38
Some concepts first...
• Test Selection Problem: suppose P has been modified
to P’. Let T be the test set used on P. Choose a subset
T’T to test P’.
• Obviously, exclude obsolete test cases: those that
can’t execute or whose oracles no longer reflect P’
semantic. Let’s assume: we can identify them.
• You want the selection to be safe : T’ includes all
test-cases in T that will execute differently on P’.
• Only attractive if the cost of calculating T’ + executing
T’ is less than simply re-executing the whole T.
39
Idea: select those that pass through
modified code
(orginal)
m(x) { if (d) y = x+1 else y=0 }
(modified)
m(x) { if (d) y = x-1 else y=0 }
• If m is the only method in P that changes, the
obvious strategy is to select only test-cases
that pass through m.
• Better: only select test-cases that pass m’s
“modified” branch.
40
Corner case
m(x) { if (d) {
y = x+1 ;
if (e) stmt ;
u=0 }
else if (e) stmt
}
m(x) { if (d) y = x+1 ;
if (e) stmt
}
• The first if is modified by removing an else-branch.
Using the previous strategy means that we have to
select all test-cases that pass m. Yet we see that the
paths [d, e, stmt] and [ d, e] present in both old
and new m; so there is actually no need to select
them.
41
Looking at it abstractly with CFG
m(x) { if (d) {
y = x+1 ;
if (e) stmt ;
u=0 }
else if (e) stmt
}
m(x) { if (d) y = x+1 ;
if (e) stmt
}
d
y=x+1
e
stmt
d
end
y=x+1
e
stmt
e
u=0
end
stmt
Notice that [d, e, stmt, end] and
[d, e, end] appear in both, and
equivalent.
42
Some concepts
• We assume: P is deterministic.  each test-case
always generate the same test path.
• Let p and p’ be the test-paths of a test-case t when
executed on P and P’; t is modification traversing if
not(p  p’).  let’s select modification traversing
test-cases.
• p  p’ if they have the same length, and for each i,
pi  p’i  the latter means they contain the same
sequence of instructions.
• So far this is not helpful, because such a selection
strategy requires us to first execute t on P’. Then it is
43
not attractive anymore!
“Intersection” Graph
G:
a
b
G’ :
G’’ = G  G’:
a
b,b
b
c
c
end
d
a,a
c
c,c
end,d
c,c
end,end
end
• First, extend CFG so that branches are labelled by the corresponding
decision value (e.g. T/F for if-branch). Label non-branch edge with some
constant value.
• Each node of G’’ is a pair (u,u’). Then G’’ is defined like this :
– The pair of initial nodes (s0,s0’)  G’’.
– If (u,u’)G’’, and uu’, and uv is an edge in G, and u’v’ and edge in
G’ both with the same label, then (u,u’)  (v,v’) should be an edge in
44
G’’.
“Intersection” Graph
G:
a
b
G’ :
G’’ = G  G’:
a
b,b
b
c
c
end
d
a,a
c
c,c
end,d
c,c
end,end
end
• Each path p in G’’ describes how a path in G would be executed on G’ if
the same decisions are taken along the way. Note that this is calculated
without re-executing any test-case on P’.
• Any path in G’’ ends either in a proper exit node (green), or in a pair (u,u’)
where not uv (red). This would be the first time a test-case would hit a
modified code when re-executed on P’.
• The old test-cases are assumed to have been instrumented, so that we
45
know which nodes/edges in G it traversed.
Selection Algorithm
G:
a
b
G’ :
G’’ = G  G’:
a
b,b
b
c
c
end
d
a,a
c
c,c
end,d
c,c
end,end
end
• Select test-cases that pass [a,b,c,end] in G  expensive to check!
• (Safe but not minimalistic) Select test-cases that pass a node u in G that is
part of a red-node in G’’.  same problem as before, it will select also
select [a,c,end] which is not modification traversal.
• (Rothermel-Harold, 1997) Select test-cases that pass an edge e in G that in
G’’ leads to a red-node in G’’.  actually the same problem.
46
Selection Algorithm
G:
a
b
G’ :
G’’ = G  G’:
a
b
c
c
end
d
b,b
c
end
c,c
end,d
a,a
c,c
end,end
G-partition
NG-partition
• (Ball algorithm,1998) Partition G’’ nodes to those can can reach a greennode (G partition), and those that cannot (NG partition). Look at edges in
G’’ that cross these partitions (so, from G to NG).
A test path p is modification traversing if and only if it passes through a
crossing edge (as meant above).  use this as the selection criterion.
47