ICTSS’11 Tutorial
Parameterized Unit Testing
Nikolai Tillmann
Microsoft Research
Outline


First part: 14:00 - 15:00pm
Second part: 15:30 - 17:00pm
Outline
 Unit Testing
 Isolation
 Parameterized Unit Testing
 Data Generation by
Dynamic Symbolic Execution
 Patterns
 Limitations and other Details
About the exercises…
 Interactive: http://pexforfun.com
 Demos: Pex for Visual Studio
http://research.microsoft.com/pex
 Requires Windows, .NET 2.0 or 4.0,
ideally Visual Studio 2008 / 2010
Unit Testing
Quiz: Unit testing

What is a unit test?
Unit Testing


A unit test is a small program with assertions
Test a single (small) unit of code
void AddAndCount() {
var list = new List();
list.Add(3);
Assert.AreEqual(1, list.Count);
}

Let’s play a game!
The Code Under Test
string ReadFooValue() {
string[] lines = File.ReadAllLines(@"t:\myapp.ini");
foreach (var line in lines) {
int index = line.IndexOf('=');
string name = line.Substring(index);
if (name == "Foo") {
string value = line.Substring(index + 1);
return value;
}
t:\myapp.ini
}
A=B
return null;
}
Foo=C
C=D
Quiz: Coverage

How much block coverage do we need?
1. 50%
2. 80%
3. 100%
4. Block coverage alone is not enough
Quiz: Coverage

How much block coverage do we need?
1. 50%
2. 80%
3. 100%
4. Block coverage alone is not enough
Quiz: Coverage
[TestMethod]
void ExistingFoo() {
File.WriteAllLines(@"t:\myapp.ini",new string[]{“Foo=b”});
Reader.ReadFooValue();
}

Do we need more tests to get 100% cov.?
[TestMethod]
void MissingFoo() {
File.WriteAllLines(@"t:\myapp.ini",new string[0]);
Reader.ReadFooValue();
}
Quiz: Assertions

Why write Assertions (or not)?
1. Documentation
2. Double check your program
3. Please your manager
4. Prevent future bugs
5. Validate user inputs
6. Catch errors early
Quiz: Assertions

Why write Assertions (or not)?
1. Documentation
2. Double check your program
3. Please your manager
4. Prevent future bugs
5. Validate user inputs
6. Catch errors early
Quiz: Assertions
[TestMethod]
void MissingFoo() {
File.WriteAllLines(@"t:\myapp.ini",new string[]{“a=b”});
string value = Reader.ReadFooValue();
Assert.IsTrue(????);
}

Which Assertions should you write?
1. Assert.IsTrue(value == “b”);
2. Assert.IsTrue(value == null);
3. Assert.IsTrue(String.IsNullOrEmpty(value))
4. Assert.IsTrue(false);
5. No assertions
Quiz: Assertions
[TestMethod]
void MissingFoo() {
File.WriteAllLines(@"t:\myapp.ini",new string[]{“a=b”});
string value = Reader.ReadFooValue();
Assert.IsTrue(????);
}

Which Assertions should you write?
1. Assert.IsTrue(value == “b”);
2. Assert.IsTrue(value == null);
3. Assert.IsTrue(String.IsNullOrEmpty(value))
4. Assert.IsTrue(false);
5. No assertions
Quiz: Coverage + Assertions

What gives you confidence in the code?
1. High coverage, no assertions
2. Low coverage, many assertions
3. High coverage, many assertions
4. Low coverage, no assertions
5. I wrote it
Quiz: Coverage + Assertions

What gives you confidence in the code?
1. High coverage, few assertions
2. Low coverage, many assertions
3. High coverage, many assertions
4. Low coverage, no assertions
5. I wrote it
The Code Under Test
string ReadFooValue() {
string[] lines = File.ReadAllLines(@"t:\myapp.ini");
foreach (var line in lines) {
int index = line.IndexOf('=');
string name = line.Substring(index);
if (name == "Foo") {
string value = line.Substring(index + 1);
return value;
}
}
return null;
}
Quiz: Isolation
string ReadFooValue() {
string[] lines = File.ReadAllLines(@"t:\myapp.ini");
...
}

In the example,
what are the external dependencies?
1. Network Share
2. Local Disk
3. No file system, all in memory
Quiz: Isolation
string ReadFooValue() {
string[] lines = File.ReadAllLines(@"t:\myapp.ini");
...
}

In the example,
what are the external dependencies?
1. Network Share
2. Local Disk
3. No file system, all in memory
Quiz: Isolation

What is the problem with a Local Disk?
1. Mapping already exists
2. Cannot run tests concurrently
3. Disk full
4. Access rights
Quiz: Isolation

What is the problem with a Local Disk?
1. Mapping already exists
2. Cannot run tests concurrently
3. Disk full
4. Access rights
Quiz: Isolation

How do you mitigate the Local Disk issues?
1. Always run on the same machine, same
2.
3.
4.
5.
hardware, same credentials, same time, same
temperature, same solar system configuration
Refactoring: use Streams
Refactoring: introduce IFileSystem
Refactoring: pass the lines as parameter
Change implementation of File.ReadAllLines
Quiz: Isolation

How do you mitigate the Local Disk issues?
1. Always run on the same machine, same
2.
3.
4.
5.
hardware, same credentials, same time, same
temperature, same solar system configuration
Refactoring: use Streams
Refactoring: introduce IFileSystem
Refactoring: pass the lines as parameter
Change implementation of File.ReadAllLines
Quiz: Definition of Unit Test

What is a Unit Test?
 A Unit Test is a program that
runs fast the code under test,
without environment dependencies,
with assertions.

What is a Unit Test Suite?
 A set of Unit Tests which achieves high code
coverage
Isolation
Dependency hell
var lines = File.ReadAllLines(@"t:\v.ini");

How do you change File.ReadAllLines?
1. Override static method
2. Changing the .NET Framework source code
(and recompiling it)
3. Rewrite application in JavaScript
4. Code instrumentation
Replace any .NET method
with a delegate
var lines = File.ReadAllLines(@"t:\myapp.ini");
What if we could replace File.ReadAllLines?
File.ReadAllLines
= delegate(string fn)
MFile.ReadAllLinesString = delegate(string fn)
{
return new string[0];
Moles
};
Motivation for Moles
 Why another isolation framework?
 Specifically designed to enable Pex
 Simple, well-defined semantics
▪ “Replace any .NET method”
Mole Types Code Generation
// System.IO
public static class File {
public static string[] ReadAllLines(string fn);
}
// System.IO.Moles
public class MFile {
public static
Func<string, string[]> ReadAllLinesString;
}
// delegate R Func<T, R>(T t);
Injecting Detours at Runtime
// System.IO
public static class File {
public static string[] ReadAllLines(string fn) {
if (MFile.ReadAllLinesString != null)
return MFile.ReadAllLines(fn);
… original code
}
}
Automatically injected
at runtime
Moles
Demo







Start from ReadFooValue problem
Create test project
Add moles for mscorlib
Write test using moles
Run test
Debug test
You can pick up project from
ReadFooValueIsolation\ReadFooValueIsolation.sln
Quiz: Moles Usage

When should you use Moles (and not)?
1.
2.
3.
4.
5.
6.
7.
8.
9.
Always use Moles to solve isolation issues
With Moles, one does not need to use interfaces anymore
Moles only should be used for 3rd party API, use interfaces for
isolation in your APIs
Moles can be used in production code
Moles lets you get away with untestable APIs
Moles make test cases more robust
With Moles, you do not need integration tests anymore
Moles make tests easier to understand
Moles is for poor programmers, real programmers rely on
interfaces
Quiz: Moles Usage

When should you use Moles (and not)?
1.
2.
3.
4.
5.
6.
7.
8.
9.
Always use Moles to solve isolation issues
With Moles, one does not need to use interfaces anymore
Moles only should be used for 3rd party API, use interfaces
for isolation in your APIs
Moles can be used in production code
Moles lets you get away with untestable APIs
Moles make test cases more robust
With Moles, you do not need integration tests anymore
Moles make tests easier to understand
Moles is for poor programmers, real programmers rely on
interfaces
Exercise (Optional / homework)




Step-by-step tutorial:
research.microsoft.com/documentation.aspx
“Unit Testing with Microsoft Moles”
“Unit Testing SharePoint Foundation with
Microsoft Pex and Moles”
What you learned so far

The Definition of Unit Testing

Unit Test Isolation
1. Identify External Dependencies
2. Replace each External API with your delegate
Parameterized
Unit Testing
The Recipe of Unit Testing

Three essential ingredients:
 Data
 Method Sequence
 Assertions
void AddAndCount() {
int item = 3;
var list = new List();
list.Add(item);
var count = list.Count;
Assert.AreEqual(1, count);
}
Quiz: list.Add(???)
list.Add(???);

Which value matters?
1.
2.
3.
4.
5.
6.
7.
0
1
int.MaxValue, int.MinValue
-1
1000
it does not matter
I don’t know until I read the code
Parameterized Unit Testing
void AddAndCount(List list, int item) {
var count = list.Count;
list.Add(item);
Assert.AreEqual(count + 1,
forlist.Count);
any list,
}
for any item,
Parameterized Unit Test =
… adding
1 item
Unit
Test
with Parameters
increments Count by 1
 Separation of concerns

 Specification of behavior
 Data to achieve coverage
Parameterized Unit Tests are
Algebraic Specifications

A Parameterized Unit Test can be read as a
universally quantified, conditional axiom.
void ReadWrite(string name, string data) {
Assume.IsTrue(name != null && data != null);
Write(name, data);
var readData = Read(name);
Assert.AreEqual(data, readData);
}
forall. string name, string data:
name not_equal null and data not_equal null implies
equals(
ReadResource(name,WriteResource(name,data)),
data)
Parameterized Unit Testing
is going mainstream
Parameterized Unit Tests (PUTs) commonly supported by
various test frameworks
 .NET: Supported by .NET test frameworks
 http://www.mbunit.com/
 http://www.nunit.org/
 …

Java: Supported by JUnit 4.X
 http://www.junit.org/
Generating test inputs for PUTs supported by tools
 .NET: Supported by Microsoft Research Pex
 http://research.microsoft.com/Pex/
 Java: Supported by Agitar AgitarOne
 http://www.agitar.com/
Problem
We cannot execute parameterized unit tests
without data.
Where does the data come from?




Random data generator
Real customer data
Ranges
Some values hand-picked by dev/tester
Data Generation
by Dynamic Symbolic Execution
Data Generation Challenge
Goal: Given a program with a set of input
parameters, automatically generate a set of input
values that, upon execution, will exercise as many
statements as possible
How would you do it?
45
Dynamic Symbolic Execution
Combines many ideas
Combines ideas from
 symbolic execution
 to capture behavior symbolically

model checking
 to perform systematic bounded state space exploration

testing
 to monitor and check concrete behavior

constraint solving
 to find inputs
Dynamic Symbolic Execution
The
high-level
algorithm
• This choice decides search order
• Search order decides how quick we
can achieve
high
code
coverage!
Combines
concrete
and
symbolic
execution
• Incomplete constraint-solver leads
to under-approximation
Algorithm:
Set J := ∅
(J is set of already analyzed program inputs)
Loop
Choose program input i ∉ J
(stop if no such i can be found)
Output i
Loop
notCterminate
if number
Execute P(i); record
pathdoes
condition
(in particular,
C(i) holds)
Set J := J ∪ C
the set { i | C(i ) } )
of execution(viewing
pathsCisasinfinite
End loop
(in the presence of loops/recursion)
47
Dynamic Symbolic Execution
Example
Choose next path
Code to generate inputs for:
void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
F
F
a.Length>0
a==null
T
Solve
Constraints to solve
Execute&Monitor
Data
Observed constraints
null
a==null
a!=null &&
!(a.Length>0)
a!=null &&
a.Length>0 &&
a[0]!=1234567890
a!=null
{}
a!=null &&
a.Length>0
{0}
a!=null &&
a.Length>0 &&
a[0]==1234567890
{123…}
a!=null &&
a.Length>0 &&
a[0]==1234567890
T
Done: There is no path left.
a[0]==123…
F
T
http://pex4fun.com/CoverMe
Dynamic Symbolic Execution
Exercises



ArrayIndexLength
Pex knows about all implicit, exceptionthrowing control-flow branches
ArrayHeap
Pex models the heap
Assert, Assert123
Assertions connect code coverage and
correctness
http://pex4fun.com/DynamicSymbolicExecutionExercises
Assertions vs. Coverage

What about finding faults?
void AssertIsTrue(bool condition) {
if (!condition)
throw new AssertionException();
}

Assertions induce branches
 Pex tries to cover branches
▪ Pex tries to fail assertions -> finds bug!
Dynamic Symbolic Execution
with Pex in Visual Studio






Create new project in Visual Studio.
Insert CoverMe method below.
Right-click on CoverMe method,
and select “Run Pex”.
Inspect results in table.
Save tests, …, have fun!
public static void CoverMe(int[] a) {
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
Dynamic Symbolic Execution
with Pex in Visual Studio
Dynamic Symbolic Execution
with Pex in Visual Studio
Generated
Test Inputs
are persisted
as C# Unit
Tests
Exercise (Optional / homework)



Step-by-step tutorial:
research.microsoft.com/documentation.aspx
“Exploring Code with Microsoft Pex”
Motivation: Unit Testing Hell
ResourceReader

How to test this code?
(Actual code from .NET base class libraries)
Motivation: Unit Testing Hell
ResourceReader
Demo
ResourceReader
[PexClass, TestClass]
[PexAllowedException(typeof(ArgumentNullException))]
[PexAllowedException(typeof(ArgumentException))]
[PexAllowedException(typeof(FormatException))]
[PexAllowedException(typeof(BadImageFormatException))]
[PexAllowedException(typeof(IOException))]
[PexAllowedException(typeof(NotSupportedException))]
public partial class ResourceReaderTest {
[PexMethod]
public unsafe void ReadEntries(byte[] data) {
PexAssume.IsTrue(data != null);
fixed (byte* p = data)
using (var stream = new UnmanagedMemoryStream(p, data.Length)) {
var reader = new ResourceReader(stream);
foreach (var entry in reader) { /* reading entries */ }
}
}
}
Dynamic Symbolic Execution
Many implementations

Defined by execution environment / programming language,
symbolic execution precision, and constraint solving
 Execution environment: C, Java, x86, .NET,…
 Precision: linear vs. non-linear arithmetic, “gods integers” vs.
bitvectors, concrete heap vs. symbolic heap., floating-point values, etc.
 Solvers: lp_solve, CVCLite, STP, Disolver, Z3,…

Examples of DSE implementations:








DART (Bell Labs), and also CUTE “concolic execution”
EXE/EGT/KLEE (Stanford) “constraint-based execution”
Vigilante (Microsoft) to generate worm filters
BitScope (CMU/Berkeley) for malware analysis
Sage (Microsoft) for security testing of X86 code
Yogi (Microsoft) to verify device drivers (integrated in SLAM)
Pex (Microsoft) for parameterized unit testing of .NET code
CREST, jCUTE, jFuzz, …
Constraint Solving
vs. Search Strategies
Initially, choose Arbitrary
Path-constraint
solving is just
hard.
Test
Inputs
Constraint
System
Reachability is
undecidable!
(Loops)
Execution Path
Known
Paths
Symbolic State Representation
Representation of symbolic values and state is similar to the ones used to
build verification conditions in ESC/Java, Spec#, …
Terms for
 Primitive types (integers, floats, …), constants, expressions
 Struct types by tuples
 Instance fields of classes by mutable ”mapping of references to values"
 Elements of arrays, memory accessed through unsafe pointers
by mutable “mapping of integers to values"
Efficiency by
 Many reduction rules, including reduction of ground terms to constants
 Sharing of syntactically equal sub-terms
 BDDs over if-then-else terms to represent logical operations
 Patricia Trees to represent AC1 operators
(including parallel array updates)
Constraint Solving

SMT-Solver (“Satisfiability Modulo Theories”)
 Decides logical first order formulas with respect to theories
 SAT solver for Boolean structure
 Decision procedures for relevant theories:
uninterpreted functions with equalities,
linear integer arithmetic, bitvector arithmetic, arrays, tuples

Model generation for satisfiable formulas
 Models used as test inputs

Limitations
 No decision procedure for floating point arithmetic

Pex uses Z3: http://research.microsoft.com/z3
Satisfiability Modulo Theories
void Foo(int x, int y, int[] a) {
if (x + 2 == y) {
a[x] = 3;
Debug.Assert(f(a[y - 2]) == f(y – x + 1));
}
}
x  2  y  f (read ( write(a, x,3), y  2)  f ( y  x  1)
Uninterpreted
Array
Arithmetic
Theory
Functions
Domain-specific Constraint Solving
Pex uses some heuristic approaches for domain-specific
problems.

Strings [TACAS’09, ICST’10]
 “uninterpreted functions” for string operations
 two-phase solving: first solve integer constraints involving
indices and lengths, then character constraints

Floating-point arithmetic
 Search-based approach: minimization of fitness function

… (more to come)
Writing
Parameterized Unit Tests
Exercise
Write PUT for ReadFooValue
Can you come up with Parameterized Unit Tests?
string ReadFooValue() {
string[] lines = File.ReadAllLines(@"t:\myapp.ini");
foreach (var line in lines) {
int index = line.IndexOf('=');
string name = line.Substring(index);
if (name == "Foo") {
string value = line.Substring(index + 1);
return value;
}
}
return null;
}
Exercise
Write PUT for ReadFooValue





Create new solution with C# class library
Write code for ReadFooValue
Create Test Project
Add moles for mscorlib
Write parameterized unit tests.
[PexMethod] // attribute marks pex tests
public void Test(...) {
...
}

Or get result from ReadFooValuePUT\ReadFooValuePUT.sln
Exercise
Crash Test

Execute the code without assertions
[PexMethod]
void Crash(string[] lines) {
MFile.ReadAllLines = () => lines;
Reader.ReadFooValue();
}
Exercise
Assert Observed Results


Manually review outputs
Pex inserts Assertions automatically
[PexMethod]
public string Regression(string[] lines) {
MFile.ReadAllLines = () => lines;
return Reader.ReadFooValue();
}
Exercise

Write and read property
[PexMethod]
void FooExist(string value1) {
PexAssume.IsTrue(value1 != null);
var lines = new string[]{“Foo=“ + value1};
MFile.ReadAllLines = () => lines;
var value2 = Reader.ReadFooValue();
PexAssert.AreEqual(value1, value2);
}
Test Generation Work-Flow
Code-Under-Test
Project
Test Project
Parameterized
Unit Tests
Pex
Generated Tests
Patterns
Patterns
how to create objects
Quiz:
Complex Objects As Inputs
[PexMethod]
public void ArrayListTest(ArrayList al, object o) {
PexAssume.IsTrue(al != null);
int len = al.Count;
al.Add(o);
PexAssert.IsTrue(al[len] == o);
}
What are the test inputs?
 Only null
 An array list created by calling a constructor
 ... and also calling Add(…) many times
 … and also calling Clear() many times
 …
Quiz:
Complex Objects As Inputs
[PexMethod]
public void ArrayListTest(ArrayList al, object o) {
PexAssume.IsTrue(al != null);
int len = al.Count;
al.Add(o);
PexAssert.IsTrue(al[len] == o);
}
Other possible problems: What if…
 ArrayList has no public constructor
 ArrayList is an abstract class (or interface)
Creating complex objects
The Problem
Problem:
 Test parameters may be classes
 Symbolic execution tracks constraints over (private)
fields of objects
 Constraint solver determines solution, assigning
concrete values to (private) fields
 However, in practice objects must be initialized by
constructor, and can be mutated only by calling
certain methods
 What sequence of method calls reaches a desired
field assignment?
 There may be no such sequence
 In general, undecidable
Object creation:
Factory methods

Factory methods
 Parameterized factories create and configure objects
 Explicit: Public static method annotated with
[PexFactoryMethod]
 Implicit: Pex guesses “best” constructor/setters

Result: Exploration of reachable states
 Reachable using factory method
 Reachable within configured bounds
 Under-approximation of possible states
Factory methods
ArrayList
Hand-written factory method:
[PexFactoryMethod(ArrayList)]
public static ArrayList Create(
int capacity,
object[] values)
{
var a = new ArrayList(capacity);
foreach (value in values)
a.Add(value);
return a;
}
Pex will explore both factory method and PUT!
Demo:
Factory methods
Open project “ArrayListSample”.
 Explore AddTestExplicit
 Explore AddTestImplicit
 Inspect generated test cases
 Notice constructor implicitly used by Pex
 Save factory method
 Edit factory method
 Explore again
Alternative
Object creation via class invariants

Write class invariant as boolean-valued
parameterless method
 Refers to private fields
 Must be placed in implementation code

Write special constructor/factory method for
testing only
 Sets fields, and assumes invariant

Result: Exploration of feasible states
 May include states that are not reachable
Patterns
for Parameterized Unit Tests
Pattern
4A

Assume, Arrange, Act, Assert
[PexMethod]
void Add(List target, T value) {
PexAssume.IsNotNull(target); // assume
var count = target.Count; // arrange
target.Add(value); // act
1.
2.
3.
}
// assert quiz
Assert(target != null);
Assert(target.Count == count + 1);
Assert(target[0] == value);
Pattern
4A

Assume, Arrange, Act, Assert
[PexMethod]
void Add(List target, T value) {
PexAssume.IsNotNull(target); // assume
var count = target.Count; // arrange
target.Add(value); // act
1.
2.
3.
}
// assert quiz
Assert(target != null);
Assert(target.Count == count + 1);
Assert(target[0] == value);
Pattern
Regression Tests

Generated test asserts any observed value
 Return value, out parameters
int AddTest(int a, int b) {
return a + b; }

When code evolves, breaking changes in observable
will be discovered
void AddTest01() {
var result = AddTest(0, 0);
Assert.AreEqual(0, result);
}
Pattern
Roundtrip

For an API f(x),
f-1(f(x)) = x
for all x
void ToStringParseRoundtrip(int value) {
string s = value.ToString();
int parsed = int.Parse(s);
1.
2.
3.
}
// assert quiz
Assert.AreEqual(value, s);
Assert.AreEqual(s, parsed);
Assert.AreEqual(parsed, value);
Pattern
Roundtrip

For an API f(x),
f-1(f(x)) = x
for all x
void ToStringParseRoundtrip(int value) {
string s = value.ToString();
int parsed = int.Parse(s);
1.
2.
3.
}
// assert quiz
Assert.AreEqual(value, s);
Assert.AreEqual(s, parsed);
Assert.AreEqual(parsed, value);
Pattern
Normalized Roundtrip

For an API f(x),
f-1(f(f-1(x)) = f-1(x)
for all x
void ParseToString(string x) {
var normalized = int.Parse(x);
var intermediate = normalized.ToString();
var roundtripped = int.Parse(intermediate);
1.
2.
3.
4.
}
// assert quiz
Assert(x == intermediate);
Assert(intermediate == roundtripped);
Assert(normalized == roundtripped);
Assert(x == roundtripped);
Pattern
Normalized Roundtrip

For an API f(x),
f-1(f(f-1(x)) = f-1(x)
for all x
void ParseToString(string x) {
var normalized = int.Parse(x);
var intermediate = normalized.ToString();
var roundtripped = int.Parse(intermediate);
1.
2.
3.
4.
}
// assert quiz
Assert(x == intermediate);
Assert(intermediate == roundtripped);
Assert(normalized == roundtripped);
Assert(x == roundtripped);
Pattern
Reachability

Indicate which portions of a PUT should be
reachable.
[PexAssertReachEventually]
public void Constructor(object input)
{
new Foo(input);
PexAssert.ReachEventually();
}
Pattern
Same Observable Behavior

Given two methods f(x) and g(x), and a method b(y)
that observes the result or the exception behavior of a
method, assert that f(x) and g(x) have same observable
behavior under b, i.e. b(f(x)) = b(g(x)) for all x.
public void ConcatsBehaveTheSame(
string left, string right)
{
PexAssert.AreBehaviorsEqual(
() => StringFormatter.ConcatV1(left, right),
() => StringFormatter.ConcatV2(left, right));
}
Pattern
Commutative Diagram

Given two implementations f and g of the same
function, each possible requiring a different number of
steps, i.e. f(x)=f1(f2(…(fn(x)…)), and g(x)=g1(g2(…
(gm(x)…)), then it should hold that
f1(f2(…(fn(x))…) = g1(g2(…(gm(x)…)) for all x.
string Multiply(string x, string y);
int Multiply(int x, int y);
void CommutativeDiagram1(int x, int y) {
string z1 = Multiply(x, y).ToString();
string z2 = Multiply(x.ToString(), y.ToString());
PexAssert.AreEqual(z1, z2);
}
Homework
Patterns



Read patterns paper:
research.microsoft.com/Pex/documentation.aspx
“Parameterized Test Patterns for Microsoft Pex”
Limitations
It’s called Parameterized Unit Testing
Testability

Pex understand managed .NET code only
 Pex does not understand native code.

Problem if branching over values obtained from the
environment
 Pex may not automatically detect all such cases.
File System?
if (!File.Exists(f)) throw ...
Hidden Complexity


Pex analyzes every executed .NET instruction
Some used libraries may be surprisingly expensive
to analyze
 XML parsing
 repeatedly converting data between different
representations
void Sum(string[] A) {
Don’t do this.
var sum = “0”;
foreach(var a in A)
sum = (int.Parse(a) + int.Parse(sum)).ToString();
if(sum == “123”) throw new Exception();
}
Exploration Boundaries

Configurable bounds include:
 TimeOut
 MaxBranches
 MaxCalls
 MaxConditions
▪ Number of conditions that depend on test inputs
 MaxRuns
 ConstraintSolverTimeOut
 ConstraintSolverMemoryLimit
Multi-threaded code



Unlike test inputs, thread-interleavings can
normally not be controlled
Thus, Pex can only explore single-threaded
code
Related approach to explore threadschedules (but not input parameters) by
controlling thread-scheduler: CHESS
http://research.microsoft.com/CHESS
Lack of Test Oracle

Unit Testing != Test Generation
Wrapping up
Conclusion
What you learned today

The Definition of Unit Testing
 Unit Tests are not Integration Tests

Unit Test Isolation
 Introduction of Abstraction Layers

Parameterized Unit Testing
 Separation of concerns: Coverage/Specification
 Patterns to write Parameterized Unit Tests
 Data Generation by Dynamic Symbolic Execution
Availability

Pex and Moles are
Visual Studio 2010 Power Tools
(academic/commercial license available)


Minimum: Windows XP, .NET 2.0
Ideally: Visual Studio 2010 Professional
http://research.microsoft.com/pex/

http://pexforfun.com

Thank you
http://research.microsoft.com/pex
http://sites.google.com/site/asergrp/