Tutorial Nikolai Tillmann, Peli de Halleux, Wolfram Schulte (Microsoft Research) Tao Xie (North Carolina State University) Unit Testing Parameterized Unit Testing Input Generation with Pex Patterns for Parameterized Unit Testing Parameterized Models Wrapping up We will use the tool Pex as part of the exercises To install it, you need Windows XP or Vista .NET 2.0 Download Pex from http://research.microsoft.com/en-us/projects/pex/downloads.aspx Two variations : pex.devlabs.msi: requires Visual Studio 2008 Team System pex.academic.msi: does not require Visual Studio, but for non-commercial use only To run the samples, you need VS Team System Introduction A unit test is a small program with assertions Test a single (small) unit of code void ReadWrite() { var list = new List(); list.Add(3); Assert.AreEqual(1, list.Count); } Design and specification By Example Documentation Short feedback loop Code coverage and regression testing Extensive tool support for test execution and management Happy path only Hidden integration tests Touch the database, web services, requires multiple machines, etc… New Code with Old Tests Redundant tests Coverage: Are all parts of the program exercised? statements basic blocks explicit/implicit branches … Assertions: Does the program do the right thing? test oracle Experience: Just high coverage of large number of assertions is no good quality indicator. Only both together are! Goal: Implement and test TrimEnd: // trims occurences of the ‘suffix’ from ‘target’ string TrimEnd(string target, string suffix); Open Visual Studio Create CodeUnderTestProject, and implement TrimEnd File -> New -> Project -> Visual C# -> Class Library Add new StringExtensions class Implement TrimEnd Create TestProject, and test TrimEnd File -> New -> Project -> C# -> Test -> Test Project Delete ManualTest.mht, UnitTest1.cs, … Add new StringExtensionsTest class Implement TrimEndTest Execute test Test -> Windows -> Test View, execute test, setup and inspect Code Coverage public class StringExtensions { public static string TrimEnd(string target, string suffix) { if (target.EndsWith(suffix)) target = target.Substring(0, target.Length-suffix.Length); return target; } } [TestClass] public class StringExtensionTest { [TestMethod] public void TrimEndTest() { var target = "Hello World"; var suffix = "World"; var result = StringExtensions.TrimEnd(target, suffix); Assert.IsTrue(result == "Hello "); } } Introduction Three essential ingredients: Data Method Sequence Assertions void Add() { int item = 3; var list = new List(); list.Add(item); var count = list.Count; Assert.AreEqual(1, count); } list.Add(3); Which value matters? Bad choices cause redundant, or worse, incomplete test suites. Hard-coded values get stale when product changes. Why pick a value if it doesn’t matter? Parameterized Unit Test = Unit Test with Parameters Separation of concerns Data is generated by a tool Developer can focus on functional specification void Add(List list, int item) { var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); } A Parameterized Unit Test can be read as a universally quantified, conditional axiom. void ReadWrite(string name, string data) { Assume.IsTrue(name != null && data != null); Write(name, data); var readData = Read(name); Assert.AreEqual(data, readData); } string name, string data: name ≠ null ⋀ data ≠ null ⇒ equals( ReadResource(name,WriteResource(name,data)), data) Parameterized Unit Tests (PUTs) commonly supported by various test frameworks .NET: Supported by .NET test frameworks http://www.mbunit.com/ http://www.nunit.org/ … Java: Supported by JUnit 4.X http://www.junit.org/ Generating test inputs for PUTs supported by tools .NET: Supported by Microsoft Research Pex http://research.microsoft.com/Pex/ Java: Supported by Agitar AgitarOne http://www.agitar.com/ Goal: Create a parameterized unit test for TrimEnd.. Refactor: Extract Method [TestMethod] public void TrimEndTest() { var target = "Hello World"; var suffix = "World"; var result = ParameterizedTest(target, suffix); Assert.IsTrue(result == "Hello "); } static string ParameterizedTest(string target, string suffix) { var result = StringExtensions.TrimEnd(target, suffix); return result; } Goal: Given a program with a set of input parameters, automatically generate a set of input values that, upon execution, will exercise as many statements as possible Observations: Reachability not decidable, but approximations often good enough Encoding of functional correctness checks as assertions that reach an error statement on failure 17 void PexAssume.IsTrue(bool c) { if (!c) throw new AssumptionViolationException(); } void PexAssert.IsTrue(bool c) { if (!c) throw new AssertionViolationException(); } Assumptions and assertions induce branches Executions which cause assumption violations are ignored, not reported as errors or test cases Human Expensive, incomplete, … Brute Force Pairwise, predefined data, etc… Semi - Random: QuickCheck Cheap, Fast “It passed a thousand tests” feeling Dynamic Symbolic Execution: Pex, Cute,EXE Automated white-box Not random – Constraint Solving Choose next path Code to generate inputs for: void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } F F a.Length>0 a==null T Solve Constraints to solve Data Observed constraints null a==null a!=null && !(a.Length>0) a!=null && a.Length>0 && a[0]!=1234567890 {} a!=null a!=null && a.Length>0 Execute&Monitor {0} condition Negated a!=null && a.Length>0 && a[0]==1234567890 {123…} a!=null && a.Length>0 && a[0]==1234567890 T Done: There is no path left. a[0]==123… F T Open CoverMe project in Visual Studio. Right-click on CoverMe method, and select “Run Pex”. Inspect results in table. public static void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } How to test this code? (Actual code from .NET base class libraries) [PexClass, TestClass] [PexAllowedException(typeof(ArgumentNullException))] [PexAllowedException(typeof(ArgumentException))] [PexAllowedException(typeof(FormatException))] [PexAllowedException(typeof(BadImageFormatException))] [PexAllowedException(typeof(IOException))] [PexAllowedException(typeof(NotSupportedException))] public partial class ResourceReaderTest { [PexMethod] public unsafe void ReadEntries(byte[] data) { PexAssume.IsTrue(data != null); fixed (byte* p = data) using (var stream = new UnmanagedMemoryStream(p, data.Length)) { var reader = new ResourceReader(stream); foreach (var entry in reader) { /* reading entries */ } } } } If the test does not throw an exception, it succeeds. If the test throws an exception, (assumption violations are filtered out), assertion violations are failures, for all other exception, it depends on further annotations. Annotations Short form of common try-catch-assert test code [PexAllowedException(typeof(T))] [PexExpectedException(typeof(T))] A brief overview Instrumentation framework “Extended Reflection” Instruments .NET code at runtime Precise data flow and control flow analysis Pex Code Under Test Z3 ExtendedReflection Pex listens to monitoring callbacks Symbolic execution along concrete execution Pex Code Under Test Z3 ExtendedReflection Constraint solving with SMT solver Z3 Computes concrete test inputs Pex Code Under Test Z3 ExtendedReflection Execute with test inputs from constraint solver Emit test as source code if it increases coverage Pex Code Under Test Z3 ExtendedReflection ExtendedReflection: Code Instrumentation Z3: SMT constraint solver Pex: Dynamic Symbolic Execution Pex Code Under Test Z3 ExtendedReflection class Point { int x; static int GetX(Point p) { if (p != null) return p.X; else return -1; } } ldtoken Point::X call __Monitor::LDFLD_REFERENCE ldfld Point::X call __Monitor::AtDereferenceFallthrough br L2 L1: ldtoken Point::GetX Prologue call __Monitor::AtBranchTarget call __Monitor::EnterMethod Record concrete values call __Monitor::LDC_I4_M1 brfalseL0 ldarg.0 to have allldc.i4.m1 information call __Monitor::Argument<Point> L2: p ==pnull L0: .try { Calls towhen buildthis method is called call __Monitor::RET .try { (The real C#with compiler path condition no proper stloc.0 context call __Monitor::LDARG_0 null Calls will perform ldarg.0 output is actually moreleave L4 call __Monitor::LDNULL } catch NullReferenceException { symbolic computation complicated.) ldnull ‘ call __Monitor::AtNullReferenceException call __Monitor::CEQ rethrow ceq call __Monitor::BRTRUE } Epilogue L4: leave L5 brtrue L1 call __Monitor::BranchFallthrough } finally { call __Monitor::LDARG_0 call __Monitor::LeaveMethod ldarg.0 Calls to build … endfinally path condition } L5: ldloc.0 ret Pex’s representation of symbolic values and state is similar to the ones used to build verification conditions in ESC/Java, Spec#, … Terms for Primitive types (integers, floats, …), constants, expressions Struct types by tuples Instance fields of classes by mutable ”mapping of references to values" Elements of arrays, memory accessed through unsafe pointers by mutable “mapping of integers to values" Efficiency by Many reduction rules, including reduction of ground terms to constants Sharing of syntactically equal sub-terms BDDs over if-then-else terms to represent logical operations Patricia Trees to represent AC1 operators (including parallel array updates) SMT-Solver (“Satisfiability Modulo Theories”) Decides logical first order formulas with respect to theories SAT solver for Boolean structure Decision procedures for relevant theories: uninterpreted functions with equalities, linear integer arithmetic, bitvector arithmetic, arrays, tuples Model generation for satisfiable formulas Models used as test inputs Limitations No decision procedure for floating point arithmetic http://research.microsoft.com/z3 Parameterized Unit Testing Code-Under-Test Project Test Project Parameterized Unit Tests Pex Generated Tests Right-click on the code-under-test project In the Solution Explorer Select “Pex” Select “Create Parameterized Unit Test Stubs” Generated Test Project References Microsoft.Pex.Framework (and more) References VisualStudio UnitTestFramework Contains source files with generated Parameterized Unit Test stubs Right-click on Parameterized Unit Test For now, it just calls the implementation, but the you should edit it, e.g. to add assertions Select “Run Pex Explorations” Input/Output table Current Exploration Row = generated test Issue bar Parameterized Column = parameterized testStatus input or Important messages Unit Test output here !!! Test outcome filtering Failing New Test Fix available Test Passing Test Exception Review test Stack trace Allow exception Pex Explorer makes it easy to digest the results of many Parameterized Unit Tests, and many generated test inputs pex.exe Rich information, used byrendered Pex developers HTML from XML report file: Full event log historyIt is easy to programmatically extract information! Parameter table Code coverage etc… Tell Pex which Type you are testing Code coverage Exception filtering Search prioritization [PexClass(typeof(Foo))] public class FooTest { … 3 Domains UserCodeUnderTest public void CodeUnderTest() { … } UserOrTestCode [PexClass] public class FooTest { [PexMethod] public void ParamTest(…) { … SystemCode public class String { public bool Contains(string value) { … Namespace, type, method filters pex.exe test.dll /tf:Foo /mf:Bar Partial match, case insensitive Full match: add “!”: /tf:Foo! Prefix: add “*”: /tf:Foo* Many other filters e.g. to select a “suite”: /sf=checkin For more information: pex.exe help filtering Goal: Explore input generation techniques with TrimEnd Write or discuss a random input generator for TrimEnd Adapt your parameterized unit tests for Pex Start over with “Create Parameterized Unit Test Stubs”, or… ▪ Add reference to Microsoft.Pex.Framework ▪ Add using Microsoft.Pex.Framework; ▪ Add [PexClass(typeof(StringExtensions))] on StringExtensionsTest ▪ Add [PexMethod] on your parameterized tests Right-click in test and ‘Run Pex Explorations’ Compare test suites It’s called Parameterized Unit Testing The yellow event bar notifies about important events, including certain Click on issue limitations kind for more information You should act on these events: Refactor your code, or tell Pex to ignore it in the future, let Pex analyze (“instrument”) more code, if possible. Understand the issue, Clickand on atake particular anevent action for more information If Pex reports that some code was uninstrumented, you may tell Pex to instrument and analyze it (if possible) Code instrumentation on Demand Instrumentation has high performance overhead Some parts of the code better ignored Use PexInstrument… attributes [assembly: PexInstrumentAssembly(“Foo”)] Pex will often suggest and insert those attributes for you Pex understand managed .NET code only Pex does not understand native code. Problem if branching over values obtained from the environment Pex may not automatically detect all such cases. File System? if (!File.Exists(f)) throw ... Pex analyzes every executed .NET instruction Some used libraries may be surprisingly expensive to analyze XML parsing repeatedly converting data between different representations void Sum(string[] A) { Don’t do this. var sum = “0”; foreach(var a in A) sum = (int.Parse(a) + int.Parse(sum)).ToString(); if(sum == “123”) throw new Exception(); } There are decision procedures for individual path conditions, but… Number of potential paths grows exponentially with number of branches Reachable code not known initially Without guidance, same loop might be unfolded forever To deal with the path explosion problem, Pex uses search heuristics. Heuristics try to create “diversity”, i.e. they prefer paths which are so far underrepresented e.g. by their coverage vectors In the presence of loops and recursion, Pex could search forever, so Pex has many ways to bound the search Named arguments of [PexMethod] [PexMethod(MaxConstraintSolverTime = 4)] Pex will often suggest and update those bounds for you. Configurable bounds include: TimeOut MaxBranches MaxCalls MaxConditions ▪ Number of conditions that depend on test inputs MaxRuns ConstraintSolverTimeOut ConstraintSolverMemoryLimit Division and modulo are expensive to reason about No decision procedures for floatingpoint arithmetic Pex currently uses heuristic search techniques for floating-point constraints Initially, choose Arbitrary Path-constraint solving is just hard. Test Inputs Constraint System Reachability is undecidable! (Loops) Execution Path (With external Known functions and Paths dynamic loading, summaries are difficult.) Pex only explores single-threaded code Related approach to explore threadschedules (but not input parameters): CHESS http://research.microsoft.com/en-us/projects/chess/ Write assertions and Pex will try to break them Without assertions, Pex can only find violations of runtime contracts causing NullReferenceException, IndexOutOfRangeException, etc. Assertions leveraged in product and test code Pex can leverage Code Contracts (discussed later) Pex only uses public methods to configure non-public object fields Heuristics built-in to deal with common types User can help if needed void (Foo foo) { if (foo.Value == 123) throw … [PexFactoryMethod] Foo Create(Bar bar) { return new Foo(bar); } Goal: Understand limitations of DSE. PexGoal/PexExpectedGoals Apply Pex to if (File.Exists(fileName)) PexGoal.Reached(); if (DateTime.Parse(s).Year == 2009) PexGoal.Reached(); class Foo { private Foo() {} } … if (foo != null) PexGoal.Reached(); Tips and tricks Problem: • Constraint solver determines test inputs = initial state of test • Most classes hide their state (private fields) • State is initialized by constructor, and can be mutated only by calling methods • What sequence of method calls reaches a given target state? • There may be no such sequence • In general, undecidable Two approaches: • (Guided) exploration of constructor/mutator methods • Testing with class invariants Specification: [PexMethod] public void ArrayListTest(ArrayList al, object o) { PexAssume.IsTrue(al != null); int len = al.Count; al.Add(o); PexAssert.IsTrue(al[len] == o); } Factory methods Parameterized factories create and configure objects Explicit: Public static method annotated with [PexFactoryMethod] Implicit: Pex guesses “best” constructor/setters Result: Exploration of reachable states Reachable using factory method Reachable within configured bounds Under-approximation of possible states We will now describe how you can create objects via the exploration of a class invariant. Write class invariant as boolean-valued parameterless method Refers to private fields Must be placed in implementation code Write special constructor for testing only May be marked as "debug only" Constructor sets fields, assumes invariant Result: Exploration of feasible states May include states that are not reachable Open project “ArrayListSample”. Explore AddTestExplicit Explore AddTestImplicit Inspect generated test cases Notice constructor implicitly used by Pex Edit code of ArrayList class: Add “ad-hoc” Invariant method: private bool Invariant() { return this._items != null && this._size >= 0 && this._items.Length >= this._size; } Add DEBUG-only constructor that allows to configure object freely, and then checks invariant: #if DEBUG public ArrayList(object[] items, int size) { this._items = items; this._size = size; if (!this.Invariant()) throw new Exception(); } #endif Tips and tricks Assume, Arrange, Act, Assert [PexMethod] void Add(List target, T value) { PexAssume.IsNotNull(target); // assume var count = target.Count; // arrange target.Add(value); // act Assert(target.Count == count + 1)//assert } For an API f(x), f-1(f(x)) = x for all x void PropertyRoundtrip(string value) { var target = new Foo(); target.Name = value; var roundtripped = target.Name; Assert.AreEqual(value, roundtripped); } For an API f(x), f-1(f(x)) = x for all x void ToStringParseRoundtrip(int value) { string s = value.ToString(); int parsed = int.Parse(s); Assert.AreEqual(value, parsed); } For an API f(x), f-1(f(f-1(x)) = f-1(x) for all x void ParseToString(string x) { var normalized = int.Parse(x); var intermediate = normalized.ToString(); var roundtripped = int.Parse(intermediate); Assert(normalized == roundtripped); } Observe a state change void ContainedAfterAdd(string value) { var list = new List<string>(); list.Add(value); Assert.IsTrue(list.Contains(value)); } Given two methods f(x) and g(x), and a method b(y) that observes the result or the exception behavior of a method, assert that f(x) and g(x) have same observable behavior under b, i.e. b(f(x)) = b(g(x)) for all x. public void ConcatsBehaveTheSame( string left, string right) { PexAssert.AreBehaviorsEqual( () => StringFormatter.ConcatV1(left, right), () => StringFormatter.ConcatV2(left, right)); } Given two implementations f and g of the same function, each possible requiring a different number of steps, i.e. f(x)=f1(f2(…(fn(x)…)), and g(x)=g1(g2(… (gm(x)…)), then it should hold that f1(f2(…(fn(x))…) = g1(g2(…(gm(x)…)) for all x. string Multiply(string x, string y); int Multiply(int x, int y); void CommutativeDiagram1(int x, int y) { string z1 = Multiply(x, y).ToString(); string z2 = Multiply(x.ToString(), y.ToString()); PexAssert.AreEqual(z1, z2); } Allowed exception -> negative test case [PexAllowedException(typeof(ArgumentException))] void Test(object item) { var foo = new Foo(item) // validates item // generated test (C#) [ExpectedException(typeof(ArgumentException))] void Test01() { Test(null); // argument check } Indicate which portions of a PUT should be reachable. [PexExpectedGoals] public void InvariantAfterParsing(string input) { ComplexDataStructure x; bool success = ComplexDataStructure.TryParse( input, out x); PexAssume.IsTrue(success); PexGoal.Reached(); x.AssertInvariant(); } Generated test asserts any observed value Return value, out parameters, PexGoal int AddTest(int a, int b) { return a + b; } When code evolves, breaking changes in observable will be discovered void AddTest01() { var result = AddTest(0, 0); Assert.AreEqual(0, result); } Goal: Identify and apply patterns Apply patterns to parameterized unit tests of TrimEnd. Unit test: while it is debatable what a ‘unit’ is, a ‘unit’ should be small. Integration test: exercises large portions of a system. Observation: Integration tests are often “sold” as unit tests White-box test generation does not scale well to integration test scenarios. Possible solution: Introduce abstraction layers, and mock components not under test AppendFormat(null, “{0} {1}!”, “Hello”, “World”); “Hello World!” .Net Implementation: public StringBuilder AppendFormat( IFormatProvider provider, char[] chars, params object[] args) { if (chars == null || args == null) throw new ArgumentNullException(…); int pos = 0; int len = chars.Length; char ch = '\x0'; ICustomFormatter cf = null; if (provider != null) cf = (ICustomFormatter)provider.GetFormat(typeof(ICustomFormatter)); … Introduce a mock class which implements the interface. Write assertions over expected inputs, provide concrete outputs public class MFormatProvider : IFormatProvider { public object GetFormat(Type formatType) { Assert.IsTrue(formatType != null); return new MCustomFormatter(); } } Problems: Costly to write detailed behavior by example How many and which mock objects do we need to write? Introduce a mock class which implements the interface. Let an oracle provide the behavior of the mock methods. public class MFormatProvider : IFormatProvider { public object GetFormat(Type formatType) { … object o = call.ChooseResult<object>(); return o; } } Result: Relevant result values can be generated by white-box test input generation tool, just as other test inputs can be generated! 92 Show AppendFormat code Show AppendFormat test Run Pex 93 Chosen values can be shaped by assumptions public class MFormatProvider : IFormatProvider { public object GetFormat(Type formatType) { … object o = call.ChooseResult<object>(); PexAssume.IsTrue(o is ICustomFormatter); return o; } } (Note: Assertions and assumptions are “reversed” when compared to parameterized unit tests.) 94 Choices to build parameterized models class PFileSystem : IFileSystem { // cached choices PexChosenIndexedValue<string,string> files; string ReadFile(string name) { var content = this.files[name]; if (content == null) throw new FileNotFoundException(); return content; }} Goal: Test a ‘CopyFile’ method with a File System model interface IFileSystem { string ReadFile(string fileName); void WriteFile(string name, string content); } Write a CopyFile method using IFileSystem Implement IFileSystem with System.IO.File and use it to test Copy. Write a parameterized model for IFileSystem and use it to test Copy. Design By Contracts meets Automated Whitebox Testing http://research.microsoft.com/en-us/projects/contracts/ Library to state preconditions, postconditions, invariants Supported by two tools: Static Checker Rewriter: turns Code Contracts into runtime checks Pex analyses the runtime checks Contracts act as Test Oracle Pex may find counter examples for contracts Missing Contracts may be suggested If present, Pex leverages [ContractInvariantMethod] to create objects via reflection and checking if invariant holds: Install Code Contracts: Add reference to Microsoft.Contracts library Add empty [ContractInvariantMethod] Run Pex, click on “Add Invariant” Repeat, edit manually [ContractInvariantMethod] protected void Invariant() { Contract.Invariant(this._items != (object[])null); Contract.Invariant( (uint)(this._size) <= (uint)(this._items.Length)); } Code Contracts allow specification of interface contracts Pex‘ stubs framework can leverage contracts to check input and restrict output of mock objects. Consider the following interfaces annotated with contracts (using Spec# notation): interface IFormatProvider { object GetFormat(Type formatType) requires formatType != null; ensures result != null && } formatType.IsAssignableFrom(result.GetType()); Create IFormatProvider interface with Code Contracts in code-under-test project Create (parameterized) test project Show stubs Create parameterized unit test using IFormatProvider Explore parameterized unit test Random input generators: many Combining random testing and constraint solving DART, CUTE, EXE, KLEE, CREST (C) jCUTE, jFuzz (Java) SAGE (X86) … Program model checkers JPF, Kiasan/KUnit (Java), XRT (.NET) Commercial tools AgitarOne, … 103 http://research.microsoft.com/pex http://codeplex.com/Pex https://sites.google.com/site/asergrp/