Isolated Parameterized Unit Testing with Pex and Moles Nikolai Tillmann, Jonathan “Peli” de Halleux Microsoft Research Learning objectives After I attend this class I will be able to... Write Unit Tests Coverage, assertions, isolation Use Moles to Isolate Unit Tests Test legacy code Write Pex Parameterized Unit Tests Achieve high code coverage Preparation We will use Pex for all exercises Pex includes Moles Visual Studio 2010 Power Tools http://research.microsoft.com/Pex http://www.pexforfun.com Works with .NET 2, 3.5, 4, x86 and x64 Visual Studio 2008, 2010, Command line (Alpha) Silverlight support Preparation Install latest public version pex.powertool.x86.msi Unit Testing Quiz: Unit testing What is a unit test? Unit Testing A unit test is a small program with assertions Test a single (small) unit of code void AddAndCount() { // Arrange int item = 3; // Act var list = new List(); list.Add(item); // Assert Assert.AreEqual(1, list.Count); } static string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); foreach (var line in lines) { int index = line.IndexOf('='); string name = line.Substring(0, index); if (name == "Foo") { string value = line.Substring(index); return value; } t:\myapp.ini } A=B return null; } Foo=C C=D Quiz: Code Coverage How much block coverage do we need? 1. 50% 2. 80% 3. 100% 4. Block coverage alone is not enough Quiz: Coverage How much block coverage do we need? 1. 50% 2. 80% 3. 100% 4. Block coverage alone is not enough ▪ Research: no correlation between high code coverage and quality Quiz: White box testing [TestMethod] void ExistingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“Foo=b”}); Reader.ReadFooValue(); } Do we need more tests to get 100% cov.? [TestMethod] void MissingFoo() { File.WriteAllLines(@"t:\myapp.ini", new string[0]); Reader.ReadFooValue(); } Quiz: Assertions Why write Assertions (or not)? 1. Documentation 2. Double check your program 3. Please your manager 4. Prevent future bugs 5. Validate user inputs 6. Catch errors early Quiz: Assertions Why write Assertions (or not)? 1. Documentation 2. Double check your program 3. Please your manager 4. Prevent future bugs 5. Validate user inputs 6. Catch errors early Quiz: Assertions [TestMethod] void MissingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“a=b”}); string value = Reader.ReadFooValue(); Assert.IsTrue(????); } Which Assertions should you write? 1. Assert.IsTrue(value == “b”); 2. Assert.IsTrue(value == null); 3. Assert.IsTrue(String.IsNullOrEmpty(value)) 4. Assert.IsTrue(true); 5. No assertions Quiz: Assertions [TestMethod] void MissingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“a=b”}); string value = Reader.ReadFooValue(); Assert.IsTrue(????); } Which Assertions should you write? 1. Assert.IsTrue(value == “b”); 2. Assert.IsTrue(value == null); 3. Assert.IsTrue(String.IsNullOrEmpty(value)) 4. Assert.IsTrue(true); 5. No assertions Quiz: Coverage + Assertions What gives you confidence in the code? 1. High coverage, few assertions 2. Low coverage, many assertions 3. High coverage, many assertions 4. Low coverage, no assertions 5. I wrote it Quiz: Coverage + Assertions What gives you confidence in the code? 1. High coverage, few assertions 2. Low coverage, many assertions 3. High coverage, many assertions ▪ Research: Experienced developers write good assertions, junior developers write ‘debugging’ assertions 4. Low coverage, no assertions 5. I wrote it string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); foreach (var line in lines) { int index = line.IndexOf('='); string name = line.Substring(0, index); if (name == "Foo") { string value = line.Substring(index + 1); return value; } t:\myapp.ini } A=B return null; } Foo=C C=D Quiz: Isolation string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); ... In the example, what are the external dependencies? 1. Network Share 2. Local Disk 3. No file system, all in memory Quiz: Isolation string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); ... In the example, what are the external dependencies? 1. Network Share 2. Local Disk 3. No file system, all in memory Quiz: Isolation What is the problem with a Local Disk? 1. Mapping already exists 2. Cannot run tests concurrently 3. Disk full 4. Access rights Quiz: Isolation What is the problem with a Local Disk? 1. Mapping already exists 2. Cannot run tests concurrently 3. Disk full 4. Access rights Unit Testing Exercise Map local directory: > mkdir c:\foo > net use t: \\[machinename]\c$\foo Create C# class library, copy in Reader snippet Create test project, write unit tests Run unit tests Optional: Measure code coverage Definition of Unit Test What is a good Unit Test? A Unit Test is a program that runs fast the code under test, without environment dependencies, with assertions What is a good Unit Test Suite? A set of Unit Tests which achieves high code coverage 10 Minutes Break Isolation with Moles Dependency hell Code under test should not depend on hard-coded environment dependencies: var lines = File.ReadAllLines(@"t:\myapp.ini"); How do you mitigate the Local Disk issues? 1. 2. 3. 4. 5. Always run on the same machine, same hardware, same credentials, same time, same temperature, same solar system configuration Refactoring: use Streams Refactoring: introduce IFileSystem Refactoring: pass the lines as parameter Change implementation of File.ReadAllLines Dependency hell Code under test should not depend on hard-coded environment dependencies: var lines = File.ReadAllLines(@"t:\myapp.ini"); How do you mitigate theReality Local check Disk issues? 1. 2. 3. 4. 5. Always run on the same machine, same Refactoring not hardware, same credentials, same time, same temperature, always an option same solar system configuration Refactoring: use Streams Refactoring: introduce IFileSystem Refactoring: pass the lines as parameter Change implementation of File.ReadAllLines Dependency hell Code under test should not depend on hard-coded environment dependencies: var lines = File.ReadAllLines(@"t:\myapp.ini"); How do you mitigate the Local Disk issues? 1. 2. 3. 4. 5. Always run on the same machine, same hardware, same credentials, same time, same temperature, same solar system configuration Refactoring: use Streams Refactoring: introduce IFileSystem Refactoring: pass the lines as parameter Change implementation of File.ReadAllLines Dependency hell Code under test should not depend on hard-coded environment dependencies: var lines = File.ReadAllLines(@"t:\myapp.ini"); How do you change File.ReadAllLines? 1. Override static method 2. Changing the CLR (and recompiling it) 3. Rewrite application in JScript 4. Code instrumentation Dependency hell Code under test should not depend on hard-coded environment dependencies: var lines = File.ReadAllLines(@"t:\myapp.ini"); How do you change File.ReadAllLines? 1. Override static method 2. Changing the CLR (and recompiling it) 3. Rewrite application in JScript 4. Code instrumentation – the Moles framework Motivation for Moles Why another isolation framework? Specifically designed to enable Pex Simple, Well-defined semantics ▪ “Replace any .NET method” Type safe Moles = Replace any .NET with a delegate var lines = File.ReadAllLines(@"t:\myapp.ini"); What if we could replace File.ReadAllLines? File.ReadAllLines = delegate(string fn) MFile.ReadAllLinesString = delegate(string fn) { return new string[0]; Moles }; Mole Types Code Generation // System.IO public static class File { public static string[] ReadAllLines(string fn); } // System.IO.Moles public class MFile { public static Func<string, string[]> ReadAllLinesString { set; } } // delegate R Func<T, R>(T t); Injecting Detours at Runtime // System.IO public static class File { public static string[] ReadAllLines(string fn) { if (MFile.ReadAllLinesString != null) return MFile.ReadAllLines(fn); … original code } } Automatically injected at runtime Demo Quiz: Func<T> Match the delegates with the methods? 1. Func<string> a) bool File.Exists(string) 2. Action b) Console.WriteLine(string) 3. Action<string> c) void Flush() 4. Func<bool,string> d) String.Empty {get;} 5. Func<string, bool> e) List<T>.Capacity {set;} 6. Action<int> f) string[] 7. Action<List<T>, int> 8. Func<string,string[]> File.ReadAllLines(string) Quiz: Func<T> Match the delegates with the methods? 1. Func<string> a) bool File.Exists(string) 2. Action b) Console.WriteLine(string) 3. Action<string> c) void Flush() 4. Func<bool,string> d) String.Empty {get;} 5. Func<string, bool> e) List<T>.Capacity {set;} 6. Action<int> f) string[] 7. Action<List<T>, int> 8. Func<string,string[]> File.ReadAllLines(string) C# 3.0 Lambdas MFile.ReadAllLinesString = delegate(string fileName) { return new string[]{“a=b”}; } C# 3.0 Lambdas MFile.ReadAllLinesString = (fileName) => { return new string[]{“a=b”}; } C# 3.0 Lambdas MFile.ReadAllLinesString = (fileName) => new string[]{“a=b”}; C# 3.0 Lambdas MFile.ReadAllLinesString = fileName => new string[]{“a=b”}; Quiz: Lambdas Match the Lambdas with the methods 1. () => “” a) bool File.Exists(string) 2. () => {} b) Console.WriteLine(string) 3. s => {} c) void Flush(); 4. (s) => “” d) String.Empty {get;} 5. (s) => false e) string ToString(); Quiz: Lambdas Match the Lambdas with the methods 1. () => “” a) bool File.Exists(string) 2. () => {} b) Console.WriteLine(string) 3. s => {} c) void Flush(); 4. (s) => “” d) String.Empty {get;} 5. (s) => false e) string ToString(); Moles Exercise I Add moles for mscorlib to the test project Add new Item Moles and Stubs for Testing mscorlib.moles Write test using moles Run test Debug test [TestMethod, ...] Exercise II void ReadFooValueTest() { MFile.BehavedAsNotImplemented(); ... static string ReadFooValue() { if (!File.Exists(@"t:\myapp.ini")) return null; string[] lines = File.ReadAllLines(@"t:\myapp.ini"); ... Exercise II Constructors and Instance methods static string ReadFooValue() { var reader = new StreamReader(“t:\myapp.ini”); string content = reader.ReadToEnd(); void ReadFooValueTest(string content) { MStreamReader.ConstructorString = delegate(StreamReader me, string file) => { var mole = new MStreamReader(me); mole.ReadToEnd = () => content; }; static string ReadFooValue() { var reader = new StreamReader(@"t:\myapp.ini")) Exercise II var lines = reader.ReadToEnd().Split(‘\n’); foreach (var line in lines) { int index = line.IndexOf('='); string name = line.Substring(0, index); if (name == "Foo") { string value = line.Substring(index + 1); return value; } } return null; }} Exercise III Quiz: Moles Usage When should you use Moles (and not)? 1. 2. 3. 4. 5. 6. 7. 8. 9. Always use Moles to solve isolation issues With Moles, one does not need to use interfaces anymore Moles only should be used for 3rd party API, use interfaces for isolation in your APIs Moles can be used in production code Moles lets you get away with untestable APIs Moles make test cases more robust With Moles, you do not need integration tests anymore Moles make tests easier to understand Moles is for poor programmers, real programmers rely on interfaces Quiz: Moles Usage When should you use Moles (and not)? 1. 2. 3. 4. 5. 6. 7. 8. 9. Always use Moles to solve isolation issues With Moles, one does not need to use interfaces anymore Moles only should be used for 3rd party API, use interfaces for isolation in your APIs Moles can be used in production code Moles lets you get away with untestable APIs Moles make test cases more robust With Moles, you do not need integration tests anymore Moles make tests easier to understand Moles is for poor programmers, real programmers rely on interfaces Exercise (optional) Step-by-Step Tutorials “Getting Started with Moles” http://research.microsoft.com/pex/molestutorial.pdf “Unit Testing SharePoint Services with Pex” http://research.microsoft.com/pex/pexsharepoint.pdf Tip: Check if the external website matches the version of Pex you installed; the latest tutorials also ship with the installer. What you learned so far The Definition of Unit Testing Unit Test Isolation through Moles 10 Minutes Break Parameterized Unit Testing The Recipe of Unit Testing void AddAndCount() { int item = 3; var list = new List(); list.Add(item); var count = list.Count; // Arrange // Act Assert.AreEqual(1, count); // Assert } Quiz: list.Add(???) list.Add(???); Which value matters? 1. 0 2. -1, 1 3. int.MaxValue, int.MinValue 4. it does not matter 5. I don’t know until I read the code Parameterized Unit Testing void AddAndCount(List list, int item) { var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, forlist.Count); any list, } for any item, Separation of … adding 1 item Specification of behavior increases Count by 1 Data to achieve coverage Problem Where does the data come from? Random data generator Real customer data Ranges Some values hand-picked by dev/tester Dynamic Symbolic Execution Data Generation by Dynamic Symbolic Execution Data Generation Challenge Goal: Given a program with a set of input parameters, automatically generate a set of input values that, upon execution, will exercise as many statements as possible How would you do it? 60 Dynamic Symbolic Execution What tests will Pex generate? Choose next path Code to generate inputs for: void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } F F a.Length>0 a==null T Solve Constraints to solve Execute&Monitor Data Observed constraints null a==null a!=null && !(a.Length>0) a!=null && a.Length>0 && a[0]!=1234567890 a!=null {} a!=null && a.Length>0 {0} a!=null && a.Length>0 && a[0]==1234567890 {123…} a!=null && a.Length>0 && a[0]==1234567890 T Done: There is no path left. a[0]==123… F T Dynamic Symbolic Execution What tests will Pex generate? What tests will Pex generate? Choose next path Solve void CoverMe2(int[] a, int i) Constraints to solve { if (a[i] == a.Length) throw new Exception("bug"); } Execute&Monitor Data Observed constraints Hint: a[i]... F F T if (a === null) throw new NullReferenceException(); T If (i < 0 | i >= a.Length) throw new ArgumentOutOfRangeException(); F *(a+i) T Dynamic Symbolic Execution What tests will Pex generate? What tests will Pex generate? Choose next path Solve void CoverMe2(int[] a, int i) Constraints to solve { a[0] = -1; if (a[i] == a.Length) throw new Exception("bug"); } Execute&Monitor Data Observed constraints Hint: a[i]... F F T if (a === null) throw new NullReferenceException(); T if (i < 0 | i >= a.Length) throw new ArgumentOutOfRangeException(); F *(a+i) T Dynamic Symbolic Execution What tests will Pex generate? Choose next path What tests will Pex generate? void Assert(bool value) { if (false == value) throw new Exception("bug"); } F Solve Constraints to solve T T F F T Execute&Monitor Data Observed constraints Dynamic Symbolic Execution What tests will Pex generate? Choose next path void Foo(int value) { Assert(value == 123); } void Assert(bool value) { if (false == value) throw new Exception("bug"); } F Solve Constraints to solve T T F F T Execute&Monitor Data Observed constraints Dynamic Symbolic Execution Demo Create new project in Visual Studio. Insert CoverMe method below. Right-click on CoverMe method, and select “Run Pex”. Inspect results in table public static void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } How does Pex monitor code? ldtoken Point::X call __Monitor::LDFLD_REFERENCE ldfld Point::X call __Monitor::AtDereferenceFallthrough br L2 class Point { int x; static int GetX(Point p) { if (p != null) return p.X; else return -1; } } ldtoken Point::GetX call __Monitor::EnterMethod brfalseL0 ldarg.0 call __Monitor::Argument<Point> L0: .try { .try { call __Monitor::LDARG_0 ldarg.0 call __Monitor::LDNULL ldnull call __Monitor::CEQ ceq call __Monitor::BRTRUE brtrue L1 call __Monitor::BranchFallthrough call __Monitor::LDARG_0 ldarg.0 … L1: call __Monitor::AtBranchTarget call __Monitor::LDC_I4_M1 ldc.i4.m1 L2: call __Monitor::RET stloc.0 leave L4 } catch NullReferenceException { ‘ call __Monitor::AtNullReferenceException rethrow } L4: leave L5 } finally { call __Monitor::LeaveMethod endfinally } L5: ldloc.0 ret (Pex / Moles Architecture) Pex Z3 SMT solver Moles ExtendedReflection Dynamic Runtime Analysis Exercise (Optional / homework) Code Digging with Pex Open digger.pdf http://research.microsoft.com/pex/digger.pdf Open VS2008 Follow PDF tutorial You can pick up initial project from digger\digger.sln Stop at section 6: A Glimpse of Parameterized Unit Tests Code Digging Exercise 1. 2. 3. 4. 5. Refactor logic in helper method (as usual) Make helper method “public” Right-click “Run Pex” in helper method Fix bugs Make helper method “private” (note: future versions of Pex will support private methods) Digger Exercise static string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); return ReadFooValue(lines); } public static string ReadFooValue(string[] lines) { foreach (var line in lines) { int index = line.IndexOf('='); string name = line.Substring(0, index); if (name == "Foo") { string value = line.Substring(index + 1); return value; } } return null; } 10 Minutes Break Pex Parameterized Unit Testing Test Generation Work-Flow Code-Under-Test Project Test Project Parameterized Unit Tests Pex Generated Tests Parameterized Unit Testing Exercise I – Crash Test Add a reference to Microsoft.Pex.Framework.dll Add the following parameterized unit test “Run Pex Explorations” using Microsoft.Pex.Framework; [TestClass, PexClass(typeof(Reader))] public partial class ReaderTest { [PexMethod] public void Crash(string[] lines) { MFile.ReadAllLines = () => lines; Reader.ReadFooValue(); } } Parameterized Unit Testing Exercise II – Crash Test Press “1 Uninstrumented method” Select row Click on “Instrument Assembly” (lower right) Run Pex Explorations again Execute All Tests in the Solution Parameterized Unit Testing Assumptions Add assumptions and run Pex again [PexMethod] public void Crash(string[] lines) { PexAssume.IsNotNull(lines); MFile.ReadAllLines = () => lines; Reader.ReadFooValue(); } Parameterized Unit Testing Pex Asserts Observed Results Manually review outputs Pex inserts Assertions automatically [PexMethod] public string Regression(string[] lines) { PexAssume.IsNotNull(lines); MFile.ReadAllLines = () => lines; // Observe return Reader.ReadFooValue(); } Parameterized Unit Testing Assert something “Ensure that we parse a Foo entry correctly” [PexMethod] void FooExist([PexAssumeNotNull]string value) { string[] lines = { “Foo=“ + value }; MFile.ReadAllLines = () => lines; var result = Reader.ReadFooValue(); PexAssert.AreEqual(value, result); } Pattern 4A – Assume, Arrange, Act, Assert Assume, Arrange, Act, Assert [PexMethod] void Add(List<T> target, T value) { // Assume PexAssume.IsNotNull(target); // Arrange var count = target.Count; // Act target.Add(value); // Assert Assert(target.Count == count + 1); } 10 Minutes Break Pex from the Command line pex.exe pex.exe test.dll HTML Reports Rich information, used by Pex developers Full event log history Parameter table Code coverage etc… Limitations and other Details It’s called Parameterized Unit Testing Event Bar The yellow event bar notifies about important events, including certain limitations Click on issue kind for more information Events View You should act on these events: Refactor your code, or tell Pex to ignore it in the future, let Pex analyze (“instrument”) more code, if possible. Instrumenting more code If Pex reports that some code was uninstrumented, you may tell Pex to instrument and analyze it (if possible) Instrumenting more code Code instrumentation on Demand Instrumentation has high performance overhead Some parts of the code better ignored Use PexInstrument… attributes [assembly: PexInstrumentAssembly(“Foo”)] Pex will often suggest and insert those attributes for you Testability Pex understand managed .NET code only Pex does not understand native code. Problem if branching over values obtained from the environment Pex may not automatically detect all such cases. File System? if (!File.Exists(f)) throw ... Hidden Complexity Pex analyzes every executed .NET instruction Some used libraries may be surprisingly expensive to analyze XML parsing repeatedly converting data between different representations void Sum(string[] A) { Don’t do this. var sum = “0”; foreach(var a in A) sum = (int.Parse(a) + int.Parse(sum)).ToString(); if(sum == “123”) throw new Exception(); } Exploration Boundaries Configurable bounds include: TimeOut MaxBranches MaxCalls MaxConditions ▪ Number of conditions that depend on test inputs MaxRuns ConstraintSolverTimeOut ConstraintSolverMemoryLimit Exercise Limitations Goal: Understand limitations. Apply Pex to public static void Y2kParser(string s) { if (DateTime.Parse(s).Year == 2000) throw new Exception(“found it”); } Patterns for Parameterized Unit Tests Pattern Roundtrip For an API f(x), f-1(f(x)) = x for all x void ToStringParseRoundtrip(int value) { string s = value.ToString(); int parsed = int.Parse(s); Assert.AreEqual(parsed, value); } Pattern Reachability Indicate which portions of a PUT should be reachable. [PexAssertReachEventually] public void Constructor(object input) { new Foo(input); PexAssert.ReachEventually(); } Pattern Same Observable Behavior Given two methods f(x) and g(x), and a method b(y) that observes the result or the exception behavior of a method, assert that f(x) and g(x) have same observable behavior under b, i.e. b(f(x)) = b(g(x)) for all x. public void ConcatsBehaveTheSame( string left, string right) { PexAssert.AreBehaviorsEqual( () => StringFormatter.ConcatV1(left, right), () => StringFormatter.ConcatV2(left, right)); } Patterns How to create objects Tips and tricks Creating objects is hard [PexMethod] void ReaderTest(Reader reader) { if (reader.Value == “foo”) ... } Pex needs an instance of Reader but… Reader is abstract… Reader has no public constructors… Reader has many constructors… … Pex has heuristics but will call for help Factory methods Reader Hand-written factory method: public static ReaderFactory { [PexFactoryMethod(Reader)] public static Reader Create(string value) { return Reader.HiddenCreate(value); } } Pex will explore both factory method and PUT! Object Creation Exercise public class Reader { string fileName; private Reader(string fileName) { this.fileName = fileName; } public static Reader Create(string fileName) { return new Reader(fileName); } string ReadFooValue() { var lines = File.ReadAllLines(this.fileName); } } Homework Patterns Read patterns paper: patterns.pdf http://research.microsoft.com/Pex/patterns.pdf 10 Minutes Break Pex and Moles Together Exercise public class IniReader { string section; public IniReader(string section) { this.section = section; } public string ParseIni(string key) { using (var reader = new StreamReader(@"t:\myapp.ini")) { bool inSection = false; string line; while((line = reader.ReadLine()) != null) { if (line[0] == '[' && line[line.Length - 1] == ']') { string currentSection = line.Substring(1, line.Length - 2); inSection = currentSection == section; } else { int index = line.IndexOf('='); if (index > -1){ string currentKey = line.Substring(0, index); if (key == currentKey) { return line.Substring(index); } } } } return null; }} Wrapping up What you learned today The Definition of Unit Testing Unit Test Isolation through Moles Parameterized Unit Tests with Pex Where to go from here Web: http://research.microsoft.com/pex/ Links to blogs, more documentation, stubs, moles General questions: http://social.msdn.microsoft.com/Forums/en-US/pex/ Thank you Become a fan on Facebook http://research.microsoft.com/pex