® IBM Software Group Rational Developer for System z – Unit Testing with zUnit Jon Sayles / IBM - jsayles@us.ibm.com © IBM Corporation IBM Trademarks and Copyrights © Copyright IBM Corporation 2013, 2014. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. This information is based on current IBM product plans and strategy, which are subject to change by IBM without notice. Product release dates and/or capabilities referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo, the on-demand business logo, Rational, the Rational logo, and other IBM Rational products and services are trademarks or registered trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others. 2 Agenda Topics: Unit Testing Value Proposition, zUnit Terms, Concepts and Vocabulary Demonstration Next Steps 3 What is Unit Testing? Unit Testing is a software testing method by which individual units of source code, sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures are tested to determine if they are fit for use. Wikipedia An individual unit of software is a single test-able logic construct or routine within a call-able program: • Date validation • Credit Card number look-up • Tax computation • Co-pay calculation This method of testing is sometimes called “white Box testing” (See Slide Notes for more on White Box Testing) 4 Black Box Testing Black Box Testing examines the functionality of an application (what the software does) without regard to its internal semantics (how the program does what it does) Analyze Input Black Box Analyze Output Classic Example - Batch File Maintenance Testing Run File Update Verify Changes Program Analyze Data Files QSAM DB2 or IMS to Data Files QSAM VSAM 5 DB2 or IMS VSAM White Box (Unit) Testing and zUnit White Box Testing is a method of testing software that tests the internal workings of an application, as opposed to its functionality or its outputs. During White Box testing, code is run with preselected input values for the validation of preselected output values. Test Harness Date Validation Routine • • • • • • Initialize Street Address Initialize City Initialize State Initialize Zip Code Call Program using… Evaluate test results Unit Test… Software Program Address Validation Test Internal Table Lookup Address Validation State Tax Calculation IBM has implemented White Box testing as “zUnit” 6 Software Program White Box Tests zUnit and the xUnit Standard xUnit is an industry-standard, program testing framework - that helps create and automatically run repeatable, self-checking unit test cases. xUnit provides: Assertion-style test validation capabilities and result reporting. Automated execution validation - instead of independent/interactive testing activity At run-time, the xUnit framework distinguishes between failures and errors: A failure is an anticipated problem (e.g., actual output does not match expected). An error is an unexpected, catastrophic problem (e.g., protection exception, null pointer) jUnit is an instance of the xUnit architecture for Java zUnit is IBM’s implementation of the xUnit architecture for System z zUnit provides the same industry-standard output reports as jUnit You can use the same XML-reporting dashboards and products that you currently use with jUnit to review and analyze results 7 Why bother to Unit Test? By testing individual logic routines in your programs: • You can move through the lifecycle more quickly, because you have precise feedback about separate logic routines • So you can better understand cause & effect • And you know that your code works and you know how your code works, which gives you confidence to make enhancements and modifications • Because you execute zUnit Tests through JCL: • The testing can be automated • The end-to-end process takes less time than interactive debugging. • And it can be more systematic • By isolating and verifying code fragments Unit Testing allows you to understand “cause & effect” in your program logic All of the above benefits allow you to catch errors earlier in the lifecycle… 8 The value of early and extensive testing “80% of development costs are spent identifying and correcting defects” ** Once released into production During the Coding or Unit Testing phases During the BUILD phase During Quality Assurance or the System Test phases $240/defect $960/defect $7,600/defect + Law suits, loss of customer trust, damage to brand $80/defect **National Institute of Standards & Technology Source: GBS Industry standard study Defect cost derived in assuming it takes 8 hours to find, fix and repair a defect when found in code and unit test. Defect FFR cost for other phases calculated by using the multiplier on a blended rate of $80/hr. 9 What do you use to Unit Test? Unit Testing is typically done using a product or a framework that provides: 1. Standardized process, methodology and a component framework (architecture) 2. Tools for code generation or some method of simplifying the development Unit Tests 3. Automation - running scripts - or in the case of zUnit running JCL and not testing interactively The industry leadership position in the Unit Testing space is occupied by the xUnit component framework • xUnit has been adopted by almost all of the major computing languages/vendors/tools providers • Including IBM • Thus… zUnit 10 How do you Unit Test - with zUnit Using zUnit you: 1. Develop Unit Tests for your programs 2. Execute the Unit Tests through JCL or interactively 3. Interpret/Analyze your Unit Test results The Unit Tests you develop are COBOL programs or PL/I procedures • RDz generates 95+ percent of the COBOL and PL/I Unit Testing code • RDz provides the test harness (Test Runner) as an installed run-time • And RDz provides both an interactive and batch test execution workflow The zUnit development process is: • Wizard-driven • Evolving - as customers adopt, and feedback next-generation enhancements and suggestions 11 zUnit Support - as of RDz v9.1 • The call-able (zUnit targeted) program is a z/OS Load Module that can be written in any z/OS language - including but not limited to: COBOL PL/I 4GL z/OS applications, generated to; COBOL or PL/I; CA-Easytrieve, SAS, IBM Report Writer, CA-Telon, NETRON/CAP™, CA-Meta COBOL, etc. Assembler (see slide notes) (what this means…) You zUnit test at the Load Module level: The underlying language used to create the targeted business logic is irrelevant. Note that this is differs significantly from jUnit and the other xUnit-based language-dependent frameworks, and product offerings • In RDz v9.1 - zUnit supports Test Case + “stub module” generation, plus automated testing and result XML generation for: COBOL batch call-able subroutines - with or without QSAM/VSAM and DB2 file I/O IMS and CICS are under development 12 zUnit Capabilities (High-level Feature/Function) zUnit Test Runner • Runs on z/OS • Installed and configured on z/OS as part of RDz Host install and customization • Fetches and runs the Test Suite referred to in a zUnit configuration file zUnit Wizard used to generate Test Cases • RDz client feature • Eclipse based wizards allow creation of: • Template Test Cases are generated in COBOL or PL/I • Simple pass/fail assertion API (RDz v9.1) Complete COBOL test cases: • Identify the interface or set of copy book(s) • Generate XML Schema to represent the interface • Generate XML files where you would specify test input and expected output • Generate a Test Case based on the XML file • (Optionally) Generate stubs for called programs RDz viewers/editors for unit test XML results 13 zUnit/Unit Testing is Not a “Silver Bullet” IBM recognizes this - and provides multiple testing solutions, each optimized for different problem areas within the software testing space: Rational Function Tester GUI 3270 Regression Testing Performance Testing Application Performance Analyzer Interactive Testing RDz Integrated Debugger Integration Testing IBM Debug Tool Automated Unit Testing zUnit Test Driven Development 14 xUnit Components and Terminology All xUnit frameworks share the following basic component architecture, with some varied implementation details. Test Runner A test runner executes test cases or test suites and makes the results available in some programmatically-consumable way (e.g., XML,...). Test Case Test cases execute within a test fixture and make assertions during test execution. Test cases contain any number of tests where each test is a separate subprogram, function, method, or procedure in the test case. Test Suite Test suites define a set of test cases for test execution. Assertions Assertions verify the behavior and state of a test case. A failed assertion should result in an exception being thrown. Test Fixture A test fixture is a context and/or environment that the programmer creates (setup()) before a test in a test case is executed, and discards (teardown()) immediately after the test is run. Test Execution Test execution involves invoking the Test Runner with input that includes, among other things, which test Cases or test suites should be run. Test Execution Launches Test Runner Reads from Test Results (XML File) Test Suite Which identifies There are hundreds of distributed xUnit products and implementations (jUnit, JSUnit, NUnit, XMLUnit,etc.). They all contain code frameworks in the underlying supported language. Many but not all are TAP compliant. Test Case Which constructs Test Fixture Assertions Which executes 15 Against… Your program and the program logic to be tested zUnit Component Implementation RDz substitutes generated z/OS-specific resources for distributed technology classes, and frameworks. A z/OS systems program developed by the RDz team, and installed with RDz. The Test Runner orchestrates the unit testing process Test Runner Test Case A generated COBOL or PL/I program compiled into a Load Module that calls your target Subroutine. Test Cases are executed (Called) by the Test Runner Test Suite An XML file that provides a list of Test Cases to be executed by the Test Runner Assertions COBOL or PL/I conditions that test individual units of source code in your program Test Fixture A generated set of COBOL or PL/I programs that implement the xUnit framework standard; for setting up, running and tearing down individual Unit Tests Test Execution z/OS JCL that invokes the Test Runner Test Execution //JCL Submit JCL EXEC PGM= Test Runner - RDz zUnit run-time OPEN INPUT READ Test Suite Test Suite - List of Test Cases (Load Modules) Which identifies Test Case Load Module Test Results (XML File) Test Case - Generated COBOL or PL/I program The Test Runner CALLs (interacts with) Test Case Load Module Which contains Test Fixtures (COBOL subprograms or PL/I procedures Each Test Fixture sets up and CALLs (executes) individual Unit Tests and releases unit test resources (Tear Down) Test Fixture - Generated COBOL or PL/I programs Assertions - Conditional Expressions coded in COBOL or PL/I Makes 16 Your program and the logic to be tested zUnit Test Case Development Wizard RDz provides a wizard-driven process to create your Test Cases and Test Fixture programs 17 zUnit Test Case Module zUnit Generated Test Case USER.ZUNIT.LOAD(UNIT0001) UNIT0001(...) The Test Case is a generated COBOL or PL/I module that contains: • Nested programs with standard xUnit names and functionality • 1 M generated Test Fixtures (programs) - One program for each of the Tests defined using the wizard ADDTESTS(...) SETUP(...) TEARDOWN(...) TEST1(...) ...... TESTn(...) You customize the Test Case programs, and add statements specific to your business logic and testing requirements: • Initialize test input variables • Evaluate test outcome RDz’s zUnit framework takes care of the rest 18 Agenda Topics: Unit Testing Value Proposition, zUnit Terms, Concepts and Vocabulary Demonstration Next Steps 19 Demonstration What you will see: 1. Develop Unit Tests for your programs • Use the zUnit Automated Testing Framework to generate a Test Case • • Which includes Test Fixtures that contain Assertion logic that unit tests your program Compile & Link the Test Case 2. Execute the Unit Test 3. Interpret/Analyze results 20 http://www.youtube.com/watch?v=vjNY7SMtVqU&list=UUdgVL1ysbledgitPTYKyJuQ&index=1&feature=plcp IBM Rational Developer for System z version 8 21 21 Review - What you Saw - zUnit Run-time Test Execution JES // Submit JCL… Launches… EXEC PGM=Test Runner //AZUCFG Test Suite //AZURES Test Results //…Test Case Load Library zUnit Test Runner Test Suite //AZUCFG DD DSN=… • Names the Test Case module(s) (Test Harness) READ Test Suite CALL (invoke) Test Case Test Case Invoke Test Fixture Interact with different nested COBOL subprograms or PL/I procedures within your Test Case to: - Setup unit test variable values - Run a test. - Evaluate the Assertion outcome - Release system resources Generate Results File Test Fixture - Target Subroutine Test Results z/OS SETUP Linkage Variables Run Unit Test Assertion Logic (CALL) Evaluate outcome Release resources Load Module to be unit tested //AZURES DD DSN=… • Stores test results in XML PDS member 22 • Testable logic routine • Testable logic routine • Testable logic routine Agenda Topics: Unit Testing Value Proposition, zUnit Terms, Concepts and Vocabulary Demonstration Next Steps 23 Review - zUnit Test Case Development Wizard RDz provides a wizard-driven process to create your Test Cases and Test Fixture programs 24 Review - xUnit Vocabulary / zUnit Implementation Test Execution: JCL that invokes the Test Runner (a “supervisor program” or “test harness” that runs your Test Suite) Test Runner: The zUnit “supervisor program” invoked by Test Execution, that: Reads a Test Configuration Test Suite Calls your Test Cases Test Suite: an XML configuration file that defines which Test Cases the Test Runner should execute (Note - a Test Case == a Load Module) Test Case: An RDz-generated COBOL or PL/I program that: Is called by the Test Runner Uses a Test Fixture to Call your Subroutine and write test outcomes to a Test Results file Test Fixture: A set of generated programs nested inside your Test Case. Test Fixtures run Tests that utilize Assertion logic to Call your Subroutine, and evaluate the result. Subroutine: The target call-able program to be Unit Tested Test Results: An output dataset that describes the results of each test run in xUnit standard (XML) file format Assertion: A simple pass/fail predicate used to test one part of your Subroutine’s logic ** Test Execution, Test Runner, Test Suite, Test Case, Test Fixture, Assertion == xUnit vocabulary 25 26 Links - to additional Unit Testing content http://lisacrispin.com/wp-content/uploads/2011/11/Agile-Testing-Quadrants.png http://1.bp.blogspot.com/-RQLtpTssY_o/UZ9CYqzflqI/AAAAAAAAAyo/9kIx6aGwSaU/s1600/TestingTrianglePished.png http://en.wikipedia.org/wiki/White-box_testing http://en.wikipedia.org/wiki/Black-box_testing http://en.wikipedia.org/wiki/XUnit http://en.wikipedia.org/wiki/Test-driven_development 27