® IBM Software Group IBM Software Development Platform Software Quality Sales 201 Training 2005 © 2003 IBM Corporation IBM Software Group Agenda Strategy IBM Software Development Platform e-Business on demand Overview of IBM Software Quality products and offerings IBM Rational Functional Tester IBM Rational Robot IBM Rational Manual Tester IBM Rational Performance Tester IBM Rational PurifyPlus IBM Rational Test RealTime Exercises Additional References 2 IBM Software Group Imperatives in Today’s World: Innovation & Productivity Innovating the business to capture new value. Business IT Improving the productivity of resources deployed. 3 IBM Software Group IT is the lever to maximize flexibility and responsiveness while containing cost An on demand operating environment is an integrated infrastructure aligned to business goals and processes in a resilient and secure manner Business-Driven Development On Demand Demand On Operating Operating Environment Environment All designed with an architecture that allows you to manage services as components Results: Simplification and Optimization of IT to meet the needs of the business responsively 4 IBM Software Group The challenge: Poor visibility, lack of cohesion across business and technology domains Business View Poor visibility and governance over IT investments Lack of actionable information Blind decision-making Operations View Inadequate service levels Inability to rapidly deploy applications Complex, multi-tier operating environments Business Analysts Operations Application Developers Application Development View Overwhelming complexity Relentless time-to-market pressure Uncontrolled change 5 IBM Software Group The solution: Breaking down the silos A shared view of the development lifecycle Business View Clear view of technology ROI Top-down and bottom-up visibility into technology projects Objective decision-making support Operations View Improved service and quality compliance Predictable deployments Accelerated diagnosis and repair Business Analysts Operations Application Developers Application Development View Rapid application development and deployment Improved collaboration Asset reuse 6 IBM Software Group Transform and simplify software development Business-driven development An integrated approach to software development that aligns line-of-business, development and operations teams to improve business performance Development as a business process • Align Technology and Business priorities • Improve efficiency and responsiveness • Create innovative products • Higher productivity - 50% + increase in developer productivity • Improved quality - 80% fewer bugs • Greater predictability Software development becomes a driver of competitive advantage 7 IBM Software Group Discover, develop, and deploy assets Discover business & technology assets Prioritize Plan Manage Measure Business priorities Business Requirements Middleware and software assets Develop at the speed of business Rapid application development Model-driven architecture Asset-based development Direct-to-middleware productivity Deploy to closed-loop environments Operations Optimize Iterate Automated applications deployment Streamlined composite application management Direct-to-operations productivity 8 IBM Software Group The business-driven development lifecycle Prioritize Plan Manage Measure Business Executive Model the Business Optimize End User Define Requirements Analyst Govern Analyze & Design Operations Manager Project Manager Architect Manage change & assets Manage Implement Application Support Deploy Developer Test Operations Deployment Manager Optimize Tester Development Iterate 9 IBM Software Group The IBM Software Development Platform A complete, open, modular, and proven solution Analyst Model, simulate, assemble, and monitor processes Project Manager Executive Architect Visually model applications and data Developer Tester Rapidly construct, transform, integrate and generate code Design, create, and execute tests Follow a common process Manage and measure projects and portfolios Manage requirements Deployment Manager Provision, configure, tune and troubleshoot applications Manage change and assets Manage quality Align investments with business objectives Analyze and monitor project portfolios 10 IBM Software Group Agenda Strategy IBM Software Development Platform e-Business on demand Overview of IBM Software Quality products and offerings IBM Rational Functional Tester IBM Rational Robot IBM Rational Manual Tester IBM Rational Performance Tester IBM Rational PurifyPlus IBM Rational Test RealTime Exercises Additional References 11 IBM Software Group Focus # 1: Discover Assets Optimize Business Investments DISCOVER IBM SOFTWARE DEVELOPMENT PLATFORM Identify and analyze Business processes Requirements Software Assets Packaged applications Existing applications Middleware Impact of potential change Business process optimization is the #1 priority in 2004 — Information Week Study 2Q04 — 400 business-technology executives 12 IBM Software Group Focus # 2: Develop Assets Improve Time-to-Value IBM SOFTWARE DEVELOPMENT PLATFORM Dramatically accelerate the development cycle Rapid application development Model-driven architecture Asset based development Direct-to-middleware productivity Service-oriented architecture Tightly integrated run-time test and deploy Anytime, anywhere, team collaborative environments 13 IBM Software Group Focus # 3: Deploy Assets Deliver Business Results Enable closed-loop deployment IBM SOFTWARE DEVELOPMENT PLATFORM Unify development and operations teams Accelerate deployment and provisioning Automate applications monitoring Monitor business performance and analyze ROI Verify and manage change 14 IBM Software Group A team-based approach All tools and team members support all development imperatives Analyst Model, simulate, assemble, and monitor processes Project Manager Executive Architect Visually model applications and data Developer Tester Rapidly construct, transform, integrate and generate code Design, create, and execute tests Follow a common process Manage and measure projects and portfolios Manage requirements Deployment Manager Provision, configure, tune and troubleshoot applications Manage change and assets Manage quality Align investments with business objectives Analyze and monitor project portfolios 15 IBM Software Group IBM Software Quality Offerings Product Description Capabilities Runtime analysis Component, functional and performance testing Cross-platform test execution support Test & planning reporting Ensures reliability, functionality, scalability Reduces cycle time Accommodates multiple skill sets, roles Shares responsibility for quality across team Business Analyst Tester Developer IBM Rational Functional Tester IBM Rational Performance Tester IBM Rational Robot IBM Rational PurifyPlus Key Products IBM Rational Manual Tester IBM Rational Test RealTime IBM Rational TestManager 16 IBM Software Group From the Industry Analysts IBM Rational Software Quality Products Ovum "IBM Rational has been focusing on strategic integration of the ASQ* toolset with the complete IBM Software Development Platform, including the use of the Eclipse framework for software tool integration. We expect over the next 18 months IBM Rational's ASQ tools will again provide best-of-breed, singlepoint test tools, which will excel by operating within a formidably integrated environment across the development, test and delivery lifecycle. As a bonus, the fact that the platform is based on Eclipse open standards means that you will not be locked into IBM Rational-only tools.“ – Source Paul Herzlich (Ovum) 08.04 - Ovum Evaluates: Software Testing Tools IDC - Noted Rational’s “extremely well established and complementary product set to IBM’s.” Source: Worldwide Distributed Automated Software Quality Tools 2004-2008 Forecast and 2003 Vendor Shares M. Webster (IDC) 08.04 * Automated Software Quality 17 IBM Software Group Agenda Strategy IBM Software Development Platform e-Business on demand Overview of IBM Software Quality products and offerings IBM Rational Functional Tester IBM Rational Robot IBM Rational Manual Tester IBM Rational Performance Tester IBM Rational PurifyPlus IBM Rational Test RealTime Exercises Additional References 18 IBM Software Group Challenges of Functional Testing Control cost of test automation Inexperienced testers face steep learning curve for tools “Our script language limits our ability to overcome some automation obstacles.” Test maintenance grows with project Testing Tools Lack Flexibility Power users hands are tied by tool limitations “I need one tool for both my expert and novice testers.” 19 IBM Software Group Functional Testing with IBM Rational Functional Tester Test automation for the novice and the professional IBM Rational Functional Tester Automated regression testing for Web, J2EE, .Net and terminal-based applications Key Benefits Minimize test maintenance with scripts resilient to application changes Wizard enhanced automation to speed test creation for the new user Powerful scripting language and IDE for the professional tester Supports Team oriented parallel development 20 IBM Software Group Challenge 1: Controlling the Cost of Automation Challenge Overcome the complexity of test automation to ensure teams spend more time on high value quality assessment rather than on low return test maintenance. Resolution Provide technologies to enable novice testers to craft complex tests with no programming Data driven wizard Dynamic data verification Reduce script maintenance Wizard driven global updates to centralized object map 21 IBM Software Group Challenge 2: Tools Lack Flexibility Challenge Fulfill both standard test needs as well as expert requirements to ensure usage limited only by the imagination of the project team. Resolution Provide two test script development languages Java Visual Basic .NET Give professional testers the flexibility to overcome testing challenges in the language of their choice 22 IBM Software Group Challenge 3: Test Team Integration Challenge Accommodate the requirements of both dedicated, independent QA teams as well as integrated project teams to ensure high operational integrity. Resolution Provide version control support in the box with ClearCase LT Provide integrations for standard version control features Check In/Out Compare with Previous Versions Show change history 23 IBM Software Group What is Automated Testing? Using automation tools to record actions taken against an application. Test scripts are derived from these recordings which are then used for subsequent test execution. Record Enhance Execute 24 IBM Software Group Challenge of Manual Testing with Short Test Cycles Tests ...the new Reruntests... the first tests and... Manual testing can’t keep pace! ...take more time Time Build 1 Build 2 Build 3, 4, 5 … Manual testing alone can’t keep pace with the rapid development of applications Quality will be sacrificed to keep on schedule IBM Software Group Effective Test Automation Creating an automated test is a three steps process Record Script Recording Capture user interactions with the system under test 26 IBM Software Group Effective Test Automation Creating an automated test is a three step process Record Enhance Script Recording Capture user interactions with the system under test Script Enhancing Modify scripts for situation specific challenges 27 IBM Software Group Effective Test Automation Creating an automated test is a three step process Record Enhance Execute Script Recording Capture user interactions with the system under test Script Enhancing Modify scripts for situation specific challenges Script Execution Ensure reliable playback for local or remote execution 28 IBM Software Group IBM Rational Functional Tester Overview GUI Technologies • Automates regression testing of Java, .NET, Web and terminal emulator-based applications Data-driven Testing • Simplified data-driven testing enabled by wizards and Excellike data tables Eclipse availability • When using Java scripting, runs inside Eclipse/RSA/RAD shell • Full Eclipse JDT features are accessible • Accessible to developers from within their IDE IBM Rational Functional Tester Tests Java, .NET, Web and terminal emulator-based apps Java and VB.NET-based test scripts Test Script Language • User has choice of two test script languages – Java and VB.NET • Use of Java implies editing and debugging in Eclipse/RAD/RSAbased editor • Use of VB.NET implies usage of Microsoft VS.NET-based editor/debugger • Identical capabilities in both editors Wizard and table-assisted data-driven tests Linux-based test authoring Improved Linux support • Users can author and modify test scripts on Linux (no script recorder) • Scripts can also be executed on Linux Eclipse Java Development Tools Manual Tester ClearCaseLT in the box • Supports parallel development of manual tests • Enables remote test access for distributed teams Automated functional regression testing ClearCase LT Manual Tester in the box • Customers receive one Manual Tester license per Functional Tester license • Improve return on manual testing efforts while adopting automated test methods 29 IBM Software Group IBM Rational Functional Tester New Positioning Lowering the Cost of Functional Testing Offers Extensive Customization Options Supports Integration of Test Team with Rest of the Project Team What’s New in Atlantic Broader environment support Adds support for testing VS.NET Winforms applications Adds ability to edit and playback scripts on Linux (no recording) Advances in ease of use Choice of script languages Wizard for data-driven testing Improved object map maintenance capabilities Ships with full-use copies of IBM Rational Manual Tester and IBM Rational ClearCaseLT and customers can download Rational TestManager 30 IBM Software Group Rational Functional Tester – Key Features and Benefits Test Java, Web & VS.NET Winform apps Lower TCO with a single tool supporting a broad array of industry standard application development technologies Java or VB.NET scripting language Easier adoption of the tool – Teams select the language they are most familiar with. Helps sell to Microsoft customers Add-on for Terminalbased application testing Central QA and IT teams can support client-server, web and legacy application testing with single tool Wizards for datadriven testing Eliminates manual coding required to vary data in test script Expanded dynamic verification Ability to use regular expressions in data pools created with new wizard for datadriven testing Advanced Object Map Maintenance Global Find and Modify tools further reduce test maintenance 31 IBM Software Group Rational Functional Tester – Key Features and Benefits Version Control “inthe-box” Collocated teams can support multiple versions of test scripts and parallel development paths with no additional tools to buy. Distributed teams can upgrade to ClearCase to facilitate remote testing, whether offsite or off-shore. Ability to install, edit and execute tests on Linux Developers and testers in diverse, heterogeneous environments benefit from a common toolset 32 IBM Software Group Summarizing Key Features in IBM Functional Tester Broad Environment Support Test Web, Java. .net and terminal-based applications Create data driven tests without coding Static data and properties verification Dynamic data validation without coding Flexible and Extensible Scripts Flexible coding language Powerful, professional debugger Object map editing flexibility Version control of scripts Script Execution Options Reliable playback with ScriptAssure Remote and local playback on various platforms 33 IBM Software Group Reduce Test Script Maintenance Reliable Playback with Script Assure Version 1.0 Version 2.0 Tester Sees No User Intervention Required With Tool TM Sees ScriptAssure Determines Match 34 IBM Software Group Case Study SAS Institute, Inc. Challenge Improve communication between developers and testers Solution IBM Rational Functional Tester Enables best-of-breed automated functional and regression testing from within an Eclipse environment, including testing from within IBM WebSphere Studio and Rational XDE Developer Benefits Productivity and quality improvements – Efficiency has gone up and the quality of software delivery has gone up Bring quality issues to the surface faster – fix these issues and deliver software on time Minimize script maintenance through resilient reusable test scripts Achieved better results by using a consistent methodology More information http://www-306.ibm.com/software/success/cssdb.nsf/CS/JENS-5WVSPY?OpenDocument&Site=software 35 IBM Software Group Competitive Advantages IBM Rational Functional Tester Part of a proven, complete, open and modular software development platform Single vendor solution Integration with other development tools Provides a shared development/test environment and common language on application development projects using Eclipse or Visual Studio.net Choice of a Java or Visual Basic.net scripting language Allows testers to leverages existing Java or VB.net skills on program teams Product wizards to help build complex tests without coding Provides a single tool to test Java, Web, .net and terminal-based applications ScriptAssure technology for building robust, resilient test scripts 36 IBM Software Group Resources IBM Rational Functional Tester IBM Rational Functional Tester Web page http://www-306.ibm.com/software/awdtools/tester/functional/index.html Rational Functional Tester evaluation copy http://www14.software.ibm.com/webapp/download/preconfig.jsp?id=2005-0119+04%3A01%3A18.402144R&S_TACT=104CBW71&S_CMP= Introduction to IBM Rational Functional Tester http://www-128.ibm.com/developerworks//rational/library/04/r-3228/ developerWorks testing resources http://www-130.ibm.com/developerworks/rational/products/xdetester Success stories http://www.ibm.com/software/success/cssdb.nsf/topstoriesFM?OpenForm&Site=rational Training http://www-306.ibm.com/software/sw-training/ Events http://www-306.ibm.com/software/sw-events/ 37 IBM Software Group Agenda Strategy IBM Software Development Platform e-Business on demand Overview of IBM Software Quality products and offerings IBM Rational Functional Tester IBM Rational Robot IBM Rational Manual Tester IBM Rational Performance Tester IBM Rational PurifyPlus IBM Rational Test RealTime Exercises Additional References 38 IBM Software Group Effective Test Automation Creating an automated test is a three step process Record Enhance Execute Script Recording Capture user interactions with the system under test Script Enhancing Modify scripts for situation specific challenges Script Execution Ensure reliable playback for local or remote execution 39 IBM Software Group Recording Scripts Create a Test Script that captures all system-user interaction Test scripts are recorded on the fly, as user navigates application Verification points are inserted to validate system response Key considerations when recording scripts Environment Support Flexible verification ability 40 IBM Software Group Environment Support for Recording Scripts IBM Rational Robot has special environment support for key development languages Java, HTML, .NET, Visual Basic, MFC PowerBuilder, Delphi, Oracle Forms Robot also has a general recording engine which can be used against other windows and terminal emulation environments 41 IBM Software Group Verification Points in Recording Scripts Robot Sees Data You See… Automated Validation Robot Sees Properties Robot captures data and properties that can be invisible to users During script execution, current results are compared to stored baselines Discrepancies are flagged and reported to user 13 different verification points are available to validate system behavior including Rational SiteCheck for web site link checking 42 IBM Software Group Enhancing Scripts Enhancing Scripts with basic coding extends their value and reach SQA Basic code is added to perform a variety of functions Typical Modifications: Conditional branching, datapooling, refactoring Key considerations when enhancing scripts Flexible coding language Extensibility of language and playback engine 43 IBM Software Group Flexible Coding Language for Enhancing Scripts IBM Rational Robot uses the SQA Basic Language Uses Visual Basic syntax Augments VB commands with test specific functions Easy to read language very suitable to those will little programming experience Flexible power to enhance scripts Programmatic access to all GUI objects Datapool facility enables data driven tests Programmatic access to Windows API Programmatic access to COM objects 44 IBM Software Group Solving Custom Problems with Extensibility Robot has an extensibility interface to allow support for custom controls Development partners have been trained Can manufacture support for ActiveX, .NET Windows Forms, Java & Win32 including MFC and ATL GUI controls Eliminates the work around Allowing testers to focus on the task of testing 45 IBM Software Group Executing Tests Scripts are executed, discrepancies are noted Scripts are executed and test logs created Test logs are highlight differences between actual and expected results Key considerations when executing scripts Multiple execution points Coordination of scripts 46 IBM Software Group Automated Testing Test Run from Robot or Command Line Remote Tests Run From Test Manager Master Script Execution Options Scripts can be run from Robot GUI Scripts can be executed from Command line Scripts can be run from TestManager Offers ability to run scripts on remote machines 47 IBM Software Group Effective Test Automation Agent Agent Agent Master Powerful and flexible scheduling TestManager will coordinate timing and dependencies Schedule execution on remote computers from central control station TestManager is configuration aware, i.e.: Tests designated as Windows XP will only be run on XP machines 48 IBM Software Group Summarizing Key Features in IBM Rational Robot Broad Environment Support Support for all major development environments Flexible and Extensible Scripts Simple, yet powerful, scripting language for enhancing scripts Support available for custom control Script Execution Options Run scripts from Robot, TestManager or command line 49 IBM Software Group Case Study Corporate Express Challenges Cut testing time and challenges in manual testing process manual regression testing found to be tedious, time-consuming and error-prone Improve repeatability of tests Improve confidence in software quality at release Solution Automated the manual process with IBM Rational Robot Benefits Cut testing time from days to weeks, over 75% 31/2 weeks of testing in 3 days Team confidence that each regression test is executed accurately and with a consistent process from one release to the next More information http://www-306.ibm.com/software/success/cssdb.nsf/CS/JENS-5WPKR2?OpenDocument&Site=software . 50 IBM Software Group Resources IBM Rational Robot Web page http://www-306.ibm.com/software/awdtools/tester/robot/index.html IBM developerWorks page for IBM Rational Robot http://www-136.ibm.com/developerworks/rational/products/robot Success stories http://www.ibm.com/software/success/cssdb.nsf/topstoriesFM?OpenForm&Site=rational Training http://www-306.ibm.com/software/sw-training/ Events http://www-306.ibm.com/software/sw-events/ 51 IBM Software Group Agenda Strategy IBM Software Development Platform e-Business on demand Overview of IBM Software Quality products and offerings IBM Rational Functional Tester IBM Rational Robot IBM Rational Manual Tester IBM Rational Performance Tester IBM Rational PurifyPlus IBM Rational Test RealTime Exercises Additional References 52 IBM Software Group What is Manual Testing? Using human labor to validate the business use cases and fitness for use of an application, system or device Includes labor to plan, design, execute and evaluate the status of the application or device under test Experience indicates over 75% of the testing done today is performed manually 53 IBM Software Group Why Teams use Manual Testing? Recruitment of subject-matter experts and end-users to test Strict regulatory or process requirements aren’t met with automation tools Poor or non-existent test automation-ready interfaces 54 IBM Software Group Rational Manual Tester Positioning Advances Manual Testing Satisfies Diverse Needs Simplifies Through Assistance Key Capabilities Supports manual testing done by analysts, subject-matter experts and technical/non-technical testers Develop detailed, easy to follow manual tests Minimize test maintenance and ensure test consistency Reduce human error during data entry and data verification Support diverse usage models and processes Ships with full copy of ClearCaseLT for version control of tests 55 IBM Software Group Challenges of Manual Testing Labor Intensive Too much time is spent creating and maintaining manual test documentation “More time is spent maintaining test documentation than on executing tests.” “We struggle to document even a minimum set of tests.” 56 IBM Software Group Challenges of Manual Testing Labor Intensive Too much time is spent creating and maintaining manual test documentation “Our results are unreliable due to human error.” Unreliable Inadvertent human-error often skews test results “Our tests aren’t detailed enough for the testers to consistently execute them.” 57 IBM Software Group Challenges of Manual Testing Labor Intensive Too much time is spent creating and maintaining manual test documentation “Each project has different standards, which are hard to manage manually.” Unreliable Accidental human-error often skews test results Resistant to Change Each project has diverse processes, tester skill sets and types of applications and devices under test “No single tool or process works for the different software and hardware systems we must test.” 58 IBM Software Group Challenge 1: Labor Intensive Challenge Creating and maintaining manual tests is labor intensive and difficult with tools such as Microsoft Word or Excel. Resolution Implement a solution that promotes the best practice of modular tests which can share common content Leverage existing Word or Excel test documents to quickstart test development Share test content 59 IBM Software Group Challenge 2: Unreliable Challenge Human error often skews tests results Resolution Assist testers with the tasks most prone to error – data entry and data verification Build manual tests that are clear, concise and easy for the tester to follow 60 IBM Software Group Challenge 3: Resistant to Change Challenge Each project has diverse processes, tester skill sets and types of applications and devices under test Resolution For all skill levels Provide a solution that testers of all skills levels can leverage Customize the tool to fit each project’s process metrics and values Leverage a solution that is independent of the software or hardware under test For any process For any target of test 61 IBM Software Group IBM Rational Manual Tester Overview Rich Text Editor • Modify test font, size, color; attach images and files • Supports creation of both test steps and verification points • Ensures clarity of test step direction and consistency of test execution Manual test authoring and execution IBM Rational Manual Tester Rich test editor Test step reuse library Tool Assistance • Automated data entry • Automated data comparison and results storage • Reduces opportunity for manual error during test execution Data entry & validation assistance Test Import / Results Export Reuse Library • Library for common test procedures • Drag-and-drop construction of new tests • Automatic global update when modifying any one instance of linked content Import/Export • Imports manual test source files from Microsoft Word, Microsoft Excel and Rational TestManager • Exports test results to CSV format for additional analysis Custom test fields Customizable • Add new data fields to individual test steps • Apply internally-defined naming conventions ClearCase LT ClearCaseLT in the box • Supports parallel development of manual tests • Enables remote test access for distributed teams 62 IBM Software Group Rational Manual Tester – Key Features and Benefits Rich Text Editor Reduces the ambiguity that often plagues other methods used to define and deliver test step directions Import From Word, Excel or TestManager Easier adoption of the tool Multiple Validation Points Allows test results to be reported and analyzed at either a high level or a granular level Reuse Palette Promotes shared content which results in time & cost saving in test maintenance & greater control over test quality and consistency Custom Data Fields Easily adapts to existing processes and metrics collection standards Assisted Data Entry and Data Verification Reduce the likelihood of human error in the areas most prone to mistakes 63 IBM Software Group Rational Manual Tester – Key Features and Benefits Translucent Execution UI The translucent window used to display test instructions makes test completion faster by eliminating the need to move the test instruction window in order to see the application under test Supports Team Use Supports individuals, teams, and mobile works through private and shared test data stores – ensuring test assets remain easily accessible and reportable – ClearCaseLT in the box Generates Spreadsheet-Ready Results Testers can use any reporting tool to analyze results - ensuring that Rational Manual Tester can easily work with a team’s processes and standards for metrics collection and reporting 64 IBM Software Group Summarizing Key Features in Manual Tester Advances manual testing Rich text editor for test authoring Test step reuse palette Test script import from Microsoft Word or Excel Satisfies diverse needs Supports usage by distributed teams Generates spreadsheet-ready results data Multiple validation points Custom data fields Simplifies through assistance Assisted data entry and data verification during test execution 65 IBM Software Group Resources IBM Rational Manual Tester IBM Rational Manual Tester Web page http://www-306.ibm.com/software/awdtools/tester/manual/index.html Rational Manual Tester evaluation copy http://www14.software.ibm.com/webapp/download/preconfig.jsp?id=2005-0116+03%3A32%3A04.405210R&S_TACT=104CBW71&S_CMP=&s= Rational Manual Tester evaluation guide http://www-106.ibm.com/developerworks/edu/i-dw-r-manualtest-i.html Article A simple tool to manage manual testing http://www-128.ibm.com/developerworks/rational/library/04/r-3232/ developerWorks testing resources http://www-128.ibm.com/developerworks/rational/products/manualtester/ Success stories http://www.ibm.com/software/success/cssdb.nsf/topstoriesFM?OpenForm&Site=rational Training http://www-306.ibm.com/software/sw-training/ Events http://www-306.ibm.com/software/sw-events/ 66 IBM Software Group Agenda Strategy IBM Software Development Platform e-Business on demand Overview of IBM Software Quality products and offerings IBM Rational Functional Tester IBM Rational Robot IBM Rational Manual Tester IBM Rational Performance Tester IBM Rational PurifyPlus IBM Rational Test RealTime Exercises Additional References 67 IBM Software Group IBM Rational Performance Tester Overview New load testing technology • New load test recorder and execution engine • Supports HTTP/S; additional protocol support (e.g. ERP) planned for Q2 2005 High scalability • High scalability, low footprint due to hyperthreaded architecture • Significant improvement over Robot technology Load and performance testing IBM Rational Performance Tester HTTP/S load testing Visual test construction and editing High scalability hyper-threaded execution architecture Graphical test scheduler and user profiler Platform Support • Multiple platform support for load generation ClearCaseLT in the box • Supports parallel development of manual tests • Enables remote test access for distributed teams Windows and Linux load generation support Eclipse Java Development Tools Visual Test Editor • Test viewer requires no coding for test modification and execution • Automated identification and variation of dynamic data • Insertion of Java code is supported for advanced customization needs Test scheduler • Graphical user profiling and transaction modeling • Eases modeling of user populations Eclipse-based • Runs inside Eclipse/RSA/RAD shell • Accessible to developers from within their IDE ClearCase LT 68 IBM Software Group IBM Rational Performance Tester New Positioning Built for Day One Productivity Provides Advanced Data Capabilities Lowers Cost of Performance Testing What’s New in Atlantic Completely re-architected performance testing solution for testing complex web application Designed for novice and advanced performance testers Real-time monitoring for immediate feedback during test execution Easy to build data driven tests Eclipse-3 integration 69 IBM Software Group Rational Performance Tester – Key Features and Benefits Visual test editor Easy for a novice to understand, with drilldown to detailed data for more expert users. Simplifies test creation and maintenance Automatic identification of variable data Allows tests to automatically re-use data sent by server and to prepare for data-driven testing Eclipse based Provides a single environment for development as well as testing activities Rich scheduling editor Novice users can easily assemble and schedule tests Built-in datapool editor for data-driven testing Easily vary data for more realistic load tests 70 IBM Software Group Rational Performance Tester – Key Features and Benefits Real-time reporting Know immediately about performance problems Remote system monitoring Collect and visualization of remote server information. Find resource constraints that cause poor performance High Scalability Low memory and processor footprint required to generate large multi-user loads. Allows larger tests with less hardware Java execution engine Tests may call custom Java provided by the user. Provides highlycustomizable and flexible performance testing 71 IBM Software Group Resources IBM Rational Performance Tester IBM Rational Performance Tester Web page http://www-306.ibm.com/software/awdtools/tester/performance/index.html Success stories http://www.ibm.com/software/success/cssdb.nsf/topstoriesFM?OpenForm&Site=rational Training http://www-306.ibm.com/software/sw-training/ Events http://www-306.ibm.com/software/sw-events/ 72 IBM Software Group Agenda Strategy IBM Software Development Platform e-Business on demand Overview of IBM Software Quality products and offerings IBM Rational Functional Tester IBM Rational Robot IBM Rational Manual Tester IBM Rational Performance Tester IBM Rational PurifyPlus IBM Rational Test RealTime Exercises Additional References 73 IBM Software Group The Problem with Trying to Pinpoint Bugs Crash Errors triggered Symptoms Crash 74 IBM Software Group What Makes Bugs Difficult to Find? Many memory-related bugs have no immediate visible symptoms Often the symptoms are difficult, if not impossible, to reproduce Manually tracing symptoms back to problems in the source code can be a daunting task Crash Crash 75 IBM Software Group Why Is My Program So Slow? Do you deliver slow code that’s “good enough?” Is it too difficult to find performance problems? Can you get repeatable, useful performance data? Can you quantify the effect of performance fixes? Does your performance profiling data help you manage your development project? What about Predictability at runtime? 76 IBM Software Group Code Coverage Challenge Can you guarantee the code that you deliver is reliable? How complete is your test suite? Are you sure that you have tested the entire application? Can you understand the summary of several tests? How Reliable is Untested Code? 77 IBM Software Group IBM Rational PurifyPlus: Developing Fast, Reliable Code To develop fast, reliable code, you need a tool that Automatically pinpoints hard-to-find bugs Highlights performance bottlenecks Keeps you from shipping untested code You need a multi-platform and multi-language tool for Unix, Windows, Linux, Java, .NET, VB6, C/C++, and more IBM Rational PurifyPlus combines three industry leaders into one box: IBM Rational Purify IBM Rational Quantify IBM Rational PureCoverage The x-ray for software Instantly become more productive 78 IBM Software Group Detect Memory Problems: Rational Purify Automatically identifies the problem while the application is running! Finds errors in third-party code, even without source Bugs in libraries that you deliver are still bugs in your application! No recompile required Much faster and easier to use Incorporate into your existing project quickly and easily Test what you are delivering No need to maintain multiple source baselines QE knows that they are testing what you are delivering Detects memory corruption and memory leaks 79 IBM Software Group Improve Code Performance: Rational Quantify Leading product that provides repeatable performance data Build better quality C/C++, Java, Visual Basic 6.0, and .NET software Get your team solving the correct problems Fast and easy to use Provides concise, graphical views of performance data “River of Time” – a straightforward way to view and understand performance profiling data Easily integrates into your nightly builds Collects performance data from code you intend to deliver No need to maintain multiple source baselines 80 IBM Software Group Identify Untested Code: IBM Rational PureCoverage Helps developers deliver higher quality code You know the unit testing is complete Helps identify “dead code” or unexercised paths in legacy code Helps improve your QA process Easily see what has and has not been tested Validate the quality and completeness of your test suite Easy to learn and easy to use out of the box With Rational PureCoverage, you know that you are covered! Avoid shipping untested code 81 IBM Software Group IBM Rational PurifyPlus Key components of IBM Rational PurifyPlus Provides run-time data collection and analysis capabilities Key for proactive debugging Supports Unix, Windows, Java, .NET and Linux environments Linux C/C++ RedHat Full 64bit Windows C/C++ VB6 .NET Java Solaris SGI-IRIX HP-UX C/C++ Java Full 64bit C/C++ Full 64bit C/C++ Full 64bit Quantify PureCoverage Not Available Yet 82 IBM Software Group IBM Rational PurifyPlus Summary To develop fast, reliable code, you need IBM Rational PurifyPlus that: Automatically pinpoints hard-to-find bugs Highlights performance bottlenecks Keeps you from shipping untested code Powerful tool for developers and testers Strong ROI when used as “medicine” Even better ROI when used as “vitamin” Used as part of the development process Instantly become more productive 83 IBM Software Group Case Study Ensemble Systems Challenge Ensemble Systems was contracted to develop an easy-to-maintain software system to perform data analysis twice as fast as an existing solution, while significantly reducing analysis errors. Solution Rational Rose, Rational Purify and Rational Quantify able to effectively model the system, detect and fix memory leaks and enhance the overall performance of the application under development. Benefits Reduced the time the system required to analyze defects by 85%, and reduced the system's error rate by 90%. Quickly pinpointed memory leaks and identified opportunities to improve performance More Information http://www-306.ibm.com/software/success/cssdb.nsf/CS/JENS-5PFQXR?OpenDocument&Site=software 84 IBM Software Group Case Study SAS Institute, Inc. Challenge Improve software quality Solution Rational Purify and Rational Quantify to help find performance bottlenecks and memory problems in the software Integrated Rational Purify and Rational Quantify into testing process through product APIs Benefits Achieve higher level of confidence in the quality of software releases More information http://www-306.ibm.com/software/success/cssdb.nsf/CS/JENS-5WVSWK?OpenDocument&Site=software 85 IBM Software Group Competitive Advantages IBM Rational PurifyPlus Part of a family of tools that comprise the IBM Software Development Platform Integration across tools Can form a more complete development and testing solution Multilanguage support Java, but also Visual C++, Visual Basic, Visual Basic.NET and Visual C#.NET in one package Module and test level coverage Offline analysis of data Merging test results 86 IBM Software Group Resources IBM Rational PurifyPlus IBM Rational PurifyPlus Web page http://www-306.ibm.com/software/awdtools/purifyplus/ developerWorks: Rational http://www.ibm.com/developerworks/ Getting Started Guides Training – Web-based or Instructor-led Articles - whitepapers – forums Success stories http://www.ibm.com/software/success/cssdb.nsf/topstoriesFM?OpenForm&Site=rational Training http://www-306.ibm.com/software/sw-training/ Events http://www-306.ibm.com/software/sw-events/ 87 IBM Software Group Agenda Strategy IBM Software Development Platform e-Business on demand Overview of IBM Software Quality products and offerings IBM Rational Functional Tester IBM Rational Robot IBM Rational Manual Tester IBM Rational Performance Tester IBM Rational PurifyPlus IBM Rational Test RealTime Exercises Additional References 88 IBM Software Group IBM Rational Test RealTime Fix your code before it breaks Rational Test RealTime is a cross-platform testing solution designed for those who write code for real-time and embedded software products Automates testing, analysis and bugs solving early during development cycle Provides a low-overhead technology for enabling targetindependent tests code code code code code code code code code code code code code code code code code code code code code code Resolve Test Analyze Extends model-driven development to include runtime analysis capabilities Fix the defects, enhance your tests and move on! 89 IBM Software Group Embedded and real-time software testing challenges Embedded and real-time systems are complex by nature Application Complexity Strong timing constraints Low memory footprints Concurrent/Distributed/Networked Environment Complexity Multiple RTOS/IDE/Chips vendors Limited host-target connectivity Low built-in debugging capabilities Process Complexity Requirements Design translation errors Difficult to maintain Poor performance 90 IBM Software Group Embedded and real-time software testing challenges It’s hard to set up and deploy an effective testing framework Are you frustrated by manual testing because of its tendency to be Labor-intensive? Resistant to reuse? Are you concerned that test creation and execution will Pull you out from your comfort zone? Add overhead to an already crowded schedule? Require to much efforts to ensure quality at both source code and system level? Are you convinced that your embedded environment is too specific for a commercial testing solution because of Design constraints? Heterogeneous build/execution tools? Strict testing techniques and activities recommended by certification standards? 91 IBM Software Group Embedded and real-time software testing challenges You need a way to test on the host AND on the actual target! If your target platform is not available, you can test on the host To simulate the operational environment and discover generic bugs But tests must be performed in the final target environment as well Since some errors are only detected under the operational conditions Cross-platform testing is difficult because the system conception… Is frequently based on multiple chips for different execution machines or boards Combines more than one programming language Presents very constrained architectural characteristics 92 IBM Software Group Embedded and real-time software testing challenges How to bring runtime analysis information at the UML level? You want to use the Unified Modeling Language (UML) to visually lay out your application's design and tests but you still don’t know how State machines and code coverage of model-level execution are related? If there are any memory leaks in the hand-coded parts of the application? To measure the performance profile of the generated code? 93 IBM Software Group How can your team overcome these challenges? Would this accelerate quality? Automated component testing at all levels of complexity Full automation of the creation and deployment of test scripts, stubs and drivers for the simplest functions up to distributed systems Faster debugging thanks to extensive runtime analysis capabilities Memory and performance profiling, code coverage and runtime tracing features Make testing an integral part of development Dynamic links between tested code, test results and potentially visual models Prioritize your tests Code complexity and static metrics calculation Fully repeatable test sessions Complete automation of regression testing 94 IBM Software Group How can your team overcome these challenges? Would this solve the verification constraints in the real target? An integration framework flexible enough to accommodate your selection of compiler, linker, debugger, and target architecture To generate test harnesses executable in the real operational conditions To avoid changing test scenarios when target environment does Automatic test download, execution and results upload on target To free developers from target management restraints To help your development team concentrating primarily on test design Low-overhead and minimally intrusive verification probes and drivers To cope with low memory footprints and limited CPU resources To encourage in-situ verification of the software despite target constraints 95 IBM Software Group How can your team overcome these challenges? Would this empower your model-centric development process? Develop, debug and analyze at the model level Graphically visualize reached and un-reached application states and transitions Track memory errors and performance in the hand-coded portions of the application while staying in the modeling environment Trace application execution flow in a UML-compliant graphical format 96 IBM Software Group Test, analyze and resolve during development Test as you code Automatic component test template and data generation Black- and white-box testing All levels of complexity: from single functions to distributed systems Static metric calculation for tests prioritization complexity estimation code code code code code code code code code code code code code code code code code code code code code code Resolve Test Analyze Full regression testing 97 IBM Software Group Test, analyze and resolve during development Test as you code Analyze while you test Code coverage analysis Memory profiling Performance profiling Runtime tracing code code code code code code code code code code code code code code code code code code code code code code Test Thread profiling Resolve Analyze 98 IBM Software Group Test, analyze and resolve during development Test as you code Analyze while you test Resolve what you have uncovered Test execution integrated with your debugger Consolidated, detailed to-the-point test reporting Test data hyperlinked to runtime analysis results and code code code code code code code code code code code code code code code code code code code code code code code Resolve Test Analyze Fix the defects, enhance your tests and move on! 99 IBM Software Group Test, analyze and resolve during development Overview of IBM Rational Test RealTime features Component Testing Code Coverage C, C++, Ada, Java C, C++ Ada, Java Memory Performance Runtime Thread Profiling Profiling Tracing Profiling C, C++ Java C, C++ Java C, C++ Java C, C++ Java Unit, Integration, Validation, Regression and System Testing System Testing C Target Deployment Port Built to achieve standards compliance DO-178B (Avionics), MISRA (Automotive), DEF STAN 00-55 (Defense) 100 IBM Software Group Test, analyze and resolve during development Test RealTime provides a mean for achieving pervasive quality Flexible to minimally impact preexisting development processes Powerful and source-code aware test languages drive robust data-driven tests Process agnostic, so you can stay in your comfort zone Shortens time-to-problem-resolution, maximizes time-to-code White-box and black-box testing for all your software modules Component testing of C/C++, Ada or Java (J2ME/J2SE) functions and classes Message-based testing of C threads, tasks, processes and system nodes Full set of runtime analysis features to enhance your debugging capabilities Enables quality verification from a single and unified testing tool Handles all test enablement activities so you test early and often Creates shared debug and test vocabulary for your team Produces testing information required for certification processes conformance 101 IBM Software Group Test and debug both host and target Target Deployment Port technology A versatile feature which fully automates Test harness building and downloading on the target Test execution and monitoring Test results uploading on the host A low-overhead feature which leverages Your choice of compiler/linker/debugger Your host-target connectivity (serial link, Ethernet connection, JTAG probe…) Test Harness Downloading Target Execution And Monitoring Hosted Build Process Test Results Uploading 102 IBM Software Group Test and debug both host and target 4-Bit to 64-Bit Cross-Development Environments Used By Our Customers WindRiver Montavista Tasking GreenHills TI CAD-UL ARM NEC Cosmic Enea Hitachi Hiware Windows CE Apex Hitex LynuxWorks Sun Symbian Lauterbach Microtec …… Languages C C++ Ada Java (J2ME/J2SE) Platforms Windows Solaris Linux HP-UX AIX 103 IBM Software Group Test and debug both host and target Target Deployment Port provides full target independence Customizable to support a complete range of embedded targets Assures tool adoption Reduces your ramp-up time when target configuration changes Guarantees the reuse of test assets despite target constraints Host, build and target environment agnostic Ensures portability of test and runtime analysis processes Simplifies multiple team deployment Optimizes ROI in comparison to home-grown test solutions Size and speed optimized to limit target impact Enables full control to minimize instrumentation overhead Frees your tests from having to compensate for target restraints Avoids overtasking your system 104 IBM Software Group Harness the power of model-driven testing IBM Rational Test RealTime - IBM Rational Rose RealTime integration Move from code-centric to model-centric development Perform test and runtime analysis at the model level Rose RealTime sequence diagrams for test scripting and Quality Architect RealTime for test generation Test RealTime plug-in for the runtime analysis of your C++ models From within Rose RealTime for C++, activate the following runtime analysis features State-machine coverage Memory and performance profiling UML runtime tracing 105 IBM Software Group Harness the power of model-driven testing From within IBM Rational Rose RealTime Covered UML state Uncovered UML transition UML Sequence Diagram as a test case 106 IBM Software Group Harness the power of model-driven testing From within IBM Rational Rose RealTime Develop and debug at the model level Seamlessly integrate Test RealTime runtime analysis capabilities Rose RealTime application execution Visualize covered states/transitions in your UML state machines 107 IBM Software Group Summary Fix your code before it breaks with IBM Rational Test RealTime A unique cross-platform solution for component testing and runtime analysis Automated functional testing at all levels of complexity to ensure your software reliability Memory and performance profiling, code coverage measurement and runtime tracing abilities to diagnose run-time issues Dynamic links between code, test assets and visual models to make testing an integral part of the development cycle Low-overhead technology for enabling target-independent tests and runtime analysis All in one tool – for any IDE and hosted on any target! 108 IBM Software Group Case Study Alcatel Space Challenge When building a communication system for a satellite project, Alcatel needed to shorten testing time yet meet stringent testing standards and overcome significant testing challenges. Solution Rational Test RealTime enabled them to successfully achieve these objectives. Benefits Increased development productivity as a result of validation testing improvements Heightened confidence in the overall quality of the application under test Improved the predictability of the application's performance further ensuring customer satisfaction More information http://www-306.ibm.com/software/success/cssdb.nsf/CS/JENS-5MHK6Z?OpenDocument&Site=software 109 IBM Software Group Competitive Advantages IBM Rational Test RealTime Full and flexible solution in a single tool Integrated with lifecycle development tools IBM Rational ClearCase IBM Rational ClearQuest IBM Rational Test Manager IBM Rational Rose RealTime Rational Unified Process Integrated with a UML model-driven development solution Test deployment at all levels of verifications and granularity Component testing and runtime analysis capabilities at the source code level Test RealTime Target Deployment Port feature Deploy tests on any target from 4-bit to 64-bit architecture 110 IBM Software Group Resources Rational Test RealTime Web page http://www-306.ibm.com/software/awdtools/test/realtime/index.html Real-Time and embedded solutions webinars http://www.ibm.com/software/rational/info/literature/whitepapers.jsp Rational Developer Network http://www-136.ibm.com/developerworks/rational/products/testrealtime Success stories http://www.ibm.com/software/success/cssdb.nsf/topstoriesFM?OpenForm&Site=rational Training http://www-306.ibm.com/software/sw-training/ Events http://www-306.ibm.com/software/sw-events/ 111 IBM Software Group Quiz Which roles on the software development team can benefit from IBM Software Quality products and offerings? What are the benefits of an automated functional testing product such as IBM Rational Functional Tester? Name three of the major features in Rational Functional Tester. Which development languages does Rational Functional Tester support for creating test scripts? What is the function of IBM Rational Robot? Name three major features of Rational Manual Tester. What does the assisted data verification feature in Rational Manual Tester do? What does the visual test editor in Rational Performance Tester do? What are the three areas of code analysis in IBM Rational PurifyPlus? What are two of the competitive advantages of IBM Rational PurifyPlus? What does Target Deployment Port technology do in Rational Test RealTime? 112 IBM Software Group © Copyright IBM Corporation 2004. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo, the e-business logo and other IBM products and services are trademarks or registered trademarks of the International Business Machines Corporation, in the United States, other countries or both. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries or both. Microsoft, Windows, Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries or both. All other trademarks, company, products or service names may be trademarks, registered trademarks or service marks of others Disclaimer: NOTICE – BUSINESS VALUE INFORMATION IS PROVIDED TO YOU 'AS IS' WITH THE UNDERSTANDING THAT THERE ARE NO REPRESENTATIONS OR WARRANTIES OF ANY KIND EITHER EXPRESS OR IMPLIED. IBM DISCLAIMS ALL WARRANTIES INCLUDING, BUT NOT LIMITED TO, IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. IBM DOES NOT WARRANT OR MAKE ANY REPRESENTATIONS REGARDING THE USE, VALIDITY, ACCURACY OR RELIABILITY OF THE BUSINESS BENEFITS SHOWN.. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGES, INCLUDING THOSE ARISING AS A RESULT OF IBM'S NEGLIGENCE.WHETHER THOSE DAMAGES ARE DIRECT, CONSEQUENTIAL, INCIDENTAL, OR SPECIAL, FLOWING FROM YOUR USE OF OR INABILITY TO USE THE INFORMATION PROVIDED HEREWITH OR RESULTS EVEN IF IBM HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. THE ULTIMATE RESPONSIBILITY FOR ACHIEVING THE CALCULATED RESULTS REMAINS WITH YOU. 113