Risk Based Validation
January 2014
© 2013 IBM Corporation
Questions
 Do you wish to be able to stay current with the latest releases of Maximo?
 Do you wish to save costs in validation and upgrade?
If you answered yes to either or both of these questions….
stay tuned, we are here to help in achieving these objectives!
© 2013 IBM Corporation
Topics
 Risk-Based Validation Program – Overview
 Approach
 Analysis Questions
 Supportive Documentation
 History and 2013 Hi-lights
 Developing a Risk Based Validation Strategy – Best Practices
 Discussion Questions
© 2013 IBM Corporation
Risk-Based Validation
 For many years, the FDA has promoted a shift to a risk-based approach to validation
– Software validation should be focused on “a justified and documented risk
assessment and a determination of the potential of the system to affect product
quality and safety, and record integrity.”
 Practical motivations behind risk-based validation:
– Companies can more rigorously test the areas of the software system that pose
the greatest risk to product quality and patient safety based on intended use
– Overall validation costs are reduced and efficiency is increased
– An industry-wide shift to risk-based validation would lead to faster adoption of
innovation in manufacturing and new technological advances
 While the risk-based approach is taking hold in regulatory industries, it can and
should be used by any and all industries
© 2013 IBM Corporation
Risk Based Validation Program - Overview
 This unique program supports customers’ implementation of a risk-based approach to Maximo
validation for all releases (e.g. new installs, upgrades, fix packs)
– Enables moving to latest fix pack/versions
– Enables reduction of costs and efforts
– Shifts $ to other areas as needed
• Customer validation is “mapped” to IBM validation; eliminate duplications
• IBM provides supportive evidence from detailed test documentation
• Risk-based testing is an effective way to prioritize risks for building validation centered around
high risk areas in your business
• Early risk assessment results in more effective testing
• IBM provides documentation and assistance, including:
• Automated & manual test execution reports
Summary report of test strategy
• Pareto analysis of defects
Traceability matrices
• Mapping analysis for test coverage
 Other benefits :
– Develop best regression testing practices, automation practices
– Directly influence IBM test efforts for Maximo by providing scenarios to include in IBM testing
5
IBM Confidential
© 2013 IBM Corporation
Approach
1. Gather business requirements
2. Categorize requirements
Criticality/Development Method
OOTB
Configured
Customized
Critical
Risk-based
Investigating
Investigating
Non-Critical
Risk-based
Investigating
Investigating
Other
Risk-based
Investigating
Investigating
3. Perform risk based testing approach
• Analyze your requirements/test cases against IBM test cases, proposed
reductions where duplications exist
4. If needed, perform audit IBM quality management system, to support acceptance of
IBM validation and verification data
5. Provide supporting documentation to support elimination of tests
6. Result - develop consistent validation strategy to be used for all release updates
© 2013 IBM Corporation
History and 2013 Hi-lights
•
Focus Areas:
•
Maximo and Calibration; Maximo 7.5 upgrade and later versions of 7.1.x
•
Wide range of business requirements (200 to 2000)
•
Benefits stem mostly from OOTB areas (critical and/or non critical)
•
Concentration in Life Sciences/regulatory sectors
•
Customer references available on request
•
Reductions in validation costs range from 20-82%
•
Provided and helped companies develop ongoing validation best practices
•
2013 Updates:
•
Doubled number of customers as compared to 2012, reductions hold same
•
Based on customer input, updated Fix Pack read-me, to allow required impact
analysis
•
Looking to work with business partners for expansion
•
Interest with Nuclear and Oil and Gas customers
•
Published QPP content on the ISM Library
© 2013 IBM Corporation
Business requirements mapped to IBM results
8
© 2013 IBM Corporation
Business Requirement mapped to IBM test case
Example client business requirements mapped to IBM Test cases
9
© 2013 IBM Corporation
Sample Data – Business Requirement
Business requirement traced to automated test result : System shall allow work order to be
auto generated from a Preventive Maintenance record
10
IBM Confidential
© 2013 IBM Corporation
Mapping Document
11
© 2013 IBM Corporation
Enhanced Fix pack Documentation
12
© 2013 IBM Corporation
Updated Fix pack readme updated to provide list of APARs provided by module vs. by APAR
prior to changes
© 2013 IBM Corporation
Fix pack documentation example
Fix pack test status tracking spreadsheet
14
© 2013 IBM Corporation
Fix Pack Documentation – List of Class File Changes
List of file changes per fix pack
15
© 2013 IBM Corporation
Fix pack documentation – Quality plan summary
Quality plan summary information
16
© 2013 IBM Corporation
Test Results
17
© 2013 IBM Corporation
Sample Data – Automated BVT Test Results
18
IBM Confidential
© 2013 IBM Corporation
Sample Data – Test Execution Summary Report
Test Execution summary reports
19
IBM Confidential
© 2013 IBM Corporation
Sample Data – Look up of specific test case
20
IBM Confidential
© 2013 IBM Corporation
Developing a Risk Based Validation Strategy – Best Practices
 Determine regression “strategy” to be used for all releases
– Articulate exactly what’s been tested/not tested, giving accountability to testing efforts
 Areas to focus:
– Analyzing the most important/critical functionalities: Priority Indicator
– Analyzing the areas where the probability of failure is high: Defect Indicator
– Analyzing the impact of failure of a particular functionality: Severity Indicator
 Analyze fix packs thoughtfully
– Review areas that are impacted; assess risk
– Often can reduce testing
 Include user “exploratory” testing
– All testing in the world does not replace having your own users test
– Include in test planning, set of users testing the release for x man-days
 Include testing for newly configured or customized areas
 Consider Automation
– Speed up validation and more easily trace results to initial risk assessment
– Best used for areas that are repeated in testing
© 2013 IBM Corporation
Discussion Questions
 What are roadblocks to getting onto the latest release cycle?
 Are there challenges in your validation strategy?
 What percentage of your application is OOTB?
 How can you streamline your regression testing?
– What validation takes up the most time?
– Are there areas that are there duplications? Do groups duplicate testing?
 Are there quality risks that you have had in past?
 Do you really need to test all Maximo areas, for every release? Where are the risk areas?
What can you do about the risk areas?
 Automation?
 Performance?
© 2013 IBM Corporation