Risk and Credibility Assessments for Computational Modeling of Medical Devices Tina Morrison, PhD tina.morrison@fda.hhs.gov Advisor of Computational Modeling Office of Device Evaluation, FDA Vice Chair ASME V&V40 Subcommittee Member MDIC CM&S Steering Committee Role of V&V for Computational Models of Medical Devices If computational models are to be increasingly relied upon in the development and evaluation of medical devices, the consistent application of V&V must be applied to establish model credibility. Need to establish … o If the model is correct and credible o Demonstrated predictive capabilities to justify use beyond domain of validation o Predictive confidence is commensurate with model risk ASME Subcommittee on V&V Standards Subcommittee o Provide procedures for assessing and quantifying the accuracy and credibility of computational modeling and simulation V&V Standards Committee in Computational Modeling and Simulation V&V-10 - Verification and Validation in Computational Solid Mechanics V&V-20 - Verification and Validation in Computational Fluid Dynamics and Heat Transfer V&V-30 - Verification and Validation in Computational Simulation of Nuclear System Thermal Fluids Behavior V&V-40 - Verification and Validation in Computational Modeling of Medical Devices Verification & Validation in Computational Modeling of Medical Devices V&V-40 Charter o Provide procedures to standardize verification and validation for computational modeling of medical devices o Charter approved in January 2011 Medical device focus o Regulated industry with limited ability to validate clinically o Want increased emphasis on modeling to support device safety and/or efficacy o Use of modeling is hindered by lack of V&V guidance and expectations within medical device community V&V Standards Committee in Computational Modeling and Simulation V&V-10 - Verification and Validation in Computational Solid Mechanics V&V-20 - Verification and Validation in Computational Fluid Dynamics and Heat Transfer V&V-30 - Verification and Validation in Computational Simulation of Nuclear System Thermal Fluids Behavior V&V-40 - Verification and Validation in Computational Modeling of Medical Devices Guide for Verification and Validation for Computational Models of Medical Devices Regulated industry with limited ability to validate clinically Want increased emphasis on modeling to support device safety and/or efficacy Use of modeling is hindered by lack of V&V guidance and expectations within medical device community Focus of the Guide o Instead of focusing on how to perform V&V (established elsewhere) … o We developed a common V&V framework to standardize definitions, processes, and documentation requirements between industry, researchers, software developers and regulators. Overall V&V Flow Purpose Define COU Assess Model Risk Establish Credibility Requirements Establish Work plan for VV NO If the plan is not achievable, you will need to redefine the scope, purpose and context of use of the CM&S, which will effect model risk, credibility requirements and the work plan. Is the plan achievable? YES Execute predefined M&S and V&V plan NO Is the CM&S Credible for COU? YES Document M&S and VV Plan and Findings Overall V&V Flow Purpose Define COU Assess Model Risk Establish Credibility Requirements Establish Work plan for VV Risk Assessment Matrix NO If the plan is not achievable, you will need to redefine the scope, purpose and context of use of the CM&S, which will effect model risk, credibility requirements and the work plan. Is the plan achievable? YES Execute predefined M&S and V&V plan NO Is the CM&S Credible for COU? YES Document M&S and VV Plan and Findings Risk Assessment Matrix (RAM) Establish Context of Use Model risk assessment o Directs/guides V&V activities o Defines model credibility requirements HIGH INFLUENCE Model Risk: combination of decision influence and consequence Decision Influence: contribution of the model outcome to the decision being made Consequence: impact if the model outcomes prove incorrect MEDIUM LOW CONSEQUENCE Overall V&V Flow Purpose Define COU Assess Model Risk Establish Credibility Requirements Establish Work plan for VV Credibility Assessment Matrix NO If the plan is not achievable, you will need to redefine the scope, purpose and context of use of the CM&S, which will effect model risk, credibility requirements and the work plan. Is the plan achievable? YES Execute predefined M&S and V&V plan NO Is the CM&S Credible for COU? YES Document M&S and VV Plan and Findings Credibility Assessment Matrix (CAM) Credibility Assessment Matrix (CAM) Establish Target Credibility Requirements based on the Context of Use ● ● ● ● ● ● ● ● ● ● ● ● ● Overall V&V Flow Purpose Define COU Assess Model Risk Establish Credibility Requirements Establish Work plan for VV NO If the plan is not achievable, you will need to redefine the scope, purpose and context of use of the CM&S, which will effect model risk, credibility requirements and the work plan. Is the plan achievable? YES Execute predefined M&S and V&V plan NO Is the CM&S Credible for COU? YES Document M&S and VV Plan and Findings Credibility Level Determination √ √ √ √ √ √ √ √ √ √ √ √ √ Example Jeff Bischoff Mehul Dharia Zimmer, Inc. Force on tibial spine of a knee implant Posterior Tibial Spine Force in Deep Flexion May Create Posterior Liftoff Anterior Tibial Spine Force in Hyperextension May Create Anterior Liftoff Locking mechanism between the (metal) tibial tray and (polyethylene) articular surface is intended to prevent disassembly (poly lift-off) of the modular tibial component during activities of daily living Anterior Lift-off Test Context of use of a test for anterior lift-off Verify that the force required for lift-off of the articular surface from the tibial tray for a new design is greater than expected physiological loading, and therefore demonstrate that the new (locking mechanism) design sufficiently mitigates that risk. Contexts of use of a model for anterior lift-off: 1. Determine the size of component within the new design family that has the smallest force required for anterior lift-off, to then be assessed in a physical test relative to a predicate. FEA followed by Physical Test, Comparison to Predicate 2. Verify that the force required for lift-off of the articular surface from the tibial tray for a new design is greater than that required for a clinically successful predicate, and therefore demonstrate that the new (locking mechanism and/or geometry) design sufficiently mitigates that risk. FEA only, Comparison to Predicate 3. 4. Determine the size of a component within the new design family that has the smallest force required for anterior lift-off, to then be assessed in a physical test without reference to predicate device. FEA followed by Physical Test, No Predicate Demonstrate through analysis alone that the worst case size can sustain physiological loading without liftoff, without reference to a predicate device. FEA only, No Predicate Risk Assessment Matrix Model influence Patient consequence LOW: Results from the computational model are a negligible factor in the decision associated with the question being answered. LOW: A poor decision would not adversely affect patient safety or health, but might result in nuisance to the physician or has other negligible impacts. MEDIUM: Results from the computational model are a moderate factor in the decision associated with the question being answered. HIGH: Results from the computational model are a significant factor in the decision associated with the question being answered. MEDIUM: A poor decision would result in minor patient injury and potentially requiring physician intervention or has other moderate impacts. HIGH: A poor decision would result in severe patient injury or death or has other significant impacts. Risk Assessment Matrix Predicate No predicate FEA + testing COU1 COU3 FEA alone COU4 COU2 Risk Assessment Matrix Predicate No predicate FEA + testing COU1 COU3 FEA alone COU4 COU2 COU4 COU3 COU1,2 COU1: Worst case determination COU2: Absolute evaluation CAM – Elements of Computational Models Verification Code (Column B) – 4 o Used commercially available validated FEA software Solution (Column C) – 4 o Mesh convergence study was performed ― Numerical effects are determined to be small on all important quantity of interests at conditions/ geometries directly relevant to the context of use o All inputs and outputs based on independently reputable source CAM – Elements of Computational Models Validation: Computational Model System Configuration (Column D) – 2 o Used mean/nominal geometry (no LMC/MMC) o Major and minor features captured o Two sizes considered Governing Equations (Column E) – 4 o Used nonlinear material (constitutive) model for UHMWPE ― Key physics (press-fit, resistance against force) was captured ― Material model did not need re-calibration/tuning System Properties (Column F) – 1 o Nominal physical properties that are representative of the comparator from literature o Sensitivity analysis on material properties was not performed Boundary Conditions (Column G) – 3 o Load applied through assumed contact patch on spine, rather than directly modeling the femoral component - Representative but simplified BCs with non-quantified effect on QOI CAM – How Well Is The Comparator Understood? Validation: Evidence-Based Comparator System Configuration (Column H) – 3 o Prescribed location o Geometries matched to machine tolerance (production parts) o Signal to noise ratio is high System Properties (Column I) – 3 o Off-the-shelf parts were tested o Environmental effects on the material are known (testing speed was modified, environment was kept the same for both groups: in air). Boundary Conditions (Column J) – 3 o No sensitivity analysis was performed. o Known (recorded) loading (perturbations) was applied and boundary condition variability (e.g. posterior slope) is known. Sample Size (Column K) – 3 o Statistically relevant sample size (n = 5) o Component size, a key parameter for lift-off, variation was considered. CAM – How Appropriate is CM to Comparator? Validation: Model-to-Comparator Discrepancy (Column L) – 4 o Equivalent input parameters, equivalent quantity of interest CAM – How Appropriate is CM to Comparator? Validation: Model-to-Comparator Discrepancy (Column L) – 4 o Equivalent input parameters, equivalent quantity of interest CAM – How Rigorously Are Outputs Compared? Validation: Qualitative or Quantitative Comparison (Column M) – 3 o Quantitative comparison, with single set of input parameters, without predictive accuracy or uncertainties available o No quantitative comparison with broad range of cases CAM – How V&V activities relates to COU? Validation: V&V to COU Applicability (Column N) – 3 o Validation activities embody relevant characteristics of the CoU sufficient overlap between the validation domain and the CoU space) What can we conclude? COU4 COU3 COU1,2 LEVEL VERIFICATION Solution No predicate FEA + testing COU1 COU3 FEA alone COU2 COU4 VALIDATION Computational Model Code Predicate System Governing System Evidence-based Comparator Boundary System System Boundary Configuration Equations Properties Conditions Configuration Properties Conditions 0 1 2 3 4 Discrepancy Comparison Applicability Sample Model-to- Qualitative or Size Comparator Quantitative V&V to COU 3 3 1 2 3 4 4 4 3 3 3 3 4 Overall V&V Flow Purpose Define COU Assess Model Risk Establish Credibility Requirements Establish Work plan for VV NO If the plan is not achievable, you will need to redefine the scope, purpose and context of use of the CM&S, which will effect model risk, credibility requirements and the work plan. Is the plan achievable? YES Execute predefined M&S and V&V plan NO Is the CM&S Credible for COU? YES Document M&S and VV Plan and Findings Public Meeting - FDA/NIH/NSF Workshop on Computer Models and Validation for Medical Devices, June 11-12, 2013 http://www.fda.gov/MedicalDevices/NewsEvents/W orkshopsConferences/ucm346375.htm Additional resources on RAM and CAM For More Information Please Contact: Tina Morrison, PhD tina.morrison@fda.hhs.gov Advisor of Computational Modeling Office of Device Evaluation, FDA OR Michael Liebschner, PhD Liebschner@bcm.edu Pre-ORS Symposium Chair Baylor College of Medicine; Exponent Failure Analysis