Risk and Credibility
Assessments for Computational
Modeling of Medical Devices
Tina Morrison, PhD
tina.morrison@fda.hhs.gov
Advisor of Computational Modeling
Office of Device Evaluation, FDA
Vice Chair
ASME V&V40 Subcommittee
Member
MDIC CM&S Steering Committee
Role of V&V for Computational Models of
Medical Devices
 If computational models are to be increasingly relied upon
in the development and evaluation of medical devices, the
consistent application of V&V must be applied to establish
model credibility.
 Need to establish …
o If the model is correct and credible
o Demonstrated predictive capabilities to justify use beyond
domain of validation
o Predictive confidence is commensurate with model risk
ASME Subcommittee on V&V
 Standards Subcommittee
o Provide procedures for assessing and
quantifying the accuracy and credibility of
computational modeling and simulation
V&V Standards Committee in
Computational Modeling and
Simulation
V&V-10 - Verification and
Validation in Computational
Solid Mechanics
V&V-20 - Verification and
Validation in Computational
Fluid Dynamics and Heat
Transfer
V&V-30 - Verification and
Validation in Computational
Simulation of Nuclear System
Thermal Fluids Behavior
V&V-40 - Verification and
Validation in Computational
Modeling of Medical Devices
Verification & Validation in
Computational Modeling of Medical Devices
 V&V-40 Charter
o Provide procedures to standardize
verification and validation for
computational modeling of medical
devices
o Charter approved in January 2011
 Medical device focus
o Regulated industry with limited ability to
validate clinically
o Want increased emphasis on modeling to
support device safety and/or efficacy
o Use of modeling is hindered by lack of
V&V guidance and expectations within
medical device community
V&V Standards Committee in
Computational Modeling and
Simulation
V&V-10 - Verification and
Validation in Computational
Solid Mechanics
V&V-20 - Verification and
Validation in Computational
Fluid Dynamics and Heat
Transfer
V&V-30 - Verification and
Validation in Computational
Simulation of Nuclear System
Thermal Fluids Behavior
V&V-40 - Verification and
Validation in Computational
Modeling of Medical Devices
Guide for Verification and Validation
for Computational Models of Medical Devices
 Regulated industry with limited ability to validate clinically
 Want increased emphasis on modeling to support device safety
and/or efficacy
 Use of modeling is hindered by lack of V&V guidance and
expectations within medical device community
 Focus of the Guide
o Instead of focusing on how to perform V&V (established
elsewhere) …
o We developed a common V&V framework to standardize
definitions, processes, and documentation requirements
between industry, researchers, software developers and
regulators.
Overall V&V Flow
Purpose
Define
COU
Assess
Model
Risk
Establish
Credibility
Requirements
Establish
Work plan
for VV
NO
If the plan is not achievable, you will need
to redefine the scope, purpose and context
of use of the CM&S, which will effect model
risk, credibility requirements and the work
plan.
Is the plan
achievable?
YES
Execute
predefined
M&S and
V&V plan
NO
Is the
CM&S
Credible for
COU?
YES
Document M&S
and VV Plan and
Findings
Overall V&V Flow
Purpose
Define
COU
Assess
Model
Risk
Establish
Credibility
Requirements
Establish
Work plan
for VV
Risk Assessment Matrix
NO
If the plan is not achievable, you will need
to redefine the scope, purpose and context
of use of the CM&S, which will effect model
risk, credibility requirements and the work
plan.
Is the plan
achievable?
YES
Execute
predefined
M&S and
V&V plan
NO
Is the
CM&S
Credible for
COU?
YES
Document M&S
and VV Plan and
Findings
Risk Assessment Matrix (RAM)
 Establish Context of Use
 Model risk assessment
o Directs/guides V&V activities
o Defines model credibility requirements
HIGH
INFLUENCE
 Model Risk: combination of decision
influence and consequence
 Decision Influence: contribution of
the model outcome to the decision
being made
 Consequence: impact if the model
outcomes prove incorrect
MEDIUM
LOW
CONSEQUENCE
Overall V&V Flow
Purpose
Define
COU
Assess
Model
Risk
Establish
Credibility
Requirements
Establish
Work plan
for VV
Credibility Assessment Matrix
NO
If the plan is not achievable, you will need
to redefine the scope, purpose and context
of use of the CM&S, which will effect model
risk, credibility requirements and the work
plan.
Is the plan
achievable?
YES
Execute
predefined
M&S and
V&V plan
NO
Is the
CM&S
Credible for
COU?
YES
Document M&S
and VV Plan and
Findings
Credibility Assessment Matrix (CAM)
Credibility Assessment Matrix (CAM)
Establish Target Credibility Requirements
based on the Context of Use
●
●
●
●
●
●
●
●
●
●
●
●
●
Overall V&V Flow
Purpose
Define
COU
Assess
Model
Risk
Establish
Credibility
Requirements
Establish
Work plan
for VV
NO
If the plan is not achievable, you will need
to redefine the scope, purpose and context
of use of the CM&S, which will effect model
risk, credibility requirements and the work
plan.
Is the plan
achievable?
YES
Execute
predefined
M&S and
V&V plan
NO
Is the
CM&S
Credible for
COU?
YES
Document M&S
and VV Plan and
Findings
Credibility Level Determination
√
√
√
√
√
√
√
√
√
√
√
√
√
Example
Jeff Bischoff
Mehul Dharia
Zimmer, Inc.
Force on tibial spine of a knee implant
Posterior Tibial Spine Force in Deep Flexion
May Create Posterior Liftoff
Anterior Tibial Spine Force
in Hyperextension
May Create Anterior Liftoff
Locking mechanism between the
(metal) tibial tray and
(polyethylene) articular surface is
intended to prevent disassembly
(poly lift-off) of the modular tibial
component during activities of
daily living
Anterior Lift-off Test
Context of use of a test for anterior lift-off
Verify that the force required for lift-off of the articular
surface from the tibial tray for a new design is greater than
expected physiological loading, and therefore demonstrate
that the new (locking mechanism) design sufficiently
mitigates that risk.
Contexts of use of a model for anterior lift-off:
1.
Determine the size of component within the new design family that has
the smallest force required for anterior lift-off, to then be assessed in a
physical test relative to a predicate.
FEA followed by Physical Test, Comparison to Predicate
2.
Verify that the force required for lift-off of the articular surface from the
tibial tray for a new design is greater than that required for a clinically
successful predicate, and therefore demonstrate that the new (locking
mechanism and/or geometry) design sufficiently mitigates that risk.
FEA only, Comparison to Predicate
3.
4.
Determine the size of a component within the new design family that
has the smallest force required for anterior lift-off, to then be assessed
in a physical test without reference to predicate device.
FEA followed by Physical Test, No Predicate
Demonstrate through analysis alone that the worst case size can sustain
physiological loading without liftoff, without reference to a predicate device.
FEA only, No Predicate
Risk Assessment Matrix
Model influence
Patient consequence
LOW: Results from the computational model are
a negligible factor in the decision associated with
the question being answered.
LOW: A poor decision would not adversely affect
patient safety or health, but might result in
nuisance to the physician or has other negligible
impacts.
MEDIUM: Results from the computational model
are a moderate factor in the decision associated
with the question being answered.
HIGH: Results from the computational model are
a significant factor in the decision associated with
the question being answered.
MEDIUM: A poor decision would result in minor
patient injury and potentially requiring physician
intervention or has other moderate impacts.
HIGH: A poor decision would result in severe
patient injury or death or has other significant
impacts.
Risk Assessment Matrix
Predicate
No predicate
FEA + testing COU1
COU3
FEA alone
COU4
COU2
Risk Assessment Matrix
Predicate
No predicate
FEA + testing COU1
COU3
FEA alone
COU4
COU2
COU4
COU3
COU1,2
COU1: Worst case
determination
COU2: Absolute evaluation
CAM – Elements of Computational Models
Verification
 Code (Column B) – 4
o Used commercially available validated FEA software
 Solution (Column C) – 4
o Mesh convergence study was performed
― Numerical effects are determined to be small on all important quantity
of interests at conditions/ geometries directly relevant to the context of
use
o All inputs and outputs based on independently reputable source
CAM – Elements of Computational Models
Validation: Computational Model
 System Configuration (Column D) – 2
o Used mean/nominal geometry (no LMC/MMC)
o Major and minor features captured
o Two sizes considered
 Governing Equations (Column E) – 4
o Used nonlinear material (constitutive) model for UHMWPE
― Key physics (press-fit, resistance against force) was captured
― Material model did not need re-calibration/tuning
 System Properties (Column F) – 1
o Nominal physical properties that are representative of the comparator from literature
o Sensitivity analysis on material properties was not performed
 Boundary Conditions (Column G) – 3
o Load applied through assumed contact patch on spine, rather than directly modeling
the femoral component - Representative but simplified BCs with non-quantified effect
on QOI
CAM – How Well Is The Comparator Understood?
Validation: Evidence-Based Comparator
 System Configuration (Column H) – 3
o Prescribed location
o Geometries matched to machine tolerance (production parts)
o Signal to noise ratio is high
 System Properties (Column I) – 3
o Off-the-shelf parts were tested
o Environmental effects on the material are known (testing speed was modified,
environment was kept the same for both groups: in air).
 Boundary Conditions (Column J) – 3
o No sensitivity analysis was performed.
o Known (recorded) loading (perturbations) was applied and boundary condition
variability (e.g. posterior slope) is known.
 Sample Size (Column K) – 3
o Statistically relevant sample size (n = 5)
o Component size, a key parameter for lift-off, variation was considered.
CAM – How Appropriate is CM to Comparator?
Validation: Model-to-Comparator
 Discrepancy (Column L) – 4
o Equivalent input parameters, equivalent quantity of interest
CAM – How Appropriate is CM to Comparator?
Validation: Model-to-Comparator
 Discrepancy (Column L) – 4
o Equivalent input parameters, equivalent quantity of interest
CAM – How Rigorously Are Outputs Compared?
Validation: Qualitative or Quantitative
 Comparison (Column M) – 3
o Quantitative comparison, with single set of input parameters, without
predictive accuracy or uncertainties available
o No quantitative comparison with broad range of cases
CAM – How V&V activities relates to COU?
Validation: V&V to COU
 Applicability (Column N) – 3
o Validation activities embody relevant characteristics of the CoU sufficient
overlap between the validation domain and the CoU space)
What can we conclude?
COU4
COU3
COU1,2
LEVEL
VERIFICATION
Solution
No predicate
FEA + testing
COU1
COU3
FEA alone
COU2
COU4
VALIDATION
Computational Model
Code
Predicate
System
Governing
System
Evidence-based Comparator
Boundary
System
System
Boundary
Configuration Equations Properties Conditions Configuration Properties Conditions
0
1
2
3
4
Discrepancy Comparison Applicability
Sample
Model-to-
Qualitative or
Size
Comparator
Quantitative
V&V to COU
3
3
1
2
3
4
4
4
3
3
3
3
4
Overall V&V Flow
Purpose
Define
COU
Assess
Model
Risk
Establish
Credibility
Requirements
Establish
Work plan
for VV
NO
If the plan is not achievable, you will need
to redefine the scope, purpose and context
of use of the CM&S, which will effect model
risk, credibility requirements and the work
plan.
Is the plan
achievable?
YES
Execute
predefined
M&S and
V&V plan
NO
Is the
CM&S
Credible for
COU?
YES
Document M&S
and VV Plan and
Findings
Public Meeting - FDA/NIH/NSF
Workshop on Computer Models
and Validation for Medical
Devices, June 11-12, 2013
http://www.fda.gov/MedicalDevices/NewsEvents/W
orkshopsConferences/ucm346375.htm
Additional resources on RAM and CAM
For More Information Please Contact:
 Tina Morrison, PhD tina.morrison@fda.hhs.gov
 Advisor of Computational Modeling
 Office of Device Evaluation, FDA
 OR
 Michael Liebschner, PhD
Liebschner@bcm.edu
 Pre-ORS Symposium Chair
 Baylor College of Medicine; Exponent Failure Analysis