Evidence and Value: Impact on DEcisionMaking (EVIDEM ) – applying
multicriteria decision analysis (MCDA) to support healthcare decisionmaking
1
PhD,
1
PhD,
1
PhD,
Mireille M Goetghebeur
Monika Wagner
Hanane Khoury
1
1,2
1
Randy Levitt PhD, Lonny J Erickson PhD, Donna Rindress PhD
1BioMedCom
Consultants inc, Montreal, Quebec Canada 2Centre Hospitalier Universitaire de Montreal – McGill University Hospital center
(CHUM-MUHC) Technology Assessment Unit, Montreal Québec Canada
T
I
M
E
Background
Decision
•
Healthcare decisionmaking is a complex process that requires simultaneous
integration of numerous disparate types of information.1
Population based and individual decisionmaking can be divided into two
steps: scientific judgment to evaluate the quality of evidence and value
judgment about the healthcare intervention.2,3
There is a need, nationally and internationally, for transparent access to
evidence and values on which healthcare decisions are made.4-11
Multi-Criteria Decision Analysis (MCDA) is an established method widely
used in numerous disciplines. It structures the decisionmaking process by
breaking down the problem into the set of criteria which are expected to
impact the value of an option.1,12 It has been applied within several areas of
healthcare decisionmaking13-16 and presents a promising approach to
reimbursement decisionmaking.
We hypothesized that healthcare decisionmaking could be facilitated by
structuring the evidence and value judgments on which it is based into a
practical and transparent architecture and that transparency would enhance
understanding and implementation of healthcare decisions.
•
•
•
•
Extrinsic value
Decisionmakers
Intrinsic value of treatment
Quality Matrix
Criteria of quality
International standards
Decisionmaking body requirements
Key info for each component of
Value Matrix
Fully referenced
Assessing
&
structuring
evidence
Investigators
& experts
Evidence
Full text source documents (published, registries, proprietary)
Development
of evidence
Researchers
& manufacturers
MCDA Value Matrix
•
Assesses the intrinsic value of a healthcare intervention from a specific perspective; it
is applicable by decisionmakers at micro (patients, clinicians), meso (payers,
institutions) and macro (healthcare policymakers) levels.
Components defining intrinsic value were identified through literature review, review of
explicit criteria used by decisionmaking bodies, and also criteria used to prioritize
technology assessment in numerous jurisdictions globally.17,18
15 components were selected to fulfill MCDA methodological criteria of completeness,
redundancy, operationality & mutual independence1,12 and grouped into 4 clusters.
Cluster
Components of value assessment (alphabetical by cluster)
Weight
Synthesized available evidence*
Score
Evaluator comments
Value
Q1
Adherence to requirements of decisionmaking body
Q2
Completeness and consistency of evidence
Q3
Relevance and validity of evidence
Quality Matrix
D1
Disease severity (risk of death & disability, acuteness)
D2
Size of population affected by disease
I1
Current clinical guidelines on intervention or similar alternatives
I2
Current interventions limitations
I3
Improvement of medical service – efficacy
I4
Improvement of medical service – safety & tolerability
I5
Improvement of medical service – PROs, convenience & adherence
I6
Public health interest (prevention & risk reduction)
I7
Type of medical service (symptom relief, prolonging life, cure etc.)
W I3
SI3
Structured summary of evidence
for intervention
E1
Budget impact of reimbursing intervention on health plan
E2
Cost-effectiveness of intervention
E3
Impact on healthcare spending excluding intervention cost
Validity of efficacy trial
Number of patients
Low
Etc…
Improvement over existing
treatment
Efficacy
Small
None
Etc…
Lower
 Based on international standards
 Comprehensive analyses can be
systematized
 Not highly dependent on evaluator
perspective
•
•
•
How does this component,
relative to other components,
impact what an intervention
would bring/add to the health
of the society?
High priority?
Higher
 Similar value system
 Requires value judgment
 Dependant on evaluator
perspective
No shared value system
Dependant on
region/institution priorities
A practical iceberg architecture was designed to support decisionmaking
using an MCDA approach to structure intrinsic value components (Value
Matrix - VM) while providing transparent access to evidence
(synthesized and full text) and quality of evidence (Quality Matrix - QM).
The architecture was built to facilitate communication between data
producers and data users
Integrated procedures were developed to ensure efficient use of
resources as this architecture is developed & applied to healthcare
intervention assessment.
www.evidem.org
Impact of intervention in
therapy
Expected role of intervention, impact on healthcare system, place of
intervention in therapy
4
Epidemiology
Epidemiology of disease treated by intervention (prevalence, incidence,
trends, sub-population etc, as applicable) and risk factors
5
Intervention characteristics
Indication, technical characteristics (e.g., for drugs, pharmacology,
pharmacokinetics, interactions, contra-indications, warnings, dosing and
administration & concomitant therapies)
6
Clinical data
Efficacy and safety data from clinical trials (published, unpublished, metaanalyses, reviews) and from documents submitted to regulatory bodies
7
Patient reported outcomes
Patient reported outcomes for intervention including quality of life,
satisfaction, convenience etc
8
Effectiveness data
Effectiveness data (naturalistic/real life trials, effectiveness estimates,
registry requirements, etc)
9
Comparator data
Data on efficacy, safety, patient reported outcomes, intervention
characteristics for comparators, where applicable
10
Price information/justification
Price of intervention and comparators; price justification
11
Economic evaluation
Economic evaluation; impact of intervention on healthcare utilization and
associated costs; impact on society and associated cost, where applicable
Budget impact
Collaboration axis
Explicit data needs
Define explicit needs of decisionmakers
Establish reasonable requirements
Analyze
requirements
Provide score &
rationale
Complete 11dimension
instrument
Provide score &
rationale
Complete 11dimension
instrument
Provide score &
rationale
Budget impact analyses if intervention reimbursed (impact on health
plan/drug plan)
Application axis
Retrospective
Application axis
Prospective
Validate process in
various jurisdictions
Explore context of
past decisions
Generate data on
quality of evidence
Adapt framework to
existing
decisionmaking
processes
Basis to develop
extrinsic value
components
1
2
3
4
5
 Low impact on value



 High impact on value
Synthesized available evidence
Score
STRUCTURED SUMMARY
Types of trial/Meta-analysis
 Intervention & comparators
 Inclusion/exclusion criteria
 Patient characteristics
 Patient disposition
 Type of analysis
Intervention
Scale anchors:
0  Lower efficacy
1 
2 
3  Major improvement in efficacy
Comparator 1
Primary outcome
Other outcomes
Subgroup analysis
Population of patients eligible for intervention:
Etc.
Value
MCDA linear model
VI3 = (W I3 / ∑Wn) x SI3
*Requirements established by the decisionmaking body

Definition of effectiveness trial based on the AHRQ criteria
Collaboration axis
Research planning
•
Planning tool for researchers and
intervention developers to meet explicit
needs
Develop methodology to generate data
tailored to critical data needs
4-step assessment:
• Literature review for all types of evidence for the healthcare intervention;
•Analysis of available evidence (public and manufacturer);
•Analysis of requirements of decisionmaking body to which manufacturer dossier is
submitted; and
•Completing instruments, scoring and providing rationale for each QM cell.
•
•The feasibility and value of EVIDEM was assessed by applying it to historical cases:
10 medicines (A-J) were assessed (cardiovascular disease, endocrinology, infectious
disease, neurology, oncology, ophthalmology) using data from literature review and
manufacturer dossiers submitted to the Canadian Common Drug Review and Québec
Conseil du Médicament. QM scoring was performed by EVIDEM investigators. VM
weights, scores and feedback on process were provided by the Canadian Value
Panel, composed of representative stakeholders from across Canada
(decisionmakers, specialists, generalists, nurses, pharmacists, health
economists/epidemiologists).
2: Some improvement in efficacy
3: Major improvement in efficacy, larger eligible
population
Value of tested medicines
on the VM scale
Proven safe and efficacious
intervention to cure endemic severe
disease resulting in major healthcare
savings (e.g. malaria in Africa)
100%
2-step assessment:
•Weighting of VM components from the societal perspective, independent of scoring
healthcare intervention; and
•Scoring of healthcare intervention using scale anchors & scoring guidelines
combined with access to synthesized data (prepared using standardized
methodology), quality of evidence scores & rationales (QM) and full text sources.
Synthesized evidence in the VM is a summarized HTA report
tailored to decisionmakers questions; it is generated on a
collaborative and on-going basis to provide timely answers
75%
J
50%
I
B
F
D G H
A C
E
Intervention for a rare disease with 25%
minimal improvement in efficacy
and major safety issues, resulting
in major increases in healthcare
spending
0%
Interpretation of the VM scale
(% maximum score)
• Was the approach feasible? An algorithm was
developed to operationalize each cell of the matrices and
this was applicable to therapeutic areas and jurisdictions
covered
• Was it practical? It took 30 min on average to apply VM
by stakeholders and about 250 hrs to build the structured
package of fully traceable information
• Value of EVIDEM according to panelists? structuring
evidence, assessing strengths and weaknesses
systematically, sharing between stakeholders – should
not be used as a formulaic approach
• Limitations of pilot? Limited access to the underlying
source data (e-icebergs in development) and extrinsic
components to be explored
Future developments include collaborative studies and iterative
processes to explore the value of EVIDEM in context as well as
development of an interactive electronic platform integrating evidence
and value for each healthcare intervention.
The expected outcome of a systematized and shareable approach for
data access and value assessment is to optimize resources, decisions
and health.
Acknowledgements
•
We wish to acknowledge the contributions of the Canadian Value Panel: Jean-François Bussières
BPharm MSc MBA FCSHP, Director, Pharmaceutical Practice Research Unit, Department of
Pharmacy, CHU Sainte-Justine Research Center; Benoit Cossette BPharm MSc, Pharmacist and
Coordinator, Drug Disease Management Program, Fleurimont Hospital, Sherbrooke University
Hospital Centre; Doug Coyle PhD, Associate Professor, Faculty of Medicine, Epidemiology and
Community Medicine, University of Ottawa; Cheri Deal MD PhD FRCPC, Associate Professor and
Program Director, Endocrine Service, CHU Sainte-Justine Research Center, University of Montréal;
Roland Grad MD CM MSc CCFP FCFP, Associate Professor, Department of Family Medicine, McGill
University; Christine Lee MD, FRCPC, Assistant Professor and Director, Microbiology Residency
Program, Department of Pathology and Molecular Medicine, McMaster University; Mitchell Levine MD
FRCPC FISPE, Professor and Director, Centre for Evaluation of Medicines, Department of Clinical
Epidemiology & Biostatistics, Department of Medicine, McMaster University; Diane Lowden RN MSc
(A), Clinical Nurse Specialist, Montreal Neurological Institute & Hospital, McGill University; G.B.John
Mancini MD FRCP, Professor and Program Director, Continuing Medical Education, Division of
Cardiology, Department of Medicine, University of British Columbia; Paul Oh MD FRCPC, Medical
Director of the Cardiac Rehab and Secondary Prevention Program, Department of Medicine, Clinical
Pharmacology and Toxicology, University of Toronto; Genevieve Tousignant N MScN, Clinical Nurse
Specialist, Montreal Neurological Institute & Hospital, McGill University; Wendy Ungar PhD, Associate
Professor, Department of Health Policy, Management & Evaluation, University of Toronto, and Senior
Scientist, Child Health Evaluative Sciences, The Hospital for Sick Children; Marie-Claude Vanier
BPharm MSc, Associate Clinical Professor, Faculty of Pharmacy, University of Montréal, Family
Medicine Clinic, Cité de la Santé de Laval.
•
This study was made possible by an unrestricted research grant from Pfizer Canada.
Where ∑W n equals
sum of weights
Scoring guidelines: Example (for information only)
0: Lower efficacy than comparators
1: Same efficacy as comparators
EVIDEM
framework
Aggregated quality score (% of maximum score)
Vulnerable vs productive
patients
Large
Major
3
•
•
Aggregated value (% of maximum value )
Type of population treated
(equity)
Low priority?
High
Treatment patterns & current practices, treatment guidelines
Q3: Relevance &
validity score
Proof of concept: Pilot study in Canadian context
Total
Weight
Extrinsic
value
of product
Treatment patterns
Q2: Completeness &
consistency score
•
VI3
Economics
Decisionmaking process
Intrinsic
value
of product
2
•
PRO: patient reported outcomes
*From public domain and manufacturer
Quality of
of
evidence
Disease description, disease progression/duration, pathophysiology,
clinical presentation, severity of disease etc
Q1: Adherence to
requirements* score
•
EVIDEM supports decisionmaking by structuring, segregating and
providing transparent access to data, and by allowing communication of
value judgments among stakeholders.
It can be applied retrospectively to generate data on quality of evidence
or on past decisions and prospectively to integrate tools into existing
decisionmaking processes and explore extrinsic components
(Application axis).
On the collaboration axis, EVIDEM provides a practical framework to
facilitate communication between those who generate data and those
who need it to make decisions, facilitating future healthcare
decisionmaking.
It can be used for any type of healthcare intervention, and by policy or
clinical decisionmakers.
Quality of evidence
Intervention
Value judgment
Disease information
Total
A conceptual framework was developed that segregated components of
decisionmaking into quality of evidence and two types of value
judgment; intrinsic value of the healthcare intervention and extrinsic or
system-related value.
It was hypothesized that segregating these concepts would make
reasoning more explicit, increase transparency and facilitate complex
healthcare decisionmaking.
Quantifiable components that shared commonly agreed direction of
scoring were organized into matrices.
Scientific judgment
Definition
1
12
Conceptual framework and architecture
•
Assess the quality of all types of evidence for a healthcare intervention;
12 QM components (rows) identified from literature review and requirements from
more than 20 decisionmaking bodies worldwide17 organized by field of research;
3 QM criteria of quality (columns) relating to: a) scientific quality (relevance and
validity of evidence available); b) quality of reporting (completeness and consistency
of evidence in high level documents and in individual studies), and c) adherence to
decisionmaking body requirements;
Questions and instruments for QM cells derived from international scientific standards
(e.g., CONSORT, CHEC, STROBE, Siegel et al, Mauskopf et al).19-23
Matrix component
Disease impact
•
•
•
Breakdown components of healthcare decisionmaking into practical tools
that structure and quantify assessment of healthcare interventions
Build an architecture to provide timely access to transparent multiple layers
of the components of decisionmaking
Ultimately, optimize healthcare and health through best use of healthcare
interventions
•
•
•
Quality of evidence
Synthesized evidence
•
•
Potential applications & developments
•
Objectives
•
Decisionmaking
body
assessment
Value Matrix
Multi Criteria Decision Analysis (MCDA)
•
•
Clinical
or policy
decision
MCDA Quality Matrix
References
1. Baltussen R, Niessen L. Priority setting of health interventions: the need for multi-criteria decision analysis. Cost Eff Resour Alloc. 2006;4:14. 2. Eddy DM. Clinical decision making: from theory to practice. Anatomy of a decision. JAMA. 1990;263(3):441-3. 3. Tunis SR. Reflections on science, judgment, and value in evidence-based decision making: a conversation with David Eddy. Health Aff (Millwood ). 2007;26(4):w500-w515. 4. Dhalla I, Laupacis A. Moving from opacity to transparency in pharmaceutical policy. CMAJ. 2008;178(4):428-31. 5.
Establishing transparency to restore trust in clinical trials. Lancet Neurol. 2006;5(7):551. 6. Rovira J. Transparency of economic evaluations of health technologies. Pharmacoeconomics. 2008;26(3):181-3. 7. Menon D, Stafinski T, Stuart G. Access to drugs for cancer: Does where you live matter? Can J Public Health. 2005;96(6):454-8. 8. Morgan SG, McMahon M, Mitton C, Roughead E, Kirk R, Kanavos P, et al. Centralized drug review processes in Australia, Canada, New Zealand, and the United kingdom. Health Aff (Millwood ). 2006;25(2):337-47.
9. Anis AH, Guh D, Wang X. A dog's breakfast: prescription drug coverage varies widely across Canada. Med Care. 2001;39(4):315-26. 10. Gregoire JP, MacNeil P, Skilton K, Moisan J, Menon D, Jacobs P, et al. Inter-provincial variation in government drug formularies. Can J Public Health. 2001;92(4):307-12. 11. Teng F, Mitton C, Mackenzie J. Priority setting in the provincial health services authority: survey of key decision makers. BMC Health Serv Res. 2007;7:84. 12. National Economic Research Associates. Multi-criteria analysis manual. On
Communities and Local Government website [updated 2000; cited 2007 Apr 11]. Available from: http://www.communities.gov.uk/pub/252/MulticriteriaanalysismanualPDF1380Kb_id1142252.pdf. 13. Le Gales C, Moatti JP. Searching for consensus through multi-criteria decision analysis. Assessment of screening strategies for hemoglobinopathies in southeastern France. Int J Technol Assess Health Care. 1990;6(3):430-49. 14. Mussen F, Salek S, Walker S. A quantitative approach to benefit-risk assessment of medicines-part 1: the development of a new
model using multi-criteria decision analysis; part 2: the practical application of a new model. Pharmacoepidemiol Drug Saf. 2007;S42-S46. 15. Baltussen R, Stolk E, Chisholm D, Aikins M. Towards a multi-criteria approach for priority setting: an application to Ghana. Health Econ. 2006;15(7):689-96. 16. Nobre FF, Trotta LT, Gomes LF. Multi-criteria decision making--an approach to setting priorities in health care. Stat Med. 1999;18(23):3345-54. 17. Biomedcom Consultants Inc. BioMedCom Interface database. On Biomedcom Consultants Inc website
[updated 2007; cited 2007 Sep 9]. Available from: http://www.biomedcom.org/en/resources-BMC-MADB.html. 18. Noorani HZ, Husereau DR, Boudreau R, Skidmore B. Priority setting for health technology assessments: a systematic review of current practical approaches. Int J Technol Assess Health Care. 2007;23(3):310-5. 19. Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA. 2001;285(15):1987-91. 20. Evers S, Goossens M, de Vet H,
van Tulder M, Ament A. Criteria list for assessment of methodological quality of economic evaluations: Consensus on Health Economic Criteria. Int J Technol Assess Health Care. 2005;21(2):240-5. 21. The STROBE group. The STROBE statement: checklist of essential items. On STROBE Statement website [updated 2005 Sep; cited 2006 Jun 15]. Available from: http://www.strobe-statement.org. 22. Siegel JE, Weinstein MC, Russell LB, Gold MR. Recommendations for reporting cost-effectiveness analyses: Panel on cost-effectiveness in health and
medicine. JAMA. 1996;276(16):1339-41. 23. Mauskopf JA, Sullivan SD, Annemans L. Principles of good practice for budget impact analysis. On ISPOR website [updated 2007; Available from: http://www.ispor.org/workpaper/budget_impact.asp.