OHTAC decision determinants:
systematic review
Ann-Sylvia Brooker, MSc, PhD
Joanna Bielecki, MISt
Murray Krahn, MD, MSc
April 14, 2015
Saskatoon, SK
Research Objective and
Research Questions:
The objective of the literature review is to examine the decisionmaking methodologies applicable to the health technology
appraisal process (decision-making process).
Research Questions :
• ***What decision criteria are used during the decision-making
process?
• What methods are used to evaluate the decision criteria (e.g.
assigned weights, ranked, rated)?
• What decision-making methods are used to integrate these
criteria in order to develop recommendations regarding
funding decisions concerning health technologies (e.g. are
decision rules used?).
Inclusion Criteria
• Information from national and international health
technology organizations and insurance agencies
making policy recommendations or funding coverage
decisions concerning the technology.
• Published research describing the decision-making
methods of HTA organizations.
• What is
• What should be
• Also – criteria for resource allocation
Exclusion Criteria
• Primary research evaluating the effectiveness of
specific aspects of the decision-making process.
• Theoretical/methodological papers that discussed
only one decision criteria (e.g. HTA and ethics) 
These articles were forwarded for consideration to
the relevant OHTAC sub committees .
• Information regarding HTA agency operations.
• Information regarding sources of evidence.
• Literature that described the details of evaluating a
technology.
Literature Search
• Search in published academic
literature; between February 2007 and
March 2013.
• Reference list of published articles.
• HTA agency website search.
• An email was sent to every member of
INA-HTA re: decision criteria or
decision-making processes.
Results
• 1479 abstracts were scrutinized from
academic literature.
• 18 members of INAHTA responded to
the email. The response rate was 32%
(18/56). 4/18 (22%) did not have
relevant documentation in English or
French. Another 5/18 (28%) responded
that their agency had an advisory role
only. However, 4 organizations had
relevant documentation.
•
Results
• 26 documents are included in this
review.
• Documents from US, Canada, Alberta,
Ghana, Germany, Australia, South
Africa, Singapore, Chile, UK, USA,
Netherlands, Brazil, Scotland, New
Zealand, England. (English speaking)
Author/Year
Title of
Publication
Duclos et al., 2012
Developing
evidence-based
immunization
recommendations
and GRADE
Goetghebeur et al.,
2012
Bridging Health
Technology
Assessment
(HTA) and Efficient
Health Care Decision
Making with Multicriteria Decision
Analysis (MCDA):
Applying the EVIDEM
Framework to
Medicines Appraisal
Decision Criteria
-Epidemiologic features of
the Disease
-Clinical characteristics
of the targeted disease
-Vaccine and
immunization
characteristics
-Economic considerations
-Health-system
considerations
-Social impacts
-Legal considerations
-Ethical considerations
-Disease Impact
-Context of Intervention
-Intervention outcomes
-Type of benefit
-Economics
-Quality of Evidence
Evaluation of Criteria
For effectiveness: GRADE
(however, risk of bias
checklist from Cochrane)
For economic & costeffective ness: WHO
For adverse effects:
Cochrane
An MCDA estimate for the
technology was calculated by
combining scores and weights
for each criterion.
Decision-Making
Method
The committee makes
recommendations by
consensus.
The weighting exercise is
not designed to be
prescriptive. The
framework is designed to
support health care
decision making by
stimulating reflection and
exchange and making the
thinking process more
explicit.
Author/Year
Title of Publication
Decision Criteria
Evaluation of Criteria
Decision-Making
Method
Husereau et al.,
2010
Priority setting for
health technology
assessment at
CADTH
-Alternatives
-Budget impact
-Clinical impact
-Disease burden
-Economic impact
-Available Evidence
Weights are given as follows:
-Alternatives (.081)
-Budget impact (.143)
-Clinical impact (.258)
-Disease burden (.216)
-Economic impact (.167)
-Available Evidence (.135)
Background report
prepared by 2
researchers provides
scores to these criteria.
But, final decision is
made by a committee.
Jehu-Appiah et al.,
2008
Balancing equity
and efficiency in
health priorities in
Ghana: the use of
multi-criteria
decision analysis
-number of potential
beneficiairies
-severity of disease
-cost-effectiveness
-poverty reduction
-vulnerable population
But also social/ethical
acceptability
Intervention complexity
Ranking of criteria as follows
(in descending order of
importance):
1.
Vulnerable populations
They suggest a
framework that
combines quantitative
and non-quantitative
analytical criteria (e.g.
ethical/social
responsibility and
intervention
complexity) and that
final approval should
occur after a number of
elaborations by
different advisory
panels
2.
Cost-effectiveness
3.
Severity of disease
4.
Number of potential
beneficiaries
5.
Poverty reduction
Additional “nonquantifiable” criteria:
-ethical/social acceptability
-intervention complexity
Author/Year
Kreis et al.,
2011
Lopert, 2009
Title of
Publication
From evidence
assessments to
coverage
decisions? The
case example
of glinides in
Germany
Evidence-Based
Decision-Making
within
Australia’s
Pharmaceutical
Benefits Scheme
Decision Criteria
Evaluation of Criteria
-needs
-costs
-safety and
effectiveness
-services are:
adequate, expedient,
cost-effective and, do
not exceed what is
necessary.
-Efficacy
-Safety
-Quality
-Cost-effectiveness
-Clinical need
-Uncertainty in costeffectiveness
-Total cost to PBS
-Ability to constrict a
restriction
-Potential for adverse
outcomes
-Affordability of drug
-Rule of Rescue
A drug can be excluded
based on inexpediency.
First, if the on the basis
of relevant studies, a
drug is inferior to a
comparable therapy
option. Or, second,
studies demonstrating a
benefit are lacking.
No fixed weight for
these factors. The
factors will be of
greater or lesser
importance in different
situations.
Decision-Making
Method
A specific decision
making method
were not found in
this article.
An expert
committee
deliberates on the
evidence and
provides its
recommendation to
the Minister. The
Minister may veto a
positive
recommendation
HTA agency
Country
PHARMAC
New Zealand
Decision
Criteria
(a) the health
needs
(b) the
particular
health needs of
Maori and
Pacific peoples;
(c) the
availability and
suitability of
existing
medicines,
and related
products and
related things;
(d) clinical
benefits/risks;
(e) costeffectiveness
(f) the
budgetary
(g) the direct
cost to health
service users;
(h) the
Government’s
priorities for
health funding,
(i) other criteria
as PHARMAC
thinks fit.
Evaluation of
Criteria
Pharmac gives
weight to each
criterion as
PHARMAC
considers
appropriate.
Decision-making method
PHARMAC makes decision after reviewing
evidence and consulting with public, groups,
that may be affected by its proposals, and with
expert advisory committees.
HTA agency
Country
Decision Criteria
NICE
England
-ICER
-certainty of the
ICER estimate
-certainty of the
health related
quality of life
measure
-benefits beyond
those captured in
the QALY measure
-whether
technology is a
“life-extending
treatment at the
end of life”
-aspects that
relate to nonhealth objectives
of the NHS (e.g.
whether a
substantial
proportion of
savings/ benefits
occur outside the
NHS and PSS.)
Evaluation of
Criteria
The appraisal
committee uses
different decision
rules depending
on whether the
ICER is (1) less
that £20,000 per
QALY gained (2) in
the range of
£20,000 to
£30,000 per QALY
gained, and (3)
above a most
plausible ICER of
£30,000 per QALY
gained.
Decision-making method
The Appraisal Committee does not
use a precise maximum acceptable
ICER above which a technology would
automatically be defined as not cost
effective or below which it would.
But a stronger case must be made for
an ICER (2) in the range of £20,000 to
£30,000 per QALY gained and even
more for an (3) ICER above a most
plausible ICER of £30,000 per QALY
gained.
Summary of Decision Criteria
Decision Criteria
Efficacy- Potential
benefit of the
intervention
(mortality, morbidity,
PRO)
Terms
Health benefit, potential health gain in terms of
mortality (saving life, life expectancy gains, average lifeyear benefit per patient, prolongation of disease-free
survival); morbidity (health benefit, enhanced health
outcome, relative advantage, incremental health gain);
patient-reported outcomes (quality of life, number of
*****
QALYs gained per patient, disability adjusted life years,
relative value to patient). Overall gain in quality of care.
Health benefits relative to current standard therapy.
Safety of the
Side (adverse) effects, unintended consequences, safety
intervention
and tolerability, risks, risk management, harm, risk of
***
event, risk of toxicity compared with standard therapy.
External impact of
Impact on patient’s family, possible harms to others,
intervention
infectious disease involved, population effect (positive or
*
negative), herd immunity, public health interest, social
impact, social benefit, prevention of ill health,
prevention.
Need (clinical)
Treatment alternatives, comparative intervention
limitations (unmet needs), availability of alternative
***
treatments, availability of effective alternative
treatments, availability of preventative measures, clinical
need, emergencies and need.
Disease determinants Factors responsible for the persistence of the burden.
Disease burdenclinical
**
Prevalence of disease, incidence of disease, number of
patients, severity of disease, impact of disease/condition
on quality of life, number of potential beneficiaries,
indirect beneficiaries,
Decision criteria - Terms
Decision Criteria Terms
Quality of
evidence (re:
effectiveness
research)
***
Relevance of
evidence/genera
lizability/
effectiveness in
real practice
*
Availability of evidence, strength of evidence, consistency
of findings, quality of data, choice of end points, validity
of data, certainty, precision of effect, selection of
studies, proof, scientific evidence, time of assessment in
technology development, therapy mechanism of action.
Relevance of evidence, representativeness of patients
(studies vs. real world), representativeness of technology
user (e.g. skill of surgeon or health care practitioner in
studies vs. real world), representativeness of context
(e.g. acute vs long term care; country differences),
response rate, patient compliance, level of
generalization, effectiveness in real practice, evidence of
effectiveness.
Decision Criteria
Ethics and moral
issues
***
Vulnerable and
needy
populations
Decision
Terms
criteria - Terms
Consistency with societal values, moral
consequences of HTA, ethical implications, rule of
rescue.
Vulnerable populations (e.g. age, gender,
geography, ethnicity, indigenous populations), life
extending treatment for end of life, social groups
with high risk and/or increased vulnerability, age of
**
targeted group, population equity, positive poverty
reduction.
Human dignity
Human integrity and dignity, basic human rights,
meets patient’s basic needs.
Patient autonomy Patient autonomy, patient preference. (e.g.
and patient
patient-centered healthcare? Is there patient &
preference **
public involvement?)
Equity, fairness
Equity, fairness, health equity, equality,
and justice
distributive justice, formal justice, procedural
*
justice, social justice, addressing health status
inequalities at population level, geographical
equity, equity of access, timeliness of access.
Utility
Utility, utilitarianism.
Solidarity
Solidarity, collectivism, cohesion.
Cultural aspects
Cultural and religious convictions.
Decision Criteria
Terms
Disease burden-cost
**
Cost to treat disease, cost to prevent disease, national cost of
the disease/condition to the health care system.
Opportunity costs
Efficiency / value for
money for patient.
Opportunity costs to the population.
Maximizing impact on health for a given level of resource
compared to available alternatives for this patient group (e.g.
cost-effectiveness, cost-utility, cost per QALY, cost-effectiveness
utility curves, cost consequence analysis.). Could include
comparisons of interventions with different objectives (e.g.
psycho-therapy vs. pain meds).
*****
Quality of evidence
(re: efficiency & cost
estimates)
*
Cost per patient.
Financial/budget
impact-costs of
intervention *
Financial impactsavings of
intervention
Costs (benefits) of
externalities
Uncertainty in QALYs, possible benefit/harms not included in the
QALY (i.e. non-health benefits, social benefits)
Cost per patient, unit cost.
Budget impact, affordability, operating and start-up costs,
national medical costs per year, financial impact on government.
Cost-savings, national savings in terms of costs of absences per
year, savings in terms of medical costs.
Costs of externalities such as: impact on patient’s family,
possible harms to others, infectious disease involved, population
effect (positive or negative), public health interest, social
impact, social benefit, prevention of ill health, prevention.
Decision Criteria
Priorities:
national, local
level etc.
Stakeholder
interests and
pressures;
political aspects.
Decision criteria - Terms
Terms
National priorities, local priorities (does it meet a
local health need? public expectations?),
international priorities, strategic direction.
Advocacy, pressure from patient groups, pressure
from physician groups, producer interests,
recommendations made by other countries, clinical
expert opinions; political pressure, political impact.
Decision Criteria
Terms
Feasibility (at the
organizational
level and at the
system level)
System requirements, physical environment, system
capacity, local capacity, ability to implement,
implementation, organization’s structure,
organizational burden, logistics, process, wellorganized, feasibility of delivery; all enablers and
barriers to diffusion within the health system
infrastructure (operational, capital, human resources,
legislative, regulatory) including ease of integration
into local community, system integration, acceptability.
Flexibility, reversibility, revisability, ability to evaluate,
provision to revision.
***
Flexibility of
implementation *
Ensuring adequate
quality and
sustainability of
intervention
Appropriate use of intervention, appropriateness,
appropriate setting/level of service, sustainability,
longevity.