Insights from ACE
Prevention: what worked
and what needs to be done
in economic evaluation
Cairns 16th October 2014
Professor Christopher Doran
Overview
• Introduction to economic evaluation
• Overview of ACE Prevention
• Introduction to impact assessment
• 2 Indigenous examples of impact assessment
Context of economic evaluation
• Resources are scarce in relation to needs / demand
• Scarcity forces choices to be made and choices imply a
sacrifice or foregone opportunity
Growth rates: health expenditure versus GDP Australia, year on year
8.0%
Health expenditure: rising faster than national economic
output
6.0%
4.0%
2.0%
0.0%
SOURCE: ABS, AIHW
GDP
Economic evaluation is …
The comparative analysis of alternative courses of action in terms of
both their costs and consequences in order to assist policy decisions”.
Program A
CostA
ConsequencesA
Program B
CostB
ConsequencesB
Choice
Types of economic evaluation
TYPE OF
ANALYSIS
COSTS
CONSEQUENCES
RESULT
Cost Minimisation
Analysis (CMA)
Dollars
Identical in all respects
Least cost
alternative
Cost Effectiveness
Analysis (CEA)
Dollars
Different magnitude of a
common measure, eg Life
years (LY) gained, blood
pressure reduction
Cost per unit of
consequence, eg
cost per LY gained
Cost Utility Analysis Dollars
(CUA)
Single or multiple effects not Cost per unit of
necessarily common.
consequence, eg
Valued as “utility”, eg QALY per QALY/DALY
Cost Benefit
Analysis (CBA)
As for CUA but valued in
money
Dollars
Net $ (P.V.)
Cost/Benefit ratio
Assessing Cost-EffectivenessPrevention
Rob Carter, Theo Vos, Chris Doran,
Alan Lopez, Andrew Wilson, Ian Anderson, Jan
Barendregt, Wayne Hall
Assessing Cost-Effectiveness (ACE)
studies in Australia
•
•
Pilot project in cancer prevention (2000)
ACE–Heart Disease (2000-2003)
•
•
ACE–Mental Health (2001-2004)
•
•
Focus on childhood interventions
ACE-Alcohol (2006-2008)
•
•
•
20 + interventions for depression, schizophrenia, anxiety and
ADHD
ACE-Obesity (2004-2005)
•
•
20 + interventions for prevention of coronary heart disease
Around 10 interventions to reduce harm from alcohol misuse
ACE-Prevention (2005-2009)
ACE-Alcohol Indigenous (2010-2014)
ACE Prevention: methods
•
•
•
•
•
•
•
•
Understand natural history of disease (from burden of
disease study)
Analyse current practice: % receiving intervention(s);
adherence
Efficacy/effectiveness from literature
Impact in routine Australian health services?
Model change in health outcomes (often over a lifetime)
in DALYs
Difference in costs of intervention & cost offsets
Cost-effectiveness ratios in $$/DALY
Mix of most cost-effective interventions
From policy to measurement of benefit
• Two-stage approach adopted in ACE
• First, a measure of health gain in relation to resources
consumed ($ cost per DALY)
 Picks up element of cost, efficacy/effectiveness and efficiency
objectives
• Second, explicitly provide for broader considerations not
in this C/E ratio
 Which we call our ‘2nd stage filters’ (equity; acceptability;
feasibility; size of the problem)
 Plus confidence in evidence base
Presenting cost-effectiveness
• Can put all costs (y-axis) and health effects (x-axis)
on a graph
• Slope of the line represents the economic
attractiveness of an intervention
Slope = CER =
costs
---------------------health effects
• The flatter the slope, the more cost-effective
Topic areas and interventions
Topic
Alcohol
Tobacco
Physical activity
Nutrition
Body mass
Blood pressure/cholesterol
Bone mineral density
Illicit drugs
Cancer
Diabetes
Renal disease
Mental disorders
Cardiovascular disease
Other
Total
Total population
Prevention Treatment
9
8
6
26
9
12
3
2
9
7
2
11
1
18
123
Indigenous
Prevention Treatment
2
5
1
1
2
10
5
6
27
7
4
2
3
19
2
Intervention pathways: ‘Ideal mix’
Alcohol intervention pathway
$100
Current practice
$0
Net lifetime costs (millions AUS$ 2003)
-
20
40
60
80
100
120
140
Lifetime DALYs averted (thousands)
-$100
Volumetric
tax
-$200
-$300
Res. treat. +
naltrexone
-$400
RBT
-$500
30% tax
Drink drive
mass media
Ad bans
-$600
Min. legal drinking age to 21 yrs
Licensing
controls
Brief
intervention
Combining everything in one model
Intervention costs - <$10,000/DALY package
Cost offsets - <$10,000/DALY package
Health gain - <$10,000/DALY package
Billions
$4
$3
1 million healthy
life years
Immediate
cost savings in
blood pressure
& cholesterol
$2
Costs
$4 billion
upfront
investment
80
$1
$0
60
40
20
0
Treatment cost
saved
-$1
2003 2010 2017 2024 2031 2038 2045 2052 2059 2066 2073 2080
Years
-20
DALYs averted
Intervention costs - Current practice
Thousands
Combined impact 43 very cost-effective
prevention measures
ACE Prevention - main findings
Areas amenable to preventive interventions to
reduce size of burden :
Substantial
CVD, diabetes, kidney disease,
tobacco, alcohol, physical inactivity,
salt
Moderate
Mental disorders, drug use,
osteoporosis
Modest to small Obesity (unless regs/tax work), F&V,
cancer screening
ACE Prevention - summary
Pros
•Good engagement with policy makers / Indigenous leaders
•Platform of recent epidemiological data
•Used sophisticated methods
•Attempted to consider equity, acceptability, feasibility
•Very good dissemination and capacity building (eg. PhDs)
Cons
•Very technical – policy makers found it hard to understand, eg. What
is a DALY?
•Focus on health outcomes – for certain risk factors (alcohol) nonhealth very important
•Relied on secondary data of mixed quality
•Modelling considered a black box – not very researcher friendly
ACE Prevention – what next?
• Funding stopped
– limited interest / funding to extend methods in Australia
– Centre for Burden of Disease and Cost-Effectiveness ceased to exist
• Majority of staff left UQ
– Prof Lopez moved to Uni Melbourne
– Prof Theo Vos and A/Prof Lim moved to Uni Washington to work on the
Global Burden of Disease study funded by Bill Gates
– Others now working at World Health Organisation, Oxford University +
other Australian Unis
• I relocated to Hunter Medical Research Institute to focus on
translation research and impact assessment
The imperative for measuring
research impact…
… there is a need to maximise the translation of effective
research outcomes into health policy, programs and
services
The generation and use of high-quality, relevant research
… this need is
central to the
Wills and
McKeon
reviews
evidence will improve health policy and program effectiveness,
achieve better health and help build efficient services.
Wills Review, 2012
In Australia the debate on improving health outcomes has relied
too much on arguments about increasing resources, and not
enough on improving productivity and effectiveness through
microeconomic reform and translation of innovations from
research.
Mckeon Review, 2013
The imperative for measuring
research impact…
• Socially responsible and good for patients
• Policy makers and the community are looking for research
that is likely to provide a positive social return on investment
(SROI)
• Policy is already changing
• Reward research that demonstrates its potential (and
actual) ‘research translation’
• Onus on researchers to demonstrate ‘value for money’
• Further evidence of this shift in policy
• NHMRC (NHMRC Advanced Health Research and Translation Centres)
• ARC (principles of research translation).
Existing work in this field
• Measuring research impact
• Payback method: Buxton& colleagues UK in 1996
• Core domains of benefit, each with metrics: knowledge, research,
political and administrative, health sector and economic. Scores to
represent success in each domain
• Other versions: Canadian Institutes of Health (2005), Research
Impact Framework (2006), Canadian Academy of Health Sciences
Framework (2009)
• Becker list (Washington University School of Medicine) (Last update
2014)
• All include a dimension of economic impact.
• AU Government
• NSW Government Evaluation Framework (2013)
• Cooperative Research Centre (Impact Tool)
Translational research pathway
(From an economic perspective)
COST
COST
Demand for the
research
Program aims
Activity (i.e.
what will the
research do?)
Outcomes (i.e.
what will the
research
‘produce’)
Use of the
outcomes in the
community
BENEFIT
Impact or
benefit (i.e.
how does the
community
benefit from
the research
outcomes )
Example: Family well-being (FWB)
• FWB program focuses on the empowerment and personal
development of Indigenous people through people sharing
their stories, discussing relationships, and identifying goals
for the future.
– Workshops are held with both adults and children to highlight the
various health and social issues experienced by Indigenous
communities and the steps that can be implemented to deal with
these issues.
• HMRI are working with James Cook University to identify
the economic impact of the program of the program
Translational research pathway
(An example from Family Wellbeing - FWB)
COST
Demand for the
research
Demand for the
FWB program is
mostly generated
by Indigenous
groups and
organisations. The
program targets
participants who
have experienced,
or are
experiencing,
distress and/or
trauma in their
lives.
COST
Program aims
to provide lifeskills for
performing
leadership roles
and dealing with
difficulties of life.
skills include :
Leadership; Basic
human needs;
Relationships; Life
journey; Conflict
resolution;
Emotions; Crises;
Beliefs and
attitudes; and,
Sensitivity as a
leader.
Activity (i.e. what
will the research
do?)
Deliver course
content, refine
content to specific
groups as needed,
provide postcourse support.
Use of the outcomes
in the community
Outcomes (i.e.
what will the
research ‘produce’)
empowerment;
better lifestyle
choices; ability to
cope with conflict;
better engagement
with family
RFDS, the Yaba Bimbie
Men’s Group,
Yarrabah, Gindaja
Treatment and
Healing Service,
Yarrabah,
Ma’Ddaimba-Balas
Mens’s Group,
Innisfail and
Queensland
Department of Com
BENEFIT
Impact or
benefit:
Reduced risks
for disease &
injury, better
health (cost
avoided),
reduced
violence (cost
avoided),
higher
wellbeing
(GEM),
increased
capacity
Translational research pathway
(An example from Family Wellbeing - FWB)
COST
Demand for the
research
Demand for the
FWB program is
mostly generated
by Indigenous
groups and
organisations. The
program targets
participants who
have experienced,
or are
experiencing,
distress and/or
trauma in their
lives.
COST
Program aims
to provide lifeskills for
performing
leadership roles
and dealing with
difficulties of life.
skills include :
Leadership; Basic
human needs;
Relationships; Life
journey; Conflict
resolution;
Emotions; Crises;
Beliefs and
attitudes; and,
Sensitivity as a
leader.
Activity (i.e. what
will the research
do?)
Deliver course
content, refine
content to specific
groups as needed,
provide postcourse support.
Use of the outcomes
in the community
Outcomes (i.e.
what will the
research ‘produce’)
empowerment;
better lifestyle
choices; ability to
cope with conflict;
better engagement
with family
RFDS, the Yaba Bimbie
Men’s Group,
Yarrabah, Gindaja
Treatment and
Healing Service,
Yarrabah,
Ma’Ddaimba-Balas
Mens’s Group,
Innisfail and
Queensland
Department of Com
BENEFIT
Impact or
benefit:
Reduced risks
for disease &
injury, better
health (cost
avoided),
reduced
violence (cost
avoided),
higher
wellbeing
(GEM),
increased
capacity
Translational research pathway
(An example from Family Wellbeing - FWB)
COST
Demand for the
research
Demand for the
FWB program is
mostly generated
by Indigenous
groups and
organisations. The
program targets
participants who
have experienced,
or are
experiencing,
distress and/or
trauma in their
lives.
COST
Program aims
to provide lifeskills for
performing
leadership roles
and dealing with
difficulties of life.
skills include :
Leadership; Basic
human needs;
Relationships; Life
journey; Conflict
resolution;
Emotions; Crises;
Beliefs and
attitudes; and,
Sensitivity as a
leader.
Use of the outcomes
in the community
Activity (i.e. what
will the research
do?)
Refine content to
specific groups as
needed, provide
post-course
support.
Outcomes (i.e.
what will the
research ‘produce’)
empowerment;
better lifestyle
choices; ability to
cope with conflict;
better engagement
with family
RFDS, the Yaba Bimbie
Men’s Group,
Yarrabah, Gindaja
Treatment and
Healing Service,
Yarrabah,
Ma’Ddaimba-Balas
Mens’s Group,
Innisfail and
Queensland
Department of Com
BENEFIT
Impact or
benefit:
Reduced risks
for disease &
injury, better
health (cost
avoided),
reduced
violence (cost
avoided),
higher
wellbeing
(GEM),
increased
capacity
Translational research pathway
(An example from Family Wellbeing - FWB)
COST
Demand for the
research
Demand for the
FWB program is
mostly generated
by Indigenous
groups and
organisations. The
program targets
participants who
have experienced,
or are
experiencing,
distress and/or
trauma in their
lives.
COST
Program aims
to provide lifeskills for
performing
leadership roles
and dealing with
difficulties of life.
skills include :
Leadership; Basic
human needs;
Relationships; Life
journey; Conflict
resolution;
Emotions; Crises;
Beliefs and
attitudes; and,
Sensitivity as a
leader.
Use of the outcomes
in the community
Activity (i.e. what
will the research
do?)
Refine content to
specific groups as
needed, provide
post-course
support.
Outcomes (i.e.
what will the
research ‘produce’)
empowerment;
better lifestyle
choices; ability to
cope with conflict;
better engagement
with family
RFDS, the Yaba Bimbie
Men’s Group,
Yarrabah, Gindaja
Treatment and
Healing Service,
Yarrabah,
Ma’Ddaimba-Balas
Mens’s Group,
Innisfail and
Queensland
Department of Com
BENEFIT
Impact or
benefit:
Reduced risks
for disease &
injury, better
health (cost
avoided),
reduced
violence (cost
avoided),
higher
wellbeing
(GEM),
increased
capacity
Translational research pathway
(An example from Family Wellbeing - FWB)
COST
Demand for the
research
Demand for the
FWB program is
mostly generated
by Indigenous
groups and
organisations. The
program targets
participants who
have experienced,
or are
experiencing,
distress and/or
trauma in their
lives.
COST
Program aims
to provide lifeskills for
performing
leadership roles
and dealing with
difficulties of life.
skills include :
Leadership; Basic
human needs;
Relationships; Life
journey; Conflict
resolution;
Emotions; Crises;
Beliefs and
attitudes; and,
Sensitivity as a
leader.
Activity (i.e. what
will the research
do?)
Refine content to
specific groups as
needed, provide
post-course
support.
Outcomes (i.e.
what will the
research ‘produce’)
empowerment;
better lifestyle
choices; ability to
cope with conflict;
better engagement
with family
Use of the outcomes
in the community
RFDS, Gindaja
Treatment and
Healing Service,
Yarrabah etc. Is there
a cost to deliver this
program?
BENEFIT
Impact or
benefit:
Reduced risks
for disease &
injury, better
health (cost
avoided),
reduced
violence (cost
avoided),
higher
wellbeing
(GEM),
increased
capacity
Translational research pathway
(An example from Family Wellbeing - FWB)
COST
Demand for the
research
Demand for the
FWB program is
mostly generated
by Indigenous
groups and
organisations. The
program targets
participants who
have experienced,
or are
experiencing,
distress and/or
trauma in their
lives.
COST
Program aims
to provide lifeskills for
performing
leadership roles
and dealing with
difficulties of life.
skills include :
Leadership; Basic
human needs;
Relationships; Life
journey; Conflict
resolution;
Emotions; Crises;
Beliefs and
attitudes; and,
Sensitivity as a
leader.
Activity (i.e. what
will the research
do?)
Refine content to
specific groups as
needed, provide
post-course
support.
Outcomes (i.e.
what will the
research ‘produce’)
empowerment;
better lifestyle
choices; ability to
cope with conflict;
better engagement
with family
Use of the outcomes
in the community
RFDS, Gindaja
Treatment and
Healing Service,
Yarrabah etc. Is there
a cost to deliver this
program?
BENEFIT
Impact or
benefit:
Reduced risks
for disease &
injury, better
health (cost
avoided),
reduced
violence (cost
avoided),
higher
wellbeing
(GEM),
increased
capacity
DEMAND
DEMAND:
E.G. A need
to enhance
wellbeing
COST
Funding to
address the
"need"
AIMS & ACTIVITIES
Develop
custom FWB
program to
address the
specific "need"
Deliver custom
FWB program
OUTCOMES
Empower
individuals
through lifeskill training
IMPACT
Better lifestyle
choices: self
management
of health, diet,
exercise,
smoking
Reduced risk
factors for
disease &
injury
Better able to
cope with
conflict
Reduction of stress; reduction in domestic violence
better quality of life (wellbeing)
Better
engagement
with family,
community,
and others
Improved
quality of life
(wellbeing)
Development of
personal
responsibility
and improved
leadership skills
Better able to
lead self /
familiy /
community
Better health outcomes;
improved life
expectancy better
quality of life / wellbeing
Improved wellbeing for
self and others
Learning
Increase capacity of self / community and
organisation
Employment
Increased income; sense of control,
wellbeing
Developing a framework to evaluate the
impact of Family well-being?
• Our framework includes a specific FWB survey that enables
us to collect information pertinent to assessing impact
• If the evaluation is conducted as a prospective exercise (rather than
retrospective – as occurs in most cases) it can also provide ongoing
feedback to researchers / service providers on performance.
• In this way it can act as a component/facilitator of continuous quality
improvement
• We acknowledge some problems but we are working on this
• The GEM is an appropriate measure of wellbeing
• Currently cannot convert changes in wellbeing scores to $ values
• Some international work is suggesting that wellbeing be included
in all cost benefit analyses and the UK Treasury have published a
paper on their attempt to convert wellbeing into $ values
• Our aim is to advance this research in Australia
Example: Institute of Urban Indigenous Health
• The Institute of Urban Indigenous Health (IUIH) was
established in July 2009 as a strategic response to the
growth and geographic dispersion of the Aboriginal and
Torres Strait Islander population in South East Queensland
(SEQ) which accounts for 38% of Queensland’s, and 10%
of Australia’s total Aboriginal and Torres Strait Islander
population.
• The role of the IUIH is to lead health service planning,
develop and co-ordinate health service delivery, and to play
a major role in the development of partnerships between
health care providers
• The IUIH activities are diverse, multifaceted and lead to a range of
outputs. The impacts of these activities are closely aligned with
the strategic goals of the IUIH: to improve access to
comprehensive primary health care; to develop an effective and
culturally aware workforce; to build sustainable partnerships; and
to contribute to building the evidence base. While some of these
COST
ACTIVITY
OUTPUT / OUTCOME
IMPACT / BENEFIT
Provision of
appropriate care to
manage chronic
conditions
Service delivery (health
assessment, chronic care plan
and review)
Service delivery
IUIH budget
Preventative health
Workforce
development
Preventative health (deadly
choices, tobacco action,
community events,
educational programs)
Workforce development
(student placements, staff
recruitment, work it out)
Sustainable partnerships
(new clinics, improved
networking, better
governance)
Building evidence base (data
management services, CQI, ehealth)
Better educated
towards chronic
diseases
Reduced risk factors for
disease & injury
Reduced complications /
fewer hospitalisations /
saving to health care system
Better health outcomes;
improved life expectancy
better quality of life /
wellbeing
Better lifestyle choices: diet,
exercise, smoking
Building cultually
aware workforce
Building sustainable
partnerships
Building the evidence
base
More effective /
productive workforce
Greenfield and
brownfield clinics;
improved governance
Data management
services
Contribution to Indigenous
health and social policy
Example: Institute of Urban Indigenous Health
• The IUIH activities are diverse, multifaceted and lead to a range of
outputs.
• The impacts of these activities are closely aligned with the
strategic goals of the IUIH: to improve access to comprehensive
primary health care; to develop an effective and culturally aware
workforce; to build sustainable partnerships; and to contribute to
building the evidence base.
• While some of these impacts may lead to quantifiable economic
return, others are more difficult to quantify.
• The next logical step for the Institute is to unpack the rich clinical
data that it collects to demonstrate longitudinal improvements in
patient and community outcomes from the range of Institute
activities.
• A better understanding of this clinical data would facilitate a more
comprehensive assessment of the economic benefit of the IUIH
and a better understanding of the IUIH contribution to closing the
gap
Summary
• In an environment of limited resources it is important to
evaluate what we do
• Economics provide a framework to identify value for money
ranging from cost-effectiveness to cost-benefit analysis
• Cost-effectiveness is appropriate when comparing health
programs
• Cost benefit is appropriate when examining return on
investment or conducting an impact assessment
• Policy makers are increasingly requiring evaluations that
make sense – what is the return on the investment?
• Good evaluation requires good data, plausible assumptions
and a robust methodology
Thank you
Chris.doran@hmri.com.au