Establishing the case for prevention in public health: QALYs, Cost benefit
analysis, Social Return on Investment, multiple criteria decision analysis and the
kitchen sink
Edwards RT, Lloyd-Williams H, McIntosh E, Jones CL, Ferguson B, Baughan S.
Abstract
Background
The way public health is organised and delivered varies across the UK. In England,
much of the public health function has been devolved to local government, whilst this
function remains within the NHS in Wales and Scotland. NICE, the Department of
Health in England, Public Health England, the King’s Fund, Public Health Wales,
NHS Health Scotland and NHS Healthcare Improvement Scotland are all currently
exploring technical methods for the evaluation of the case for prevention, producing
case studies to guide future resource allocation.
Objective
From a theoretical and pragmatic policy perspective, to review the types of economic
appraisal that might be relevant in particular circumstances, drawing upon illustrative
case studies in the area of prevention. The strengths and weaknesses of different
approaches are explored.
Methods
We present a review of prioritisation frameworks and approaches to the economic
evaluation of public health interventions, spanning QALYs, Social Return on
Investment, and multiple criteria decision analysis.
Results
A review of current methods for the evaluation of public health interventions and their
strengths and weaknesses in particular circumstances. Presentation of case studies of
economic evaluations of public health interventions designed to illustrate theoretical,
operational and policy support issues.
Conclusion
At present there is a need for a range of methods to be applied to the evaluation of
public health interventions, and a definite need for some standardisation of techniques,
including wider use of thresholds, as is current practice in health technology
assessment. We would welcome discussion of these issues at HESG in Glasgow.
1
Background
Public health is the science and art of promoting and protecting health and well being,
preventing ill-health and prolonging life through the organised efforts of society
(Faculty of Public Health 2014). This paper considers current NICE guidance on the
evaluation of public health preventive interventions, considers the reason why
markets for preventative goods and services fail, and argues the case for government
investment in health promoting activities. We review a range of evaluative methods
currently in use, which expand the traditional health economics toolbox and provide
case studies.
The way public health is organised and delivered varies across the UK. In England,
much of the public health function has been devolved to local government, whilst this
function remains within the NHS in Wales, Scotland and Northern Ireland. NICE, the
Department of Health in England, Public Health England, the King’s Fund, Public
Health Wales, NHS Health Scotland and NHS Healthcare Improvement Scotland are
all currently exploring technical methods for the evaluation of the case for prevention.
There are an increasing number of case studies being published to demonstrate the use
of these methods (Buck and Gregory, 2013).
Economic evaluation alongside trials of Public Health interventions are often more
expensive due to the need for large sample sizes (Fischer et al, 2012). However this
depends on the study design and the complexity and resources required in the
identification, measurement and valuation of all costs and outcomes across all
affected sectors. Finally, economic models of Public health interventions are complex
(Squires, 2013).
To some extent, all these technical methods consider, and attempt to make explicit,
the concept of opportunity cost, that is, to identify the benefits foregone from
investing in one activity and not another. Resources are finite and should be used in a
way that maximises the health gain for the population, the implicit goal of NICE, or
another such societal objective such as reducing health inequalities, a central tenet of
public health. Techniques such as multiple-criteria decision analysis (MCDA), social
return on investment (ROI/SROI) and the consideration of capabilities as an outcome
2
measure, are being added to the traditional economic evaluation toolbox with
reference to the evaluation of public health interventions.
MCDA in particular is a potentially useful tool when decision makers are faced with
conflicting criteria and the complexity of balancing health gain and reducing
inequalities can be addressed using this method. Health economists are looking to
fields of environmental economics, transport economics and economic evaluation by
central government e.g. the UK Treasury (Edwards et al, 2013) that commonly use
broader measures of evaluation such as cost-benefit analysis and SROI. These
methods can use information from randomised trials (or in public health, often cluster
randomised trials), or use the results of longer term modelling of the costs and
outcomes from investment in prevention. Concepts such as ‘rule of rescue’ (an ethical
imperative to save individual lives even when money might be more efficiently spent
to prevent deaths in the larger population (Doughety, 1993, quoted in McKie and
Richardson, 2003)) means that prevention with its often-distant time horizon,
struggles to gain an equitable share of resources.
NICE guidance on the Economic Evaluation of Public Health interventions
Since the responsibility for many public health programmes has been devolved to
local authorities in England the approach that is now taken by NICE in their latest
guidance (NICE 2012) is to broaden its approach to appraisal. The tools of economic
evaluation must reflect the fact that local authorities are now responsible for the
overall ‘well-being’ of its citizens and decision making that is much more
decentralised. There is a wider remit than just health and a greater local element
which has led NICE to advocate more emphasis be placed on cost consequence and
cost benefit analysis rather than cost-effectiveness and cost-utility analysis. This is
outlined by NICE in their technical guidance for new drugs and health technologies.
A scoping study commissioned by NICE has also explored the potential of SROI as a
method of evaluation in public health (NICE 2011).
Public Health and preventative Strategies
In his seminal book, Geoffrey Rose distinguished between population strategies and
high risk strategies to prevention (Rose, 1992). Population strategies aim to achieve a
small change in behaviour across the whole population (e.g. salt consumption); high
3
risk strategies aim to change the behaviour of those in the extreme tails of a
population behaviour distribution (e.g. drug or alcohol misuse).
Getting the public health message across is increasingly part of the role of GPs. GPs
are now under increasing pressure to provide their patients with up to date
information, and proactive brief intervention lifestyle-changing advice that may
prevent ill-health in future. In 1990 a contract was introduced for GPs that offered
payments for achieving certain public health prevention targets (such as cervical
screening and immunisation). This role was strengthened with the introduction of
incentive payments to GPs through the Quality and Outcomes framework (QOF) in
2004.
It is estimated that only about 4% of NHS spending is on prevention (NHS 2013). In
terms of shifting this balance in resource allocation, Hale et al (2012) argue that
public health interventions in general, and those providing preventive measures
specifically, are expected to demonstrate that they are cost saving as well as than
merely cost-effective at the NICE thresholds. That is, in order to justify investment of
resources, they are expected to use less resources and be more effective than
treatment as usual. Shurcke et al (2007) argue that prevention should not be judged
against a higher criterion than medical care. The question often asked is whether
preventive interventions can prevent admissions to hospital in the short and medium
term. The answer to this is often ‘no’.
Market Failure for Prevention goods
Hale et al (2012) base their economic approach to prevention on an idea of market
failure. Governments, it is said in economic theory, should intervene in the supply of
goods and services when there is some form of market failure in their provision. This
leads to an outcome that is not Pareto optimal, that is, one can make someone better
off without making at least one other person worse off. In the case of prevention,
three reasons for market failure can occur (Hale et al 2012). The first is time
inconsistent preferences. This is where individuals prefer instant gratification over
future long-term interests. This is such that any commitment to behave in a certain
way now will not be honoured in the future when the time comes. The second source
4
of market failure is when people do not act rationally, in accordance with the
assumptions of micro-economic theory. This, it is argued is especially true in children
and young people where as economic agents they are not making decisions to
maximise their utility over the long-term but seek behaviour that maximises utility
over the short term and may damage their health. There is an argument that there is an
economic case for prevention to prevent them from harming themselves. The third
type of market failure comes in the form of imperfect information. Information on
healthy living is a public good and therefore expected to be under supplied by the free
market. A public good displays aspects of non-rivalry in consumption and nonexcludability. Non-rivalry means that once a good is consumed this does not diminish
the amount left for others to consume – this is true of information on health harming
consumer goods. Non-excludability means that one cannot exclude anyone from
consuming the good – also true of information. There is therefore a case for
government intervention in the provision and production of health information. We
see therefore that by recourse to economic theory there is a strong case for prevention
within public health based on the possibility that provision by the free market will be
sub-optimal and lead to inefficient outcomes.
This market failure in preventive activity, and the reasons behind it, may lead to the
need to consider different forms of evaluation on behalf of the government or other
payer for preventive services. Within the NHS, NICE provides evidence of relative
cost-effectiveness of new drugs and technologies using a threshold of £20,000 £30,000, which may be revised downwards given recent evidence (Claxton 2013).
Owen et al 2011, using a cost-utility analysis framework, review and present evidence
that many public health prevention initiatives fall well below this threshold and some
are cost-saving (Owen et al, 2011). Acknowledgement that QALYs may not capture
all benefits derived from public health interventions, and that many such interventions
may have wide reaching costs and outcomes has led to interest in methods to identify,
measure and value the full and broad range of costs and outcomes relevant to the
evaluation of public health interventions (Weatherly et al, 2009; Payne et al,2012;
Edwards et al, 2013).
Many public health interventions have the potential to have an impact on health,
housing, transport, education and the environment. Central and local government
5
departments have not traditionally used a framework of cost-per QALY - using
concepts of “a business case”, or “the return on investment”. It is this kind of
evidence that local government in England is now asking for, in order to guide
commissioning and investment decisions (Buck et al, 2013).
The key question in this paper is therefore: How do alternative methods of economic
evaluation, within and outwith the health economists’ toolbox help to address
questions about the cost-effectiveness of investment in prevention? The following
section considers some approaches to both formal evaluation and those that simply
facilitate organisation of costs and benefits within a public health context.
NOISE Analysis
NOISE analysis is a variant of SWOT Analysis. NOISE1 analysis allows researchers
or teams within an organisation to set out explicitly the Need, Opportunity,
Improvements, Strengths and Exceptions of various options. We apply this in Table 1
to a range of methods of economic evaluation that are currently being applied to the
evaluation of Public Health prevention interventions.
NOISE analysis begins by looking across all techniques of analysis and highlights the
need by service commissioners for evidence of the cost effectiveness of prevention
initiatives. Table 2 provides additional information on the theoretical basis of these
methods of evaluation and signposts the analyst to useful websites. In terms of
opportunities we see that funding streams such as the PHR enables more trials and
cohort studies in public health. There is also guidance from NICE on the application
of economic evaluation of public health interventions. All techniques could be
improved by extending their application to more examples, and thus increasing the
evidence base. Applying these evaluation techniques leads to a more robust, evidencebased allocation of resources. It can be argued that an exception to the need for
economic evaluation is in the cases of low cost interventions with strong evidence of
effectiveness- this still needs to be documented as part of evidence based policy.
1
http://create-learning.com/article/manager-training/noise-analysis-an-alternative-toswot-strategic-planning
6
Cost-effectiveness Analysis
Cost-effectiveness Analysis (CEA) –This method is useful when a clear unidimensional metric of effectiveness across a Public Health prevention programme is
known e.g. number of quitters per year from a smoking prevention programme. It
does not allow for comparison across prevention programmes tackling different health
problems across different populations.
Cost Utility Analysis (CUA)
Cost Utility Analysis (CUA) – In a single metric the QALY brings together two of the
most important features of an intervention, that is, the length of life extended by the
intervention and the quality of that life. It is a utility based measure focusing as it does
on the weights assigned to particular health states which come from utility theory.
The main advantage of using QALYs is that CUA can be used to compare across as
well as within Public Health prevention programmes. Some proponents of the QALY
approach argue that one of its strengths is the potential to compare across different
types of interventions. Owen 2011 delivers a powerful message that of the 200 Public
Health interventions reviewed, 85 percent had a cost-per QALY ratio well below the
current NICE threshold.
Case study 1 is based on an example from Dalziel et al (2006) looking at the cost
utility of physical activity counselling in general practice in New Zealand and
Australia. It is noted that physical activity acts as a preventative measure to reduce the
risk of a number of ailments including coronary heart disease, stroke, high blood
pressure, non-insulin diabetes, osteoporosis and obesity. In addition, the costs of
disorders associated with physical inactivity are high - $24billion or 2.4% of total
healthcare expenditure in the US in 1995. It is generally postulated that preventive
interventions that aim to increase physical activity will improve quality of life, reduce
mortality and generate savings in downstream costs of care. It can be said that a main
drawback with the QALY approach is where the incremental benefit is very small it
yields a very large incremental cost-effectiveness ratio (ICER). In this situation, along
with when ICERs are negative, a net benefit approach is preferable. This approach
aims to give a monetary value to QALYs by multiplying them with different values
for willingness to pay thresholds and then uses regression to regress net benefit on the
7
treatment variable thus giving an incremental net benefit that does not depend on
small changes in QALYs.
The authors chose the cost utility framework to assess whether a ‘Green Prescription’
intervention was cost effective compared to usual care as this intervention had not
previously been evaluated. A cost utility method was chosen and their reasoning for
using QALYs was that it allowed comparison with other approaches to prevention
and management and so policy makers are able to make better decisions on resource
allocation. This is because the QALY combines the impact on quality of life and
mortality into a single measure. The main outcomes used to inform the economic
model were changes in the proportion of people who became physically active
(defined as having at least two and a half hours of activity in a week) and change in
quality of life over 12 months. Impact on mortality was gleaned from the literature on
the relationship between physical activity and mortality. They found a 90% likelihood
that the program is more cost effective than usual care when the threshold is set at NZ
$7,500 per QALY. The cost per QALY gained was $NZ 2,053 which falls well below
the generally accepted funding thresholds. This intervention, therefore, can be seen to
be highly cost effective. Further a Markov model was employed to extrapolate quality
of life beyond the 12 month duration of the trial. Individuals were allocated into one
of three Markov states: active, inactive and dead. The transition probabilities between
each state at each cycle were derived from the published literature. The results were
found to be consistent with other studies thus improving the authors’ confidence in
the results. The main limitation of the study was the short follow up period (12
months) which meant that modelling had to be used to extrapolate results beyond the
time frame of the trial. Another one of the main limitations of the study was that they
used SF-36 data to calculate quality of life. The authors admit that utility data based
on directly collected utility scores (such as EQ5D) would have been the preferred
approach. However, it can be seen that the decision to use cost utility through QALYs
was very apt in this case.
Cost Benefit Analysis
Cost benefit analysis can be seen as the most comprehensive and theoretically robust
approach to economic evaluation (Robinson 1993). It allows the researcher to
estimate the total cost and total benefit of an intervention and see whether total
8
benefits exceed total costs and so whether society experiences a net gain from the
intervention in question. It differs from cost effectiveness and cost utility analysis in
that it attempts to put a monetary valuation on benefits. It does this through a number
of avenues including the human capital approach and contingent valuation methods.
The human capital approach was the more traditional approach, has a long history in
economics and is based on the idea that people’s working lives yield a rate of return
similar to that of physical capital. In terms of valuing health,
“…the benefits of health care can be measured in terms of the future flow of
income that would otherwise have been forgone because of ill health”
(Robinson 1993, p. 924) in McIntosh, 2010.
The contingent valuation (CV) method is a stated preference approach designed to
directly estimate welfare gains/losses as appropriate.
Individuals are asked to
consider a hypothetical scenario where they are asked to imagine that a market exists
for the benefits or losses of a public programme. The exercise proceeds on the
hypothetical contingency that such a market exists. Various design instruments can
then be applied to ask individuals to state their willingness to pay (WTP) to ensure a
welfare gain occurs or willingness to accept (WTA) to tolerate the welfare loss from
the programme. The WTP or the WTA amount is then taken as a measure of the
individual perceived value of the programme (i.e. the demand), which is then
aggregated across all individuals (McIntosh et al 2010).
There have been criticisms of both these methods. First, the human capital approach
can be seen by some critics as unethical as it attempts to give a monetary value on
human life. There may therefore be some distrust from decision makers about the
method and uneasiness in applying its results. However, it must be noted that decision
makers should be made aware of the fact that economists are not valuing human life
per se but rather placing a monetary value on the costs of extending human life
beyond a certain point. Another criticism is that the approach, focusing as it does on
rates of pay, is not able to capture the health benefits that accrue to those not in
employment.
9
The contingent valuation method also has a large number of methodological
drawbacks mainly related to the various types of bias that arise when designing stated
preference questions. For a comprehensive summary of such biases, see Mitchell and
Carson, 1989.
The following case study is based on work by Nichol (2001) who expored the costbenefit of vaccination for influenza among 18-64 year olds in the US. The objective
of the study is to assess the economic implications of vaccinating this group on an
annual basis. According to data from the National Health Interview Survey, influenza
was responsible for 100 million days of bed disability, 75 million days of absenteeism
and 25 million health visitor visits for this group in 1995. It is therefore apparent that
there are potentially large savings to be made from a programme of vaccination. Here
a CBA was carried out from a societal perspective allowing the researcher to look at
wider costs and benefits. It measured the net cost or savings associated with the use of
influenza virus vaccine in healthy working adults compared with no vaccination. This
is not a traditional cost-benefit study whereby the authors attempt to put a monetary
value on benefits. Here the benefits are seen as averted costs and vaccination can lead
to direct and indirect costs being averted. Direct costs are those such as physician
visits and hospitalisation while indirect costs include work absenteeism, reduced
productivity and forgone future earnings (the human capital method of valuing
benefits). Parameters for worker absenteeism and lower productivity were estimated
from the National Health Survey. It was estimated that influenza on average results in
3.2 to 3.4 days of work loss. Productivity is measured on a scale of effectiveness (1 to
10) and the mean score for UK workers with influenza like symptoms was 4.6
according to Keech et al (1998). This was converted into a number of days lost due to
reduced productivity. This was then compared with the cost of vaccination which
includes the cost of the vaccination itself and cost of a health care provider visit. The
metric of analysis was then cost savings (whereby costs averted were greater than
costs incurred) or net costs (vice versa) and it was found that vaccination was cost
saving. However there were notable limitations to the study. Perhaps the most
obvious was that they failed to take into account the economic value of leisure time or
the value that people place on the avoidance of suffering. Also the findings here may
not be applicable to part time workers as they were based on full time, year round
workers. However, if we view prevention as performing some kind of cost saving
10
function in the NHS then clearly this type of cost-benefit analysis is a good clear way
of getting the message across about cost effectiveness.
Cost consequence analysis
Cost consequence analysis lists the costs and benefits associated with an intervention
in a disaggregated way. NICE advocates the use of CCA in its technical guidance and
this support can be seen as one of its strengths. However it does not provide decision
makers with a single metric or ‘bottom line’ for evaluation and this may be seen as a
weakness or exception of the method. Having said this the lack of a single criterion
such as cost per QALY can be seen as a strength. Decision makers rarely make
decisions based on one criterion and maybe the strength of CCA is that it measures
impact across a range of criteria and compares it to cost. As such it shapes the
analysis to the decision making context, whereas other techniques try and shape the
decision making context to the analysis.
Cost capability analysis
This may be a novel concept and is suggested by the authors as a possible avenue of
investigation in the future. Maximising QALY gains is not the only aim of public
health interventions. NICE also considers the effect of interventions on the
distribution of health within society. Reducing health inequalities is sometimes
viewed as being equally important to maximising health for the population. One way
of capturing changes in inequality is through the capability approach (Sen 1985). This
is defined as the ability to function given the choice and looks to distribute capability
equally across society. The development of the ICECAP (Flynn 2013) (Al Janabi
2012) measures and a planned version for children represents a big opportunity for
this method. The concern with this approach however is the idea that consumers of
preventive goods are essentially healthy today and whether the capability approach
will pick up any effects. (We plan to add a case study after discussion of the concept
of “cost-capability” analysis at HESG Glasgow).
Multiple Criteria Decision Analysis (MCDA)
Multiple criteria decision analysis (MCDA) is aimed at supporting decision makers
faced with evaluating alternatives taking into account multiple, sometimes conflicting,
criteria. The first step of MCDA is to identify the alternatives to be appraised and the
11
criteria against which the alternatives are to be assessed. This is known as problem
structuring and is performed by the stakeholders involved through a process of
decision conferencing. The final stage involves scoring each alternative’s expected
performance on each criterion and then using weights to adjust each criterion’s
importance.
Perhaps the traditional way of thinking about decision making has been by recourse to
a single objective function whereby the decision makers seek to maximise (or
minimise) this function e.g. profit maximisation (Figueira et al 2005). Most of the
techniques of priority setting, e.g. evidence-based medicine, burden of disease
analysis, cost effectiveness analysis and equity analysis are based on a single criterion.
However MCDA has developed over the last thirty years and sets decision problems
in the context of a multitude of points of view taking into account their pros and cons.
It is argued (Baltussen et al 2006) that public health interventions need to be
evaluated in this broader context. For example, the need to address equity issues, to
maximise population health, all to budget, requires the decision maker to be able to
rationally choose which interventions deliver across a range of criteria.
MCDA is not confined to simple weighting and scoring and approaches are available
that are more relevant to NICEs health technology evaluation process. It can be seen
that the first step of MCDA, that is problem structuring, is akin to the scoping stage in
health technology assessments. Where the two methods differ is in the decision
making stage. With the NICE approach the ICER is used to capture and evaluate the
evidence whereas with the MCDA approach this evidence is inputted into
mathematical models to identify the best alternatives. The way these models are
specified accounts for the difference in the techniques that can be used. There are
three main approaches classified as value measurement models, outranking models
and goal, aspiration or reference models. The first approach uses and compares
numerical scores to assess the degree to which one option is preferred over another.
This approach is also the basis for programme budgeting marginal analysis (PBMA).
The outranking approach compares alternatives pair wise in terms of each criterion.
This is to establish the preference for each alternative for each specific criterion. The
preference information is then aggregated in order to arrive at the total preference for
12
each alternative. The last method is involved with setting pre-determined levels of
achievement for each criterion and identifying those alternatives that are closest to
matching these.
The case study for this section comes from the Netherlands. In it the authors (Bots &
Hulshof 2000) use MCDA to assess a) the most important health problems in the
Netherlands and b) efficiency gains in the health sector. Let us concentrate on the first
of these. This case was developed by initially clustering the diseases that needed to be
taken into account. The authors then determined the criteria to be used to rate these
clusters by importance. These criteria were prevalence, potential years of life lost and
cost of health care and the disease clusters were importance rated according to these
criteria. The next step was to find data on all the criteria for all the disease clusters
and using this data the three criteria scores were aggregated in order to determine the
rank order. This was the basic structure of the MCDA in this case. The rest of the
process involved policy design and this is where the case for prevention comes to the
fore. MCDA can be used to rank diseases in order of importance which can then be
used to inform on policy goals and instruments. These instruments were categorised
into ‘cure’, ‘care’ and ‘prevention’ and were defined for the 14 most ‘important’
disease clusters resulting in 42 distinct instruments. The case for prevention is
highlighted therefore in these policy instruments.
Return on Investment (ROI) and Social Return on Investment (SROI)
ROI and SROI are both pragmatic versions of CBA. They seek to link the inputs of a
process with its outputs. These methods report results as a ratio of the value created
by an intervention (£) in relation to the investment made £ for £. Within the NHS, the
aim of ROI analysis is to use economics to guide decision makers in the NHS as to
which public health prevention interventions should be invested in in order to receive
the highest returns possible. NICE have developed a return on investment tool for
smoking cessation2 and a tobacco control tool will be launched soon. They have also
launched alcohol3 and physical activity4 tools.
2
(http://www.nice.org.uk/usingguidance/implementationtools/returnoninvesment/TobaccoROITool.jsp)
3
http://beta.nice.org.uk/About/What-we-do/Into-practice/Return-on-investment-tools/Alcohol-return-oninvestment-tool
4
http://beta.nice.org.uk/About/What-we-do/Into-practice/Return-on-investment-tools/Physical-activity-return-oninvestment-tool
13
SROI takes a wider societal perspective generating a triple bottom line - financial,
social and environmental. According to the Cabinet Office (2010),
“Social Return on Investment (SROI) is a framework for measuring and
accounting for this much broader concept of value; it seeks to reduce inequality and
environmental degradation and improve wellbeing by incorporating social,
environmental and economic costs and benefits.”
Olsen & Lingane (2003) offer the following definition,
“A term originating from return on investment (ROI) used by traditional
investors. It describes the social impact of a business or nonprofit’ operation in dollar
terms, relative to the investment required to create that impact and exclusive of its
financial return to investors.” (Olsen & Lingale, p. 4)
We are interested in measuring inputs, outputs and outcomes. The inputs are those
that are needed to make the intervention work and are measured in terms of costs.
Outputs relate to the direct result of the intervention on patients in terms of metrics
such as life years gained, reduced prevalence of some disease etc. Outcomes measure
longer term benefits. These are adjusted to take account of deadweight – what would
have happened anyway without the intervention.
In order to investigate how SROI analysis works in real terms we will look at the
feasibility of applying this type of analysis to the field of work site health promotion
programs. This is a type of prevention based at work places intended to promote the
health of the workforce. Goetzel and Ozminkowski (2008) have conducted a review
of ROI studies in this field and this will serve as our case study.
Healthcare interventions in the workplace can produce health benefits and reduce
healthcare costs. Ill-health in the workplace is seen to affect absenteeism, staff
turnover and decreased productivity. The question is whether these work related
health programmes can improve employees’ health, reduce health costs, limit
14
absenteeism and decrease productivity loss in relation to the cost of the intervention.
A 1998 review of work place health programmes yielded estimated ROI savings of
between $1.40 and $3.14 per dollar spent with a median ROI of ~$3.00 saved per
dollar spent. It can be seen that SROI is particularly apt in this case for prevention as
it allows the researcher to isolate specific program elements and see which elements
deliver the “biggest bang for the buck” or highest ROI ratio. Resources can then be
channelled into those elements that seem to perform better and have the greatest effect
on health for the cost involved. SROI ratios also provide a comparable framework in
which to assess these interventions. A downside to this method however is the fact
that many public health interventions, especially preventative ones, do not show a
positive ROI for many years – the return is far from being instant. This may put
employers off and make them reluctant to implement work place health promotion.
Another weakness with the method is that negative results, or ROI ratios less than one,
are unlikely to be reported in the literature. Finally an important element to consider
in ROI analysis is selection bias. This may inflate cost savings and ROI ratios in work
site studies. It may be seen that changes in medical expenditures or absenteeism can
be attributed to underlying characteristics which may be independent of the program
being evaluated. It is sometimes unclear whether the participants are healthier than
nonparticipants to begin with. There are methods available however, such as those
suggested by Heckman (propensity matching e.g.), that correct for this selection bias.
There is growing evidence that these types of preventive interventions generate a
positive ROI to employers that invest in them. The authors note however that the ROI
research in this field is largely based on employer-sponsored health programmes.
Most of the research is therefore funded by these employers and therefore likely to
provide artificially positive ROI metrics, as the employers want a positive assessment
to justify their investment decisions.
To conclude, the authors note that in order for an intervention to have a positive ROI
it needs to be funded at an optimal investment level so that the savings from the
programme are equal to or greater than the programme’s costs. Most employers do
not know where this tipping point is and so further research is required on the optimal
design and cost of these interventions.
15
Discussion
At present there is a need for a range of methods to be applied to the evaluation of
public health prevention interventions, and a clear need for some standardisation of
techniques, including wider use of thresholds, as is current practice in health
technology assessment.
We would welcome discussion of these issues at HESG in
Glasgow.
16
Table 1: A NOISE Analysis of Techniques of Economic Analysis and related techniques being applied to the evaluation
of Public Health intervention
EVALUATION
NEED
METHOD
Service commissioners in
local government
(England) and in the NHS
Applies to all
and local government
techniques of
(Scotland, Wales and N
analysis
Ireland) increasingly
asking for evidence of the
cost effectiveness of
prevention initiatives
OPPORTUNITIES
IMPROVEMENTS
NEEDED
STRENGTHS
EXCEPTIONS
1.Funding streams such as PHR
mean more opportunity for trials
and cohort studies in public
health
2. NICE centre for public health
Application to more
excellence proposes methods for
examples
economic evaluation of public
Avoids ad hoc
investment decision
making
Arguably no need for economic
evaluation of low cost interventions
with strong evidence of
effectiveness
health interventions, ongoing
consultation
CMA rarely applicable as it is rare
that 2 prevention initiatives have
Cost Analysis
exactly the same outcome
Useful within a single
Cost
effectiveness
As above
analysis
public health
Cannot be used to compare across
prevention
public health prevention
programme (e.g.
programmes for different health
smoking cessation
problems or population groups
programme).
Cost utility
analysis
As above
Increased evidence base of cost
Further use of utility
CUA allows
Measurement of utility may be very
per QALY for public health
weights in particular
comparison of value
difficult in prevention initiatives
17
interventions Owen et al 2010
population groups e.g.
for money of public
where consumers are essentially
children.
health prevention
healthy now, with ceilings effects
initiatives with
e.g. EQ-5D. Also measurement of
medical
utility may be difficult in certain
interventions. Also
groups such as children or the very
allows comparison
elderly.
across prevention
programmes for
different health
problems and in
different population
groups
Application of
treasury approach to
CBA places public
Cost benefit
analysis
As above
health prevention at
NICE supports use of CBA in
cross-government
its technical guidance
level and allows for
comparison with
other government
investment decisions
Cost
consequence
As above
NICE supports use of CCA in
Advocated by NICE
its technical guidance
18
Does not provide
commissioners/policymakers with a
“bottom-line” or ratio of benefits to
analysis
cost
Cost capability
analysis
As above
Development of capability
Need for the planned
measures, ICECAP-O and A
ICECAP-C for children
Capability theory fits
Consumers of prevention goods
and planned version for
to allow for evaluation
very well with public
essentially healthy today, will
children. Applicable to public
of prevention initiatives
health concerns over
capability approach pick-up
health and provide alternative to
for children and young
inequalities in health.
anything?
QALYs
people
Traditional CEA,
CUA and CBA
Multiple criteria
decision analysis
Public health initiatives
methods have one
often have many
metric of outcome.
simultaneous goals.
MCDA allows for
multiple criteria to be
used
1. Increased volume of SROIs
Negative results or
relating to Public Health.
SROI ratios less than
Return on
Organisations such as New
£1/£1 rarely reported in
investment /
Economics Foundation teaching
the literature. Need for
this technique and accrediting
standardisation of
researchers round UK
discount rates, time
2. Increased range of proxy
horizon and range of
values/ shadow prices available
stakeholders
Social return on
investment
As above
19
Collation of evidence
for local government
in England e.g.
King’s Fund Report,
2013
from NEF and other sources
3. NICE developing a suite of
ROI tools for NHS(e.g.
smoking, alcohol misuse and
obesity prevention)
20
Table 2: Key Features of Evaluative Methods
EVALUATION
THEORETICAL
PERSPECTIVE
TIME
METHOD
FRAMEWORK
OF ANALYSIS
HORIZON
Accounting
NHS
Any
SOURCES OF
TECHNICAL
WEBSITE SUPPORT
GUIDANCE
Applies to all
techniques of analysis
Cost Analysis
Cost effectiveness
analysis
NHS or public
Extra-welfarist
sector multi
agency
NHS or public
Cost utility analysis
Extra-welfarist
sector multi
agency
Cost benefit analysis
Cost consequence
analysis
Any
Welfarist
Societal
Public sector
Extra-welfarist
multi agency or
societal
n/a
www.cdc.gov/owcd/eet/cost/1.html
NICE technical
NICE www.nice.org.uk
guidance 2008
www.cdc.gov/owcd/eet/costeffect2/fixed/1.html
Any, often
NICE technical
lifetime
guidance 2008
NICE
NICE public
Any, often
health guidance
lifetime
NICE Centre for Public Health Excellence
2012
NICE public
Any, often
health guidance
lifetime
NICE
2012
ICECAP website
Cost capability analysis
Capability theory
Societal (?)
Any
n/a
http://www.birmingham.ac.uk/research/activity/mds/projects/
HaPS/HE/ICECAP/index.aspx
Multiple criteria
Operations
decision analysis
Research
Stakeholder
Any
n/a
21
www.ncsu.edu/nrli/decision-making/MCDA.php
New Economics Foundation (NEF) www.neweconomics.org
Return on investment /
Social return on
investment
Public sector
Stakeholder or
Typically 1 to
Cabinet Office
economics
societal
3 years
2010
Social Return on Investment Network
www.thesroinetwork.org
Treasury https://www.gov.uk/government/organisations/hmtreasury
22
References
Al Janabi H, Flynn T, Coast J. Development of a self-report measure of capability
wellbeing for adults: The ICECAP-A. Quality of Life Research 2012; 21:167-176
Baltussen et al (2006) “Priority setting of health interventions: the need for multi
criteria decision analysis” Cost Effectiveness and Resource Allocation 2006, 4:14
Bots PWG, Hulshof JAM, (2000) “Designing Multi-Criteria Decision Analysis
Processes for Priority Setting in Health Policy”. Journal of Multi Criteria Decision
Analysis, 9: 56-75
Boyce, T, Peckham, S, Hann, A, Trenholm, S (2010), “A proactive approach. Health
promotion and ill-health prevention”, The Kings Fund.
Buck D, Gregory S, “Improving the public’s health: a resource for local authorities”,
King’s Fund, 2013.
Butterfield R., Henderson, J. and Scott R. (2009) Public Health and Prevention
Expenditure in England – Health England Report No.4. Health England.
Cabinet Office 2010. A Guide to Social Return on Investment. Prepared by the new
Economics Foundation. Available at:
http://www.bond.org.uk/data/files/Cabinet_office_A_guide_to_Social_Return_on_Inv
estment.pdf
Claxton, K et al (2013) “Methods for estimation of the NICE cost-effectiveness
threshold”, Centre for Health Economics, University of York.
Cohen (1984) Utility model of preventive behaviour. J Epidemiol Community Health.
1984 Mar;38(1):61-5.
Dalziel et al (2006) “Cost utility analysis of physical activity counselling in general
practice” Australian and New Zealand Journal of Public Health. Vol 30, No 1.
Drummond M, Weatherly H, Claxton K, Cookson R, Ferguson B, Godfrey C, Rice N,
Sculpher M, & Sowden A: Assessing the Challenge of Applying Standard Methods of
Economic Evaluation to Public Health Interventions.
[http://phrc.lshtm.ac.uk/papers/PHRC_D1-05_Final_Report.pdf]
Edwards RT: Paradigms and Research Programmes: Is it Time to Move From Health
Care Economics to Health Economics? Health Econ 2001 10:635-649
Edwards RT, Charles J, Lloyd-Williams H, (2013). “Public health economics: a
systematic review of guidance for the economic evaluation of public health
interventions and discussion of key methodological issues”, BMC Public Health, 13:
1001.
Faculty of Public Health (2014). http://www.fph.org.uk/
23
Figuieira, J et al (2005) “Multiple Criteria Decision Analysis: State of the Art
Surveys”, 78ed Springer New York, 133-153.
Fischer AJ et al (2013). “The appraisal of public health interventions: an overview”,
Journal of Public Health, 1-7.
Flynn T, Huynh E, Peters T, Al Janabi H, Clemens S, Moody A et al. Scoring the
ICECAP_A Capability instrument: Estimation of a UK general population tariff.
Health Economics 2013; Early View Online.
Goetzel RZ, Ozminkowski RJ, “The health and cost benefits of work site healthpromotion programs”, Annual Review of Public Health, 2008, 29:303-23
Hale, J, Phillips, CJ, Jewell, T, (2012) “Making the economic case for prevention – a
view from Wales”, BMC Public Health, 12:460.
Keech M, Scott AJ, Ryan PJ The impact of influenza and influenza-like illness on
productivity and healthcare resource utilization in a working population. Occup Med
(Lond). 1998;4885- 90
Kelly, M.P., McDaid, D., Ludbrook, A., & Powell, J. (2005). “Economic appraisal of
public health interventions”. London: Health Development Agency. Retrieved from
http://www.nice.org.uk/nicemedia/documents/Economic_appraisal_of_public_health_
interventions.pdf
Kelly MP (2005). “Public health guidance and the role of new NICE”, Public Health
119, 960-968.
McIntosh, E et al, “Applied Methods of Cost-Benefit Analysis in Health Care”,
Oxford Handbooks In Health Economic Evaluation. 2010
McKie, J & Richardson J (2003). "The Rule of Rescue" Social Science & Medicine
Volume 56, Issue 12, June 2003, 2407-2419
Mitchell RC, Carson RT. Using surveys to value public goods. 3rd ed. Resources for
the Future; 1989.National Institute for Health and Clinical Excellence (NICE) (2012)
Methods for the development of NICE public health guidance (third edition).
National Institute for Health and Clinical Excellence (NICE) (2011): Supporting
Investment in Public Health: Review of Methods for Assessing Cost Effectiveness,
Cost Impact and Return on Investment.
[http://www.nice.org.uk/media/664/AC/cost_impact_proof_of_concept.pdf].
New Economics Foundation. Measuring Social Impact: the foundation of social return
on investment (2004). Available at http://sroi.london.edu/Measuring-SocialImpact.pdf
24
NHS England (2013) “A call to action: commissioning for prevention”, Available at
http://www.england.nhs.uk/wp-content/uploads/2013/11/call-to-action-com-prev.pdf
Nichol KL “Cost-Benefit Analysis of a Strategy to Vaccinate Healthy Working
Adults Against Influenza.” Internal Medicine. 2001. 161 (5). 749-759
Olson, S, Lingane, A, “Social Return on Investment: Standard Guidelines”, (2003),
Centre for Responsible Business. Working Paper Series. Available from:
http://escholarship.org/uc/item/6xp540hs
Owen L, Morgan A, Fischer A, Ellis S, Hoy A & Kelly M:. The cost-effectiveness of
public health interventions, Journal of Public Health 2011, doi:
10.1093/pubmed/fdr075 [Published Online First].
Payne K, McAllister M, & Davies LM: Valuing the economic benefits of complex
interventions: When maximising health is not sufficient. Health Econ 2012,
doi:10.1002/hec.2795 [Published Online First].
Robinson R, “Cost Benefit Analysis”, BMJ , 307, 1993.
Rose, G (1992). “The Srtategy of Preventive Medicine” OUP, Oxford.
Sen AK: Commodities and Capabilities. Oxford: Oxford University Press; 1985
Squires, H (2013). “A Methodological Framework for developing the structure of
Public Health models”. PhD theses, University of Sheffield.
Suhrcke M, Urban D, Moesgaard Iburg K, Schwappach D, Boluarte T, McKee
M: The economic benefits of health prevention in a high-income country: the
example of Germany. Gutersloh: Bertelsmann Foundation; 2007.
Weatherly H, Drummond M, Claxton K, Cookson R, Ferguson B, Godfrey C, Rice N,
Sculpher M, & Sowden A (2009). “Methods for assessing the cost-effectiveness of
public health interventions: Key challenges and recommendations”. Health Policy
93:85-92.
25