The Payback Framework:
& developments in assessing policy and economic impacts
of health research & application to development research
Stephen Hanney
Health Economics Research Group,
Brunel University
stephen.hanney@brunel.ac.uk; claire.donovan@brunel.ac.uk
UKCDS/DFID/IDRC workshop:
Evaluation of Research Impact: Options and Challenges
Wellcome Trust: 18 October 2012
Overview of presentation

HERG Payback Framework to trace impacts from
specific bodies of health research
- 2 elements; applications; strengths/weaknesses.

Payback Framework stream developed in several ways
including:
- research impact on policy
- economic benefits of medical research

More speculatively: would it be possible to combine
approaches for development research?
- use the Payback Framework to assess impacts as ICDDR,B doing
- address the inevitable controversies around the economic benefits,
and explore the possibilities of studies on this
HERG’s Payback Framework
• Payback Framework has 2 main elements (Buxton &
Hanney, 1996)
• Multidimensional categorisation of benefits or paybacks:
- knowledge production
- targeting future research, capacity building, & absorption
- informing policies & product development
- health and health sector benefits, eg better health, health equity etc
- broader economic benefits (overlaps and changing definitions)
• Logic model of how to assess the benefits:
- 7 stages
- 2 interfaces
The payback framework:
logic model for assessing benefits
Adapted from: Hanney S, Gonzalez-Block M, Buxton M and Kogan M, 2003.
Payback framework to assess the impacts
research: corticosteroids for prevention of RDS



Used payback framework to organise data to show benefits from work of
Liggins on ante-natal use of corticosteroids for prevention of Respiratory
Distress Syndrome
Researcher moved from animals to humans: randomised the first
patients to receive the treatment (unusual, important in attribution)
Considerable impact in most categories (Hanney et al, 2005)
- knowledge production: major, widely cited publications
- informed research of many others (these studies inspired the Cochrane logo)
- influenced clinical policies in many countries
- thousands of babies survived, plus reduced morbidity & possible cost savings
- broader economic benefits in terms of human capital more complex to show

Raises nationality/attribution issues in impact assessment
Cochrane logo:
Liggins’ RCT and subsequent trials
Payback framework to assess research benefits:
corticosteroids for prevention of RDS - evidence






The evidence to attribute the impacts to Liggins research gathered in various
ways including:
Archival analysis – access to some documents from funder
Bibliometric analysis shows many later researchers cited this work, and in many
later trials this was the key reference
Review of policy statements such as guidelines, consensus statements shows
this work is cited as a key paper
Review of series of textbooks shows how early this work cited and leading
clinicians were influenced by it
Interviews with some key players in this story (in this instance did not use the
surveys that often used in other studies)
Payback Framework: limitations, strengths


Limitations:
- often difficult to identify impacts in later categories eg health gain
- the full impacts might not have occurred by time of assessment
- tends to be resource intensive to apply
- limitations of working forwards from individual projects which can
downplay role of other contributions
Strengths:
- includes range of benefit categories relevant to funders (Bacon, 1627)
- builds on major UK analysis of the collaborative approach
- provides a conceptual framework for organising application of various
methods (surveys, interviews) in a consistent way across a programme
- often successfully engages researchers in data collection
- facilitates presentation of data in a consistent way
- provides a framework for working forwards, addressing many
attribution issues, & maximising opportunities to identify impacts that
have occurred
- facilitates analysis of factors associated with impacts
Payback Framework & policy impacts

Alternative opening of Jane Austen’s Pride and Prejudice:
‘It is a truth universally acknowledged that a single researcher in possession of a
good finding must be in want of a policymaker’ (Hanney, 2005)

Rothschild experiment in government departments in 1970s: formative
evaluation in health department by Kogan & Henkel (’83;’06)
- collaborative approach developed & analysed: importance but difficulty of
getting policymakers & researchers to work together on agendas etc;
- knowledge brokerage role in getting research used by policy receptors;
- importance of research evaluation criteria recognising policy relevance.

Ideas informed Framework on ways to assess research impact on policy:
- collaboration might help to illustrate attribution/contribution of policy impact;
- Framework can help analysis of research’s policy impact in a variety of
systems

But key issue for evaluation: is research impact from collaboration and
pushing own findings always good – what about SR?
- the role of mechanisms/receptors to absorb SRs & other research.
Development of framework: economic returns


Reviewed literature for WHO, very little in LMIC (Buxton et al., 2004)
Then study for MRC/Wellcome Trust/Academy of Medical Sciences:
Economic returns from health research to the UK consist of two main
elements:




The UK health gain attributable to the research
UK GDP gains [This assessed by OHE]
Cardiovascular (CVD) and mental health (MH) research used as
exemplars [Just CVD shown here]
2 main approaches possible:
- top-down macro approaches that take health gain of nation and value it;
- estimate health gain by detailed ‘bottom-up’ approach (Buxton et al., 2008)
Development of framework: economic returns

Estimate public and charitable expenditure on CVD research

Estimate net value of CVD health gains attributable to research in
each year from 1986 – 2005:
- started with 45 specific health interventions in CVD (eg use of statins for
secondary prevention of CHD post MI), calculated health gain of each.
- used the opportunity cost of achieving health gain value of a QALY (Quality
Adjusted Life Year) consistent with NICE and allowed for costs of treatment

Estimate % of health change attributable to UK research and timelag from research to health benefit

Health gains to UK residents deliver a rate of return for CDV of 9%;
Non-health gains (GDP spill over gains) best estimate 30%
Best estimate of total gain equivalent to IRR of 39% per year
Are these approaches relevant for development research?

The UK economic returns study with its data requirements etc not replicable
in LMIC? But is it acceptable/feasible at all to consider the value of the
health gain from health research and build on specific examples?

Our report and the 39% IRR figure was the first reference in the Royal
Society’s submission to the DFID’s 2009 consultation on eliminating world
poverty. This might give some legitimacy to the general approach.

Meltzer D (2006) chapter on ‘Economic approaches to valuing global health
research.’ in: World Bank report by Jamison et al., eds, Disease Control
Priorities in Developing Countries:
- describes studies showing value of a statistical life increases with income;
- claims the value of recent health gains in countries such India is larger
than for the USA, even if a local statistical value of a life used;
- to move from the finding that gains in health are highly valued to the
finding that health research is highly valued, gains in health must be
connected to health research. [So, back to the start]
Combining approaches for development research?

Can we use the Payback Framework to show examples of research impact (eg
ICDDR,B) elements of the Buxton (2008) approach for development research?

Key point from our study is that we do not rely on valuing a statistical life.

Instead we rely on identifying the amount a system is willing to spend to
generate health gains.

Factors to consider:
- might the time lags might be shorter if much development research is applied
research?
- how far is local research or the global stock of knowledge contributing to
achieving health gains?

Could case studies using Payback Framework help address some of these
issues of impacts in relation to specific examples?
References
 Buxton M, Hanney S (1996) How can payback from health services research be
assessed? J Health Serv Res & Policy, 1: 35-43
 Buxton M, Hanney S, Jones T (2004) Estimating the economic value to societies of the
impact of health research: a critical review. Bull WHO, 82: 733-739.
 Buxton M, Hanney S, Morris S, et al (2008) Medical Research - What's it worth?
Estimating the economic benefits from medical research in the UK. London: UK
Evaluation Forum (Academy of Medical Sciences, MRC, Wellcome Trust).
 Hanney S, Gonzalez-Block M, Buxton M, Kogan M (2003) The utilisation of health
research in policy-making: Concepts, examples and methods of assessment. Health
Res Policy Syst, 1:2
 Hanney S, Mugford M, Grant J, Buxton M (2005) Assessing the benefits of health
research: Lessons from research into the use of antenatal corticosteroids for the
prevention of neonatal respiratory distress syndrome. Soc Sci Med, 60: 937-47.
 Hanney S (2005) Personal interaction with researchers or detached synthesis of the
evidence: modelling the health policy paradox. Evaluation and Research in Education,
18: 72-82.
 Hanney S, Buxton M, Green C, Coulson D, Raftery J (2007) An assessment of the
impact of the NHS Health Technology Assessment Programme. Health Technol Assess,
11(53).
 Kogan M, Henkel M, Hanney S (2006) Government and Research; Thirty Years of
Evolution. (Second edition). Dordrecht: Springer.