Making an Impact
A Preferred Framework and Indicators
to Measure Returns on Investment in Health Research
Full Report available at
http://www.cahs-acss.ca/e/assessments/completedprojects.php
Canadian Academy of Health Sciences
Académie canadienne des sciences de la santé
scientific advice for a healthy Canada
Table of Contents
Background
The Report

Impact categories and Impacts

Three case examples

Recommendations
scientific advice for a healthy Canada
2
Background
scientific advice for a healthy Canada
3
CAHS: Who We Are

Created in 2004

Non-profit charitable organization

One of three founding member academies

Unique collaboration of 6 health disciplines and the
full spectrum of academic health sciences

Elected Board (13) plus >260 Fellows
scientific advice for a healthy Canada
4
What is CAHS?

A new collaborative body

Multidisciplinary, accomplished health scientists

Not an advocacy group

“Scientific advice for a healthy Canada”

Publications including this report on CAHS web site:
http://www.cahs-acss.ca
scientific advice for a healthy Canada
5
Scientific Advice for a Healthy Canada

Tackle urgent and complex problems

Use knowledge to address apprehension

Congregate the best minds

Assemble the best science

Listen, deliberate, debate

Provide the best advice

Unbiased

Non-vested

Balanced

Feasible
scientific advice for a healthy Canada
6
Evolution of Return on Investment Assessment
CAHS Executive or Board
Assessment Panel and staff
Assessment Standing Committee
Sponsors
External Reviewers
Others
Meeting
Sponsor’s input
Invite panelists, refine question , comprehensive literature and expertise
searches, panel meetings, draft assessment and related information,
interview experts, identify areas where information is lacking,
commission papers to fill gaps, obtain consensus on recommendations,
address issues raised by external reviewers, help disseminate report
Jan-09
Dec-08
Nov-08
Oct-08
Approval,
publication
preparation,
translation,
meeting planning,
dissemination
Sep-08
Aug-08
Jul-08
Jun-08
May-08
Apr-08
Mar-08
Feb-08
Jan-08
Dec-07
Nov-07
Oct-07
Sep-07
Aug-07
Jul-07
Jun-07
May-07
Apr-07
Mar-07
Feb-07
Jan-07
Dec-06
Reviewed
recommended
approval
Progress report
sponsors
Initial literature and expertise searches,
drafting prospectus, fundraising,
communication to sponsors,
forum planning, financial management
Select & appoint chair
7
Recommend
&
Recruit
External
Reviewers
7 commissioned
papers
External
review
Refinement of prospectus,
approval of question, define
expertise appoint panelists
ROI Sponsors
Major Sponsors




Canadian Health Services Research
Foundation (CHSRF)
Canadian Institutes of Health Research
(CIHR)
Canada’s Research-Based Pharmaceutical
Companies (Rx&D)
Public Health Agency of Canada (PHAC)
Sponsors








Alberta Heritage Foundation for Medical
Research (AHFMR)
Association of Canadian Academic
Healthcare Organizations (ACAHO)
Association of Faculties of Medicine of
Canada (AFMC)
BIOTECanada
Canadian Agency for Drugs and
Technologies in Health (CADTH)
Fonds de la recherche en santé du Québec
(FRSQ)
Government of Ontario, Ministry of Research
and Innovation; Ministry of Health and LongTerm Care
Heart & Stroke Foundation of Canada
(HSFC)
Sponsors (continued)







Manitoba Health Research Council (MHRC)
Michael Smith Foundation for Health Research
(MSFHR)
National Cancer Institute of Canada (NCIC)
Nova Scotia Health Research Foundation
(NSHRF)
Ontario Neurotrauma Foundation (ONF)
Saskatchewan Health Research Foundation
(SHRF)
Western Economic Diversification Canada (WD)
Contributors







Canada Foundation for Innovation (CFI)
Canadian Association of Schools of Nursing
(CASN)
Canadian Medical Association (CMA)
Canadian Nurses Association (CNA)
Canadian Nurses Foundation (CNF)
Newfoundland & Labrador Centre for Applied
Health Research (NLCAHR)
Research Canada
scientific advice for a healthy Canada
8
Why ROI in Health Research? (1)

Lack of public understanding of the value of research
applicability to current issues in health

Concern about accessible, affordable, high quality
health care in a publicly funded system

Need to adequately measure & meaningfully convey
benefits of health research to policy-makers & public

Increasingly common view that health care / health
research) is a cost-driver consuming an ever greater
share of resources at expense of other sectors

Concern about expenditure accountability in both the
public and private sectors in Canada and abroad
scientific advice for a healthy Canada
9
Why ROI in Health Research? (2)

Lack of consensus on how and when to best evaluate
return on research expenditures

Questions from policy makers about tangible results
attributable to recent increases in public investment in
health research e.g. CIHR, CFI, CRC programs

Uncertainty about appropriateness of Canada’s health
research expenditures versus those of analogous
contributions in other industrialized countries

Need to acquire appropriate evidence to strike right
funding balance between investigator-initiated
“discovery” & targeted “strategic” health research
scientific advice for a healthy Canada
10
Mobilizing Science and Technology to
Canada’s Advantage 2007
The Science and Technology Framework
Vision: We will build a sustainable national competitive advantage based
on science and technology and the skilled workers whose aspirations,
ambitions, and talents bring innovations to life.
To achieve this vision, we will create three S&T Advantages for Canada:
Entrepreneurial Advantage
Canada must translate knowledge into
practical applications to improve our wealth,
wellness, and well-being.
Knowledge Advantage
People Advantage
Canada must build upon our research
and engineering strengths, generate
new ideas and innovations, and
achieve excellence by global
standards.
Canada must grow its base of
knowledge workers by developing,
attracting, and retaining the highly
skilled people we need to thrive in
the modern global economy.
Mobilizing Science and Technology to
Canada’s Advantage 2007
Policy Commitments
Canada’s federal government will increase its
accountability to Canadians by:
Improving its ability to measure and report on the
impact of S&T expenditures. The government will
improve its understanding of Canadian S&T
developments and the impact of federally performed
S&T, and will work with the OCED and other
countries to develop metrics that will enable
comparisons against international benchmarks of
success.
12
CAHS Standing Committee on
Assessments











13
Andreas Laupacis, MD (Chair), Executive Director, Li Ka Shing Knowledge Institute
of St. Michael'sHospital; Professor, Faculty of Medicine, University of Toronto
John A. Cairns, MD, Professor of Medicine and Dean Emeritus, UBC
Timothy Caulfield, LLM, CRC in Health Law and Policy; Professor, Faculty of Law
and School of Public Health; Senior Health Scholar AHFMR and Research Director,
Health Law Institute, University of Alberta
André-Pierre Contandriopoulos, PhD, Professeur Titulaire, Département
d'Administration de la santé, Université de Montréal
Alastair Cribb, DVM, PhD, Dean, Faculty of Veterinary Medicine, U of Calgary
Jean Gray, CM,LLD, DSc, Professor Emeritus, Dalhousie University
Pavel Hamet, MD, PhD, CRC, Predictive Genomics; Chief, Gene Medicine Services,
Centre de recherche Centre hospitalier de l’Université de Montreal
Dorothy Pringle, OC, RN, PhD, Professor Emeritus, Faculty of Nursing, U of
Toronto
Matthew Spence, OC, MD, PhD, Retired President and CEO, AHFMR
Peter S. L. Tugwell, MD, CRC in Health Equity; Director, Centre for Global Health
(Institute of Population Health); Professor of Medicine & Epidemiology, U Ottawa
Sharon L. Wood Dauphinee, PhD, PT, Professor, McGill University, Montreal, QC
Steps in a CAHS Assessment

Choice of topic

Choice of chair

Choice of assessment panel

Panel’s independent work

Review of draft report by external reviewers
and CAHS Assessment Committee

Revision of report

Approval by CAHS Board
scientific advice for a healthy Canada
14
External Reviewers

Joseph B. Martin. MD, PhD, Edward R. and Anne G.
Lefler Professor of Neurobiology and former Dean of the
Harvard Faculty of Medicine, Harvard Medical School,
Boston, MA, USA

The Honourable Michael J. L. Kirby, MA, PhD, LLD(Hon),
Chair, Mental Health Commission of Canada, Senator
(former), Ottawa, ON, Canada

John W. Frank, MD, CCFP, MSc, FRCP(C ), Director,
Scottish Collaboration for Public Health Research
andPolicy, MRC Human Genetics Unit, Edinburgh,
United Kingdom
scientific advice for a healthy Canada
15
The Report
scientific advice for a healthy Canada
16
The Panel (1)








Cyril Frank, MD (Chair), McCaig Professor of Joint Injury and Arthritis
Research; Professor, Division of Orthopaedics University of Calgary
Renaldo Battista, MD, MPH, ScD, Professor and Director of the
Department of Health Administration, Université de Montréal
Linda Butler, Fellow and Head, Research Evaluation and Policy Project,
Australian National University
Martin Buxton, BA, Professor, Health Economics, Brunel University, UK
Neena Chappell, PhD, Canada Research Chair; Social Gerontology,
Professor of Sociology and Centre on Aging, University of Victoria
Sally C. Davies, Director General, Research and Development,
Department of Health and National Health Service, UK
Aled Edwards, PhD, Banbury Professor, Banting and Best Department of
Medical Research, University of Toronto
Chris Henshall, PhD, Pro-Vice-Chancellor, External Relations, University
of York, UK
scientific advice for a healthy Canada
17
The Panel (2)





Yann Joly, LLB, LLM, project manager, Centre de recherche
en droit public Université de Montréal
Gretchen Jordan, PhD, Principal Member of Technical Staff,
Science and Technology Strategic Management Unit,
Department of Energy, Washington, DC
Terence Kealey, MB, BS, PhD, Vice Chancellor & Clinical
Bio-Chemist, University of Buckingham, UK; author The
Economic Laws of Scientific Research
Michael C. Wolfson, PhD, Assistant Chief Statistician,
Analysis and Development, Statistics Canada
Steven H. Woolf, MD, MPH, Professor, Departments of
Family Medicine, Epidemiology, and Community Health,
Virginia Commonwealth University, US
scientific advice for a healthy Canada
18
The Staff

Eddy Nason – Health Research Evaluation Analyst and
Writer (Toronto) –

Larissa Sommerfeld – Research Assistant (Calgary)

Linda Marchuk – Research Associate , Admin Support
and Finance Officer (Calgary)

Rhonda Kennedee – Meeting and Events Coordinator
(Calgary)
scientific advice for a healthy Canada
19
The Remit

Is there a “best way”
(method) to evaluate the
impacts of health
research in Canada &
are there ‘best metrics’
for assessing those
impacts (or improving
them)?

Useful to a full range of
funders/research types

Compatible with what is
already in place in
Canada

Transferrable to
international
comparisons

Able to identify the full
spectrum of potential
impacts
scientific advice for a healthy Canada
20
The Approach






Sponsor interviews by panelists to be certain of needs
Literature review
Expert interviews
Seven commissioned papers in areas of special interest and/or
perceived gaps:
 Public perspective
 Ethics
 International frameworks
 Pillar II, Pillar III, and Pillar IV research
 ‘Meso-level’ metrics for impact
Working Groups on panel
Face-to-face meetings x 3
scientific advice for a healthy Canada
21
Sponsor Expectations
Metric
Development
Cover all
types of
funding
Cover long
range and
global
impacts
Framework
Development
Metrics for
human
resources
Understand
evaluation
work in
place in
Canada
Must allow
learning
not just
audit
Metrics
relevant to
all four
pillars
Understand
lag-times
for impacts
Evaluation
issues
Cover nonmonetary
impacts too
Metrics for
commercialization
Understand
knowledge
translation
Understand
the
attribution
problem
scientific advice for a healthy Canada
22
Different Evaluation Needs
Evaluation for
Accountability
External
audience
Evaluation for
Advocacy
Mission
linked
Identify
‘best’
Comprehensive
Evaluation
= Our Target
Evaluation for
Learning
scientific advice for a healthy Canada
23
Many Complexities
For example: the (nearly) infinite number
of potential evaluation questions

Evaluation questions from one funder and one
program only:

Have we increased the skill set of Canadian health
research?

Have we increased the number of skilled
researchers working in Canada?

Are our trainees producing high quality research?

Are our trainees disseminating their findings to a
variety of appropriate stakeholders?
scientific advice for a healthy Canada
24
Experts Interviewed

Douglas Barber

Ilse Treurnicht

Stefan Ellenbroek

Sharon Manson Singer

Greg Webster

Teren Clarke

Wendy Baldwin

Cheryl L. Koehn

Egon Jonsson

Greg Tassey

Noralou P.Roos

Muhajarine Nazeem

John Cairns

Carol Dahl
scientific advice for a healthy Canada
25
Selected Literature
>260 articles, books, web links, etc










Buxton MJ, Hanney SR. How can payback from health services research be assessed? J
Health Serv Res Pol. 1996;1(1):35-43. ******
US Senate. Joint Economic Committee. The benefits of medical research and the role of the
NIH. Washington, D.C. 2000.
Nason E, Janta B, Hastings G, Hanney S, O'Driscoll M, Wooding S. 2008. Health research:
Making an Impact. The Economic and Social Benefits of HRB Funded
Research.Dublin:Ireland.
Oortwijn W, Hanney S, Ligtvoet A, Hoorens S, Wooding S, Grant J, et al. Assessing the
impact of health technology assessment in the Netherlands. Int J Technol Assess Health
Care. 2008;24(3):259-69.
Lavis J, Ross S, McLeod C, Gildiner A. Measuring the impact of health research. J Health
Services Res Pol. 2003;8(3):165-170.
Funding First.; 2000. Exceptional Returns: The economic value of America's investment in
medical research. New York, New York: The Lasker Foundation. ******
Cutler DM, Kadiyala S.; 1999. The Economics of Better Health: The Case of Cardiovascular
Disease. New York, New York: The Lasker Foundation.
Access Economics; 2003. Exceptional Returns: the value of investing in health R&D in
Australia. 2003.
Access Economics; 2008.Exceptional returns: The value of investing in health R&D in
Australia II.Canberra.AccessEconomics;2008 ********
Buxton M, Hanney S, Morris S, Sundmacher L, Metre-Ferrandiz J, Garau M, et al. Medical
Research – What’s it worth? Estimating the economic benefits from medical research in the
UK. Report to the UK evaluation forum 2008. London, UK. ********
scientific advice for a healthy Canada
26
Many Options for Methods

Econometric approaches and models

Performance measurement systems

Logic models and frameworks

Implementation evaluation models

Balanced scorecards
scientific advice for a healthy Canada
27
Many ‘Technical Complexities’
Issues in Determining Health Research Impacts


Attribution issues (effects of factors other than research) and
the ‘counterfactual’ (what would have happened without the
research being done)?

Need collaboration

Need research on those topics
Time lags to impact are very long


Double-counting of health research impacts


Need indicators that can track longitudinally
Need to determine contributions wherever possible
The ‘Halo effect’ (only consider positive impacts of research)

Need to consider negative impacts of research
scientific advice for a healthy Canada
28
Big Challenge

To address the diverse needs of the sponsors

Identify a method that can work for all

To identify a method that can help resolve the
technical challenges noted above

To synthesize all available information on this topic

To not ‘reinvent the wheel’ as many other groups are
also working in this area in the world right now

Avoid simply advocating ‘more research’ on the topic
scientific advice for a healthy Canada
29
RAND EUROPE
Health Research Evaluation
Frameworks –
An International Comparison
Philipp-Bastian Brutscher, Steven
Wooding, Jonathan Grant
scientific advice for a healthy Canada
30
Reviewed 8 evaluation frameworks
Frameworks
Country
Description
Leiden University Medical
Center (LUMC)
NL

Measure of Research Impact
and Achievement (MORIA)
AUS



Program Assessment Rating
Tool (PART)
US


Vinnova (Swedish Govern.
Agency for Innovation Systems)
S

Looks at “societal impact” (rather than scientific quality)
Can be seen as part of a broader movement to correct
for the “serious imbalance in the research portfolio”.
Was developed at the Australian NHMRC as an analytic
instrument in the peer review process for grant
applications.
It builds on the Record of Research Achievement
framework.
Was introduced in 2001, as part of the Bush
administration’s agenda to improve government
management.
PART is used to assess the effectiveness of around 800
federal programmes.
Consists of two main parts: an ongoing evaluation
process and an impact analysis.
scientific advice for a healthy Canada
31
Reviewed 8 evaluation frameworks
(Cont’d)
Frameworks
Country
Description
Payback
UK +
Canada +
Ireland

UK Department for
Innovation, Universities and
Skills (DIUS)
UK

European Union Framework
Programme (EU)
EU




Congr. Directed Medical
Research Programs
(CDMRP)
US


Has been applied in a number of contexts.
Is an input-process-output-outcome frame-work. It
typically comprises two parts: evaluation criteria (potential impact criteria) and a logic model.
Aims to “assess the overall health of the science and
innovation system…”.
Is the latest stage in a process of developing S&I
performance appraisal methods.
Is meant as a system for tracking the results of
research programmes
Intended as a way to identify what needs to be
improved for these programmes to be more effective.
Part of the US Army Medical Research and Material
Command (USAMRMC).
Consists of a grants management system, a product
database and an Award Survey.
scientific advice for a healthy Canada
32
Payback Logic Model “Flow”
Adapted from: Hanney S, Gonzalez-Block M, Buxton M and Kogan M. The Utilisation of
health research in policy-making: concepts, examples and methods of assessment. Health
Research Policy Systems 2003, 1:2
scientific advice for a healthy Canada
33
Five Categories of Health Research Impact
Original
Payback
1996
CIHR 2005
CIHR 2008
Knowledge Production
Knowledge
Production
Advancing Knowledge
Research Targeting,
Capacity and Absorption
Research Targeting
and Research
Capacity
Research
Capacity
Informing Policies and
Product Development
Informing Policy
Health and health sector
benefits
Health and Health
Sector Benefits
Health Benefits
Broader Economic Benefits
Economic Benefits
Economic Benefits
Informing
Decision Making
Payback Model Advantages
(for Canada)







Built on a logic model and framework
Strong for accountability and management
In use in Canada (CIHR and AHFMR)
 Similar to some others in use in Canada (eg FRSQ)
and builds on elements of others (Lavis, etc)
Can allow adequate time scale (longitudinal
evaluation)
Can track from individual level upward
Can potentially track from inputs to outputs and
onwards to outcomes
Considers social and economic impacts
scientific advice for a healthy Canada
35
CAHS Logic Model Framework
built on payback logic model (across bottom)
Initiation and Diffusion of Health Research Impacts
Global
Research
Canadian
Health
Research
•Biomedical
•Clinical
•Health
Services
•Population
and Public
Health
•Cross-Pillar
Research
Interactions/Feedback
Health
Status,
Function,
Well-being,
Economic
Conditions
Health
Industry
Health care
Appropriateness,
Access, etc.
Other
Industries
Government
Research
Agenda
Public
Information,
Groups
Improvements
in Health and
Well-being
Prevention and
Treatment
Determinants of
Health
Economic
and Social
Prosperity
Research
Capacity
36
Dissemination
•Topic
Identification
•Selection
•Inputs
•Process
Primary Outputs
PAYBACK
FRAMEWORK
Impacts feed back into inputs for future research
Secondary
Outputs
Adoption
Final
Outcomes
Initiation and Diffusion of Health Research Impacts
Research activity
That produces
results
Health Industry
-Products/drugs
-Services, databases
-Practitioners’ behaviour
-Clinical/manager’s guidelines
-Institutional policies
-Social care practices
Global Research
•Topic Identification
PAYBACK •Selection
FRAMEWORK •Inputs
•Process
37
Consultation/ Collaborations
Research Capacity
-Increased understanding
-Methodological
advances
-Larger, more
comprehensive data sets
-Human capital
(absorptive capacity)
-Student and faculty
career paths
-Reputation
-Research revenues
-Cross-fertilization of
ideas/research
-Education curriculum
Other
Industries
-Products/services
-Built infrastructure
-Work environment
Research Decision Making
-R&D agendas/investment
(industry/gov’t/foundations)
-Identify issues, gaps
-Evidence problems are being
addressed
-Tackle harder problems
Secondary
Outputs
That affect
healthcare, health
risk factors, and
other health
determinants
That contribute to
changing health,
well -being and
economic and
social prosperity
Health care
-Appropriateness
-Acceptability
-Accessibility
-Competence
-Continuity
-Effectiveness
-Safety
Occur through
prevention and
treatment
For disease, illness,
injury, or progressive
condition
-Prevention
-Diagnosis/prognosis
-Treatment/palliation
-Post-treatment
Government
(multiple levels)
-Resource allocation
-Regulation
-Policy
-Intervention programs
-Taxes and subsidies
Public Information,
Groups
- Advocacy groups
- Media coverage
- General knowledge
- Confidence in data
Primary
Outputs
Dissemination
Health
status and
function,
well-being,
economic
conditions
Knowledge Pool
Canadian
Health
Research
•Biomedical
•Clinical
•Health services
•Population and
public health
•Cross-pillar
research
That influence decision
making in…
Determinants of
health
-Personal behaviour
-Social/cultural
determinants
-Environmental
determinants
-Living and working
conditions
Improvements
in health and
well-being
(disease
prevalence and
burden)
Economic and
social
prosperity
External Influences:
Interests, Traditions
Technical limitations,
Political dynamics
Impacts feed back into inputs for future research
Adoption
Final
Outcomes
Impact Categories
and Impacts
scientific advice for a healthy Canada
38
Original
Payback
1996
CIHR 2005
CIHR 2008
CAHS 2009
(Impacts)
Knowledge
Production
Knowledge
Production
Advancing
Knowledge
Advancing
Knowledge
Research
Targeting,
Capacity and
Absorption
Research
Targeting and
Research
Capacity
Research
Capacity
Building
Capacity
Informing
Decision Making
Informing
Decision Making
Health Benefits
Health Benefits
Informing
Policies and
Product
Development
Informing Policy
Health and
health sector
benefits
Health and
Health Sector
Benefits
Broader
Economic
Economic
Benefits
Economic
Benefits
Broad Economic
and Social
scientific advice for a healthy Canada
Biomedical Research
Initiation and Diffusion of Health Research Impacts
Health
Industry
Interactions/Feedback
Health
Status,
Function,
Wellbeing,
Economic
Conditions
Canadian
Health
Research
•Biomedical
•Clinical
•Health
Services
•Population and
Public Health
•Cross-pillar
Research
Research Results
Knowledge Pool
Global Research
Healthcare
Appropriateness,
Access, etc.
Other
Industries
Government
Research
Agenda
Public
Information,
Groups
Improvements
in Health and
Well-being
Prevention and
Treatment
Determinants of
Health
Economic
and Social
Prosperity
Research
Capacity
Impacts feed back into inputs for future research
Advancing
Knowledge
Capacity Building
40
Informing Decision
Making
Health Benefits
Economic
Benefits
Clinical Research
Initiation and Diffusion of Health Research Impacts
Global
Research
Interactions/Feedback
Knowledge Pool
Research Results
Health
Status,
Function,
Well-being,
Economic
Conditions
Canadian
Health
Research
•Biomedical
•Clinical
•Health Services
•Population and
Public Health
•Cross-pillar
Research
Health
Industry
Other
Industries
Government
Research
Agenda
Public
Information,
Groups
Healthcare
Appropriateness,
Access, etc.
Improvements
in Health and
Well-being
Prevention and
Treatment
Economic
and Social
Prosperity
Determinants of
Health
Research
Capacity
Impacts feed back into inputs for future research
41
Advancing
Knowledge
Informing Decision
Making
Capacity Building
Health Benefits
Economic
Benefits
Health Services Research
Initiation and Diffusion of Health Research Impacts
Global
Research
Interactions/Feedback
Knowledge Pool
Research Results
Health
Status,
Function,
Well-being,
Economic
Conditions
Canadian
Health
Research
•Biomedical
•Clinical
•Health
Services
•Population
and Public
Health
•Cross-pillar
Research
Health
Industry
Other
Industries
Government
Research
Agenda
Public
Information,
Groups
Healthcare
Appropriateness,
Access, etc.
Improvements
in Health and
Well-being
Prevention and
Treatment
Determinants of
Health
Economic
and Social
Prosperity
Research
Capacity
Impacts feed back into inputs for future research
42
Advancing
Knowledge
Informing Decision
Making
Capacity Building
Health Benefits
Economic Benefits
Population and Public Health Research
Initiation and Diffusion of Health Research Impacts
Health
Industry
Interactions/Feedback
Knowledge Pool
Health
Status,
Function,
Well-being,
Economic
Conditions
Canadian
Health
Research
•Biomedical
•Clinical
•Health
Services
•Population
and Public
Health
•Cross-pillar
Research
Research Results
Global Research
Other
Industries
Government
Research
Agenda
Public
Information
, Groups
Healthcare
Appropriateness,
Access, etc.
Improvements
in Health and
Well-being
Prevention and
Treatment
Determinants of
Health
Economic
and Social
Prosperity
Research
Capacity
Impacts feed back into inputs for future research
43
Advancing
Knowledge
Informing Decision
Making
Capacity Building
Health Benefits
Economic
Benefits
Cross Pillar Research
Initiation and Diffusion of Health Research Impacts
Health
Industry
Research
Capacity
44
Interactions/Feedback
Knowledge Pool
Health
Status,
Function,
Well-being,
Economic
Conditions
Canadian
Health
Research
•Biomedical
•Clinical
•Health
Services
•Population
and Public
Health
•Cross-pillar
Research
Research Results
Global
Research
Other
Industries
Healthcare
Appropriateness,
Access, etc.
Government
Prevention and
Treatment
Research
Agenda
Public
Information,
Groups
Improvements
in Health and
Well-being
Determinants of
Health
Economic
and Social
Prosperity
Impacts feed back into inputs for future research
Advancing
Knowledge
Informing Decision
Making
Capacity Building
Health Benefits
Economic
Benefits
Appropriate Use of the Framework
understanding the logic model and impact categories
FRAMEWORK
Health
R&D
Primary
Outputs/
Dissemination
Advancing
Knowledge
• New
molecular
technique
developed
• Publication of research
results in a journal
Research
Capacity
• Research
PhD gained by
team member
IMPACT
CATEGORIES
Informing
Decision
Making
Secondary
Outputs
Final
outcomes
• Further research
in Industry
• Discussions between
researchers and pharma
define direction of
pharma research
• Pharma company
initiates research
program to
develop a drug
Health
Impacts
Broad
Economic and
Social Impacts
Adoption
• Employment in
the pharma
company
• Drug developed
passed for use by
the health system
• Adoption by the
health system
causes increased
cost of drugs
• Decreased readmission for
condition
• Reduced
condition burden
in the population
• Sales of drugs
by pharma
• Improved
workplace
productivity
• Social and
economic benefit
of “wellness”
scientific advice for a healthy Canada
45
Indicators vs Metrics

Indicators ‘indicate’ impact; they do not
attempt to quantify that impact

Metrics are ‘numeric indicators’; they allow
putting some numbers on impact

A combination of indicators and metrics
are recommended
scientific advice for a healthy Canada
46
REFER to MENU of INDICATORS*
Table of CAHS Indicators and Metrics
arranged by impact category with advice
about how to use each one
Shows subcategories, indicators within each
subcategory, indicator description, level of
recommended application , comments and pillars
* To begin a library with references on the web
scientific advice for a healthy Canada
47
Sets of indicators and metrics chosen
should meet FABRIC criteria:






Focussed on the organization’s objectives that will
use them
Appropriate for the stakeholders who are likely to
use the information
Balanced to cover all significant areas of work
performed by an organization
Robust enough to cope with organizational changes
(such as staff changes)
Integrated into management processes
Cost-effective (balancing the benefits of the
information against collection costs)
scientific advice for a healthy Canada
48
Indicators proposed meet qualities of
attractiveness and feasibility

Attractiveness: validity, relevance,
behavioural impact, transparency, coverage,
recency, methodological soundness,
replicability, comparability

Feasibility: data availability, cost of data,
compliance costs, timeliness, attribution,
avoids gamesmanship, interpretation, welldefined
scientific advice for a healthy Canada
49
‘Steps’ for Users of the Framework
and Indicators
1.
Define and prioritize specific evaluation question(s).
2.
Use the framework to determine where to look for impacts
3.
Based on question(s) choose the impact categories (and
subcategories) of interest: advancing knowledge, capacity
building, informing decision making, health impacts, and
broad economic and social impacts.
4.
a.
Be as specific as possible about where impacts are
expected to occur and at what level (individual, group,
institution, provincial, federal, international).
b.
Choose (or develop) attractive and feasible indicators and
metrics from the appropriate categories of interest that will
address the evaluation questions at the right level.
Choose sets of indicators that are appropriate.
Avoid inappropriate uses: attribution,
Halo, counterfactual, double-counting
50
scientific advice for a healthy Canada
Three Case Examples
To demonstrate potential uses at
different levels of complexity
scientific advice for a healthy Canada
51
Example #1
National funder: Is our research getting
commercialized and can we improve that?


Step 1

Q: Are our researchers commercializing
their research effectively right now?

Q: What proportion of projects we fund
lead to a commercialized product each
year?
Step 2

Use the framework to determine where to
look for impacts.
scientific advice for a healthy Canada
52
Biomedical Research
Initiation and Diffusion of Health Research Impacts
Health
Industry
Interactions/Feedback
Health
Status,
Function,
Wellbeing,
Economic
Conditions
Canadian
Health
Research
•Biomedical
•Clinical
•Health
Services
•Population and
Public Health
•Cross-pillar
Research
Research Results
Knowledge Pool
Global Research
Healthcare
Appropriateness,
Access, etc.
Other
Industries
Government
Research
Agenda
Public
Information,
Groups
Improvements
in Health and
Well-being
Prevention and
Treatment
Determinants of
Health
Economic
and Social
Prosperity
Research
Capacity
Impacts feed back into inputs for future research
Advancing
Knowledge
Capacity Building
53
Informing Decision
Making
Health Benefits
Economic
Benefits
Example #1
National funder: Is our research getting
commercialized and can we improve that?

Step 3

Choose Sets of Indicators for each category of interest.

Informing decision making (health products industry vs other
industries)
– # patents licensed (per year)/ #projects funded (or per program)
– # funded researchers consulted by industry (per year)/# projects

Economic Benefits
– Licensing returns ($)
– Valuation of spin-out companies ($)
– Product sales revenues ($)

Step 4

Review evaluation results to identify potential enablers and
barriers to commercialization and improve the system
scientific advice for a healthy Canada
54
Example #2
Provincial Funder: Are we building Research
Capacity in our province?


Step 1

Q1: Are we developing and retaining highly
qualified research personnel in our province?

Q2: Are researchers from more than one pillar of
research being retained?
Step 2

Use the framework to identify where to look for
impacts and clarify what you mean by “research
capacity”

e.g. You could use direct indicators (people/$) and/or
indirect indicators (impacts on decision makers)
scientific advice for a healthy Canada
55
Biomedical Research
Initiation and Diffusion of Health Research Impacts
Health
Industry
Interactions/Feedback
Health
Status,
Function,
Wellbeing,
Economic
Conditions
Canadian
Health
Research
•Biomedical
•Clinical
•Health
Services
•Population and
Public Health
•Cross-pillar
Research
Research Results
Knowledge Pool
Global Research
Healthcare
Appropriateness,
Access, etc.
Other
Industries
Government
Research
Agenda
Public
Information,
Groups
Improvements
in Health and
Well-being
Prevention and
Treatment
Determinants of
Health
Economic
and Social
Prosperity
Research
Capacity
Impacts feed back into inputs for future research
Advancing
Knowledge
Capacity Building
56
Informing Decision
Making
Health Benefits
Economic
Benefits
Health Services Research
Initiation and Diffusion of Health Research Impacts
Global
Research
Interactions/Feedback
Knowledge Pool
Research Results
Health
Status,
Function,
Well-being,
Economic
Conditions
Canadian
Health
Research
•Biomedical
•Clinical
•Health
Services
•Population
and Public
Health
•Cross-pillar
Research
Health
Industry
Other
Industries
Government
Research
Agenda
Public
Information,
Groups
Healthcare
Appropriateness,
Access, etc.
Improvements
in Health and
Well-being
Prevention and
Treatment
Determinants of
Health
Economic
and Social
Prosperity
Research
Capacity
Impacts feed back into inputs for future research
57
Advancing
Knowledge
Informing Decision
Making
Capacity Building
Health Benefits
Economic Benefits
Cross Pillar Research
Initiation and Diffusion of Health Research Impacts
Health
Industry
Research
Capacity
58
Interactions/Feedback
Knowledge Pool
Health
Status,
Function,
Well-being,
Economic
Conditions
Canadian
Health
Research
•Biomedical
•Clinical
•Health
Services
•Population
and Public
Health
•Cross-pillar
Research
Research Results
Global
Research
Other
Industries
Healthcare
Appropriateness,
Access, etc.
Government
Prevention and
Treatment
Research
Agenda
Public
Information,
Groups
Improvements
in Health and
Well-being
Determinants of
Health
Economic
and Social
Prosperity
Impacts feed back into inputs for future research
Advancing
Knowledge
Informing Decision
Making
Capacity Building
Health Benefits
Economic
Benefits
Example #2
Provincial Funder: Are we building Research
Capacity in our province?

Step 3

Choose sets of indicators for categories of
interest (Capacity Building)

Infrastructure being built ($/year)
– Infrastructure grant $ attracted ($/year)

Funding attracted ($/year)
– Levels of ‘additional funding’ attracted ($/year)

Personnel trained/attracted in the province
– Graduated students per year (MSc or PhD or MD-PhD)
– # hospital staff with MSc or PhD or MD/PhD)
– # provincial government staff with MSc or PhD
scientific advice for a healthy Canada
59
Example #2
Provincial Funder: Are we building Research
Capacity in our province?

Step 3 (cont’)
 Choose sets of indicators for categories of interest (Eg.
Decision-Making and subcategories in domains of potential
impact interest)

Health related
– Use of research in provincial health care guidelines
– Survey of health policy makers

Research
– Citation analysis of successful funding applications

Health Products Industry
– Use of research in stage reports by provincial industries

General Public
– Media citation analysis

Step 4 – Use results to improve capacity development
scientific advice for a healthy Canada
60
Example #3
Federal Government: Are we achieving national health
benefit from funding Canadian health research?

Step 1

Q: How much health benefit are we achieving per $
invested in (any area of) health research in
Canada?


e.g. Cardiovascular research
Step 2:

Use the framework to define the potential
outcomes and (if attribution to Canadian health
research alone is to be determined) – try to trace
the stream that leads to Canadian ‘health benefits’
scientific advice for a healthy Canada
61
Cross Pillar Research
Initiation and Diffusion of Health Research Impacts
Health
Industry
Interactions/Feedback
$$
Knowledge Pool
Health
Status,
Function,
Well-being,
Economic
Conditions
Canadian
Health
Research
•Biomedical
•Clinical
•Health
Services
•Population
and Public
Health
•Cross-pillar
Research
Research Results
Global
Research
Other
Industries
Healthcare
Appropriateness,
Access, etc.
Government
Prevention and
Treatment
all Cdn CV Research $$
Research
Capacity
Research
Agenda
Public
Information,
Groups
Determinants of
Health
Improvements
in Health and
Well-being
Economic
and Social
Prosperity
Impacts feed back into inputs for future research
The Gross Approach
Advancing
Knowledge
Capacity Building
62
Informing Decision
Making
Health Benefits
Economic Benefits
Example #3
Federal Government: Are we achieving national health benefit
from funding Canadian health research?

Step 3: Choose sets of indicators for categories of
interest (Economic and Social Benefits)
 Indicators of Health Status



Indicators of “Determinants of Health”


Prevalence of modifiable risk factors (eg. hypertension)
Indicators of Health Benefit


Mortality (potential years of life lost)
Quality adjusted mortality (quality adjusted life years)
QALY gain per health care dollar spent in CV research
and/or CV research $ plus CV treatment $
Step 4: Improve by defining Canadian attribution
scientific advice for a healthy Canada
63
Cross Pillar Research
Initiation and Diffusion of Health Research Impacts
A
Health
Industry
t
Interactions/Feedback
$$
Knowledge Pool
Health
Status,
Function,
Well-being,
Economic
Conditions
Canadian
Health
Research
•Biomedical
•Clinical
•Health
Services
•Population
and Public
Health
•Cross-pillar
Research
Research Results
Global
Research
t
Other
Industries
r
i
Government
b
Cdn CV Research $$
u
Research
Agenda
t
i
Public
Information,
o
Groups
n
Research
Capacity
Healthcare
Appropriateness,
Access, etc.
Improvements
in Health and
Well-being
Prevention and
Treatment
Determinants of
Health
Economic
and Social
Prosperity
Impacts feed back into inputs for future research
The Attribution Approach
Advancing
Knowledge
Capacity Building
64
Informing Decision
Making
Health Benefits
Economic Benefits
What this framework can do

With strategic selection of appropriate sets of
indicators the framework can be used to trace
impacts within any of the four “pillars of health
research” or domains that cut across these pillars.

It can also be used to describe impacts at various
levels: individual, institutional, provincial, national, or
international and define the “returns on investment”
of funders by (eventually) quantifying the values of
impacts as a function of dollars put in (ROI).
scientific advice for a healthy Canada
65
What this framework can do

The combination of breadth, depth, and
flexibility suggests that the new framework
fulfils our sponsor-requested criteria to be
useful to a full range of funders/research
types, compatible with what is already in
place in Canada, transferable to international
comparisons, and able to identify the full
spectrum of potential impacts.
scientific advice for a healthy Canada
66
What this framework cannot do

The framework can help guide evaluations but it does NOT
provide the questions (or refine them).

The framework cannot resolve all complexities easily or
immediately


It is only as good as the selection of domains, indicator sets
and data within indicators



If applied prospectively, it will take time and depending on
the scope, it could become costly to apply it
to progressively resolve attribution, to achieve a balanced
perspective, to avoid the Halo effect
The menu of 66 indicators provided is only a start – more are
required.
Balanced perspectives may not provide ‘black and white’
answers
scientific advice for a healthy Canada
67
Recommendations
scientific advice for a healthy Canada
68
Recommendation #1

The framework and indicators identified
by this assessment should be used by
all funders of health research in Canada
for evaluation of their health research
impacts.
scientific advice for a healthy Canada
69
Recommendation #2

Sets of indicators and metrics chosen
from our menu should be used by all
funders of health research in Canada
for evaluation of their health research
impacts.
scientific advice for a healthy Canada
70
Recommendation #3

Canadian health research funders should begin
collaborations immediately to advance the practical
(methodological) prerequisites for measuring returns
on investment in Canada.



To standardize and refine methods, and to routinely collect
high quality and appropriate data (eg. common CV; end-ofgrant reports, etc).
A library of impact indicators and metrics should be created,
beginning with the starting menu developed here.
A core set of key health research impact questions – based
on what is practical and feasible – should be developed.


Strategic and ethically sound selection of indicator sets is
required in order to avoid biasing future health research
Evaluation questions and the choice of indicators and metrics
to be used reflect political and social choices; the motives
underpinning such choices should be made transparent.
scientific advice for a healthy Canada
71
Recommendation #4

Canada should immediately initiate a national
collaborative effort to begin to measure the impacts
of Canadian health research

Leaders from national organizations, industry &
government should organize a comprehensive
evaluation effort that engages research
communities, other stakeholders and members of
the public

One option = funders, led by one national
organization, form a national council to lead
planning and execution with a formal secretariat &
commissioned data collectors to begin this work
scientific advice for a healthy Canada
72
Recommendation #5

Canadian health research funders should collaborate
internationally to advance the “basic science of health research
impacts.”
 an international funding stream – Collaborative international
funding would be beneficial in advancing the elements
identified in Recommendation #3.
 a research plan to identify “contribution indicators” and
close “attribution gaps” – The distal indicators of the
impacts of health research are very broad, and many factors
other than funded health research can and do affect these
outcomes. It is fundamental that methodologies be
developed to separate the contribution of health research
from other causal factors.
 a research plan to use the recommended framework for
learning (impact improvement) purposes
scientific advice for a healthy Canada
73
How recommendations address
6 evaluation issues






‘Attribution’ (things other than health research
impacting health) – solution R5: research on attribution
‘Counterfactual’ (what would have happened without
the research) solution R5: controls are required
Using isolated indicators (distorts impressions) –
solution R2: indicator sets must be used
Costs of Evaluation (hard to justify) – solution R3-R5:
collaboration nationally and internationally
Double-Counting (two disciplines taking credit for same
advance) – solution R5: contributions defined
‘Halo Effect’ (seeking only positive impacts) – solution
R5: consider negative impacts within indicator sets
scientific advice for a healthy Canada
74
Many Lessons from Process

There are a number of ROI in health research projects
in progress internationally right now



we had many of the global experts involved in our
process & established excellent relationships
There are common international needs & opportunities

many potential synergies (*future potential)

accurate content and a consistent approach are
required to make informed decisions
The ‘principles’ of measuring ROI go far beyond health
research

Involving ‘others’ (e.g. Energy, etc) can add value
scientific advice for a healthy Canada
75
Making an Impact
A Preferred Framework and Indicators
to Measure Returns on Investment in Health Research
Full Report available at
http://www.cahs-acss.ca/e/assessments/completedprojects.php
Canadian Academy of Health Sciences
Académie canadienne des sciences de la santé
scientific advice for a healthy Canada