Applicability and
Transferability
Role of review authors
Belinda Burford
Rebecca Armstrong
Emma Tavender
The McCaughey Centre
VicHealth Centre for the Promotion of Mental Health and Community Wellbeing
Melbourne School of Population Health
Some definitions
• applicability and transferability are two very
important and related concepts (often used
interchangeably)
• Two sides of the one coin:
– Applicability: whether or not it would be feasible to implement this
intervention in a particular setting
• OR “Is it possible?”
– Transferability: whether we can expect to see similar results after
implementing this intervention in a particular setting
• OR “Even if it’s possible, will it work?”
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health and Community Wellbeing
Melbourne School of Population Health
Some challenges for complex reviews
• Heterogeneity, heterogeneity, heterogeneity!
• Population
– Interactions between intervention, population characteristics and
individual preferences/attitudes
– Groups rather than individuals
• Intervention
– Multiple components
• ‘active’ components?; interactions between components?
– Fidelity/integrity difficult to measure
• Comparisons
– Poorly described
• Outcomes
– Proxy/surrogate outcomes
• Setting
– Important! Often poorly described
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health and Community Wellbeing
Melbourne School of Population Health
Image from: http://www.harlemschool.com/10TH/math_pdf/math.html
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health and Community Wellbeing
Melbourne School of Population Health
Organising the information
What are some approaches you have taken in
your reviews to help the reader consider
applicability/transferability?
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health and Community Wellbeing
Melbourne School of Population Health
What do end-users need from our reviews?
• Who are your end-users?
– Differing needs
• What do they need to know1:
–
–
–
–
Could it work in my jurisdiction?
Will it work?
What would it take to make it work?
Is it worth it?
• Many resources for end-users for assessing applicability of
systematic reviews:
– SUPPORT tools to support policy-making
– Applicability/transferability checklist2
– Many, many more…
1
2
Lavis et al. Use of research to inform public policymaking Lancet 2004; 364: 1615–21 Lancet 2005
Wang et al 2005. Applicability and transferability of interventions in evidence-based public health. Health Promotion International, 21(1):76-83.
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health and Community Wellbeing
Melbourne School of Population Health
Starting from the end-user perspective:
Applicability
End-user question1
What can the review provide?2
Does the political environment of the local society allow for this intervention to be
implemented?
•Study setting/context
Is there any political barrier to implementing this intervention?
Would the general public and the targeted (sub) population accept this
intervention? Does any aspect of the intervention go against local social norms? Is
it ethically acceptable?
• Study setting/context
•Characteristics of study
population
Study setting/context
Can the contents of the intervention be tailored to suit the local culture?
•Intervention components
Are the essential resources for implementing this intervention available in the
local setting?
•Resources required
Does the target population in the local setting have a sufficient educational level
to comprehend the contents of the intervention?
•Characteristics of study
population
Which organisation will be responsible for the provision of the intervention in the
local setting?
•Organisational context
Is there any possible barrier to implementing this intervention due to the structure •Barriers to implementation
of that organisation?
identified in studies
•Who delivered the
Does the provider of the intervention in the local setting have the skill to deliver
intervention? Was training
this intervention? If not will training be available?
provided?
1
2
Questions from: Wang et al 2005. Applicability and transferability of interventions in evidence-based public health. Health Promotion International, 21(1):76-83.
Table from: Burford et al. Forthcoming
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health and Community Wellbeing
Melbourne School of Population Health
Starting from the end-user perspective:
Transferability
1
2
End-user question1
What can the review provide?2
What is the baseline prevalence of the health problem of interest in the local
setting? What was the difference in prevalence between the study setting and
the local setting?
•Baseline prevalence in study
populations
Are the characteristics of the target population comparable between the study
setting and the local setting? Is it possible that the characteristics of the target
population, such as ethnicity, socioeconomic status, educational level etc will
have an impact on the effectiveness of the intervention?
•Characteristics of study population
•Outcomes analysed by relevant
socio-demographic factors
Is the capacity to implement the intervention comparable between the study
setting in such matters as political environment, social acceptability, resources,
organisational structure and the skills of the local providers?
•Study setting/context
•Resources required
•Intervention deliverers
Questions from: Wang et al 2005. Applicability and transferability of interventions in evidence-based public health. Health Promotion International, 21(1):76-83.
Table from: Burford et al. Forthcoming
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health and Community Wellbeing
Melbourne School of Population Health
Organising the information
Place
•
•
•
•
•
Geographical
Setting
Scale
plus…
consider factors important for
your review…
Population
•
•
•
•
•
•
Intervention Characteristics
•
•
•
•
•
Types of roles involved
Interaction between roles
Delivery
Flexibility/tailoring
Component within a broader
system/program
• plus…
Age
Gender
Race/Ethnicity
Income-level
Education
plus…
Intervention Components
•
•
•
•
Number of components
Type of components
Frequency
plus…
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health and Community Wellbeing
Melbourne School of Population Health
Approaches for dealing with this in your review
• Extrinsic Approach
– Give context-dependent guidance for applying
the review findings
– 1-2 paragraphs in the discussion
– Implementation tables
• Intrinsic Approach
– Build into the review planned subgroup
analyses
• Good to do both
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health and Community Wellbeing
Melbourne School of Population Health
EXAMPLE:
Interventions for preventing obesity in children1
• Data extraction
– “implementation factors” based on “practicerelevant tool”
• Extrinsic approach
– Described these across studies in results
section and summarised in the discussion
• Intrinsic approach
– Meta analyses to explore heterogeneity
1 Waters
E, de Silva-Sanigorski A, Hall BJ, Brown T, Campbell KJ, Gao Y, Armstrong R, Prosser L, Summerbell CD. Interventions for preventing
obesity in children. Cochrane Database of Systematic Reviews 2011, Issue 12. Art. No.: CD001871. DOI: 10.1002/14651858.CD001871.pub3.
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health and Community Wellbeing
Melbourne School of Population Health
Extrinsic Approach - Results
• Effectiveness
– Maintenance / Sustainability of effects
– Equity
• PROGRESS-Plus
– Harm-adverse/unintended effects
• Implementation
– Intervention design and theoretical basis
– Process evaluation (including: intensity/dose, acceptability,
barriers/facilitators)
– Resources needed (including: who delivered, funding, hours of
implementation, other?)
– Strategies to address disadvantage/diversity
Waters E, de Silva-Sanigorski A, Hall BJ, Brown T, Campbell KJ, Gao Y, Armstrong R, Prosser L, Summerbell CD. Interventions for preventing obesity in children.
Cochrane Database of Systematic Reviews 2011, Issue 12. Art. No.: CD001871. DOI: 10.1002/14651858.CD001871.pub3.
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health and Community Wellbeing
Melbourne School of Population Health
Extrinsic approach: Discussion
Applicability of the evidence
This review included studies from high income countries as well as lower-middle- and upper-middleincome countries, with five studies conducted in countries within the latter two groupings (Thailand,
Brazil, Chile and Mexico). This means that, while predominantly conducted within high-income settings,
the findings from this review may be generalisable to a number of settings. A total of nineteen studies
specifically reported incorporating strategies to target socio-economic and/or cultural diversity or
disadvantage. One such study was conducted outside of the high-income country setting, in Chile, an
upper-middle-income country. Of the remaining eighteen studies, seven studies conducted in the USA
were of interventions targeting African American children and their communities and another two
studies targeted Native American communities. Other studies targeted participants of low socioeconomic status, or were implemented in areas of social disadvantage. By far the most common setting
for interventions included in this review were schools (43 studies). Other interventions were (or
included) home-based (14 studies), community-based (six studies), or were set in a health service (two
studies) or care setting (two studies). Eleven studies incorporated interventions across multiple settings.
Most interventions took a combined dietary and physical activity approach to obesity prevention (31
studies). As a single strategy, targeting physical activity alone was more popular (17 studies) than
targeting diet alone (seven studies). The predominant theoretical basis for interventions in this review
was behaviour change theory. Other theories represented include environmental change strategies, the
socio-ecological framework, social learning theory, health promotion theory, transtheoretical models,
and youth development and resiliency based approaches. The theoretical basis for interventions was
explicitly reported in approximately half of the included studies.
Waters E, de Silva-Sanigorski A, Hall BJ, Brown T, Campbell KJ, Gao Y, Armstrong R, Prosser L, Summerbell CD. Interventions for preventing obesity in children.
Cochrane Database of Systematic Reviews 2011, Issue 12. Art. No.: CD001871. DOI: 10.1002/14651858.CD001871.pub3.
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health and Community Wellbeing
Melbourne School of Population Health
Intrinsic approach: Synthesis
• Meta analyses to explore heterogeneity:
– Age group of participants
– Setting:
• Education only
• Education plus other settings
• Non-education
– Intervention type:
• PA
• Diet
• PA/diet combined
– Intervention duration:
• <=12 months
• > 12 months
Waters E, de Silva-Sanigorski A, Hall BJ, Brown T, Campbell KJ, Gao Y, Armstrong R, Prosser L, Summerbell CD. Interventions for preventing obesity in children.
Cochrane Database of Systematic Reviews 2011, Issue 12. Art. No.: CD001871. DOI: 10.1002/14651858.CD001871.pub3.
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health and Community Wellbeing
Melbourne School of Population Health
The next update?
• Intrinsic
– meta-regression: investigate multiple factors simultaneously
(continuous and categorical) for their possible influence on the
size of the intervention effect
• Caution:
– Generally not considered when <10 studies in meta-analysis
– Characteristics should be pre-specified (supported by scientific rationale)
[ours will be post-hoc]
– Select a small number of characteristics
– Careful interpretation (Ch 9 of Cochrane Handbook)
• Extrinsic
– Implementation tables
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health and Community Wellbeing
Melbourne School of Population Health
Using qualitative synthesis to explore
heterogeneity of complex interventions
2 reviews:
• Qualitative review on HIV/AIDS patients perspectives on adherence
• Cochrane Review of intervention trials to improve adherence to therapy in HIV/AIDS patients
Tabulated whether the interventions in the CR corresponded with the patients’ perspectives on how to help them
Candy et al. BMC Medical Research Methodology 2011, 11:124
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health and Community Wellbeing
Melbourne School of Population Health
Vindigni et al. Systematic review: handwashing behaviour in low- to-middle-income countries:
outcome measures and behaviour maintenance. Tropical Medicine and International Health.
2011;16(4):466-477
Exploring temporal study design:
Handwashing behaviour in LMIC
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health and Community Wellbeing
Melbourne School of Population Health
Handwashing: Temporal study design
• Community settings (n=16 studies)
– Formative research: average = 4.8 months
– Intervention: average = 17.2 months
– Evaluation: average = 9.6 months
• Schools (n=4 studies)
– Formative research: 1 study
– Intervention: average = 4.8 months
– Evaluation: average = 5 months
• Health care settings (n=7 studies)
– Formative research: 1 study
– Intervention: average = 2.5 months
– Evaluation: average = 3.2 months
Vindigni et al. Systematic review: handwashing behaviour in low-tomiddle-income countries: outcome measures and behaviour maintenance. Tropical
Medicine and International Health. 2011;16(4):466-477
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health and Community Wellbeing
Melbourne School of Population Health
Example from the EPOC Group
Are there different amounts of relevant evidence?
Eg. Specialist outreach clinics
in primary care and rural hospital settings
Studies
satisfying EPOC
Design Criteria
Comparative studies
not satisfying EPOC
Design Criteria
Descriptive
studies only
Total
Urban
7
12
16
35
Urban Disadvantaged
0
1
6
7
Rural
1
4
14
19
Rural Disadvantaged
1
0
11
12
9
17
47
73
Population type
Total
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health and Community Wellbeing
Melbourne School of Population Health
Specialist Outreach review
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health and Community Wellbeing
Melbourne School of Population Health