Randomised trials of complex public
health interventions: challenges,
prospects, and example: NPRI trial of the
Strengthening Families Programme 10-14
Laurence Moore
MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
What is a complex (social / public health)
intervention?

Complex interventions involve “multiple, synergistic
components” [Bonell et al, 2012]

Complex interventions interact with context

There is therefore complexity both within the
intervention model AND in its dependence on context,
receipt, implementation; the wider system

This presents challenges for the conduct of RCTs and
the value of RCT evidence
MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
RE-AIM framework
Adoption
Efficacy
Effectiveness
Building Programs
and Policies
with a Large Public
Health Impact
Reach
Implementation
Maintenance
3
MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
RECOMMENDED PURPOSE OF
TRANSLATION/EFFECTIVENESS RESEARCH
To determine the characteristics of interventions /
policies / programmes
that can:
•
Reach large numbers of people, especially those
who can most benefit
•
Be widely adopted by different settings
•
Be consistently implemented by staff members
with moderate levels of training and expertise
•
Produce replicable and long-lasting effects (and
minimal negative impacts) at reasonable cost
Glasgow RE et al Am J Public Health. 2003;93:1261–1267
MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
RE-AIM
• ‘Effective interventions’ that are not adopted will have
no impact
• Frequent failure of ‘evidence based interventions’ which
may not be transferable and cannot be implemented
without commitment, engagement, resources
• ‘highly unlikely that interventions that are successful in
efficacy studies will do well in effectiveness studies or
real world application’
• A lot of information on efficacy and very little on
effectiveness
• Call for a moratorium on efficacy trials (Kessler and
Glasgow 2011)
MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Standard translational model
• T1, T2, bench to bedside…….
• Efficacy, Feasibility, Pilot, Effectiveness……
• Far from optimal for complex interventions
• Production line for ‘effective interventions’ that
generally don’t work!
• Provides a strong basis for critique of value of
RCTs of complex interventions
MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
We need complex interventions that:
• Are resilient to contextual variation and therefore more
transferable
• Flexible
• Standardised function, flexible form
and / or
• Have a clear program theory and thus a clear
understanding of contextual dependencies, (target
group, resources, system requirements etc) and system
impacts
MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Not just ‘what works?’ but Why?
• Theory of the problem
• Causal processes
• Intervention / logic model
• Key components and their impact
• Program theory
• How will the intervention bring about
change, interact with context
MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Realist critiques & alternatives to RCTs
[e.g. Pawson & Tilley 1997; Pawson, 2013]
MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Realist Evaluation
– Pawson &
Tilley
• Purpose of evaluation “as informing
the development of policy and
practice”
• Experiments can identify the mean
effect but this is rarely, if ever,
evenly produced
• Mechanism + Context = Outcome
• “what works, for whom, and in what
circumstances?” and even better, to
also help us to understand “why?”
• Need to develop and continually
update program theory
MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
What would a “Realist(ic) RCT” look
like?
1. Examine the effects of intervention
components separately and in
combination
2. Examine pathways via which change
occurs
3. Examine how effects vary by sub-groups
and with context more systematically
4. Draw on complementary quantitative and
qualitative data to answer different RQs.
5. Build and test mid-level, program theories
about how interventions work in context
MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
New MRC Process Evaluation Guidance is focused
on “what works for whom in what context & why?”
Context
• Moofactors which affect (and may be affected by) implementation, intervention mechanisms and
Contextual
outcomes. Causal mechanisms present within the context which act to sustain the status quo, or lead
to change.
Description of
intervention
and its causal
assumptions
Logic model
Theory of
action/ change
Implementation
How delivery is achieved
(training, resources etc..)?
What is delivered?

Fidelity

Dose

Adaptations

Reach
Moore et al 2015
MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Mechanisms of impact
Participant responses
Intervention mediators
Unanticipated pathways / consequence
Outcomes
Explanatory - Pragmatic
• Explanatory
• Efficacy
• Mechanisms
• Standardised
delivery & dose
• Highly specified
inclusion and
exclusion criteria
• Per protocol
analysis
MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
• Pragmatic
•
•
•
•
Effectiveness
Outcomes
‘Black box’
Some variation in
adherence
• Less restrictive
inclusion/exclusion
• Intention to treat
analysis
Pragmatic - Realistic
• Pragmatic
•
•
•
•
Effectiveness
Outcomes
‘Black box’
Some variation in
adherence
• Less restrictive
inclusion/exclusion
• Intention to treat
analysis
MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
• Realistic
• Effectiveness
• Context,
Mechanisms,
Outcomes
• Program theory
• Heterogeneity
• Mediation and
moderation
analyses
• Mixed methods
• Hypothesis testing
and generation
“Realistic RCT” examples:
ASSIST peer-led smoking intervention
59 schools randomised, found to be effective & now rolled out
across UK [Campbell et al., 2008]
Fruit tuck shop trial
43 schools randomised to run fruit-only shops. Significant
interaction between shops and school policy [Moore et al., 2008]
Primary School Free Breakfasts Initiative
111 schools randomised and found to be effective in most
deprived areas [Murphy et al., 2010]
Strengthening Families Programme
748 families randomised and results due in 2014 to inform Welsh
Government policy [Segrott et al., 2014]
MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Conclusions
• Complex interventions depend on and influence external
factors
• Traditional translational model is not optimised for
complex interventions; external factors are considered
too late
• Program theories, which include external factors, rather
than closed logic / intervention models, are required
• Exploratory (Feasibility and Pilot) studies need to assess
and optimise evaluation methods AND program theory
before Phase III trials are commissioned
• Realistic trials, which not only provide an unbiased
average effect estimate but also test and refine
program theory, are a potential important extension of
existing pragmatic trial methods
MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Laurence Moore
J. Segrott, D. Gillespie, I.Humphreys, J. Holliday, S. Murphy,
Z. Roberts, J. Scourfield, D. Foxcroft, H. Rothwell, M.Lau,
C. Hurlow, C. Phillips, H. Reed, K. Hood
NPRI Randomised controlled trial of the
Strengthening Families Programme 10-14 UK in
Wales UK: Results
DECIPHer & South East Wales Trials Unit, Cardiff University,
Swansea University, Oxford Brookes University, University of Glasgow
SFP10-14 intervention
• Universal substance misuse prevention intervention
• Delivered to groups of families with children aged 10-14
• Focuses on parenting, family functioning and young
people’s peer-resistance skills
• Weekly sessions of 2.5 hours for seven weeks, child & family
sessions
• Evidence of effectiveness from US trials
– Validated ‘effective’ program
– Blueprints for Healthy Youth Development provides a registry of
evidence-based positive youth development programs
– Communities that Care
SFP10-14 intervention
• Contested evidence base
– Does it work?
– “It works!“– but why? And for whom and under what
circumstances?
– Limited evidence of successful transferability
– No formal logic model
– Flawed implementation - Sweden
– Variation in context and targeting
• Variable TAU
• Universal / high risk
• SFP10-14 adapted for use in UK
– Initially targeted at high risk families in UK implementation
(Barnsley)
SFP10-14: group composition strategy
•
•
•
•
•
•
•
Adapted UK programme delivered in Cardiff
Identified implementation risks from comprising groups of families with
high levels of challenge
Developed ‘mixed families’ (70/30) approach (Proportionate
Universalism)
Aims to form groups of (5-12 families) comprised of
– Families from the General Population with no challenges in a group
setting (70%)
– Families who may experience/present challenges within a group
setting (30%)
Examples of challenges: young person not attending school; ADHD; low
literacy skills; learning difficulties
70/30 mix aims to maximise fidelity, promote positive group
dynamics/behaviour change, and maximise retention
Terminology used this morning: ‘General Population’ (GP) / ‘Families
with Challenges’ (FWC), 70/30
Trial Design and Implementation
Trial Design
•
Pragmatic RCT with families as the unit of randomisation
•
Embedded process and economic evaluations
•
Comparing normal care with normal care + Strengthening Families
Programme 10-14
•
SFP10-14 delivered by agencies such as charities and local government
•
Research fieldworkers embedded in programme delivery teams during
recruitment
•
Knowledge exchange structures built into trial from outset: strong
support from, and partnership with Welsh Government
Trial objectives
Primary Objective
• To ascertain the impact of the SFP10-14UK on alcohol misuse in
adolescents.
Secondary objective: Ascertain the impact of SFP10-14Uk on …
• drug misuse, and smoking behaviour in adolescents
• school performance
• alcohol initiation, drink-related problems, and other alcohol-related
behaviours
Tertiary objectives
• What impact does SFP10-14UK have on mental health and well being?
• What impact does the SFP10-14UK have on protective factors for
alcohol and tobacco use/misuse located in the family, such as family
functioning, parenting and young people’s peer resistance skills?
• What are the costs associated with the SFP10-14UK and to what extent
can it be regarded as an efficient use of public funds?
• How can SFP10-14UK best be implemented and is there important
variation in delivery and receipt?
Trial Outcomes
•
Primary outcomes: number of occasions that young people report
having drunk alcohol and been drunk during the last 30 days,
dichotomised as ‘never’ and ‘1-2 times or more’
•
Secondary outcomes: use of cannabis, weekly smoking, age of alcohol
use initiation, frequency of drinking, frequency of different types of
alcoholic drinks, drink related problems, and GCSE performance
•
Tertiary outcomes: age of initiation of use of drugs and tobacco; family
functioning, parenting, and peer pressure resistance skills in young
people; wellbeing and stress; and depression.
Data collection from families
Questionnaires with parents
and young people
0 months
Observation of programme
sessions
Telephone interviews with
parents
9 months
Telephone interviews with
parents
15 months
Questionnaires with
parents & young people
24 months
Recruitment and Retention
Recruitment
•
Family recruitment completed in June 2012
•
715 families were recruited in total:
– 361 families (50.5%) allocated to intervention
– 354 families (49.5%) allocated to control
•
The 715 families were made up of:
– 919 adults (459 control, 460 SFP)
– 931 young people (454 control, 477 SFP)
Findings: Recruitment and retention
715 families randomised
931 Young People (YP), 919 Parents/Carers (PC)
354 families randomised to Control
454 YP, 459 PC
361 families randomised to SFP
477 YP, 460 PC
240 PCs completed 9 month
telephone follow-up (52%)
288 PCs completed 9 month
telephone follow-up (63%)
166 PCs completed 15 month
telephone follow-up (36%)
220 PCs completed 15 month
telephone follow-up (48%)
354 YPs completed 24 month faceto-face follow-up (78%)
316 PCs completed 24 month faceto-face follow-up (69%)
403 YPs completed 24 month faceto-face follow-up (85%)
377 PCs completed 24 month faceto-face follow-up (82%)
Project SFP Cymru Main Trial Findings: UK SBM Annual Scientific Meeting, Nottingham 2014
24 month follow up
•
Follow up completed July 2014
•
24 month follow up (adults):
– 693 adults completed 24 month interview (316 Control, 377
SFP)
– The final completion rate of 75.4% (68.8% Control, 82.0% SFP)
•
24 month follow up (young people):
– 757 young people completed 24 month interview (354 Control,
403 SFP)
– The final completion rate of 81.3% (78.0% Control, 84.5% SFP)
Findings: Baseline Characteristics
• No differences of major note between trial arms
Demographic
Young person (YP)
Parent/carer (PC)
Median age (IQR)
12 (10 to 13)
37 (32 to 43)
% Female
46
77
% White British
85
81
Substance use in YPs
%
Substance use in PCs
%
Ever tried a cigarette
27
Smoker
52
Usually smoke > 6 cigarettes a week
5
Never drink alcohol
19
Had a proper alcoholic drink
31
Been a little bit drunk
17
High risk from problematic
drinking (AUDIT-C)
34
Been very drunk
7
Tried drugs
5
Used drugs at least once in
lifetime
34
Project SFP Cymru Main Trial Findings: UK SBM Annual Scientific Meeting, Nottingham 2014
Economic and Process Evaluations
Programme cost, Attendance
•
C. £1550 per family, variable across areas
•
361 families were randomised to SFP
•
119 (33%) attended all 7 weeks
•
218 (60%) received the intervention, defined as attending at least 5
sessions without missing more than 1 session in a row
•
74 (20%) did not attend any sessions
•
Overall, 287 attended at least 1 session
Process Evaluation
•
•
•
•
•
•
Described fidelity, dose delivered, dose received, reach and context
Used mixed methods approach to explain variation across trial sites
and key influences on implementation, and interpret outcomes
Fidelity (content coverage ): facilitators reported 95.8% of activities
as mostly/fully covered; for sample of sessions also observed by
researchers their scores agreed with facilitators 83% of the time
Group size: 84% (n=47) of programmes enrolled 5-12 families
Group composition: 37% (n=21) programmes achieved intended
group composition (70% families from General Population, 30%
families with challenge in group setting), but remaining groups also
achieved mix of families
Staffing: good adherence to staffing levels and delivery of Weeks 17 by same staff
Analysis
Outcome Results
Summary
– Challenging trial conducted with high follow-up rates over
two years and minimal risk of bias
• MAJOR EFFORT
– Pragmatic trial, good fidelity of programme delivery,
although groups sometimes small and variable
composition
– No significant differences between groups in primary
outcomes
• CONSISTENT WITH RECENT EUROPEAN STUDIES
– SFP group had better parenting/family outcomes but
higher anxiety/discomfort
– Consistent pattern of benefit for families with challenges
but poorer outcomes for general population
Summary
• Realistic trial:
– Mediation analysis of tertiary parenting variables on
outcomes
– Analysis of variations in programme delivery and
relationship with outcomes
– Investigate candidate explanations for the differential
effects by FWC
– Recommendations for logic model and future
implementation
– Update programme theory and understanding of for
whom and under what circumstances the programme’s
effectiveness may be maximised
Acknowledgements
Project SFP Cymru is funded by the National Prevention
Research Initiative (http://www.npri.org.uk)
Funding partners: Alzheimer’s Research Trust; Alzheimer’s
Society; Biotechnology and Biological Sciences Research
Council; British Heart Foundation; Cancer Research UK;
Chief Scientist Office, Scottish Government Health
Directorate; Department of Health; Diabetes UK; Economic
and Social Research Council; Engineering and Physical
Sciences Research Council; Health & Social Care Research &
Development Office for Northern Ireland; Medical Research
Council; The Stroke Association; Welsh Government; and
World Cancer Research Fund
Additional funding for programme delivery was provided by
the Welsh Government. Cardiff Drug and Alcohol team
provide financial support for recruitment through schools.
MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Acknowledgements: Rona Campbell, Simon Murphy, Liz Waters, Chris Roberts, Jeremy
Segrott, James White, Kerry Hood, Sharon Simpson, Chris Bonell and Adam Fletcher
• Bonell, C., Fletcher, A. et al. (2012) Realist randomised controlled trials: a new approach to evaluating
complex public-health interventions. Social Science & Medicine 75(12): 2299-2306.
• Bonell, C., Fletcher, A. et al. (2013) Methods don’t make assumptions, researchers do: A response to
Marchal et al. Social Science & Medicine 94: 81-82.
• Campbell, M., et al., (2000) Framework for design and analysis of complex interventions to improve
health. BMJ 321(7262):694-6.
• Campbell, R, et al., (2008) An informal school-based peer-led intervention for smoking prevention in
adolescence (ASSIST): a cluster randomised trial. Lancet 371:1595-602
• Craig, P., et al. (2008) Developing and evaluating complex interventions: new guidance. London: MRC.
• Glasgow, R., et al., (2003) Why don’t we see more translation of health promotion research to practice?
Rethinking the efficacy-to-effectiveness transition. Am J Public Health. 2003;93:1261–1267
• Kessler and Glasgow (2011) A proposal to speed translation of healthcare research into practice. Am J
Prev Med 40(6):637-644.
• Marchal, B., et al., (2013) Realist RCTs of complex interventions - an oxymoron. Social Science & Medicine
94: 124-128.
• Moore, G., et al., (2015) Process evaluation of complex interventions: MRC guidance BMJ 350;h1258
• Moore, L, Tapper, K. (2008) The impact of school fruit tuck shops and school food policies on children's
fruit consumption: a cluster randomised trial of schools in deprived areas. J Epidemiol Community Health
62:926-31
• Murphy, S., et al. (2010) Free healthy breakfasts in primary schools: a cluster randomised controlled trial
of a policy intervention in Wales, UK. Public Health Nutrition14(2):219.
• Pawson, R., & Tilley, N. (1997). Realistic evaluation. London: Sage.
• Pawson, R. (2013) The Science of Evaluation. London: Sage.
• Segrott, J., et al., (2014) Preventing substance misuse: study protocol for a RCT of SFP 10-14 UK. BMC
Public Health 14:49.
MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.