Exploring the Potential of
Mixed Methods Studies through the
COPD Year in the Life Evaluation
Abigail Baim-Lance & Francesca Solmi
on behalf of the CLAHRC YiL Evaluation Team
Internal DAHR Seminar
15 January 2015
Overview
I.
Brief Background to ‘Year in the Life’ (YiL) evaluation
II.
Using a mixed methods framework
III. Integrating quantitative and qualitative evaluation components in YiL
IV. Challenges & potential arising from our mixed methods approach:
“data” as the case study
V.
Concluding thoughts
Overview
I.
Brief Background to Year in the Life (YiL) evaluation
II.
Mixed methods framework
III. Integrating quantitative and qualitative evaluation components in YiL
IV. Challenges & potential arising from our mixed methods approach:
“data” as the case study
V.
Concluding thoughts
COPD ‘Year in the Life’ Programme (2011-2012) –
BACKGROUND
 Aimed to improve the quality of primary care for
Chronic Obstructive Pulmonary Disease
(COPD) by increasing adherence to NICE 2010
guidelines
 Took place in the four boroughs of the Outer
Northeast London (ONEL) cluster, comprising
189 practices
 Consisted of educational activities underpinned
by professional engagement & informatics
system providing benchmarking reports
NIHR CLAHRC North Thames
COPD ‘Year in the Life’ Programme – EVALUATION
OVERALL AIMS
To evaluate:
• If and which changes in primary & secondary care occurred as a result of
YiL
•
Which factors and processes influenced the Programme’s impact
• How much the programme cost
QUAL
Using a mixed methods approach
QUANT
NIHR CLAHRC NORTH THAMES
Overview
I.
Background to Year in the Life evaluation
II.
Using a mixed methods framework
III. Integrating quantitative and qualitative evaluation components in YiL
IV. Challenges & potential arising from our mixed methods approach:
“data” as the case study
V.
Concluding thoughts
Using a mixed methods framework
 Health service research evaluates interventions happening in complex
and multi-level contexts.
 Mixed-methods designs can be a valuable tool to untangle these
processes through:
Integration of
qualitative and
quantitative approaches
at all stages of research
Strengthening reliability
of data and validity of
findings
Broadening and
deepening
understanding of
processes and contexts
Despite its many advantages, integration of qualitative and quantitative research is
often difficult to achieve and limited in use.
In YiL evaluation, we aimed to apply a strong mixed methods framework to
inform our methods, and interpret our results.
Bamberger M. 2012. Introduction to Mixed Methods in Impact Evaluation. Impact Evaluation Series (no.3),
Interaction and the Rockefeller Foundation.
Fetters, M. Curry, L. & Creswell, J. 2013. “Achieving Integration in Mixed Methods Design.” HSR 48:6
Using a mixed methods framework – RESEARCH
DESIGN
Basic designs:
 Exploratory sequential
QUALITATIVE
QUANTITATIVE
 Explanatory sequential
QUANTITATIVE
QUALITATIVE
 Convergent
QUANTITATIVE
QUALITATIVE
Advanced designs:

Multistage mixed-methods frameworks: combination of basic designs
at different stages of the study
Methods and analyses:




Connecting:
Building:
Merging:
Embedding:
One database links to another through sampling
One database informs data collection approach of the other
Two databases are brought together for analysis
Data collection and analysis link at multiple points
NIHR CLAHRC NORTH THAMES
Fetters, M. Curry, L. & Creswell, J. 2013. “Achieving Integration in Mixed Methods Design.” HSR 48:6
Overview
I.
Background to and aims of Year in the Life evaluation
II.
Mixed methods framework
III. Integrating quantitative and qualitative components in YiL
IV. Challenges & potential arising from our mixed methods approach:
“data” as the case study
V.
Concluding thoughts
Integrating quantitative and qualitative components in YiL
Multi-stage mixed-methods framework
PHASE 1
Exploratory Sequential
PHASE 2
Convergent
PHASE 3
Explanatory Sequential
(ongoing)
QUAL
QUAL
QUAL
QUANT
QUANT
QUANT
NIHR CLAHRC NORTH THAMES
Integrating quantitative and qualitative components in YiL
PHASE 1
Exploratory Sequential
AIMS
To describe YiL (what was YiL a
nd how did it work?)
METHOD
QUAL
Design, implementation: 14 planner
interviews, 300+ documents review
How Qualitative results informed quantitative analyses
1. YiL Interventions and theories of change (BUILDING)
2. Key COPD processes of care (BUILDING)
3. timeline to determine ‘before, during, after’ periods (BUILDING)
(i) To evaluate changes in
Emergency Hospital
Admissions for COPD;
(ii) To evaluate changes
in Primary care
processes; (iii) costs
QUANT
Design: variable and analysis model
selection
NIHR CLAHRC NORTH THAMES
Integrating quantitative and qualitative components in YiL
PHASE 2
Convergent
AIMS
Phase II interviews to: 1) explore YiL impac
t on primary care; 2) compare with PC QI s
trategies; 3) speak to emerging metrics
(admissions, processes of care, contexts of
implementation of programme)
METHOD
QUAL
Design, implementation: 10 primary
care interviews, ranging in YiL
participation
How Qualitative and Quantitative research interacted
1. Initial QUANT results showing limited impact of YiL on changing hospital admission rates led to questions about providers’ views on
admissions (BUILDING)
1. Preliminary results on practice involvement informed both phase II interviews and development of new metrics (CONNECTING/BUILDING)
Overall aims maintained + Better
understand the role of practice involvement
in YiL on primary and secondary care
outcomes
QUANT
NIHR CLAHRC NORTH THAMES
Implementation: Data analyses
also expanded to capture the effect
of practice involvement on the
outcomes
Integrating quantitative and qualitative components in YiL
AIMS
Using increasingly convincing QUANT
results to explore interview data
PHASE 3
Explanatory Sequential
(ongoing)
QUAL
METHOD
Implementation: triangulation and
further learning
How Quantitative results informed qualitative analyses
1. examining why number of service visits per patient might be declining while at the same time, annual reviews are increasing (MERGING)
QUANT
Implementation: finalisation and
refinement of analyses
Interpretation of results
NIHR CLAHRC NORTH THAMES
Overview
I.
Background to and aims of Year in the Life evaluation
II.
Mixed methods framework
III. Integrating quantitative and qualitative evaluation components in YiL
IV. Challenges & potential arising from our mixed methods approach:
“data” as the case study
V.
Concluding thoughts
Challenges in mixed-methods integration findings:
INFORMATICS/GP DATA EXTRACTION
QUALITATIVE REVEALS

Phase I: data/informatics system drove YiL potential, BUT
!

Raft of problems with implementation, questions raised about validity of
primary care extracted information
Phase II: confirmation that GP-led data recording does not always conform with
regional-level extraction assumptions (e.g., templates tailored to make clinical
sense)
QUANTITATIVE CONUNDRUMS

Data issues have also emerged throughout the study (extractions, complexities
in interpreting dataset) AND

How to take on board concerns about data? Data checks
Potential and challenges
Advantages
 Structured methodological approach to evaluation
 Better understanding of ‘knowns’ and ‘known unknowns’
Challenges
 Retrospective design

Having to be an ‘archeologist’!

Recall patterns
 Space for non-integration?
Overview
I.
Background to and aims of Year in the Life evaluation
II.
Mixed methods framework
III. Integrating quantitative and qualitative evaluation components in YiL
IV. Challenges & potential arising from our mixed methods approach:
“data” as the case study
V.
Concluding thoughts
Concluding Thoughts
 Enriches, though also complicates understandings of research topics
 Requires flexibility and unexpected adjustments
 (might) Expand the scope of analyses to include what is typically
outside of quantitative work
 What have been your experiences with mixed methods?
COPD research team
 Dr Jessica Sheringham (UCL)
 Prof Naomi Fulop (UCL)
 Prof Steve Morris (UCL)
 Dr Cono Ariti (The Nuffield Trust)
 Dr Martin Bardsley (The Nuffield Trust)
 Dr Abigail Baim-Lance
 Dr Francesca Solmi
‘Year in the Life’ Steering Group