How qualitative research
contributes to evaluation
Professor Alicia O’Cathain
ScHARR University of Sheffield
22 June 2015
What is qualitative research?
• Normal in evaluation
• Understanding not
measuring
• Set of methods
– Focus groups
– Semi-structured or in
depth interviews
– Non-participant
observation
– Diaries
What is evaluation?
• Researcher led
evaluation
• Policy evaluation
Researcher-led
• MRC Framework developing
an evaluating complex
interventions
• ACTIF programme
– 5 years
– RCT
– Qualitative at each phase
O’Cathain A, Thomas KJ, Drabble SJ, Rudolph A, Hewison J. What can
qualitative research do for randomised controlled trials? A systematic
mapping review. BMJ Open 2013;3:e002889
Intervention
Trial design
and conduct
Measures
Outcomes
Health
conditions
Intervention n=254
Develop
n=48
Describe it
n=10
Implementation in
real world n=4
Fidelity,
reach and
dose n=12
Understand
how it works
n=23
Feasibility and
acceptability
n=83
Acceptability
in principle
n=32
Value and
benefits
n=42
Trial design and conduct n=54
Recruitment
Diversity
Impact on staff,
researchers, patients
Adapting to local
circumstances
Participation in
trials
Ethics/informed
consent
Acceptability in
principle
Acceptability in
practice
Potential value
Bias
Efficiency
Potential value
Avoidance of measurement bias
Faster recruitment
Ethics
Saves money
Trials sensitive to human beings
Improved informed consent
Implementation
Facilitates replicability of intervention in the real world
Facilitates transferability of findings in the real world
Interpretation
Relevance
Explains trial findings
Ensures interventions meet the needs of health
professionals and patients
Success
Validity
Makes a trial successful, feasible, viable
Improves internal validity
Improves external validity
Maximising value…
1. Do it early
– 28% pre-trial
•
•
•
•
•
Intervention development 100%
Acceptability of intervention in principle 25%
Acceptability of intervention in practice 24%
Recruitment 18%
Breadth of outcomes 0%
…otherwise its about future trials
2. Publish learning for specific trial or future trials
3. Think beyond interviews: non-participant
observation
4. Try iterative or dynamic or participatory
approaches at feasibility phase
5. Not just complex interventions
• 38% of 104 data extracted were drugs or devices
6. Think about the range of work
Problems with quantitative only
• Null RCTs….explain findings
(context, mechanisms of
action, implementation)
• Failed trials….prevent this at
pilot stage
• It works but what is ‘it’?
…..qualitative can fix
Policy evaluation
• Learning from early adopters (feasibility)
• Stakeholder reception (acceptability)
• Service delivered (implementation, workforce)
Useful but challenges remain
– Fast evaluation
– When to evaluate
– Moving target
– Replacement of difficult-to-measure outcomes
with understanding of processes
Conclusions
• Useful contribution no matter what type of
evaluation – essential due to complexity
• Can help to fix problems faced in researcherled evaluation
• Challenges in policy evaluation need reflection