Systematic reviews in the social sciences
Sarah Whitehead, Cardiff Institute of Society and Health
Outline
• Why do systematic reviews in the social
sciences?
• Examples of types of reviews:
 Correlates of physical activity
 Effectiveness of tobacco harm reduction approaches
 Barriers and facilitators to tobacco harm reduction
• Overview of different types of evidence generated
• Lessons learnt
Why do systematic reviews in the social
sciences? Chaos in the brickyard
•
•
•
•
•
•
Goal of scientific research to discover explanations
Research findings to be assembled like bricks to create edifices
Initially bricks carefully made and only those required for the edifice
produced
Over time, brick making became an end in itself until the land was
flooded with bricks, many of them random and useless
Now we need to search through so many bricks it’s difficult to find
proper ones for the task
“Saddest of all, sometimes no effort was made even to maintain the
distinction between a pile of bricks and a true edifice”1
1. Forscher 1963
Why do systematic reviews in the social
sciences?
• Systematic reviews allow us to make sense of the
chaos in the brickyard
• They allow us to:
 develop a comprehensive understanding of research
to date
 identify areas of evidence that can be translated into
policy and practice
 determine which areas are in need of further research
Examples of types of reviews
Example 1: correlates of physical activity
• Research question:
 What factors are associated with physical activity
participation among adolescent girls?
• Suitable design:
 Correlates review of quantitative studies reporting a
measure of physical activity and at least one potential
correlate
Example 1: methods
• 4 databases searched, manual searches of journals
• Inclusion criteria:






Quantitative research design
Published 1999-2003
Participants in age range 10-18 years
Results available for females separately
Articles published in English language
Any type of physical activity
Example 1: methods (cont’d)
• Potential correlates classed as demographic/biological,
psychological, behavioural, social/cultural, or physical
environmental
• Variables classed as related/ not related to physical activity
• Strength of association noted1
• Variables only included if studied three or more times2
• Quality of physical activity measure coded2:
 Self-report of poor or unknown reliability/validity
 Self-report with acceptable reliability/validity
 Acceptable objective measure
1. Cohen 1988
2. Sallis et al. 2000
Example 1: findings
• 50 published papers yielding 51 independent
samples
• Majority of studies (63%) carried out in US
• Mean sample size (females only) = 1,280
(range 22-9,039)
• 41 (80.4%) cross-sectional in design
• Majority (61%) used self-report with acceptable
reliability/validity
Example 1: findings
• Demographic variables:
 Non-white ethnicity (-)
 Age (-)
 SES (+)
• Psychological variables:




Enjoyment (+)
Perceived competence (+)
Self-efficacy (+)
Physical self-perceptions (+)
• Behavioural variables:
 Smoking (-)
 Organised sport
involvement (+)
• Social/cultural variables:
 Family/parental support (+)
 Father’s physical activity (+)
• No consistent trends for
environmental variables
Example 1: conclusions
• Several variables related with PA across 3+ studies
• Many potential correlates understudied
• Several weak designs and measures of poor or
unknown validity
• Most promising modifiable correlates:
 Family support
 Positive psychology
 Organised sport
Example 2: effectiveness of tobacco harm
reduction approaches
• Research questions:
 How effective are tobacco harm reduction approaches with the
intention of quitting (i.e. ‘cutting down to quit’ or ‘reduction to stop
smoking’), with and without assistance?
 How effective are long-term harm reduction approaches without
the prior intention of quitting (i.e. reducing consumption without
the aim of quitting), with and without assistance?
• Suitable design:
 Two effectiveness reviews of experimental studies of tobacco
harm reduction approaches, e.g. randomised controlled trials,
controlled trials
Example 2: methods
• 21 databases, 28 websites, range of 'snowballing’ techniques
• Included populations:
 People of all ages wanting to quit smoking but unable to do so
abruptly / not willing to quit but wanting to reduce/ stop temporarily
 Participants in cut down to quit/ reduction/ temporary abstinence
interventions
• Included interventions:




Pharmacotherapies licensed for cutting down/ temporary abstinence
Other non-tobacco nicotine containing products, e.g. e-cigarettes
Behavioural support or counselling
Self-help
Example 2: methods
• Data extraction to capture:




Study design, setting and participants
Intervention details
Methods of analyses
Outcomes
• Quality appraisal to assess:
 Methodological quality
 External validity
 Narrative synthesis and meta-analyses to summarise
findings
Example 2: included studies and types
Cut down to quit
Cut down without quitting
•
•
•
•
11 studies
 10 RCTs or quasi-RCTs
 1 uncontrolled before &
after
Quality
 Generally moderate
Applicability
 Only one UK study
•
•
•
45 studies
 29 RCTs or quasi-RCTs
 3 non RCTs
 2 controlled before & after
 10 uncontrolled before & after
 1 secondary analysis
1 systematic review
Quality
 Generally moderate
Applicability
 5 UK studies
 6 in UK-comparable countries
Example 2: types of evidence examined
• Effectiveness of pharmacotherapies in helping
people cut down to quit or cut down indefinitely
without the aim of quitting
• The effectiveness of behavioural support,
counselling or self-help
• Whether different approaches had differential
impacts on different groups
Example 3: barriers and facilitators to
tobacco harm reduction approaches
• Research question:
 What are the barriers and facilitators to implementing
tobacco harm reduction approaches, including users’
and providers’ perspectives?
• Suitable design:
 Mixed methods review of qualitative and quantitative
views and opinions studies
Example 3: methods
• 21 databases, 28 websites, range of 'snowballing’ techniques
• Included populations:
 Smokers interested in THR or who take part in THR interventions, or
those who provide opinions or experiences regarding THR
approaches
 Service providers, healthcare personnel and policy makers who may
deliver/ commission/ refer smokers to THR interventions
• Included study types:
 Views studies relating to cut down to quit/ long term reduction/
temporary abstinence intervention studies previously identified
 Views studies relating to tobacco harm reduction approaches in
general
Example 3: methods
• Data extraction to capture:
 Study design, setting and participants
 Methods and analytical approach
 Outcomes / key themes
• Quality appraisal to assess:
 Methodological quality
 External validity
• Thematic synthesis of views, experiences and perspectives
• Mapping of whether effectiveness studies addressed the
identified barriers and facilitators to THR approaches
Example 3: included studies and types
• 41 studies





22 cross-sectional surveys
13 qualitative studies
3 mixed methods studies
2 process evaluations
1 longitudinal study
• Quality
 Generally moderate
• Applicability
 9 UK studies and 5 UK applicable
Example 3: types of evidence examined
• Factors that might act as barriers or facilitators to tobacco
harm reduction approaches
• Smokers’/ their families’ and healthcare professionals'
views and experiences on whether specific tobacco harmreduction approaches are perceived to have a differential
impact on particular groups
• Smokers’/ their families’ and healthcare professionals'
views and experiences on the potential or actual
unintended consequences from adopting a harmreduction approach
Summary of different types of evidence that
systematic reviews can generate
• Correlates of behaviours or outcomes
• Effectiveness of interventions / intervention type
studies
• Views and opinions
• All can be useful in informing policy and practice
and identifying evidence gaps where further
research is needed
Lessons learnt
• Develop a solid search strategy
• Identify appropriate tools for the type of review
being undertaken, for example quality
assessment measures and approaches to
synthesising the data;
• Most importantly – work with a team of
experienced information specialists
Acknowledgement
Sarah Whitehead’s role at
Cardiff Institute of Society and
Health is supported by the
Welsh Assembly Government’s
National Institute for Social
Care and Health Research