Conducting Systematic Reviews
from a KT Perspective
Important information before we start…
• Turn speakers off
• Mute your phone when you are not speaking
• Turn on the captioning
Conducting Systematic Reviews from
a KT Perspective:
Developing evidence-based information for
researchers, clinicians and consumers
Model Systems Knowledge
Translation Center
University of Washington
The Webcast Series
•
Part 1: Conducting Systematic Reviews from a KT Perspective:
lessons learned from 14 systematic reviews collaboratively written
with faculty from the Model Systems in TBI, SCI and Burn about
conducting reviews from a KT perspective.
•
Part 2: The MSKTC Approach to Systematic Reviews: using tools
and forms developed by the MSKTC to conduct systematic reviews
of the evidence.
•
Part 3: Using the MSKTC Systematic Review database: a
demonstration of a database with web interface to expedite the
creation of tables of evidence. This database will be available to
NIDRR grantees.
Using Elluminate
Elluminate is an interactive, real-time, online system.
• Interact with moderators
• Raise hand
• Laugh
• Applaud
• Show disapproval
• Show confusion
• Chat
• Ask questions, offer comments
– Using the Chat space, tell us which Model System you are associated with
• Poll
• Respond to quick polls or surveys
Quick Poll
• Have you ever conducted a systematic review?
Goals for this Webcast
• Conduct systematic reviews from a knowledge
translation perspective
– Learn about different types of and approaches to
systematic reviews
– Learn about conducting systematic reviews that
result in information that is useful to multiple
audiences, including clinicians and consumers
– Learn about some of the major challenges in
conducting systematic reviews in rehabilitation
What is Knowledge Translation?
• An active process of ensuring that new
knowledge gained through the course of
research ultimately improves the lives of
people with disabilities, and furthers
their participation in society (NIDRR, 2005).
• Process that takes us “from knowledge
creation through implementation and
impact.”(Sudsawad, 2007)
Current Barriers to KT
• Consumers: difficulty finding and evaluating research to
guide health care and rehabilitation decision making.
• Health care professionals: lack time, resources, or skill to
translate research to practice and rate evidence
• Researchers: academic culture does not support publication
of non-refereed materials such as consumer education
materials
• Policy makers: disconnect from researchers—want “sound
bytes” rather than complex analysis
• Gov’t agencies: political or agency constraints may outweigh
evidence in guiding research and policy agendas
8
What is a systematic review?
A systematic review is an attempt to gather all empirical
evidence that fits pre-specified eligibility criteria in
order to answer a specific research or clinical question.
• Key characteristics:
– A clearly stated set of objectives with pre-defined eligibility
criteria;
– An explicit, reproducible methodology;
– A systematic search that attempts to identify all the studies
that meet the eligibility criteria;
– An assessment of the validity of the findings
Source: Higgins JPT, Green S (editors). Cochrane Handbook for Systematic
Reviews of Interventions Version 5.0.1 [updated September 2008]. The
Cochrane Collaboration, 2008. Available from www.cochranehandbook.org.
Systematic Reviews from a KT Perspective
• Create a framework
– The systematic review is not the end product
– It is one source of information
• Other sources include professional judgment or consensus
• What do you do when the evidence isn’t adequate?
– Gather other best evidence to communicate to your
audience, .e.g., consumers
– Saying ‘there is no evidence’ is not a satisfactory or useful
answer
• Ensure the clinical relevance of the question
• Use a pragmatic approach in conducting systematic
reviews from a KT perspective
How does a Systematic Review differ
from a Narrative or Lit Review?
Feature
Narrative
Systematic
Question
Often broad in scope
Often a focused clinical
question
Sources and Search
Not usually specified
Comprehensive sources &
explicit search strategy
Selection
Not usually specified
Criterion-based selection;
uniformly applied
Appraisal
Variable
Rigorous critical appraisal
Synthesis
Often a qualitative summary
Quantitative summary
Inferences
Based on a sample of the
evidence
Based on all available evidence
Grading
May or may not be graded
All evidence is graded
Adapted from: Cook, D., Mulrow, C. & Haynes, R. (1997). Systematic Reviews: Synthesis of best evidence for clinical
decisions. Annals of Internal Medicine, 126(5), p.376-380).
What are the Most Common Types of
Systematic Reviews?
• Intervention
– To assess the evidence about the effects of a
healthcare intervention.
• Prognostic
– To assess the evidence of models or predictors of
patient outcomes.
• Measurement
– To assess the properties of health-status
instruments or tools
Example #1: TBI & Depression Interventions
(http://www.liebertonline.com/doi/pdfplus/10.1089/neu.2009.1091)
Purpose:
• critically evaluate the evidence on interventions for
depression following TBI
• provide recommendations for clinical practice & future
research
Outcome:
• There is a paucity of RCTs for depression following TBI.
• Serotonergic antidepressants and cognitive behavioral
interventions appear to have the best preliminary evidence
for treating depression following TBI.
• More research is needed to provide evidence based
treatment recommendations for depression following TBI
What is Evidence Based?
• There are many types of evidence, e.g.
– RCTs (large & small)
– Nonrandomized trials
– Case series
– Expert consensus
• It’s an active process
• Evidence you will trust
• Conveying the evidence
Issues in rehab systematic reviews
• Reviews report finding very little or no
evidence
– Evidence standards & methods not a good match
to rehab
• Few RCTs in rehab
• Johnston et al list multiple reasons, including….
Issues in Rehabilitation Research
• Complexity of rehab interventions
– Behavioral, psychosocial, environmental
• Participatory research is highly valued but may be considered less rigorous
• Small sample sizes; need for highly individualized interventions (e.g., AT)
• Blinding & placebo control (again, AT is a good example)
• Control group or denial of services?
• Rehab perceived to be a clinical service not research & developments
• Many of our important issues cannot be manipulated experimentally
(attitudes, access, economic factors)
Steps in Conducting a Systematic Review from a KT Perspective
Develop a clinical
question
Create consumer &
clinical materials
Conduct a quick
scan
External review
& publish review
Create a plan
Grade the
evidence
Categorize
Create tables of
evidence
Extract data
Create consumer
& clinical
materials
Develop
a clinical
question
Conduct a
quick scan
External review
& publish
review
Develop a Clinical Question
Create a plan
Grade the
evidence
Categorize
Create tables
of evidence
Extract data
• To write a good clinical question,
you should ask yourself
• Why am I conducting the review?
– What do I hope to learn?
– Who is this information for?
» Consumers, clinicians, researchers,
others?
A good clinical question will….
• Save time when researching
• Keep the focus directly on the needs of the
target audience
• Suggest the appropriate form that a useful
answer may take
Clinical Question
Which of the following questions would result in the
most useful clinical findings for patients with SCI?
A. Is cardiovascular exercise beneficial to patients with SCI?
B. What are the adverse effects of cardiovascular exercise in
individuals with SCI?
C. What are the parameters/components of a safe
cardiovascular exercise program?
Clinical Question Poll Response
Clinical Question—MSKTC Suggestions
Which of the following questions would result in the
most useful clinical findings for patients with SCI?
A. Is cardiovascular exercise beneficial to patients with SCI?
a.
oversimplified, we already know this is not a Yes/No question
B. What are the adverse effects of cardiovascular exercise in
individuals with SCI?
b.
only focuses on the adverse effects, patients may also want to know
what to do, not just what not to do
C. What are the parameters/components of a safe
cardiovascular exercise program?
c.
BEST ONE!
Develop a
clinical
question
Create consumer
& clinical
materials
Conduct a
quick scan
Create
a plan
External review
& publish
review
Grade the
evidence
Categorize
Create tables
of evidence
Create a Plan
Your development plan should include:
1. Your clinical question
2. Criteria for the literature search
–
–
–
Extract data
3.
4.
5.
Key words
Databases
Dates to include (how far back do you go?)
Inclusion and exclusion criteria
Types of studies you want to include
A timeline for your project
Key Words
• Key Text words for the condition, e.g.,
– Spinal Cord Injury, Paraplegia, Tetraplegia, Quadriplegia
• Key Text words for the intervention, e.g.,
–
–
–
–
Training
Exercise conditioning
Activity limitation
FES (Functional Electrical Stimulation)
Criteria
• Outcomes to be included…
– Cardiovascular events, autonomic dysreflexia
• And excluded
– Beneficial effects of exercise
• Types of studies to be included
– RCTs, case control, case cohort
Study Timeline
Project Tasks
Timeline
1. Project Development Plan
Month 1
2. Perform Literature Search
Month 1
3. Inclusion/exclusion of Articles
Month 1-2
4. Table of Evidence/Data Extraction
Month 2-6
5. Table of Evidence Review
Month 7-9
6. First Draft of Review
Month 11-12
7. Committee Review
Month 12-14
8. External Review
Month 14
Develop a
clinical
question
Create consumer
& clinical
materials
Conduct a
quick scan
External review
& publish
review
Create a plan
Grade the
evidence
Categorize
Create tables
of evidence
Extract data
Categorize
• Create categories for the articles that
you have found according to your
inclusion and exclusion criteria.
• Once you have categorized your
articles, you can determine how many
levels to include in your review
Deciding which Articles to Include
Categories
1
The primary outcome or one of the primary outcomes of the experimental study is related to maintaining
cardiovascular health after SCI2 (NOT interventions specifically aimed to help gain movement or walk). These
interventions could include upper body endurance exercise, strength & endurance training, etc.
2
The primary outcome or one of the primary outcomes of the observational study is related to maintaining
cardiovascular health after SCI2 (NOT interventions specifically aimed to help gain movement or walk).
3
A secondary outcome of the experimental study is related to maintaining cardiovascular health after SCI2 (could
be interventions specifically aimed to help gain movement or walk, but cardiovascular health is secondary
outcome).
4
A secondary outcome of the observational study is related to maintaining cardiovascular health after SCI2
(could be interventions specifically aimed to help gain movement or walk, but cardiovascular health is a
secondary outcome.).
5
A prevalence/incidence study, not fitting in the first 4 categories, where the main focus of the study is to learn
about the rate of exercise & overall health in those with SCI.
Develop a
clinical
question
Create consumer
& clinical
materials
Conduct a
quick scan
External review
& publish
review
Next Steps
Create a plan
Grade the
evidence
Categorize
Create
tables of
evidence
Extract
data
• Data Extraction
• Create Tables of Evidence
Example: Study Flow Chart
Develop a
clinical
question
Conduct a
quick scan
Create consumer
& clinical
materials
External review
& publish
review
Create a plan
Grade the
evidence
Categorize
Create
tables of
evidence
Extract data
Grade the Evidence
• Most common approaches to
classification and grading of the
evidence:
– American Academy of Neurology
(AAN)
– GRADE (Grading of
Recommendations Assessment,
Development and Evaluation)
• Adopted by Cochrane Collaboration,
WHO, and many others)
AAN
– Evidence classified class I (low risk of bias) through
IV (very high risk of bias)
– Evidence graded A (consistent with class 1 studies)
and recommend treatment as established as safe
and effective
– through D (class 4 studies or a mix of 4 classes).
Insufficient evidence to support a conclusion of
effectiveness
GRADE
• Classified high, moderate, low and very low
– Randomized trial = high
– Observational study = low
– Any other evidence = very low
– Note: in GRADE, look at study design first but
there are many other criteria to consider in
assigning a grade of evidence
• Recommendations (strong or weak) based on
classification
Steps in Conducting a Systematic Review
Develop a clinical
question
Create consumer &
clinical materials
Conduct a quick
scan
External review
& publish review
Create a plan
Grade the
evidence
Categorize
Create tables of
evidence
Extract data
Moving from article to next step
• Hasn’t traditionally been researchers' role
• Requires new partnerships
– With consumer and advocacy groups
– With organizations that create clinical guidelines
• NIDRR’s response
– Create KT centers such as the MSKTC
– Our role is to assist researchers in this new role
Systematic Review to User Product
Using the synthesis of evidence resulting from you review of the
adverse effects of exercise for individuals with SCI
– What would you take as a next step for each of the
following audiences
• CLINICIANS
• RESEARCHERS
• CONSUMERS
Consumer & Clinician Materials:
Develop a
clinical
question
Create consumer
& clinical
materials
Conduct a
quick scan
Lessons learned
External review
& publish
review
Create a plan
Categorize
Grade the
evidence
Create
tables of
evidence
Extract data
•
•
•
•
•
•
Provide resources to learn more
Use visuals
For consumers, use and define medical terms
Make it understandable for everyone
Use appropriate tone
Validate findings with target audience
– e.g., cognitive interviews
Best Practices in Developing Consumer Information:
http://msktc.washington.edu/infoneeds/BestPractices_ConsumerInfo.
pdf
Key Points To Remember
Systematic reviews
– assemble, critically appraise, and synthesize the results of primary
investigations addressing a specific topic or problem
– are prepared using strategies that limit bias and random error
– can help practitioners keep up to date with the overwhelming volume
of medical literature
– can help ground clinical decisions in research evidence, although they
neither make decisions nor obviate the need for sound,
compassionate clinical reasoning
– help consumers understand what best evidence is
Summary of Lessons Learned
•
•
•
•
Ask the right clinical question (pragmatic)
When there is not adequate evidence yet…
Get consumer input
Don’t stop when the review is completed
For More Information
• Dijkers, M. When the Best is the Enemy of the Good: The
Nature of Research Used in Systematic Review and Guidelines
(http://www.sedl.org/pubs/catalog/items/dis125.html).
• Johnston, M., Vanderheiden, Farkas, Rogers, Summers &
Westbrook, The Challenge of Evidence in Disability and
Rehabilitation Research and Practice (online soon at
www.ncddr.org/kt/products/tfpapers/tfse_challenge/)
• Sudsawad, P. (2007) Knowledge Translation: Introduction to
Models, Strategies and Measures
(http://www.sedl.org/pubs/catalog/items/dis112.html)
• American Academy of Neurology (AAN) Clinical Practice
Guideline Process Manual
(http://www.aan.com/globals/axon/assets/3749.pdf)
• GRADE (http://www.gradeworkinggroup.org/)