Critical Appraisal: An Introduction
Melanie Browne HBHSc, MLIS
Information Specialist
Agenda
•
•
•
•
•
•
•
•
Introduction
Define Evidence-based clinical practice
How to read a research article
Search for best evidence
Apply critical appraisal in your library
Our new role
Case presentation
Resources
Evidence-Based Medicine
"...the conscientious, explicit, and
judicious use of current best evidence in
making decisions about the care of
individual patients.“
from Sackett, DL, et al. "Evidence based
medicine: What it is and what it isn't."
(BMJ 1996; 312: 71-2)
• "Evidence based medicine (EBM) is an approach
to health care that promotes the collection,
interpretation, and integration of valid,
important and applicable patient-reported,
clinician-observed and research-derived
evidence. The best available evidence,
moderated by patient circumstances and
preferences, is applied to improve the quality
of clinical judgements" (McKibbon et al 1995).
Why EBM?
• A clinician needs to read 17 peer
reviewed articles per day, every day of
the year, to stay current (Haynes 1993).
Haynes, R. (1993) Where's the meat in clinical journals?
ACP Journal Club, 119: A23-4.
There are 20 million pieces of 'evidence' of varying quality and
sometime of contradictory conclusions.
How Can You Access, Distil and Apply
Research/Evidence?
• Develop summaries
• Train clinicians and other health care
decision makers to find and appraise
relevant evidence
• Decision support systems
Summaries With Rigorous Methodologies
• Cochrane Collaboration
• National Centre for Clinical Excellence
• Centre for Clinical Effectiveness at
Monash University
The Argument for Evidence-Based Medicine
• Stay up to date with the current
literature
• Communicate effectively
• Make the best use of information from
the history, physical examination, and
diagnostic testing
• Avoid common pitfalls of clinical
decision-making
Critical Appraisal
• The assessment of evidence by
systematically reviewing its relevance,
validity and results to specific situations.
- Chambers, R. (1998).
What is ‘best’ evidence?
• Using critical appraisal skills you can
understand the methods and results of
the research and then be able to assess
the quality of the research.
Critical appraisal
Critical appraisal
Exact science
“right” answer
Information Mastery
Slawson and Shaughnessy Formula
Usefulness of Medical Information =
Relevance x Validity
____________________
Work to Access
• Smith R. What clinical information do
doctors need? Br Med J 1996; 313: 1062-8.
• When doctors see patients they usually
generate at least one question
• Most of the questions concern treatment
• Many of the questions are highly
complex, simultaneously asking about
individual patients and particular areas of
medical knowledge
• Often doctors are asking not simply for
information but for support, guidance,
affirmation, and feedback
• Doctors are most likely to seek answers
to these questions from other doctors
• The best information sources provide
relevant, valid material that can be
accessed quickly and with minimal effort
Basic elements of clinical decision making
Haynes RB. Loose connections between peer-reviewed clinical journals and clinical
practice. Ann Intern Med 1990;113:724-8.
Why Should we Critically Appraise?
• Published research is not always reliable
• Published research is not always relevant
• To improve clinical effectiveness, we
need a systematic framework to interpret
research
Difficulties with Critical Appraisal
• Can be time consuming initially
• Doesn’t provide an “easy” answer
• It could show a lack of good evidence in a
particular topic
Key Steps to Effective Critical Appraisal
1. Are the results valid?
2. What are the results?
3. How will these results be relevant to
the patient?
Validity and Reliability
• A test is valid when it measures what it’s
supposed to.
• If a test is reliable, it yields consistent
results.
• A test can be both reliable and valid, one
or the other, or neither.
• Reliability is a prerequisite for
measurement validity.
Reliable, but Not Valid!
Not Reliable, Not Valid!
Reliable and Valid
Mark Newman - Middlesex University
updated 04/2001
What a Physician can be Faced With on a
Daily Basis?
• ~1 - 2 questions per patient (clinics)
• 15/patient/day (wards)
• ~30% of questions are followed
– People
– Books
– Electronic resources
• 40% easy to answer
• 30% tough to answer
• 30% cannot answer
Source: Dawes M, Sampson U. Knowledge management in clinical practice: a systematic review of
information seeking behaviour in physicians. Int J Med Inform. 2003 Aug; 71(1):9-15. Review.
PMID: 12909153
Steps to EBM
1. Formulate a clear, focused clinical question
2. PICO model
3. Search the literature for the best external
evidence
4. Critically appraise the evidence for its validity
and usefulness
5. Implement the useful evidence in clinical
practice
6. Evaluate the results
Question Formulation
Not easy but EXTREMELY important
Good questions will
– Focus/clarify your information need
– Give you some idea of where to look for
information
– Give you searching concepts and terms
Background Questions
…Often broad in nature
…Often not patient-specific but fact based
…May not need to integrate knowledge
…More common early in training/new
situations
Foreground Questions
…Often for a specific patient or clinical
situation
…Narrow in focus
…Need to integrate external information
with clinical/situational data
PICO Structure
Patients
Intervention
Comparison
Outcome
P
I
C
O
Clinical Scenario
What therapeutic agents can be used for
rate control of atrial fibrillation (AF) in a
patient with congestive heart failure
(CHF)?
Starting Point
Department: Emergency
Population: Patients with atrial fibrillation
and congestive heart failure
Intervention: Rate control
Comparison: N/A
Outcome: Mortality, effectiveness of rate
control
PICO Schematic Model
Rate control
Patient w/ Atrial
Fibrillation in
congestive heart
failure
Comparison
Rhythm control /
No treatment
Effectiveness
of rate
control
Clinical problem
General Search Strategy
Define the search question
Try another relevant resource
Choose a resource/database
Create a search strategy
Create a search strategy
Summarize the evidence
Apply the evidence
Poor
yield
Summarize the evidence
Try another relevant resource
Adapted from: Sackett, D. et al. 2000. Evidence-Based Medicine: How to Practice
and Teach EBM. 2nd Edition. Toronto: Churchill Livingstone.
Study Types for Question Types
Diagnosis
Therapy
Etiology/
Harm
Prognosis
Prospective cohort study with good
quality validation against “Gold
Standard”
Randomized controlled clinical trial
(RCT)
RCT, cohort or case-control study
(probably retrospective)
Prospective cohort study
Does this treatment work?
systematic review, RCT
How good is a diagnostic test?
(prospective) cohort study
Should we screen?
RCT
What causes this disease?
RCT, prospective cohort study, case
control study (rare diseases)
What did people think or do?
cohort study, cross-sectional survey,
qualitative study
How can we do Critical Appraisal?
• Use common sense
• Use simple checklists
• Use different checklists depending on the
different types of studies (i.e., RCTS,
systematic reviews etc)
• Checklists help you focus on the
important parts of the article
Research Methodology
•
•
•
•
•
•
Who were the participants of the study?
How were they recruited?
Was there bias in the recruiting methods?
How was the data collected?
What statistical tests were used?
Where the data collection methods
accurate?
Critical appraisal questions
•
•
•
•
What is the paper about?
Why was the study done?
What type of study was done?
Was it primary research (experiment,
RCT, cohort, case-control, crosssectional, longitudinal, case
report/series)?
Critical appraisal questions
• Was it secondary research (overview,
systematic review, meta-analysis,
decision analysis, guidelines
development, economic analysis)?
• Was the design appropriate (for study
on treatment, diagnosis, screening,
prognosis, or causation)?
Critical appraisal questions
• Was the study ethical?
• Is the design right?
(BMJ Editor's checklists)
How to Read a Research Article? – First
Glance.
• Purpose of reading the paper.
• Do not read the abstract.
• Read the title, find out who the authors are and where they work, look for
sources of funding and conflicts of interest
• Look at the tables and figures
• Is there a diagram to show the flow of participants through each stage of
the study
• Why was the study done and what hypothesis were the authors testing?
• What is broadly the topic of research?
• Therapy, Diagnosis, Screening, Prognosis, Causation?
• What type of study was done?
• Is this a primary (experimental, clinical trial, survey) or a secondary paper
(review, meta-analysis, guideline, economic analysis)?
• Was the study design appropriate?
• Now proceed to a critical appraisal of the paper
Case Presentation
• Read the Ray et al(2008) paper and divide
the group into 2 – for a debate on the
paper “Breast size and risk of type 2
diabetes mellitus.” CMAJ
• Ray and colleagues studied data from 92
106 women in conjunction with the
Nurses' Health Study II and found that
breast size at age 20, assessed by recall
of bra cup size, correlated positively with
the incidence of type 2 diabetes
• After adjustment for relevant factors,
such as body mass index, waist
circumference and family history of
diabetes, the hazard ratio dropped to
1.58 but remained significant.
• Nurses' Health Study II population data
emanates from women who were mainly
of white ancestry, and that their analysis
is based on recall and self-report.
New Role of Health Sciences Librarians
• Teaching access to the literature and
other information resources
• Teaching use of technology as a means to
access and manage information
• Teaching skills in information
organization and critical appraisal
Role of Librarians as Information Clinicians
• Information Clinician
• Medical Informatics Tutor
• EBM Educator
Informationist
• A discipline requiring a combination of
the skills of a librarian, a clinical
epidemiologist and a medical scientist
Davidoff, F. & Florance,V. (2000) The
Informationist: A New Health Profession?
Annals of Internal Medicine, 132: 996998.
Effective INFORMAtician
• Knows categories of information resources
…therapy, diagnosis, prognosis
…synthesized resources vs original studies
• Knows strengths/weaknesses of information
resources
• Knows when to use each category
• Knows that some resources are better than
others in certain situations
…Cochrane diagnostic accuracy data?
• Knows what to do with the results
• Practical solutions - Librarian as a partner
on the teaching team; teachable
moments
• Formal hospital library educational
programs have focused on orientation and
instruction in the use of bibliographic
tools. i.e.- LATCH (Literature Attached to
Charts) and clinical medical librarianship
Critical Appraisal Skills Training
• Research methodology
• Statistical techniques
Courses for Librarians Out There
CILIP: the Chartered Institute of Library
and Information Professionals
• Critical appraisal skills for healthcare
librarians: building on the basics
http://www.cilip.org.uk/training/trainin
g/2008/libinfo/criticalappraisalskillsforhe
althcarelibrariansbuildingonthebasics.htm
General Resources:
• CASP (Critical Appraisal Skills Programme) - part of the Public
Health Resource Unit based at Oxford, CASP runs training
workshops on critical appraisal skills.
• Evidence-Based Medicine Toolkit - hosted by the University of
Alberta, this is an online "box" of handy tools to help you find,
appraise, and apply in practice, evidence-based research
• How to read a paper - a set of ten guides from the BMJ (individual
links given in the relevant section below).
• Levels of Evidence - a ranking system used to rank various study
designs in order of evidence-based merit: systematic
reviews/meta-analyses and well conducted randomised controlled
trials (RCTs) are usually seen as the best form of "evidence", with
research based on the outcome of a case series placed somewhere
near the bottom
• Netting the Evidence - search for the keyword "appraisal" to find a
quality assessed list of appraisal resources
• User's guides to evidence based practice - based on a series of
articles published in JAMA, these guides give comprehensive advice
on how to find, appraise and apply research in practice
Keeping Current
• Daily InfoPOEMs http://www.infopoems.com/
• bmjupdates+
http://bmjupdates.mcmaster.ca/index.asp
• Tables of contents of journals
(e.g. JAMA)
– From journal itself
– My NCBI from PUBMED
Predictive Value of tests
Relative Risk Reduction
P Value