Introduction to
Critical Appraisal
Jo Hunter
Clinical Outreach Librarian
University of Oxford
Health Care Libraries
Aims of this session
Define “critical appraisal”
Discuss why it is important
Discover how to incorporate it in
your own research/practice
Is this evidence good enough?
What is “good” evidence?
• We need to use evidence from
research studies to support our
arguments or decisions
• How do we recognise a good
quality research study?
How can you tell whether a
research study is “good quality”?
Factors that tell you about the
quality of a study include…
Extrinsic factors:
• Journal
• Author
• Institution
• Quoted by other
authors
• Quoted in
newspapers
Intrinsic factors:
• Study design
• Number of
subjects
• Confounders
• Bias (author,
experimental)
Critical appraisal looks at the
intrinsic factors
“Critical appraisal is the process of
systematically examining
research evidence to assess its
validity, reliability and
relevance before using it to
inform a decision.”
(Adapted from Hill H & Spittlehouse C, EBM 2001; 3:2)
Validity, reliability, relevance
1) Validity: To what extent is the
study a close representation of
the “truth”?
2) Reliability: Are the results
credible and repeatable?
3) Relevance: Will the results help
me in my own research or
practice?
Pros of critical appraisal
• Systematic way of assessing a
paper’s validity, reliability,
relevance
• Helps to close the gap between
research and practice
• Encourages objective
assessment
• Skills are not difficult – common
sense & user-friendly checklists
Cons of critical appraisal
• Can be time-consuming initially
• Does not always provide the
“easy” answer – may provide
more questions
• Can be dispiriting if it highlights
poor quality studies
Examples of appraisal
checklists
• CASP for experimental studies,
systematic reviews
• DISCERN for consumer health
information (papers about
treatment choices
• CriSTAL for librarianship studies
(information needs analysis, user
survey)
Getting started
1) Set up a journal club
2) Remember/devise a suitable
scenario
3) Find a relevant paper
4) Choose an appropriate checklist
1) Why set up a journal club?
• Regular protected time for honing
your “quality-detector” skills
• Facilitates discussing and
evaluating new research and its
application to your research or
practice
• Helps to unravel complex
problems
1) Five steps to journal club
success
1.
2.
3.
4.
Identify a leader
Identify the goals of the club
Set up a regular meeting time
Send the scenario and paper to
everyone before the meeting
5. Encourage active participation
by using a checklist
2) Why bother with a scenario?
• Provides a practical focus
• Helps you to answer the
applicability question
• Helps you to find a paper
2) Example of a scenario
A 32 year old man is found in
garage with engine running. He is
brought to the ED by paramedics.
He is awake and alert. He
complains of a headache. His
exam is unremarkable. Skin
colour is normal. Neurologic
exam is normal.
What is the appropriate therapy?
2) Another example of a
scenario
You have been invited to join a
local implementation group to
look at how to improve
information services to those
studying ethnomusicology at the
University.
The group agrees that they need to
find out what ethnomusicologists
need, before making any
improvements.
2) Another example of a
scenario
3) Find a relevant paper
• Use key words from the scenario
for database searching
• Browse your favourite journals
• Ask your librarian for help
4) Choose an appropriate
checklist
• Identify the study type of the
paper
• Google! www.google.co.uk
• Adapt a checklist that is nearly
appropriate
• Remember: validity, reliability,
applicability
Finally..
• Write up a Critically Appraised
Topic (CAT) for future reference
www.ouls.ox.ac.uk/hcl
• Evaluate: Did you find the
process helpful? If not, why not?
How could it be improved?