EBD for Dental Staff Seminar 2:
Core Critical Appraisal
Dominic Hurst
r.d.hurst@qmul.ac.uk
Structure of the seminars
Seminar 1
• Recap of EBD
• Using an educational prescription
• Structured questions and search
Seminar 2
• Critical appraisal
Seminar 3
• Communicating evidence to patients
• Getting evidence into practice
Workshop
• Focused work on one or more of these
Why critically appraise a study?
Put rubbish in, get rubbish out
Critical appraisal is the
process of carefully and
systematically examining
research to judge its
trustworthiness, and its
value and relevance in a
particular context.*
Value
Trustworthiness
Relevance
*http://www.medicine.ox.ac.uk/bandoli
er/painres/download/whatis/What_is_cr
itical_appraisal.pdf
Critical appraisal
Three key questions
• Is the study valid?
• Internal validity
• What are the results?
• Size and precision of the results
• Can I use it with the patients I manage in my context?
• External validity and usefulness in this setting
Is the study valid?
Are the results what they claim to be?
Internal validity
• In intervention or risk factor questions:
• The degree of confidence we have that the difference in outcomes between 2
or more groups is due to the intervention / exposure and not something else
• Regular fluoride varnish reduces caries incidence in children by 30-40%
• TMJD resolves of its own accord in 70% of patients after 3 months
Does a red pill make a mouse grow more than
a blue pill?
What messes up internal validity?
• Confounder variables or factors
• Any variable other than the one of interest, that can influence the outcome
Study concludes that “night shifts cause dental caries”.
Is there any other explanation?
• Confounders lead to bias
• A systematic deviation from the truth
• High internal validity if bias is low / not present
• Low internal validity if bias is high
Using critical appraisal checklists
• RCTs
• Systematic reviews
What are the results?
Think absolutes rather than relatives…
Size (or magnitude) of the effect
• In a trial with 2000 patients having non-surgical extractions
• 1000 get the socket rinsed with chlorhexidine
• 1000 get the socket rinsed with water
• We follow up for 14 days
• We count how many get dry socket
•
•
•
•
•
I
C
No dry
socket
950
900
1850
Dry
socket
50
100
150
1000
1000
2000
Risk of getting a dry socket with the
intervention?
• 50/1000=5%
Risk of getting a dry socket with the
comparison?
• 100/1000=10%
Absolute risk reduction (ARR)?
• 10%-5%=5% (0.05)
Relative risk reduction (RRR)?
• (10%-5%)/10%=50%
Number needed to treat (NNT) with the
chlorhexidine rather than water to prevent
one additional dry socket?
• 1/ARR=1/0.05=20
I
C
No dry
socket
955
990
1985
Dry
socket
5
10
15
1000
1000
2000
•
•
•
•
•
Risk of getting a dry socket with the
intervention?
• 5/1000=0.5%
Risk of getting a dry socket with the
comparison?
• 10/1000=1.0%
Absolute risk reduction (ARR)?
• 1.0%-0.5%=0.5% (0.005)
Relative risk reduction (RRR)?
• (1.0%-0.5%)/1.0%=50%
Number needed to treat (NNT) with
the chlorhexidine rather than water to
prevent one additional dry socket?
• 1/ARR=1/0.005=200
Other ways results are expressed
• Mean difference
• E.g. mean difference in probing depth
• Odds ratios
• Odds: the chance that something happens / the chance that it doesn’t
I
C
No dry
socket
950
900
1850
Dry
socket
50
100
150
1000
1000
2000
• Odds of a dry socket if using Intervention?
• 50/950=0.053
• Odds of a dry socket if using comparison?
• 100/900=0.111
• Odds ratio?
• 0.053/0.11=0.477
Results in Systematic Reviews: Forrest Plot
Precision and statistical significance
• 95% Confidence intervals
• We are 95% confident that the population result would lie within this range
either side of the study result
• P values
• If P<0.05 = a less than 1/20 chance that the result is due to chance
• P<0.005 means there is a less than 1 in 200 chance the result is due to chance
Can you use the results locally?
External validity, your skills, your patients and your resources
Things to consider of any study
• Are all the outcomes that would be of interest to you reported?
• Think of harms as well as benefits
• Are the patients that were in the study so different?
• Is what is being done feasible by a student / you in this setting?
Key messages
• Critical appraisal asks 3 questions:
• Is the study valid?
• What are the results?
• Can they be used locally?
• Internal validity affected by confounding factors that cause bias
• Relative outcomes can be misleading…so opt for absolutes if you can
Thanks for participating
Dominic Hurst
r.d.hurst@qmul.ac.uk