www.askdoctorclarke.com
CRITICAL READING: AN OVERVIEW
Critical reading and critical reflection
Critical reading usually takes place after some aspect of clinical work reminds us that
we have a learning need. Critical reflection follows. Here the clinician considers the
extent to which the results of different studies might be useful in practice (Clarke & Croft
1998). How far can the study conditions be generalised to the real world and what
would the consequences be of introducing a change in practice? Any plan for change
should ideally make explicit ideal standards of practice and audit can assess the impact
of change against these.
Critical reading
Problem definition
Critical reflection
Action
Reflection
Practice
Self-directed, problem based learning
Sackett showed that there were three independent dimensions, which can predict
effective learning in clinical medicine (Sackett et al. 1995).
a) Clinical problem
b) Clinical evidence
c) Critical appraisal
Learning was more effective where searching for evidence was based on a real patient
with a real problem, rather than a topic ("let's find out about renal medicine") or a case
history of someone else's patient. Self-directed learning was also effective as searching
for evidence gave a sense of ownership of the problem-solving process. Learning how
to search may also have helped in stimulating life-long learning, as the skills are easily
transferable from one clinical problem to another. Finally, the ability to read critically and
to consider the implications for practice (critical reflection) formed a third dimension.
1
www.askdoctorclarke.com
This was the basis of Sackett's cube (Sackett et al. 1995).
l
isa
a
r
pp
Self-directed, problem-based learning
a
Critical
al
reflection
c
i
it
Can critically
Cr
appraise
Clinical problem
Intuition
Actual
patient
Case
history
None
Supplied Guided Independent
search search
Clinical evidence
After Sackett et al 1991
Critical appraisal
1) Searching for evidence
2) Design / methodology
3) Bias
4) Implications
5) Systematic reviews and meta-analysis
Searching for evidence
Question
You are planning a policy on prescribing antibiotics for urinary tract infections in
young children. You wish this process to be evidence based.
Outline how you would gather your evidence.
Question
You are interested in studying the psychosocial impact of the diagnosis of
genital chlamydia. You wish the process to be evidence based.
How would you gather the evidence?
2
www.askdoctorclarke.com
How would you gather evidence? eg acute otitis media
Formulating an answerable question
1) Population
2) Intervention
3) Comparison
4) Outcome
How would you gather evidence?
Eg psychosocial impact of diagnosis of chlamydia?
Formulating an answerable question: qualitative version of PICO
1) Population: ? purposive sampling
2) Investigation: ? grounded theory ?interview ?focus group ?questionnaire
3) Comparison: may not be needed
4) Outcome: ? what outcomes relevant; ? respondent validation
3
www.askdoctorclarke.com
Search strategy
1) Who should search?
2) Choice of databases, journals and opinions
3) Start broad (to avoid missing important evidence)
4) Then narrow-down (to focus on what you want)
5) Hierarchy of evidence
Hierarchy of evidence
1) Systematic review of randomised controlled trials (RCT's)
2) Single RCT
3) Systematic review of observational studies addressing important patient outcomes
4) Single observational study
5) Unsystematic clinical observations
You are considering using a medical librarian to gather the evidence for you. What are
the strengths and weaknesses of using this approach?
Strengths
Weaknesses
Common questions on study design
Comment on the design of this study
Discuss the strengths and weaknesses of the methodology
Critically assess the methods
4
www.askdoctorclarke.com
Selection bias
Look at the pathway from potential study population to population actually studied.
Is it clear how many were excluded ?
Do the same selection conditions apply to compared groups ?
Selection bias questions
Comment on the recruitment of subjects
Comment on the sampling methods used
Comment on the selection of cases
Drop-out bias
A study showed that patients' knowledge of asthma was improved by being given a
booklet. A small table gave the following information:
Baseline
4 weeks
Booklet
n=50
n=25
No booklet
n=50
n=45
How would you interpret this table?
Could this result in a biased conclusion (deviation from the truth)?
Measurement (information) bias
Observer error
Validity of measurements
Reliability of measurements
5
www.askdoctorclarke.com
R eliability
(repeatability)
V alidity
(closeness
to truth)
H igh
Low
H igh
L ow
Numbers
Are they clear ?
Is the result statistically significant ?
How big is the result ?
Is it clinically significant ?
Patients with sore throats reviewed after 5 days:
Pen V
Placebo
Still sore
50
65
Better
450
435
Total
500
500
Chi square 5.2, p = 0.02
How do you interpret these results?
What is meant by statistical significance?
What is the NNT? (numbers needed to treat)?
Statistically significant but note that you have to treat 500 to make a difference of 15.
Probably all better by ten days anyway! Questionable clinical significance.
NNT= 500/ 15 ie approximately 33
6
www.askdoctorclarke.com
3 types of data
Nominal, categorical eg better, worse
Ordinal- ranked data- cured, much better, no change, a bit worse, much worse, dead
Interval, numerical- measured on a numerical scale eg height, weight, haemoglobin
Two statistical tests
Chi square test- used on categorical data (non-parametric)
t test- used on parametric data (interval data if normally distributed)
Statistical significance
• means only that the difference is unlikely to have arisen by chance
• does not mean that the result is clinically important
• cannot correct for or overcome bias
Common questions on implications
Discuss the implications of these results
How far can these results be generalised?
What factors would you consider before introducing this into your practice?
Implications
* Generalisation
* Consequences
Consequences checklist
P * People and ethics
R * Resources
A * Audit, protocols and quality
T * Time and training
S * Safety and society
Smash and grab questions
To what extent are the author's conclusions justified?
Write short notes on the importance of these results
What would make you want to read the rest of this article?
Critical appraisal checklist: a new tool for critical reading
P
* PRESENTATION
E
* EXPERIMENTAL DESIGN
N
* NUMBERS
I
* INFORMATION BIAS
S
* SELECTION BIAS
7
www.askdoctorclarke.com
Task
Below is the abstract of a large hospital-based trial performed in the USA. Comment
on the implications for diabetes care in the UK.
“The effect of intensive treatment of diabetes on the development and progression of
long-term complications in insulin-dependent diabetes mellitus."
The diabetes control and complications trial research group (N Engl J Med 1993;329,977-86)
Abstract
Background
Long-term microvascular and neurological complications cause major morbidity and
mortality in patients with insulin-dependant diabetes mellitus (IDDM). We examined
whether intensive treatment with the goal of maintaining blood glucose
concentrations close to the normal range could decrease the frequency and severity
of these complications.
Methods
A total of 1441 patients with IDDM – 726 with no retinopathy at base line (the
primary-prevention cohort) and 715 with mild retinopathy (the secondary-intervention
cohort) and 715 with mild retinopathy (the secondary-intervention cohort) were
randomly assigned to intensive therapy administered either with an external insulin
pump or by three or more daily insulin injections and guided by frequent blood
glucose monitoring or to conventional therapy with one or two daily insulin injections.
The patients were followed for a mean of 6.5 years and the appearance and
progression of retinopathy and other complications were assessed regularly.
Results
In the primary-prevention cohort, intensive therapy reduced the adjusted mean risk
for the development of retinopathy by 76 percent (95 percent confidence interval, 62
to 85 percent), as compared with conventional therapy. In the secondaryintervention cohort intensive therapy slowed the progression of retinopathy by 54
percent (95 percent confidential interval, 39 to 66 percent) and reduced the
development of proliferative or severe non-proliferative retinopathy by 47 percent (95
percent confidence interval 14-67 percent). In the two cohorts combined, intensive
therapy reduced the occurrence of microalbuminuria (urinary albumin excretion of
>40mg per 24 hours) by 39 percent (95 percent confidence interval, 21 to 52
percent), that of albuminuria (urinary albumin excretion >300mg per 24 hours) by
54% (95 percent confidence interval, 19-74 percent), and that of clinical neuropathy
by 60 percent (95 confidence interval, 38-74 percent). The chief adverse event
associated with intensive therapy was a two-to-threefold increase in severe
hypoglycaemia..
Conclusions
Intensive therapy effectively delays the onset and slows the progression of diabetic
retinopathy, nephropathy and neuropathy in patients with IDDM.
8
www.askdoctorclarke.com
The Three E’s
• Efficacy: impact of an intervention in the bets
possible circumstances eg in a trial with wellmotivated staff, selected compliant patients
and unlimited resources
• Could it work?
• Effectiveness: impact on everyday practice:
• Does it work?
• Efficiency: cost effectiveness
• How much does it cost to work?
9
www.askdoctorclarke.com
Diabetes trial (DCCT: implications
Design
Randomised trial of clearly defined patient groups with insulin-dependent diabetes.
Therefore the results are potentially generalisable to diabetics in the UK.
Intensity of care and reduction of complications
It is likely that only a very small minority of patients in the UK are receiving this
intensity of care. If the results are reliable, the first implication is that changes in
diabetic care could prevent and slow down the current level of complications in
insulin-dependent diabetics. The potential for improvement in the incidence of
retinopathy, nephropathy and neuropathy looks to be considerable.
Process of care: cost, workload, acceptability
The provision of a pump, or an increased regularity of insulin injections, together with
the need for frequent blood glucose monitoring has many potential implications for
practice in the UK: additional costs, additional workload for health care professionals
in training and supervising, patient acceptability, patient compliance.
The overall effect of the intervention would be less than that seen in the trial. Being
in a trial is a highly motivating process and this may influence the result. Such an
effect is sometimes referred to as the Hawthome effect.
Large size of the effect of intervention
Against this must be weighed the fact that the changes in relative terms were quite
large, so that even if a less efficient regime was introduced or even if there were a
considerable number of patients who refused or did not wish to adhere to the regime,
it still might have an impact – both to the individual patient but also overall – on the
occurrence of diabetic complications. So an implication might be that more modest
but useful benefits might emerge even if the trial protocol could not be put into
practice universally in the UK.
Side-effects of treatment
Of more concern is the increase in severe hypoglycaemia. Under the highly
controlled conditions which a trial demands, the level of hypoglycaemia events is
likely to represent the most optimistic situation. Putting the regime into practice in
the real world is likely to mean many more such events, and probably more serious
outcomes. Since hypoglycaemia kills, the universal adoption of this approach might
result in unacceptable mortality rates – more costs than benefits in fact.
Patient autonomy
The need for careful and highly controlled intervention also goes against the general
approach to diabetics which encourages their independence and self-monitory, so
any attempt to introduce this regime in the UK would need to be slow, in chosen
patient groups, and with careful monitoring of effects. This is the conflict between
what Professor Alberti (Newcastle) has called “good control or a happy life”.
10
www.askdoctorclarke.com
Numbers Needed To Treat (NNT)
Based on examples in an excellent book on evidence based medicine (Sackett et al. 1997).
Does the treatment work ?
• relative risk reduction (RRR)
• absolute risk reduction (ARR)
• numbers needed to treat (NNT)
• three letter acronyms (TLA)
Consider the DCCT trial
• risk of neuropathy 9.6% in control group
- control event rate (CER)
• intensive insulin for 6 years in intervention group reduces rate to 2.8%
- experimental event rate (EER)
• relative risk reduction (RRR) = CER-EER/ CER
Consider the DCCT trial
• risk of neuropathy 9.6% (CER)
• intensive insulin for 6 years in intervention group reduces rate to 2.8% (EER)
• relative risk reduction (RRR) = CER-EER/ CER
= 9.6-2.8 / 9.6
= 71%
Disadvantage of RRR
• relative risk reduction (RRR)
•
•
•
= CER-EER/ CER
= 9.6-2.8/9.6
= 71%
same result would apply if 96% got neuropathy in control group and 28% in
intervention group: 96-28/96= 71%
fails to show baseline risk or size of effect
tiny effect can be made to look impressive
Consider the DCCT trial
• risk of neuropathy 9.6% in control group
- control event rate (CER)
• intensive insulin for 6 years in intervention group reduces rate to 2.8%
- experimental event rate (EER)
= CER-EER
• Absolute risk reduction
= 9.6-2.8
= 6.8%
11
www.askdoctorclarke.com
Number needed to treat (NNT)
• Number of patients a clinician needs to treat in order to prevent one additional
adverse event
• Inverse of ARR
NNT is inverse of ARR
• ARR is 6.8%
• This means you have to treat 100 to prevent 6.8 additional cases of neuropathy
• How many would you have to treat intensively in order to prevent ONE additional
case?
• Answer: 100/6.8 = 14.7
Consider the DCCT trial
• 6.8/100 ARR equivalent to: NNT= 100/6.8=14.7
• usually rounded up
• treat 15 patients for 6 years to prevent one extra neuropathy
Number needed to treat (NNT)
• helps clinician but not much help to patient who is interested in a series of n=1
• practical tangible result
• easier to remember than ARR which is often a figure less than one
• can be used to compare interventions
Number needed to treat (NNT)
• Breast examination plus mammography after 9 years
• Thrombolysis + aspirin for MI: death 2yrs
• Carotid endarterectomy: death/ CVA at 2yrs
NNT = 1000
NNT = 20
NNT =
10
12
www.askdoctorclarke.com
Task
You are considering how best to treat patients with non-rheumatic atrial fibrillation
(AF). You wish the process to be evidence based. A literature search has identified
three studies with the following titles:
A)
Taylor FC, Cohen H, Ebrahim S
Systematic review of long term anticoagulation or antiplatelet treatment
in patients with non-rheumatic atrial fibrillation
BMJ 2001 Feb 10; 322: 321-326.
B)
Peters RW, Mergner W, Ghiorzi T, Benitez RM
A 70-year-old man with atrial fibrillation and stroke
Maryland Med J. 1999 Mar-Apr; 48 (2): 74-8.
C)
Fuster V et al American College of Cardiology/ American Heart
Association/ European Society of Cardiology
Guidelines for the management of patients with atrial fibrillation
J Am Coll Cardiol. 2001: 38(4): 1231-1266
Copies of the papers listed above are not provided. Your answers should be
based upon the principles associated with the various types of evidence that
have been identified.
1a)
Consider study A. Comment on the strengths and weaknesses of
this type of study in answering your question.
Strengths
Weaknesses
13
www.askdoctorclarke.com
1b)
Consider study B. Comment on the strengths and
weaknesses of this type of study in answering your
question.
Strengths
1c)
Weaknesses
Consider guideline C. Comment on the strengths and
weaknesses of using this type of paper in answering your
question.
Strengths
Weaknesses
14
www.askdoctorclarke.com
Task
Please read the accompanying reference material, an excerpt from a paper entitled
‘Systematic review of long term anticoagulation or antiplatelet treatment in patients
with non-rheumatic atrial fibrillation’. (see next page)
Explain the meaning of the figure.
15
www.askdoctorclarke.com
Systematic review of long term anticoagulation or antiplatelet treatment in patients with non-rheumatic
atrial fibrillation (Taylor, Cohen, and Ebrahim BMJ 2001; 322: 321-326)
Introduction
In the past decade there has been widespread implementation of oral anticoagulation (warfarin) in
preference to antiplatelet treatment (aspirin/indoprofen) to reduce the risk of cardiovascular mortality
1
and morbidity in patients with non-rheumatic atrial fibrillation. This choice of treatment has been
based on evidence from randomised control trials that compared long term anticoagulation with
placebo and antiplatelet treatment with placebo. Reviews pooling efficacy estimates of such trials for
anticoagulation have shown considerable benefits, with an odds reduction for non-fatal stroke of 68%
2-4
with low rates of bleeding. Pooled efficacy estimates for antiplatelet treatment have shown less
benefit. The combined fatal and non-fatal vascular event rates were 10.8% for aspirin versus 13.5%
5 6
for placebo, a 28% reduction in risk.
From these reviews it might be concluded that long term
anticoagulation results in substantially greater benefits in non-rheumatic atrial fibrillation, indeed more
than twice as great a relative effect on treatment. To make this judgment, however, direct head to
head comparison of the effects of long term anticoagulation and antiplatelet drugs in comparable
patients is necessary. Indirect comparisons of the effects of the two treatment options may be biased
by different selection criteria used in trials, leading to differences in prognosis unrelated to treatment
7
and overestimation of treatment effects. We undertook a systematic review of evidence from
randomised controlled trials to determine unbiased estimates of which is the better treatment option.
Results
No trials were found from before 1989. There were five randomised controlled trials published
between 1989-99. There were no significant differences in mortality between the two treatment
options (fixed effects model: odd ratio 0.74 (95% confidence interval 0.39 to 1.40) for stroke deaths;
0.86 (0.63 to 1.17) for vascular deaths). There was a borderline significant difference in non-fatal
stroke in favour of anticoagulation (0.68 (0.46 to 0.99)); and 0.75 (0.50 to 1.13) after exclusion of one
trial with weak methodological design. A random effects model showed no significant difference in
combined fatal and non-fatal events (odds ratio 0.79 (0.61 to 1.02)). There were more major bleeding
events among patients on anticoagulation than on antiplatelet treatment (odds ratio 1.45 (0.93 to
2.27)). One trial was stopped prematurely after a significant difference in favour of anticoagulation
was observed. The only trial to show a significant difference in effect (favouring anticoagulation) was
methodologically weaker in design than the others.
Comparison of fatal vascular outcomes in trials of anticoagulation versus antiplatelet treatment
16
www.askdoctorclarke.com
Checklist For A Systematic Review (Greenhalgh 1997)
1) Did the review examine an important clinical question?
2) Was a thorough search done of the appropriate databases and were other potentially
important sources explored?
3) Was methodological quality assessed and the trials weighted accordingly?
4) How sensitive are the results to the way the review has been done?
5) Have the numerical results been interpreted with common sense and due regard to the
broader aspects of the problem?
What is systematic about a systematic review?
1) Clearly formulated question
2) Explicit methods to
- identify studies
- select studies
- appraise studies
- analyse data
•
•
•
•
All of these can be applied to qualitative as well as quantitative studies.
Meta-analysis is a statistical technique for combining quantitative date.
Statistics cannot overcome bias- so depends on quality of individual studies.
One way of checking that combined studies are reasonably similar is to check
for heterogeneity.
Heterogeneity
One would expect the results of different studies to be slightly different due to
random variation (chance). If such small differences are observed, it is likely that the
populations in each study and the results of the interventions are similar- or
homogeneous.
On the other hand, if the differences are greater than would be expected by chance
variation, the studies are said to be heterogeneous. The source of those differences
might be due to the experimenters asking a different question, selecting subjects
differently, making different measurements etc.
In practice, the statistician does a "Chi square test for heterogeneity"- and everyone
hopes the result will be non-significant, making homogeneity likely. If the test of
heterogeneity is significant, the studies need to be re-examined for bias and to ask
whether it is really reasonable to compare them.
17
www.askdoctorclarke.com
Probability and Odds
In a bag are 3 oranges and 1 apple = 4 pieces of fruit
Probability
If you choose a piece of fruit at random, then the probability of picking an apple is:
1 in 4
and of picking an orange is: 3 in 4
This relates the chance of picking one type of fruit to the whole number (all the fruit).
Odds
The odds of picking one type of fruit are calculated by making a comparison within the
sample. The odds of picking an apple are:
1 to 3
Try these examples:
Diagnosis
Eczema
Also hayfever
141
No hayfever
420
Total
561
What is the probability that a child with eczema also has hayfever?
(answer: 141:561)
If a child has eczema, what are the odds of also having hayfever?
(answer 141:420)
Odds ratio
Trials of nicotine replacement therapy (NRT) show:
All trails combined
Given NRT:
proportion quitting
1456 / 7834 (18%)
Controls:
proportion quitting
1016 / 9600
(10%)
Odds ratio
1.9
In the NRT group, the number not quitting is: 7834-1456=6378
Odds of quitting when given NRT= 1456: 6378 = 0.228
Odds in control group= 1016: 8584 = 0.118
Odds ratio = 0.228 ÷ 0.118 = 1.9
Note this gives no idea of the actual size of the difference because it is a ratio.
(Similar problem to that with relative risk)
18
www.askdoctorclarke.com
References
Clarke, R. & Croft, P. (1998) Critical reading for the reflective practitioner, Oxford:
Butterworth Heinemann.
Greenhalgh, T. (1997) How to read a paper, London: BMJ Publishing Group.
Sackett, D., Haynes, R., Guyatt, G., & Tugwell, P. (1995) Clinical epidemiology: a basic
science for clinical medicine, Second edn, London: Little Brown and Company.
Sackett, D., Richardson, W., Rosenberg, W., & Haynes, R. (1997) Evidence-based
medicine: how to practice and teach EBM, New York: Churchill Livingston.
19