www.cebm.net
Evidence Based Health Care Course
Oxford, 2010
Appraising diagnostic studies
Dr Matthew Thompson
Senior Clinical Scientist & GP
What is diagnosis?
www.cebm.net
• Increase certainty about
presence/absence of disease
• Disease severity
• Monitor clinical course
• Assess prognosis – risk/stage
• Stall for time!
• 2/3 legal claims against GPs
in UK
• 40,000-80,000 US hospital
deaths from misdiagnosis per
year
• Adverse events, negligence
cases, serious disability more
likely to be related to
misdiagnosis than drug errors
www.cebm.net
• Diagnosis uses <5% of
hospital costs, but influences
60% of decision making
What we will cover today
www.cebm.net
• Diagnostic reasoning
• Basic design of diagnostic studies
• Appraising a diagnostic study in 3 easy
steps
• What do all the numbers mean?
How do clinicians make diagnoses?
High pressure, stressful, busy, difficult,
continual new and complex information…
Diagnostic reasoning


For example, what causes cough?
Comprehensive history…examination…differential
diagnosis…final diagnosis

Cardiac failure, left sided , Chronic obstructive pulmonary disease , Lung abscess
Pulmonary alveolar proteinosis, Wegener's granulomatosis, Bronchiectasis
Pneumonia, Atypical pneumonia, Pulmonary hypertension
Measles, Oropharyngeal cancer, Goodpasture's syndrome
Pulmonary oedema, Pulmonary embolism, Mycobacterium tuberculosis
Foreign body in respiratory tract, Diffuse panbronchiolitis, Bronchogenic carcinoma
Broncholithiasis, Pulmonary fibrosis, Pneumocystis carinii
Captopril, Whooping cough, Fasciola hepatica
Gastroesophageal reflux, Schistosoma haematobium, Visceral leishmaniasis
Enalapril, Pharyngeal pouch, Suppurative otitis media
Upper respiratory tract infection, Arnold's nerve cough syndrome, Allergic bronchopulmonary aspergillosis
Chlorine gas, Amyloidosis, Cyclophosphamide
Tropical pulmonary eosinophilia, Simple pulmonary eosinophilia, Sulphur dioxide
Tracheolaryngobronchitis, Extrinsic allergic alveolitis, Laryngitis
Fibrosing alveolitis, cryptogenic, Toluene di-isocyanate, Coal worker's pneumoconiosis
Lisinopril, Functional disorders, Nitrogen dioxide, Fentanyl
Asthma, Omapatrilat, Sinusitis
Gabapentin, Cilazapril

……diagnostic reasoning

















53!
Diagnostic reasoning strategies

Aim: identify types and frequency of diagnostic
strategies used in primary care

6 GPs collected and recorded strategies used on 300
patients.
(Diagnostic strategies used in primary care. Heneghan, et al,. BMJ 2009)
Diagnostic stages & strategies
Stage
Initiation of the
diagnosis
Refinement of
the diagnostic
causes
Defining the
final diagnosis
Strategy
Spot diagnoses
Self-labelling
Presenting complaint
Pattern recognition
•Restricted Rule Outs
•Stepwise refinement
•Probabilistic reasoning
•Pattern recognition fit
•Clinical Prediction Rule
Known Diagnosis
Further tests ordered
Test of treatment
Test of time
No label
Spot diagnosis
Almost instant recognition, often at the start of the
consultation, does not need more history or examination
or diagnostic tests.
*Brooks LR. Role of specific similarity in a medical diagnostic task. J Exp
Psychol Gen 1991;220:278-87
Initiation: Self-labelling



“Its tonsillitis doc– I’ve
had it before”
“I have a chest infection
doctor”
20% of consultations


Accuracy of selfdiagnosis in recurrent UTI
88 women with 172 selfdiagnosed UTIs
Uropathogen in 144 (84%)
 Sterile pyuria in 19 cases
(11%)
 No pyuria or bacteriuira in
9 cases (5%)
(Gupta et al Ann Int Med
2001)

Stage
Initiation of the
diagnosis
Refinement of
the diagnostic
causes
Defining the
final diagnosis
Strategy
Spot diagnoses
Self-labelling
Presenting complaint
Pattern recognition
•Restricted Rule Outs
•Stepwise refinement
•Probabilistic reasoning
•Pattern recognition
•Clinical Prediction Rule
Known Diagnosis
Further tests ordered
Test of treatment
Test of time
No label
Refining: Restricted rule-out (or Murtagh’s)
process

A learned diagnostic strategy for each presentation




Think of the most common/likely condition
AND… what needs to be ruled out also?
Example: patient with headache …learn to check for
migraine, tension type headache, but to rule out
temporal arteritis, subarachnoid haemorrhage etc
Used in 30% consultations
Murtagh. Australian Fam Phys 1990. Croskerry Ann Emerg Med 2003
Refining: Probabilistic reasoning


The use of a specific but
probably imperfect symptom,
sign or diagnostic test to rule
in or out a diagnosis.
E.g. urine dipstick test for
UTI
Pattern recognition
61 yr old
Male smoker
Chest pain
Signals a shift from the
study of judgment to the
study of the organisation
and retrieval of
memories
ST wave elevation
on ECG
A number of symptoms and signs are
sought, the way that they intersect
represents a recognisable pattern.
Schmidt H. Academic Medicine. 1990;65:611–21.
Regehr G.Academic Medicine. 1996;71:988–1001.
Refining: Pattern recognition



Symptoms and signs volunteered or elicited
from the patient are compared to previous
patterns or cases and a disease is recognized
when the actual pattern fits.
Relies on memory of known patterns, but no
specific rule is employed.
Used in 40% cases
Refining: Clinical prediction rules



Formal version of pattern
recognition based on a well
defined and validated
series of similar cases.
Examples: Ottawa ankle
rule, streptococcal sore
throat,
Rarely used <10% cases
Stage
Initiation of the
diagnosis
Refinement of
the diagnostic
causes
Defining the
final diagnosis
Strategy
Spot diagnoses
Self-labelling
Presenting complaint
Pattern recognition
•Restricted Rule Outs
•Stepwise refinement
•Probabilistic reasoning
•Pattern recognition fit
•Clinical Prediction Rule
Known Diagnosis
Further tests ordered
Test of treatment
Test of time
No label




Known diagnosis
Order further tests
Test of treatment
Test of time
Can’t label
0%

10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Final diagnostic stage
Defining the final diagnoses
Known
Further
tests
Diagnosis
ordered
Test of
Test of
treatment time
No Label
What we will cover
www.cebm.net
• Diagnostic reasoning
• Basic design of diagnostic studies
• Appraising a diagnostic study in 3 easy
steps
• What do all the numbers mean?
Basic structure of diagnostic studies
Series of patients
Index test
Reference (“gold”) standard
Compare the results of the
index test with the reference
standard, blinded
www.cebm.net
Read this abstract
www.cebm.net
• Scan in UTI abstract
www.cebm.net
• Scan in UTI abstract
Gold
standard
Accuracy
Series
of
patients
Index
test
www.cebm.net
Series of patients
Index test
Reference (“gold”) standard
Compare the results of the
index test with the reference
standard
Women presenting
with history of
recurrent UTIs
Self diagnosis based
on symptoms
Positive urine culture
172 episodes:
Positive urine culture
in 144 (84%)
What we will cover
www.cebm.net
• Diagnostic reasoning
• Basic design of diagnostic studies
• Appraising a diagnostic study in 3
easy steps
• What do all the numbers mean?
Appraising diagnostic tests: 3 easy steps
www.cebm.net
1. Are the results valid?
2. What are the results?
3. Will they help me
look after my patients?
Appraising diagnostic tests: 3 easy steps
www.cebm.net
•Appropriate spectrum of patients?
•Does everyone get the gold standard?
1. Are the results valid?
2. What are the results?
3. Will they help me
look after my patients?
•Is there an independent, blind or
objective comparison with the gold
standard?
Appropriate spectrum of patients?
• Ideally, test should be performed on group of
patients in whom it will be applied in the real
world clinical setting
www.cebm.net
? Appropriate spectrum
www.cebm.net
• Scan in UTI abstract
All patients have the gold standard?
www.cebm.net
• Ideally all patients get the gold /reference standard
test
Gold standard in all?
www.cebm.net
• Scan in UTI abstract
www.cebm.net
Independent, blind or objective comparison with the gold
standard?
• Ideally, the gold standard is independent, blind
and objective
Appraising diagnostic tests: 3 easy steps
www.cebm.net
1. Are the results valid?
•Sensitivity, specificity
2. What are the results?
•Likelihood ratios
•Predictive values
3. Will they help me
look after my patients?
Appraising diagnostic tests: 3 easy steps
www.cebm.net
1. Are the results valid?
2. What are the results?
3. Will they help me
look after my patients?
•Can I do the test in my setting?
•Do results apply to the mix of patients
I see?
•Will the result change my
management?
•Costs to patient/health service?
Will the result change management?
0%
Probability of disease
No action
Testing
threshold
Test
100%
Action
(e.g. treat)
Action
threshold
Will the test apply in my setting?





Will the results change my management?
Reproducibility of the test and interpretation in
my setting
Impact on outcomes that are important to
patients?
Where does the test fit into the diagnostic
strategy?
Costs to patient/health service?
What we will cover
www.cebm.net
• Diagnostic reasoning
• Basic design of diagnostic studies
• Appraising a diagnostic study in 3 easy
steps
• What do all the numbers
mean?!!!......HELP
www.cebm.net
• In the ideal world, a test would have perfect
discrimination…
i.e. all the patients who HAVE the disease
are identified by the test
AND all the patients who DO NOT have the
disease test have a negative test
How about this way??
www.cebm.net
What do all the numbers mean?!!!
Use a computer?!
How about this way??
www.cebm.net
What do all the numbers mean?!!!
Sensitivity and specificity
2 by 2 table
+
+
Test
-
Disease
-
2 by 2 table
+
+
Disease
-
True
positives
Test
-
True
negatives
2 by 2 table
+
+
Disease
-
True
positives
False
positives
False
negatives
True
negatives
Test
-
2 by 2 table: sensitivity
+
+
Test
-
Disease
-
a
Proportion of people
with the disease who
have a positive test
result.
True
positives
c
False
negatives
Sensitivity = a / a + c
So, a test with 84%
sensitivity….means
that the test identifies
84 out of 100 people
WITH the disease
2 by 2 table: sensitivity
+
+
Disease
-
84
Test
-
Proportion of people
with the disease who
have a positive test
result.
16
Sensitivity = 84/ 100
So, a test with 84%
sensitivity….means
that the test identifies
84 out of 100 people
WITH the disease
2 by 2 table: specificity
Disease
+
+
Test
b
False
positives
Proportion of people
without the disease
who have a negative
test result.
d
-
True
negatives
Specificity = d / b + d
2 by 2 table: specificity
+
+
Test
Disease
-
b
False
positives
d
-
True
negatives
Proportion of people
without the disease
who have a negative
test result.
So, a test with 75%
specificity will be
NEGATIVE in 75 out
of 100 people without
the disease
Specificity = d / b + d
2 by 2 table: specificity
Disease
+
+
Test
b
25
d
-
75
Proportion of people
without the disease
who have a negative
test result.
So, a test with 75%
specificity will be
NEGATIVE in 75 out
of 100 people without
the disease
Specificity = 75/100
Tip…..

Sensitivity is useful to me


Specificity seems a bit confusing!


‘The new rapid chlamydia test was positive in 47 out of 56 women
with chlamydia (sensitivity =83.9%)’
‘The new rapid chlamydia test was negative in 600 of the 607
women who did not have chlamydia (specificity = 98.8%)’
So…the false positive rate is sometimes easier


False positive rate = 1 – specificity
So a specificity of 98.8% means that the new rapid test is wrong (or
falsely positive) in 1.2% of women
2 by 2 table: specificity
Chlamydia Lab
PCR
+
rapid
test
vaginal
swab
+
-
7
600
-
There were 607
women who did not
have chlamydia…
the rapid test was
falsely positive in 7
of them
607
Specificity = 600/607 = 98.8%
False positive rate = 1-specificity = 1.2%
How about this way??
www.cebm.net
What do all the numbers mean?!!!
Natural frequencies
Natural frequencies
Your father went to his doctor and was told that his
test for a disease was positive. He is really
worried, and comes to ask you for help!

After doing some reading, you find that for men
of his age:
The
prevalence of the disease is 30%
The test has sensitivity of 50% and specificity of 90%

“Son, tell me what’s the chance
I have this disease?”
Prevalence of 30%, Sensitivity of 50%, Specificity of 90%
Disease +ve
Sensitivity
= 50%
30
100
Disease -ve
15
Testing +ve
70
False
positive
rate = 10%
7
22 people
test
positive……….
of whom 15
have the
disease
So, chance of
disease is
15/22 about
70%
Try it again



A disease with a prevalence of 4% must be
diagnosed.
It has a sensitivity of 50% and a specificity of
90%.
If the patient tests positive, what is the
chance they have the disease?
Prevalence of 4%, Sensitivity of 50%, Specificity of 90%
Disease +ve
Sensitivity
= 50%
4
100
Disease -ve
2
Testing +ve
9.6
96
False
positive
rate = 10%
11.6 people
test
positive……….
of whom 2
have the
disease
So, chance of
disease is
2/11.6 about
17%
Doctors with an average of 14 yrs experience
….answers ranged from 1% to 99%
….half of them estimating the probability as 50%
Gigerenzer G BMJ 2003;327:741-744
How about this way??
www.cebm.net
What do all the numbers mean?!!!
Likelihood ratios
Likelihood ratios


Can use in situations with more than 2
test outcomes
Direct link from pre-test probabilities to
post-test probabilities
Likelihood ratios
Positive likelihood ratio (LR+)
How much more likely is a positive test to be found in a person
with the disease than in a person without it?
LR+ = sens/(1-spec)
Negative likelihood ratio (LR-)
How much more likely is a negative test to be found in a person
without the condition than in a person with it?
LR- = (1-sens)/(spec)
2 x 2 table: positive likelihood ratio
+
+
a
Disease
-
How much more often a positive
test occurs in people with
compared to those without the
disease
b
LR+ = a/a+c / b/b+d
Test
-
c
d
or
LR+ = sens/(1-spec)
What do likelihood ratios mean?
LR<0.1 = strong
negative test
result
LR=1
No diagnostic
value
LR>10 = strong
positive test
result
McGee: Evidence based Physical Diagnosis (Saunders Elsevier)
%
Bayesian
reasoning
Pre test 5%
Post test 20%
? Appendicitis:
McBurney tenderness
LR+ = 3.4
%
Fagan
nomogram
What’s wrong with PPV and NPV?

Depend on accuracy of the test and
prevalence of the disease
Summary: diagnostics
www.cebm.net
• Clinical decision making
• All diagnostic studies share a similar structure
• Appraise in 3 steps:
1. Are the results valid (do I believe them)?
2. What are the results (sens/specificity etc)?
3. Can the test be used in our setting?
Appraising diagnostic tests: 3 easy steps
www.cebm.net
1. Are the results valid?
2. What are the results?
3. Will they help me
look after my patients?