Lost in Translation:
Innumeracy in Medicine and
Dealing with Uncertainty
Brian Chan MD
IM PGY3
Senior Talk Jan 25 2013
Objectives
• Characterize innumeracy and difficulties in
risk communication
• Illustrate physicians own innumeracy and
approach to uncertainty
• Describe tools to deal with innumeracy to
help ourselves and our patients
• Demonstrate how we use statistical thinking
in practice of modern medicine
Relevance:
QuickTime™ and a
decompressor
are needed to see this picture.
Quic kTime™ and a
dec ompres sor
are needed to see this pic ture.
QuickTime™ and a
decompressor
are needed to see this picture.
Data Requires Context:
QuickTime™ and a
decompressor
are needed to see this picture.
QuickTime™ and a
decompressor
are needed to see this picture.
QuickTime™ and a
decompressor
are needed to see this picture.
Case 1:
41 year old female “Ms. Odds’ with
history of hypertension, otherwise
healthy comes into clinic for follow up.
She has a friend whose mother died
from breast cancer and is wondering if
she should get a screening
mammogram…
Choices:
• She returns a month later, stating she went
ahead and got a mammogram from a
screening fair-- its come back positive, what
is her chance she has breast cancer?
A. 90%
B. 70%
C. 50%
D. 30%
E. 10%
F. 1%
Background Data:
• Age 40-50 yo women
• Incidence of breast cancer in 10 years= 1.4%
• Sensitivity of mammogram test (If B+ then
M+) = 75%
• False positive rate of mammogram (If B- then
M+) =10%
• If Ms. Odds has a positive mammogram, what
is her chance of breast cancer?
Choices:
•
•
•
•
•
•
A. 90%
B. 70%
C. 50%
D. 30%
E. 10%
F. 1%
Encountering a Cougar
Answer:
•
•
•
•
•
•
A. 90%
B. 70%
C. 50%
D. 30%
E. 10%
F. 1%
Out of 100 women in
age 40 group with a
positive screening
mammogram, 10 will
have diagnosis of
breast cancer
Who is this man and why should we care?
QuickTime™ and a
decompressor
are needed to see this picture.
Google 2013
QuickTime™ and a
decompressor
are needed to see this picture.
QuickTime™ and a
decompressor
are needed to see this picture.
Innumeracy
• Inability to reason about uncertainties
and risk*
– Illusion of certainty
– Ignorance of risk
– Miscommunication of risk
– Clouded thinking/drawing inference
Uncertainty versus Risk
Risk is the quantification of uncertainty
into a probability or frequency based on
empiric data
Quantification of risk can take a number
of forms….
How We Quantify Risk
A. Degrees of belief - subjective probability
B. Propensity - intrinsic properties of an object
C. Frequency - “incidence”
-
-
Relative frequency of event in specified
reference class (icu population, population of
BMT patients, population of intubated pts)
For frequentists, risks can only be defined in
situations where a large body of empirical data
exists (what about in situations of few cases)
Risk “mis”communication
• A. single event probabilities
– “you have 30% chance of side effect”
• Lack of reference causes confusion
B. Comparing treatment effects
- RRR - largest effect, drug companies
- ARR - how patients like to hear risks from MD
- NNT - how we communicate to peers
C. Conditional probabilities
- sensitivity is not the same as PPV of a test
Overcoming innumeracy
• 1. Defeat illusion of certainty
• 2. Learn about the actual risks of
relevant events and actions
• 3. Communicate the risk in an
understandable way and draw
inferences without falling to “clouded
thinking”
Illustrating innumeracy with
my favorite San Diegan
QuickTime™ and a
decompressor
are needed to see this picture.
Anchorman. Dreamworks picture. Accessed from youtube.
Back to our case:
•
•
•
•
•
Data: age 40-50 yo women
P (B+) = 1.4% (incidence)
If B+ then M+ = 75% (sensitivity)
If B- then M+ =10% (false positive)
If Ms. Odds has a M+, what is her
chance of B+ P(B+|M+) ?
Bayes Theorem
• P(disease|positive) = a / a+b
•
•
P(disease|positive) = p(disease)p(positive|disease) /
P(disease)p(positive|disease) + p(1-disease)p(positive|no disease)
Test result
Disease
Yes
No
Positive
(a) Sensitivity
(b) False positive
rate
Negative
(c) False negative
(d) specificity
Conditional probabilities
versus natural frequencies
Method 1: probability (x,y,z)
P(disease) = x = .014
P(pos | disease) = y = .75
P(pos | no disease) = z = .10
= xy / (xy + (1-x) z
= .014 x .75 / (.014 x .75 + .986 x .10) = 10%
Method 2:
Representing Bayes Theorem through Frequencies
1,000 people
14
disease
12
positive
2
negative
986
no disease
99
positive
887
negative
P(B+|M+) = 12 / (12 + 99) = 10%
Physicians And Innumeracy
QuickTime™ and a
decompressor
are needed to see this picture.
A Way Forward to Numeracy
• Presenting data in the right context can
lead to insight
• 1. Single event probabilities
• 2. Conditional probabilities
• 3. Relative Risks
Common Scenarios
Data type:
Example:
Potential Solution:
Single Event
Probabilities
“30% risk of side effect”
Use frequency statement: “3 out of
10 patients have side effects”
Conditional
probabilities
Sensitivity, specificity,
positive predictive value
Use natural frequencies to
represent Bayes Theorem
Relative Risk
4 of 1000 mortality in
group A; 3 of 1000
mortality in group B:
Relative risk reduction is
25%
Use absolute risks (with pts) or
number needed to treat (with
peers): 1000 patients need to be
screened/treated to prevent 1
death
Adapted from gigerenzer, edwards, “simple tools for understanding risks: from innumeracy to insight.” BMJ 2003
Photo break
QuickTime™ and a
decompressor
are needed to see this picture.
Part II.
From Numeracy to Prediction
• Numeracy allows us to realize
uncertainty in medicine
• Probability and prediction as a waypoint
between ignorance and knowledge
• Prediction = hypothesis testing
• Separating ‘Signal’ and ‘Noise’
Bayes (again):
clinical reasoning
• 1. “Prior” (pre-test probability) - “x”
• 2. Conditional probability of new data
given prior hypothesis is true - “y”
• 3. Conditional probability of new data
given prior hypothesis is false - “z”
________________________________
= Posterior probability xy / [ xy + z(1-x) ]
Dealing with uncertainty
QuickTime™ and a
decompressor
are needed to see this picture.
Probability of a terror attack, given
plane crashed into the WTC
• “prior”: 1 in 20,000 (.005%) terror attacks
• New event occurs: plane hits WTC
– probability that a plane hits a building if terrorists were to
attack (100%, conservative)
– Probability that a plane hits a building if terrorists were not to
attack (1 in 12,500, .008%)
• Posterior probability that terror attack occurred given
plane crash is now is 38%
2nd plane crash
• Prior P(terror attack): .005%--> 38%
• New event: plane crash
– P(Crash | terror) 100%
– P(Crash | non-terror) .008%
P(terror attack | plane) --> 99%
pre-test probability is key
QuickTime™ and a
decompressor
are needed to see this picture.
Bayes vs Fisher
QuickTime™ and a
decompressor
are needed to see this picture.
• Fisher = frequentism - the p value
– Bayesian prior is “too subjective”
– Develop methods to free that bias
– Uncertainty is derived from sampling, intrinsic to
the experiment, not intrinsic to our understanding
of the world
– Collect more data to decrease uncertainty
Applicability to clinical
medicine
• Can we truly leave subjectivity out of
clinical reasoning?
– Prior experience with patients
– Our medical knowledge
– Our values of practice
Photo break 2
QuickTime™ and a
decompressor
are needed to see this picture.
QuickTime™ and a
decompressor
are needed to see this picture.
Conclusion:
Risks of Data Overload
• Explosion of new diagnostic tests heighten
risks for false positives
• Explosion of published research (most which
may be questionable*) - how can we sort
through signal and noise?
• False positives make our predictions more
prone to fail
• “Before we demand more of our data, we
need to demand more of ourselves.”
Ionnidis Jpa. Why most published research findings are false. Plos med 2005 august;2(8): e124
QuickTime™ and a
decompressor
are needed to see this picture.
Conclusions (cont)
A first step is to appreciate innumeracy
exists in patients and clinicians alike.
- innumeracy prevents meaningful
quantification of uncertainty
- Impairs our ability to communicate risk
to patients and ourselves
Conclusions (cont)
Acknowledge subjectivity and uncertainty
in medicine, yet strive to minimize bias,
minimize irrationality through practice
– Making predictions based on our beliefs
allows us to test our priors
– Use evidence based medicine wisely
– Prediction, testing, revision will allow us to
converge toward truth/signal
In Summary
.. [BS] is inevitable in situations that require us to talk or write
about something we do not understand. This is one of the
dangers in perpetuating the myth of the omnisicent physician
In some situations, we may not know with sufficient confidence
what is true and what is not. In those cases, we should be as
truthful as we can… we must be vigilant in our efforts to assess
the truth of what we suppose we know. When we are wrong,
we should be the first to admit it. In all cases, it is vital that we
commit to veracity.”
thanks
•
•
•
•
•
•
•
Sima desai
Marc gosselin
Elizabeth allen
Laura zeigen - ohsu library services
The senior residents whose shoulders I stand upon: marissa, darcy, sam,
sarah, nancy, andre
Jess bordley, my first co-intern
Pete sullivan
references
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Calculated Risks. Gerd gigerenzer. Simon&schuster. 2002
The signal and the noise. Nate silver. The pengin press 2012
Presentation. Is a picture worth a 1000 words? Communicating evidence to patients. Courtesy of elizabeth allen.
Bayes calculator http://psych.fullerton.edu/mbirnbaum/bayes/BayesCalc.htm
http://www.cdc.gov/cancer/breast/statistics/age.htm
Gigerenzer g, galesi m. why do single event probabilities confuse people. BMJ Jan 12. 1-3
Galesic M, Gerg Gigerenzer. Natural frequencies help older adults and people with low numaracy to evaluate medical
tests
Ahmed H, Naik G. Communicating risk. BMJ 2012; 344: e3996.
Kent DM, Hayward RA. Limitations of applying summary results of clinical trial to individual patients. JAMA,Sept 12,
2007- vol 298. 10.
How doctors think. Karthyn montgomery. Excerts. 2006
Gigerenzer, G, Edwards A. Simple tools uor understanding risks: from innumeracy to insight. BMJ vol 327. 27
September 2003
Odette w, schwartz L, et al. do physician understand cancer screening statistics? A national survey of primary care
physicians in the US. Annals of Internal medicine 2012;156:340-349.
Editorial. What we don’t know can hurt our patients: physician innumeracy and overuse of screening tests. Annals of
internal medicine 392-3
Education and the art of uncertainty. Richard Gunderman. Radiology 2005.
Sonnenberg A, Gogel. “translating vaguecomplaintsinto precise systmes.” euro journal of gastroenterology and
hepatology 2002; 14317-321.
Gunderman RB. Bullshit. Journal of the american college of radiology 2010. vol 7, issue 1. p 13-15
Innumeracy is a barrier to risk
communication
• Miscommunication of single event
probabilities (lacking reference class)
– Ex. 30% chance of side effect
– Out of 10 patients, 3 patients tend to report
side effect
– Risk communication requires a clear
statement of a what a probability refers to
• Frequency statements can help reduce
confusion
Use Framing Wisely
QuickTime™ and a
decompressor
are needed to see this picture.
Final Exam
•
•
•
•
•
Heme occult test -(ER GI bleed special)
Prevalence 0.3%
Sensitivity 50%
False positive rate 3%
Positive predictive value?
Example Using Conditional
Probabilities*
• Probability that person has insulin dependent
diabetes 0.5%
• IF patient has diabetes, will have a 95%
positive screening genetic test.
• IF patient does not have diabetes, 50% still
test positive on a screening genetic test.
• Estimate probability that a patient has
diabetes if positive test.
•Galesic, Gigerenzer. Natural frequences help older adults and people with low numeracy to evaluate medical screening tests.
•Med Decis making 2009 29;368
Case 2:
• You get a call from the surgical floor intern
about a 79 yo patient here for LOA with
increasing oxygen requirement likely needing
bipap. No other history (of course!). Cxr
obtained before night states:
• “support hardware noted. Bilateral consolidations
are appreciated, concern for pneumonia, chf,
hemothorax, ILD, malignancy cannot be excluded.
Clinical correlation is advised.”