Knight Health & Medical
Journalism
April 18, 2008
Grady College, The University of Georgia
jeanne Lenzer
jeanne.lenzer@gmail.com
Jeanne Lenzer April 18, 2008
How statistics and spin can distort
science and what you as journalists
can do about it
Jeanne Lenzer April 18, 2008
How do we get results…
From:
To:
A new drug,
(tPA, a clot
buster) “saves
lives” in acute
stroke
Jeanne Lenzer April 18, 2008
Clotbusters
increased
deaths or had
no effect on
mortality in
acute stroke
How do we get results…
From:
To:
“Antidepressants “Antidepressants
prevent suicides” cause suicides”
Jeanne Lenzer April 18, 2008
How do we get results…
To:
From:
Lowering
cholesterol
reduced
cardiovascular
deaths
Jeanne Lenzer April 18, 2008
Cholesterol
lowering had no
effect on
mortality
An analysis of the most-cited studies
showed that the results of onethird (14 of the 49) of original
research studies, were either “false
or exaggerated.”
Jeanne Lenzer April 18, 2008
How they (try to) fool us
Hide data in plain sight
Suppress data
Jeanne Lenzer April 18, 2008
Hiding in plain sight – Page 1
 Randomization bias, publication bias
 Healthy volunteer bias, inclusion bias, spectrum
bias…
 Subgroup analysis, data dredging, post-hoc
analysis
 Straw man comparators
 Combination endpoints
 Multiple comparators
 Unblinded arms
 Shifting goalposts
Jeanne Lenzer April 18, 2008
Hiding in plain sight – Page2
• Underpowering
•
•
•
•
Segregation of data
Surrogate endpoints
Disease-specific mortality
Failure to balance harms with reported benefits: The
word “benefit” does not mean “net benefit”
• Framing
• Lying
• Run-in and wash-out
Jeanne Lenzer April 18, 2008
M’English
Plain language meanings are not
medical language meanings
Jeanne Lenzer April 18, 2008
Some M’English words that should
make your ears jerk forward
 The words:
 “And”
 “Significant”
 “Benefit”
 “Mortality”
 A special shout-out to the word “may”
Jeanne Lenzer April 18, 2008
A story about “And”
It all started with a rumor
Jeanne Lenzer April 18, 2008
M’English
 AHA claimed drug “saved lives”
 How did they make the claim without lying?
Jeanne Lenzer April 18, 2008
Genentech and the AHA
Despite enormous controversy over the safety and
efficacy of the clot buster, tPA, the American Heart
Association rated tPA as a class I (definitely
recommended) intervention for acute stroke in August
2000
Jeanne Lenzer April 18, 2008
The house that Genentech built
Jeanne Lenzer April 18, 2008
Genentech and the AHA
 Genentech, the US manufacturer of alteplase,
contributed over $11 million to the American Heart
Association in the decade before its recommendation
of tPA for stroke – giving $250,000 to rebuild its
headquarters in Houston.
 Seven of the nine AHA panelists had ties to Genentech
(and its marketing partner, BI)
Jeanne Lenzer April 18, 2008
Some conflicts over tPA
 Six controlled studies showed tPA either





increased mortality or had no effect on mortality
NINDS, the single trial used to obtain FDA
approval, was showed no effect on mortality
There were significant baseline imbalances in the trial
Enrollment was artificially tilted to enhance possible
benefit (more patients in early treatment)
Real world experience with tPA has been poor
Potential benefit was marginal and came with a
dramatic increased risk of cerebral hemorrhage
Jeanne Lenzer April 18, 2008
It didn’t “save lives”??
 Walk, Talk or Die
 “Brain attack” campaign is an industry now with new
Joint Commission “stroke center” certification
 NYT and others still claiming tPA “saves lives” despite
AHA withdrawal of the claim
Jeanne Lenzer April 18, 2008
So how did they do it?
 Back to M’English 101:
 The word “and” and combination
endpoints
Jeanne Lenzer April 18, 2008
“We don’t know how another trial would
turn out. And if we don’t come out ahead,
we would have a tremendously selfinflicted wound…[Another study] may
be a good thing for America, but it
wasn’t going to be a good thing for
us.”
- Genentech scientist commenting on tPA for myocardial
infarction
Marsa L. Prescription for Profits: How the Pharmaceutical
Industry Bankrolled the Unholy Alliance Between Science and
Business. New York: Scribner, 1997
Jeanne Lenzer April 18, 2008
Significant to me may not be…
significant to you…
especially if I am a statistician and
you are a patient
Jeanne Lenzer April 18, 2008
Why tPA – why not gyms?
 Only three to four of every 1,000 stroke patients can
potentially benefit from tPA;
 Yet the “brain attack” campaign has diverted millions
of dollars to help stroke patients get tPA;
 Why not invest in workplace gyms since they are
proven to improve employee health and regular
exercise leads to a 27% reduction in strokes – not just
for a few – but for all, according to the American Heart
Association.
Jeanne Lenzer April 18, 2008
How do we get results…
From:
To:
“Antidepressants “Antidepressants
prevent suicides” cause suicides”
Jeanne Lenzer April 18, 2008
Antidepressants
…and the word “benefit”
Jeanne Lenzer April 18, 2008
In M’English…
the word, “Benefit” does not imply
net benefit.
Jeanne Lenzer April 18, 2008
Jeanne Lenzer April 18, 2008
Benefits may be reported
 On one scale without reporting negative outcomes on
the same scale (see “Journalists on Prozac”)
 From unblinded arms
 Without regard to harms – separate clinical trials are
conducted to test “efficacy” versus “safety”
Jeanne Lenzer April 18, 2008
Safety “signals” and ADs
 Multiple efficacy studies showed a “signal” of
increased suicidality, but because they were
underpowered for safety analyses, they were dubbed
“non-significant.”
 The problem of under-powering and publication bias
resulted in concealment of what the FDA eventually
found in pooled analyses of RCTs: doubling of
suicidality
Jeanne Lenzer April 18, 2008
Efficacy
 Publication bias (Turner and Kirsch) revealed that
most negative studies are never reported and if they
are, they aren’t reported as negative;
 Virtually all positive trials were reported;
 When unpublished studies are considered, roughly
half of all trials were negative.
Jeanne Lenzer April 18, 2008
Current claims…
 Since black-box warning issued, suicides have
increased among teens…
 Heirarchy of evidence; observational versus
randomized controlled trials
Jeanne Lenzer April 18, 2008
Systematic reviews
Hierarchy of study methods for
clinical decision-making
How do we get results…
To:
From:
Lowering
cholesterol
reduced
cardiovascular
deaths
Jeanne Lenzer April 18, 2008
Cholesterol
lowering had no
effect on
mortality
A nosy neighbor who is watering a bush outside
the apartment building says to a woman who lives
in one of the apartments, “I see that you sure do
come and go a lot.”
The woman responded, “Well, you do stand right
outside the building all the time. If you stood
next to my toilet you’d say I pee all the time.”
From Sara Silverman
Jeanne Lenzer April 18, 2008
Spot the problems…
 “The researchers reported a 32%
reduction in cardiovascular mortality in
the group of men who took statins.”
 Des Spence, BMJ, 2008;336:100 (12 January)
Jeanne Lenzer April 18, 2008
The cholesterol debate
 Framing and relative risks vs absolute risks;
 Cause-specific vs all-cause mortality
 Which populations?
 The use of surrogate markers
Jeanne Lenzer April 18, 2008
Relative vs Absolute Risk Reduction
 RRR of 32% in cardiovascular mortality translates into
->
 ARR of 0.7% in cardiovascular mortality -> NNT of
715, meaning that each year, of 715 men taking statins,
714 will receive no benefit.
Jeanne Lenzer April 18, 2008
Once again “benefit” is not…
 necessarily net benefit – the study only reported a
reduction in cardiovascular mortality – not all-cause
mortality;
 “Dying with a corrected cholesterol level is not a
successful outcome in my book.” John Abramson
Jeanne Lenzer April 18, 2008
All-cause morbidity and mortality…
 “In all groups except middle aged men with heart
disease – or at high risk for it – whatever benefits
statins have to the heart are completely offset by [an]
increase in other causes of death and serious
problems.”
 Beatrice Golomb, MD, PhD, researcher UCSD
Jeanne Lenzer April 18, 2008
Surrogate markers
 High cholesterol is not a disease –
 Nor is high blood pressure –
 To show benefit we must examine clinical endpoints
Jeanne Lenzer April 18, 2008
Costs over lives?
 FDA wants to increase the use of surrogate markers to
reduce research costs – but what is the cost of
ezitimibe (Zetia and Vytorin) and years of ineffective
and possibly dangerous treatment?
Jeanne Lenzer April 18, 2008
Niche marketing: The Race Drug
BiDil: Original study, V-HEFT, was of all people with
CHF
 Benefit in V-HEFT reached marginal statistical
significance
 Subgroup analysis suggested benefit among selfidentified Black people
Jeanne Lenzer April 18, 2008
Men/ White people
Women/Afric
an Americans
Men
Women
BiDil: Miracle drug or miracle of marketing?
NitroMed announced that “[D]eath rates
from heart failure are more than twice
as high in black patients than in white
patients.”
How a Drug Becomes “Ethnic”: Law, Commerce, and the
Production of Racial Categories in Medicine by Jonathan
Kahn, J.D., PhD., Yale Law Review
Jeanne Lenzer April 18, 2008
 A-HEFT trial which “proves” benefit did not include
white people (no comparator).
Jeanne Lenzer April 18, 2008
Dr Jay Cohn’s happy find
 Applied for a patent using combination hydralazine
and isosorbide dinitrate in 1987 with no mention of
race
 FDA said “too many variables specified as primary
endpoints” and did not approve BiDil for general
population
 Stock in Medco plunged 25% next day
Jeanne Lenzer April 18, 2008
 1997 estimate suggested $60,000,000 in annual sales of
BiDil would be possible
 March 2001 BiDil became the first “race drug” – FDA
approved it for use in African Americans only
 What is BiDil? Two old standbys, isosorbide dinitrate
and hydralazine, used for ages without regard or
reason for concern about race in patients with CHF
Jeanne Lenzer April 18, 2008
Booyah! (Or: how to market race
when drugs run off patent)
 Costs? Four times more expensive:
 30 days of BiDil®: $195 ($2.17 / dose)
 30 days of generic equivalents of same drugs: $52 ($0.45 /
dose)
 Plus the usual greasing of palms: NitroMed gave
money to a number of African American groups that
subsequently praised BiDil.
Jeanne Lenzer April 18, 2008
Exaggerating risk of disease; minimizing risk
of drugs
 CDC data show the two-fold mortality rate among self-
identified blacks does not exist – rate is closer to 1.1
and reduced with standard therapy
 Exaggerate role of “genes” – some study authors
specifically minimized the role of economic,
environmental, and psychosocial factors.
Jeanne Lenzer April 18, 2008
NEJM Editorial by Bloche
 “A-Heft has not established that adding isosorbide
dinitrate and hydralazine to conventional therapy for
heart failure yields greater benefits for blacks than for
other racial or ethnic groups.”
 This “race-specific drug was driven in large measure by
regulatory and market incentives.”
Jeanne Lenzer April 18, 2008
The creation of a race drug
 “We may be finding that in some blacks, when the
choke hold is applied, the veins or arteries do not
open up like in normal people.”

Daryl Gates, LAPD Commissioner: May 6,
1982
Jeanne Lenzer April 18, 2008
On the FDA and race:
 “Apparently the people at NitroMed felt the same way
when they developed BiDil, as did the FDA when they
approved it.”
 Joe Lex, M.D.
Jeanne Lenzer April 18, 2008
Recapping Bidil (how they did it)
 Subgroup analysis (underpowering)
 Multiple comparators V-HEFT
 Lack of comparator group in A-HEFT
 Exploiting potential mean group variances as if groups
are mutually exclusive
 Exaggerating risk of disease; minimizing risk of
drugs: doubled death rate doesn’t exist; CDC data
show RRR 1.1 – and that even diminishes with tx; onethird of pts taking hydralazine develop lupus within
three years
Jeanne Lenzer April 18, 2008
Drugs on market with no clinical
evidence of benefit…
 Ezitmibe approved on the basis of its ability to lower
cholesterol only.
Jeanne Lenzer April 18, 2008
History instructs…
 The cholesterol-lowering drug, torcetrapib, was very
effective at lowering cholesterol levels. But a study
underway during the FDA approval process was shut
down when the drug was seen to increase total
mortality.
Jeanne Lenzer April 18, 2008
Critics observe:
 “If torcetrapib had been approved solely on the basis of
its ability to [affect] cholesterol levels, the drug might
have been marketed for many years before the increase
in cardiovascular events was detected.”
Psaty and Lumley JAMA 2008;299:1474;
Jeanne Lenzer April 18, 2008
The operation was a success!*
*But the patient died
Jeanne Lenzer April 18, 2008
What journalists can do
 Think devious (it’s a must)
 Read:
 Evaluating Clinical Research, by Furberg and Furberg
 Epidemiology in Medicine by Hennekens and Buring
 Medical Uses of Statistics by Bailar and Mosteller
 Consult independent experts:
 List available from Overtreated.com or
Brownlee.Lenzer@gmail.com
Jeanne Lenzer April 18, 2008
What journalists can do
 Don’t allow experts to intimidate you
 Always inquire about financial cois
 Follow 10-point guidelines of HealthNewsReview.org
for good journalism
Jeanne Lenzer April 18, 2008