Foolish & Fatal Flaws
1
When medical science goes bad
Outline
2





Background on statistical and methodological error
On Pigs and PCCs
 Crossing the species barrier
Correct group assignment
 What? You mean some of the control patients actually got transfused?
Large starting group consents, but few randomized & deceptive
statistics
 You mean only 6.8% of patients consenting to the trial were actually
randomized? You mean there are missing statistical calculations?
‘Temporal ambiguity’
 Outcome before exposure – What? The patient got pneumonia before
the transfusion?
3
The use of statistics in medical diagnoses and
biomedical research may affect whether individuals live
or die, whether their health is protected or jeopardized,
and whether medical science advances or gets
sidetracked. [...] Because society depends on sound
statistical practice, all practitioners of statistics,
whatever their training and occupation, have social
obligations to perform their work in a professional,
competent, and ethical manner.”
[Ethical Guidelines for Statistical Practise, American Statistical
Association,1999].
When transfusion medicine gets
sidetracked
4
Recombinant factor VIIa
 Transfusion-related immunomodulation
 Formula-driven resuscitation
 Albumin use in critical care
 RBC transfusion in patients with ischemic heart
disease

Where can you go wrong?
Strasak AM, et al. Swiss Med Wkly 2007; 137: 44-49
5

Study design





Failure to a priori define outcomes
Failure to perform sample calculations
Failure to blind (or disclose who the blinding was done)
Control and treatment groups not comparable
Data analysis



Messing up on simple statistical tests – i.e. using a t-test without
meeting test requirements or appropriate correction tests
When comparing multiple groups – can’t use ‘two-group’ tests
Post-hoc subgroup analysis – ‘shopping’ for statistically significant
results
Where can you go wrong?
Strasak AM, et al. Swiss Med Wkly 2007; 137: 44-49
6

Documentation
 ‘Where
appropriate’ use of a statistical test
 Failure to state the number of ‘tails’ used for a specific
analysis

Presentation
 Correct
use of standard deviation vs. 95% confidence
interval vs. inter-quartile ranges
 Median vs. mean
 P values – record exactly – not ‘ns’, ‘<0.05’, ‘>0.05’ –
no cheating when you round your p values
Incongruence between test statistics and P values
in medical papers
BMC Med Res Methodol. 2004; 4: 13.
7




11.6% and 11.1% of the statistical results published in
Nature and BMJ respectively during 2001 were incongruent,
mostly due to rounding, transcription, or type-setting errors
At least one such error appeared in 38% and 25% of the
papers of Nature and BMJ, respectively
In 12% of the cases, the significance level might change one
or more orders of magnitude
The frequencies of the last digit of statistics deviated from
the uniform distribution and suggested ‘digit preference’ in
rounding and reporting (a.k.a. lying)
Should see uniform distribution
8
More haste, less science?
Nature. 1999 Aug 5;400(6744):498
9
10
Details 74 types of medical bias
Name
Centripetal Bias
Mode for mean bias
Obsequiousness bias
Example
Healthcare access bias
Frequency-quantity Q to
assess EtOH intake, subject
reports modal rather than
mean intake (closer to zero)
Subjects alter responses in the
direction they perceive is
desired by the investigator
What are the Contributing Factors to Misuse?
11




Pressures to publish, produce results, or obtain
grants
Career ambitions or aspirations
Conflicts of interest and economic motives
Inadequate supervision, education, or training
Gardenier JS, Resnik DB. The misuse of statistics: concepts,
tools, and a research agenda. Account Res. 2002;9:65–74.
12
Bottom line: Be on the look out for
medical science flaws!
On Pigs and PCCs
13
Prothrombin complex concentrate vs fresh
frozen plasma for reversal of dilutional
coagulopathy in a porcine trauma model.
Dickneite G, Pragst I.
Brit J Anesthesia 2009; 102: 345-54.
The study was funded and conducted by representatives
of the company that makes the PCC product
The study design
14


47 pigs, 20-30 kg, anesthetized, 70% isovolemic blood loss
with replacement with RBCs/HES
The pigs were randomized to (5-7 each group):






15 mL/kg of porcine FFP
40 ml/kg of porcine FFP
25 U/kg of human-derived PCC (Beriplex P/N)
15 mL/kg saline
Porcine FFP was used instead of human FFP because infusion
of human FFP into pigs can result in transfusion reactions
Following this resuscitation, the pigs underwent a controlled
injury

3 mm hole into the femur or 7 cm x 1cm incision into the spleen
Outcomes – measured by blinded observers
15




Time to hemostasis and blood loss were monitored
for 120 min after injury
Skin bleeding time (SBT) in duplicate
Blood samples were collected at baseline, after the
completion of hemodilution and 5 min after study
treatment administration
Coagulation factors (2/7/9/10) were measured
 They
did not measure the non-vitamin K dependent
factors
The effect on the PT – all 3 work
16
Factor X – only PCC works
17
Expected
results
Human equiv
11 units FFP
Proportion with hemostasis
Better off with saline than FFP?
18
PCC
Saline
FFP - low
FFP – high
Spleen blood loss
19
Bone blood loss
20
FFP-low
Saline
FFP-high
PCC
Skin bleeding time
21
Fibrinogen level
22
2 ½ doses
Of FFP?
The investigators conclusion
23

In view of the unmet clinical need for
more efficacious haemostatic agents in
such patients, clinical studies are now
justified to confirm the observed
favorable effects of PCC in the present
preclinical model system
A lot of unanswered questions
24

The laboratory tests used to measure clotting factor levels
are based on human factor-deficient plasma




Do the lab assays work for human and pig samples for all factor
assays performed?
Perhaps this explains why PT corrects but not factor assays
Why would massive FFP exposure increase blood loss in
these pigs? Was there something wrong with their pig FFP?
Is hemodilution with FFP a bad thing? Did they fail to give
enough RBCs?
Why would human PCC immediately stop bleeding in these
pigs and FFP would make you bleed more than saline?
Comment
25

Pig vs. Human comparison




Infusion of human factors into pig recipients via PCC may have
induced enhanced coagulation compared with what those same
proteins would achieve in a human recipient?
The clinical bleeding outcomes observed between animals infused
with pig FFP versus animals infused with human PCC in this
experiment are difficult to compare directly
The real value of PCCs compared with FFP in human trauma
cannot be answered by this experimental design
If you are a pig and you get injured, you want human PCC!
Caution
26

Make sure your trauma surgeons do not take this
study at face value
Correct group assignment
When you untransfused control group might be transfused
27
Intraoperative transfusion of 1 U to 2 U
packed red blood cells is associated with
increased 30-day mortality, surgical-site
infection, pneumonia and sepsis in general
surgery patients.
Bernard AC,Davenport DL,Chang PK, et
al. J Am Coll Surg 2009; 208: 931-37
Design
28



Prospective study of patients undergoing major
surgical procedures at 121 hospitals
Nurses prospectively collected preoperative,
intraoperative and postoperative variables for 30
days after the operation on the first 40 operations
in each 8-day cycle
Database was queried for 05-06 for all general
surgery procedures
Transfused vs. not transfused
29

‘Transfusion’ =
 number
of PRBC units transfused intraoperatively
 Transfused >4 U RBCs is the first 72 hours post-op

2nd group = yes/no definition, not number of units
Time course
30
Data variable only
72 hours
OR
>4 units
yes/no
>4 units
yes/no
=Transfused
Intraoperative
number
transfused
30 days
Outcomes over 30 days
31







Composite surgical-site infection (superficial, deep, or
organ/space)
Urinary tract infection
Pneumonia (without preoperative pneumonia)
Sepsis/septic shock
Composite morbidity – all of the above
1 or more of 20 adverse events uniformly defined by the
ACS-NSQIP, excluding bleeding requiring transfusion
Mortality
Analysis
32


Estimated probability for a patient to receive a transfusion
(propensity) was calculated by MLR analysis of all the
available patient and operative risk factors
Risks for outcomes by level of intraoperative transfusion were
calculated using logistic regression, with adjustment for:





transfusion propensity, procedure group, and complexity
other ACS-NSQIP risk factors
operative duration (proxies for technique),
level of transfusion received intraoperatively
postoperative transfusion of >4 U PRBCs
Results
33



125,223 general surgery patients at 121 hospitals
were retrieved
4,788 patients (3.8%) received intraoperative RBC
or >4 in the 72 hours post
Risk variables most predictive of transfusion were
inpatient procedure, procedure group, ASA class,
hematocrit >38!, preoperative transfusion >4 U,
emergent procedure, esophageal varices, and age
Results
34


Patients receiving a single unit of intraop RBCs had higher
rates of surgical-site infections, urinary tract infection,
pneumonia, sepsis/shock, composite morbidity, and 30-day
mortality
After adjustment for transfusion propensity, procedure group
and complexity, wound class and operative duration, and all
other important risk variables, transfusion significantly
(p<0.05) increased the risk of mortality (OR 1.32),
composite morbidity (OR 1.23), pneumonia (OR 1.24), and
sepsis/shock (OR 1.29), but not surgical-site infection
Their conclusions – Can’t argue with
these motherhood statements
35





RBC transfusions should be used very selectively during
surgical procedures.
Mild anemia should be tolerated.
Blood-conservation strategies and appropriate indicators
for transfusion should be used.
Additional studies must determine the mechanisms by which
transfusion of PRBCs and other blood components contribute
to poor patient outcomes.
Although deleterious effects are evident and some
mechanisms have been suggested, reversible causes and
effective treatments have yet to be definitively determined.
The flaw
36


Their statistical model was based on patients
requiring intraoperative transfusions and if a
patient was given >4 transfusion within 72 hours
post-operatively.
As such, a patient could have received up to 8
transfusions in the perioperative period and would
have been considered as not having a transfusion!
Large population screened (few enrolled)
Deceptive statistics, ‘underpowered’
37
Safety and efficacy of recombinant
activated factor VII. A randomized
placebo-controlled trial in the
setting of bleeding after cardiac
surgery. Gill R, Herbertson M,
Vuylsteke A, et al. Circulation 2009;
120: 21-27.
industry-funded
Design
38




Phase 2 dose-escalation study
Safety and possible benefits
30 sites in 13 countries
Aug 2004 - Nov 2007
Cohort 1 (n=70)
1:1 randomization
Cohort 2a (n=51)
1:2 randomization
placebo
Cohort 2b (n=51)
DSMB required repeat
due to safety concerns
placebo
placebo
80 ug/kg rVIIa
80 ug/kg rVIIa
40 ug/kg rVIIa
Cohort 3 (planned, but not done)
Steering/safety Comm advised against
Placebo vs. 160ug/kg
Primary endpoint
39


The primary endpoint was the incidence of critical
serious adverse events (death, myocardial
infarction, stroke, and venous thromboembolic
complications)
Note: Don’t get too excited – the study is stopped
early before we get the answer (maybe)
Patient population
40
Inclusion criteria:
 Pt admitted to CVICU  30 min (excluded patients bleeding intraop)
 Post-op bleeding into drains in the cardio-thoracic cavity:
200 ml/hr or 2ml/kg for 2 consecutive hours
 Urgent re-op not required ‘per investigator judgment’
Examples of exclusion criteria:
 History of CVA/DVT/PE
 Hereditary thrombophilia
 VAD, ECMO, aortic arch +/or descending thoracic aorta
 1st time CABG + none or only 1 antiplatelet medication within 5 days
 Unacceptable thrombotic risk ‘as per site investigator’
Patient outcome
Consented (n=2619)
6.8%
41
Randomized (n=179)
Randomized and not dosed (n=7)
Randomized and dosed (n=172)
Placebo (n=68)
Death (n=4, 6%)
40 ug/kg (n=35)
Death (n=4, 11%)
80 ug/kg (n=69)
Death (n=6, 9%)
Study terminated prior to introduction of cohort 3
“based on the data within the expanding cardiac literature in
which doses of rVIIa were in the range of 60 ug/kg” ???
The time to drug administration was 2.8 hrs arriving in ICU
Overview of adverse events
42
SAEs
Placebo (68) 40 ug/kg (35) 80 ug/kg (69)
Death
Cerebral
infarction
MI
PE
Other TEs
P
OR
4 (6)
0
4 (11)
2 (6)
6 (9)
2 (3)
1 (2)
0
0
3 (9)
0
1 (3)
0.25
2.16 (.6-8.1)
1 (1)
0
2 (3)
0.43
1.61 (.5-5.3)
Sample size calculation
Poorly written
Data on 1st 35 control
Not provided
43

Sample size was chosen to have <20% risk of seeing >14
(of 35) on active versus <7 (of 35) on placebo in cohort 1


What they saw: 5/68 (7.35%) placebo vs. 10/35 (28.57%)
treatment – 21.22% absolute risk increase
<16.7% risk of 13 (of 34) on active versus 2 (of 17) on
placebo or 7 on active versus 8 on placebo in cohorts 2a,
2b and 3, all assuming no differences and 21 events in
cohort 1 and 15 events in cohorts 2a, 2b, and 3.

What they saw: 5/68 (7.35%) placebo vs. 11/69 (15.94%)
treatment – 8.59% absolute risk increase
Sample size - continued
44

“Additionally, the sample size was chosen to give
adequate power to detect a 35% reduction in the need
for any allogeneic transfusions.”



The power for the efficacy evaluation is based on a comparison
of (all) placebo patients with the highest dose of rFVIIa (ie, cohort
3) – never done
11% (placebo) vs. 32% (40 ug/kg) vs. 27% (80 ug/kg) – 20%
absolute risk reduction
“This simple comparison between 2 groups (86 on
placebo versus 34 on rFVIIa) then has 80% power
assuming 80% transfusion rate on placebo and a 35%
reduction to 52%” (89% placebo, 68% on 40 ug/kg,
73% on 80 ug/kg)
r7a was associated with a statistically higher
frequency of Serious Adverse Events
45
Placebo
Drug
Any SAE*
5
21
No SAE
63
83
Total
68
104
p=0.0284, Fisher
* SAE = Death, Cerebral Infarction, MI, PE, or other TE
Efficacy
46
placebo 40 ug/kg 80 ug/kg
Re-op for bleeding (%)
25
14
P=0.21
12
P=0.04
Volume allogeneic blood (ml)
825
640
P=0.05
500
P=0.04
Drainage from intra-thoracic cavity
15mins-4hrs post dose (ml/hr)
51
35
P=0.76
24
P=0.018
Re-op rates
47
Avoiding RBCs after treatment
48
Major limitations
49




Statistically increased serious adverse events in rVIIa
groups, although they failed to do a simple Fisher
Exact test
Potential efficacy only seen at 80 ug/kg dose
Very select group of 6.8% of their population
Limitations:
 Small
sample size
 Industry sponsored trial
 Authors “compensated” by company
 Study never completed
50
Outcome before intervention
(a.k.a. Temporal ambiguity)
Transfusion and pneumonia in the
trauma intensive care unit: An
examination of the temporal
relationship. Vandromme MJ,
McGwin G, Marques MB, et al. J
Trauma 2009; 67: 97-101
Background
51




Retrospective studies have found an association between
transfusion in ICU patients and pneumonia
Associations between transfusion and ALI, MOF, tumor
recurrence and mortality
These reports have been used as evidence to support the
notion of ‘transfusion-related immunomodulation’ or TRIM
There are 2 limitations with these studies
1.
2.
Retrospective and therefore this association may not represent
true causation
The timing of transfusion is often unknown and therefore
potentially some of these patients developed pneumonia before
they even received the first unit of RBCs
Causation or Association?
52
Design
53





Patients admitted to the trauma ICU at the University of Alabama
University Hospital between 2004-07 who had an overall length of
stay of >4 days, and who had spent 1 or more days in the ICU
receiving mechanical ventilation
All transfusions to these patients were also categorized as ‘young’ (<14
days) or ‘old’ (>14 days) – as if something magical happens on day 14
All RBC units were prestorage leukoreduced
The incidence of pneumonia was the primary outcome of interest, and
was defined as positive culture >105 cfu/mL on bronchoalveolar
lavage
The analysis of the effect of the transfusion on pneumonia was
performed for all units transfused (like all the studies that came before),
and then only for units that were transfused before the first episode of
pneumonia
Patient population
54






1,615 patients met their study criteria
Of these patients, 73% were transfused at least 1 U of RBC
The mean number of units transfused per transfused patient
was 8.94 units
A total of 270 (16.7%) patients developed pneumonia
The population was as expected for a civilian trauma cohort
(age of 40 years, 74% male, 81% with a blunt injury)
The length of stay was 22 days, with a mean duration of
ventilation of 12 days
Pneumonia analysis
55


The overall adjusted relative risk for pneumonia for
all transfusions was 1.99 (95% CI 1.39-2.86)
When transfusions that occurred after the diagnosis
of pneumonia were excluded, the adjusted relative
risk for pneumonia was no longer statistically
significant (1.33, 95% CI 0.98-1.80)
Age of blood
56

They also analyzed the effect of different ages of blood on
the risk of pneumonia




This analysis was problematic because those who received
exclusively young blood were transfused only a mean of 4.24
units, compared to 4.74 units for those receiving old blood, and
12.07 units for those receiving a combination of old and young
units (different populations of patients)
The only statistically significant relative risk they could find was
that the receipt of exclusively old blood, compared to no blood,
increased the risk of pneumonia, albeit only slightly (RR 1.42,
95% CI 1.01-2.02)
There was no difference between patients who received
exclusively young, when compared to those who received
exclusively old blood
Statistically significant ‘shopping’ may be going on here
Author’s conclusions
57


The finding of an association between transfusion
and pneumonia may be an erroneous one, reflecting
transfusions received as a consequence of
developing pneumonia during a long stay in ICU
These authors methodologically improved on
previous TRIM reports by excluding transfusions that
occur after pneumonia onset, and not surprisingly
the effect is less prominent
Some tips when reading
58






Read from cover to cover – don’t be an abstract
reader
Think really hard about how they got their results if
the results are surprising or unexpected or too good
to be true
Look for statistically significant ‘shopping’
Do your own calculations
Check tables for errors
Read the stats section and look for missing info
Any questions?
59