Making Randomized Clinical Trials
Seem Less
Random
Andrew P.J. Olson, MD
Assistant Professor
Departments of Medicine and Pediatrics
University of Minnesota Medical School
olso5714@umn.edu
Disclosures
• I have no financial interests to disclose.
• I will not discuss off label or investigational
product use.
Learning Objectives
• Identify a framework for analyzing RCT’s in the
learning environment
• Discuss major sources of bias in RCT’s
• Define validity and generalizability and be able
to begin to assess these in real-world article.
• Use these skills in a small group environment
$
Agenda
•
•
•
•
•
Overview of RCT’s
Randomization
Blinding
Outcome measurement
Analysis – Intention to treat?
Let’s start with a roadmap.
RCT Roadmap
Randomization
Outcome
Treatment
Follow-up
Population
People at risk for
heart attacks
1000
Statins
25 heart
attacks
1000
Placebo
50 heart
attacks
Sample
Randomization is Key
• By randomizing subjects to different groups,
both known (measured) and unknown
(unmeasured) variables should be randomly
distributed.
• This controls for known and unknown
confounding variables
What is Confounding?
• A confounding variable is associated with the
receipt of treatment and the outcome.
• Statin trial:
– Smoking, exercise, hypertension medications
Validity and Generalizability
• Validity: In the studied population, was the
study performed in a way that the results are
valid?
• Generalizability: Are these results applicable
to my patients?
Elements of a Randomized Controlled Trial
Element
Best Case Scenario
(Described in paper)
Validity or
Generalizability
Subject Selection
Recruitment Procedures
and Entrance Criteria
specified
Generalizability
Randomization
Random Sequence
Allocation Concealment
Validity
Treatment
Feasible, safe, delineated
Generalizability
Element
Best Case Scenario
(Described in paper)
Validity or Generalizability
Follow up
Complete (all accounted
for) and similar between
groups
Validity
Co-intervention
Same between groups and
relevant co-interventions
described
Validity
Blinding
Subjects, Providers, and
Outcome assessors
Validity
Element
Outcomes
Analysis and Power
Best Case Scenario
(Described in paper)
Validity or Generalizability
Measurable?
Validity
Meaningful?
Generalizability
Intention to treat?
Validity
Adequately powered?
Validity
Statistical Methods
described and
appropriate?
Validity
Randomized Controlled Trials
• Overview of RCT’s
• Randomization
• Blinding
• Outcome measurement
• Analysis – Intention to treat?
Randomization
• Is the randomization of a subject to a group really
random?
– If allocation is truly random, it cannot be predicted
– Random number table or generator
– Examples of non-random allocation:
•
•
•
•
Even or odd MRN
Days of the week
Morning or afternoon patients
First patient the day
Randomization
Allocation Concealment
• The sequence of allocation to different groups
cannot be seen by subjects or providers
• Examples:
– Sealed, opaque envelopes
– Central voice-response system
– Online systems
Randomized Controlled Trials
• Overview of RCT’s
• Randomization
• Blinding
• Outcome measurement
• Analysis – Intention to treat?
Blinding
• Ideally, the only difference between groups is the
treatment (which no one knows about!)
• Triple Blinding is Ideal
– No one knows the treatment group allocation
• Provider
• Subject
• Outcomes assessor
• Blinding protects against bias from:
– Different receipt of co-interventions between groups
– Differential outcome ascertainment
Co-interventions
60% take aspirin
Population
People at risk for
heart attacks
1000
Statins
25 heart
attacks
1000
Placebo
50 heart
attacks
Sample
30% take aspirin
Co-interventions
• By not knowing which group a subject is
assigned to, subjects in different groups
should be treated the same
• Neither those giving or receiving treatment
know the assignment
Blinding in Treatment Studies
Blinding
Randomization
Outcome
Treatment
Follow-up
Population
People at risk for
heart attacks
1000
Statins
25 heart
attacks
1000
Placebo
50 heart
attacks
Sample
In treatment studies, it is usually necessary to have a placebo or
sham procedure
Blinding
• If subjects and providers are unaware of which
group the patient is allocated to, cointerventions should be the same on average
• Differences in co-interventions, if there is
proper randomization and blinding, will be
due to chance.
Don’t forget about the third blind team
member!
• Subjects and Providers can be difficult to
blind, especially with certain treatments
• However, those who are analyzing the
outcomes can almost always be blinded
– Analysis of medical records
– If they know the group assignment, their view of
an outcome can be biased
Randomization and Blinding
Randomization
Outcome
Treatment
Follow-up
Population
Treatment
A
# Events
Treatment
B
# Events
Sample
Similar at baseline?
Similar during followup?
Randomized Controlled Trials
•
•
•
•
•
Overview of RCT’s
Randomization
Blinding
Outcome measurement
Analysis – Intention to treat?
Outcomes
• Outcomes are prespecified
– Measurable - Validity
– Meaningful – Generalizability
• Easily measurable:
– Mortality, MI, cancer recurrence, blood pressure, lipids
• Less easily measured:
– Quality of life, pain, disability
– Validated tool?
• Meaningful:
– Do they matter to a patient?
Surrogate Outcomes
• Sometimes a meaningful outcome is difficult
to measure:
– Time for followup, hard to quantify
• So a surrogate outcome is used:
– FDA Definition:
• A laboratory measurement or physical sign that is used
as a substitute for a clinically meaningful outcome
because it is expected to predict the effect of therapy
on a clinically meaningful outcome.
Surrogate Outcome
Population
Treatment
A
Surrogate
# Events
Treatment
B
Surrogate
# Events
Sample
Surrogate Outcome
↑Harm
Treatment
↑Surrogate
Treatment
↓Surrogate
No Change
Events
An Example of Surrogate Outcomes
• There is significant mortality from arrhythmias
after myocardial infarctions
• PVC’s can be a marker of arrhythmias
• Antiarrhythmic medications decrease PVC’s
• Thus, it makes sense that using antiarrhythmic
medications after myocardial infarctions might
decrease mortality
A Classic Example of
Surrogate Outcomes
CAST Trial
Cardiac Mortality
All Cause Mortality
Randomized Controlled Trials
•
•
•
•
•
Overview of RCT’s
Randomization
Blinding
Outcome measurement
Analysis – Intention to treat?
Intention to Treat Analysis
5000 Patients Screened
1000 Randomized
500 Metoprolol
23 withdraw
consent
14 lost to
followup
22 stop taking
medicine
500 Placebo
?
Outcome
Outcome
10withdraw
consent
4 lost to followup
13 stop taking
medicine
Intention to Treat
• All randomized subjects are included in the
analysis, regardless of actual receipt of
treatment
• This means some subjects who didn’t get the
intervention are still included in the analysis
• Preserves the randomization
Intention to Treat
• All subjects should be able to be accounted for while
you read the paper
– High rate of participation
– Few are “lost to followup”
• If a subject is lost to followup:
– Search for vital statistics
– Perform advanced analyses to determine what probably
happened to these subjects
• Most importantly, patients must NOT be removed from
the study in a non-random way!
Small Group Activity
Small Group Activity
• Was the assignment of patients to treatments randomized?
• Were the groups similar at the start of the trial?
• Except for the allocated treatment, were the groups treated
equally?
• Were all patients who entered the trial accounted for and were
they analyzed in the groups to which they were randomized?
• Were the measures objective?
• Were the patients and clinicians kept blind to which treatment was
being received?
• How large was the treatment effect?
• How precise was the estimate of the treatment effect?
• Will the results help me in caring for my patients?