Research Methods
Clinical Trial Design and Analysis
Jay Horrow, MD, MS
Professor, Anesthesiology & Perioperative Medicine, College of Medicine
Professor, Epidemiology & Biostatistics, School of Public Health
Drexel University, Philadelphia, PA
Executive Director, Global Clinical Development
Merck, North Wales, PA
Goals
• Principles of Experimental Design
• Appreciate the value of a statistician
• Awareness of challenges in research
Why learn about clinical trials?
How to stay current?
• Vigilance
• Discipline
• Skill: weeding out published material that is
• Not applicable
• Biased
• Flawed in design, methods, or analysis
Who do you believe?
To add MEANINGFULLY to the vast literature
http://www.nytimes.com/2008/02/16/business/16drug.html?_r=1&ref=todayspaper&oref=slogin#
Types of Data
• Observational
• Experimental
•Retrospective
•Survey data
•Lack intervention
•Lack active randomization
•Prospective
•Active randomization
•Intervention
•Controls / blinding
Types of Data: Example
• Observational
ASSOCIATION
200 patients received chemotherapy drug A or drug B.
Tumor regression for each is recorded.
• Experimental
CAUSATION
200 patients randomly receive either drug A or B. Tumor
regression for each is recorded.
Types of Data: Implication
• Observational
ASSOCIATION
Many conscious or unconscious factors, known or unknown,
could systematically affect results in one group v. the other.
• Experimental
CAUSATION
All factors, conscious or unconscious, known or unknown,
should affect results in both groups equally.
How to Insult Someone’s Data
“Oh, those data are observational.”
Statistical inference is very limited.
Lack randomization.
Probably substantial bias.
May lack independence of observations.
What kind of study will you do?
•
•
•
•
•
Survey or chart review?
Historical control?
Case control?
Incidence / prevalence measures (RWE)?
Sensitivity / specificity / predictive value?
• Answer a hypothesis?
• Compare ways to detect disease
• Compare ways to treat disease
Basic Designs
• Between-patient comparison: Parallel Group
• Within-patient comparison
• Before and after
• Simultaneous comparison
• Cross-over trials
• Factorial design
• Latin square designs
Parallel Group Design
Ximelagatran 36 mg bid
n = 1960
Baseline
R
Warfarin od titrated to INR
n = 1962
Albers G et al.: Stroke prevention with the oral direct thrombin inhibitor ximelagatran
compared with warfarin in patients with non-valvular atrial fibrillation (SPORTIF V). JAMA
2005; 293: 690-8
Within-patient: Before and After
also known as “paired observations”
t
Enroll
Before
Treat
After
no randomization; no blinding; no control for “time”
Sources of a difference:
• Treatment
• Time
• Trt x time interaction
Within-patient: Before and After
Better design?
PLACEBO
Effect
of time
ACTIVE
Effects
of time
& trt
R
Enroll
Before
Treat
After
Within-patient: cross-over trial
Placebo
before
after
<washout>
Treatment
before
no randomization; no blinding; controlled
after
Sources of a difference:
• Treatment
• Time: disease stable?
• Trt x time interaction
Within-patient: cross-over trial
Better design?
Oxprenolol
Baseline
R
Washout
1st day
Placebo
assess
Placebo
2nd day
Washout
Oxprenolol
assess
randomized; blinded; controlled
James IM et al. Effect of oxprenolol on stage-fright in musicians. Lancet 1977; ii: 1317
Parallel v. Cross-over trial
Oxprenolol
Baseline
R
Washout
Placebo
1st day
Placebo
2nd day
Washout
Oxprenolol
assess
Advantage:
Disadvantage:
inter-patient variability minimized
possible period effect, carry-over effect
assess
Simultaneous comparison
•
•
•
•
Study something we have 2 of: eyes, legs, etc.
Initial status identical to both disease, outcome
Apply treatment to one only
Follow both
Must assure identical status maintained
Blocking Factor: Examples
Lo-Carb Diet Regular Diet
Philadelphia
Group 1
Group 2
Boston
Group 3
Group 4
Analysis provides effects of
diet after subtracting effect
of city.
N.B. Any “differences” by city not interesting or valuable to learn.
Clopidogrel
Prasugrel
Ticagrelor
NSTEMI
Group 1
Group 2
Group 3
STEMI
Group 4
Group 5
Group 6
Analysis provides effects of antiplatelet choice after accounting for
effect of ACS type
N.B. You already know affect of ACS type– it’s a “nuisance.”
Factorial Designs
Lo-Carb Diet Regular Diet
Analysis provides effects of:
• Diet
• Medication
• Diet x Medication
Appetite suppressor
Group 1
Group 2
Placebo
Group 3
Group 4
Clopidogrel
Prasugrel
Ticagrelor
Rivaroxaban
Group 1
Group 2
Group 3
Placebo
Group 4
Group 5
Group 6
Analysis provides effects of:
• Antiplatelet
• Anticoagulant
• Interaction
Advanced Stuff: Latin Squares
Study 3 factors in 2-dimensional design
Diet 1
Diet 2
Diet 3
Diet 4
Group Rx
Trt A: G1
Trt B: G2
Trt C: G3
Trt D: G4
Hypnosis
Trt B: G5
Trt C: G6
Trt D: G7
Trt A: G8
Behavioral Trt C: G9
Sham Rx
Trt D: G10 Trt A: G11 Trt B: G12
Trt D: G13 Trt A: G14 Trt B: G15 Trt C: G16
Analysis provides effects of:
• Diet
• Psychotherapy
• Treatment (A  D)
• Interactions (if replicates in each cell)
Trial Design  Analysis
• Multitude of possible designs
• Randomization, Blinding, Control
• Does design facilitate data analysis?
• Often, many possible analytic techniques
• Specify analysis BEFORE unblinding
• CONSULT A STATISTICIAN:
• Analytic wizardry cannot substitute for good design
• Generally impartial and unbiased
• Mathematical training  less “wishful thinking”
Learning by Example: Design a Study
You wish to determine whether or not providing a decision support tool
(DST) affects adherence to Practice Guidelines, via a written, multiplechoice examination to residents. You hope using the DST improves
performance on the exam. Guidelines training occurs 2 weeks before the
testing period.
1.
2.
3.
4.
Can every resident take the same exam twice?
Can the DST be used always on the 2nd exam?
Can every resident take only 1 exam? Is this an “efficient” design?
Would 2 separate, similar exams help?
• Might exam “order” affect performance?
• How can you account for an affect due to “exam”?
5. What if all residents cannot come from a single institution? Is this good?
bad? Can your analysis account for this?
Hand WR et al. Anesthesiology 2014; 120: 1339
Summary of Design Elements
• 2 exams (similar, well-edited, critiqued)
• Cross-over: each resident takes each exam
• Time between exams to reduce “learning” effect
• Half take exam “A” first, half take exam “A” second
• Half use DST first, half use DST second
• Thus, 2x2 FACTORIAL component: “exam” x “DST”
Hand WR et al. Anesthesiology 2014; 120: 1339
A cross-over factorial design
DST 1st
No DST 1st
Exam A first
Group 1
Group 2
Exam B first
Group 3
Group 4
Analysis provides effects of:
• DST
• Exam
• Order
WHAT ABOUT USING 4 DIFFERENT SITES?
Is “site” a variable of interest?
Any more than Exam? or Order?
What are the “blocking” factors in this design?
How are residents randomized to Groups if there are 4 sites?
Summary of Analysis Effects
• DST (the primary outcome variable)
• Exam (A or B) – a blocking factor
• Site (1, 2, 3, 4) – a blocking factor
• Order (1st ; 2nd) – a blocking factor
• Could also look at interaction terms, e.g.
• DST x order: might be less useful if used 2nd?
Hand WR et al. Anesthesiology 2014; 120: 1339
LEARNING BY Q&A: Study “B”
You wish to determine which of 3 methods of post-operative pain management is
most successful following shoulder arthroscopy: patient-controlled analgesia (PCA),
constant bupivacaine infusion via brachial plexus catheter, or intra-articular morphine
during surgery. No current data suggest one treatment is best. You plan a
randomized trial with trained personnel, blinded to study treatment, evaluating pain
scores by telephone call on post-operative day 3.
Although the evaluators are blinded to study treatment, the patients cannot
be blinded. This study is:
a)
b)
c)
d)
e)
Not blinded
Single-blinded
Double-blinded
Mixed-blinded
Randomized, and therefore blinding is not relevant
LEARNING BY Q&A: Study “B”
You wish to determine which of 3 methods of post-operative pain management is most
successful following shoulder arthroscopy: patient-controlled analgesia (PCA), constant
bupivacaine infusion via brachial plexus catheter, or intra-articular morphine during surgery. No
current data suggest one treatment is best. You plan a randomized trial with trained personnel,
blinded to study treatment, evaluating pain scores by telephone call on post-operative day 3.
A colleague suggests that patients with neurologic symptoms should not receive
plexus catheters for medico-legal reasons, but may randomly be assigned to either of
the other two groups. Which is MOST correct?
a) If other patients are randomly assigned to all 3 groups, then the randomization
scheme maximally reduces bias.
b) Since those evaluating post-operative pain are blinded to group assignment, the
proposed modification does not impact study bias.
c) The Institutional Review Board (IRB) has jurisdiction over medico-legal issues, so
their directive should be sought and followed.
d) The design confounds patient’s symptoms with method of pain management, thus
preventing conclusions regarding the best method of pain management.
e) Conclusions based on data from the entire cohort regarding the best method are
valid for all patients EXCEPT those with neurologic symptoms.
LEARNING BY Q&A: Study “B”
You wish to determine which of 3 methods of post-operative pain management is most
successful following shoulder arthroscopy: patient-controlled analgesia (PCA), constant
bupivacaine infusion via brachial plexus catheter, or intra-articular morphine during surgery. No
current data suggest one treatment is best. You plan a randomized trial with trained personnel,
blinded to study treatment, evaluating pain scores by telephone call on post-operative day 3.
Assume patients with neurologic symptoms are excluded. Citing safety concerns, the IRB
approves the study only if patients are randomized in a 2:1:1 ratio (twice as many in the PCA
group as in the other groups). Which one of the following is MOST correct?
a) The study is no longer blinded because observers recording pain scores know that the
chance of having PCA is twice that of the other two treatments.
b) Since those evaluating post-operative pain are blinded to group assignment, the proposed
modification does not impact study bias.
c) The IRB has jurisdiction over medico-legal issues, not safety issues, so their directive may
be ignored.
d) Investigators should select every other (half) of the PCA patients to participate in the final
analysis.
e) This imbalanced randomization scheme compromises study equipoise (ethical balance).
LEARNING BY Q&A: Study “B”
You wish to determine which of 3 methods of post-operative pain management is
most successful following shoulder arthroscopy: patient-controlled analgesia (PCA),
constant bupivacaine infusion via brachial plexus catheter, or intra-articular morphine
during surgery. No current data suggest one treatment is best. You plan a
randomized trial with trained personnel, blinded to study treatment, evaluating pain
scores by telephone call on post-operative day 3.
Midway through this study, intended to enroll 300 patients total, you note 5 instances
of pneumothorax, an unusually high incidence, and prolonging hospitalization. You
are blinded to study treatments. The best course of action is:
a) Complete the study, then analyze the data to determine whether or not the
pneumothoraces occurred more in one group compared to the others.
b) Call the FDA to report adverse drug reactions for morphine and bupivacaine.
c) Stop and unblind all the data collected to date to see which method was best.
d) Stop enrollment and have a “fire-wall”-ed safety monitoring group view un-blinded
data to recommend a course of action.
e) Ask the IRB to review collected data and recommend whether or not to halt
enrollment.
Success in Clinical Research
•
•
•
•
[Observe  hypothesize] = CREATIVITY
Trust your judgment: challenge status quo
Find collaborators: a team sport
Consult an “un”-enthusiast = statistician
General References
Pocock SJ. Clinical Trials. A Practical Approach. Chichester: Wiley. 1983
Montgomery DC. Design and Analysis of Experiments, 5th ed., New York:
Wiley. 2001
Taubes G: Unhealthy Science. NY Times Magazine, 16-Sep-2007
Mlodinow L. The Drunkard’s Walk. How Randomness Rules Our Lives.
New York: Random House. 2008
Silver N. The Signal and the Noise. New York: Penguin. 2012
Kahneman D: Thinking, fast and slow. New York: Farrar, Straus and
Giroux. 2011