Lost Opportunities for Design
Theory in Drug Development
Stephen Senn
(C)Stephen Senn
1
Basic Thesis
• Design theory has great potential in drug
development
• But this potential is unrealised
• Those working in so-called optimal design are
so ignorant of application realities that where
their influence is not zero it is harmful
• On the other hand the understanding of design
theory by biostatisticians is pitifully inadequate
• We must cooperate properly to cure this parlous
state of affairs
(C)Stephen Senn
2
Outline
• Quick tutorial on cross-over trials
• I shall then give two introductory examples
of nonsense
– By leading design theoreticians
– By leading biostatisticians
• I shall then consider ‘design nonsense’
further
• Some conclusions
• After lunch a case-study
(C)Stephen Senn
3
Warning
• I am a biostatistician
• We are used to thinking of data matrices
with rows as subjects and columns as
measurements
• That means that we write sequences for
designs with rows representing subjects
and columns representing periods
(C)Stephen Senn
4
Cross-over Trials
Definition: A cross-over
trial is one in which
subjects are given
sequences with the
object of studying
differences between
individual treatments.
(C)Stephen Senn
5
An Example of an AB/BA cross-over in asthma
Sequence
Period 1
for/sal
formoterol
salbutamol
sal/for
salbutamol
formoterol
(C)Stephen Senn
Wash-out
Period 2
Perio
6
An Example from Rheumatism:
2 doses of diclofenac and placebo
Period 1
D1
P
D2
D2
P
D1
(C)Stephen Senn
Period 2
D2
D1
P
D1
D2
P
Period 3
P
D2
D1
P
D1
D2
7
Carry-over
Definition: Carry-over is the persistence (whether physically or
in terms of effect) of a treatment applied in one period in
a subsequent period of treatment.
If carry-over applies in a cross-over trial we shall, at some
stage, observe the simultaneous effects of two or more
treatments on given patients.
We may, however, not be aware that this is what we are
observing and this ignorance may lead us to make errors in
interpretation.
(C)Stephen Senn
8
Simple Carry-over
• Carry-over lasts for exactly one period
• It depends only on the engendering
treatment and is unmodified by the
perturbed treatment
• There is a huge literature proposing
‘optimal’ designs for this model
• There is no empirical evidence that any of
this has been useful
(C)Stephen Senn
9
Three Period Bioequivalence
Designs
• Three formulation designs in six sequences
common.
• Subjects randomised in equal numbers to six
possible sequences.
For example, 18 subjects, three on each of the
sequences ABC, ACB, BAC, BCA, CAB, CBA.
– A = test formulation under fasting conditions,
– B = test formulation under fed conditions
– C = reference formulation under fed conditions.
–
(C)Stephen Senn
10
Weights for the Three Period Design:
not Adjusting for Carry-over
Period
Sequence
1
2
3
ABC
A
0
B
1/6
C
-1/6
ACB
A
0
C
-1/6
B
1/6
BAC
B
1/6
A
0
C
-1/6
BCA
B
1/6
C
-1/6
A
0
CBA
C
-1/6
A
0
B
1/6
CAB
C
-1/6
B
1/6
A
0
(C)Stephen Senn
11
Properties of these weights
• Sum 0 in any column,
– eliminates the period effect.
• Sum 0 in any row
– eliminates patient effect
• Sum 0 over cells labelled A
– A has no part in definition of contrast
• Sum to 1 over the cells labelled B and to -1
over the cells labelled C
– Estimate contrast B-C
(C)Stephen Senn
12
Weights for the Three Period Design:
Adjusting for Carry-over
Period
Sequence
1
2
3
ABC
A
-1/24
Ba
4/24
Cb
-3/24
ACB
A
1/24
Ca
-4/24
Bc
3/24
BAC
B
4/24
Ab
2/24
Ca
-6/24
BCA
B
5/24
Cb
-2/24
Ac
-3/24
CBA
C
-4/24
Ac
-2/24
Ba
6/24
CAB
C
-5/24
Bc
2/24
Ab
3/24
(C)Stephen Senn
13
Weights for the Three Period Design:
Adjusting for Carry-over
Period
Sequence
1
2
3
ABC
A
-1/24
Ba
4/24
Cb
-3/24
ACB
A
1/24
Ca
-4/24
Bc
3/24
BAC
B
4/24
Ab
2/24
Ca
-6/24
BCA
B
5/24
Cb
-2/24
Ac
-3/24
CBA
C
-4/24
Ac
-2/24
Ba
6/24
CAB
C
-5/24
Bc
2/24
Ab
3/24
(C)Stephen Senn
14
Properties of These Weights
• As before
– Estimates B-C contrast
– Eliminates, period and patient effect
– Eliminates A
• Sum to zero over cells labelled a,b, and c
– Eliminate simple carry-over
(C)Stephen Senn
15
Have We Got Something for
Nothing?
• Sum of squares weights of first scheme is
1/3 (or 4/12)
• Sum of squares of weights of second
scheme is 5/12
• Given independent homoscedastic withinpatient errors, there is thus a 25%
increase in variance
• Penalty for adjusting is loss of efficiency
(C)Stephen Senn
16
First Example
Some Design Theory Nonsense
John, J. A., Russell, K. G., and Whitaker, D. (2004), "Crossover: An Algorithm for
the Construction of Efficient Cross-over Designs," Statistics in Medicine, 23,
2645 - 2658.
A cross-over experiment involves the application of sequences of treatments to
several subjects over a number of time periods. It is thought that the observation
made on each subject at the end of a time period may depend on the direct
effect of the treatment applied in the current period, and the carry-over effects of
the treatments applied in one or more previous periods. Various models have
been proposed to explain the nature of the carry-over effects. An experimental
design that is optimal under one model may not be optimal if a different model is
the appropriate one. In this paper an algorithm is described to construct efficient
cross-over designs for a range of models that involve the direct effects of the
treatments and various functions of their carry-over effects. The effectiveness
and flexibility of the algorithm are demonstrated by assessing its performance
against numerous designs and models given in the literature.
(C)Stephen Senn
17
What’s wrong here?
“Sometimes in a clinical trial it may be necessary to modify or extend an ongoing trial.
For example, suppose that after a few periods have been completed one of the
treatments is dropped from the trial. It will then be necessary to re-allocate the other
treatments to the remaining periods of the trial.”
(John et al, 2004 p. 2653)
“Jones and Donev [17] also consider augmenting a design to account for the
removal of a treatment. The initial trial to compare four treatments A, B, C, and D
in five periods using four groups of subjects used a Williams square [3] with the fourth
period repeated. After the first two periods had been completed it was decided to
drop treatment D from the remainder
of the trial.”
(C)Stephen Senn
18
(C)Stephen Senn
19
The reality
..in a single-dose cross-over trial in asthma in 12 patients reported by Palmqvist
et al. [5] patients were treated in the first period of a cross-over on dates
ranging from 5 May to 12 November 1987 [4]. They were treated in a second
period on dates ranging from 18 May to 26 November 1987. Eleven of the
patients had completed period two of treatment before the 12th patient was
recruited. (Senn, 2005, p3675.)
A basic fact of clinical trials
You treat patients when they fall ill
(C)Stephen Senn
20
Multi-Story
Aspect
Single-Dose
Multi-Dose
When?
Phase I/II
Phase II/III
Why?
PD, Dose
Therapeutic
Primitive
Constraints
Carry-over
Number of
Length of
periods
treatment
Not a problem Potential
Problem
(C)Stephen Senn
21
Conclusion
• Multi-dose trials real scope for design
theory.
• These will employ active wash-out
• Design problem is trade-off between
exploiting correlation and eliminating
carry-over.
• Short vs long active wash-out periods
(C)Stephen Senn
22
Second Example
Some Biostatistics Nonsense
Chow, S. C., and Liu, J. P. (2000), Design and Analysis of Bioavailability and
Bioequivalence Studies (2nd ed.), New York: Marcel Dekker.
Have several discussions of efficiency of designs in their book which are completely
beside the point. They compare designs in terms of residual degrees of freedom!
They conclude that Balaam’s design, which uses sequences
TR/RT/TT/RR
Is similar in efficiency to the more conventional
TR/RT design.
They write “The degrees of freedom for the intrasubject residuals for the 2 × 2 and 4
× 2 design are 22 and 21 respectively. Therefore there is little difference in testing
power.”
This is nonsense
(C)Stephen Senn
23
Second Example
Some Biostatistics Nonsense
Design
.
Source
Between
2×2
4×2
2×3
2×4
23
23
23
23
Seq
1
3
1
1
Res
22
20
22
22
Within
24
24
48
72
Period
1
1
2
3
Form
1
1
1
1
1
1
1
Carry
Res
22
21
44
67
Total
47
47
71
95
(C)Stephen Senn
4×4
24
What is wrong 1.
It’s not correct design theory
• As any design expert knows residual
degrees of freedom are (nearly) irrelevant
to efficiency
• It is the impact of adjustment on the
degree of orthogonality of the design
matrix that is important
(C)Stephen Senn
25
What is wrong 2.
It’s not realistic biostatistics
• In fact as any biostatistician who has had to
think about it will know from a practical point of
view far from being optimal Balaam’s design is
simply inadmissible
• The reasons is that only half of the resources
are devoted to actually measuring the treatment
• The rest are devoted to providing an adjustment
for a form of carry-over that is itself implausible
(C)Stephen Senn
26
Allocation of patients for two
designs
Sequence
AB/BA
Balaam
AB
n/2
n/4
BA
n/2
n/4
AA
0
n/4
BB
0
n/4
(C)Stephen Senn
27
Investigation of the real efficiency
of Balaam’s design
Mathcad 2001 Program to compare
design correcting for simple carry
correcting for carry-over.
Take simplest comparable case of four patients: on
Balaam's design or two to each of the sequences o
Set up elements of design matrix
Patient and period dummies common to both
Designs: AB/AB/BA/BA (standard design) or AB/BA
Design matrix rows in order patient 1 period 1, patie
(C)Stephen Senn
28
Five Reasons why the Simple Carry-over
Model is not Useful
• If it applies then the investigator can design a trial
which eliminates it . (double the periods)
• Implausible given pk/pd theory. (obvious)
• Leads to inefficient estimators. (see investigation
to follow)
• Can lead to poor designs. (ditto)
• The models which incorporate it are selfcontradicting. (example: factorial X-overs)
(C)Stephen Senn
29
100
Dose response: the
pharmacokineticist’s
version
50
Response
0
0
2
Concentration/EC50
4
0.5
1
2
5
(C)Stephen Senn
30
Dose Response:
The Statistician’s Version
Response
This is what the
simple carryover model
implies
(C)Stephen Senn
Dose
31
The Models Which use Simple Carry-over
are Inconsistent
Consider a factorial cross-over in four periods comparing A, B
and the combination of A and B to placebo. We can represent the
four treatments by: **, A*, *B and AB.
Suppose we consider a patient who has received the sequence
AB ** A* *B. A standard parameterisation for treatment and
carry-over would be as in the following table.
(C)Stephen Senn
32
Paramaterisation of a factorial cross-over
Period
I
II
III
IV
Treatment
Combination
AB
**
A*
*B
Treatment
Parameters
A, B,
AB
A
B
Carry-over
Parameters
(C)Stephen Senn
A, B,
AB
A
33
The Rhinoceros
The rhinoceros has a kind heart, if you doubt it
here’s the proof
That thing on his nose is for taking stones out of a
horse’s hoof
He seldom ever meets a horse, it is this that makes
him sad
When he does, then it hasn’t a stone in its hoof
But he would, if he did and it had
Flanders and Swann
(C)Stephen Senn
34
The Phoenix Bioequivalence
Trials
•
•
•
•
•
•
Analysed by D’Angelo & Potvin
20 drug classes
1989-1999
12 or more subjects
96 three period designs
324 two period designs
(C)Stephen Senn
35
Three Treatment Designs
P-Values for Carry-Over
AUC
0 : 115567899
1 : 01458999
2 : 01225568999
3 : 011335577
4 : 24688
5 : 35667788
6 : 00336667888
7 : 14444566999
8 : 011233468888
9 : 13335667899
(C)Stephen Senn
Cmax
0 : 223557888
1 : 4677799
2 : 000124566899
3 : 011124689
4 : 01223455799
5 : 00045599
6 : 000166667778
7 : 0345566779
8 : 2345779
9 : 13444556889
36
Two Treatment Designs
AUC
0 : 00111111222222234444
0 : 5666777777789999
1 : 00000112222223333
1 : 5556667777899999
2 : 0011112223344444
2 : 555666788899999
3 : 00001112233344
3 : 5556666666777778888899999
4 : 001111112222223334
4 : 55666666777777788999
5 : 00000111222333344444
5 : 566677888899
6 : 000001134
6 : 55666667777888889999
7 : 111233333344
7 : 555556777888899
8 : 0000112234444
8 : 55666778888999
9 : 00011112233334444
9 : 555567777788999
(C)Stephen Senn
Cmax
0 : 00122222344
0 : 55555556666677999999
1 : 0001122233333344444444
1 : 55566667778888899
2 : 00011111122344
2 : 566667788889999
3 : 111112222233444444
3 : 555566666777778888999
4 : 000001112222333334444
4 : 5557888889999
5 : 00001122233
5 : 5555666678999
6 : 0000111222233334
6 : 55555566677788889999
7 : 000000112223344
7 : 6666777777889
8 : 0122233444
8 : 55666677888899
9 : 1111111222333444
9 : 555555556666677778889999
37
Test of Uniformity of P-Values
Study
Design
Variable
Total
number
of studies
KS
statistic
pvalue*
2-way
AUC0-t
Cmax
324
324
0.0645
0.0496
0.1354
0.4040
3-way
AUC0-t
Cmax
96
96
0.1048
0.0542
0.2424
0.9407
*
H0: true cdf U[0,1] vs. H1: true cdf NOT U[0,1]
(C)Stephen Senn
38
Conclusions
• Distribution of P-values uniform
– no evidence of carry-over
• Carry-over a priori implausible
– presence testable by assay
• No point is testing for it
– leads to bias
• Or adjusting for it
– increased variance
(C)Stephen Senn
39
Do Bayesians do Better?
• In principle the Bayesian approach ought
to allow us to be more flexible about
nuisance parameters such as carry-over
• However, the Bayesian track record is not
impressive here
• Realistic models have not been employed
(C)Stephen Senn
40
Hills and Armitage Eneuresis Data
Cross-over trial in
Eneuresis
14
12
Two treatment periods of
14 days each
10
Treatment effect
significant if carry-over
not fitted
8
6
2.037 ( 0.768, 3.306)
4
Treatment effect not
significant if carry-over
fitted
2
0
0.451 (-2.272, 3.174)
0
2
4
6
8
10
12
Dry nights placebo
Sequence Drug Placebo
Sequence placebo drug
Line of equality
(C)Stephen Senn
1.
Hills, M, Armitage, P. The two-period
cross-over clinical trial, British Journal of Clinical
Pharmacology 1979; 8: 7-20.
41
Identical ‘uninformative’
prior placed on carry-over
as for treatment
NB Parameterisation here
means that values of 
need to be doubled to
compare to conventional
contrasts
(C)Stephen Senn
42
Identical Priors for Treatment and Carryover?
•
•
•
•
•
Patients treated repeatedly during trial
Fourteen day treatment period
Average time to last treatment plausibly 4 hours
Average time to previous treatment seven days
Saying that it is just as likely that carry-over
could be greater than treatment is not coherent
• In any case the two cannot be independent
• Is negative carry-over as likely as positive carryover?
(C)Stephen Senn
43
So What are Acceptable Models for
Carry-over?
• Ignoring carry-over altogether (not allowing for it
because one believes one has taken adequate
steps to eliminate it)
– This is always a reasonable strategy
• Using an integrated pharmacokinetic
pharmacodynamic model (Sheiner et al, 1991)
– This may work for dose-finding trials
– Very difficult to implement where more than one
molecule is involved
(C)Stephen Senn
44
The Sheiner model
DEij 
Emax i d ij
PD dose response
D50i  d ij
j

d ij   Dil 1  e
l 1
 tki

e

 ki T j Tl

PK model for doseconcentration as a
consequence of previous
dosing history
Steady state concentration for
patient i in period l
(C)Stephen Senn
45
Pharmacodynamic model
1
1
Response
0.909
3
DE( d  1)
DE( d  2)
0.5
0.5
DE( d  3)
0
0
0
5
0
d
10
10
Dose
(C)Stephen Senn
46
Dosing levels achieved
5
4
Dose
3
2
1
0
1
2
3
4
Period
Seq 1 Actual
Seq 1 Theory
Seq 2 Actual
Seq 2 Theory
Seq 3 Actual
Seq 3 Theory
Seq 4 Actual
Seq 4 Theory
(C)Stephen Senn
1

2
D  
3
4

2 3 4

4 1 3
1 4 2
Possible set of sequences for a design. These follow a
Williams square. NB This is probably not a good idea

3 2 1
47
Carry-over by sequence
Carry-over
0.1
0
0
0.1
0.2
1
2
3
4
Period
Seq 1
Seq 2
Seq 3
Seq 4
Carry from 4
(C)Stephen Senn
48
The difference between
mathematical and applied
statistics is that the former is full
of lemmas whereas the latter is
full of dilemmas
(C)Stephen Senn
49
Advice for Design-Theoreticians
• Resist the temptation to give advice if you
are unfamiliar with the application area
• Seek collaborators
• Ground your models in pharmacology
• Remember that the goal is good medicine
not elegant mathematics
• Don’t defend the indefensible
(C)Stephen Senn
50
Advice for biostatisticians
• Remember that design theoreticians have
many powerful results
• It’s just conceivable that some of them
may even be useful
(C)Stephen Senn
51
(C)Stephen Senn
52
References
1. Senn, S.J., Is the 'simple carry-over' model useful?
[published erratum appears in Statistics in Medicine 1992 Sep 15;11(12):1619].
Statistics in Medicine, 1992. 11(6): p. 715-26.
2. Senn, S.J., The AB/BA cross-over: how to perform the two-stage analysis
if you can't be persuaded that you shouldn't., in Liber Amicorum Roel van Strik,
B. Hansen and M. de Ridder, Editors. 1996, Erasmus University:
Rotterdam. p. 93-100.
3. Senn, S.J., Cross-over Trials in Clinical Research. Second ed. 2002, Chichester:
Wiley.
4. Senn, S.J., Statistical Issues in Drug Development. Statistics in Practice, ed. V.
Barnett. 2007, Chichester: John Wiley.
(C)Stephen Senn
53