Pharmacogenetics:
New dawn or false dawn
Stephen Senn
(c) Stephen Senn 2011
1
Genes, Means and Screens
It will soon be possible for patients in clinical trials to undergo genetic tests
to identify those individuals who will respond favourably to the drug
candidate, based on their genotype…. This will translate into smaller, more
effective clinical trials with corresponding cost savings and ultimately better
treatment in general practice. … individual patients will be targeted with
specific treatment and personalised dosing regimens to maximise efficacy
and minimise pharmacokinetic problems and other side-effects.
Sir Richard Sykes, FRS, 1997
(c) Stephen Senn 2011
2
Soon?
(c) Stephen Senn 2011
3
Articles on pharmacogenetics by publication year
1750
1500
1250
1000
Articles
750
500
Source: Web of Science 9 June 2011
250
0
1960
1970
1980
1990
2000
2010
Year
(c) Stephen Senn 2011
4
Articles on pharmacogenetics by publication year
17500
15000
12500
10000
7500
5000
Source: Web of Science 9 June 2011
Cumulative Numbers of Article
2500
0
1960
1970
1980
1990
2000
2010
Year
(c) Stephen Senn 2011
5
The Pharmacogenomic Revolution?
• Clinical trials
– Cleaner signal
– Non-responders eliminated
• Treatment strategies
– “Theranostics”
• Markets
– Lower volume
– Higher price per patient day
(c) Stephen Senn 2011
6
Implicit Assumptions
• Most variability seen in clinical trials is genetic
– Furthermore it is not revealed in obvious phenotypes
• Example: height and forced expiratory volume (FEV 1) in one second
• Height predicts FEV1 and height is partly genetically determined but you
don’t need pharmacogenetics to measure height
• We are going to be able to find it
– Small number of genes responsible
– Low (or no) interactive effects (genes act singly)
– We will know where to look
• In fact we simply don’t know if most variation in clinical trials
is due to individual response let alone genetic variability
(c) Stephen Senn 2011
7
My Opinion
• Most of the hype is due to a failure to
understand response
• Responder analysis is to blame
• And related to this is an obsession with
Numbers Needed to Treat
– Which is increasing the pressure to use
dichotomies
(c) Stephen Senn 2011
8
A Thought Experiment
• Imagine a cross-over trial in hypertension
• Patients randomised to receive ACE II inhibitor
or placebo in random order
• Then we do it again
• Each patient does the cross-over twice
• We can compare each patient’s response
under ACE II to placebo twice
(c) Stephen Senn 2011
9
Difference to placebo in DBP mmHg
Second cross-over
10
10
5
5
0
0
-5
-5
-10
-10
-15
-15
-20
-20
-25
-25
-30
-30
-30
-25
-20
-15
-10
-5
0
5
...
..
..
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
....
...
...
...
...
...
..
....
..
..
..
...
.
..
..
...
...
....
..
...
.....
...
..
....
.
10
First cross-over
...............................................
...........
..............
........
..............
.
.
.
.
.
.
.
.
.
................
.............
...............
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.....................
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
..............................................................
......................................................
-30
-25
-20
-15
-10
-5
0
5
(c) Stephen Senn 2011
10
10
795
= 0.95
832
46
= 0.27
168
(c) Stephen Senn 2011
NB These are conditional
probabilities of response on the
second occasion. They are not
conditional probabilities of
being a ‘true’ responder.
11
Difference to placebo in DBP mmHg
Second cross-over
10
10
5
5
0
0
-5
-5
-10
-10
-15
-15
-20
-20
-25
-25
-30
-30
-30
-25
-20
-15
-10
-5
0
5
..
..
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
.
..
...
.
....
..
..
..
...
.
..
..
...
.
..
..
...
..
...
....
...
...
....
.
.....
...
..
...
.
10
First cross-over
......................................................
..............
.......
.........
..........
.
.
.
.
.
.
.
..................
............
.
..................
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
..
..........................................
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.................................
...........................................
-30
-25
-20
-15
-10
-5
0
(c) Stephen Senn 2011
5
10
12
678
= 0.82
826
140
= 0.81
174
(c) Stephen Senn 2011
13
?
........................................
................
.
.......
.
.
.
.
.
.
.
.
.
..
........
.
.
.
.
.
.
.
.
...............
.
.
.
.
.
.
.
.
.
.
..................
...
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.....................
.
.
...
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
........................................................
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
........................................................
-30
-25
-20
-15
-10
(c) Stephen Senn 2011
-5
0
5
10
14
NOTE FOR GUIDANCE ON
CLINICAL INVESTIGATION OF MEDICINAL PRODUCTS
IN THE TREATMENT OF HYPERTENSION
2004
P2 Section 3.1
Arbitrarily, response criteria for antihypertensive therapy include the
percentage of patients with a normalisation of blood pressure (reduction
SBP < 140 mmHg and DBP < 90 mmHg) and/or reduction of SBP ≥ 20
mmHg and/or DBP ≥ 10 mmHg. Results obtained should be discussed in
terms of statistical significance and in relation to their clinical relevance.
The first word in this paragraph is the most important
(c) Stephen Senn 2011
15
Dichotomania
• Continuous measurements taken and referred to
baseline
• Patients dichotomised as responder/non- responder
– Inefficient
– Arbitrary
• Sheep versus goats
– Ignores geep and shoats
• Analysis on risk difference scale to calculate NNT
(c) Stephen Senn 2011
16
Figure 1 Illustration of response region
Outcome DBP
100
Region( X )
90
80
90
95
100
105
110
X
Baseline DBP
Y = outcome, X = baseline
If (Y < 90 ∩ X > 95) ∪ (Y < 0.9X) patient ‘responds’
(c) Stephen Senn 2011
17
Why I mistrust the NNT
• It has very poor properties as a scale
– Reciprocal of risk difference
• It is theoretically unlikely to be stable from study to
study
– And this theoretical instability has been demonstrated by
empirical research
• It is an impatient measure
– It tries to shortcut the steps from study to practice
• It is an illusion that this can be done
• Those who advocate it are preferring an easy lie to a
difficult truth
(c) Stephen Senn 2011
18
Pharmacogenetics: A cutting-edge
science that will start delivering miracle
cures the year after next.
(c) Stephen Senn 2011
19
Moerman and Placebos
• Paper of 1983
• Investigated 31 placebo-controlled trials of
cimetidine in ulcer
• Found considerable variation in response
• Considered placebo response rate was an
important factor
• Has been cited by others as proof of variation
in treatment effect from trial to trial
(c) Stephen Senn 2011
20
(c) Stephen Senn 2011
21
Lessons from Moerman
•
There is no evidence of variation in the treatment
effect from trial to trial
We should be wary about concluding that apparent
variation signals true variation
We need to be cautious and think carefully about
analysis
Of course…it is always possible that there was exactly
the same genetic mix in each trial
•
•
•
–
•
in which case gene by treatment would not manifest itself as
trial by treatment interaction
We need to understand components of variation
(c) Stephen Senn 2011
22
Pharmacogenomics:
A subject with great promise.
(c) Stephen Senn 2011
23
What you learn in your first ANOVA
course
• Completely randomised design
– One way ANOVA
• Randomised blocks design
– Two way ANOVA
• Randomised blocks design with replication
– Two way ANOVA with interaction
• No replication, no interaction
(c) Stephen Senn 2011
24
Sources of Variation in Clinical Trials
Label
Source
Description
A
Between
treatments
The average difference between patients over all treatments
and all randomisations
B
Between
patients
The difference between patients given the same treatment.
(Averaged over both experimental and control treatments.)
C
Patient by
treatment
interaction
The extent to which the ’true’ difference between
treatments will vary from patient to patient. (Equivalently
the extent to which the difference between patients will
vary from treatment to treatment.)
D
Pure within
patient error
The extent to which the ‘response’ would vary for a given
patient when given the same treatment on different
occasions. (Averaged over all patients and both treatments.)
1.
Senn SJ. Individual Therapy: New Dawn or False Dawn. Drug Information Journal
2001;35(4):1479-1494
Identifiability and Clinical Trials
Type of Trial
Description
Identifiable effects
Error term
Parallel
Each patient
receives one
treatment
A
B+C+D
Cross-over
Each patient
receives each
treatments
A+B
C+D
Repeated crossover
Each patient
receives each
treatment at least
twice
A+B+C
D
A Word of Caution
• What is additive on one scale is not additive on
another
• The Moerman example suggests a constant effect on
the log-odds ratio scale
• If the background risk varies this translates into a
varying effect on the risk-difference scale
• The biological interpretation of this is then moot
• However the practical implication of this is
summarise on the additive scale
(c) Stephen Senn 2011
27
20/09/11
(c) Stephen Senn 2011
28
The Mottos
• Additive at the point of study
• Relevant at the point of application
• If NNTs have their place it is in decision making for individual
patients
• Not in reporting results from individual trials
• The additive scale has to be transformed into the relevant
scale at the point of treatment
• The fact that NNTs might be relevant when making an
individual decision is not an excuse for summarising results
this way
(c) Stephen Senn 2011
29
Tiotropium v Placebo
in Chronic Obstructive Pulmonary Disease
From the UPLIFT Study, NEJM, 2008
Significant differences in favor of tiotropium were observed at all time points for
the mean absolute change in the SGRQ total score (ranging from 2.3
to 3.3 units, P<0.001), although the differences on average were below what is
considered to have clinical significance (Fig. 2D). The overall mean
between-group difference in the SGRQ total score at any time point was
2.7 (95% confidence interval [CI], 2.0 to 3.3) in favor of tiotropium
(P<0.001). A higher proportion of patients in the tiotropium group than in
the placebo group had an improvement of 4 units or more in the SGRQ
total scores from baseline at 1 year (49% vs. 41%), 2 years (48% vs. 39%), 3
years (46% vs. 37%), and 4 years (45% vs. 36%) (P<0.001 for all comparisons).
(My emphasis)
(c) Stephen Senn 2011
30
Two Normal
distributions with the
same spread but the
Active treatment has a
mean 2.7 higher.
If this applies every
patient under active
can be matched to a
corresponding patient
under placebo who is
2.7 worse off
20/09/11
(c) Stephen Senn 2011
31
A cumulative plot
corresponding to
the previous
diagram.
If 4 is the threshold,
placebo response
probability is 0.36,
active response
probability is 0.45.
20/09/11
(c) Stephen Senn 2011
32
In summary…this is rather silly
• If there is sufficient measurement error even
if the true improvement is identically 2.7,
some will show an ‘improvement’ of 4
• The conclusion that there is a higher
proportion of true responders by the standard
of 4 points under treatment than under
placebo is quite unwarranted
• So what is the point of analysing ‘responders’?
(c) Stephen Senn 2011
33
Who are the authors?
1.
Tashkin, DP, Celli, B, Senn, S, Burkhart, D, Kesten, S, Menjoge, S,
Decramer, M. A 4-Year Trial of Tiotropium in Chronic Obstructive Pulmonary
Disease, N Engl J Med 2008.
Personal note. I am proud to have been involved in this important study and
have nothing but respect for my collaborators. The fact that, despite the fact
that two of us are statisticians, we have ended up publishing something like
this shows how deeply ingrained the practice of responder analysis is in
medical research. We must do something to change this.
(c) Stephen Senn 2011
34
In conclusion
• Responder analysis is the source of much confusion
• It is leading trialists to overestimate the individual element of
response to treatment
• The key to understanding response is replication and careful
analysis
• Stupid dichotomies do not help this understanding
• NNTs may be relevant at the point of application but they are
not relevant at the point of study
• Personalised medicine may be about to happen ‘soon’ for
quite a few years to come yet
(c) Stephen Senn 2011
35