A General Framework for Model-Based Drug
Development Using Probability Metrics for
Quantitative Decision Making
Ken Kowalski, Ann Arbor Pharmacometrics Group (A2PG)
Outline
Population Models


Basic Notation and Key Concepts
Basic Probabilistic Concepts
General Framework for Model-Based Drug Development
(MBDD)
Examples
Final Remarks/Discussion
Bibliography





2
PaSiPhIC 2012
A2PG
Population Models
Basic Notation
General Form of a Two-Level Hierarchical Mixed Effects
Model:
y i  f  i   h  i ,   i ,
 i ~ N 0 ,  

 i  g  ,  i  (e.g.,  i   i or  i e ),
i

 i ~ N 0 ,  
Definitions:
y i  vector of observatio ns for individual i
 i  vector of subject - specific parameters for individual i
  vector of fixed effects parameters
 i  vector of inter - individual random effects for individual i
 i  vector of intra - individual random effects for individual i
3
  covariance
matrix of inter - individual
random effects
  covariance
matrix of intra - individual
random effects
PaSiPhIC 2012
A2PG
Population Models
Key Concepts
Typical Individual Prediction:
 E  y i  i  0   f  


Easy to compute, same functional form as f
Population Mean Prediction:
 E  y i    E  y i  i  p  i d  i



Integral is often intractable when f is nonlinear
Typically requires Monte-Carlo integration (simulation)
The typical individual and population mean predictions
are not the same when f is nonlinear



4
Cannot observe a ‘typical individual’
Can observe a sample mean
PaSiPhIC 2012
A2PG
Basic Probabilistic Concepts
Statistical intervals (i.e., confidence and prediction
intervals)
Statistical power
Probability of achieving the target value (PTV)
Probability of success (POS)
Probability of correct decision (POCD)





5
PaSiPhIC 2012
A2PG
What’s the difference between a confidence
interval and prediction interval?
A confidence interval (CI) is used to make inference
about the true (unknown) quantity (e.g., population
mean)



Reflects uncertainty in the parameter estimates
Typically used to summarize the current state of knowledge
regarding the quantity of interest based on all available data
used in the estimation of the quantity
A prediction interval (PI) is used to make inference for a
future observation (or summary statistic of future
observations)


6
Reflects both uncertainty in the parameter estimates as well as
the sampling variation for the future observation
PaSiPhIC 2012
A2PG
Relationship Between CIs and PIs
Prediction Limits if
E(X ) Located Here
Distribution of
sampling variation
Prediction Limits Recognizing
Uncertainty in E(X )
X
Confidence Limits for X
Note: Prediction intervals are always wider than confidence intervals.
7
PaSiPhIC 2012
A2PG
Confidence interval for the mean based on a
sample of N observations
X  t1  
s
2,N
1
Sample mean
(parameter estimate)
8
N
Standard error of the mean
(parameter uncertainty)
PaSiPhIC 2012
A2PG
Prediction interval for a single future
observation
X  t1  
Sample mean
(parameter estimate)
s
2 , N 1
2
s
2
N
Sample variance of the mean
(parameter uncertainty)
Sample variance of a
future observation
(sampling variation)
Note: The sample mean based on N previous observations is the best estimate
for a single future observation.
9
PaSiPhIC 2012
A2PG
Prediction interval for the mean of M future
observations
X  t1  
Sample mean
(parameter estimate)
s
2 , N 1
2

N
Sample variance of the mean
(parameter uncertainty)
s
2
M
Sample variance of the mean
of M future observations
(sampling variation)
Note 1: The sample mean based on N previous observations is the best estimate
for the mean of M future observations.
Note 2: A prediction interval for M=∞ future observations is equivalent to a
confidence interval (see Slide 8). This will also be referred to as ‘averaging
out’ the sampling variation.
10
PaSiPhIC 2012
A2PG
A Conceptual Extension of Confidence and
Prediction Intervals to Population Modeling
Measure/Quantity
Parameters
Prediction
Sampling Variation
Simple Mean Model
Population Model
, 
, Ω, 

X
yˆ  f ˆ
s  ˆ
ˆ ,
ˆ  ˆ 2 I

s

ˆ , ˆ
Cov ˆ , 

Parameter Uncertainty*
sX 
Confidence Interval
See Slide 8
Stochastic simulations with
sufficiently large M
Prediction Interval
See Slide 10
Stochastic simulations with
finite M
N
* Note for the simple mean model the standard error of the mean does not take into account
uncertainty in the sampling variation (s) whereas in population models we typically take into
account the uncertainty in Ω and .
11
PaSiPhIC 2012
A2PG
Quantifying Parameter Uncertainty in
Population Models – Nonparametric Bootstrap



Randomly sample with replacement subject data vectors
to preserve within-subject correlations to construct
bootstrap datasets
Re-estimate model parameters for each bootstrap dataset
to obtain an empirical (posterior) distribution of the
parameter estimates (, Ω, )
May require stratified-resampling procedure (by design
covariates) for a pooled-analysis with very heterogeneous
study designs

12
E.g., limited data at a high dose in one study may be critical to
estimation of Emax
PaSiPhIC 2012
A2PG
Quantifying Parameter Uncertainty in
Population Models – Parametric Bootstrap

Draw random samples from multivariate normal distribution
with


Mean vector = [ ˆ ˆ ˆ ]
 ,,
Covariance matrix = Cov(ˆ , 
ˆ ,
ˆ)



Based on Fisher’s theory (Efron, 1982)
Assumes asymptotic theory (large sample size) that maximum
likelihood estimates converge to a MVN distribution


Obtained from Hessian or other procedure (e.g., COV step in NONMEM)
See Vonesh and Chinchilli (1997)
Based on Wald’s approximation that likelihood surface can be
approximated by a quadratic model locally around the
maximum likelihood estimates


13
Approximations are dependent on parameterization
Improved approximations may occur by estimating the natural
logarithm of the parameter for parameters that must be positive
PaSiPhIC 2012
A2PG
Non-parametric Versus Parametric
Bootstrap Procedures

The non-parametric bootstrap procedure is widely
used in pharmacometrics

Often used as a back-up procedure to quantify parameter
uncertainty when difficulties arise in estimating the
covariance matrix (eg., NONMEM COV step failure)


In this setting issues with a large number of convergence failures in
the bootstrap runs may call into question the validity of the
confidence intervals (i.e., Do they have the right coverage
probabilities?)
This form of parametric bootstrap procedure is less
computationally intensive than other bootstrap
procedures that require re-estimation

14
Requires successful estimation of the covariance matrix
(NONMEM COV step) but can lead to random draws
outside the feasible range of the parameters unless
appropriate transformations are applied
PaSiPhIC 2012
A2PG
Non-parametric Versus Parametric
Bootstrap Procedures (2)

Developing stable models that avoid extremely high
pairwise correlations (>0.95) between parameter
estimates and have low condition numbers (<1000)
can help



Ensure successful covariance matrix estimation
Reduce convergence failures in non-parametric
bootstrap runs
Choice of bootstrap procedure should focus on the
adequacy of the parametric assumption

15
Random draws from MVN versus the more computationally
intensive re-estimation approaches (e.g., non-parametric
bootstrap)
PaSiPhIC 2012
A2PG
Simulation Procedure to Construct Statistical
Intervals for Population Model Predictions
Obtain random draw
of , Ω,  from
bootstrap procedure
for kth trial
Summarize predictions
(e.g., mean)
stratified by design
(dose ,time, etc.)
Simulate subject
level data
Yi | , Ω, 
for M subjects
k<K
Note 1: To construct confidence interval consider
sufficiently large M (e.g., ≥2000 subjects) to
average out sampling variation in Ω and .
Note 2: For prediction intervals, M is chosen based on
observed or planned sample size.
16
PaSiPhIC 2012
Repeat for
k=1,…,K
trials
k=K
Use percentile
method to obtain
statistical interval
from K predictions
A2PG
To describe other probabilistic concepts we
need to define some additional quantities


True (unknown) treatment effect or quantity ()
Target value (TV)


A reference value for 
Data-analytic decision rule (e.g., Go/No-Go criteria)

17
Based on an observed treatment effect or quantity (T)
PaSiPhIC 2012
A2PG
Treatment Effect ()


 is the true (unknown) treatment effect
Models provide a prediction of 



ˆ , ˆ
ˆ  g ˆ , 

Uncertainty in the parameter estimates of the model
provides uncertainty in the prediction of 

18
P(ˆ ) denotes the distribution of predictions of 
PaSiPhIC 2012
A2PG
Example of Model-Predicted 
Dose-Response Model for Fasted Plasma Glucose (FPG)
Semi-mechanistic model of inhibition of glucose production



D
  K out FPG , R 0 
 K in  1 
ED 50  D 

dFPG
dt
Week 0
Week 2
Week 4
Week 6
Week 8
Week 12


D
Model-Predicted
Placebo
 1  e  K t 
R0 1 

Corrected
FPG
Versus
ED

D
50


Dose at Week 12
o ut
Dose (mg)
Population Mean Prediction
5th Percentile (90% LCL)
95th Percentile (90% UCL)
Dose (mg)
Observed Mean
Typical Individual Prediction (PRED)
19

Placebo-Corrected Delta FPG (mg/dL)
Delta FPG (mg/dL)
Mean Model Fit of FPG Versus Dose
FPG
(integrates data across dose and time)
PaSiPhIC 2012
A2PG
Target Value (TV)

Suppose we desire to develop a compound if the true
unknown treatment effect () is greater than or equal to
some target value (TV)

TV may be chosen based on:




Target marketing profile
Clinically important difference
Competitor’s performance
If we knew truth we would make a Go/No-Go decision
to develop the compound based on:


20
Go:
No-Go:
 ≥ TV
 < TV
PaSiPhIC 2012
A2PG
Data-Analytic Decision Rule

But we don’t know truth…

So we conduct trials and collect data to obtain an estimate of
the treatment effect (T)


T can be a point estimate or confidence limit on the estimate or
prediction of  (e.g., placebo-corrected change from baseline FPG)
We might make a data-analytic Go/No-Go decision to
advance the development of the compound if:


21
Go:
No-Go:
T ≥ TV
T < TV
PaSiPhIC 2012
A2PG
Statistical Power

Power is a conditional probability based on an assumed
fixed value of the treatment effect ()



Power = P(T ≥ TV | ) where P(T ≥ TV |  = TV) =  (false
positive)
TV=0 for statistical tests of a treatment effect
Power is an operating characteristic of the design based
on a likely value of 

22
No formal assessment that the compound can achieve the
assumed value of 
PaSiPhIC 2012
A2PG
Simulation Procedure to Calculate Power
Based on a Population Model-Predicted 
Use the same final
estimates (, Ω, )
for each
simulated trial
Simulate subject
level data
Yi | , Ω, 
for M subjects
Analyze simulated
data as per SAP
to test
Ho:  = TV
Ha:   TV
k<K
Note 1: Typically TV=0 when assessing whether the
compound has an overall treatment effect.
Note 2: When using simulations to evaluate power it is
good practice to also simulate data under the null
(e.g., no treatment effect or placebo model) to
verify that the Type 1 error () is maintained.
23
PaSiPhIC 2012
Repeat for
k=1,…,K
trials
k=K
Power is calculated
as the fraction of the
K trials in which
Ho is rejected
A2PG
Probability of Achieving the Target Value
(PTV)

Probability of achieving the target value is defined as the
proportion of trials where the true  ≥ TV

PTV = P( ≥ TV)



PTV is a measure of confidence in the compound at a
given stage of development


Does not depend on design or sample size
Based only on prior information through the model(s) and its
assumptions
Can change as compound progresses through development
PTV can be calculated from the same set of simulations
used to construct confidence intervals of the predicted
treatment effect ()
24
PaSiPhIC 2012
A2PG
Simulation Procedure to Calculate PTV Based
on Population Model Predictions
Obtain random draw
of , Ω,  from
bootstrap procedure
for kth trial
Summarize simulated
data to obtain
population mean
predictions of 
Simulate subjectlevel data Yi | , Ω, 
for arbitrarily
large M
k<K
Note: To calculate PTV use an arbitrarily large M (e.g.,
≥2000 subjects) to average out sampling variation in
Ω and . PTV should only reflect the parameter
uncertainty based on all available data used in the
model estimation.
25
PaSiPhIC 2012
Repeat for
k=1,…,K
trials
k=K
Calculate PTV as
proportion of K
trials in which
 ≥ TV
A2PG
Probability of Success (POS)

Probability of success is defined as the proportion of
trials where a data-analytic Go decision is made


POS is an operating characteristic that evaluates both the
performance of the compound and the design


POS = P(Go) = P(T ≥ TV)
In contrast to Power = P(T ≥ TV | ) which is an operating
characteristic of the performance of the design for a likely
value of 
POS is sometimes referred to as ‘average power’ where a
Go decision is based on a statistical hypothesis test

26
P Go   P T  TV
   P T
 TV  P   d 
PaSiPhIC 2012
A2PG
Simulation Procedure to Calculate POS Based
on a Population Model-Predicted 
Obtain random draw
of , Ω,  from
bootstrap procedure
for kth trial
Summarize simulated
data to obtain estimate
of  (T) and perform
hypothesis test
Simulate subjectlevel data Yi | , Ω, 
for planned sample
size (M)
k<K
Note: POS integrates the conditional probability of a
significant result over the distribution of plausible
values of  reflected through the uncertainty in the
parameter estimates for , Ω, and .
27
PaSiPhIC 2012
Repeat for
k=1,…,K
trials
k=K
Calculate POS as
proportion of K
trials in which
T ≥ TV
A2PG
Probability of Correct Decision (POCD)

A correct data-analytic Go decision is made when


A correct data-analytic No-Go decision is made when


T < TV and  < TV
Probability of a correct decision is calculated as the
proportion of trials where correct decisions are made


T ≥ TV and  ≥ TV
POCD = P(T ≥ TV and  ≥ TV) + P(T < TV and  < TV)
POCD can only be evaluated through simulation where
the underlying truth () is known based on the datageneration model used to simulate the data
28
PaSiPhIC 2012
A2PG
Simulation Procedure to Calculate POCD
Based on a Population Model-Predicted 
Obtain random draw
of , Ω,  from
bootstrap procedure
for kth trial
Simulate subjectlevel data Yi | , Ω, 
for planned sample
size (M)
Summarize
simulated data to
obtain estimate of
 (T)
Classify
Classify
Go:
Go:≥TV
≥TV
No
No Go:
Go:<TV
<TV
Under
UnderTruth
Truth
Compare Truth
Versus
Data-Analytic
Decision
Classify
Go: T≥TV
No Go: T<TV
Under Trial Data
k<K
Note: Classification of trial under
truth is obtained from the
29 PTV simulations.
Repeat for
k=1,…,K
trials
PaSiPhIC 2012
k=K
Calculate POS as
proportion of K
trials in which
T ≥ TV
A2PG
General Framework for MBDD







Basic assumptions of MBDD
Six components of MBDD
Clinical trial simulations (CTS) as a tool to integrate
MBDD activities
Table of trial performance metrics
Improving POCD
Setting performance targets
Comparing performance targets between early and late
stage clinical drug development
30
PaSiPhIC 2012
A2PG
Basic Assumptions of MBDD

Predicated on the assumptions:



That we can and should develop
predictive models
That these models can be used in CTS to
predict trial outcomes
Think of MBDD as a series of learnpredict-confirm cycles



31
Update models based on new data
(learn)
Conduct CTS to predict trial outcomes
(predict)
Conduct trial to obtain actual outcomes
and evaluate predictions (confirm)
PaSiPhIC 2012
Learn
Confirm
Predict
A2PG
Six Components of MBDD
Quantitative
Decision Criteria
PK/PD &
Disease Models
Trial Performance
Metrics
Evaluate
probability
of
Leverage
Implement
SAP,and
Evaluate
designs
Understand
achieving
Explicitly
andof
understanding
evaluate
alternative
dose
selection;
competitive
landscape
target
value
(PTV),
quantitatively
defined
pharmacology/disease
analysis
methods
incorporate
trial –
from
a
dose-response
success
(POS),
criteria
– useful
for such
ANCOVA,
MMRM,
execution
models
perspective
correct
decisions
(POCD)
“draw
line in the
sand”
regression,
NLME
as extrapolation
dropout
models
MBDD
Meta-Analytic Models
(Meta-Data from
Public Domain)
Data-Analytic Models
Design & Trial
Execution Models
32
PaSiPhIC 2012
A2PG
Clinical Trial Simulations (CTS)


Just as a clinical trial is the basic building block of a clinical drug
development program, clinical trial simulations should be the
cornerstone of an MBDD program
CTS allows us to assume (know) truth for a hypothetical trial


Based on simulation model we know 
Mimic all relevant design features of a proposed clinical trial




Analyze simulated data based on the proposed statistical analysis plan
(SAP)
Calculate T (test statistic for treatment effect) and apply data-analytic
decision rule
CTS should be distinguished from other forms of stochastic
simulations


Sample size, treatments (doses), covariate distributions, drop out rates,
etc.
E.g., CIs for dose predictions, PTV calculations, etc.
CTS can be used to integrate the components of MBDD and
the various probabilistic concepts (including POS and POCD)
33
PaSiPhIC 2012
A2PG
Table of Trial Performance Metrics
Trial No Go
Trial Go
Total
“True” Correct No Go
No Go
Incorrect Go
P(True No Go)
“True”
Go
Incorrect No Go
Correct Go
P(True Go)
P(Trial No Go)
P(Trial Go)
1.0
Total
POCD
34
POS
PaSiPhIC 2012
PTV
A2PG
Improving the probability of making correct
decisions

Change the design




Change the data-analytic model



 n/group
Regression-based designs ( # of dose groups,  n/group)
Consider other design constraints (cross-over, titration, etc.)
Regression versus ANOVA
Longitudinal versus landmark analysis
Change the data-analytic decision rule

Alternative choices for T


Point estimate, confidence limit, etc.
All of the above can be evaluated in a CTS
35
PaSiPhIC 2012
A2PG
Setting Performance Targets

PTV will change over time as model is refined and new
data emerge

Bring forward compounds/treatments with higher PTV as
compound moves through development


Industry average Phase 3 failure rate is approximately 50%



PTV may be low in early development
It is difficult to improve on this average unless we can routinely
quantify PTV
Strive to achieve PTV>50% before entering Phase 3
Strive to achieve high POCD in decision-making
throughout development
36
PaSiPhIC 2012
A2PG
Comparing performance targets between
early and late stage clinical drug development
Trial
No Go
Trial
Go
Total
True
No Go
High
Low
High
True Go
Low
Low
Low
Total
Low
Low
100%
Trial
No Go
Trial
Go
True
No Go
Low
Low
Low
True Go
Low
High
High
Total
Low
High
100%
Late Development
POCD
should be high
Early Development
POCD should be high
PTV
PTV
may be low
should be high
Advance good compounds /
treatments to registration
Kill poor compounds /
treatments early
37
Total
PaSiPhIC 2012
A2PG
Examples

Rheumatoid Arthritis Example


Urinary Incontinence Example


Phase 3 development decision
Potency-scaling for back-up to by-pass Phase 2a POC trial and
proceed to a Phase 2b dose-ranging trial
Acute Pain Differentiation Case Study

38
Decision to change development strategy to pursue acute pain
differentiation hypothesis
PaSiPhIC 2012
A2PG
Example – Rheumatoid Arthritis

Endpoints:



DAS28 remission (DAS28 < 2.6)
ACR20 response (20% improvement in ACR score)
Models developed based on Phase 2a study:


Continuous DAS28 longitudinal PK/PD model with Emax directeffect drug model
ACR20 logistic regression PK/PD model with Emax drug model


Conducted clinical trial simulations for a 24-week Phase 2b
placebo-controlled dose-ranging study (placebo, low, medium
and high doses)



Both direct and indirect-response models evaluated
At Week 12 non-responders assigned to open label extension at
medium dose level
Primary analysis at Week 24; Week 12 responses for non-responders
carried forward to Week 24
Evaluated No-Go/Hold/Go criteria for Phase 3 development
39
PaSiPhIC 2012
A2PG
Example – Rheumatoid Arthritis (2)
ACR20 (Difference from placebo)
DAS28-CRP Remission
(Difference from
placebo)
<20%
20-25%
25-30%
30%
<10%
No Go
No Go
Hold
Hold
10-16%
No Go
Hold
Hold
Hold
16-20%
Hold
Hold
Hold
Go
20%
Hold
Hold
Go
Go



No Go:
Hold:
Go:
40
Stop development
Wait for results of separate Phase 2b active comparator trial
Proceed with Phase 3 development without waiting for
results from comparator trial
PaSiPhIC 2012
A2PG
Example – Rheumatoid Arthritis (3)
Treatment
Probability (%)
No Go
Hold
Go
Low Dose
96.1
3.9
0.0
Medium Dose
28.1
62.9
9.0
High Dose
18.4
65.6
16.0
CTS results suggest a high probability that the team will have to wait for
results from the Phase 2b active comparator trial before making a decision
to proceed to Phase 3. Very low probability of taking low dose into Phase 3.
41
PaSiPhIC 2012
A2PG
Example – Urinary Incontinence

Endpoint:


Daily micturition (MIC) counts
Models developed:

Longitudinal Poisson-Normal model developed for daily MIC counts for
lead compound


Potency scaling for back-up based on:



Time-dependent Emax drug model using AUC0-24 as measure of exposure
In vitro potency estimates for lead and back-up (back-up more potent than lead)
Equipotency assumption between lead and back-up
Conducted CTS to evaluate Phase 2b study designs for back-up
compound (placebo and four active dose levels)



42
Evaluated various dose scenarios of low (L), medium #1 (M1), medium
#2 (M2) and high (H) doses levels
Implemented SAP (constrained MMRM analysis with step down trend
tests)
Quantified POS for the L, M1, M2 and H doses for the various dose
scenarios and potency assumptions
PaSiPhIC 2012
A2PG
Example – Urinary Incontinence (2)
Dose
L
Scenario
M1
M2
H
Comment
1
1X
2.5X
12.5X
25X
2
1X
2.5X
12.5X
37.5X
Doses selected favor in vitro
potency assumption (i.e., back-up
more potent than lead compound)
3
1X
5X
25X
50X
4
2.5X
5X
25X
75X
5
2.5X
12.5X
37.5X
75X
6
5X
12.5X
50X
100X
Doses selected favor equipotent
assumption
Note: Low (L) dose was selected to be a sub-therapeutic response. Design was
not powered to detect a significant treatment effect at this dose.
43
PaSiPhIC 2012
A2PG
Example – Urinary Incontinence (3)

CTS results:

High POS (>95%) demonstrating statistical significance at the H dose
for all 6 dose scenarios


High POS (>88%) demonstrating statistical significance at the M2
dose for all 6 dose scenarios


Insensitive to potency assumptions
POS varied substantially for demonstrating statistical significance of
the M1 dose


Insensitive to potency assumptions
Depending on dose scenario and potency assumption
POS < 60% for demonstrating statistical significance at the L dose

Except for dose scenarios 4 – 6 for the in vitro potency assumption
CTS results provided guidance to the team to select a range of doses that would have a
high probability of demonstrating dose-response while being robust to the uncertainty
in the relative potency between the back-up and lead compounds. Provided confidence
to bypass POC and move directly to a Phase 2b trial for the back-up.
44
PaSiPhIC 2012
A2PG
Case Study – Acute Pain Differentiation
Background


SC-75416 is a selective COX-2 inhibitor
Capsule dental pain study conducted



Poor pain response relative to active control (50 mg rofecoxib)
Lower than expected SC-75416 exposure with capsule relative
to oral solution evaluated in Phase 1 PK studies
PK/PD models developed to assess whether greater
efficacy would have been obtained if exposures were
more like that observed for the oral solution


45
Pain relief scores (PR) modeled as an ordered-categorical
logistic normal model
Dropouts due to rescue therapy modeled as a discrete survival
endpoint dependent on the patient’s last observed PR

Assumes a missing at random (MAR) dropout mechanism
PaSiPhIC 2012
A2PG
Case Study – Acute Pain Differentiation
Background (2)

PK/PD modeling predicted greater efficacy with oral
solution relative to capsules



A 6-fold higher SC-75416 dose (360 mg) than previously
studied predicted to have clinically relevant improvement in
pain relief relative to active control (400 mg ibuprofen)
Model extrapolates from capsules to oral solution and
leverages in-house data from other COX-2s and NSAIDs
Project team considers change in development strategy
to pursue a high-dose efficacy differentiation hypothesis

Original strategy was to determine an acute pain dose that was
equivalent to an active control and then scale down the dose
for chronic pain (osteoarthritis)

46
Based on well established relationships that chronic pain doses for
NSAIDs tend to be about half of the acute pain dose
PaSiPhIC 2012
A2PG
Case Study – Acute Pain Differentiation
Proposed POC Dental Pain Trial

Proposed conducting a proof of concept oral solution
dental pain study

Demonstrate improvement in pain relief for 360 mg SC
relative to 400 mg ibuprofen



Perform ANOVA on observed LOCF-imputed TOTPAR6
response and calculate LS mean differences



Primary endpoint is TOTPAR6 (SC vs. ibuprofen)
TOTPAR6 = 3 (TV) is considered clinically relevant
T = LS mean (SC) – LS mean (ibuprofen)
LCL95 = 2-sided lower 95% confidence limit on T
Compound and data-analytic decision rule:


47
Truth:
Data:
Go if ≥3, No-Go if <3
Go if T≥3 and LCL95>0, No-Go if T<3 or LCL95≤0
PaSiPhIC 2012
A2PG
Case Study – Acute Pain Differentiation
Simulation Procedure to Calculate PTV
Simulate PR Model
Parameters
Simulate PR Scores
M=2,000 patients
per treatment
(PR,2) ~ MVN
Simulate Dropout
Times
M=2,000 patients
per treatment
Simulate Dropout
Model Parameters
DO ~ MVN
k<K
Summarize
Distribution of
TOTPAR6 ()
48
k=K
Repeat for
k = 1,…,K=10,000
trials
PaSiPhIC 2012
Perform LOCF
Imputation and
Calculate TOTPAR6
Calculate Population
Mean TOTPAR6 &
TOTPAR6
Across M=2,000 pts
Determine
True Decision
Go: 3
No Go: <3
A2PG
Case Study – Acute Pain Differentiation
Posterior Distribution of TOTPAR6
360 mg SC-75416 vs 400 mg Ibuprofen
2500
PTV = P(  3) = 67.2%
F re q ue nc y
2000
Mean Prediction = 3.2
1500
1000
500
0
0
1
2
3
4
5
Delta-TOTPAR6
PTV = 67.2% sufficiently high to warrant recommendation to conduct oral
solution dental pain study to test efficacy differentiation hypothesis.
49
PaSiPhIC 2012
A2PG
Case Study – Acute Pain Differentiation
CTS Procedure to Evaluate POC Trial Designs
Simulate PR Scores
for k-th Trial
n pts / treatment
Perform LOCF
Imputation &
Calculate TOTPAR6
Calculate Mean
TOTPAR6 (T),
SEM & 95% LCL
Compare Truth vs.
Data-Analytic
Decision
Apply Decision Rule
Simulate Dropout
Times for k-th Trial
n pts / treatment
k<K
50
Repeat for
k=1,…,K=10,000
trials
Go: LCL>0 and T3
No Go: LCL0 or T<3
k=K
Calculate Metrics
POS
POCD
PaSiPhIC 2012
A2PG
Case Study – Acute Pain Differentiation
CTS Trial Performance Metrics
Trial
Truth
Trial No Go
LCL95  0 or T<3
Trial Go
LCL95> 0 and T3
Total
<3
20.81%
11.99%
32.80%
3
17.29%
49.91%
67.20%
61.90%
100%
(out of 10,000 trials)
Total
38.10%
POCD = 70.72%
POS = 61.90%
PTV = 67.20%
A sufficiently high POCD and POS supported the recommendation and
approval to proceed with the oral solution dental pain study.
51
PaSiPhIC 2012
A2PG
Case Study – Acute Pain Differentiation
Comparison of Observed and Predicted (About 9 months later…)
Placebo
F re qu e nc y
400
60 mg SC-75416
Pred = -7.0
Obs = -9.6
300
200
100
0
-12
-8
-4
0
4
8 -12
-8
DELTA
F re qu e nc y
400
0
4
8
4
8
360 mg SC-75416
Pred = 3.2
Obs = 3.3
Pred = 2.0
Obs = 2.6
300
-4
DELTA
180 mg SC-75416
200
100
0
-12
-8
-4
0
4
8 -12
DELTA
52
Pred = -0.9
Obs = -1.8
-8
-4
0
DELTA
PaSiPhIC 2012
A2PG
Case Study – Acute Pain Differentiation
Summary of Results

360 mg SC-75416 met pre-defined Go decision criteria





Confirmed model predictions
Demonstrated statistically significant improvement relative to 400
mg ibuprofen
MBDD approach provided rationale to pursue acute pain
differentiation strategy that might not have been pursued
otherwise
Allowed progress to be made while reformulation of solid
dosage form was done in parallel
Validation of model predictions provided confidence to pursue
alternative pain settings for new formulations without
repeating dental pain study

53
Model could be used to provide predictions for new formulations
PaSiPhIC 2012
A2PG
Final Remarks/Discussion


Some thoughts on implementing MBDD
Challenges to implementing MBDD
54
PaSiPhIC 2012
A2PG
Final Remarks/Discussion
Some thoughts on implementing MBDD

We need to clearly define objectives


We need explicit and quantitatively defined decision criteria


It’s difficult to know how to apply the models if decision criteria are
ambiguous or ill-defined
We need complete transparency in communicating model
assumptions



What questions are we trying to address with our models?
Entertain different sets of plausible model assumptions
Evaluate designs for robustness to competing assumptions
We need to routinely evaluate the predictive performance of
the models on independent data

55
Modeling results should be presented as ‘hypothesis generating’
requiring confirmation in subsequent independent studies
PaSiPhIC 2012
A2PG
Final Remarks/Discussion
Some thoughts on implementing MBDD (2)

Conduct CTS integrating information across disciplines



Provide graphical summaries of CTS results for
recommended design prior to the release of the actual
trial results


Implement key features of the design and trial execution (e.g.,
dropout)
Implement statistical analysis plan (SAP)
Perform quick assessment of predictive performance when
actual trial reads out
Update models and quantification of PTV after actual trial
reads out

56
i.e., Begin new learn-predict-confirm cycle
PaSiPhIC 2012
A2PG
Final Remarks/Discussion
Challenges to implementing MBDD

Focus on timelines of individual studies and a ‘go-fast-at-risk’
strategy (i.e., minimizing gaps between studies) can be counterproductive to a MBDD implementation


Integration of MBDD activities in project timelines will require focus
on integration of information across studies



M&S (learning phase) is a time-consuming effort
Not just tracking of individual studies
May need processes to allow modelers to be un-blinded to interim
results to begin modeling activities earlier to meet aggressive
timelines
Insufficient scientific staff with programming skills to perform CTS


57
Pharmacometricians and statisticians with such skills should be identified
CTS implementation often requires considerable customization to
address the project’s needs (i.e., no two projects are alike)
PaSiPhIC 2012
A2PG
Final Remarks/Discussion
Challenges to implementing MBDD (2)


Insufficient modeling and simulation resources to
implement MBDD on all projects
Reluctance to be explicit in defining decision rules (i.e.,
reluctance to ‘draw line in the sand’)


Due to complexities and tradeoffs in making decisions
Can be difficult to achieve consensus


http://www.ascpt.org/Portals/8/docs/Meetings/2012%20Annual%20Me
eting/2012%20speaker%20presentations/ASOP%20TUE%20CHERRY%
20BLOS%20SESSION%201.pdf
Reluctance to use assumption rich models


58
We make numerous assumptions now when we make
decisions…we’re just not very explicit about them
MBDD can facilitate open debate about explicit assumptions
PaSiPhIC 2012
A2PG
Bibliography








Neter, J., and Wasserman, W. Applied Linear Statistical Models, Irwin Inc., IL, 1974, pp.
71-73.
Efron, B. The Jackknife, the Bootstrap, and Other Resampling Plans, Society for
Industrial and Applied Mathematics, PA, 1982, pp. 29-30.
Vonesh, E.F., and Chinchilli, V.M. Linear and Nonlinear Models for the Analysis of
Repeated Measurements, Marcel Dekker, Inc., NY, 1997, pp. 245-246.
Kowalski, K.G., Ewy, W., Hutmacher, M.M., Miller, R., and Krishnaswami, S. “ModelBased Drug Development – A New Paradigm for Efficient Drug Development”.
Biopharmaceutical Report 2007;15:2-22.
Lalonde, R.L., et al. “Model-Based Drug Development”. Clin Pharm Ther 2007;82:2132.
Chuang-Stein, C.J., et al. “A Quantitative Approach to Making Go/No Go Decisions
in Drug Development”. DIJ 2011;45:187-202.
Smith, M.K., et al. “Decision-Making in Drug Development – Application of a ModelBased Framework for Assessing Trial Performance”. Book chapter in Clinical Trial
Simulations: Applications and Trends, Kimko H.C. and Peck C.C. eds. , Springer Inc., NY,
2011, pp. 61-83.
Kowalski, K.G., Olson, S., Remmers, A.E., and Hutmacher, M.M. “Modeling and
Simulation to Support Dose Selection and Clinical Development of SC-75416, a
Selective COX-2 Inhibitor for the Treatment of Acute and Chronic Pain”. Clin
Pharm Ther, 2008; 83: 857-866.
59
PaSiPhIC 2012
A2PG