Ram C. Tiwari
Associate Director
Office of Biostatistics, CDER, FDA
Ram.Tiwari@fda.hhs.gov
Disclaimer
This presentation reflects the views of the author and
should not be construed to represent FDA’s views or
policies.
Benefit-risk Assessment
2
Benefit-risk Assessment
3
Outline
 Introduction
 Commonly-used Benefit-risk (BR) measures
 Methodology
 BR measures based on Global benefit-risk (GBR) scores and
a new measure
 Bayesian approaches

Power prior
 Illustration and simulation study
 Future work
Benefit-risk Assessment
4
Introduction
 The benefit-risk assessment is the basis of regulatory decisions in the pre-
market and post market review process.
 The evaluation of benefit and risk faces several challenges.
Benefit-risk Assessment
5
Commonly used B-R measures
 Various measures have been proposed to assess benefit and risk
simultaneously:
 Q-TWiST by Gelbert et al. (1989)
 Ratio of benefit and risk by Payne (1975)
 The Number Needed to Treat and the Number Needed to Harm by
Holden et al. (2003)
 Global Benefit Risk (GBR) scores by Chuang-Stein et al. (1991)
Benefit-risk Assessment
6
BR categories
 A five-category multinomial random variable to capture the benefit and
risk of a drug product on each individual simultaneously:
Table 1: Possible outcomes of a clinical trial with binary response data
Benefit
No benefit
No AE
Category 1
Category 3
AE
Category 2
Category 4
withdrawal
Benefit-risk Assessment
Category 5
7
Example: Hydromorphone
Data was provided by Jonathan Norton.
Benefit-risk Assessment
8
GBR scores
2
5
i 1
i 3
BRScore _ Linear   wi pi   wi pi
2
BRScore _ Ratio 
( wi pi ) e
i 1
5
w p
i 3
i
BRScore _ Cmp _ Ratio 
i
w1 p1
w2 p2
(
)f
w5 p5 w3 p3  w4 p4
where w1  2, w2  1, w3  0, w4  1, w5  2
Benefit-risk Assessment
9
Methodology: BR measures
 BR measures based on the global scores proposed by Chuang-Stein et al.
2
5
i 1
i 3
2
5
i 1
i 3
BR _ Linear  ( wi pi ,T   wi pi ,T )  ( wi pi ,C   wi pi ,C )
2
BR _ Ratio 
( wi pi ,T ) e
i 1
5
w p
i 3
i
BR _ Cmp _ Ratio 
2

( wi pi ,C ) e
i 1
5
w p
i ,T
w1 p1,T
i 3
(
i
i ,C
w2 p2,T
w5 p5,T w3 p3,T  w4 p4,T
)f 
w1 p1,C
(
w2 p2,C
w5 p5,C w3 p3,C  w4 p4,C
)f
where w1  2, w2  1, w3  0, w4  1, w5  2


BR measures based on the global scores are for each arm
(treatment and comparator) separately.
BR_Linear can take a continuous value on a scale of -4 to 4
(inclusive).
Benefit-risk Assessment
10
Methodology: New BR measure
 A new indicator based measure is proposed:
2
5
i 1
i 3
BR _ Indicator  wi  ( pi ,T , pi ,C )   wi  ( pi ,T , pi ,C )


1
if
a

b






where  (a, b)   0 if a  b 





1
if
a

b






BR_Indicator compares two arms simultaneously.
It takes a integer value between -6 to 6 (inclusive).
Benefit-risk Assessment
11
Methodology: Dirichlet prior
 Dirichlet distribution is used as the conjugate prior for multinomial
distribution, and the posterior distribution of the five-category random
variable is derived at each visit using sequentially updated posterior as a
prior.
Benefit-risk Assessment
12
Methodology: Sequential Updating
 Sequential updating of the posteriors are given by:
 The posterior mean (i.e., Bayes estimate) and 95% credible
interval for each of the four measures are obtained using a
Markov chain Monte Carlo (MCMC) technique.
Benefit-risk Assessment
13
Methodology: Decision Rules

For a BR measure,
 If the credible interval include the value zero, the benefit does not
outweigh the risk;
 If the lower bound of the credible interval is greater than zero, the
benefit outweighs the risk;
 If the upper bound of the credible interval is less than zero, the risk
outweighs the benefit.
Benefit-risk Assessment
14
Methodology: Power Prior
 Power prior (Ibrahim and Chen, 2000) is used through the likelihood function
to discount the information from previous visits, and the posterior
distribution of the five-category random variable is obtained using the
Dirichlet prior for p and a Beta (1, 1) as a power prior for a0 .
Benefit-risk Assessment
15
Methodology: Model Fit
 The model fit of the two models (with and without power prior) is assessed
through the conditional predictive ordinate (CPO) and the logarithm of the
pseudo-marginal likelihood (LPML). The larger the value of LPML, the
better fit the model is. Here, n(i) is the data with ni removed.
Benefit-risk Assessment
16
Back to our example: Hydromorphone
Benefit-risk Assessment
17
3
Illustration: Posterior Means and 95% Credible Intervals for
BR_Linear Measure
Benefit
4
1
0
1
2
3
4
5
6
7
8
-1
-2
without power prior
with power prior
-3
-4
Risk
BR Linear Measure
2
Visit
Benefit-risk Assessment
18
Illustration: Posterior Means and 95% Credible Intervals for
BR_Indicator Measure
Benefit
6
2
0
1
2
3
4
5
6
7
8
-2
-4
without power prior
with power prior
Risk
BR Indicator Measure
4
-6
Visit
Benefit-risk Assessment
19
Illustration: Results
a. The model without power prior
Benefit-risk Assessment
b. The model with power prior
20
Illustration: Posterior Means and 95% Credible Intervals for Power
Prior Parameter
Benefit-risk Assessment
21
Illustration: Model Fit
LPML values
Model without power prior
Model with power prior
Benefit-risk Assessment
Treatment
Control
-14.230
-14.209
-6.432
-6.190
22
Simulation study
 Correlated longitudinal multinomial data are simulated using the R
package SimCorMultRes.R, which uses an underlying regression model to
draw correlated ordinal response.
 Two scenarios are simulated:
 The treatment arm is similar to the control arm in terms of benefit-risk;
 The treatment arm is better than control arm in the sense that the
benefit outweighs risk.
Benefit-risk Assessment
23
Simulation study: Scenarios
Scenario 1: Treatment benefit does not outweigh risk compared to control
Scenario 2: Treatment benefit outweighs risk compared to control
Benefit-risk Assessment
24
Simulation study: Scenario 1
Treatment benefit does not outweigh risk compared to control
a. The model without power prior
Benefit-risk Assessment
b. The model with power prior
25
Simulation study: Scenario 2
Treatment benefit outweighs risk compared to control
a. The model without power prior
Benefit-risk Assessment
b. The model with power prior
26
Simulation study: Results
Scenario 1: Treatment benefit does not outweigh risk compared to control
Scenario 2: Treatment benefit outweighs risk compared to control
Benefit-risk Assessment
27
Simulation study: Model Fit
LPML values
Treatment
Control
-23.536
-23.354
-8.472
-7.667
-27.099
-21.840
-8.532
-8.393
Scenario 1:
Model without power prior
Model with power prior
Scenario 2:
Model without power prior
Model with power prior
Benefit-risk Assessment
28
Benefit-risk Assessment
29
Future work in BR assessment
 Frequentist approaches:
 Bootstrap approach
 General linear mixed model (GLMM) approach
 Other Bayesian approaches:
 Normal priors
 Dirichlet process
Benefit-risk Assessment
30
Bootstrap Approach
 Approximate underlying distribution using the empirical
distribution of the observed data;
 Resample from the original dataset;
 Calculate the estimates and confidence intervals (CIs) of
the BR measures based on the bootstrap samples;
 Percentile bootstrap CIs;
 Basic bootstrap CIs;
 Studentized bootstrap CIs;
 Bias-Corrected and Accelerated CIs.
 Apply the decision rules.
Benefit-risk Assessment
31
Bootstrap Approach-Results
Benefit-risk Assessment
32
General linear mixed model (GLMM) approach
 Within each arm (T or C), the ith subject falls in the jth
category (vs. the first category) at kth visit can be modeled
as,
P(Yik  j )
log
  0   j   k   ik
P(Yik  1)
 where, α0 is the baseline effect assumed common across
all categories, βj is the category effect, and γk is the
longitudinal effect at kth visit, with
and,  ik ~ N (0,  2.)
Benefit-risk Assessment
33
GLMM approach
 Note that different variance-covariance structures can be
used for (γ1,γ2,…γ8), to model the longitudinal trend.
 Compound-symmetry
 Power covariance structure
 Unstructured covariance structure
 The estimates of the confidence intervals of the global
measures can be derived from Monte Carlo samples, and
the decision rules can be determined based on the
confidence intervals.
Benefit-risk Assessment
34
General linear mixed model approach-Results
Benefit-risk Assessment
35
Bayesian approaches with GLMM
log
P(Yik  j )
  0   j   k   ik
P(Yik  1)
 (α0 , βj ; j=1,…,5)~ independent Normal with means 0 and large variances;
 Variance parameters~ IG
 Dirichlet Process Approach: Let α0 to depend on subjects, that is, assume
that α0i |G ~ iid G, with G~ DP(M, G0), M>0 concentration parameter and
G0 a baseline distribution such as a normal with mean 0 and large
variance. βj ; j=1,…,5 are independent normal with means 0, and large
variances.
 The posterior distributions of the probability and the global measures can
be derived, and the decision rules can be determined based on the
credible intervals.
Benefit-risk Assessment
36
Discussion




Quantitative measure of benefit and risk is an important aspect in the
drug evaluation process.
The Bayesian method is a natural method for longitudinal data by
sequentially updating the prior; Power prior can be used to discount
information from previous visits.
Frequentist approaches such as bootstrapping method and general linear
mixed model can be applied for benefit risk assessment.
Continuous research in longitudinal assessment of drug benefit-risk is
warranted.
Benefit-risk Assessment
37
Benefit-risk Assessment
38
Selected References
 Gelber RD, Gelman RS, Goldhirsch A. A quality-of-life oriented endpoint for comparing
treatments. Biometrics. 1989;45:781-795
 Payne JT, Loken MK. A survey of the benefits and risks in the practices of radiology. CRC Crit
Rev Clin Radiol Nucl Med. 1975; 6:425-475
 Holden WL, Juhaeri J, Dai W. “Benefit-Risk Analysis: A Proposal Using Quantitative Methods,”
Pharmacoepidemiology and Drug Safety. 2003; 12, 611–616. 154
 Chuang-Stein C, Mohberg NR, Sinkula MS. Three measures for simultaneously evaluating
benefits and risks using categorical data from clinical trials. Statistics in Medicine. 1991;
10:1349-1359.
 Norton, JD. A Longitudinal Model and Graphic for Benefit-risk Analysis, with Case Study. Drug
Information Journal. 2011; 45: 741-747.
 Ibrahim, JG, Chen, MH. Power Prior Distributions for Regression Models. Statistical Science.
2000; 15: 46-60.
Benefit-risk Assessment
39
Q&A
Benefit-risk Assessment
40