Mitigating Risk of
Out-of-Specification Results
During Stability Testing of
Biopharmaceutical Products
36th Annual
Midwest Biopharmaceutical Statistics Workshop
May 21, 2013
Jeff Gardner
Principal Consultant
Outline
What Is an OOS Result?
Business Consequences of OOS Results
- Regulatory requirements
- Marketing/business impact
Risks of OOS Associated with Statistical Approaches to
Estimating Shelf Life
- ICH Q1E Guidance
- Mixed model (random slopes) estimation
Recommendations for Managing OOS Risk
What Is an OOS Result?
From 21 CFR 314.81(b)(1)(ii):
“Any failure of a distributed batch to meet [i.e. conform to]
any of the specifications established in an application.”
From ICH Q6A/Q6B:
“’Conformance to specifications’ means that the drug
substance and/or drug product, when tested according to
the listed analytical procedures, will meet the listed
acceptance criteria.”
Expectation is that any test result, not just the average of
results or majority of results, meets the specified acceptance
criterion.
Business Impact of OOS Results
Regulatory Requirements
• Field alert required within 72 hours of identification
Commercial Impact
• Potential product recall
• Possible reduction of shelf-life
Illustration
105
Regression
90% CI
90% PI
105
104
S
R-Sq
R-Sq(adj)
103
102
0.885278
57.1%
49.9%
Potency
101
100
99
98
97
96
95
95
94
0
6
12
18
24
30
36
Months
As seen here, a portion of the population of results extends into the OOS region,
meaning that some risk exists of seeing OOS results.
ICH Q1E Guidance
“An appropriate approach to… shelf life estimation is to analyze a
quantitative attribute… by determining the earliest time at which the
95 percent confidence limit for the mean intersects the proposed
acceptance criterion.”
The above applies in either of two cases (contingent on assumption
of common slope using “poolability” test):
• Linear regression analysis is performed on data from a single batch
•
In this case, mean = batch mean
• Linear regression analysis is performed on data from multiple
batches
•
•
In this case, mean = mean of all batches in analysis
This mean is then taken to be an estimate of the product mean
Numerous critiques exist in literature which address “flaws” or limitations of the
statistical approach outlined in ICH Q1E.
ICH Q1E Limitations
Methodology treats batch as a fixed effect, thereby limiting
inference to only the sample of batches being evaluated.
Calculation of a (1- α)% prediction interval about the mean is
only useful in defining the region of likely results from the
batch(es) being evaluated.
This is despite the stated aim of the guidance which is to
provide a “high degree of confidence” that all future batches
will remain within acceptance criteria.
A statistical methodology is needed that more closely aligns with stated objectives of
the ICH guidance documents.
Mixed Effects Model
Consider the model
𝑦𝑖𝑗 = 𝐴 + 𝑎𝑖 + 𝐵𝑥𝑗 + 𝑏𝑖 𝑥𝑗 + 𝑒𝑖𝑗
A = Population intercept (mfg. process mean)
B = Mean product degradation rate
ai = Initial value for batch i
bi = Degradation rate for batch i
xj = Stability test interval j
eij = Random error when measuring batch i at test interval j
𝑎𝑖 ~ N(0,σ2𝑖𝑛𝑡 ), 𝑏𝑖 ~ N(0,σ2𝑠𝑙𝑜𝑝𝑒 ) , 𝑒𝑖𝑗 ~ N(0,σ2𝑒𝑟𝑟 )
Inclusion of random effects ai and bi in the model allows for calculation of prediction
limits which more closely align with stated objective of ICH guidance.
Prediction Intervals Using Mixed Model
When the model parameters are known, (1- α)% prediction intervals
can be constructed at each xj such that
𝜑𝑏𝑎𝑡𝑐ℎ = 𝐴 + 𝐵𝑥𝑗 ± 𝑧𝛼
and
𝜑𝑖𝑛𝑑𝑖𝑣 = 𝐴 + 𝐵𝑥𝑗 ± 𝑧𝛼
σ2𝑖𝑛𝑡 + 𝑥𝑗2 σ2𝑠𝑙𝑜𝑝𝑒
Reflects test result
(i.e. analytical method)
variability
σ2𝑖𝑛𝑡 + 𝑥𝑗2 σ2𝑠𝑙𝑜𝑝𝑒 + σ2𝑒𝑟𝑟
Constructing the intervals in this way assumes that σ2𝑖𝑛𝑡 , σ2𝑠𝑙𝑜𝑝𝑒 and
σ2𝑒𝑟𝑟 are all independent.
The first equation above is “philosophically” closest to ICH Q1E in that it governs the
distribution of individual batch means at each stability testing interval.
Prediction Intervals Using Mixed Model
When the model parameters are unknown (which is pretty
much always the case),
𝜑𝑏𝑎𝑡𝑐ℎ = 𝐴 + 𝐵𝑥𝑗 ± 𝑡𝛼,𝑑𝑓
σ2𝑖𝑛𝑡 + 𝑥𝑗2 σ2𝑠𝑙𝑜𝑝𝑒
and
𝜑𝑖𝑛𝑑𝑖𝑣 = 𝐴 + 𝐵𝑥𝑗 ± 𝑡𝛼,𝑑𝑓
σ2𝑖𝑛𝑡 + 𝑥𝑗2 σ2𝑠𝑙𝑜𝑝𝑒 + σ2𝑒𝑟𝑟
For sake of discussion, we will use the first equation as the basis for determining
shelf life since it aligns closest with ICH Q1E.
Determination of Shelf Life
Using α=0.05, shelf life = maximum xj such that
𝜑𝑏𝑎𝑡𝑐ℎ ≤ Upper Acceptance Limit (UAL)
or
𝜑𝑏𝑎𝑡𝑐ℎ ≥ Lower Acceptance Limit (LAL)
Once shelf life has been determined, calculate
Pj(OOS) = P(𝜑𝑖𝑛𝑑𝑖𝑣 ≤ LAL | xj = Shelf Life)
+
P(OOS) = P(𝜑𝑖𝑛𝑑𝑖𝑣 ≥ UAL | xj = Shelf Life)
Shelf life and Pj(OOS) were determined as described above for n = 25 = 32
experimental settings in order to examine the relationship between the two.
Experimental Settings
𝜑batch and 𝜑indiv were calculated for each combination of
settings for A, B, σ𝑖𝑛𝑡 , σ𝑠𝑙𝑜𝑝𝑒 , and σ𝑒𝑟𝑟 as specified below.
A: 0, -0.4
B: -0.015, -0.04
σ𝑖𝑛𝑡 : 0.15, 0.33
σ𝑠𝑙𝑜𝑝𝑒 : 0.008, 0.024
σ𝑒𝑟𝑟 : 0.05, 0.15
All numbers are expressed as multiples
of unit scale where
1=
UAL − LAL
2
(two-sided)
or
1 = UAL (one-sided)
Example 1: P(OOS) at 24-Month Expiry
A
σ𝑖𝑛𝑡
B
σ𝑠𝑙𝑜𝑝𝑒
σ𝑒𝑟𝑟
0
0
0
0
0
0
0
0.33
0.33
0.33
0.33
0.33
0.33
0.33
-0.015
-0.015
-0.015
-0.015
-0.015
-0.015
-0.015
0.008
0.008
0.008
0.008
0.008
0.008
0.008
0.15
0.15
0.15
0.15
0.15
0.15
0.15
Month Mean
(xj)
(A + Bxj)
0
0.00
3
-0.05
6
-0.09
9
-0.14
12
-0.18
18
-0.27
24
-0.36
𝜑𝑏𝑎𝑡𝑐ℎ P(OOS)
-0.543
-0.589
-0.639
-0.691
-0.745
-0.862
-0.988
0.0029
0.0043
0.0064
0.0096
0.0144
0.0306
0.0594
Since parameters are pre-defined (known), z statistics were
used in lieu of t for these calculations.
𝜑batch and 𝜑indiv were calculated in this manner at 0 through 24 months for each of
the 32 experimental settings.
Probability of OOS at Expiry
Probability of OOS Result
Relationship Between Estimated Shelf Life and Probability of OOS at Expiry
for Varying Settings of Random Coefficients Model Parameters
0.09
0.08
0.07
0.06
0.05
0.04
0.03
0.02
0.01
0
0
3
6
9
12
15
Supported Expiry
18
21
24
Determining Cumulative P(OOS)
The cumulative P(OOS) at xj is given by
𝑗
1−
(1 − 𝑃𝑗 (𝑂𝑂𝑆))
1
Note: this probability significantly increases if testing is performed
more frequently later in shelf life
Therefore the cumulative probability is most useful when calculated
only for those intervals that are specified when establishing the
stability testing protocol.
Determining cumulative probability of OOS provides a direct measure of business
risk of field alert/product recall.
Example 2: Cumulative P(OOS)
A
σ𝑖𝑛𝑡
B
σ𝑠𝑙𝑜𝑝𝑒
σ𝑒𝑟𝑟
0
0
0
0
0
0
0
0.33
0.33
0.33
0.33
0.33
0.33
0.33
-0.015
-0.015
-0.015
-0.015
-0.015
-0.015
-0.015
0.008
0.008
0.008
0.008
0.008
0.008
0.008
0.15
0.15
0.15
0.15
0.15
0.15
0.15
Month Mean
(xj)
(A + Bxj)
0
0.00
3
-0.05
6
-0.09
9
-0.14
12
-0.18
18
-0.27
24
-0.36
P(OOS)
0.0029
0.0043
0.0064
0.0096
0.0144
0.0306
0.0594
Cum.
P(OOS)
0.0029
0.0072
0.0135
0.0230
0.0371
0.0666
0.1220
Note: same model parameters used here as in Example 1.
Cumulative Probability of OOS
Relationship Between Estimated Shelf Life and Cumulative P(OOS)
for Varying Levels of Random Coefficients Model Parameters
Cumulative Probability of OOS
0.14
0.12
0.1
0.08
0.06
0.04
0.02
0
0
3
6
9
12
15
Supported Expiry
18
21
24
This chart illustrates that determining shelf life based on mean alone (x-axis) does
not address the risk of OOS results.
Application
Example 3: 12 Month Assay Data
Consider the following:
Plot of Assay (% Label Claim) vs Month
105
Spec Limit
103
Assay
101
Lot
A1341
A1342
A1343
99
97
95
Spec Limit
0
3
6
9
Month
12
15
18
Example 3 cont’d: Q1E Approach
• Per ICH Q1E, p-value for the month*lot effect (<0.25) requires
determination of individual confidence limit for each lot.
• Shelf life is based on the earliest time at which any lot’s
confidence limit intersects the lower acceptance limit (shown in
previous graph).
• In this case, the data supports a maximum shelf life of 15 months
which per ICH Q1E would have to be verified with real-time data.
Example 3 cont’d: Mixed Model Approach
σ2𝑖𝑛𝑡
σ2𝑠𝑙𝑜𝑝𝑒
σ2𝑒𝑟𝑟
A
B
The point
estimates for A,
B, σ2𝑖𝑛𝑡 , σ2𝑠𝑙𝑜𝑝𝑒 ,
and σ2𝑒𝑟𝑟
can be used to
determine
Pj(OOS).
(Note: Output is
from SAS®
PROC MIXED)
Example 3 cont’d
Using 𝑡9.88 =
95 −(𝐴+𝐵𝑥𝑗 )
σ2𝑖𝑛𝑡 + 𝑥𝑗2 σ2𝑠𝑙𝑜𝑝𝑒 +σ2𝑒𝑟𝑟
,
Based on a nominal
α=0.05, the cumulative
P(OOS) suggests a shelf
life of 12 months.
Summary
Summary of ICH Guidance
Estimation of shelf-life is based on the earliest point at which a
batch’s true mean is likely to intersect a specification limit.
• Substantial risk exists for observing individual OOS analytical
results during the expiry period
• OOS risk is greatest toward the end of product shelf-life
Estimating Expiry Using Mixed Model
A prediction interval based on a mixed effects model can be
constructed with either of two aims:
• philosophical alignment with ICH guidance (i.e. including
only variance components for batch intercept & slope
variability)
• minimizing risk of OOS results (by including residual variance
component as well as batch variance components)
Either of the above methods are an improvement upon the
statistical approaches outlined in ICH guidance.
Impact of Analytical Method Variability
It is recommended that both prediction interval calculations are
employed for preliminary shelf life estimation.
• Difference in shelf life estimates = impact of analytical
method variability on shelf-life
• Additional analysis may be considered for evaluating how
this impact can be lessened via increased replication
• Such understanding may provide guidance toward optimal
method validation
Formally Assessing Risk of OOS
It’s recommended that a cumulative probability of OOS
occurrence accompanies each preliminary shelf life estimate
1. calculate P(OOS) at each scheduled stability testing
interval
2. calculate cumulative P(OOS) across all stability testing
intervals
It may be worth considering whether the final proposed shelf
life should be based on first determining a maximum
cumulative P(OOS) rather than by focusing on lower bound on
the population of batch slopes.
Future Research
Using experimental settings defined earlier, examine the
sampling distributions of model parameter estimates via Monte
Carlo simulations:
• How does sampling error contribute to over-estimation or
under-estimation of P(OOS)?
• What impact does analytical method variability have on shelf
life estimates if P(OOS) is used instead of the distribution of
batch means at a given testing interval?
End of Presentation