10th Annual
Pharma&Biotech
BioProcess International
Conference & Exhibition
Hynes Convention Center
Boston, MA
16Sep - 19Sep 2013
Quality Risk Assessment and Management
Strategies for BioPharmaceuticals
Ben Locwin, PhD, MBA
Head of Training & Development / Risk and Decision Sciences,
Quality Assurance
Lonza Biopharmaceuticals
© Lonza
CASE STUDY/UNPUBLISHED DATA
Risk Analysis – A Brief Primer

Where does risk assessment come from?

Why is it important?

What is the overall utility?
2
Apr-15
Why Do We Need Formal Tools?

We’re human!

We have fallibilities and produce errors when
trying to estimate likelihoods and outcomes
“We often think that when we have
completed our study of “one” we
know all about “two,” because “two”
is “one and one.” We forget that we
still have to make a study of “and.”
-- Sir Arthur Eddington
3
Apr-15
Different Types of Cultures

Immature vs. Mature risk-view cultures

NOT termed “-averse” or “-seeking” here for a
reason
4
Apr-15
Loss Analysis
 What is the theory?
 To get a better handle on risk!
R = Pf x M c
How likely
is it?
How big is
it?
Uncertainty in Risk
 Uncertainty in estimates makes us less confident in our decisions
The “uncertainty path” of a hurricane shows visually how
much we don’t know

More information could make a big difference
Magnitude in Risk
 Magnitude gives us an estimate of the potential size of the outcome
 Hindsight would have us avoiding or preventing all catastrophes
(not possible!)
 Very low probability, high magnitude events (Black Swan
events) shape our world
Hazard-Loss Estimation
 Any serious evaluation of risk takes into account the following four
factors:
 1. Hazard
 2. Inventory
 3. Vulnerability
 4. Loss
 A hazard’s potential impact on inventory is the inherent vulnerability
 The vulnerability of the inventory gives a sense of the magnitude of
loss
Active Advancement of Risk Methods

JPL – for Optimal Mars Entry, Descent, and
Landing

Optimal sequential decisions within a specified
risk bound

Not standard dynamic programming (achieves
risk aversion by arbitrary penalties on failure
states)

Instead uses “risk allocation” (RC-DP) to
decompose a joint chance into a set of
individual chance constraints and distributes
risk over them.
Reference: Ono, M. and Kuwata, Y. (2013). NASA Tech Briefs. Information Technology, 37(7), 52.
9
Apr-15
Further Extraterrestrial Example

Paths over Martian terrain NO DIFFERENT
from paths through our latent risks!
.72
.38 .07
.81
.80
.40
Overall certainty diminishes
.28
.10
.45
10
Apr-15
Reference: Jet Propulsion Laboratory. (2013). Covariance Analysis Tool (G-CAT) for Computing Ascent, Descent, and Landing Errors. Software Tech Briefs.
11
Apr-15
Probability-Impact Matrix

First tool: Assess at a high-level with a
probability-impact matrix

Uses horizontal resolution to maximum effect

3 x 3 matrix can blur risks
12
Apr-15
OT-Matrix

This matrix layout does not overlook
opportunities in the interest of threats (which is
typical)
P
P
I
I
13
Apr-15
Ask “Why?” 7 ±2 Times to Get to Needed
Action

Second tool:
Understand the
depth with 5 Whys
14
Apr-15
Fishbone Diagram

Third tool: Assess cause-and-effect structure
with a Fishbone Diagram
15
Apr-15
Contradiction Matrix (Is/Is Not Analysis)

Fourth tool: Disambiguate with a Contradiction Matrix
16
Apr-15
FMEA

Fifth tool: Prioritize risks with
Failure Modes and Effects
Analysis

Caveat: It’s not a tool for
every risk assessment –
beware the Concreteness
Fallacy

Sometimes with many factors
and multiplicative scores, a
weighted decision matrix
(e.g., Pugh) works much
better
17
Apr-15
Fallacies and Pitfalls

The Manager’s Fallacy

The Buyer’s Fallacy

The Fallacy of Control
18
Apr-15
Risk Perception
 How do people perceive risk?
 Not very logically, it turns out
 Unfamiliar technologies with catastrophic potential (e.g., nuclear
power) cause people to way overestimate risk
Reference: Wordpress. (2009). UAE gets green light for nuclear power. Retrieved 13Jan10 from: http://seeker401.files.wordpress.com/2009/05/nuclear10b.jpg
Risk Reality
 This misperception takes place at the expense of likelihood data
 In a study where a group was faced with hypothetical managerial
decisions, fewer than 1 in 4 asked for probabilistic information, none
sought precise likelihood data
 In a group presented with precise likelihood data, fewer than 1 in 5
drew upon probabilistic concepts when making choices about
alternate courses of action
Decide with evidence
Thinking Longer-Term
 Consider this scenario:
 A company is considering flood insurance for the 25-year life of
a production facility
 Managers are more likely to take seriously the risk of a 1-in-100
year flood if it’s presented as having greater than a 1-in-5 chance
of occurring in a 25-year period, rather than a 1-in-100 chance of
it occurring during the coming year
Using Evidence for Better Decisions
 Homeowners in California purchased earthquake insurance most
often after personally experiencing an earthquake, even though most
responded (correctly) that the likelihood of another quake was lower
now that stress on the geologic fault had been reduced
 People generally dismiss low probability events unless they
personally experience them
Reference: NASA. (2009). Earth observatory. Retrieved 13Jan10 from: http://earthobservatory.nasa.gov/Newsroom/NasaNews/ReleaseImages/20041004/04_LOMA-Prieta2.jpg
9-11 Aftermath
 Research indicates the perception of risk has an enormous impact
on behavior
 After 9-11, how many people drove to avoid death by airliner who
then died by automobile accident?
Neglecting Probabilities
 The answer, it turns out, is a sobering 725 more people died than
normal in the three months following 9-11 (October, November, and
December) by increased driving habits
Reference: Blalock, G., Kadiyali, V., Simon, D.H. (2005). The impact of 9/11 on driving fatalities: The other lives lost to terrorism. Ithaca, NY: Cornell University.
What 9-11 Did to Our Perception of Risk
 The risk of suffering a fatal airline event is 1 in 9.4 billion miles
traveled
 The risk of suffering a fatal automobile event is 1 in 70.4 million
miles traveled
 An individual is thus 133 times more likely per mile traveled to have
a fatal event driving versus flying
Risk of Flying vs. Driving
Fatalities/Hundred Mio Passenger Miles
1.60
1.420
1.40
1.20
1.00
0.80
0.60
0.40
0.20
0.011
0.00
Flying
References: NHTSA. (2009).
Driving
Fatality analysis reporting system. Retrieved from: http://www-fars.nhtsa.dot.gov/Main/index.aspx.
Sivak, M., Flannagan, M. (2002). Flying and driving after the September 11 attacks. UMTRI Research Review, 33(3). Ann Arbor, MI: University of Michigan.
Case Study – Unpublished Data

Sherlock holmes movie video clip
http://www.youtube.com/watch?v=9b3KM2p1nHs
26
Apr-15
Uncertainty Distributions

Much like the beta-distribution (why?)

Uncertainty distributions (especially with small
sample sizes) tend to be triangular or
rectangular (uniform) distributions (all values
equally likely – rolling a non-loaded die
27
Apr-15
Monte Carlo – Probability Distribution
In-Practice

Monte Carlo integrates risk and uncertainty into
schedule, is well-validated
28
Apr-15
Understanding Impact of Risk and
Uncertainty on Project Schedule
29
Apr-15
Option Analysis – From Passive to Active

This could be patient recruitment or new hire
recruitment
30
Apr-15
Option Analysis – From Passive to Active
When to visit Chicago?
Fitted Line Plot
Residual Plots for Temp
Normal Probability Plot
99
30
80
50
70
10
-25
0
Residual
Histogram
Frequency
3
25
2
60
-10
0
Residual
10
40
44
48
52
Fitted Value
56
Versus Order
40
S
R-Sq
R-Sq(adj)
5.26148
3.78088
93.6%
97.1%
92.2%
96.0%
18.9477
8.3%
0.0%
60
50
40
30
15
20
-20
-15
50
30
-30
S
R-Sq
R-Sq(adj)
70
0
50
1
0
80
-30
Residual
-50
Temp
1
Temp = 39.29 + 1.507 C3
15
Residual
Percent
90
Fitted Line Plot
Temp
Temp== -6.020
8.705++10.69
22.08C3
C3
Versus Fits
+ 0.5214 -C3**2
1.582 C3**2
- 0.1079 C3**3
Temp

30
0
-15
20
0
-30
20
1
10
10
0
2
2
3
4 5 6 7 8 9
Observation Order
4
2
10 11 12
6
Month
8
Reference: Vickers, A. (2010). What is a p-value anyway? Boston: Addison-Wesley, p. 78.
10
4
6
Month
8
10
12
12
31
Apr-15
Uncertainty – How Much Don’t We Know?

“It is also true that for extremely rare events,
correct uncertainty estimates may lead us to
conclude that we know virtually nothing. This is
not such a bad thing. If we really know nothing,
we should say so!”
Summary for BPI data
A nderson-Darling N ormality Test
5
10
15
20
25
30
A -S quared
P -V alue
1.08
0.005
M ean
S tDev
V ariance
S kew ness
Kurtosis
N
14.893
6.984
48.776
1.57659
2.57563
14
M inimum
1st Q uartile
M edian
3rd Q uartile
M aximum
35
6.000
10.750
13.000
17.500
33.000
95% C onfidence Interv al for M ean
10.860
18.925
95% C onfidence Interv al for M edian
10.949
16.309
95% C onfidence Interv al for S tDev
9 5 % C onfidence Inter vals
5.063
11.252
Mean
Median
10
12
14
16
18
20
Reference: Locwin, B. (2013). Quality Risk Assessment and Management Strategies for BioPharmaceuticals. BioProcess International. Forthcoming. 32
Apr-15
Uncertainty – How Much Don’t We Know?

Risk contour plot

Concreteness Fallacy
SPC Slight Risk Climatology. Retrieved from: www.pmarshwx.com
Relative risks?
33
Apr-15
Uncertainty – How Much Don’t We Know?

Study 1: Participants completed a health screening for Type 2
diabetes, took a survey either before or after choosing to learn
their risk. Fewer in the contemplation group avoided learning
their risks (χ2(1, N = 146) = 5.57, p < .02, Φ = .20.

Study 2: Participants completed a health screening for
cardiovascular risk (CVD), completed another contemplation
survey (or not). Fewer in the contemplation group avoided
learning their risks (χ2(1, N = 130) = 10.05, p < .01, Φ = .28.)

Study 3: Participants learned about a fictitious disease (thioamine
acetlyase (TAA) deficiency), the groups then either learned that
the disease was treatable (by a pill) or not treatable. Completed
another contemplation survey (or not). For those who learned
TAA was treatable, fewer participants avoided learning their risks
in the contemplation condition (χ2(1, N = 78) = 9.02, p = .003, Φ
= .34). For those who learned TAA was untreatable, there was no
difference between the two conditions (χ2(1, N = 80) = 1.75, p =
.19, Φ = .14).
Reference: Howell, J. L. & Sheppard, J.A. (2013). Reducing Health-Information Avoidance Through Contemplation. Psychological Science, X(X), 1-8. 34
Apr-15
Uncertainty – How Much Don’t We Know?

Reports that say that something hasn't
happened are always interesting to me,
because as we know, there are known knowns;
there are things we know we know. We also
know there are known unknowns; that is to say
we know there are some things we do not
know. But there are also unknown unknowns -the ones we don't know we don't know. And if
one looks throughout the history of our country
and other free countries, it is the latter category
that tend to be the difficult ones.

Donald Rumsfeld, United States Secretary of Defense,
2001-2006. Department of Defense briefing, 12 February
2002
35
Apr-15