Market Modelers
Forecasting and valuation for the life sciences
Monte Carlo Simulations and Executive Decision Making
The implied ethical demands of building simulation models and reporting findings
Robert Ameo, PhD
Initially presented - October 22, 2009
Palisade User Conference, NJ/NYC
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
1
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
2
“Since the great meltdown of 2008” people like us who develop and
report quantitative models to assess and manage uncertainty (i.e., risk)
have been popular subjects for news articles.
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
3
“Beware of geeks bearing formulas”
Warren Buffett
The “quants” of Wall Street were classified as either the dumbest smart people
on the planet, or worst of questionable character. Mercenary math geeks with no
Gordon Gecko panache.
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
4
and therefore never send to know for whom the bell tolls;
it tolls for thee. . . .
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
5
The essence, the basic nature of a Monte Carlo simulation requires a
higher degree of professional diligence from the analyst than scenario
models. An incomplete or poorly constructed simulation can result in
serious undesirable consequences for companies, stockholders, and the
executives who use their output for decision making.
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
6
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
7
The question is simply, “What is the one and only reason to create and
run a Monte Carlo simulation?”
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
8
A Monte Carlo simulation is simply the best and only way to answer the
question “How likely is it that X outcome will be met or exceeded in the
future?”
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
9
The goal of a Monte Carlo is not to produce an mean or other point
estimate of an outcome, but rather to describe the range and relative
frequency of all outcomes.
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
-1,000 -600
-200
Copyright 2009, Market Modelers, LLC
200
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
600
1,000 1,400 1,800 2,200 2,600 3,000 3,400 3,800 4,200
10
Definition of Probability
The theory of chance consists in reducing all
the events of the same kind to a certain
number of cases equally possible, that is to
say, to such as we may be equally undecided
about in regard to their existence, and in
determining the number of cases favorable
to the event whose probability is sought. The
ratio of this number to that of all the cases
possible is the measure of this probability,
which is thus simply a fraction whose
numerator is the number of favorable cases
and whose denominator is the number of all
the cases possible.
P(X)
=
s
me
outco
oX
t
e
l
rab
es
Favo
m
o
c
t
e ou
l
b
i
s
os
All p
– Pierre-Simon Laplace, A Philosophical Essay on Probabilities
Laplace, P. S., 1814, English edition 1951, A Philosophical Essay on
Probabilities, New York: Dover Publications Inc.
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
11
Let me make sure I understand. You want
your median case to be 90% likely to be
achieved?
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
12
The big difference between scenario modeling and simulation modeling
•
Aside from outright errors, scenarios can’t be wrong. They are simply if X then Y analyzes.
The modeler has plausible deniability, “they told me these were their most likely inputs”.
•
In a simulation, inputs are integral to the model. Building a simulation on unreliable, biased,
or just plain WAGS is a silly waste of time, energy and intellect. It might be enjoyable, like
solving Rubic’s cube or playing a video game on your iPhone, but it is about as
meaningless.
•
Your results are making a statement of fact about the probability of something occurring or
not. In the long run heads should come up half the time.
•
If any part of your simulation design, inputs, execution, or interpretation is flawed, then you
will be wrong in your probability statement. Like most complex systems MCS are
unforgiving of errors.
•
The implications of these things keeps me humble with each simulation I build
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
13
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
14
Simulations are statistical and construct experiments. Like all
experiments, to be valid (truthful) they need to be reliable (repeatable)
and to manage threats to internal and external validity
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
15
How to build a automated graph wiggler & scenario generator
•
Take a ‘base case’ and Monte Carlo it.
•
Pick your key variables, the rest aren’t important.
•
Don’t worry too much about fitting the uncertainty to a distribution. PERTs are
great.
•
Correlations who needs them?
•
Out of bounds - modeling the impossible future
•
Quality assurance – oh yeah, I checked it and the numbers look good to me.
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
16
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
17
Monte Carlo as a verb “…he took the team’s base case model and
monte carlo’d it the night before the presentation”
•A standard deterministic model is unlikely to be constructed to be fully
parametized. That is, to have all output lines driven from formulas that accept
variables as inputs. This includes time variables as many key calculations like
technology adoption, disruption, market share, pricing changes all are sensitive to
temporal uncertainties like launch dates.
•How are the stakeholders going to react when you tell them that their ‘most likely
case’ is highly unlikely to occur?
•Good elicitation practice would avoid having the experts first give their most likely
estimate as it is more than likely to anchor them and restrict the range of
uncertainty.
Parameterization is the process of deciding and defining the parameters necessary for a complete or relevant specification of a model
or geometric object.
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
18
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
19
A well know pharmaceutical consulting firm produced a standard Monte
Carlo model for a mega-client that organized annual sales uncertainty as
independent events.
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
20
A little unusual, uncertainty not increasing with time as would be
expected, but it might pass until …
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
21
…one look at a few specific iterations were it is obvious that a timeseries has been treated as independent annual events. Next year’s sales
have no connection to this year’s.
300
250
200
150
100
50
0
2010
2011
2012
Copyright 2009, Market Modelers, LLC
2013
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
2014
2015
2016
2017
2018
2019
2020
22
Correlations, why bother?
1/99:10/90 view of 5 related adoptions without and with a 75% coefficient of determination
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
23
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
24
Most “correlations” are dependencies unnamed. For example the relationship
between U.S. and EU adoption of a new technology can easily be assessed as a
shared dependency on the technology’s attractiveness. The relationship can be
created as an adjustable fuzzy one.
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
25
Completeness is the quality of a simulation that refers to the degree to
which the inputs encompass all uncertainties driving the outputs. There
are two different type of incompleteness: omission and hybridization.
Input parameters in an incomplete simulation
Uncertain
Deterministic
Interpretation of simulated outcomes?
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
26
Average annual growth rates are often delegated to ‘minor’ variable status and
left as a point estimate in simulation models. Even leaving the uncertainty in one
of these ‘minor’ variables unspecified can dramatically truncate the range of
possible outcomes.
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
27
Even a ‘small’ variable like an annual average growth rate applied to a
sensitive variable can have a dramatic effect on the range of possible
outcomes
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
28
From @Risk Manual, Chapter 2 – Section on Interpreting the Results – a
definitive stance
•
In an @RISK Risk Analysis, the output probability distributions give the decisionmaker a complete picture of all the possible outcomes…
•
…because you have more rigorously defined the uncertainty associated
with every input variable…
•
… A probability distribution shows the relative likelihood of occurrence for each
possible outcome.
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
29
More than one author feels differently on the subject.
“The first step in any simulation (after building a deterministic spreadsheet) is to
select parameters to treat as uncertain. …economy is called for in selecting
uncertain parameters.” - Stephen G. Powell & Kenneth R. Baker authors of The Art of Modeling with Spreadsheets, 2004, Wiley.
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
30
Granularity is the degree to which the input variables have been broken down into
their predicate components. Granularity can be manipulated to control the amount
of time it takes to construct a simulation at the cost of validity.
Pre-clinical
research
Begin study 1
Time until last
patient follow-up
complete
Product launch date
Clinical studies
Time to lock database
Time to prepare FDA
submission
Regulatory review
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
FDA submission date
31
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
32
The motivation for the development of Monte Carlo Simulations was to
model systems that were too complex for algorithmic treatment.
Physicists at Los Alamos Scientific Laboratory were investigating radiation shielding
and the distance that neutrons would likely travel through various materials. Despite
having most of the necessary data, such as the average distance a neutron would
travel in a substance before it collided with an atomic nucleus or how much energy
the neutron was likely to give off following a collision, the problem could not be
solved with theoretical calculations.
John von Neumann and Stanislaw Ulam suggested that the problem be solved by
modeling the experiment on a computer using chance. Being secret, their work
required a code name. Von Neumann chose the name "Monte Carlo". The name is a
reference to the Monte Carlo Casino in Monaco where Ulam's uncle would borrow
money to gamble.[4][5][6]
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
33
Even a small change in how we record a distribution of an uncertain
input can impact the outcome. Here four variables are multiplied using
either a Pert P5/P95 or a Pert Min/Max. The change to a the 5/95 inputs
doubles the probability of failure (NPV < 0) and misses the multi-billion
dollar upside.
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
34
Pert distributions
Pert 5/95 distributions
P(Gain) = 89%
Mean Gain = $991
Mean Loss = $178
P(Gain) = 79%
Mean Gain = $1,575M
Mean Loss = $309M
Return on Monies at Risk (ROMR)
ROMR Pert = 46 / 1
ROMR Pert 5/95 = 19 / 1
Interpretation – Modeling with PERTs, when actual
judgments were on a 5/95 basis, doubled the
expected return on monies at risk.
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
35
How many of your organizations require independent QA of decision
critical spreadsheets?
•
Coopers and Lybrand in England, which found that 90% of all spreadsheets with
more than 150 rows that it audited contained errors. One Price-Waterhouse
consultant audited four large spreadsheets and found 128 errors (Ditlea, 1987).
•
Assume a 5% human error rate for complex calculations
What does your organization do when a spreadsheet error is detected
AFTER the numbers have been presented to senior management?
http://panko.shidler.hawaii.edu/SSR/Mypapers/whatknow.htm
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
36
200 hours of modeling, a million dollars of
multi-national quantitative research, a panel of
50 bona fide experts including three Nobel
laureates, a full-time team of a dozen top
people, and you don't like the answer because
"it seems a little light".
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
37
Who are we? Soothsayers? Forecasters? Simulationists? Should we have
standards? Ethical codes? Will people have faith in simulations if so many
are done so poorly?
I would like to engage each and everyone of you to have a conversation
about the professionalization of simulation.
As I said when I started today, I believe that simulation is one of the
most powerful tools we have to navigate uncertain futures. It and its
practitioners (i.e., us) are in danger of disrepute.
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
38
Market Modelers
Forecasting and valuation for the life sciences
www.marketmodelers.com
908 279-7925
Copyright 2009, Market Modelers, LLC
Includes 3rd party licensed images, DO NOT COPY or DISTRIBUTE without permission
39