Simulation Forecast Models Using @Risk
A focus on Market Models for Product Forecasts
Copyright 2007 Applied Quantitative Sciences, Inc.
About AQS
• Applied Quantitative Sciences, Inc. is a service
organization that specializes in assisting clients
understand and navigate the process of making
important decisions in the context of risk and uncertainty.
• Our Healthcare Division has as its clients global market
leaders in:
–
–
–
–
Pharmaceuticals
Biotechnology
Medical device
Healthcare delivery systems
• Corporate office located in Broward County, FL
AQS Healthcare Division Clients
Agenda
• Why forecast?
• What is simulation forecasting and how is it different from
traditional forecasting methods
• Basic truths about forecasting
• Roles and responsibilities within the forecasting team
• Components of a representative forecast
• Managing validity, reliability and credibility of the forecast
• Pitfalls to avoid when reporting outputs
• Tools for simplifying the task of simulation forecasting
4
A lot to cover....
• I’ll try to be both informative and brief
5
Why Forecast?
• You simply cannot plan without it
• Virtually every function in a company must plan
–
–
–
–
–
–
Business development
Sales
Finance
R&D
Manufacturing and production
Supply chain
6
Just because you have a forecast...
7
Doesn’t make it useful
8
Traditional Method
•
Characteristics:
– Inputs are single point
estimates
– The model relates the
inputs mathematically
– Outputs are
deterministic and
represent a single
scenario
•
Can answer questions
such as “If things happen
the way we hope, we can
expect…”
Monte Carlo Method
•
Characteristics:
– Inputs are distributions
of uncertainty
– The model relates the
inputs mathematically
– Outputs are
representative
distributions of what is
possible, with
likelihoods able to be
calculated for any value
•
Can answer questions
such as “What is the
likelihood of achieving…”
Why Use Simulation Forecasting
11
“Any realistic model of a real-world phenomena must
take into account the possibility of randomness.”
--- Sheldon M. Ross
12
Some Truths About Forecasting
•
•
•
•
•
No-one can predict the future
The myth of accuracy
You cannot effectively plan without it
Few do it well
In many large corporations, various functions use
different forecasts for presumably the same outcomes
13
No-one can predict the future: forecasting is
not fortune telling
14
When one tries to pretend it is, they look
more like this...
15
And, when actuals come in...
16
Accuracy vs. Representativeness
• Accuracy refers to how well a model predicts actuals
– Within sample (to establish the model)
– out of sample (to validate model)
• Accuracy is irrelevant when discussing simulation
forecasts
– A well designed simulation model will incorporate all potential
futures, and is therefore always “accurate”
– The precision of the forecast is determined by the level of
uncertainty surrounding inputs
– The purpose of a simulation forecast is to provide insight into the
range of potential futures and inform decisions based on the
likelihood of achieving any level within that range
– Simulation embraces uncertainty
17
However, validity and reliability is critical
• Predicated on
–
–
–
–
A sound, reproducible forecasting process
Appropriate model specification
Appropriate assumption sources
Testing
18
The AQS Modeling Process
19
AQS Modeling Standards
•
•
•
•
•
•
•
•
Transparency
– Assumptions
– Structure
Modularity
– Increases structural flexibility
– Models are more maintainable
– Lower total cost of ownership
Documentation
– All assumptions are documented for source, rationale and date
Scaleability
Adaptivity to evolving business requirements
Portability of results
– Designed to provide specific decision support with explicit levels of confidence
– Able to integrate with existing enterprise applications
Quality Assurance (Every model is reviewed by an independent modeler, with signoff / certification
by the individual performing QA)
– Model structure
– Assumption documentation
– Code integrity (every line of code is reviewed for mathematical and referential integrity)
Version Control and Traceability
20
Roles and Responsibilities of the
Forecasting Team
• Sponsor: The person who defines the scope of the
project and who will be the primary consumer of the
outputs. Responsible for engaging the appropriate
resources to establish a valid and reliable forecast
relevant to the business domain(s) of interest.
• Forecaster: Takes a leadership role in project
management, identifying appropriate sources of inputs,
and managing potential conflicts of interest or bias
• Modeler: Translates information about market dynamics
(whether verbal or otherwise) into mathematical
representations that produce consistent with what one
would expect in the real world
21
Often, more than one role is assumed by a
single individual
22
Components of a Representative Forecast
•
•
•
•
•
•
•
•
Segmentation
Key events
Target population
Exclusions
Target pool (Target population - Exclusions)
Technology adoption
Market share
Average Selling Price
23
A Model Framework
Revenue = Target Pool * Adoption * Market Share * Price
Disruptive Risk
Instability in
population
Uncertainty
Disruptive Risk
Emerging
technologies
Healthcare
economics
Study data
Correlations
Disruptive Risk
Disruptive Risk
Time of entry
Relative advantage
Contracting &
Distribution
Relative sales force
size/tenure
Study data
Disruption to
COGS
Competitive
pressure
The fascinating impressiveness of rigorous mathematical analysis, with its atmosphere of
precision and elegance, should not blind us to the defects of the premises that condition the
whole process. There is perhaps no beguilement more insidious and dangerous than an
elaborate and elegant mathematical process built upon unfortified premises. - T.C. Chamberlain
Forecast Inputs
•
•
•
•
•
In many strategic forecasts, there
are no data from which to
extrapolate
Garbage in; garbage out
Identify who should (and who
should not) own inputs
Perform formal assumption
elicitation interviews
Revisit as new data / information
are available
26
Incorporating Risk and Uncertainty
• None of us know the future
• Many of the variables of interest for the model represent
uncertain future events.
• Someone, whether implicitly or explicitly, will make
assumptions and decisions based on these uncertain
quantities.
• It is preferable to have the most qualified individual make
explicit the assumptions that drive the model, so that
they may be documented, understood, discussed and
(when more current knowledge is acquired) updated.
Even a good forecast may not be credible
• Simulation forecast models can incorporate a significant
amount of complexity
• No matter how complex the model becomes, consumers
will not trust a “black box”
• It is critically important to communicate modeled
dynamics and outputs in an effective, actionable fashion
– Making a good bet
– Risk analysis and mitigation
28
Without Making Their Heads Explode
Benefits Beyond the Numbers
• Assumption transparency, traceability and updateability
• Explicit probability revealed for continuum of potential
values
• Consistency
• Rigor
• Credibility
• Reproducibility
• Provides a framework for valid and reliable management
of opportunity portfolio
30
Why is this not routinely done today?
• It is hard work
– Perceived lack of tools to simplify
• No common language
• The desire for “the number” (elective ignorance of
uncertainty)
• Corporate innumeracy
31
Commercial Simulation Engines Make
Probabilistic Long-Term Forecasting
Accessible to Any Organization
32
Sophisticated Business Models
Made Easy
•
A business-modeling tool that dramatically simplifies the task of building
sophisticated, complex business models, with or without Monte Carlo
Simulation. AQS Model Builder instantly creates model components for:
–
–
–
–
–
–
–
–
–
–
•
Technology adoption
Market share
Probability and timing of product launch
Probability and timing of key market events
Disruption
Dynamic transitions from one value to another
Dynamic transitions by a percentage of the base value
Replacement/Replenishment model
Dynamic pricing models
And MORE!
Cut the time you spend building models to a FRACTION of what it takes
today.
Decide With Confidence
Forecasting
Simulation
Optimization
Statistical Analysis
•Portfolio Management
•Strategic Planning
•Supply Chain Management
•Resource Planning
•Pharmacoeconomic Analysis
•Risk Management
Applied Quantitative Sciences, Inc.
649 San Remo Drive
Weston, FL 33326
561-433-8206
www.aqs-us.com
Thank you.