Sensitivity analysis
Marko Tainio
Decision analysis and Risk Management
course in Kuopio
21.3.2011
Sensitivity analysis studies on what
happens inside the Black Box
Data
Black Box
(the model)
Results
Outline of lecture
• What is sensitivity analysis?
– Why to use sensitivity analysis
– What options there are?
• Example of two sensitivity analysis methods
– Nominal Range Sensitivity
– Rank-order Correlation
• Other sensitivity analysis methods
What is sensitivity analysis
Definition of sensitivity analysis
• http://en.wikipedia.org/wiki/Sensitivity_analysis
• Sensitivity analysis (SA) is the study of how the
variation (uncertainty) in the output of a
mathematical model can be apportioned,
qualitatively or quantitatively, to different
sources of variation in the input of the model.
• Put another way, it is a technique for
systematically changing parameters in a model
to determine the effects of such changes.
Main idea of SA
• In sensitivity analysis you change the input
parameters to see how the model results
response to these changes
• Thus, sensitivity analysis resembles
laboratory research where you control input
and measure the outcome
– Same statistical methods are applied in laboratory
studies and in sensitivity studies (correlation,
regression analysis, ANOVA)!
Different sensitivity analysis methods
Frey and Patil, 2002 divides sensitivity analyses
to three broad categories:
1. Mathematical
•
Suitable for deterministic models
2. Statistical (or probabilistic)
•
Usually based on simulation and statistical
parameters.
3. Graphical
•
Presenting of sensitivity with graphs, charts etc.
When to use sensitivity analysis
• In simple, always when making risk or decision
models!
• Two main advantages:
– You can guide your own modeling work by testing
the sensitivity of the model while doing the
assessment;
– You can also communicate to possible users the
main uncertainties related to assessment
Calculation of sensitivity analysis:
- Nominal Range Sensitivity
- Rank-order Correlation
Setting
• How many square meters of tables we have in
this building?
• The model is simple:
– (Number of tables) x (average width) x (average
height) = n x w x h
• Since we don’t know any of these parameters,
we assume some distributions for them
Parameter
Number of tables (#)
Height (m)
Width (m)
Best guess
150
1
1,5
Min
50
0,5
1
Max
300
1,2
2
Nominal Range Sensitivity Analysis
Method
• NRSA is used to evaluate the effect on model
outputs of varying only one of the model
inputs across its entire range of plausible
values, while holding all other inputs at their
nominal or base-case values
• Equation:
Page 14, Frey 2
NRSA sensitivity analysis
Parameter
Number of tables (#)
Height (m)
Width (m)
Result (all)
Results (number of tables)
Results (height)
Results (width)
Nominal input
150
1
1,5
Nominal output
225
225
225
225
Min input
50
0,5
1
Max input
300
1,2
2
Min output Max output
25
720
75
450
113
270
150
300
NRSA
1,7
0,7
0,7
Model is most sensitive to Number of tables parameter.
Qualities of NRSA analysis
• Advantages:
– Works with deterministic models (no need for Monte
Carlo)
– Easy to use and apply in number of models
• Disadvantages:
– Works only with linear models
– Doesn’t take into account interactions/correlations
between input parameters
• NRSA is a good screening level sensitivity analysis
tool
Sample and Rank Correlation
Number of tables
Coefficients
50
150
300
Height
Result
Model (aka. Black Box)
0.5
1.0
1.2
104
Width
1
1.5
2.0
214
384
Sample and Rank Correlation
Coefficients
Two options for correlation analysis:
1. Parametric or Pearson
– For linear models
2. Non-parametric or Spearman or rank
– Also for non-linear models
– Importance analysis
• Correlation varies between -1 and 1
– The value of -1 represents a perfect negative
correlation while a value of +1 represents a perfect
positive correlation
In Monte Carlo, correlation is
calculated between samples
Sample
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Number of tables
101,732485
232,0156268
131,4324874
209,13953
275,3905506
209,9895839
229,2318928
229,3910416
281,1751494
240,1094749
139,4776508
165,4660823
143,5213879
115,9071316
249,5953871
155,7281212
198,3111363
74,6221445
184,3685813
98,38646505
Height
0,852764085
0,78158924
0,844481494
0,855728127
0,994355641
1,052050684
0,569119462
1,112370096
0,634582688
0,814638682
0,748580973
0,85503873
0,827288099
1,017827005
1,048682453
0,907887852
0,722974214
0,996051913
1,000092521
0,712855585
Width
1,627474833
1,611766823
1,526740029
1,494292424
1,50905703
1,337379608
1,154838626
1,363145976
1,264338798
1,685794653
1,144135353
1,579910724
1,803340396
1,740529385
1,90421378
1,51237822
1,061032778
1,622607631
1,68074305
1,459809743
Correlation
Sample
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Results
141,1896
292,2793
169,4564
267,4284
413,2344
295,4535
150,6606
347,8309
225,5946
329,7456
119,4595
223,5256
214,117
205,3362
498,4209
213,8256
152,1243
120,6044
309,9049
102,3843
Result of the rank-order correlation
sensitivity analysis
The uncertainty in the results correlates 80% with the uncertainty of „Number
of tables” parameter.
Qualities of correlation sensitivity
analysis
• Advantage:
– Easy to compute
– Correlation available in most of the computer
modeling tools (including Excel)
• Disadvantage:
– Correlation is not causation
– Non-linear and non-monotonic models are
problematic
Other sensitivity analysis
methods
Research on sensitivity analysis
• Frey et al. 2003 (Evaluation of Selected
Sensitivity Analysis Methods Based Upon
Applications to Two Food Safety Process Risk
Models) lists 11 different sensitivity analysis
methods
• They also made recommendations on which
sensitivity analysis to use in which situation
• Report available:
http://www.ce.ncsu.edu/risk/Phase2Final.pdf
Different sensitivity analysis methods
• Mathematical Methods for Sensitivity Analysis
– Nominal Range Sensitivity Analysis Method
– Differential Sensitivity Analysis (DSA)
• Statistical Methods for Sensitivity Analysis
–
–
–
–
–
–
–
Sample and Rank Correlation Coefficients
Regression Analysis
Rank Regression
Analysis of Variance
Classification and Regression Tree
Sobol’s Indices
Fourier Amplitude Sensitivity Test (FAST)
• Graphical Methods for Sensitivity Analysis
– Scatter Plots
– Conditional Sensitivity Analysis
Selection of the sensitivity analysis
(Frey et al. 2004)
• Some selection criteria's:
– What are the objectives of sensitivity analysis?
– Based upon the objectives, what information is needed from
sensitivity analysis?
– What are the characteristics of the model that constrain or
indicate preference regarding method selection?
– How detailed is the analysis?
– What are the characteristics of the software that may constrain
selection of methods?
– What are the specifications of the computing resources?
– Can “push-button” methods adequately address characteristics
of interest in the analysis?
– Is the implementation of the selected sensitivity analysis
method post-hoc?
Frey et al. pages 46-47, http://www.ce.ncsu.edu/risk/Phase3Final.pdf
Some objectives of sensitivity analysis
• Rank ordering the importance of model
inputs (e.g., critical control points);
• Identifying combination of input values that
contribute to high exposure and/or risk
scenarios;
• Identifying and prioritizing key sources of
variability and uncertainty;
• Identifying critical limits;
• Evaluating the validity of the model.
What Information is Needed from
Sensitivity Analysis?
• Qualitative or quantitative ranking of inputs
• Discrimination of the importance among different
inputs
• Grouping of inputs that are of comparable
importance
• Identification of inputs that are not important
• Identification of critical limits
• Identification of inputs and ranges that produce
high exposure or risk
• Identification of trends in the model response
Frey et al. Pages 58, http://www.ce.ncsu.edu/risk/Phase3Final.pdf
Further reading
• Frey et al. 2004. Recommended Practice Regarding
Selection, Application, and Interpretation of Sensitivity
Analysis Methods Applied to Food Safety Process Risk
Models: http://www.ce.ncsu.edu/risk/Phase3Final.pdf
• Frey et al. 2003. Evaluation of Selected Sensitivity
Analysis Methods Based Upon Applications to Two
Food Safety Process Risk Models:
http://www.ce.ncsu.edu/risk/Phase2Final.pdf
• Patil and Frey 2004. Comparison of sensitivity analysis
methods based on applications to a food safety risk
assessment model. Risk Analysis 24 (3): 573-585