Chapter 6
Statistical Analysis of Output from
Terminating Simulations
Statistical Analysis of Output from
Terminating Simulations
• Random input leads to random output (RIRO)
• Run a simulation (once) — what does it mean?
– Was this run “typical” or not?
– Variability from run to run (of the same model)?
• Need statistical analysis of output data
– From a single model configuration
– Compare two or more different configurations
– Search for an optimal configuration
• Statistical analysis of output is often ignored
– This is a big mistake – no idea of precision of results
– Not hard or time-consuming to do this – it just takes a little planning
and thought, then some (cheap) computer time
Simulation with Arena
Chapter 5 – Detailed Modeling and
Terminating Statistical Analysis
2
Output Analysis
Output analysis is concerned with
• Designing replications
Obtain most reliable info with minimum number of replications and
minimum run length.
• Computing statistics
Point and confidence interval estimation
Size and independency issues
• Presenting them in a textual and graphical
format.
Aim is to understand the system behavior and
generate predictions for it!
Simulation with Arena
Chapter 5 – Detailed Modeling and
Terminating Statistical Analysis
3
Time Frame of Simulations
• Terminating: Specific starting, stopping conditions
– Run length will be well-defined (and finite)
• Steady-state: Long-run (technically forever)
– Theoretically, initial conditions don’t matter (but
practically they usually do)
– Not clear how to terminate a simulation run
• This is really a question of intent of the study
• Has major impact on how output analysis is done
• Sometimes it’s not clear which is appropriate
Simulation with Arena
Chapter 5 – Detailed Modeling and
Terminating Statistical Analysis
4
Model 6.1
• Same as Model 5.3
Number of trunk lines=26
No additional staff during 5-8 hrs.
• 10 runs are made
For terminating case, make IID replications
Run>Setup>Replication Parameters:
Number of Replications =10
Check both boxes for Initialize Between Replications
• Outputs are saved to .dat files
Statistics Module, Type=output,
Data file name= Filename.dat
Asli Sencer
5
Outputs of Model 6.1
• Category Overview report will have some statisticalanalysis results of the output across the replications
Asli Sencer
6
Output Precision in Model 6.1
This information (except standard deviation) is in Category
Overview report
If > 1 replication specified, Arena uses cross-replication data as above
For other confidence levels or graphics – Output Analyzer
Asli Sencer
7
Confidence Interval Estimation
Notation:
𝑛 =number of replications
𝑥 =sample mean
𝑠 =sample standard deviation
𝑡𝑛−1,1−∝/2 =critical value for ∝ significance level
(1-α)x100% Confidence Interval (CI):
for n < 30, 𝑥 ± 𝑡𝑛−1,1−∝/2
for n ≥ 30, 𝑥 ± 𝑧∝/2
𝑠
𝑛
𝑠
𝑛
8
Interpretation of a Confidence Interval
• A CI is an interval with random (data-dependent) endpoints
that’s supposed to have stated probability (1-α) of
containing, or covering, the expected values
– “Target” expected value is a fixed, but unknown, number
– Expected value = average of infinite number of replications
• A CI is not an interval that contains, say, 95% of the data
generated by simulation outputs.
– That’s a prediction interval … useful too, but different
• 95% CI means:
If we make infinitely many batch runs of size 𝑛 and
construct C.I. for α=5%, 95% of the time this interval will
contain the true (but unknown) population parameter.
Simulation with Arena
Chapter 5 – Detailed Modeling and
Terminating Statistical Analysis
9
Required Number of Replications to
Achieve a Certain Precision
• Half-width =
𝑠
𝑡𝑛−1,1−∝/2
𝑛
Want this to be “small,” say
< h where h is prespecified
• We prefer confidence intervals with smaller half width — for
higher precision
• Question: How many replications are needed to get the
required (better) precision?
We can’t control t or s. We may just increase n — how much?
– Trial and error (now)
– Approximate number for acceptable precision (below)
– Sequential sampling (Chapter 11)
Simulation with Arena
Chapter 5 – Detailed Modeling and
Terminating Statistical Analysis
10
Half Width and Number of
Replications
2
𝑠
• Let ℎ =required half-width, solve for 𝑛 = 𝑡 2 𝑛−1,1−∝/2 2
ℎ
• Not really solved for n (t, s depend on n)
• Approximation:
• Replace t by z, corresponding normal critical value
– Pretend that current s will hold for larger samples
𝑠2
– Get 𝑛 ≅ 𝑧 1−∝/2 2
ℎ
2
s = sample standard
deviation from “initial”
number n0 of replications
• Easier but different approximation:
ℎ0 2
𝑛 ≅ 𝑛0 2
ℎ
Simulation with Arena
h0 = half width from
“initial” number n0 of
replications
Chapter 5 – Detailed Modeling and
Terminating Statistical Analysis
n grows
quadratically
as h
decreases.
11
Number of Replications Needed
• If we require h=$250 rather than $812 for total cost,
ℎ0 2 10 812.76
𝑛 ≅ 𝑛0 2 =
ℎ
250 2
2
= 105.7
replications are needed.
Asli Sencer
12
Model 6.2
• 110 Runs
• 95% CI for the total cost=
22,175.19 +- 369.54, half width=369.54>250!
Required accuracy is still not met, why?
As more replications are made, in ℎ =
𝑠
𝑧∝/2
𝑛
formula,
𝑧∝/2 does not change, 𝑛 increases; but 𝑠, the sample
standard deviation might increase. So ℎ might decrease.
• 95% CI for the %rejected calls=
11.73.96 +- 0.51, accuracy is quite satisfactory!
Asli Sencer
13
Model 6.3
• 1000 Runs-as a trial
• Save the output to Total cost.dat
• Open Output analyzer as a separate application
File>Data File>Export
Export binary data in .dat file to a plain ASCII text file and
save.
• Open Arena Input Analyzer
Plot the histogram of the Total Costs
Asli Sencer
14
Histogram of 1000 Total Cost Values
• Since Total Cost values is a sum, law of large numbers apply. We see
that the distribution approaches normal as the number of
replications increase!
• Same is true for average statistics due to central limit theorem.
• It is not true for extreme value statistics like maximum or minimum.
Asli Sencer
15
Confidence Intervals (cont’d)
• Usual formulas assume normally-distributed
data

Never true in simulation

Might be approximately true if output is an
average, rather than an extreme

Central limit theorem

Issues of robustness, coverage, precision –
details in book
Simulation with Arena
Chapter 5 – Detailed Modeling and
Terminating Statistical Analysis
16
Comparison of Alternatives
Statistical Hypothesis Test
H 0  1   2  0
H 1  1   2  0
Reject Ho if
X1 X
2
 i is the mean performance of system i
is significantly large or small, i.e.,
performance of system 1 is significantly different than system 2!
Here: x 1 : Total Cost of Base Model (110 observations)
x 2 : Total Cost of Alternative Model (110 observations)
Simulation with Arena
Chapter 5 – Detailed Modeling and
Terminating Statistical Analysis
17
Comparing Two Scenarios
• Base Scenario: Model 6.4 (Same as in Model 5.3)
-110 runs
-26 Trunk Lines, No New Staff between 12:00-16:00
• Alternative scenario: Model 6.4 (More-resources scenario)
-110 runs
-29 Trunk Lines, (Change the capacity from 26 to 29)
-Hire three for each of Larry, Moe, Curly, Hermann and
Sales Resources. (Change these variables from 0 to 3)
• Tradeoff is between increased salary cost but decreased excess
waiting costs. Will the total costs decrease?
• Percent Rejected calls will decrease, but how much?
Asli Sencer
18
Comparison of Scenarios
• Runs both models for 110 times.
• Statistics Data Module
Save output files –BaseCase.dat or -MoreResources.dat.
• 95% CI for total costs are
Base model: 22,175.19 +- 369.54=[21,805, 22,544]
Increased resources: 24,542.82 +- 329.11=[24,213, 24,871]
Intervals do NOT overlap, hence
Total Costs are significantly different at 5% significance level.
• 95% CI for percent rejected are
Base model: 11.74 +- 0.51=[11.23, 12.25]
Increased resources: 1.73 +- 0.31=[1.42, 2.04]
Intervals do NOT overlap, hence
Percent Rejected are significantly different at 5% significance level.
Asli Sencer
19
Arena Output Analyzer
• Separate application in Arena
• Operates in output files (.dat) generated by Arena
through the Statistics data module
• Data in .dat file is in binary format to be opened
by Arena Output Analyzer only!
• Provides confidence intervals on expected output
statistics as also appear in Arena output reports.
• Provides statistical comparison of two scenarios,
and others.
Asli Sencer
20
Comparison of Scenarios with Arena
Output Analyzer
• Open Output Analyzer
• Select File>New to open a data group, i.e., list of .dat files
• Add
TotalCost-BaseCase.dat
TotalCost-MoreResources.dat
PercentRejected-BaseCase.dat
PercentRejected-MoreResources.dat
• Can save this data group as .dgr file to refer easily
afterwards
• Analyze>Compare Means
Add each pair of comparisons by choosing ‘lumped’ so
that all 110 values are considered in the analysis
Asli Sencer
21
Hypothesis Tests
• Ho: Mean TC of base case = Mean TC of more resources
Ha: Mean TC of base case ≠ Mean TC of more resources
• Ho: Mean % rejected of base case = Mean % rejected of
more resources case
Ha: Mean % rejected of base case ≠ Mean % rejected of
more resources case
Asli Sencer
22
Output Report-Compare Means
Confidence interval on difference misses 0, so conclude that there is a (statistically)
significant difference between the base model and the alternative at α=5%
Asli Sencer
23
Evaluating Many Scenarios with
Process Analyzer
• Separate application in Arena
• Allows making multiple pairwise scenario comparisons
at a time.
• PAN operates on Arena program files with .p extension,
generated when .doe model is run.
• A PAN scenario includes a program file, a set of values
for the input controls (decision variables in the form of
variables and resources), a set of output responses.
• A PAN project is a collection of such scenarios that can
be saved by .pan extension for future reference.
Asli Sencer
24
Development of a PAN Project
•
•
•
•
•
Use Model 6.5
110 runs
Output data files are deleted since they will be
useless in PAN.
Open a PAN project
File > New, File > Open
Add a new scenario.
Double click on the raw
Name=Base Case, Program File=Model 6.5.p Replications=110
Add contols
Right click in this line OR Insert > Control
Under Resources: The capacity of trunk line
Under User Specified: New Tech 1, New Tech 2, New Tech 3, New Tech All,
New Sales
Add responses
Right click on this line OR Insert > Responses
Under user specified: Total Cost, Percent Rejected
Asli Sencer
25
New Scenarios
• Suppose you have $1360/week to spend on all additional
resources. To which of the six expandable resources should
you allocate the new money?
• Then following 6 alternative scenarios apply in addition to
Base Case.
13 more trunk lines ($98 each)
4 more tech 1,2,3 people ($320 each)
3 more tech all people ($360 each)
4 more sales people ( 340 each)
• Run the scenarios
Check the scenarios to run
Run > Go OR play button OR F5 function key
Asli Sencer
26
PAN screen
Asli Sencer
27
Generating Reports for Multiple
Comparison in PAN
• Insert > Chart OR right click on a response column.
Chart type=Box whiskers
Check Identify Best Scenarios box
Select ‘smaller is the better’
• Red boxes are significantly better than blue ones at 5%
significance level.
• To decrease the half width of a scenario, increase the number
of replications of that specific one.
• Error tolerance is a positive value that represents an amount
small enough that you don’t care if the selected scenarios are
actually inferior to the true best one by at most this amount.
A positive error reduce the number of selected scenarios at
the risk of being off by a little bit.
Asli Sencer
28
A PAN Report
Asli Sencer
29