SIMULATION MODELING AND ANALYSIS
WITH ARENA
T. Altiok and B. Melamed
Chapter 9
Output Analysis
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
1
Output Analysis Activities
• Output Analysis is the modeling stage concerned with
• simulation replications
• computing statistics from replications
• presenting statistics in textual or graphical format
• Output Analysis activities consist of the following activities:
• replication design
• estimation of performance metrics
• system analysis and experimentation
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
2
Types of Simulation Models
• Simulation models can be classified into two main classes,
based on their time horizon:
• terminating models
• steady-state models
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
3
Terminating Simulation Models
• A terminating simulation model has a natural termination time
for its replications
• the time horizon is inherent in the system and finite
• the modeler is interested in short-term system dynamics
• statistics are computed within the system’s natural time horizon
• The number of replications is a critical parameter of the
associated Output Analysis
.
• it is the only means of controlling the sample size of any given estimator
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
4
Steady-State Simulation Models
• A steady state simulation model has no natural termination time
for its replications
• the time horizon could be potentially infinite
• the modeler is interested in long-term dynamics and statistics
• While long-term statistics are of interest, initial system
conditions tend to bias its long-term statistics
• it, therefore, makes sense to start statistics collection after an initial
period of system warm-up, namely, after the biasing effect of the initial
conditions decays to insignificance
• the transient-state regime is characterized by statistics that vary as
function of time, while the steady-state regime prevails when statistics
stabilize and do not vary over time
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
5
Sample Steady-State Simulation Output
1.6
1.4
1.2
Throughput
1
0.8
0.6
0.4
Average Time in Buffer
0.2
0
1
51
101
151
201
251
301
351
401
451
Time
Workstation throughput and average number of jobs as functions of time
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
6
Steady-State Simulation Issues
• Since steady-state models have no natural termination time,
how does one select a replication length?
• the replication can be stopped when statistics at the end of several
successive increments are sufficiently close
(e.g., within some difference, to be determined by the analyst)
• Since a warm-up period is needed to eliminate statistical bias,
how does one select the warm-up length?
• in a similar vein, the length of a warm-up period is determined by
observing experimentally when the time variability of the statistics of
interest largely disappears
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
7
Statistics Collection From Replications
• Suppose we are interested in a parameter q of the system
• e.g., mean flow time or blocking probability
• The simulation will then be programmed to produce a (variate)
estimator, Q€ , which evaluates to some estimate Q€ = q€
• For example, let X j (r ) denote the random variable of flow time
through the system of job j in replication r , and let x j (r ) be a
realization of X j (r )
• if the estimator of the mean flow time is the sample mean of flow times
in each replication, then the corresponding replication estimates are
1
x (r ) =
l (r )
l (r )
еj = 1 x j (r ),
r = 1, ј , n
where l (r ) is the number of flow time observations recorded in replication r
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
8
Batch Means Statistics Collection
• The Batch Means method is a practical way of collecting
multiple estimates from a single replication
• the idea is to group observations into batches, which are iid or
approximately so, and then collect one estimate from each batch
• Let the observed history be a discrete sample {x 1, K , x n }
• the total of n observations is divided into m batches of size k each,
such that n = m k (the batch size is selected to be large enough so as
to ensure that the corresponding estimators are iid or approximately so)
• this results in the following partition into m sub-samples (batches)
{x 11, K , x 1k }, {x 21, K , x 2k }, K , {x m 1, K , x m k }
• for each batch j = 1, , m , an estimate q€ is formed from the
j
j
1
jk
observations {x , K , x } of that batch only
• the replication thus yields a set of estimates {q€ , K , q€ }
m
1
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
9
Point Estimation for
Discrete Samples
• Let a replication collect observations from a discrete sample, where
• {X 1, K , X n } is a sequence of n variates,
• {x 1, K , x n } is the corresponding sample of observations
• The estimator for the mean value is the sample mean
1
X =
n
n
еj = 1 X j
• The sample mean above is classified as a point estimator,
because it estimates a scalar
• In a similar vein, the corresponding point estimate is obtained
when the sample of observations (realizations) are substituted into
the formula for the sample mean above
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
10
Point Estimation for
Continuous Samples
• Let a replication collect observations from a continuous sample
over time, where
• {X : 0 Ј t Ј T } is a stochastic process over some time interval [0, T ]
t
t
• {x : 0 Ј t Ј T } is the corresponding sample of observations
• The estimator for the mean value is the time average
point estimate
1
X =
T
T
т0
X t dt
so called because the variates involved are indexed by time
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
11
Point Estimation for
Continuous Samples (Cont.)
• Examples of time-continuous variates in queueing context
• the number of jobs, N t , at time t in a queueing system (buffer and server)
• the server utilization process, {U t : 0 Ј t Ј T } , is defined as the indicator
variate
Ut
м
0,
п
п
= н
п
1,
п
о
if N t = 0
if N t > 0
where U = 1 when the server is busy, and U t = 0 when the server is idle
t
• the utilization statistic is the time average
1
U =
T
T
т0
U t dt
which is the fraction of time in [0, T ] that the server is busy, and as such
is an estimator of the probability that the server is busy
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
12
Point Estimation in Arena
• The Statistic module allows the user to obtain estimates for
• Tally statistics
• Time Persistent statistics
• Both Tally and Time Persistent statistics permit user access
to a number of related Arena variables, such as
• TAVG(X) = tally average of variable X
• DAVG(X) = time average of variable X
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
13
Example: Point Estimation in Arena
• Consider a workstation, subject to failure, where
• jobs arrive with exponential inter-arrival times of mean 1 hour
• jobs have a fixed processing time of 0.75 hours
• the workstation goes through up/down cycles as follows:
• while busy, it fails randomly with exponentially distributed
time-to-failure of mean 20 hours
• on failure, repair times are uniformly distributed between 1 and 5 hours
• Based on analytical calculations, we expect
• the throughput to be 1 job per hour (same as the arrival rate)
• the down time probability to be 0.1125
• the average job delay (queue delay) to be 4.11 hours per job
[see Altiok (1997), Chapter 3]
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
14
Example: Point Estimation in Arena
(Cont.)
• The results of 5 replications from a simulation of a workstation
subject to failures are displayed in the table below
Replication
Number
1
2
3
4
5
Average
Throughput
1.0096
0.9858
1.0002
1.0176
0.9999
Average
Job Delay
3.9111
3.9307
3.4373
3.4243
3.5828
Probability of
Down State
0.1144
0.1133
0.1038
0.1116
0.1147
• Note that the estimates vary from replication to replication
• the underlying estimator is a random variable!
• Any of the values can be used to estimate the true (unknown)
parameter
• but how confident can the modeler be in their accuracy?
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
15
Confidence Interval Estimation
• Confidence Interval (CI) estimation quantifies the confidence
(probability) that an interval “covers” the true (but unknown)
statistic
• the boundaries of the confidence interval are estimated using
appropriate point estimates
• therefore, those boundaries are random variables, and the CI is a
random interval which varies across experiments (replications)!
• the modeler predetermines the desired probability that the CI “covers”
the true statistic (the larger the probability, the wider the interval)
• Confidence intervals can be computed for
• terminating simulations
• steady-state simulations
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
16
Confidence Intervals for Terminating
Simulations
• Assume that to estimate a parameter q by simulation,
• n independent fixed-length replications of the model were run
• the runs produced a sample { q€(1), K , q€(n )} , where q€(r ) is the
point estimate of q in replication r
• The pooled point estimator Q for q is the sample mean across
replications
1 n €
Q = е Q (r )
n r =1
• this estimator is a random variable with mean m and variance s 2 / n
• thus, increasing the number of replications, n , would decrease the
variance of Q , and consequently, increase our confidence in the
corresponding point estimate value, q
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
17
Confidence Intervals for Terminating
Simulations (Cont.)
• We wish to quantify the confidence in an estimate of the true
parameter q by computing (at least approximately) the
probability of events of the form
€ Ј qЈ Q
€ } = 1- a
Pr{ Q
1
2
where
• the estimators Q€ and Q€ define a (random) confidence interval [Q€ , Q€ ]
1
2
2
1
for the true parameter, q
• 1- a is the probability that the confidence interval [Q€1, Q€2 ]
does not include the true parameter, q
• a is a small probability, (often 0.01 or 0.05), called the
significance level
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
18
Confidence Intervals for Terminating
Simulations (Cont.)
• By the Central Limit Theorem,
n
1 n €
€
• the random variable е Q (r ) in the estimator Q = е Q (r )
n r=1
r=1
is approximately normally distributed with mean n m and variance n s 2
• It follows that Q : Norm( m, s 2 / n )
• From properties of the normal distribution,
• the approximation improves as the sample size, n , increases
€ (r )] = q
• we have m = E[Q ] = E[Q
(or at least approximately so, if the bias is tolerable)
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
19
Confidence Intervals for Terminating
Simulations (Cont.)
• It is known that Z =
Q- m
s / n
: Norm( 0, 1) has the standard normal
distribution with quantiles z a / 2 , where Pr{Z Ј z a } = a
1- a
a / 2
za/ 2
a / 2
z 1- a / 2
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
20
Confidence Intervals for Terminating
Simulations (Cont.)
• A confidence interval for the mean parameter, q = m ,
at significance level a is given by
Pr{Q - z 1- a / 2 s 2 / n Ј m Ј Q + z 1- a / 2 s 2 / n } = 1- a
• In case the variance s
2
is unknown, proceed as follows:
• estimate the variance by S
2
Q€
n
1
€(r ) - Q ]2
=
[
Q
е
n - 1 r=1
Q- m
• use the fact that the Student t distribution satisfies T
n- 1 = S / n
€
Q
• the requisite confidence interval is Q ± t
n - 1, 1- a / 2
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
S 2€ / n
Q
21
Confidence Intervals for
Steady-State Simulations
• Consider confidence interval estimation for some mean q = m
in Batch Means setting with a single replication of m batches
(the underlying history may be discrete or continuous)
• The corresponding set of m estimators, { Q€ 1, K , Q€ m } , gives rise to
• sample mean Q = 1
m
• sample variance
S 2€
Q
m
еj = 1 Q€j
m
1
=
[Q€j - Q ]2
е
m - 1 j=1
• The confidence interval for q = m at significance level a is
Q ± t m - 1, 1- a / 2 S Q2€ / m
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
22
Confidence Interval Estimation
in Arena
• Standard Arena output provides 95% Batch Means confidence
intervals for each replication
• computed for both Tally and Time Persistent statistics in terms of
half widths under the Half Width column heading
• if, however, the estimated batch means are significantly dependent
or the underlying sample history is too short to yield a sufficient number
of batches, then that column will display the message (Correlated)
• Arena also supports the computation of confidence intervals
from multiple replications in the Outputs element of the
SIMAN summary report
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
23
Example: Output Analysis via
Standard Arena Output
• As a working example, consider a single-machine finishing
operation workstation that processes two types of parts,
denoted by G_1 and G_2, where
• parts of type G_1 arrive according to an exponential inter-arrival time
distribution with mean 2 hours, and each part has a fixed processing time
of 1 hour
• parts of type G_2 arrive according to an exponential inter-arrival time
distribution with mean 4 hours, and each part has a fixed processing time
of 1.4 hours
• all parts are processed in FIFO order
• all part types have equal service priorities
• We wish to simulate the finishing operation for 10,000 hours
in order to understand the behavior of the number of parts in the
workstation buffer and the buffer delay for each part type
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
24
Example: Output Analysis via
Standard Arena Output (Cont.)
Figure 9.3 Arena model of a finishing operation with two types of parts
Arena model of a finishing operation with two types of parts
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
25
Example: Observation Collection
Record module with Tally statistics for part delay in the buffer
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
26
Example: Observation Collection
(Cont.)
Dialog boxes for the Set module to tally delay times for each part type (bottom)
and its members (top)
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
27
Example: Observation Collection
(Cont.)
Dialog box for the Statistic module with a Tally statistic for each part delay
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
28
Example: Output Summary
Output summary report for Resources
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
29
Example: Output Summary (Cont.)
Output summary report for Queues
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
30
Example: Output Analysis
• From the summary report we observe the following partial
utilizations in the finishing machine:
• r 1 = 1 /2 = 0.5 is utilization due to type G_1 parts workload
• r 2 = 1.4 /4 = 0.35 is utilization due to type G_2 parts workload
• r = 0.85 is utilization due to overall workload
• We, therefore, expect the probability of the finishing machine
being in the Busy state to be around 0.85
• The estimated probability in the summary report for Resources
is actually 0.84 over a replication of length 10,000 hours.
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
31
Example: Output Analysis (Cont.)
• The table below shows the behavior of three performance
measures as functions of replication length
Replication
Length
10,000
100,000
1,000,000
10,000,000
Finishing
Machine
Busy
0.84038
0.84652
0.84855
0.85009
G_1 Buffer
Delay
G_2 Buffer
Delay
2.9603
3.1245
3.3017
3.2986
2.9206
3.1264
3.3081
3.3020
• Observe how the estimates appear to converge to respective limiting
values as the simulation length increases
• convergence is indicated by the fact that the values appear to stabilize and
change very little for the higher range of replication lengths
• Since we know the true value of machine utilization, we can take
advantage of this knowledge in deciding on the smallest replication length
that gives rise to sufficiently accurate estimates
• high accuracy is indicated in the table by low variability in the estimates
as function of replication length
.
• we naturally seek the smallest value, since we would like to reduce the
computational effort as much as possible
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
32
Example: Output Analysis (Cont.)
Performance
Measure
Machine Busy
G_1 Buffer
Delay
G_2 Buffer
Delay
Average
Value
0.84697
3.1366
Half
Width
0.00669
0.28652
3.1421
0.30979
Minimum Maximum Number of
Value
Value
Replications
0.83346
0.85974
10
2.6073
4.1624
10
2.6811
4.1841
10
Summary statistics based on 10 replications
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
33
Output Analysis via
the Arena Output Analyzer
• The Arena Output Analyzer tool provides the following
capabilities:
• Data transfer
• data import from ASCII files
• data export to ASCII files
• Statistical analysis
• batching, correlogram, and basic statistics
• point estimation and confidence interval estimation for means and
standard deviations
• statistical tests for comparing parameters of different samples
• Graphing
• data plots and charts
• statistics plots and charts
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
34
Example: Data Collection
Collection Time
0.00000000e+000
3.80000000e+000
6.20000000e+000
8.60000000e+000
9.60000000e+000
1.06000000e+001
1.30000000e+001
1.40000000e+001
1.64000000e+001
2.16000000e+001
2.26000000e+001
2.36000000e+001
2.46000000e+001
2.70000000e+001
2.80000000e+001
3.18000000e+001
3.28000000e+001
3.38938754e+001
3.72061345e+001
4.21822596e+001
4.45822596e+001
4.97822596e+001
-1.00000000e+000
Observed G_1_Q Queue
Time
0.00000000e+000
1.06411972e+000
5.70854437e-002
2.10152691e+000
2.80815078e+000
3.61637065e+000
3.10610560e+000
1.00060690e+000
1.83680646e+000
5.07139068e+000
4.37504831e+000
4.69624875e+000
5.49921187e+000
4.10759217e+000
4.53584068e+000
1.88719423e+000
1.41925206e-001
0.00000000e+000
4.24329579e-001
1.04363036e+000
7.33678306e-001
1.33942112e+000
-1.00000000e+000
Tallied buffer delay observations for parts of type G_1
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
35
Example: Graphical Statistics
Graphical statistics for parts of type G_1
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
36
Example: Batching Data for
Independent Observations
Batch/Truncate dialog box
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
37
Example: Batching Data for
Independent Observations (Cont.)
Batched observations stored in file : G_1_Delay_Batched.flt
Initial Observations Truncated :
0
Number of Batches :
499
Number of Observations Per Batch :
1000
Number of Trailing Obs'ns Truncated :
486
Estimate of Covariance Between Batches : -0.002459
Batch/Truncate summary report for delay times of G_1 type parts
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
38
Example: Confidence Intervals for
Means and Variances
Confidence intervals for mean delays of parts of type G_1 and G_2
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
39
Comparing Means and Variances
• The Analyze menu in the Arena Output Analyzer tool provides
the following options:
• Compare Means for comparing the means of two samples
drawn from two populations
• Compare Variances for comparing the variances of two samples
drawn from two populations
• These options perform statistical analysis for testing statistically
for the equality of means and variances, respectively
• For example, to test the null hypothesis that the means are equal,
• a confidence interval is sets up for their difference
• since under the null hypothesis the difference is zero, one accepts equality
if the confidence interval includes 0, and rejects it, otherwise
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
40
Example: Comparing Means
Dialog boxes for comparing mean buffer delays of type G_1 and G_2 parts
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
41
Example: Testing for
Equality of Means
Test results for the equality of mean buffer delays of type G_1 and G_2 parts
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
42
Point Estimation for Correlation
• The Analyze menu in the Arena Output Analyzer tool provides
the Correlogram option to gauge the statistical dependence
among observations within a sample
• performs point estimation of the correlation between two samples
by estimating the coefficient of correlation between the respective
random variables
• This activity is called correlation analysis
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
43
Example: Correlation Analysis
• The left-hand correlogram
was computed from
simulation observations
of successive delays
• The right-hand correlogram
was computed from
the sample means of
batches of 1000 delays
• The two correlograms
are strikingly different:
the successive delays
are strongly positively
correlated, while the
sample means of the
batches are very nearly
uncorrelated
Results of correlation analysis for type G_1 parts
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
44
Parametric Analysis via
the Arena Process Analyzer
• The term Parametric Analysis refers to the following set of
activities:
• running a model multiple times with a different set of input parameters
for each run
• then comparing the resultant performance measures
• Its purpose is to carry out to Sensitivity Analysis
• understand the impact of parameter changes on system behavior
• often use this understanding to find the optimal configuration (parameter set)
with respect to one or more performance measures, or combination thereof
• In Arena Process Analyzer parlance
• input parameters are called controls
• the resultant performance measures are called responses
• a collection of controls and responses for a given set of runs is referred to
as a scenario
• a collection of scenarios is termed an Arena project
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
45
The Process Analyzer GUI
A Process Analyzer window with an initial grid
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
46
The Process Analyzer GUI (Cont.)
A Process Analyzer window with a populated grid
Altiok / Melamed Simulation Modeling and Analysis with Arena
Chapter 9
47