IE 306
Systems Simulation
Output Analysis - 2
IE 306 – Systems Simulation
G. Yücel
Steady-state example
• A web-based company, operates 24 hours/day
• Considering additional servers
• Aim is to accommodate sustained periods of peak load
– Long-run average throughput and utilization under the current peak
load conditions
IE 306 – Systems Simulation
G. Yücel
Steady-state Behavior: Motivation
• Sometimes we are interested in the long-run behavior of the
system
– Steady-state: Long-run (technically forever)
• Observations collected in a single run are not only autocorrelated but they also depend on initial conditions
IE 306 – Systems Simulation
G. Yücel
Warm Up and Run Length
• Most models start empty and idle
– Empty: No entities present at time 0
– Idle: All resources idle at time 0
– In a terminating simulation this is OK if realistic
– In a steady-state simulation, though, this can bias the output for a
while after startup
• Bias can go either way
• Usually downward (results are biased low) in queueing-type models that
eventually get congested
• Depending on model, parameters, and run length, the bias can be very severe
IE 306 – Systems Simulation
G. Yücel
Warm Up and Run Length (cont’d.)
• The period up to 1500 minutes is less congested
• Thus average output measures will be biased down
• How can we get rid of this bias?
IE 306 – Systems Simulation
G. Yücel
Initial Phase
• The upward trend at the beginning is the result of initial
conditions (empty & idle).
• Collecting statistics after T0 time units have passed seems to be
a better idea.
• How to determine T0 ?
• For each collected statistics T0 might be different!
• Use moving averages for plotting mean plots.
IE 306 – Systems Simulation
G. Yücel
Intelligent Initial Conditions
• Collect data
– Observe an actual state of the real system that has been running for a
reasonable period of time
– Use this state as the initial conditions
– Not possible if system does not exist or you are modifying the system
• Use another model
– Queuing models, inventory models etc.
– Give steady-state results under more restrictive assumptions than
simulation
– Use these results as initial conditions
IE 306 – Systems Simulation
G. Yücel
Determining Warm-up Times
• Ensemble averages
– The average across replications of the first,
second, third, … observations
– Each ensemble average is an iid sample from
the distribution of that observation
– Put t-distribution confidence interval around
each average
– See when the ensemble averages settle
down
Y11, Y12 , , Y1m Y1•
Y21, Y22 ,, Y2 m Y2•
Yn1 , Yn 2 , , Ynm Yn•
Y•1 , Y•2 ,, Y•m
Series
Averages
s•21 , s•22 ,, s•2m
Ensemble
Averages
IE 306 – Systems Simulation
G. Yücel
Mean plot
Across 10 replications
IE 306 – Systems Simulation
G. Yücel
Determining Warm-up Times
IE 306 – Systems Simulation
G. Yücel
Determining Warm-up Times
IE 306 – Systems Simulation
G. Yücel
Truncated Replications
• If you can identify appropriate warm-up and run-length times,
just make replications as for terminating simulations
– Only difference: Specify Warm-Up Period in Simulate module
Y1,mW +1 ,, Y1,mW +m1
Y2,mW +1 ,, Y2,mW +m2
Y2•
Yn ,mW +1 , ,, Yn ,mW +mn
IE 306 – Systems Simulation
Y1•
Yn•
Y
s2
s
t n−11
, − / 2
n
G. Yücel
Batch Means Method
• One disadvantage of the replication method is that initial
phase is deleted from all replications (waste of data!).
• Instead,
– let’s have a single long run (will be correlated)
– Throw away the data that corresponds to the initial phase
– Divide the data into k batches and find their means separately (will
be approximately independent)
IE 306 – Systems Simulation
G. Yücel
Batching in a Single Run
• Alternative: Just one R E A L L Y long run
– Only have to “pay” warm-up once
Y1 , Y2 ,, YmW , YmW +1 ,...YmW + m
throw away
sample
– Problem: Have only one “replication” and you need more than that
to form a variance estimate (the basic quantity needed for statistical
analysis)
• Big mistake: Use the individual points within the run as “data” for variance
estimate
• Usually correlated (not indep.), variance estimate biased
IE 306 – Systems Simulation
G. Yücel
Batching in a Single Run (cont’d.)
• Break each output record from the run into a few large batches
Y1 ,..., YmW , YmW +1 ,..., YmW +k , YmW +k +1 ,.., YmW +2 k , YmW +2 k +1 ,..., YmW +ck
Warm-up
IE 306 – Systems Simulation
Batch 1
Batch 2
1 k
Y1 = YmW +i
k i =1
1 k
Y2 = YmW +k +i
k i =1
…...
1 k
Y j = YmW +( j −1) k +i
k i =1
G. Yücel
Batch Means cont’d
Y1 ,..., Yd , Yd +1 ,..., Yd + m , Yd + m+1 ,..., Yd + 2 m ,..., Yd + ) k −1) m+1 ,..., Yd + km
deleted
Y1
Y2
Yk
(Y − k Y )
S
1 k (Y j − Y )
j =1
=
=
k
k j =1 k − 1
k (k − 1)
2
2
k
2
j
2
WhereY is the overall sample mean.
The batch means are approximately independent.
The variance estimator is approximately unbiased.
IE 306 – Systems Simulation
G. Yücel
Back to M/M/1 Example
• We take the responses for 5000 customers
• Remove first 20 observations
• Divide the remaining into 30 batches (of 166 observations)
1
5
9
13
17
21
25
29
33
37
41
45
49
53
57
61
65
69
73
77
81
85
89
93
97
101
105
109
113
117
121
125
129
133
137
141
145
149
153
157
161
165
16
14
12
10
8
6
4
2
0
• CI at (95%)
– Mean=10.3262
– Half Width=0.9416
IE 306 – Systems Simulation
G. Yücel
Batching in a Single Run (cont’d.)
• Rules of thumb
– Schmeiser (1982) found that for a given run length, there was little benefit in more than 30
batches
– However, less than 10 batches was too few
Y1 , Y2 , , Yc
– There may well be correlation between all lags, looking at the lag 1 correlation is usually enough
to ascertain independence
– Auto-correlation estimates are not very good for sample sizes like 30
– Use smaller batches (say c > 100) and if the independence test is passed then the bigger batches
will be fine
IE 306 – Systems Simulation
G. Yücel
0
