Module 3
Descriptive Time Series Statistics
and Introduction to Time Series Models
Class notes for Statistics 451: Applied Time Series
Iowa State University
Copyright 2015 W. Q. Meeker.
November 11, 2015
17h 20min
3-1
Module 3
Segment 1
Populations, Processes, and
Descriptive Time Series Statistics
3-2
Populations
• A population is a collection of identifiable units (or a specified characteristic of these units).
• A frame is a listing of the units in the population.
• We take a sample from the frame and use the resulting data
to make inferences about the population.
• Simple random sampling with replacement from a population implies independent and identically distributed (iid)
observations.
• Standard statistical methods use the iid assumption for observations or residuals (in regression).
3-3
Processes
• A process is a system that transforms inputs into outputs,
operating over time.
inputs
outputs
• A process generates a sequence of observations (data) over
time.
• We can use a realization from the process to make inferences about the process.
• The iid model is rarely appropriate for processes (observations close together in time are typically correlated and the
process often changes with time).
3-4
Process, Realization, Model,
Inference, and Applications
Description
Process
Realization
Generates Data
Statistical Model
for the Process
(Stochastic Process Model)
Z1 , Z2 , ..., Zn
Estimates for Model
Parameters
Inferences
Forecasts
Control
3-5
Stationary Stochastic Processes
Zt is a stochastic process. Some properties of Zt include
mean µt, variance σt2, and autocorrelation ρZt,Zt+k . In general,
these can change over time.
• Strongly stationary (also strictly or completely stationary):
F (zt1 , . . . , ztn ) = F (zt1+k , . . . , ztn+k )
Difficult to check.
• Weakly stationary (or 2nd order or covariance stationary)
requires only that µ = µt and σ 2 = σt2 be constant and that
ρk = ρZt,Zt+k depend only on k.
Easy to check with sample statistics.
Generally, “stationary” is understood to be weakly stationary.
3-6
5
0
-5
Billions of Dollars
10
15
Change in Business Inventories 1955-1969
Stationary?
1955
1957
1959
1961
1963
1965
1967
1969
Year
3-7
Estimation of Stationary Process Parameters
Some Notation
Process
Parameter
Mean of Z
Variance of Z
Standard
Deviation
Notation
µZ = E(Z)
2 = Var(Z)
γ0 = σZ
σZ
Autocovariance
γk
Autocorrelation
ρk = γγk
0
Variance of Z̄
2 = Var(Z̄)
σZ̄
Estimate
b Z = z̄ =
µ
γb0 =
Pn
γbk =
t=1 zt
n
t=1(zt −z̄)
bZ =
σ
Pn−k
Pn
2
n
√
γb0
t=1 (zt −z̄)(zt+k −z̄)
n
ρbk = γγbbk
0
γ
b0
S2
=
n [· · · ]
Z̄
3-8
5
0
-5
Billions of Dollars
10
15
Change in Business Inventories 1955-1969
bZ
with z̄ and z̄ ± 3σ
1955
1957
1959
1961
1963
1965
1967
1969
Year
3-9
Variance of Z̄ with Correlated Data
The Variance of Z̄
γ0
2
σZ̄ = Var(Z̄) =
n
n−1
X
Ã
!
|k|
1−
ρk
n
k=−(n−1)
with correlated data is more complicated than with iid data
where
γ0
σ2
2
σZ̄ = Var(Z̄) =
=
n
n
Thus if one incorrectly uses the iid formula, the variance is
under estimated, potentially leading for false conclusions.
3 - 10
Module 3
Segment 2
Correlation and Autocorrelation
3 - 11
Correlation [From “Statistics 101”]
Consider random data y and x (e.g., sales and advertising):
x
y
x1
x2
..
xn
y1
y2
..
yn
ρx,y denotes the “population” correlation between all values
of x and y in the population.
To estimate ρx,y , we use the sample correlation
Pn
(xi − x̄)(yi − ȳ)
,
ρbx,y = qP i=1
P
n (x − x̄)2 n (y − ȳ)2
i=1 i
i=1 i
−1 ≤ ρbx,y ≤ 1
(Stat 101)
3 - 12
Sample Autocorrelation
Consider the random time series realization z1, z2, . . . , zn
t
1
2
3
..
n−2
n−1
n
zt
z1
z2
z3
..
zn−2
zn−1
zn
Lagged Variables
zt+1 zt+2 zt+3 zt+4
z2
z3
z4
z5
z3
z4
z5
z6
z4
z5
z6
z7
..
..
..
..
zn−1
zn
–
–
zn
–
–
–
–
–
–
–
Assuming weak (covariance) stationarity, let ρk denote the
process correlation between observations separated by k time
periods. To compute the “order k” sample autocorrelation
(i.e., correlation between zt and zt+k )
γbk
=
ρbk =
γb0
Pn−k
t=1 (zt − z̄)(zt+k − z̄) ,
Pn
2
(z
−
z̄)
t
t=1
Note: −1 ≤ ρbk ≤ 1
k = 0, 1, 2, . . .
3 - 13
Sample Autocorrelation (alternative formula)
Consider the random time series realization z1, z2, . . . , zn
t
1
2
3
4
..
n
zt
z1
z2
z3
z4
..
zn
Lagged Variables
zt−1 zt−2 zt−3 zt−4
–
–
–
–
z1
–
–
–
z2
z1
–
–
z3
z2
z1
–
..
..
..
..
zn−1 zn−2 zn−3 zn−4
Assuming weak (covariance) stationarity, let ρk denote the
process correlation between observations separated by k time
periods.
To compute the “order k” sample autocorrelation (i.e., correlation between zt and zt−k )
Pn
γbk
t=k+1(zt − z̄)(zt−k − z̄)
ρbk =
,
=
Pn
2
γb0
(z
−
z̄)
t=1 t
Note: −1 ≤ ρbk ≤ 1
k = 0, 1, 2, . . .
3 - 14
0
50
100
150
Wolfer Sunspot Numbers 1770-1869
0
20
40
60
80
100
3 - 15
Lagged Sunspot Data
t
1
2
3
4
5
..
99
100
101
102
103
104
Spot
101
82
66
35
31
..
37
74
–
–
–
–
Lagged Variables
Spot1 Spot2 Spot3 Spot4
–
–
–
–
101
–
–
–
82
101
–
–
66
82
101
–
35
66
82
101
..
..
..
..
7
16
30
47
37
7
16
30
74
37
7
16
–
74
37
7
–
–
74
37
–
–
–
74
3 - 16
Wolfer Sunspot Numbers
Correlation Between Observations Separated by One
Time Period
150
•
•
Autocorrelation = 0.806244
Correlation = 0.81708
•
•
•
•
•
spot.ts[-100]
100
•
•
•
• •
•
0
50
•
••
••
••
••
•
•
•
•• •
• •
• • •
••
••
• • • •• • • •
•
•
•
•
•• • •
0
• •
•
• •
•
• •
•
•
•
•
•
•
•
•
•
•
•
•
•
•••
••
•• •
•
•• •
•
•
•
•
• •
•
•
•
50
100
150
spot.ts[-1]
3 - 17
Wolfer Sunspot Numbers
Correlation Between Observations Separated by k
Time Periods [show.acf(spot.ts)]
50
150
0
50
•
100
150
0
150
50
100
100
50
Series lag 4
•
0
150
•
0
•
100
Series lag 3
•
•
•
•
50
150
100
50
0
Series lag 2
Series lag 1
100
150
100
•
• •
•• • • •
•
•
•• • •
•
•
•••• •• • •• •
••
•
••• •
•
• ••
•
••• ••••••• • • •
•
• •
•••
• •
•• • •
••
•
•
•
•
• • •• •• ••• • • • ••
••
•
•
• ••
150
••
•••• ••
•
••• •
•
•• ••••••
• •
•
••
•
•••••• • • • •• • • • •
• •
• •
•••
•• •
• •• • •
•• •
••
•
•
•• • •• ••• •
••• •
• •
•• • •
0
50
100
100
150
0
50
100
0
50
100
150
••
• •
•
• • • ••• •
•••
•
•
•
• ••• • • • •
• • ••
• ••
••
• ••• ••••
••••• •• •• •• • •••
• • ••• •••• ••
•
0
50
100
100
•
•
•
Series lag 9
•
•• •
•
50
•
••
150
•
150
•
•
• •
• •
•
0
•
100
•
•
•
•
• •
• • • ••
•• • •
• ••
• ••
•
•
••••• • • •
•
••
•• •
• •
•
• •• • • •• •
•••• • • •• • • ••
••• •••••• •• ••
•••• • •
Series lag 8
100
150
•
•
••
•• •
•
50
•
50
•
• •••• • •
• •
• •
•
• •
•
••• • •
• • •• •
•
••••
•
• •
•
•
• •
••
• ••
••• •
••
•
••• ••• ••
•
•
•
•
•• • •• •• •• •• • •• •
• •• •
•
0
0
•
•
•
•
•• •
••
0
100
50
Series lag 6
••••
Series lag 7
•
•
• •
150
Lag 9
150
Lag 8
150
Lag 7
150
Lag 6
•
•
•
• •
•
•
••• •
•
• •• •
•
•
•• ••• • •• •
• • •• ••• • •• •
••• •••
•
•• •• •• ••••• • •
•
•••• • •••• ••
•
•
0
•
•
20
•
60
100
•
••
•
•
140
series
Lag 10
Lag 11
Lag 12
Lag 13
Series : timeseries
60
series
140
0
20
60
series
100
• •
140
0
20
60
series
100
•
••
• •
140
•
•
•
•
•
•
•
•
•
••
•• •
••
• • •••
•
••• •
•
• • •
• • • •• • • • • •• ••
• • • •
•• • •
• •••• •
• •
••
•••••••• ••• ••
• •
0
20
•
ACF
0.2
0.6
•
• • • •••• ••• •
•
•
•• •
••
• • ••
•
•
•• •• •• •• •
•
•
•
• •
•
••••• •• • •
•
•••••••• •• • •• •
•
100
•
50
•
•
•
•
••
•
60
100
•
•
•
• ••
140
-0.2
•
•
•
•
Series lag 13
•
•
100
•
• •
•
•
•
•
•
•
•
•
•
••
• ••• •• •
•
• • ••••
•• •••
• •• • ••
•
••
•
••••
•
•
•
• •• •
•••••• •••• •••• • •
•
•
50
•
•
0
•
•
••
0
•
•
100
•
•••
Series lag 12
••
•
•• • •
•• ••
••• •••• ••
• • • ••
• •
• •
•• • •
•• •••• • • •• • • •
••••• • •••• •
•
•
Series lag 11
• •
•
••
•
•• • •
• •
•
•
••
•
1.0
series
150
series
150
series
150
series
150
100
50
Series lag 10
•
••• •
•••• • •• • •
••
•
•
• •• •
••• • • • • •
•••• •••• •• •
••• • • • •
•
•
•••• •• •• • •••• •• • •
••
••
•• •
•
Lag 5
•
20
•
•
•
•
•
•
series
•
0
•
•
•
•
•
•
•
series
••
•
•
•
•
•
•
•••
•
••
••• • •
•
•
Lag 4
series
50
•
50
• •
Lag 3
series
•• • •
••
• ••• •
•
•
•
•••• ••
•
• • •
• •
•
•
•
• • ••
••••••••
•
• •
•
• •
•• • •
• •• •••
•
•• •
•
•
•
•
•
••
••
• •• •••••• • • • • • •
••
•
•
•
• •
•
•
•• • • •
•••
•
••• •• • ••••
•
•
•
•• •••• •
•
•• •• • • • •
••• • •
• •
•
•• •
••• • ••
••••••••••••• ••• •
•••
0
0
100
Series lag 5
50
0
•••
0
0
150
100
•
•
•
100
Lag 2
timeseries
50
150
0
0
•••
••••
••••
••
•
•
•
•••
••••
•••
••••
•
•
•
••
•••
••••
•••
50
•
•
••
•••
100
50
0
timeseries
150
Lag 1
0
5
10
Lag
15
20
series
3 - 18
Wolfer Sunspot Numbers Sample ACF Function
-0.2
0.0
0.2
ACF
0.4
0.6
0.8
1.0
Series : spot.ts
0
5
10
Lag
15
20
3 - 19
Module 3
Segment 3
Autoregression, Autoregressive Modeling,
and and Introduction to Partial Autocorrelation
3 - 20
0
50
100
150
Wolfer Sunspot Numbers 1770-1869
0
20
40
60
80
100
3 - 21
Autoregressive (AR) Models
• AR(0): zt = µ + at (White noise or “Trivial” model)
• AR(1): zt = θ0 + φ1zt−1 + at
• AR(2): zt = θ0 + φ1zt−1 + φ2zt−2 + at
• AR(3): zt = θ0 + φ1zt−1 + φ2zt−2 + φ3zt−3 + at
• AR(p): zt = θ0 + φ1zt−1 + φ2zt−2 + · · · + φpzt−p + at
at ∼ nid(0, σa2)
3 - 22
Data Analysis Strategy
Tentative
Identification
No
Time Series Plot
Range-Mean Plot
ACF and PACF
Estimation
Least Squares or
Maximum Likelihood
Diagnostic
Checking
Residual Analysis
and Forecasts
Model
ok?
Yes
Use the Model
Forecasting
Explanation
Control
3 - 23
AR(1) Model for the Wolfer Sunspot Data
Sunspot Residuals, AR(1) Model
0
-20
50
20
100
60
150
Data and 1-Step Ahead Predictions
0
20
40
60
80
100
0
20
40
Year Number
60
80
100
Year Number
0
5
10
Lag
15
60
• •
20
-20
-0.4
ACF
0.2
0.8
residuals(spot.ar1.fit)
Series : residuals(spot.ar1.fit)
•
•
•
••
•
••••
••
•
••
••••••••
•
•
•
•
•
•
••••••
••••••••
•
•
•
•
•
•
••
•
•
•
•
•
•
•••
••••••••••••••••••
• • •••••
-2
-1
0
1
2
Quantiles of Standard Normal
3 - 24
AR(2) Model for the Wolfer Sunspot Data
Sunspot Residuals, AR(2) Model
0
-40
50
0 20
100
150
Data and 1-Step Ahead Predictions
0
20
40
60
80
100
0
20
40
Year Number
60
80
100
Year Number
0
5
10
Lag
15
0 20
•
-40
-0.2
ACF
0.4
0.8
residuals(spot.ar2.fit)
Series : residuals(spot.ar2.fit)
•
•
••••••
•
•
•
•
•
•••••••••
•••••••••••••
•
•
•
•
•
••••••••
••••••••••••••••••••••
•
•
•
•
•
•
•
•
• • ••
• •
•
-2
-1
0
1
2
Quantiles of Standard Normal
3 - 25
AR(3) Model for the Wolfer Sunspot Data
Sunspot Residuals, AR(3) Model
0
-40
50
0
100
20 40
150
Data and 1-Step Ahead Predictions
0
20
40
60
80
100
0
20
40
Year Number
60
80
100
Year Number
0
5
10
Lag
15
0 20
•
-40
-0.2
ACF
0.4
0.8
residuals(spot.ar2.fit)
Series : residuals(spot.ar3.fit)
•
•
••••••
•
•
•
•
•
•••••••••
•••••••••••••
•
•
•
•
•
••••••••
••••••••••••••••••••••
•
•
•
•
•
•
•
•
• • ••
• •
•
-2
-1
0
1
2
Quantiles of Standard Normal
3 - 26
AR(4) Model for the Wolfer Sunspot Data
Sunspot Residuals, AR(4) Model
0
-20
50
0
20
100
40
150
Data and 1-Step Ahead Predictions
0
20
40
60
80
100
0
20
40
Year Number
60
80
Year Number
0
5
10
Lag
15
0 20
•
-40
-0.2
ACF
0.4
0.8
residuals(spot.ar2.fit)
Series : residuals(spot.ar4.fit)
•
•
••••••
•
•
•
•
•
•••••••••
•••••••••••••
•
•
•
•
•
••••••••
••••••••••••••••••••••
•
•
•
•
•
•
•
•
• • ••
• •
•
-2
-1
0
1
2
Quantiles of Standard Normal
3 - 27
Sample Partial Autocorrelation Function
The sample PACF φbkk can be computed from the sample ACF
ρb1, ρb2, . . . using the recursive formula from Durbin (1960):
φb1,1 = ρb1
φbkk =
where
ρbk −
k−1
X
φbk−1 ,j ρbk−j
j=1
k−1
X
1−
j=1
, k = 2, 3, ...
φbk−1,j ρbj
φbkj = φbk−1,j − φbkk φbk−1,k−j (k = 3, 4, ...; j = 1, 2, ..., k − 1)
Thus the PACF contains no new information—just a different way of looking at the information in the ACF.
This sample PACF formula is based on the solution of the
Yule-Walker equations (covered module 5).
3 - 28
Summary of Sunspot Autoregressions
Order
Regression Output
Model
p
R2
S
AR(1)
AR(2)
AR(3)
AR(4)
1
2
3
4
.6676
.8336
.8407
.8463
21.53
15.32
15.12
15.01
φbp
.810
−.711
.208
−.147
PACF
tφb
p
13.96
−9.81
2.04
−1.41
φbpp
.8062
−.6341
.0805
−.0611
φbp is from ordinary least squares (OLS).
φbpp is from the Durbin formula.
3 - 29
Sample Partial Autocorrelation
The “true partial autocorrelation function,” denoted by φkk ,
for k = 1, 2, . . . is the process correlation between observations separated by k time periods (i.e., between zt and
zt+k ) with the effect of the intermediate zt+1, . . . , zt+k−1
removed.
We can estimate φkk , k = 1, 2, . . . with φbkk , k = 1, 2, . . .
• φb1,1 = φb1 from the AR(1) model
• φb2,2 = φb2 from the AR(2) model
• φbkk = φbk from the AR(k) model
The Durbin formula gives somewhat different answers due
to the basis of the estimator (ordinary least squares versus
solution of the Yule-Walker equations).
3 - 30
Module 3
Segment 4
Introduction to ARMA Models
and Time Series Modeling
3 - 31
Autoregressive-Moving Average (ARMA) Models
(a.k.a. Box-Jenkins Models)
• AR(0): zt = µ + at (White noise or “Trivial” model)
• AR(1): zt = θ0 + φ1zt−1 + at
• AR(2): zt = θ0 + φ1zt−1 + φ2zt−2 + at
• AR(p): zt = θ0 + φ1zt−1 + · · · + φpzt−p + at
• MA(1): zt = θ0 − θ1at−1 + at
• MA(2): zt = θ0 − θ1at−1 − θ2at−2 + at
• MA(q): zt = θ0 − θ1at−1 − · · · − θq at−q + at
• ARMA(1, 1): zt = θ0 + φ1zt−1 − θ1at−1 + at
• ARMA(p, q): zt = θ0 + φ1zt−1 + · · · + φp zt−p
− θ1at−1 − · · · − θq at−q + at
at ∼ nid(0, σa2)
3 - 32
Graphical Output from RTSERIES Function
iden(spot.tsd)
Wolfer Sunspots
0
50
w
100
150
w= Number of Spots
1780
1800
1820
1840
1860
time
ACF
0.0
-1.0
120
ACF
140
0.5
1.0
Range-Mean Plot
100
10
20
30
Lag
0.5
0.0
-1.0
60
Partial ACF
1.0
80
PACF
40
Range
0
20
30
40
50
Mean
60
70
0
10
20
30
Lag
3 - 33
Tabular Output from RTSERIES Function
iden(spot.tsd)
Identification Output for Wolfer Sunspots
w= Number of Spots
[1] "Standard deviation of the working series= 37.36504"
ACF
Lag
ACF
se
t-ratio
1
1 0.806243956 0.1000000 8.06243956
2
2 0.428105325 0.1516594 2.82280693
3
3 0.069611110 0.1632975 0.42628402
.
.
.
Partial ACF
Lag Partial ACF se
t-ratio
1
1 0.806243896 0.1 8.06243896
2
2 -0.634121358 0.1 -6.34121358
3 - 34
Standard Errors for Sample Autocorrelations and
Sample Partial Autocorrelations
vÃ
!
u
u 1+2ρb21+···+2ρb2
k−1
• Sample ACF standard error: Sρbk = t
n
Also can compute the “t-like” statistics t = ρbk /Sρbk .
√
• Sample PACF standard error: Sφb = 1/ n
kk
Use to compute “t-like” statistics t = φbkk /Sφb and set dekk
cision bounds.
• In long realizations from a stationary process, if the true
parameter is really 0, then the distribution of a “t-like”
statistic can be approximated by N(0, 1)
• Values of ρbk and φbkk may be judged to be different from
zero if the “t-like” statistics are outside specified limits (±2
is often suggested; might use ±1.5 as a “warning”).
3 - 35
Drawing Conclusions from Sample ACF’s and PACF’s
PACF
Dies Down
PACF
Cutoff after p > 1
ACF Dies Down
ARMA
AR(p)
ACF Cutoff after q > 1
M A(q)
Impossible
Technical details, background, and numerous examples will
be presented in Modules 4 and 5.
3 - 36
Comments on Drawing Conclusions from Sample
ACF’s and PACF’s
The “t-like” statistics should only be used as guidelines in
model building because:
• An ARMA model is only an approximation to some true
process
• Sampling distributions are complicated; the “t-like” statistics do not really follow a standard normal distribution (only
approximately, in large samples, with a stationary process)
• Correlations among the ρbk values for different k (e.g., ρb1
and ρb2 may be correlated).
• Problems relating to simultaneous inference (looking at many
different statistics simultaneously, confidence/significance
levels have little meaning).
3 - 37