Module 4
Stationary Time Series Models Part 1
MA Models and Their Properties
Class notes for Statistics 451: Applied Time Series
Iowa State University
Copyright 2015 W. Q. Meeker.
February 14, 2016
20h 58min
4-1
Module 4
Segment 1
ARMA Notation, Conventions, Expectations, and
other Preliminaries
4-2
Deviations from the Sample Mean
• The sample mean is computed as
Pn
zt
t=1
b
µ = z̄ =
n
• We use the centered realization zt − z̄ in the computation
of many statistics. For example
v
uP
u n (z − z̄)2
t
bZ = γ
b0 = SZ = t t=1
σ
n−1
• Also used in computing sample autocovariances and autocorrelations. Subtracting out the mean (z̄) does not change
the sample variance (γb0), sample autocovariances (γbk ), or
sample autocorrelations (ρbk ) of a realization.
4-3
Change in Business Inventories 1955-1969 and
Centered Change in Business Inventories 1955-1969
10
5
0
-5
Billions of Dollars
Change in Inventories
1955
1957
1959
1961
1963
1965
1967
1969
1967
1969
Year
10
5
0
-10
Billions of Dollars
Centered Change in Inventories
1955
1957
1959
1961
1963
1965
Year
4-4
Deviations from the Process Mean
• The stochastic process is Z1, Z2, . . . .
• The process mean is µ = E(Zt).
• Some derivations are simplified by using Żt = Zt − µ to
compute model properties (only the mean has changed).
• In particular,
E(Żt) = E(Zt − µ) = E(Zt) − E(µ) = µ − µ = 0
Similarly,
2
γ0 = Var(Żt ) = Var(Zt −µ) = Var(Zt )+Var(µ) = Var(Zt) = σZ
and
γk = Cov(Żt , Żt+k ) = Cov(Zt − µ, Zt+k − µ) = Cov(Zt , Zt+k )
ρk = γk /γ0
4-5
Backshift Operator Notation
Backshift operators:
BZt = Zt−1
B2Zt = BBZt = BZt−1 = Zt−2
B3Zt = Zt−3, etc.
If C is a constant, then BC = C
Differencing operators:
(1 − B)Zt = Zt − BZt = Zt − Zt−1
(1 − B)2Zt = (1 − 2B + B2)Zt = Zt − 2Zt−1 + Zt−2
Seasonal differencing operators:
(1 − B4)Zt = Zt − B4Zt = Zt − Zt−4
(1 − B12)Zt = Zt − B12Zt = Zt − Zt−12
4-6
Polynomial Operator Notation
φ(B) = φp(B) ≡ (1 − φ1B − φ2B2 − · · · − φpBp)
is useful for expressing the AR(p) model.
For example, with p = 2, an AR(2) model.
φ2(B)Żt = at
(1 − φ1B − φ2B2)Żt = at
Żt − φ1Żt−1 − φ2Żt−2 = at
or
Żt = φ1Żt−1 + φ2Żt−2 + at
Other polynomial operators:
θ(B) = θq (B) ≡ (1−θ1B−θ2B2 −· · ·−θq Bq ) [as in Żt = θq (B)at ]
ψ(B) = ψ∞(B) ≡ (1 + ψ1B + ψ2B2 + · · · ) [as in Żt = ψ(B)at ]
π(B) = π∞(B) ≡ (1 − π1B − π2B2 − · · · ) [as in π(B)Żt = at]
4-7
General ARMA(p, q) Model in Terms of Żt
Żt ≡ Zt − µ
φp(B)Żt = θq (B)at
(1 − φ1B − φ2B2 − · · · − φp Bp)Żt =
(1 − θ1B − θ2B2 − · · · − θq Bq )at
Żt − φ1Żt−1 − φ2Żt−2 − · · · − φpŻt−p =
at − θ1at−1 − θ2at−2 − · · · − θq at−q
or
Żt = φ1Żt−1+φ2Żt−2+· · ·+φpŻt−p−θ1at−1−θ2at−2−· · ·−θq at−q +at
4-8
Mean (µ) and Constant Term (θ0)
for an ARMA(2,2) Model
Replace Żt with Żt = Zt − µ and solve for Zt = · · · .
For example, using p = 2 and q = 2, giving an ARMA(2,2)
model:
Żt = φ1Żt−1 + φ2Żt−2 − θ1at−1 − θ2at−2 + at
and
Zt − µ = φ1(Zt−1 − µ) + φ2(Zt−2 − µ) − θ1at−1 − θ2at−2 + at
Zt = µ − φ1µ − φ2µ + φ1Zt−1 + φ2Zt−2 − θ1at−1 − θ2at−2 + at
Zt = θ0 + φ1Zt−1 + φ2Zt−2 − θ1at−1 − θ2at−2 + at
where θ0 = µ − φ1µ − φ2µ = µ(1 − φ1 − φ2) is the ARMA model
“constant term.” Also, E(Zt) = µ = θ0/(1 − φ1 − φ2).
Note that E(Zt) = µ = θ0 for an MA(q) model.
4-9
Module 4
Segment 2
Properties of the MA(1) Model and the PACF
4 - 10
−4
−2
0
2
Simulated MA(1) Data with θ1 = 0.9, σa = 1
plot(arima.sim(n=250, model=list(ma=0.90)),
ylab=””)
0
50
100
150
200
250
Time
4 - 11
−4
−2
0
2
4
Simulated MA(1) Data with θ1 = −0.9, σa = 1
plot(arima.sim(n=250, model=list(ma=-0.90)),
ylab=””)
0
50
100
150
200
250
Time
4 - 12
Mean and Variance of the MA(1) Model
Model: Zt = θ0 − θ1at−1 + at,
at ∼ nid(0, σa2)
Mean: µZ ≡ E(Zt) = E(θ0 − θ1at−1 + at ) = θ0 + 0 + 0 = θ0.
2 ≡ Var(Z ) ≡ E[(Z − µ )2 ] = E(Ż 2)
Variance: γ0 ≡ σZ
t
t
Z
t
t
γ0 = E(Żt2)
= E[(−θ1at−1 + at)2]
= E[(θ12a2
t−1
= θ12E(a2
t−1 )
− 2θ1at−1 at
+ a2
t )]
− 2θ1E(at−1at ) + E(a2
t)
= θ12σa2
− 0
= (θ12 + 1)σa2
+ σa2
4 - 13
Autocovariance and Autocorrelation Functions
for the MA(1) Model
Autocovariance:
γk ≡ Cov(Zt, Zt+k ) = E[(Zt − µZ )(Zt+k − µZ )] = E(ŻtŻt+k )
γ1 ≡ E(ŻtŻt+1)
= E[(−θ1at−1 + at)(−θ1at + at+1)]
= E(θ12at−1 at − θ1at−1at+1 − θ1a2
t + at at+1)
= 0
− 0
− θ1σa2 + 0
= −θ1σa2
Thus ρ1 = γγ1 = −θ12 .
(1+θ1 )
0
Using similar operations, it is easy to show that γ2 ≡ E(ŻtŻt+2) = 0
and thus ρ2 = γ2/γ0 = 0. In general, for MA(1), ρk = γk /γ0 =
0 for k > 1.
4 - 14
Partial Autocorrelation Function
For any model, the process PACF φkk can be computed from
the process ACF from ρ1, ρ2, . . . using the recursive formula
from Durbin (1960):
φ1,1 = ρ1
ρk −
φkk =
k−1
X
φk−1 ,j ρk−j
j=1
k−1
X
1−
, k = 2, 3, ...
φk−1,j ρj
j=1
where
φkj = φk−1,j − φkk φk−1,k−j (k = 3, 4, ...; j = 1, 2, ..., k − 1)
This formula is based on the solution of the Yule-Walker
equations (covered later).
Also, the sample PACF can be computed from the sample
ACF. That is, φbkk is a function of ρb1, ρb2, . . ..
4 - 15
Hints For Using Durbin’s formula
φ11 = ρ1
ρ − φ11ρ1
φ22 = 2
1 − φ11ρ1
depends on ρ1
depends on ρ1, ρ2 and φ11
• φ33 depends on ρ1, ρ2, ρ3, φ21, and φ22
• φ44 depends on ρ1, ρ2, ρ3, ρ4, φ31, φ32, and φ33
• φ55 depends on ρ1, ρ2, ρ3, ρ4, ρ5, φ41, φ42, φ43, and φ44
• and so on.
4 - 16
True ACF and PACF for MA(1) Model with θ1 = 0.95
True ACF
0.0
-1.0
True ACF
1.0
ma: 0.95
0
5
10
15
20
15
20
Lag
True PACF
0.0
-1.0
True PACF
1.0
ma: 0.95
0
5
10
Lag
4 - 17
True ACF and PACF for MA(1) Model with θ1 = −0.95
True ACF
0.0
-1.0
True ACF
1.0
ma: -0.95
0
5
10
15
20
15
20
Lag
True PACF
0.0
-1.0
True PACF
1.0
ma: -0.95
0
5
10
Lag
4 - 18
Simulated Realization (MA(1), θ1 = 0.95, n = 75)
Graphical Output from Function iden
Simulated data
-3
-2
-1
w
0
1
2
w= Data
0
10
20
30
40
50
60
70
time
ACF
4.5
0.0
-1.0
ACF
0.5
5.0
1.0
Range-Mean Plot
0
5
10
15
20
15
20
Range
Lag
0.5
0.0
-1.0
3.5
Partial ACF
1.0
4.0
PACF
-0.8
-0.6
-0.4
-0.2
Mean
0.0
0.2
0
5
10
Lag
4 - 19
Simulated Realization (MA(1), θ1 = 0.95, n = 300)
Graphical Output from Function iden
Simulated data
-2
0
w
2
w= Data
0
50
100
150
200
250
300
time
ACF
0.0
5
-1.0
ACF
6
0.5
1.0
Range-Mean Plot
0
5
10
15
20
15
20
Range
Lag
0.5
0.0
-1.0
3
Partial ACF
1.0
4
PACF
-0.1
0.0
0.1
Mean
0.2
0
5
10
Lag
4 - 20
Simulated Realization (MA(1), θ1 = −0.95, n = 75)
Graphical Output from Function iden
Simulated data
-2
0
w
2
4
w= Data
0
10
20
30
40
50
60
70
time
ACF
0.0
-1.0
5
ACF
6
0.5
1.0
Range-Mean Plot
5
10
15
20
15
20
4
0
Range
Lag
0.5
0.0
-1.0
1
2
Partial ACF
1.0
3
PACF
-0.6
-0.4
-0.2
0.0
Mean
0.2
0.4
0.6
0
5
10
Lag
4 - 21
Simulated Realization (MA(1), θ1 = −0.95, n = 300)
Graphical Output from Function iden
Simulated data
w
-4
-2
0
2
4
w= Data
0
50
100
150
200
250
300
time
ACF
5
0.0
-1.0
ACF
6
0.5
1.0
Range-Mean Plot
5
10
15
20
15
20
4
Lag
0.5
0.0
Partial ACF
3
1.0
PACF
-1.0
2
Range
0
-0.5
0.0
0.5
Mean
1.0
0
5
10
Lag
4 - 22
Module 4
Segment 3
Inverting the MA(1) Model and MA(q) Properties
4 - 23
Geometric Power Series
1
= (1 − θ1B)−1 = (1 + θ1B + θ12B2 + · · · )
(1 − θ1B)
This expansion provides a convenient method for re-expressing
parts of some ARMA models. When |θ1| < 1 the series converges.
Similarly,
1
2
=(1 − φ1B)−1 = (1 + φ1B + φ2
1B + · · · )
(1 − φ1B)
Again, when |φ1| < 1 the series converges.
4 - 24
Using the Geometric Power Series to Re-express the
MA(1) Model as an Infinite AR
Żt = (1 − θ1B)at
at = (1 − θ1B)−1 Żt
at = (1 + θ1B + θ12B2 + · · · )Żt
Żt = −θ1Żt−1 − θ12Żt−2 − · · · + at
Thus the MA(1) can be expressed as an infinite AR model.
If −1 < θ1 < 1, then the weight on the old observations is
decreasing with age (having more practical meaning). This
is the condition of “invertibility” for an MA(1) model.
4 - 25
Using the Back-substitution to Re-express the
MA(1) Model as an Infinite AR
Żt = −θ1at−1 + at
at−1 = Żt−1 + θ1at−2
at−2 = Żt−2 + θ1at−3
at−3 = Żt−3 + θ1at−4
Substituting, successively, at−1, at−2, at−3 . . ., shows that
Żt = −θ1Żt−1 − θ12Żt−2 − · · · + at
This method works, more generally, for higher-order MA(q)
models, but the algebra is tedious.
4 - 26
Notes on the MA(1) Model
Given ρ1 = γγ1 = −θ12 we can see
(1+θ1 )
0
• −0.5 ≤ ρ1 ≤ 0.5
• Solving the ρ1 = · · · quadratic equation for θ1 gives
r

1
 −1 ±
−1
2ρ1
(2ρ1 )2
θ1 =

0
ρ1 6= 0
ρ1 = 0
The two solutions are related by
1
θ1 = ′
θ1
For any −0.5 ≤ ρ1 ≤ 0.5, the solution with −1 < θ1 < 1 is
the “invertible” parameter value.
• Substituting ρb1 for ρ1 provides an estimator for θ1.
4 - 27
Mean, Variance, and Covariance of the MA(q) Model
Model: Zt = θ0 + θq (B)at = θ0 − θ1at−1 − · · · − θq at−q + at
Mean: µZ ≡ E(Zt) = E(θ0 − θ1at−1 − · · · − θq at−q + at) = θ0.
Variance: γ0 ≡ Var(Zt) ≡ E[(Zt − µZ )2] = E(Ż 2)
Centered MA(q): Żt = θq (B)at = −θ1at−1 − · · · − θq at−q + at
Var(Zt ) ≡ γ0 = E(Żt2) = (1 + θ12 + · · · + θq2)σa2
Cov(Zt , Zt+k ) = γk = E(ŻtŻt+k ) = · · ·
ρk =
γk
γ0
4 - 28
Re-expressing the MA(q) Model as an Infinite AR
More generally, any MA(q) model can be expressed as
Żt = θq (B)at = (1 − θ1B − θ2B2 − · · · − θq B)at
1
Żt = π(B)Żt = (1 − π1B − π2B2 − . . . )Żt = at
θq (B)
Żt = π1Żt−1 + π2Żt−2 + · · · + at =
∞
X
πk Żt−k + at
k=1
Values of π1, π2, . . . depend on θ1, . . . , θq . For the model to
have practical meaning, the πj values should not remain large
as j gets large. Formally, the invertibility condition is met if
∞
X
i=1
|πj | < ∞
Invertibility of an MA(q) model can be checked by finding the
roots of the polynomial θq (B) ≡ (1−θ1B−θ2B2 −· · ·−θq Bq ) =
0. All q roots must lie outside of the “unit-circle.”
4 - 29
Module 4
Segment 4
Checking Invertibility and Polynomial Roots
4 - 30
Checking the Invertibility of an MA(2) Model
Invertibility of an MA(2) model can be checked by finding
the roots of the polynomial θ2(B) = (1 − θ1B − θ2B2) = 0.
Both roots must lie outside of the “unit-circle.”
B=
−θ1 ±
q
θ12 + 4θ2
2θ2
Roots have the form
z = x + iy
where i =
√
−1. A root is “outside of the unit circle” if
|z| =
q
x2 + y 2 > 1
This method works for any q, but for q > 2 it is best to use
numerical methods to find the q roots.
4 - 31
Roots of (1 − 1.5B − 0.4B2) = 0
arma.root(c(1.5, 0.4))
Coefficients= 1.5, 0.4
Roots and the Unit Circle
•
•
-2
0
Imaginary Part
-10
-15
-4
-20
Function
-5
2
0
4
Polynomial Function versus B
-8
-6
-4
B
-2
0
-4
-2
0
2
4
Real Part
4 - 32
Roots of the Quadratic (1 − 0.5B + 0.9B2) = 0
and the Unit Circle
Unit Circle
0.0
−0.5
−1.0
−1.5
Imaginary Part
0.5
1.0
1.5
Coefficients= 0.5, −0.9
−1.5
−1.0
−0.5
0.0
0.5
1.0
1.5
Real Part
4 - 33
Roots of (1 − 0.5B + 0.9B2) = 0
arma.root(c(0.5, −0.9))
Coefficients= 0.5, -0.9
Roots and the Unit Circle
1.02
1.5
Polynomial Function versus B
0.5
-0.5
Imaginary Part
0.98
0.96
•
-1.5
0.94
Function
1.00
•
0.0
0.2
0.4
B
0.6
-1.5
-0.5
0.5
1.5
Real Part
4 - 34
Roots of (1 − 0.5B + 0.9B2 − 0.1B3 − 0.5B4) = 0
arma.root(c(0.5, −0.9, 0.1, 0.5))
Coefficients= 0.5, -0.9, 0.1, 0.5
Roots and the Unit Circle
1
•
•
•
-1
0
Imaginary Part
-40
-60
-80
•
-100
Function
-20
0
Polynomial Function versus B
-4
-2
0
B
2
-1
0
1
Real Part
4 - 35
Checking the Invertibility of an MA(2) Model
Simple Rule
Both roots lying outside of the “unit-circle” implies
θ2 + θ1 < 1
θ2 < 1 − θ 1
θ2 − θ 1 < 1
θ2 < 1 + θ1
−1 < θ2 < 1
−1 < θ2 < 1
and the inequalities define the MA(2) triangle.
4 - 36
Module 4
Segment 5
MA(2) Model Properties
4 - 37
Autocovariance and Autocorrelation Functions
for the MA(2) Model
γ1 ≡ Cov(Zt, Zt+1) = E[(Zt − µZ )(Zt+1 − µZ )] = E(ŻtŻt+1)
= E[(−θ1at−1 − θ2at−2 + at)(−θ1at − θ2at−1 + at+1)]
2
= E(θ1θ2a2
t−1 − θ1at ) + 0 + 0 + 0 + 0 + 0 + 0 + 0
2
2
= θ1θ2E(a2
)
−
θ
E(a
)
=
(θ
θ
−
θ
)σ
1
1
2
1
t
a
t−1
Thus
γ
θ θ −θ
ρ1 = 1 = 1 22 1 2 .
γ0
1 + θ1 + θ2
Using similar operations, it is easy to show that
−θ2
γ2
=
ρ2 =
γ0
1 + θ12 + θ22
and that, in general, for MA(2), ρk = γk /γ0 = 0 for k > 2.
4 - 38
True ACF and PACF for MA(2) Model with
θ1 = 0.78, θ2 = 0.2
True ACF
0.0
-1.0
True ACF
1.0
ma: 0.78, 0.2
0
5
10
15
20
15
20
Lag
True PACF
0.0
-1.0
True PACF
1.0
ma: 0.78, 0.2
0
5
10
Lag
4 - 39
True ACF and PACF for MA(2) Model with
θ1 = 1, θ2 = −0.95
True ACF
0.0
-1.0
True ACF
1.0
ma: 1, -0.95
0
5
10
15
20
15
20
Lag
True PACF
0.0
-1.0
True PACF
1.0
ma: 1, -0.95
0
5
10
Lag
4 - 40
Simulated Realization (MA(2), θ1 = 0.78, θ2 = 0.2, n = 75)
Graphical Output from Function iden
Simulated data
-3
-2
-1
w
0
1
2
w= Data
0
10
20
30
40
50
60
70
time
ACF
0.0
-1.0
4.6
ACF
4.8
0.5
1.0
Range-Mean Plot
4.4
5
10
15
20
15
20
Lag
0.5
-1.0
0.0
4.0
Partial ACF
1.0
4.2
PACF
3.8
Range
0
-0.04
-0.02
0.0
0.02
Mean
0.04
0.06
0.08
0
5
10
Lag
4 - 41
Simulated Realization
(MA(2), θ1 = 0.78, θ2 = 0.2, n = 300)
Graphical Output from Function iden
Simulated data
-2
0
w
2
4
w= Data
0
50
100
150
200
250
300
time
ACF
0.0
6
-1.0
7
ACF
0.5
1.0
Range-Mean Plot
0
5
10
15
20
15
20
Range
Lag
3
0.5
0.0
-1.0
4
Partial ACF
1.0
5
PACF
-0.2
-0.1
0.0
0.1
Mean
0.2
0
5
10
Lag
4 - 42
Simulated Realization (MA(2), θ1 = 1, θ2 = −0.95, n = 75)
Graphical Output from Function iden
Simulated data
-2
0
w
2
4
w= Data
0
10
20
30
40
50
60
70
time
ACF
0.0
-1.0
6
ACF
0.5
7
1.0
Range-Mean Plot
5
10
15
20
15
20
5
Lag
0.5
-1.0
0.0
Partial ACF
1.0
PACF
4
Range
0
-0.2
0.0
0.2
Mean
0.4
0
5
10
Lag
4 - 43
Simulated Realization
(MA(2), θ1 = 1, θ2 = −0.95, n = 300)
Graphical Output from Function iden
Simulated data
-6
-4
-2
w
0
2
4
w= Data
0
50
100
150
200
250
300
time
ACF
0.0
-1.0
8
ACF
0.5
10
1.0
Range-Mean Plot
0
5
10
15
20
15
20
Range
Lag
0.5
0.0
-1.0
4
Partial ACF
1.0
6
PACF
-0.6
-0.4
-0.2
0.0
Mean
0.2
0.4
0.6
0
5
10
Lag
4 - 44
MA(2) Triangle Relating ρ1 and ρ2 to θ1 and θ2
4 - 45
MA(2) Triangle Showing ACF and PACF Functions
4 - 46