Module 4
Stationary Time Series Models Part 1
MA Models and Their Properties
Class notes for Statistics 451: Applied Time Series
Iowa State University
Copyright 2015 W. Q. Meeker.
February 14, 2016
20h 58min
Deviations from the Sample Mean
n
Pn
t=1 zt
4-1
Module 4
Segment 1
4-2
ARMA Notation, Conventions, Expectations, and
other Preliminaries
1955
1955
1957
1957
1959
Year
1963
1965
Change in Inventories
1961
1961
Year
1963
1965
Centered Change in Inventories
1959
1967
1967
1969
1969
Backshift Operator Notation
Backshift operators:
BZt = Zt−1
B2Zt = BBZt = BZt−1 = Zt−2
B3Zt = Zt−3, etc.
If C is a constant, then BC = C
Differencing operators:
(1 − B)Zt = Zt − BZt = Zt − Zt−1
(1 − B)2Zt = (1 − 2B + B2)Zt = Zt − 2Zt−1 + Zt−2
Seasonal differencing operators:
(1 − B4)Zt = Zt − B4Zt = Zt − Zt−4
(1 − B12)Zt = Zt − B12Zt = Zt − Zt−12
4-6
4-4
Change in Business Inventories 1955-1969 and
Centered Change in Business Inventories 1955-1969
10
5
• The sample mean is computed as
b = z̄ =
µ
• We use the centered realization zt − z̄ in the computation
of many statistics. For example
v
uP
u n (z − z̄)2
t
bZ = γ
b0 = SZ = t t=1
σ
n−1
4-3
• Also used in computing sample autocovariances and autocorrelations. Subtracting out the mean (z̄) does not change
the sample variance (γb0), sample autocovariances (γbk ), or
sample autocorrelations (ρbk ) of a realization.
Deviations from the Process Mean
• The stochastic process is Z1, Z2, . . . .
• The process mean is µ = E(Zt).
• Some derivations are simplified by using Żt = Zt − µ to
compute model properties (only the mean has changed).
• In particular,
E(Żt) = E(Zt − µ) = E(Zt) − E(µ) = µ − µ = 0
Similarly,
2
γ0 = Var(Żt ) = Var(Zt −µ) = Var(Zt )+Var(µ) = Var(Zt) = σZ
and
ρk = γk /γ0
γk = Cov(Żt , Żt+k ) = Cov(Zt − µ, Zt+k − µ) = Cov(Zt , Zt+k )
4-5
Billions of Dollars
Billions of Dollars
0
-5
10
5
0
-10
Polynomial Operator Notation
φ(B) = φp(B) ≡ (1 − φ1B − φ2B2 − · · · − φpBp)
is useful for expressing the AR(p) model.
Żt − φ1Żt−1 − φ2Żt−2 = at
(1 − φ1B − φ2B2)Żt = at
φ2(B)Żt = at
For example, with p = 2, an AR(2) model.
or
Żt = φ1Żt−1 + φ2Żt−2 + at
Other polynomial operators:
4-7
θ(B) = θq (B) ≡ (1−θ1B−θ2B2 −· · ·−θq Bq ) [as in Żt = θq (B)at ]
ψ(B) = ψ∞(B) ≡ (1 + ψ1B + ψ2B2 + · · · ) [as in Żt = ψ(B)at ]
π(B) = π∞(B) ≡ (1 − π1B − π2B2 − · · · ) [as in π(B)Żt = at]
Mean (µ) and Constant Term (θ0)
for an ARMA(2,2) Model
150
200
250
General ARMA(p, q) Model in Terms of Żt
Żt ≡ Zt − µ
φp(B)Żt = θq (B)at
(1 − φ B − φ B2 − · · · − φp Bp)Żt =
1
2
(1 − θ1B − θ2B2 − · · · − θq Bq )at
at − θ1at−1 − θ2at−2 − · · · − θq at−q
Żt − φ1Żt−1 − φ2Żt−2 − · · · − φpŻt−p =
or
4-8
Żt = φ1Żt−1+φ2Żt−2+· · ·+φpŻt−p−θ1at−1−θ2at−2−· · ·−θq at−q +at
Module 4
Replace Żt with Żt = Zt − µ and solve for Zt = · · · .
For example, using p = 2 and q = 2, giving an ARMA(2,2)
model:
Segment 2
100
0
50
100
Time
150
200
250
4 - 12
Simulated MA(1) Data with θ1 = −0.9, σa = 1
plot(arima.sim(n=250, model=list(ma=-0.90)),
ylab=””)
4 - 10
Properties of the MA(1) Model and the PACF
Żt = φ1Żt−1 + φ2Żt−2 − θ1at−1 − θ2at−2 + at
and
Zt = µ − φ1µ − φ2µ + φ1Zt−1 + φ2Zt−2 − θ1at−1 − θ2at−2 + at
Zt − µ = φ1(Zt−1 − µ) + φ2(Zt−2 − µ) − θ1at−1 − θ2at−2 + at
Zt = θ0 + φ1Zt−1 + φ2Zt−2 − θ1at−1 − θ2at−2 + at
4-9
where θ0 = µ − φ1µ − φ2µ = µ(1 − φ1 − φ2) is the ARMA model
“constant term.” Also, E(Zt) = µ = θ0/(1 − φ1 − φ2).
Note that E(Zt) = µ = θ0 for an MA(q) model.
50
4 - 11
Simulated MA(1) Data with θ1 = 0.9, σa = 1
plot(arima.sim(n=250, model=list(ma=0.90)),
ylab=””)
0
Time
4
2
0
−2
−4
2
0
−2
−4
at ∼ nid(0, σa2)
Mean and Variance of the MA(1) Model
Model: Zt = θ0 − θ1at−1 + at,
4 - 13
Mean: µZ ≡ E(Zt) = E(θ0 − θ1at−1 + at ) = θ0 + 0 + 0 = θ0.
+ at2)]
2 ≡ Var(Z ) ≡ E[(Z − µ )2 ] = E(Ż 2)
Variance: γ0 ≡ σZ
t
t
Z
t
t
γ0 = E(Żt2)
− 2θ1at−1 at
= E[(−θ1at−1 + at)2]
2
= E[(θ 2at−1
1
2 )
= θ12E(at−1
− 2θ1E(at−1at ) + E(at2)
+ σa2
= θ12σa2
− 0
= (θ12 + 1)σa2
Partial Autocorrelation Function
=
ρk −
φ1,1 = ρ1
φkk
k−1
X
φk−1,j ρj
φk−1 ,j ρk−j
j=1
j=1
k−1
X
1−
, k = 2, 3, ...
For any model, the process PACF φkk can be computed from
the process ACF from ρ1, ρ2, . . . using the recursive formula
from Durbin (1960):
where
φkj = φk−1,j − φkk φk−1,k−j (k = 3, 4, ...; j = 1, 2, ..., k − 1)
This formula is based on the solution of the Yule-Walker
equations (covered later).
5
5
4 - 15
Autocovariance and Autocorrelation Functions
for the MA(1) Model
Autocovariance:
γk ≡ Cov(Zt, Zt+k ) = E[(Zt − µZ )(Zt+k − µZ )] = E(ŻtŻt+k )
γ1 ≡ E(ŻtŻt+1)
−θ1
.
(1+θ12 )
= E[(−θ1at−1 + at)(−θ1at + at+1)]
= E(θ 2at−1 at − θ1at−1at+1 − θ1at2 + atat+1)
1
− 0
− θ1σa2 + 0
= 0
= −θ1σa2
0
Thus ρ1 = γγ1 =
Using similar operations, it is easy to show that γ2 ≡ E(ŻtŻt+2) = 0
and thus ρ2 = γ2/γ0 = 0. In general, for MA(1), ρk = γk /γ0 =
0 for k > 1.
4 - 14
4 - 16
depends on ρ1, ρ2 and φ11
depends on ρ1
Hints For Using Durbin’s formula
φ11 = ρ1
ρ − φ11ρ1
φ22 = 2
1 − φ11ρ1
• φ33 depends on ρ1, ρ2, ρ3, φ21, and φ22
• φ44 depends on ρ1, ρ2, ρ3, ρ4, φ31, φ32, and φ33
• φ55 depends on ρ1, ρ2, ρ3, ρ4, ρ5, φ41, φ42, φ43, and φ44
• and so on.
5
10
Lag
Lag
ma: -0.95
True PACF
10
15
15
20
20
4 - 18
True ACF and PACF for MA(1) Model with θ1 = −0.95
0
5
ma: -0.95
20
0
True ACF
15
20
ma: 0.95
Lag
15
True ACF
10
Lag
ma: 0.95
True PACF
10
1.0
Also, the sample PACF can be computed from the sample
ACF. That is, φbkk is a function of ρb1, ρb2, . . ..
1.0
True ACF and PACF for MA(1) Model with θ1 = 0.95
0
0
4 - 17
True ACF
True PACF
0.0
-1.0
1.0
0.0
-1.0
True ACF
True PACF
0.0
-1.0
1.0
0.0
-1.0
10
Mean
20
-0.2
Range-Mean Plot
-0.4
0.0
30
0.2
0.2
0.4
30
w= Data
time
40
0
0
Simulated data
0.6
50
5
5
60
ACF
10
Lag
PACF
10
Lag
15
15
70
70
20
20
50
Mean
0.1
Range-Mean Plot
0.0
100
0.2
time
150
0
0
5
5
200
Lag
10
PACF
Lag
10
ACF
250
15
20
20
4 - 20
15
300
Simulated Realization (MA(1), θ1 = 0.95, n = 300)
Graphical Output from Function iden
0
0
50
Mean
0.5
Range-Mean Plot
0.0
100
1.0
w= Data
5
200
0
5
Simulated data
time
150
0
Geometric Power Series
Again, when |φ1| < 1 the series converges.
ACF
10
Lag
PACF
10
Lag
250
15
20
20
300
15
4 - 22
4 - 24
1
2 B2 + · · · )
=(1 − φ1B)−1 = (1 + φ1B + φ1
(1 − φ1B)
Similarly,
This expansion provides a convenient method for re-expressing
parts of some ARMA models. When |θ1| < 1 the series converges.
1
= (1 − θ1B)−1 = (1 + θ1B + θ12B2 + · · · )
(1 − θ1B)
-0.5
Simulated Realization (MA(1), θ1 = −0.95, n = 300)
Graphical Output from Function iden
-0.1
w= Data
ACF
60
20
4 - 19
20
Simulated data
50
10
Lag
PACF
10
Lag
15
w= Data
40
5
15
w
Simulated data
time
0
0
5
Simulated Realization (MA(1), θ1 = 0.95, n = 75)
Graphical Output from Function iden
0
-0.6
20
Range-Mean Plot
0.0
Mean
1.0
0.5
-0.8
10
-0.2
4 - 21
1.0
0.5
0.0
-1.0
1.0
0.5
0.0
-1.0
ACF
Partial ACF
ACF
Partial ACF
Simulated Realization (MA(1), θ1 = −0.95, n = 75)
Graphical Output from Function iden
0
-0.4
Module 4
Segment 3
4 - 23
Inverting the MA(1) Model and MA(q) Properties
-0.6
2
0
-2
6
5
4
3
4
2
0
Range
w
Range
1.0
0.5
1.0
0.5
-2
-4
6
5
4
0.0
-1.0
1.0
0.5
0.0
-1.0
ACF
Partial ACF
0.0
-1.0
1.0
0.5
0.0
-1.0
3
2
ACF
Partial ACF
0.0
-1.0
1.0
0.5
0.0
-1.0
2
1
0
-1
-2
-3
5.0
4.5
4.0
3.5
4
w
Range
w
Range
2
0
-2
6
5
4
3
2
1
Using the Geometric Power Series to Re-express the
MA(1) Model as an Infinite AR
Żt = (1 − θ1B)at
at = (1 − θ1B)−1 Żt
at = (1 + θ1B + θ12B2 + · · · )Żt
Żt = −θ1Żt−1 − θ12Żt−2 − · · · + at
Thus the MA(1) can be expressed as an infinite AR model.
−θ1
we can see
(1+θ12 )
Notes on the MA(1) Model
0
ρ1 6= 0
ρ1 = 0
4 - 25
If −1 < θ1 < 1, then the weight on the old observations is
decreasing with age (having more practical meaning). This
is the condition of “invertibility” for an MA(1) model.
0
Given ρ1 = γγ1 =
• −0.5 ≤ ρ1 ≤ 0.5
r

1
 −1 ±
−1
2ρ1
(2ρ1 )2

• Solving the ρ1 = · · · quadratic equation for θ1 gives
θ1 =
The two solutions are related by
1
θ1 = ′
θ1
For any −0.5 ≤ ρ1 ≤ 0.5, the solution with −1 < θ1 < 1 is
the “invertible” parameter value.
• Substituting ρb1 for ρ1 provides an estimator for θ1.
4 - 27
Re-expressing the MA(q) Model as an Infinite AR
More generally, any MA(q) model can be expressed as
Żt = θq (B)at = (1 − θ1B − θ2B2 − · · · − θq B)at
k=1
∞
X
πk Żt−k + at
1
Żt = π(B)Żt = (1 − π1B − π2B2 − . . . )Żt = at
θq (B)
Żt = π1Żt−1 + π2Żt−2 + · · · + at =
|πj | < ∞
Values of π1, π2, . . . depend on θ1, . . . , θq . For the model to
have practical meaning, the πj values should not remain large
as j gets large. Formally, the invertibility condition is met if
∞
X
i=1
4 - 29
Invertibility of an MA(q) model can be checked by finding the
roots of the polynomial θq (B) ≡ (1−θ1B−θ2B2 −· · ·−θq Bq ) =
0. All q roots must lie outside of the “unit-circle.”
Using the Back-substitution to Re-express the
MA(1) Model as an Infinite AR
Żt = −θ1at−1 + at
at−1 = Żt−1 + θ1at−2
at−2 = Żt−2 + θ1at−3
at−3 = Żt−3 + θ1at−4
Substituting, successively, at−1, at−2, at−3 . . ., shows that
Żt = −θ1Żt−1 − θ12Żt−2 − · · · + at
This method works, more generally, for higher-order MA(q)
models, but the algebra is tedious.
4 - 26
Mean, Variance, and Covariance of the MA(q) Model
Model: Zt = θ0 + θq (B)at = θ0 − θ1at−1 − · · · − θq at−q + at
Mean: µZ ≡ E(Zt) = E(θ0 − θ1at−1 − · · · − θq at−q + at) = θ0.
Variance: γ0 ≡ Var(Zt) ≡ E[(Zt − µZ )2] = E(Ż 2)
4 - 28
Centered MA(q): Żt = θq (B)at = −θ1at−1 − · · · − θq at−q + at
Var(Zt ) ≡ γ0 = E(Żt2) = (1 + θ12 + · · · + θq2)σa2
γk
γ0
Cov(Zt , Zt+k ) = γk = E(ŻtŻt+k ) = · · ·
ρk =
Module 4
Segment 4
Checking Invertibility and Polynomial Roots
4 - 30
Checking the Invertibility of an MA(2) Model
−θ1 ±
q
−0.5
θ12 + 4θ2
q
x2 + y 2 > 1
Unit Circle
Real Part
0.0
0.5
Coefficients= 0.5, −0.9
−1.0
•
1.0
-1
1.5
•
•
0
Real Part
1
•
Roots and the Unit Circle
Coefficients= 0.5, -0.9, 0.1, 0.5
2
4 - 31
4 - 33
-4
B
-2
0.4
0
0.6
-2
•
Real Part
0
2
4
Roots and the Unit Circle
-4
-1.5
•
•
0.5
Real Part
-0.5
4 - 32
4 - 36
4 - 34
1.5
Roots and the Unit Circle
Coefficients= 0.5, -0.9
Roots of (1 − 0.5B + 0.9B2) = 0
arma.root(c(0.5, −0.9))
B
Polynomial Function versus B
0.2
θ2 < 1 − θ 1
θ2 < 1 + θ1
−1 < θ2 < 1
and the inequalities define the MA(2) triangle.
−1 < θ2 < 1
θ2 − θ 1 < 1
θ2 + θ1 < 1
Both roots lying outside of the “unit-circle” implies
Checking the Invertibility of an MA(2) Model
Simple Rule
0.0
•
Coefficients= 1.5, 0.4
Roots of (1 − 1.5B − 0.4B2) = 0
arma.root(c(1.5, 0.4))
-6
Polynomial Function versus B
-8
4
2
Invertibility of an MA(2) model can be checked by finding
the roots of the polynomial θ2(B) = (1 − θ1B − θ2B2) = 0.
Both roots must lie outside of the “unit-circle.”
B=
2θ2
z = x + iy
|z| =
−1. A root is “outside of the unit circle” if
Roots have the form
√
0
Imaginary Part
where i =
−1.5
B
4 - 35
Imaginary Part
-2
0
0
-5
-4
1.5
0.5
-0.5
Function
Function
-10
-15
-1.5
This method works for any q, but for q > 2 it is best to use
numerical methods to find the q roots.
Function
-20
1.02
1.00
0.98
0.96
Roots of the Quadratic (1 − 0.5B + 0.9B2) = 0
and the Unit Circle
Imaginary Part
Roots of (1 − 0.5B + 0.9B2 − 0.1B3 − 0.5B4) = 0
arma.root(c(0.5, −0.9, 0.1, 0.5))
-2
Imaginary Part
0.94
1.5
1.0
0.5
0.0
−0.5
−1.0
−1.5
Polynomial Function versus B
-4
1
0
-1
0
-20
-40
-60
-80
-100
20
0.04
Range-Mean Plot
Mean
0.02
0.06
60
ACF
10
Lag
PACF
10
Lag
15
15
70
20
20
Autocovariance and Autocorrelation Functions
for the MA(2) Model
γ1 ≡ Cov(Zt, Zt+1) = E[(Zt − µZ )(Zt+1 − µZ )] = E(ŻtŻt+1)
= E[(−θ1at−1 − θ2at−2 + at)(−θ1at − θ2at−1 + at+1)]
-0.2
0
-0.1
50
5
5
0.1
Range-Mean Plot
0.0
100
0.2
10
Lag
Lag
ma: 1, -0.95
True PACF
10
w= Data
15
20
20
5
200
0
5
Simulated data
time
150
0
ACF
10
Lag
PACF
10
Lag
250
4 - 38
4 - 40
20
300
15
20
4 - 42
15
Simulated Realization
(MA(2), θ1 = 0.78, θ2 = 0.2, n = 300)
Graphical Output from Function iden
15
True ACF and PACF for MA(2) Model with
θ1 = 1, θ2 = −0.95
and that, in general, for MA(2), ρk = γk /γ0 = 0 for k > 2.
−θ2
γ
ρ2 = 2 =
γ0
1 + θ12 + θ22
Using similar operations, it is easy to show that
Thus
2
= E(θ1θ2at−1
− θ1at2) + 0 + 0 + 0 + 0 + 0 + 0 + 0
50
5
5
0
ma: 1, -0.95
20
True ACF
15
ma: 0.78, 0.2
Lag
w= Data
40
Simulated data
time
0
0
0
True ACF
10
ma: 0.78, 0.2
Lag
20
γ
θ θ −θ
ρ1 = 1 = 1 22 1 2 .
γ0
1 + θ1 + θ2
Module 4
4 - 37
2
= θ1θ2E(at−1
) − θ1E(at2) = (θ1θ2 − θ1)σa2
30
0.08
1.0
0.0
-1.0
1.0
0.0
-1.0
True PACF
10
15
4 - 39
Segment 5
MA(2) Model Properties
5
5
True ACF and PACF for MA(2) Model with
θ1 = 0.78, θ2 = 0.2
0
10
0.0
4
2
0
-0.02
Mean
1.0
0.5
1
True ACF
True PACF
w
Range
2
Simulated Realization (MA(2), θ1 = 0.78, θ2 = 0.2, n = 75)
Graphical Output from Function iden
0
-0.04
4 - 41
ACF
Partial ACF
1.0
0
-2
7
6
0.0
-1.0
1.0
0.5
0.0
-1.0
ACF
Partial ACF
0.5
0.0
-1.0
1.0
0.5
0.0
-1.0
5
4
3
1.0
0.0
-1.0
1.0
0.0
-1.0
True ACF
True PACF
w
Range
0
-1
-2
-3
4.8
4.6
4.4
4.2
4.0
3.8
10
Mean
30
0.4
w= Data
40
Simulated data
time
0
0
50
5
5
60
ACF
10
Lag
PACF
10
Lag
15
15
70
20
20
4
2
0
-2
-0.6
0
-0.4
50
Range-Mean Plot
0.0
0.2
100
0.4
w= Data
time
150
0
5
5
200
0
Simulated data
0.6
ACF
10
Lag
PACF
10
250
20
300
15
20
4 - 44
15
Simulated Realization
(MA(2), θ1 = 1, θ2 = −0.95, n = 300)
Graphical Output from Function iden
-0.2
Lag
4 - 46
MA(2) Triangle Showing ACF and PACF Functions
Mean
1.0
0.5
0.0
20
0.2
Range-Mean Plot
0.0
w
Range
4
Simulated Realization (MA(2), θ1 = 1, θ2 = −0.95, n = 75)
Graphical Output from Function iden
0
-0.2
4 - 43
MA(2) Triangle Relating ρ1 and ρ2 to θ1 and θ2
4 - 45
ACF
Partial ACF
1.0
-4
-6
10
8
6
4
-1.0
1.0
0.5
0.0
-1.0
ACF
Partial ACF
0.5
0.0
-1.0
1.0
0.5
0.0
-1.0
w
Range
2
0
-2
7
6
5
4