ST414 – Spectral Analysis of
Time Series Data
Lecture 4
13 February 2014
Last Time
• The periodogram
• The smoothed periodogram
• Asymptotic considerations
2
Today’s Objectives
• Estimating the AR parameters
• The AR spectrum
3
AR(1)
𝑋 𝑡 = 𝜙𝑋 𝑡 − 1 + 𝑍 𝑡
𝑍 𝑡 is white noise (0, 𝜎 2 )
𝜎2
𝑓 𝜔 = 2
𝜙 − 2𝜙 cos 2𝜋𝜔 + 1
𝑓 𝜔 =
𝜎2
𝜙 2 − 2𝜙 cos 2𝜋𝜔 + 1
4
Some Preliminaries
X(t) is an ARMA(p,q) process if X(t) is stationary
and if for every t,
𝑝
𝑋 𝑡 −
𝑞
𝜙𝑗 𝑋 𝑡 − 𝑗 = 𝑍 𝑡 +
𝑗=1
𝜃𝑘 𝑍 𝑡 − 𝑘 ,
𝑘=1
where Z(t) is white noise (0, 𝜎 2 ) and the
𝑝
𝑞
𝑗
polynomials (1 − 𝑗=1 𝜙𝑗 𝑧 ) and (1 + 𝑗=1 𝜃𝑗 𝑧 𝑗 )
have no common factors.
5
Some Preliminaries
An ARMA(p,q) process X(t) is said to be
causal if there exists a sequence of
constants {𝜓𝑗 } with ∞
𝑗=0 |𝜓𝑗 | < ∞ such that
𝑋 𝑡 =
∞
𝑗=0 𝜓𝑗 𝑍(𝑡
− 𝑗) for all t.
6
The Sample Mean
𝑇
𝑋 = 𝑇 −1
𝑋(𝑡)
𝑡=1
𝑇
𝑉𝑎𝑟 𝑋 = 𝑇
−1
ℎ=−𝑇
ℎ
1−
𝑇
𝛾(ℎ)
7
The Sample Autocovariance
A “natural” estimator:
𝑇−|ℎ|
𝛾 ℎ = 𝑇 −1
(𝑋 𝑡 + ℎ − 𝑋)(𝑋 𝑡 − 𝑋) ,
𝑡=1
−𝑇 < ℎ < 𝑇
The divisor by T (instead of T-h) ensures
that the sample covariance matrix Γ =
[𝛾 𝑗 − 𝑘 ]𝑗,𝑘 is nonnegative definite.
8
The Yule-Walker Equations
For an AR(p) causal model:
Γ𝑝 𝜙𝑝 = 𝛾𝑝
𝑇
𝛾 0 − 𝜙𝑝 𝛾𝑝 = 𝜎 2 ,
where
Γ𝑝 = 𝛾 𝑗 − 𝑘
𝑝
𝑗,𝑘=1
(𝜙1 , … , 𝜙𝑝 )𝑇
𝑇
𝜙𝑝 =
𝛾𝑝 = (𝛾 1 , … 𝛾 𝑝 )
9
Conditional MLE
Consider a causal AR(1)
𝑋 𝑡 = 𝜙𝑋 𝑡 − 1 + 𝑍 𝑡
with 𝑍 𝑡 𝑖𝑖𝑑 𝑁 0, 𝜎 2 , 𝑡 = 1, … , 𝑇.
Likelihood function:
𝐿 𝜙, 𝜎 2 = 𝑓 𝑋 1 , … , 𝑋 𝑇 𝜙, 𝜎 2
= 𝑓(𝑋 1 )𝑓(𝑋 2 |𝑋 1 ) ∙∙∙ 𝑓(𝑋(𝑇)|𝑋 𝑇 − 1 )
10
Conditional MLE
Note that
𝑋 𝑡 |𝑋 𝑡 − 1 ~𝑁(𝜙𝑋 𝑡 − 1 , 𝜎 2 )
for 𝑡 = 2, … , 𝑇.
Then the conditional (on 𝑋 1 ) likelihood is
𝑇
𝐿 𝜙, 𝜎 2 |𝑋(1) =
𝑓(𝑋(𝑡)|𝑋 𝑡 − 1 )
𝑡=2
2 −(𝑇−1)/2
= (2𝜋𝜎 )
−𝑆 𝜙
exp(
)
2
2𝜎
11
Conditional MLE
𝑇
𝐿 𝜙, 𝜎 2 |𝑋(1) =
𝑓(𝑋(𝑡)|𝑋 𝑡 − 1 )
𝑡=2
2 −(𝑇−1)/2
= (2𝜋𝜎 )
−𝑆 𝜙
exp(
)
2
2𝜎
where
𝑇
(𝑋 𝑡 − 𝜙𝑋(𝑡 − 1))2
𝑆 𝜙 =
𝑡=2
12
Conditional MLE
The conditional MLE approach reduces to a
regression problem!
This approach can be generalised to the
AR(p) model.
13
The AR Spectrum
The spectral density for an AR(p) is
𝜎2
𝑓 𝜔 =
,
2
|Φ(exp −𝑖2𝜋𝜔 )|
where
𝑝
𝜙𝑗 𝑧 𝑗
Φ 𝑧 =1−
𝑗=1
14
The AR Spectrum
Let g 𝜔 be the spectrum of a weakly
stationary process. Then for 𝜀 > 0, there
exists a time series with representation
𝑝
𝑋 𝑡 =
𝜙𝑗 𝑋 𝑡 − 𝑗 + 𝑍 𝑡 ,
𝑗=1
where 𝑍 𝑡 is white noise (0, 𝜎 2 ) such that
𝑓𝑋 𝜔 − 𝑔 𝜔 < 𝜀 for all 𝜔𝜖[−0.5,0.5].
15
The AR Spectrum
Problem: how large must the order be for
the approximation to be reasonable?
16
Model Selection
• PACF
• AIC
𝐴𝐼𝐶 = log 𝜎 2 + (𝑇 + 2𝐾)/𝑇
where K is the number of parameters
• BIC
𝐵𝐼𝐶 = log 𝜎 2 + (𝐾 log(𝑇))/𝑇
17
Example 1
18
Example 1
19
Example 1
20
Example 1
21
Example 1
22
Example 1
23
Example 2
24
Example 2
25
Example 2
26
Example 2
27
Example 2
28
The AR Spectrum
𝑉𝑎𝑟 𝑓 𝜔
2𝑝 2
≈
𝑓 (𝜔)
𝑇
As the order increases:
• the bias decreases, i.e., more complex
spectra can be modeled
• the variance increases linearly
29
The Whittle Likelihood
Parametric spectrum:
𝑓 𝜔 = 𝑓 𝜔; 𝜃
Can we optimize the parameters in the
frequency domain?
30
The Whittle Likelihood
Recall for Gaussian white noise:
𝑅𝑒(𝑑 𝜔𝑗 )
~𝑁(0,0.5diag(𝑓 𝜔𝑗 , 𝑓 𝜔𝑗 )
𝐼𝑚(𝑑 𝜔𝑗 )
In general,
𝑑 𝜔𝑗
𝑑
𝑁 𝐶 (0, 𝑓 𝜔𝑗 )
and approximately independent at distinct
frequencies.
31
The Whittle Likelihood
The Whittle Likelihood:
log 𝐿 𝜃 𝑋
1
≈−
2
0<𝜔𝑗 <0.5
|𝑑 𝜔𝑗 |2
log 𝑓(𝜔𝑗 ; 𝜃) +
𝑓(𝜔𝑗 ; 𝜃)
32
Comparisons
AR spectrum
• Good frequency resolution, even for low-order
models
• Potential for model misspecification
Periodogram
• Frequency resolution is a function of the length
of the time series
• Some form of smoothing is necessary for a
stable estimate
33