STTN326
Analysis of Dependent Data
Gerrit Grobler
Modisane Seitshiro
30 September 2020
Subject Group Statistics, North-West University, Potchefstroom Campus
Subject Group Statistics, North-West University, Vanderbijlpark Campus
Chapter 3. Autocorrelation and partial autocorrelation
3. Autocorrelation and partial autocorrelation
3.1 ACF of stationary processes
3.2 Partial ACF of stationary processes
3.3 Estimation of the mean, autocovariance and ACF
3.4 Testing for autocorrelation and model identification
Chapter 3. Autocorrelation and partial autocorrelation
Upon completion of this study unit you should be able to understand the following
concepts:
I Section 3.1: ACF of a general stationary process.
I Section 3.1: The recurrence equation for ACF of AR processes.
I Section 3.2: The Partial ACF (PACF) of a stationary process.
I Section 3.3: The sample mean, sample ACF and sample PACF and it distributional
properties.
I Section 3.4: Hypothesis tests for autocorrelation.
I Section 3.4: Model identification using sample ACF and sample PACF.
Study material
Shumway et al. 2011: Chapter 3.4
Box-Jenkins approach1
The Box-Jenkins modelling approach is an iterative process of model identification
and selection, parameter estimation, model checking and forecasting. The approach is
executed using the following steps:
1. Data preparation involves transformations and differencing.
2. Model selection uses ACF and PACF graphs from transformed and differenced
data to identify potential models.
3. Parameter estimation involve estimating parameters of the model using
methods such as method of moments, Maximum Likelihood (ML) and Conditional
Least Squares (LS).
4. Model diagnostics or checking involves testing the assumptions of the model to
identify any areas where the model is inadequate. Upon a model been
inadequate, return to step 2 in order to identify a better model.
5. Model forecasting is the main end goal purpose of the approach after all the
steps are executed.
1
Box, G. E. P. and Jenkins, G. M. (1970). Time Series Analysis Forecasting and Control. San Francisco: Holden-Day.
Study unit 3.1
3.1 ACF of stationary processes
By the end of this section you will need to be able to:
I Derive and apply the general formula for the ACF of stationary ARMA processes.
I State and derive the recurrence equation for the autocovariance function (and
ACF) of an AR(p) model.
I Apply the recurrence equation to calculate the ACF of a specific AR model.
I Derive the ACF of an ARMA(1,1) model.
Chapter 3. Autocorrelation and partial autocorrelation
I We will be using ARMA models to describe time series dynamics:
φ(B)Xt = θ(B)Wt
or
Xt = φ1 Xt−1 + . . . + φp Xt−p + Wt + θ1 Wt−1 + . . . + θq Wt−q
I Model must be causal, stationary and invertible
I Equivalently, all roots of φ(z) = 0 and θ(z) = 0 have |z| > 1.
I Next, want to know the dependence structure (i.e. ACF) that we get from an ARMA
model
I Want to match that to sample ACF from data
3.1 ACF of stationary processes
Theorem 3.1
The ACF of a stationary process Xt =
P∞
j=0 ψj Wt−j is given by
P∞
ρ(h) =
j=0 ψj ψj+h
P∞ 2 , h = 0, 1, 2, ...
j=0 ψj
I This is a result of (See proof of Theorem 2.2)
γ(h) = σw2
∞
X
ψj ψj+h .
j=0
I To apply this method calculate ψj weights from: ψ(B) = φ−1 (B)θ(B).
I If model has MA part only, then ψ(B) = θ(B) ⇒ ψj = θj , j = 1, ..., q.
I If model has AR part only, then ψ(B) = φ−1 (B) ⇒ infinite number of ψj weights.
I To apply this method to either AR or ARMA model we can approximate ρ(h) using
only a large but finite number of ψj weights.
Example
Example 3.1
Find an expression for the ACF for a MA(2) model.
A recurrence equation for the autocovariance function of an AR model
3.1 ACF of stationary processes
I To calculate the ACF of an AR model we find and solve a recurrence equation
directly in terms of ρ(h)
Theorem 3.2
A recurrence equation for the autocovariance function of an AR(p) model is given by
γ(h) = φ1 γ(h − 1) + φ2 γ(h − 2) + ... + φp γ(h − p), h = 1, 2, ...
(1)
Proof. Let {Xt } be an AR(p) process, such that
Xt+h = φ1 Xt+h−1 + φ2 Xt+h−2 + ... + φp Xt+h−p + Wt+h ,
where {Wt } is a white noise process. Multiplying both sides with Xt and taking
expectations we have
E (Xt Xt+h ) = φ1 E Xt Xt+h−1 + φ2 E Xt Xt+h−2 + ... + φp E Xt Xt+h−p .
Assuming, without loss in generality, that E (Xt ) = 0 we obtain the result that
Cov (Xt , Xt+h ) = φ1 Cov Xt , Xt+h−1 +φ2 Cov Xt , Xt+h−2 +...+φp Cov Xt , Xt+h−p ,
since Cov Xt , Xt+h−i = E Xt Xt+h−i , for i = 0, 1, ..., p
ACF of AR model
I By dividing with γ(0) in eq. (1) we get the following result
Corollary 3.3
A recurrence equation for the ACF of an AR(p) model is given by
ρ(h) = φ1 ρ(h − 1) + φ2 ρ(h − 2) + ... + φp ρ(h − p), h = 1, 2, ...
I We first use this recurrence equation to find {ρ(1), ρ(2), ..., ρ(p − 1)} by setting
up p − 1 equations with p − 1 unknowns.
I Secondly, we use the recurrence equation to find {ρ(p), ρ(p + 1), ...} by
substitution.
I Hint: ρ(0) = 1 and ρ(h) = ρ(−h).
ACF of AR(1) model
Example 3.2
The ACF of an AR(1) model is given by
ρ(h) = φh
Example
Example 3.3
Find the ACF for h = 1, 2, 3 of the AR model given by
Xt =
1
1
Xt−1 −
Xt−3 + Wt
2
10
Self evaluation exercise
Self Evaluation Exercise 3.1
Find the ACF for h = 1, 2, 3 of the AR model given by
Xt =
1
3
Xt−1 −
Xt−2 + Wt
2
10
ACF of ARMA model
Theorem 3.4
The ACF of an ARMA(1,1) model is given by
ρ(h) = φh−1
(φ + θ)(1 + φθ)
, h≥1
1 + 2φθ + θ2
Proof. Let {Xt } be an ARMA(1,1) process, such that Xt = φXt−1 + Wt + θWt−1 . The
autocovariance function of {Xt } is given by
γ(h) = Cov(Xt+h , Xt )
= Cov φXt+h−1 + Wt+h + θWt+h−1 , Xt
= φ Cov Xt+h−1 , Xt + Cov (Wt+h , Xt ) + θ Cov Wt+h−1 , Xt
= φγ(h − 1) + Cov (Wt+h , Xt ) + θ Cov Wt+h−1 , Xt .
The causal representation for an ARMA(1,1) model is given by (see Theorem 2.7)
Xt = Wt + (φ + θ)
∞
X
φj−1 Wt−j .
j=1
Substituting this into γ(h) above we have
γ(h) = φγ(h − 1) + Cov (Wt+h , Wt + (φ + θ)Wt−1 + ...)
+ θ Cov Wt+h−1 , Wt + (φ + θ)Wt−1 + ... .
Proof(continued). For h = 0 we have
γ(0) = φγ(1) + Cov (Wt , Wt ) + θ Cov (Wt−1 , (φ + θ)Wt−1 )
= φγ(1) + σw2 [1 + θ(φ + θ)]
For h = 1 we have
γ(1) = φγ(0) + θ Cov (Wt , Wt ) = φγ(0) + σw2 θ.
Substituting this into γ(0) above we get
γ(0) = φ2 γ(0) + σw2 φθ + σw2 [1 + θ(φ + θ)] .
Solving for γ(0) we obtain
γ(0) = σw2
1 + 2φθ + θ2
,
1 − φ2
and
1 + 2φθ + θ2
+θ
1 − φ2
(φ + θ)(1 + φθ)
.
= σw2
1 − φ2
γ(1) = σw2
φ
For h ≥ 2 we have
γ(h) = φγ(h − 1).
Proof(continued). Dividing by γ(0) we obtain the ACF as


1
if h = 0

 (φ+θ)(1+φθ)


if h = 1

2

1+2φθ+θ
 (φ+θ)(1+φθ)



if h = 2
 φ 1+2φθ+θ2
2 (φ+θ)(1+φθ)
ρ(h) =
φ
if
h=3 ,
2
1+2φθ+θ



 .
.




 .
.



.
.
from where we obtain the final result that
ρ(h) = φh−1
(φ + θ)(1 + φθ)
, h≥1
1 + 2φθ + θ2
ACF of ARMA model
I For general ARMA(p,q) models we have the following recurrence equation for γ(h)
that can be used to obtain the ACF:
ρ(h) = φ1 ρ(h − 1) + φ2 ρ(h − 2) + ... + φp ρ(h − p), h ≥ max(p, q + 1)
with initial conditions:
γ(h) =
p
X
j=1
φj γ(h − j) + σw2
q
X
θj ψj−h , 0 ≤ h < max(p, q + 1)
j=h
I To apply this method the values for ψj needs to be derived for 0 < j ≤ q.
Study unit 3.2
3.2 Partial ACF of stationary processes
By the end of this section you will need to be able to:
I Define a the partial autocorrelation function (PACF) for a process.
I Derive the PACF of an AR(1) process
I Distinguish between the behavioural properties of the PACF and ACF for various
stationary processes.
ACF for MA vs AR and ARMA models
I For pure MA(q) model, ρ(h) = 0 for h > q.
I The result of the ACF for an ARMA(1,1) model, shows that ρ(h) tails off to 0 as h
increases.
I This makes it difficult to distinguish between an AR and ARMA model when using
the sample ACF to identify models.
I The partial autocorrelation function (PACF) is another dependence measure that
can expose differences between pure AR and ARMA models.
PACF - motivation
I If {Xt } is an AR(1) model given by.
Xt = φXt−1 + Wt ,
then the covariance between Xt and Xt+2 is non-zero as a result of the
relationship that Xt and Xt+2 have with Xt+1 .
I We want to have a measure of linear dependence when the impact of in-between
variables are taken away.
I For lag h the best linear predictors of Xt and Xt+h , based on {Xt+1 , ..., Xt+h−1 }, is
given by
X̂t+h = β1 Xt+h−1 + β2 Xt+h−2 + ... + βh−1 Xt+1
X̂t = β1 Xt+1 + β2 Xt+2 + ... + βh−1 Xt+h−1 .
I These predictors are best in that they minimise the mean square errors
2
2
E Xt − X̂t
and E Xt+h − X̂t+h .
Definition of PACF
Definition 3.1
The partial autocorrelation function (PACF) of a stationary time series {Xt }, denoted
by ϕhh for h = 1, 2, ... is defined as:
ϕ11 = Cor (Xt+1 , Xt ) = ρ(1)
ϕhh = Cor Xt+h − X̂t+h , Xt − X̂t for h ≥ 2,
where X̂t+h and X̂t are the best linear predictors of Xt and Xt+h .
PACF of AR(p) model i
Theorem 3.5
The PACF of an AR(1) model is given by
(
ϕhh =
φ,
0,
if h = 1
.
if h ≥ 2
Proof. For an AR(1) model we have
ϕ11 = ρ(1) = φ
To derive an expression for ϕ22 , we define the best linear predictors of Xt+2 and Xt as
X̂t+2 = βXt+1 , and X̂t = βXt+1 ,
where β minimises
2
2
f (β) = E Xt+2 − X̂t+2 , and g(β) = E Xt − X̂t .
PACF of AR(p) model ii
Therefore,
f (β) = E (Xt+2 − βXt+1 )2
2
2
= E Xt+2
− 2βXt+2 Xt+1 + βXt+1
2
= γ(0) − 2βγ(1) + β 2 γ(0).
Setting the first derivative of f to zero
∂f
= −2γ(1) + 2βγ(0) = 0,
∂β
we obtain
β=
γ(1)
= ρ(1) = φ.
γ(0)
Similarly, setting the first derivative of g to zero we get the same result. Therefore,
X̂t+2 = φXt+1 , and X̂t = φXt+1 ,
PACF of AR(p) model iii
and
ϕ22 = Cor Xt+2 − X̂t+2 , Xt − X̂t
= Cor (Xt+2 − φXt+1 , Xt − φXt+1 )
= Cor (Wt+2 , Xt − φXt+1 )
= 0.
It can also be shown, for h ≥ 3 that ϕhh = 0
PACF of AR(p) model
I The PACF of an AR(p) model has the following properties
I ϕhh not necessarily zero for 1 ≤ h < p.
I ϕhh = ϕpp = φp for h = p.
I ϕhh = 0 for h > p.
I For MA / ARMA model, the PACF is not necessarily 0, but tails off to 0 as h
increases.
PACF of MA(1) model
Figure 33: PACF of MA1 model
1
2
> ma1pacf = ARMAacf (ma = c ( . 7 ) , l a g . max = 3 6 , p a c f =TRUE )
> p l o t ( ma1pacf , type = ” h ” , main = ” T h e o r e t i c a l PACF o f MA ( 1 ) w i t h t h e t a = 0 . 7 ” )
PACF of ARMA(1,1) model
Figure 34: PACF of ARMA(1,1) model
1
2
> arma11pacf = ARMAacf ( ar = c ( 0 . 3 ) ,ma = c ( . 7 ) , l a g . max = 3 6 , p a c f =TRUE )
> p l o t ( arma11pacf , type = ” h ” , main = ” T h e o r e t i c a l PACF o f ARMA ( 1 , 1 ) w i t h phi = 0 . 3 and
theta = 0.7 ” )
Summary of ACF and PACF
This table summarises the behaviour of the ACF and PACF for AR, MA and ARMA models
ACF
PACF
AR(p)
Tails off
Cuts off (for h > p)
MA(q)
Cuts off (for h > q)
Tails off
ARMA(p,q)
Tails off
Tails off
Example
1
2
> ma3acf = ARMAacf (ma = c ( 0 . 3 , 0 . 5 , 0 . 7 ) , l a g . max = 3 6 )
> ma3pacf = ARMAacf (ma = c ( 0 . 3 , 0 . 5 , 0 . 7 ) , l a g . max = 3 6 , p a c f =TRUE )
Example
1
2
> a r 1 a c f = ARMAacf ( ar = c ( 0 . 7 ) , l a g . max = 3 6 )
> a r 1 p a c f = ARMAacf ( ar = c ( 0 . 7 ) , l a g . max = 3 6 , p a c f =TRUE )
Example
1
2
> arma22acf = ARMAacf ( ar = c ( 0 . 3 , 0 . 2 ) ,ma= c ( 0 . 3 , 0 . 5 ) , l a g . max = 3 6 )
> arma22pacf = ARMAacf ( ar = c ( 0 . 3 , 0 . 2 ) ,ma= c ( 0 . 3 , 0 . 5 ) , l a g . max = 3 6 , p a c f =TRUE )
Home work 3
1. Find the autocovariance function (ACVF) of the process
1
1
1
Xt = 1 − B
1+ B
1 − B Wt ,
2
2
4
where Wt ∼ WN(0, 1).
2. Find the autocorrelation function (ACF) of the following time series. Also, specify
whether the model is invertible?
2.1 Xt = Wt + 25 Wt−1 − 23 Wt−2 , where Wt ∼ WN(0, 2).
1 e
et − 1 W
e
e
2.2 Xt = W
6 t−1 − 6 Wt−2 , where Wt ∼ WN(0, 4).
3. Find ρ(h), h = 0, 1, 2, 3, 4 for the following AR models.
3.1 Xt = 0.3Xt−1 − 0.25Xt−2 + Wt
3.2 Xt = − 21 Xt−1 + 51 Xt−2 − 41 Xt−3 + Wt
3.3 Xt = −2.5Xt−1 + 1.5Xt−2 + Wt
4. Find the autocovariance function (ACVF), for h = 0, 1, . . . 5, of the following
stationary AR process
Xt − 0.2Xt−1 + 0.4Xt−2 = Wt ,
where Wt ∼ WN(0, 1).
Study unit 3.3
3.3 Estimation of the mean, autocovariance and ACF
By the end of this section you will need to be able to:
I Define the sample mean, sample autocovariance function and sample ACF.
I Derive the variance of the sample mean, when observations are a realisation of a
stationary process.
I Show that the sample autocovariance is a biased estimator of the autocovariance
functions.
I State the asymptotic distributions of the sample ACF under certain conditions.
Sample mean
Suppose you have n observations of a time series x1 , x2 , ..., xn .
I The sample mean is given by
n
x̄ =
1 X
xt
n t=1
I The sample mean is a natural unbiased estimator of the mean µ = E(Xt ), where
{Xt } is a stationary process.
Variance of sample mean i
Theorem 3.6
Suppose {Xt } is a stationary process and x1 , x2 , ..., xn are n observations from this
process. The variance of the sample mean x̄ is given by
n−1
Var (x̄) =
1
2 X
γ(0) + 2
(n − h)γ(h)
n
n
h=1
Proof Since {Xt } is stationary, we have the variance of the sample mean x̄ given by
!
n
1 X
Var (x̄) = Var
xt
n t=1
!
n
n
X
X
1
= 2 Cov
xt ,
xs
n
s=1
t=1
n
=
=
n
1 XX
Cov (xt , xs )
n2 t=1 s=1
1
(nγ(0) + (n − 1)γ(1) + (n − 2)γ(2) + ... + γ(n − 1)
n2
+(n − 1)γ(−1) + (n − 2)γ(−2) + ... + γ(1 − n)) ,
Variance of sample mean ii
Since the autocovariance function γ(h) is symmetric, we have that
1
(nγ(0) + 2(n − 1)γ(1) + 2(n − 2)γ(2) + ... + 2γ(n − 1))
n2
n−1
2 X
1
(n − h)γ(h)
= γ(0) + 2
n
n
Var (x̄) =
h=1
Sample autocovariance function
Suppose you have n observations of a time series x1 , x2 , ..., xn .
Definition 3.2
The sample autocovariance function is defined by
n−h
γ̂(h) =
1 X
(xt − x̄) (xt+h − x̄)
n t=1
I This estimator of γ(h) is biased
biasedness of sample autocovariance functions i
Theorem 3.7
Suppose {Xt } is a stationary process and x1 , x2 , ..., xn are n observations from this
process. The sample autocovariance function is a biased estimator for the
population autocovariance function γ(h) of {Xt }.
Proof First note that
n−h
X
(xt − x̄) (xt+h − x̄)
t=1
=
n−h
X
({xt − µ} − {x̄ − µ}) ({xt+h − µ} − {x̄ − µ})
t=1
=
n−h
X
t=1
≈
n−h
X
t=1
(xt − µ) (xt+h − µ) − (x̄ − µ)
n−h
X
(xt+h − µ) − (x̄ − µ)
t=1
(xt − µ) (xt+h − µ) − (n − h) (x̄ − µ)2 .
n−h
X
t=1
(xt − µ) +
n−h
X
t=1
(x̄ − µ)2
biasedness of sample autocovariance functions ii
Now, the expected value of γ̂(h) is given by
n−h
E (γ̂(h)) =
1 X
n−h
E{(xt − µ) (xt+h − µ)} −
E{(x̄ − µ)2 }
n t=1
n
n−h
1 X
n−h
γ(h) −
Var (x̄)
n t=1
n
n−h
n−h
=
γ(h) −
Var (x̄)
n
n
=
Sample ACF
Definition 3.3
The sample autocorrelation function is defined by
ρ̂(h) =
γ̂(h)
γ̂(0)
Theorem 3.8
Suppose {Xt } is an i.i.d. sequence satisfying E Xt2 < ∞, then ρ̂(h) is asymptotically
1
normal with mean zero and variance n for any fixed positive integer h.
Theorem 3.9 (Bartlett’s theorem)
If Xt is a weakly stationary time series satisfying
Xt = µ +
q
X
ψi Wt−i ,
i=0
where ψ0 = 1 and {Wt } is a sequence of i.i.d. random variables with mean zero, then
ρ̂(h) is asymptotically normal with mean zero and variance
1+2
Pq
h=1
n
ρ(h)2
, for h > q
Study unit 3.4
3.4 Testing for autocorrelation and model identification
By the end of this section you will need to be able to:
I Formulate a test for individual ACF as well as Portmanteau tests.
I Detect non-stationarity using the sample ACF.
I Apply the test for individual ACF as well as Portmanteau tests to test for
autocorrelation.
I Use the sample ACF and PACF for model identification.
Hypothesis Test 3.1 (Testing for individual ACF)
Hypotheses:
H0 : ρ(h) = 0 vs Ha : ρ(h) 6= 0
Test statistic and asymptotic distribution:
t=
√
nρ̂(h) ∼ N(0, 1)
Portmanteau Tests
Hypothesis Test 3.2 (Box-Pierce Test)
(Testing for a group of autocorrelations)
Hypotheses:
H0 : ρ(1) = ρ(2) = ... = ρ(m) = 0 vs H0 : ρ(i) 6= 0 for some i = 1, 2, ..., m
Test statistic and asymptotic distribution: Q∗ (m) = n
Pm
2
2
h=1 ρ̂(h) ∼ χm
Hypothesis Test 3.3 (Ljung-Box Test)
(Testing for a group of autocorrelations)
Hypotheses:
H0 : ρ(1) = ρ(2) = ... = ρ(m) = 0 vs H0 : ρ(i) 6= 0 for some i = 1, 2, ..., m
Test statistic and asymptotic distribution: Q(m) = n(n + 2)
ρ̂(h)2
2
h=1 n−h ∼ χm
Pm
I Choosing m = log n provides better power performance.
I You can also choose a range of values for m and apply this test on all of the
values within the range.
Testing autocorrelation
I We can use the test for individual ACF as well as the Portmanteau tests to test for
autocorrelation.
I A version of the capital asset pricing model (CAPM) theory is that the log return
{rt } of an asset is not predictable and should have no autocorrelations. Testing
for zero autocorrelations has been used as a tool to check the efficient market
hypothesis.
I The way stock prices and index returns are determined might introduce
autocorrelations.
> l o g r e t u r n data = dlog f b [ −1]
> n= l e n g t h ( l o g r e t u r n data )
3 > Box . t e s t ( l o g r e t u r n data , l a g = round ( l o g ( n ) ) , type = ’ Ljung ’ )
1
2
4
5
Box−Ljung t e s t
6
7
8
data :
l o g r e t u r n data
X−squared = 6 . 6 9 1 6 , df = 7 , p−value = 0 . 4 6 1 7
Testing autocorrelation
Model identification
I We can use the properties of ACF and PACF as well as Hypothesis test 3.1 to
identify the order of an ARMA model.
I The following table describes the behaviour of the sample ACF and sample PACF.
sample ACF
sample PACF
AR(p)
Tails off
Cuts off (for h > p)
MA(q)
Cuts off (for h > q)
Tails off
ARMA(p,q)
Tails off
Tails off
Model identification
Model identification
Model identification
Detecting non-stationarity
I The sample ACF can be used to detect whether a process is a random walk
process (with or without drift)
I For a stationary AR(1) model, the ACF is given by ρ(h) = φh . By letting φ → 1, the
ACF approaches one for all lags h: To identify that a model has a unit root, the
sample ACF will have a slow decay.