STAT 497
LECTURE NOTES 2
1
THE AUTOCOVARIANCE AND THE
AUTOCORRELATION FUNCTIONS
• For a stationary process {Yt}, the
autocovariance between Yt and Yt-k is
 k  CovYt ,Yt k   EYt   Yt k   
and the autocorrelation function is
k
 k  Corr Yt , Yt k    ACF
0
2
THE AUTOCOVARIANCE AND THE
AUTOCORRELATION FUNCTIONS
PROPERTIES:
1.  0  VarYt   0  1.
2.  k   0   k  1.
3.  k   k and k  k , k.
4. (necessary condition) k and k are positive semidefinite
n n
   i j ti t j  0
i 1 j 1
n n
   i j  ti t j  0
i 1 j 1
3
for any set of time points t1,t2,…,tn and any real numbers 1,2,…,
n.
THE PARTIAL AUTOCORRELATION
FUNCTION (PACF)
• PACF is the correlation between Yt and Yt-k after
their mutual linear dependency on the
intervening variables Yt-1, Yt-2, …, Yt-k+1 has
been removed.
• The conditional correlation
Corr Yt , Yt k Yt 1 , Yt 2 ,, Yt k 1   kk
is usually referred as the partial autocorrelation
in time series.
e.g., 11  Corr Yt , Yt 1   1
22  Corr Yt , Yt 2 Yt 1 
4
CALCULATION OF PACF
1. REGRESSION APPROACH: Consider a model
Yt k  k1Yt k 1  k 2Yt k  2    kkYt  et k
from a zero mean stationary process where ki
denotes the coefficients of Ytk+i and etk is the
zero mean error term which is uncorrelated
with Ytk+i, i=0,1,…,k.
• Multiply both sides by Ytk+j
Yt kYt k  j  k1Yt k 1Yt k  j    kkYtYt k  j  et kYt k  j
5
CALCULATION OF PACF
and taking the expectations
 j  k1 j 1  k 2 j 2    kk j k
diving both sides by 0
 j  k1 j 1  k 2  j 2    kk  j k
PACF
6
CALCULATION OF PACF
• For j=1,2,…,k, we have the following system
of equations
1  k1  k 2 1    kk  k 1
 2  k11  k 2    kk  k 2

 k  k1 k 1  k 2  k 2    kk
7
CALCULATION OF PACF
• Using Cramer’s rule successively for k=1,2,…
11  1
1
22 
1
1
1
1
2
 2  2  1

1 1  12
1

8
CALCULATION OF PACF
1
1
1
1


 k 1  k 2
kk 
1
1
1
1


 k 1  k 2
2
1

  k 2 1
  k 3  2



 1  k
  k 2  k 1
 k 3
2
1   k 3

 k 3



1
 k 2

1
9
CALCULATION OF PACF
2. Levinson and Durbin’s Recursive Formula:
k 1
kk 
 k   k 1, j  k  j
j 1
k 1
1   k 1, j  k  j
j 1
where kj  k 1, j  kkk 1,k  j , j  1,2,, k  1.
10
WHITE NOISE (WN) PROCESS
• A process {at} is called a white noise (WN)
process, if it is a sequence of uncorrelated
random variables from a fixed distribution
with constant mean {E(at)=}, constant
2
variance {Var(at)= a } and Cov(Yt, Yt-k)=0 for
all k≠0.
Yt  at
11
WHITE NOISE (WN) PROCESS
• It is a stationary process with autocovariance
function
 a2 , k  0
k  
 0, k  0
ACF
PACF
 1, k  0
k  
0, k  0
 1, k  0
kk  
0, k  0
Basic Phenomenon: ACF=PACF=0, k0.
12
WHITE NOISE (WN) PROCESS
• White noise (in spectral analysis): white light is
produced in which all frequencies (i.e., colors)
are present in equal amount.
• Memoryless process
• Building block from which we can construct
more complicated models
• It plays the role of an orthogonal basis in the
general vector and function analysis.
13
ESTIMATION OF THE MEAN, AUTOCOVARIANCE
AND AUTOCORRELATION
• THE SAMPLE MEAN:
n
 yt
y  t 1
n
with E Y    and Var Y  
0
n
n 1 
k
 1  k .
n
k    n 1 

Because VarY  n
 0, Y is a CE for  .
lim Y  
n



in mean square
if this holds, the process is ergodic for the mean.
14
ERGODICITY
• Kolmogorov’s law of large number (LLN) tells that if
Xii.i.d.(μ, 2) for i = 1, . . . , n, then we have the
following limit for the ensemble
average
n
 Yi
Yn  i 1  .
n
• In time series, we have time series average, not
ensemble average. Hence, the mean is computed by
averaging over time. Does the time series average
converges to the same limit as the ensemble
average? The answer is yes, if Yt is stationary and
ergodic.
15
ERGODICITY
• A covariance stationary process is said to
ergodic for the mean, if the time series
average converges to the population mean.
• Similarly, if the sample average provides an
consistent estimate for the second moment,
then the process is said to be ergodic for the
second moment.
16
ERGODICITY
• A sufficient condition for a covariance
stationary process to be ergodic for the mean

is that   k   . Further, if the process is
k 0
Gaussian, then absolute summable
autocovariances also ensure that the process
is ergodic for all moments.
17
THE SAMPLE AUTOCOVARIANCE
FUNCTION
1 nk
ˆk   Yt  Y Yt k  Y 
n t 1
or
1 nk
ˆk 
 Yt  Y Yt k  Y 
n  k t 1
18
THE SAMPLE AUTOCORRELATION
FUNCTION
nk
ˆ k  rk 
 Yt  Y Yt k  Y 
t 1
n
 Yt  Y 
, k  0,1,2,...
2
t 1
• A plot ˆ k versus k a sample correlogram
• For large sample sizes, ˆ k is normally
distributed with mean k and variance is
approximated by Bartlett’s approximation for
processes in which k=0 for k>m.
19
THE SAMPLE AUTOCORRELATION
FUNCTION

1
Var  ˆ k   1  2 12  2  22    2  m2
n

• In practice, i’s are unknown and replaced by
their sample estimates,ˆi. Hence, we have the
following large-lag standard error of ˆ k :

1
2
sˆ k 
1  2ˆ 12  2 ˆ 22    2ˆ m
n

20
THE SAMPLE AUTOCORRELATION
FUNCTION
• For a WN process, we have
sˆ k
1

n
• The ~95% confidence interval for k:
1
ˆ k  2
n
For a WN process, it must be close to zero.
• Hence, to test the process is WN or not, draw a
2/n1/2 lines on the sample correlogram. If all ˆ k
are inside the limits, the process could be WN
(we need to check the sample PACF, too).
21
THE SAMPLE PARTIAL
AUTOCORRELATION FUNCTION
ˆ11  ˆ1
k 1
ˆkk 
ˆ k   ˆk 1, j ˆ k  j
j 1
k 1
1   ˆk 1, j ˆ k  j
j 1
where ˆkj  ˆk 1, j  ˆkkˆk 1,k  j , j  1,2,, k  1.
• For a WN process,
1
ˆ
Var kk  
n
• 2/n1/2 can be used as critical limits on kk to
test the hypothesis of a WN process.
22
BACKSHIFT (OR LAG) OPERATORS
• Backshift operator, B is defined as
B Yt  Yt  j , j  0 with B  1.
j
0
BYt  Yt 1
B 2Yt  Yt  2
B12Yt  Yt 12
e.g. Random Shock Process:
Yt  Yt 1  et
Yt  Yt 1  et
Yt  BYt  et
1  B Yt  et
23
MOVING AVERAGE REPRESENTATION
OF A TIME SERIES
• Also known as Random Shock Form or Wold
(1938) Representation.
• Let {Yt} be a time series. For a stationary
process {Yt}, we can write {Yt} as a linear
combination of sequence of uncorrelated
(WN) r.v.s.
A GENERAL LINEAR PROCESS:

Yt    at  1at 1  2at 2        j at  j
j 0

2

where 0=I, {at} is a 0 mean WN process and  j  .
j 0
24
MOVING AVERAGE REPRESENTATION
OF A TIME SERIES

Yt    at  1Bat  2 B at        j B j at
2

j 0

   1  1B  2 B 2   at



     B at where   B   1  1B  2 B      j B j
2
j 0
25
MOVING AVERAGE REPRESENTATION
OF A TIME SERIES
E Yt   

2
Var Yt    0   a   2j
j 0
 k  E Yt   Yt  k    
E at  1at 1  2 at  2  .... k at  k  k 1at  k 1  ...at  k  1at  k 1  2 at  k  2  ....

2
2
2
2
 k  a  1k 1 a  2 k  2 a  ...   a  i k  i
i 0

 i k  i
 k  i  0
2
 j
j 0
26
MOVING AVERAGE REPRESENTATION
OF A TIME SERIES
• Because they involve infinite sums, to be
statinary
 k  E Yt   Yt  k     Var Yt Var Yt  k 
1/ 2




Cauchy Schwarz Inequality
  a2

  2j  
j 0
2

• Hence,  j   is the required condition for
j 0
the process to be stationary.
• It is a non-deterministic process: A process
contains no deterministic components (no
randomness in the future states of the system)
that can be forecast exactly from its own past.
27
AUTOCOVARIANCE GENERATING
FUNCTION
• For a given sequence of autocovariances k,
k=0,1, 2,… the autocovariance generating
function is defined as 
 B     k B k
k  
where the variance of a given process 0 is the
coefficient of B0 and the autocovariance of lag
k, k is the coefficient of both Bk and Bk.
  B      2 B 2   1B 1   0   1B   2 B 2  
2
1
28
AUTOCOVARIANCE GENERATING
FUNCTION
• Using

2
 k   a  i i  k
i 0
and stationarity
  
 k
2
 B    a    i i  k B
k    i  0

j i  k
  
 j i
2
  a    i  j B
j 0  i 0



2
i
  a  i B   j B j
i 0
j 0
 
  a2  B 1  B 
where j=0 for j<0.
29
AUTOCORRELATION GENERATING
FUNCTION
 B 
 B    k B 
0
k  

k
30
EXAMPLE


Yt  Yt 1  at where   1 and at ~ iid 0, a2 .
a) Write the above equation in random shock
form.
b) Find the autocovariance generating function.
31
AUTOREGRESSIVE REPRESENTATION
OF A TIME SERIES
• This representation is also known as INVERTED
FORM.
• Regress the value of Yt at time t on its own
past plus a random shock.
Yt     1Yt 1      2 Yt 2       at

1  1B   2 B 2  Yt     at

B 

 
j
   j B  Yt     at with  0  1 and 1    j  .
j 1
 j 0

32
AUTOREGRESSIVE REPRESENTATION
OF A TIME SERIES
• It is an invertible process (it is important for
forecasting). Not every stationary process is
invertible (Box and Jenkins, 1978).
• Invertibility provides uniqueness of the
autocorrelation function.
• It means that different time series models can
be re-expressed by each other.
33
INVERTIBILITY RULE USING THE
RANDOM SHOCK FORM
• For a linear process,
Yt  B at
to be invertible, the roots of (B)=0 as a
function of B must lie outside the unit circle.
• If  is a root of (B), then ||>1.
(real number) || is the absolute value of .
2
2


c

d
.
(complex number)  c  id  || is
34
INVERTIBILITY RULE USING THE
RANDOM SHOCK FORM
• It can be stationary if the process can be rewritten in a RSF, i.e.,
1
Yt    
at  B at
 B 

 B B   1 where   2j  .
j 0
35
STATIONARITY RULE USING THE
INVERTED FORM
• For a linear process,
B Yt     at
to be invertible, the roots of (B)=0 as a
function of B must lie outside the unit circle.
• If  is a root of (B), then ||>1.
36
RANDOM SHOCK FORM AND
INVERTED FORM
• AR and MA representations are not the model
form. Because they contain infinite number of
parameters that are impossible to estimate
from a finite number of observations.
37
TIME SERIES MODELS
• In the Inverted Form of a process, if only finite
number of  weights are non-zero, i.e.,
1  1,  2  2 ,,  p   p and Πk  0, k  p,
the process is called AR(p) process.
38
TIME SERIES MODELS
• In the Random Shock Form of a process, if only
finite number of  weights are non-zero, i.e.,
1  1, 2   2 ,, q   q and k  0, k  q,
the process is called MA(q) process.
39
TIME SERIES MODELS
• AR(p) Process:
Yt     1Yt 1        p Yt  p     at
Yt  c  1Yt 1     pYt  p  at where  
c
.
1-1     p


• MA(q) Process:
Yt    at  1at 1     q at q .
40
TIME SERIES MODELS
• The number of parameters in a model can be
large. A natural alternate is the mixed AR and
MA process  ARMA(p,q) process
Yt  c  1Yt 1     pYt  p  at  θ1at 1    θq at-q
1   B    
1
pB
p
Y  c  1  θ B    θ B a
q
t
1
q
t
• For a fixed number of observations, the more
parameters in a model, the less efficient is the
estimation of the parameters. Choose a simpler
model to describe the phenomenon.
41