Dates for term tests
1. Friday, February 5
2. Friday, March 5
3. Friday, March 26
The Moving Average Time series of order q, MA(q)
Let {xt|t  T} be defined by the equation.
x t    u t   1u t 1   2 u t  2 
  qutq
where {ut|t  T} denote a white noise time series
with variance s2.
Then {xt|t  T} is called a Moving Average time series of
order q. (denoted by MA(q))
The mean value for an MA(q) time series
E  xt   
The autocovariance function for an MA(q) time series

s
s h   


qh


2
   i i  h 
 i0

0
if i  q
iq
The autocorrelation function for an MA(q) time series
 q  h

s  h      i i  h 
 h  
  i0

s 0   
0

 q
2 
   i 
 i0

if i  q
iq
Comment
The autocorrelation function for an MA(q) time series
 q  h

s  h      i i  h 
 h  
  i0

s 0   
0

 q
2 
   i 
 i0

if i  q
iq
“cuts off” to zero after lag q.
1.2
1
q
0.8
0.6
0.4
0.2
0
0
-0.2
-0.4
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
The Autoregressive Time series of order p, AR(p)
Let {xt|t  T} be defined by the equation.
x t   1 x t 1   2 x t  2     p x t  p    u t
where {ut|t  T} is a white noise time series with
variance s2.
Then {xt|t  T} is called a Autoregressive time series of
order p. (denoted by AR(p))
The mean value of a stationary AR(p) series
E  xt    

1  1   2     p
The Autocovariance function s(h) of a stationary AR(p) series
Satisfies the equations:
s 0    1s 1      p s  p   s
Y u le W alk er
E q u a tion s
2
s 1    1s 0      p s  p  1 
s  2    1s 1      p s  p  2 

s  p    1s  p  1      p s 0 
and
s  h    1s  h  1      p s  h  p 
for h > p
The Autocorrelation function (h) of a stationary AR(p) series
Satisfies the equations:
 1    11     p   p  1 
  2    1  1      p   p  2 

  p    1   p  1     p 1
with
 h    1  h  1     p  h  p 
and s  0  
s
1    1  1  
2
  p   p 
for h > p
or:
h
h
 1 
1
 1 
  h   c1    c 2      c p  
r 
r
r
 1
 2
 p 
h
where r1, r2, … , rp are the roots of the polynomial
 x   1  1x     p x
p

x 
x  
x 

  1    1     1 


r
r
r
1 
2 
p 


and c1, c2, … , cp are determined by using the starting
values of the sequence (h).
Conditions for stationarity
Autoregressive Time series of
order p, AR(p)
If 1 = 1 and  = 0.
i.e.
x t   1 x t 1  u t
The value of xt increases in magnitude and ut
eventually becomes negligible.
The time series {xt|t  T} satisfies the equation:
x t   1 x t 1
The time series {xt|t  T} exhibits deterministic
behaviour.
For a AR(p) time series, consider the polynomial
 x   1  1x     p x
p

x 
x  
x 

  1    1     1 


r
r
r
1 
2 
p 


with roots r1, r2 , … , rp
then {xt|t  T} is stationary if |ri| > 1 for all i.
If |ri| < 1 for at least one i then {xt|t  T} exhibits
deterministic behaviour.
If |ri| ≥ 1 and |ri| = 1 for at least one i then {xt|t  T}
exhibits non-stationary random behaviour.
since:
h
h
 1 
1
 1 
  h   c1    c 2      c p  
r 
r
r
 1
 2
 p 
h
and |r1 |>1, |r2 |>1, … , | rp | > 1 for a stationary
AR(p) series then
lim   h   0
h 
i.e. the autocorrelation function, (h), of a stationary
AR(p) series “tails off” to zero.
1.2
1
0.8
0.6
0.4
0.2
0
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
Special Cases: The AR(1) time
Let {xt|t  T} be defined by the equation.
x t   1 x t 1    u t
Consider the polynomial

x
  x   1   1 x   1  
r1 

with root r1= 1/1
1. {xt|t  T} is stationary if |r1| > 1 or |1| < 1 .
2. If |ri| < 1 or |1| > 1 then {xt|t  T} exhibits
deterministic behaviour.
3. If |ri| = 1 or |1| = 1 then {xt|t  T} exhibits nonstationary random behaviour.
Special Cases: The AR(2) time
Let {xt|t  T} be defined by the equation.
x t   1 x t 1   2 x t  2    u t
Consider the polynomial

x 
x 
  x   1   1 x   2 x   1    1  
r1  
r2 

2
where r1 and r2 are the roots of (x)
1. {xt|t  T} is stationary if |r1| > 1 and |r2| > 1 .
This is true if 1+2 < 1 , 2 –1 < 1 and 2 > -1.
These inequalities define a triangular region for
1 and 2.
2. If |ri| < 1 or |1| > 1 then {xt|t  T} exhibits
deterministic behaviour.
3. If |ri| ≥ 1 for i = 1,2 and |ri| = 1 for at least on i then
{xt|t  T} exhibits non-stationary random behaviour.
Patterns of the ACF and PACF of AR(2) Time Series
In the shaded region the roots of the AR operator are complex
2
The Mixed Autoregressive Moving Average
Time Series of order p,q
The ARMA(p,q) series
The Mixed Autoregressive Moving Average Time
Series of order p, ARMA(p,q)
Let 1, 2, … p , 1, 2, … p ,  denote p +
q +1 numbers (parameters).
Let {ut|t  T} denote a white noise time series with
variance s2.
– independent
– mean 0, variance s2.
Let {xt|t  T} be defined by the equation.
x t   1 x t 1   2 x t  2     p x t  p  
 u t   1u t 1   2 u t  2     q u t  q
Then {xt|t  T} is called a Mixed AutoregressiveMoving Average time series - ARMA(p,q) series.
Mean value, variance,
autocovariance function,
autocorrelation function of an
ARMA(p,q) series
Similar to an AR(p) time series, for certain
values of the parameters 1, …, p an
ARMA(p,q) time series may not be stationary.
An ARMA(p,q) time series is stationary if the
roots (r1, r2, … , rp ) of the polynomial
(x) = 1 – 1x – 2x2 - … - p xp
satisfy | ri| > 1 for all i.
Assume that the ARMA(p,q) time series {xt|t  T}
is stationary:
Let  = E(xt). Then
E  x t    1 E  x t 1    2 E  x t  2      p E  x t  p   
 E u t    1 E u t 1    2 E u t  2      q E u t  q 
  1   2      p   
 0  1  0   2  0     q  0
1  
or
1
  2     p   
E  xt    

1  1   2 
 p
The Autocovariance function, s(h), of a stationary
mixed autoregressive-moving average time series {xt|t
 T} be determined by the equation:
x t   1 x t 1   2 x t  2     p x t  p  
 u t   1u t 1   2 u t  2     q u t  q
now   1   1   2     p 
Thus
x t     1  x t 1        p  x t  p   
 u t   1u t 1   2 u t  2     q u t  q
Hence
s h   E  x t  h    x t   

 E   1  x t  h 1        p  x t  h  p   

 u t  h   1u t  h 1   2 u t  h  2     q u t  h  q  x t   


  1 E  x t  h 1    x t        p E  x t  h  p    x t   
 E u t  h  x t      1 E u t  h 1  x t        q E u t  h  q  x t   
  1s  h  1      p s  h  p 
 s ux  h    1s ux  h  1      q s ux  h  q 
where
s ux h   E u t  h  x t   
note
s ux h   E u t  h  x t   

 E u t  h  1  x t 1        p  x t  p   
 u t   1u t 1   2 u t  2     q u t  q


  1 E u t  h  x t 1        p E u t  h x t  p  

 E u t  h u t    1 E u t  h u t 1      q E u t  h u t  q 
  1s ux  h  1     p s ux  h  p 
 s uu  h    1s uu  h  1     q s uu  h  q 
where
and
s ux h   E u t  h  x t     0 if h  0 .
s 2 if h  0 .
s uu  h   E u t  h u t   
 0 if h  0 .
We need to calculate:
s ux 0 , s ux  1,  , s ux  q 
note
s ux h    1s ux  h  1     p s ux  h  p 
 s uu  h    1s uu  h  1     q s uu  h  q 
and
s 2 if h  0 .
s uu  h   
 0 if h  0 .
s ux h   0 if h  0 .
s ux 0   s
2
s ux   1   s ux 0    s
2
  s
2
  s
s ux   2    s ux   1   2s ux 0    2s
         s
2
  2s
2
2
       s
2
  2s
            2   2 s
2
2
etc
2
The autocovariance function s(h) satisfies:
s  h    1s  h  1      p s  h  p 
 s ux  h    1s ux  h  1      q s ux  h  q 
For h = 0, 1. … , q:
s 0    1s 1      p s  p   s ux 0    1s ux   1     q s ux   q 
s 1    1s 0      p s  p  1    1s ux 0      q s ux   q  1

s  q    1s  q  1      p s  q  p    q s ux 0 
for h > q:
s  h    1s  h  1      p s  h  p 
We then use the first (p + 1) equations to determine:
s(0), s(1), s(2), … , s(p)
We use the subsequent equations to determine:
s(h) for h > p.
Example:The autocovariance function, s(h), for an
ARMA(1,1) time series:
s  h    1s  h  1   s ux h    1s ux h  1
For h = 0, 1:
s 0    1s 1   s ux 0    1s ux  1
s 1    1s 0    1s ux 0 
or
s 0    1s 1   s   1  1   1 s
2
s 1    1s 0 
for h > 1:
  1s
2
s  h    1s  h  1 
2
Substituting s(0) into the second equation we get:
s 1   1  1s 1  s   1  1   1 s
2
or
s 1  
1   1  1  1   1 
1  1
2
s
2
  s
2
1
2
Substituting s(1) into the first equation we get:
s 0    1
1   1  1  1   1 
1  1
2
s  s   1  1   1 s
2
2
 1 1   1  1  1   1   1   1  1   1  1  1   1 
2

2
1  1
2
1   1  2 1  1
2

1  1
2
s
2
2
s
2
for h > 1:
s  h    1s  h  1 
s  2    1s 1    1
s 3    1s  2   
2
1
1   1  1  1   1 
1  1
2
s
1   1  1  1   1 
1  1
2
s
2
2

s  h    1s  h  1  
h 1
1
1   1  1  1   1 
1  1
2
s
2
The Backshift Operator B
Consider the time series {xt : t  T} and Let M denote
the linear space spanned by the set of random variables
{xt : t  T}
(i.e. all linear combinations of elements of {xt : t  T}
and their limits in mean square).
M is a vector space
Let B be an operator on M defined by:
Bxt = xt-1.
B is called the backshift operator.
Note:
1. B c x
1
t1
 c 2 xt2    c k xtk

 c1 Bx t1  c 2 Bx t 2    c k Bx t k
 c1 x t1  1  c 2 x t 2  1    c k x t k  1
2. We can also define the operator Bk with
Bkxt = B(B(...Bxt)) = xt-k.
3. The polynomial operator
p(B) = c0I + c1B + c2B2 + ... + ckBk
can also be defined by the equation.
p(B)xt = (c0I + c1B + c2B2 + ... + ckBk)xt .
= c0Ixt + c1Bxt + c2B2xt + ... + ckBkxt
= c0xt + c1xt-1 + c2xt-2 + ... + ckxt-k
4. The power series operator
p(B) = c0I + c1B + c2B2 + ...
can also be defined by the equation.
p(B)xt = (c0I + c1B + c2B2 + ... )xt
= c0Ixt + c1Bxt + c2B2xt + ...
= c0xt + c1xt-1 + c2xt-2 + ...
5. If p(B) = c0I + c1B + c2B2 + ... and q(B) = b0I + b1B
+ b2B2 + ... are such that
p(B)q(B) = I
i.e. p(B)q(B)xt = Ixt = xt
than q(B) is denoted by [p(B)]-1.
Other operators closely related to B:
1. F = B-1 ,the forward shift operator, defined
by Fxt = B-1xt = xt+1
and
2. D = I - B ,the first difference operator,
defined by Dxt = (I - B)xt = xt - xt-1 .
The Equation for a MA(q) time series
xt= 0ut + 1ut-1 +2ut-2 +... +qut-q + 
can be written
xt= (B) ut + 
where
(B) = 0I + 1B +2B2 +... +qBq
The Equation for a AR(p) time series
xt= 1xt-1 +2xt-2 +... +pxt-p +  + ut
can be written
(B) xt=  + ut
where
(B) = I - 1B - 2B2 -... - pBp
The Equation for a ARMA(p,q) time series
xt= 1xt-1 +2xt-2 +... +pxt-p +  + ut + 1ut-1
+2ut-2 +... +qut-q
can be written
(B) xt= (B) ut + 
where
(B) = 0I + 1B +2B2 +... +qBq
and
(B) = I - 1B - 2B2 -... - pBp
Some comments about the Backshift
operator B
1. It is a useful notational device, allowing us to
write the equations for MA(q), AR(p) and
ARMA(p, q) in a very compact form;
2. It is also useful for making certain
computations related to the time series
described above;
The partial autocorrelation
function
A useful tool in time series analysis
The partial autocorrelation function
Recall that the autocorrelation function of an AR(p)
process satisfies the equation:
x(h) = 1x(h-1) + 2x(h-2) + ... +px(h-p)
For 1 ≤ h ≤ p these equations (Yule-Walker) become:
x(1) = 1 + 2x(1) + ... +px(p-1)
x(2) = 1x(1) + 2 + ... +px(p-2)
...
x(p) = 1x(p-1)+ 2x(p-2) + ... +p.
In matrix notation:
1
  x 1   

 
 2 
 x 1 
 x
  
   


 
  x  p    x  p  1 
 x 1 

1



 x  p  2 
 x  p  1    1 
 
 x  p  2  2 

  



1
   p 
These equations can be used to find 1, 2, … , p,
if the time series is known to be AR(p) and the
autocorrelation x(h)function is known.
If the time series is not autoregressive the
equations can still be used to solve for 1, 2, … ,
p, for any value of p 1.
In this case
p
p
p
1 ,  2 , ,  p
are the values that minimizes the mean square
error:
 
M .S . E .  E   ( x t   x ) 
 
p

i 1
p
i
( x t i

  x )

2



Definition: The partial auto correlation function
at lag k is defined to be:
 kk  
(k )
k

1
 x 1 

 x 1 
 x 1 
1

 x 2 




 x k  1
 x k  2  
 x k 
1
 x 1 

 x k  1
 x 1 
1

 x k  2 




 x k  2  
1
 x k  1
Comment:
The partial auto correlation function, kk is
determined from the auto correlation function,
(h)
Some more comments:
1. The partial autocorrelation function at lag k,
kk, can be interpreted as a corrected
autocorrelation between xt and xt-k conditioning
on the intervening variables xt-1, xt-2, ... ,xt-k+1 .
2. If the time series is an AR(p) time series than
kk = 0 for k > p
3. If the time series is an MA(q) time series than
x(h) = 0 for h > q
A General Recursive Formula for
Autoregressive Parameters and the
Partial Autocorrelation function
(PACF)
Let
 ,  ,  , , 
k
1
k
2
k
3
k
k
  kk
denote the autoregressive parameters of order k
satisfying the Yule Walker equations:
       2      k 1 
k
1
k
2
k
3
    1     k 2 
k
1
k
2
k
3
k
k
k
k
 
 2

 k 1    k  2    k  3     
k
1
k
2
k
3
k
k
 k
Then it can be shown that:
k
 k     j  k 1 j
k
j 1
k 1
 k 1   k 1, k 1 
k
1   j  j
k
j 1
and

k 1
j
 
k
j
  k 1, k 1 
k
k  j 1
j  1, 2 ,  , k
Proof:
The Yule Walker equations:
 1     2   2  3     k 1  k   
k
k
k
k
1   2  1 3     k 2  k   2
k
k
k
k

 k 1  1   k  2  2   k  3  3     k   k
k
k
k
k
In matrix form:
 1



 

  k 1


1



 k 2

or
 1


k
Ρ 
 

  k 1
 k 1    1k 
  
 k   
 k 2  2 
2

  
      
 k   
1    k    k 
Ρ β ρ
k


1



 k 2

k
k
 k 1 
 
β  Ρ
k
k 1
ρ
k
  1k 
  
 k

 
 k 2
2

k
k
2


, β 
and ρ   
  
  
 
 k

 
1 
  k 
k 
The equations for  1k  1 ,  2k  1 ,  3k  1 ,  ,  kk11
 1



 

k


1



 k 1

 k    1k  1 
  
  k 1  

 k 1   2 
2



      
  k 1  

1    k  1    k  1 
  k 1, k 1
or
A ρ   β 1k  1   ρ k 



1    k  1, k  1    k  1 
 Ρk

k 
 A ρ

k

0

0

A 


1
where
β
k 1
1

 
k 1
1
0

0



0

,
k 1
2
,
The matrix A reverses order
1

0



0
k 1
3
and
, , 
k 1
k


The equations may be written
k 1
k
Ρ β1

ρ 
k
  k 1, k 1 A ρ  ρ
k
k
k 1

A β 1   k 1, k 1   k 1
Ρ 
k 1
Multiplying the first equations by
β
or
k 1
1
 
  k 1, k 1 Ρ
β
k 1
1
k 1
 β   k 1, k 1
k 1
 β   k 1, k 1 A β
k
k 1
  Aρ
A Ρ  ρ
 β   k 1, k 1 Ρ
k
k
 
Aρ  Ρ
k
k 1
k
ρ β
k
k
k
k
Substituting this into the second equation

ρ  A β
k
k
  k 1, k 1 A β
k
 
k 1, k 1
  k 1
or

k  k 
k 
k
 k 1, k 1  1  ρ β    k 1  ρ A β


 
and
 k 1, k 1 
 
 k 1  ρ

 Aβ

1  β  ρ
k
k
k
k
Hence

k
 k      k 1 j
k
j
k 1
k 1
  k 1, k 1 
j 1
k
1   j  j
k
j 1
and
β
k 1
 β   k 1, k 1 A β
k
k
or

k 1
j
 
k
j
  k 1, k 1 
k
k  j 1
j  1, 2 ,  , k
Some Examples
Example 1: MA(1) time series
Suppose that {xt|t  T} satisfies the following
equation:
xt = 12.0 + ut + 0.5 ut – 1
where {ut|t  T} is white noise with s = 1.1.
Find:
1. The mean of the series,
2. The variance of the series,
3. The autocorrelation function.
4. The partial autocorrelation function.
Solution
Now {xt|t  T} satisfies the following equation:
xt = 12.0 + ut + 0.5 ut – 1
Thus:
1. The mean of the series,
 = 12.0
The autocovariance function for an MA(1) is
 1    s

2
s  h     s

0

2
1
2
  1  0.5 2  1.1  2
h0

2

h 1  
0.5  1.1 
h 1 
0


h0
1.5125

h  1   0.605
h  1  0
h0
h 1
h 1
Thus:
2. The variance of the series,
s(0) = 1.5125
and
3. The autocorrelation function is:
 1
s  h   0.605
 h 
  1.5125
s 0 
 0
h0
 1

h  1   0.4
h  1  0
h0
h 1
h 1
4.
The partial auto correlation function at lag k is defined to be:
 kk  
Thus
 22  
(k )
k

 11  
(2)
2

1
 1 
 1 
 1 
1
 2
  k  1
 k  2
 k 
1
 1 
  k  1
 1 
1
 k  2
  k  1
 k  2
1
(1)
1
1
 1 
 1 
 2
1
 1 
 1 
1

 1 
1

  1   0.4
 2  
1 
2
1   1 
2

 0.4
2
1  0.4
2

 0.16
0.84
  .19048
 33  
 44   4
(4)
(3)
3

 55   5

1
 1 
 1 
1
0.4
0.4
 1 
1
 2
0.4
1
0
 2
 1 
 3
0
0.4
0
1
 1 
 2
 1 
1
 2
 1 

1
.4
0
 1 
.4
1
.4
1
0
.4
1

0.064
1
 1 
 2
 1 
1
.4
0
.4
 1 
1
 1 
 2
.4
1
.4
0
 2
 1 
1
 3
0
.4
1
0
 3
 2
 1 
 4
0
0
.4
0
1
 1 
 2
 3
1
.4
0
0
 1 
1
 1 
 2
.4
1
.4
0
 2
 1 
1
 1 
0
.4
1
.4
 3
 2
 1 
1
0
0
.4
1
(5)

0.01024
0.4368
 0.0234

 0.0941
0.68

 0.0256
0.5456
  0.0469
 66   0.0117,  77  0.0059,  88   0.0029,  99  0.0015
 10 ,10   0.0007,  11,11  0.0004,  12 ,12   0.00029
Graph: Partial Autocorrelation function kk
1 .2
1
0 .8
0 .6
0 .4
0 .2
0
0
-0 .2
-0 .4
1
2
3
4
5
6
7
8
9
10
11
Exercise: Use the recursive method to calculate kk
k
 k     j  k  1 j
k
k 1
 k  1   k  1, k  1 
j 1
k
1   j  j
k
j 1
and

k 1
j
 
k
j
  k  1, k  1 
k
k  j 1
w e start w ith    1,1   1
1
1
j  1, 2,
, k
Exercise: Use the recursive method to calculate kk
   2 ,2 
2
2
and
2   
1
1 1
1
1 1
1  

  0.4 
1   0.4 
     2.2 
2
1
2
1
1
1
1
2
  .19048
j 1
  1    .1 9 0 4 8    0 .4 
  1.19048   0.4   .0.476192
 3  1  2  1 1
2
 3   3,3 
3
2
1   1    2
2
1
2
2
 0.0941,
etc
Example 2: AR(2) time series
Suppose that {xt|t  T} satisfies the following equation:
xt = 0.4 xt – 1 + 0.1 xt – 2 + 1.2 + ut
where {ut|t  T} is white noise with s = 2.1.
Is the time series stationary?
Find:
1. The mean of the series,
2. The variance of the series,
3. The autocorrelation function.
4. The partial autocorrelation function.
1. The mean of the series
 

1  1   2

1.2
1  0.4  0.1
 2.4
3. The autocorrelation function.
Satisfies the Yule Walker equations
 1   1  1    2  1  0.4  0.1  1
 2   1  1   2  1   0.4  1  0.1
then  h  1   1  h   2  h 1  0.4  h  0.1  h 1
w here  h    h 
hence
1 
0.4
 0.4444
0.9
 2  0.4
0.4
 0.1  2.778
0.9
then  h  1   1  h   2  h 1  0.4  h  0.1  h 1
w here  h    h 
h
0
1
2
3
4
5
6
1.0000 0.4444 0.2778 0.1556 0.0900 0.0516 0.0296
h
7
8
9
10
11
12
13
0.0170 0.0098 0.0056 0.0032 0.0018 0.0011 0.0006
h
h
2. the variance of the series
s 0 
s
2
1  1 1   2 1

2.1
2
1  0.4  0.4444   0.1  0.2778 
 5.7522
4. The partial autocorrelation function.
 1,1   1  0.4444
 2 ,2 
1
1
1
0.4444
1
2
0.4444
.2778
1
1
1
0.4444
1
1
0.4444
1

 0.1000
 3 ,3 
1
1
1
1
0.4444
0.4444
1
1
2
0.4444
1
0.2778
2
1
3
0.2778
0.4444
0.1556
1
1
2
1
0.4444
0.2778
1
1
1
0.4444
1
0.4444
2
1
1
0.2778
0.4444
1
in fact  k , k  0

0
for k  3
The partial autocorrelation function of an AR(p) time
series “cuts off” after p.
Example 3: ARMA(1, 2) time series
Suppose that {xt|t  T} satisfies the following equation:
xt = 0.4 xt – 1 + 3.2 + ut + 0.3 ut – 1 + 0.2 ut – 1
where {ut|t  T} is white noise with s = 1.6.
Is the time series stationary?
Find:
1. The mean of the series,
2. The variance of the series,
3. The autocorrelation function.
4. The partial autocorrelation function.