Non-Seasonal Box-Jenkins Models
1
Four-step iterative procedures
1) Model Identification
2) Parameter Estimation
3) Diagnostic Checking
4) Forecasting
2
Step One: Model Identification
3
Model Identification
I.
II.
Stationarity
Theoretical Autocorrelation Function
(TAC)
III. Theoretical Partial Autocorrelation
Function (TPAC)
IV. Sample Partial Autocorrelation Function
(SPAC)
V. Sample Autocorrelation Function (SAC)
4
Stationarity (I)
 A sequence of jointly dependent random
variables
{ yt :   t  }
is called a time series
5
Stationarity (II)
 Stationary process
Properties :
(1) E ( yt )  u y for all t.
(2) Var ( yt )  E[( yt  u y ) ]   for all t.
2
2
y
(3) Cov( yt , yt  k )   k for all t.
6
Stationarity (III)
 Example: The white noise series {et }

e’s are iid as N(0,e2). Note that
(1) E (e t )  0 for all t.
(2) Var (e t )  E (e t )   e for all t.
2
2
(3) Cov(e t , e t s )  0 for all t.
7
Stationarity (IV)
 Three basic Box-Jenkins models for a
stationary time series {yt } :
(1) Autoregressive model of order p (AR(p))
yt    1 yt 1  2 yt 2    p yt  p  e t ,
i.e., yt depends on its p previous values
(2) Moving Average model of order q (MA(q))
yt    e t 1e t 1  2e t 2   qe t q ,
i.e., yt depends on q previous random error terms
8
Stationarity (V)
 Three basic Box-Jenkins models for a
stationary time series {yt } :
(3) Autoregressive-moving average model of order
p and q (ARMA(p,q))
yt    1 yt 1  2 yt 2     p yt  p
 e t  1e t 1   2e t 2     qe t q ,
i.e., yt depends on its p previous values and q
previous random error terms
9
AR(1) (I)
 Simple AR(1) process without drift
y t  1 y t 1  e t
( where e t is white noise)
y t  1Lyt  e t
( where L is the back  shift operator)
 (L)  (1  1L) y t  e t
or
et
yt 
1  1L
 e t (1  1L   L  )
2 2
1
 e t  1e t 1  12 t 2  .
10
AR(1) (II)
 Now,
(1) E ( yt )  0
for all t.
 e2
(2) Var ( yt ) 
1  12
for all t.
 e21s
(3) Cov( yt , yt  s ) 
1  12
for all t.
 Var(yt) and cov(yt, yt-s) are finite if and only if
|1| < 1, which is the stationarity requirement
for an AR(1) process.
11
AR(1) (IV)
 Special Case: 1 = 1
yt  yt 1  e t .
 It is a “random walk” process. Now,
t 1
yt   e t  j .
 Thus,
j 0
(1) E ( yt )  0 for all t.
(2) Var ( yt )  t e2 for all t.
(3) Cov( yt , yt  s )  | t  s |  e2 for all t.
12
AR(1) (V)
 Consider,
yt  yt  yt 1
 et .
 yt is a homogeneous non-stationary series.
The number of times that the original series
must be differenced before a stationary series
results is called the order of integration.
13
Theoretical Autocorrelation
Function (TAC) (I)
 Autoregressive (AR) Processes
Consider an AR(1) process without drift :
yt  1 yt 1  e t .
Recall that
(1) E ( yt )  0
for all t.
e
(2) Var ( yt ) 
0
2
1  1
2
 e21s
(3) Cov( yt , yt  s ) 
s
2
1  1
for all t.
for all t.
14
Theoretical Autocorrelation
Function (TAC) (II)
The autocorrelation function at lag k is
k
k 
0
for k  0,1, 2, ...
 1k .
So for a stationary AR(1) process, the TAC
dies down gradually as k increases.
15
Theoretical Autocorrelation
Function (TAC) (III)
Consider an AR(2) process without drift :
yt  1 yt 1  2 yt 2  e t .
The TAC functions are
1
1 
,
1  2
 2  2 

2
1
1  2
 k  1  k 1   2  k  2
for k  2.
16
Theoretical Autocorrelation
Function (TAC) (IV)
Then the TAC dies down according to a
mixture of damped exponentials and/or
damped sine waves.
In general, the TAC of a stationary AR
process dies down gradually as k increases.
17
Theoretical Autocorrelation
Function (TAC) (V)
 Moving Average (MA) Processes
Consider a MA(1) process without drift :
yt  e t 1e t 1 .
Recall that
(1) E ( yt )  0
for all t.
(2) Var(yt )   0   e2 (1  12 )
for all t.
  1 e2 s  1
(3) Cov( yt , yt  s )   s  
s  1.
0
18
Theoretical Autocorrelation
Function (TAC) (VI)
Therefore the TAC of the MA(1) process is
k
k 
0
  1
k 1

2
  1  1
0
k  1.
The TAC of the MA(1) process “cuts off”
after lag k=1.
19
Theoretical Autocorrelation
Function (TAC) (VII)
Consider a MA(2) process :
 1 (1   2 )
1 
,
2
2
1  1   2
 2
2 
,
2
2
1  1   2
 k  0 for k  2 .
The TAC of a MA(2) process cuts off after 2
lags.
20
Theoretical Partial Autocorrelation
Function (TPAC) (I)
 Autoregressive Processes
By the definition of the PAC, the parameter k is
the kth PAC kk. Therefore, the partial
autocorrelation function at lag k is
 kk  k .
As mentioned before, if k=1, then
1  1  11 .
That is, PAC=AC.
The TPAC of an AR(1) process “cuts off” after
lag 1.
21
Theoretical Partial Autocorrelation
Function (TPAC) (II)
 Moving Average Processes
Consider
yt  e t  1e t 1

 1j yt  j  e t ,
j 1
which is a stationary AR process with infinite
order. Thus, the partial autocorrelation decays
towards zero as j increases.
22
Summary of the Behaviors of
TAC and TPAC (I)
Behaviors of TAC and TPAC for general non-seasonal models
Model
Autoregressive of order p
zt    1 zt 1  2 zt 2   p zt  p  e t
Moving Average of order q
zt    e t 1e t 1  2e t 2   qe t q
Mixed Autoregressive-Moving Average of order (p,q)
TAC
TPAC
Dies down Cuts off
after lag p
Cuts off
Dies down
after lag q
Dies down Dies down
zt    1 zt 1  2 zt 2   p zt  p
 e t  1e t 1  2e t 2    qe t q
23
Summary of the Behaviors of
TAC and TPAC (II)
Behaviors of TAC and TPAC for specific non-seasonal models
Model
TAC
TPAC
First-order autoregressive
zt    1 zt 1  e t
Dies down in a damped exponential
fashion; specifically:
Cuts off
after lag
1
Second-order autoregressive
Dies down according to a mixture of
damped exponentials and/or damped
sine waves; specifically:

1  1 ,
1  2
Cuts off
after lag
2
zt    1 zt 1  2 zt 2  e t
k  1k for k  1
12
 2  2 
,
1  2
k  1 k 1  2 k 2 for k  3
24
Summary of the Behaviors of
TAC and TPAC (III)
Behaviors of TAC and TPAC for specific non-seasonal models
Model
First-order moving average
zt    e t  1e t 1
TAC
TPAC
Cuts off after lag 1; specifically: Dies down in a fashion
 1
dominated by damped
1 
,
2
1  1
exponential decay
 k  0 for k  2
Second-order moving average
Cuts off after lag 2; specifically: Dies down according
 1 (1   2 )
to a mixture of
zt    e t 1e t 1  2e t 2
1 
,
2
2
damped exponentials
1  1   2
and/or damped sine
 2
2 
,
waves
2
2
1  1   2
 k  0 for k  2.
25
Summary of the Behaviors of
TAC and TPAC (IV)
Behaviors of TAC and TPAC for specific non-seasonal models
Model
Mixed autoregressivemovingaverage of order (1,1)
zt    1 zt 1  e t  1e t 1
TAC
Dies down in a damped
exponential fashion;
specifically:
1 
(1  11 )(1  1 )
,
2
1  1  211
TPAC
Dies down in a
fashion dominated
by damped
exponential decay
 k  1  k 1 for k  2
26
Sample Autocorrelation
Function (SAC) (I)
 For the working series zb, zb+1, , zn, the
sample autocorrelation at lag k is
nk
rk 
 z
t b
t
 z  zt  k  z 
n
 z
t b
where
 z
2
t
n
z
z
t b
t
n  b 1
27
Sample Autocorrelation
Function (SAC) (II)
 rk measures the linear relationship between
time series observations separated by a lag of
k time units
k 1
 The Standard error of rk is sr 
k
1  2 r
j 1
2
j
n  b 1
.
rk
 The trk statistic is t rk 
.
srk
28
Sample Autocorrelation
Function (SAC) (III)
 Behaviors of SAC
(1) The SAC can cut off. A spike at lag k exists
in the SAC if rk is statistically large. If
t rk  2
Then rk is considered to be statistically large.
The SAC cuts off after lag k if there are no
spikes at lags greater than k in the SAC.
29
Sample Autocorrelation
Function (SAC) (IV)
(2) The SAC dies down if this function does not
cut off but rather decreases in a
‘steady
fashion’. The SAC can die down in
(i) a damped exponential fashion
(ii) a damped sine-wave fashion
(iii) a fashion dominated by either one of
or a combination of both (i) and (ii).
The SAC can die down fairly quickly or
extremely slowly.
30
Sample Autocorrelation
Function (SAC) (V)
 The time series values zb, zb+1, …, zn should
be considered stationary, if the SAC of the
time series values either cuts off fairly
quickly or dies down fairly quickly.
 However if the SAC of the time series values
zb, zb+1, …, zn dies down extremely slowly,
then the time series values should be
considered non-stationary.
31
Sample Partial Autocorrelation
Function (SPAC) (I)
 The sample partial autocorrelation at lag k is
r1

k 1

 rk   rk 1, j  rk  j
rkk  
j 1
k 1

 1   rk 1, j  rk

j 1

where rkj  rk 1, j
for j = 1, 2, …, k-1.
if k  1,
if k  2,3, 
 rkk rk 1,k  j
32
Sample Partial Autocorrelation
Function (SPAC) (II)
 rkk may intuitively be thought of as the
sample autocorrelation of time series
observations separated by a lag k time units
with the effects of the intervening
observations eliminated.
 The standard error of rkk is sr
kk
 The trkk statistic is trkk
rkk

.
srkk
1

.
n  b 1
33
Sample Partial Autocorrelation
Function (SPAC) (III)
 Behaviors of SPAC similar to its of the SAC.
The only difference is that rkk is considered to
be statistically large if
t rkk  2
for any k.
34
Sample Partial Autocorrelation
Function (SPAC) (IV)
 The behaviors of the SAC and the SPAC of a
time series data help to tentatively identify a
Box-Jenkins model.
 Each Box-Jenkins model is characterized by
its theoretical autocorrelation (TAC) function
and its theoretical partial autocorrelation
(TPAC) function.
35
Step Two: Parameter Estimation
36
Parameter Estimation
 Given n observations y1, y2, …, yn, the likelihood
function L is defined to be the probability of
obtaining the data actually observed.
 For non-seasonal Box-Jenkins models, L will be a
function of the , ’s, ’s and e2 given y1, y2, …, yn.
 The maximum likelihood estimators (m.l.e.) are
those value of the parameters for which the data
actually observed are most likely, that is, the values
that maximize the likelihood function L.
37
Step Three: Diagnostic Checking
38
Diagnostic Checking
 Often it is not straightforward to determine a
single model that most adequately represents
the data generating process. The suggested
tests include
(1) residual analysis,
(2) overfitting,
(3) model selection criteria.
39
Residual Analysis
 If an ARMA(p,q) model is an adequate
representation of the data generating process,
then the residuals should be uncorrelated.
 Use the Box-Pierce statistic
k
Q(k )  (n  d ) rl 2 (e) ~  (2k  p q )
l 1
 or the Ljung-Box-Pierce statistic
2
r
*
l (e)
Q (k )  (n  d )(n  d  2)
~  (2k  p q )
l 1 n  d  l
k
40
Overfitting
 If an ARMA(p,q) model is specified, them
we could estimate an ARMA(p+1,q) or an
ARMA(p,q+1) process.
 Then we check the significance of the
additional parameters (but be aware of
multicollinearity problems),
41
Model Selection Criteria
 Akaike Information Criterion (AIC)
AIC = -2 ln(L) + 2k
 Schwartz Bayesian Criterion (SBC)
SBC = -2 ln(L) + k ln(n)
where L = likelihood function
k = number of parameters to be estimated,
n = number of observations.
 Ideally, the AIC and SBC will be as small as
possible
42
Step Four: Forecasting
43
Forecasting
 Given a the stationary series zb, zb+1, , zt,
we would like to forecast the value zt+l.
zˆt l (t ) = l-step-ahead forecast of zt+l made at time t,
etl (t )= l-step-ahead forecast error
= zt l  zˆt l (t ).
 The l-step-ahead forecast is derived using
the minimum mean square error forecast and
is given by
zˆt l (t )  E( zt l | zb , zb1 ,, zt )
44
Forecasting with AR(1) model (I)
 The AR(1) time series model is
zt    1 zt 1  e t
where et ~ N(0,e2).
 1-step-ahead point forecast
zˆt 1 (t )  E (  1 zt  e t 1 | zb , zb1 ,, zt )
   1 zt  E (e t 1 | zb , zb1 ,, zt ).
45
Forecasting with AR(1) model (II)
 Recall that et+1 is independent of zb, zb+1, …,
zn and it has a zero mean. Thus,
zˆt 1 (t )    1 zt .
 The forecast error is
et1 (t )  zt 1  zˆt 1 (t )
   1 zt  e t 1  (  1 zt )
 e t 1 .
 Then the variance of the forecast error is
2
var[et1 (t )]  var(e t 1 )   e .
46
Forecasting with AR(1) model (III)
 2-step-ahead point forecast
zˆt 2 (t )    1 zˆt 1 (t ).
 The forecast error is
et 2 (t )  zt  2  zˆt  2 (t )
   1 zt 1  e t  2  (  1 zˆt 1 (t ))
 e t  2  1 ( zt 1  zˆt 1 (t ))
 e t  2  1et1 (t )  e t  2  1e t 1 .
 The forecast error variance is
var[et2 (t )]  var(e t 2  1e t 1 )   e2  12 e2   e2 (1  12 ) .
47
Forecasting with AR(1) model (IV)
 l-step-ahead point forecast
zˆt l (t )    1 zˆt l 1 (t )
 The forecast error is
for l  2.
2

et l (t )  e t l  1e t l 1  1 e t l  2
  e
l 1
1
t 1
for l  1.
 The forecast error variance is
1  12l 2
var[etl (t )] 
 e for l 1.
2
1  1
48
Forecasting with MA(1) model (I)
 The MA(1) model is
zt    e t  1e t 1
where et ~ N(0,e2).
 l-step-ahead point forecast
  1e t
zˆt l (t )  

for l  1,
for l  2 .
49
Forecasting with MA(1) model (II)
 The forecast error is
e t 1
etl (t )  
e t 1  1e t l 1
for l  1,
for l  2 .
 The variance of forecast error is

 e
var[etl (t )]  
2
2

(1  1 ) e
2
for l  1,
for l  2 .
50