UNIVERSITY OF LIMPOPO
TURFLOOP CAMPUS
FACULTY OF SCIENCE AND AGRICULTURE
SCHOOL OF MATHEMATICAL AND COMPUTER SCIENCES
DEPARTMENT OF STATISTICS & OPERATIONS RESEARCH
SSTA031
TIME SERIES ANALYSIS LECTURE NOTES
2019
TIME SERIES ANALYSIS LECTURE NOTES
2019
COURSE DESCRIPTIVE
In this course we look at the applied issues of Time Series Analysis. We will combine theoretical work
with the use of computer techniques for model solution. The purpose of this series of lectures is to
provide students with sufficient background in modern Time Series Analysis to choose techniques
suited both to the data-source and the Time Series model. The course places some emphasis on the
link between Time Series theory and forecasting estimation and deals explicitly with interpretation and
critical appraisal of forecasting estimates.
LEARNING OUTCOMES
After successful completion of the module, the student should be able to :
i.
Understand the basic theory of time series analysis and forecasting approaches;
ii.
Synthesize the relevant statistical knowledge and techniques for forecasting;
iii.
Use procedures in popular statistical software for the analysis of time series and forecasting;
iv.
Interpret analysis results and make recommendations for the choice of forecasting methods;
v.
Produce and evaluate forecasts for given time series;
vi.
Present analysis results of forecasting problems.
Lecturer
: Mr K.N Maswanganyi
Office Number
: 2004D (Maths Building)
Tel Number
: 015 268 3680
Lecturing Venue
: Check the Main Timetable
Lecture Notes
: A copy of Lecture notes will be available on UL blackboard. The notes do
not include proofs of stationarity/invertibility, so students are advised to take
additional notes in class.
2|Page
TIME SERIES ANALYSIS LECTURE NOTES
2019
READING LIST
Although lecture notes are provided, it is important that you reinforce this material by referring to more
detailed texts.
RECOMMENDED TEXTS



Chris Chatfield (2004).The Analysis of Time Series, An Introduction, Six Edition, Chapman
& Hall/CRC.
Cryer, D. C. and Chan, K (2008). Time Series Analysis with Application in R, 2nd Edition,
Springer.
Wei W.W.S (2006) Time Series Analysis, Univariate and Multivariate Methods, Second
Edition, Pearson Addison Wesley.
SUPPLEMENTARY TEXTS




Cryer, D. J. (1986) Time Series Analysis, Duxbury Press
Abraham B. and Ledolter J. (1983) Statistical Methods for Forecasting
Wiley Series, New York
Peter J. Brockwell and Richard A. Davis (2002), Introduction to Time Series and Forecasting,
Second Edition, Springer.
TIME SCHEDULE
The lecture topics within the semester are as in the following schedule:
Week
1
2
3
4
5
6
7
8
Dates
28 Jan-01 Feb
04-08 Feb
11-15 Feb
18 -22 Feb
25 Feb-01 Mar
04 -08 Mar
11-15 Mar
18-22 Mar
Topics
Introduction to Time Series
Introduction to Time Series
The Model Building Strategy
Models for Stationary Time Series
Models for Stationary Time Series
Models for Stationary Time Series
Models for Non-Stationary Time Series
Models for Non-Stationary Time Series
Chapters
1
1
2
3
3
3
4
4
AUTUMN RECESS :25-29 March
9
10
11
12
13
14
15
01-05 April
08 -12 Apr
15-19 Apr
22 -26 Apr
29 -03 May
06-10 May
13-17 May
3|Page
Models for Non-Stationary Time Series
Parameter Estimation
Parameter Estimation
Model Specification and Diagnostics
Forecasting
Forecasting
REVISION WEEK
4
5
5
6
7
7
TIME SERIES ANALYSIS LECTURE NOTES
2019
NOTATIONS
Symbol
Description
xt
Observed time series at time t
xt 
Observed time series for all t
k
k -th order differencing
 ks
Seasonal differencing
 x 
Mean of x
 k and ACVF
Autocovariance function
 k and ACF
Autocorrelation function
 kk and PACF
Partial autocorrelation function

WN 0,  2

N 1,  2


White noise with mean 0 and variance  2
Normally distributed with mean 1 and variance  2
IID
Independent, identically distributed
AR( p )
Autoregressive model of order p
MA( q )
Moving average model of order q
ARMA( p, q )
Autoregressive moving average model of order ( p, q )
ARIMA( p, d , q )
Intergrated ARMA( p, q ) model
SARIMA
Seasonal ARIMA
B
Backward shift operator.
4|Page
TIME SERIES ANALYSIS LECTURE NOTES
2019
CONTENTS
_____________________________________________________
1
Course Descriptive
2
Reading list & Time schedule
3
Notations
4
Introduction to Time Series
9
1.1
Introduction
9
1.2
Why do we need Time Series?
9
1.3
Time Series plots
10
1.4
General Approach to Time Series Modelling
10
1.5
Component of a Time Series
10
1.5.1 Trend (T)
10
1.5.2 Seasonal variation (S)
11
1.5.3 Cyclical variation (C)
11
1.5.4 Irregular variation (I)
11
1.6
Decomposition of a time series
12
1.7
Smoothing Methods
12
1.7.1 Moving Averages
12
1.7.2 Running Median
12
1.7.3 Exponential Smoothing
12
Trend Analysis
13
1.8.1 Methods for Trend Isolation
13
1.8.2 Calculating Moving Average
13
1.8
5|Page
TIME SERIES ANALYSIS LECTURE NOTES
1.9
2
1.8.3 Regression Analysis
14
Seasonal Analysis
15
1.9.1 Ratio-to-Moving Average Method
16
The Model Building Strategy
18
2.1
Introduction
18
2.2
The Box-Jenkins Technique
18
2.2.1 Model Specification
18
2.2.2 Model Fitting
19
2.2.3 Model Diagnostic
19
Stationary Time Series
19
2.3.1 Transformations
20
2.3.2 Stationary through differencing
21
Analyzing Series Which Contains a Trend
21
2.4.1 Filtering
21
2.5
Stochastic Processes
22
2.6
Mean,Variance & Covariances
22
2.3
2.4
3
2019
Models for Stationary Time Series
23
3.1
Introduction
23
3.1.1 Strictly Stationary Processes
23
3.1.2 Weakly Stationary Processes
23
3.2
Autocorrelation Function of Stationary Processes
24
3.3
Purely Random Process
24
3.4
Random Walk
25
3.5
Moving Average Processes
26
3.5.1 MA(1)
26
3.5.2 MA(2)
26
3.5.3 MA( q )
27
6|Page
TIME SERIES ANALYSIS LECTURE NOTES
3.6
Partial Autocorrelation Function
28
3.7
Autoregressive Processes
28
3.7.1 AR(1)
28
3.7.2 AR(2)
29
3.7.3 AR( p )
30
3.8
The Dual Relationship between AR and MA Processes
30
3.9
Yule-Walker Equation
32
3.10
Mixed ARMA Models
33
3.10.1 ARMA(1,1)
34
3.10.2 Weights ( ,  )
34
Seasonal ARMA Models
35
3.11
4
5
Model for Non Stationary series
35
4.1
ARIMA Models
35
4.2
Non Stationary Seasonal Process
36
4.2.1 SARIMA Model
36
Parameter Estimation
37
5.1
Introduction
37
5.2
Methods of Moments
37
5.2.1 Mean
37
5.2.2 Autoregressive Model
39
5.2.3 Moving Average Models
39
The Least Square Estimation (LSE)
39
5.3.1 Autoregressive Models
40
Confidence Interval for Mean
40
5.3
5.4
6
2019
Model Diagnostics
41
6.1
41
7|Page
Introduction
TIME SERIES ANALYSIS LECTURE NOTES
6.2
6.3
7
6.1.1 Residual Analysis
41
6.1.2 Test of Independence
42
6.1.3 Test for Normality
42
6.1.4 Test of Constant Variance
42
Autocorrelation of Residuals
42
6.2.1 Test for combined residual ACF
42
Over Fitting
43
Forecasting
44
7.1
MME
44
7.1.1 ARMA Model Forecast
44
Computation of Forecasting
44
7.2.1 Forecast Error and Forecast Error Variance
44
7.3
Prediction Interval
45
7.4
Forecasting AR( p ) and MA( q ) Models
45
7.5
Forecasting ARMA( p, q ) Models
46
7.6
Forecasting ARIMA ( p, d , q ) Models
46
7.2
9
2019
APPENDIX A
8|Page
47
TIME SERIES ANALYSIS LECTURE NOTES
2019
CHAPTER 1
INTRODUCTION TO TIME SERIES
1.1 Introduction
What is time series? A time series may be defined as a set of observations of a random variable
arranged in chronological (time) order. We also say it’s a series of observations recorded sequentially
at equally spaced intervals of time. Let us look at a few examples in order to appreciate what we
mean by a time series.
Example: 1.1.1
The daily temperature recorded over a period of a year. There are a time series because they are
recorded at equally spaced intervals of time and they are recorded regularly.
Example: 1.1.2
The hourly temperature readings of a machine in a factory constitutes a time series. The fact that the
temperature readings are taken every hour makes the temperature readings a time series.
1.2 Why do we need time series?
The aim of time series is “ to identify any recurring patterns which could be useful in estimating future
values of the time series”. Time series analysis assumes that the actual values of a random variable
in a time series are influenced by a variety of environmental forces operating over time.
Time series analysis attempts to isolate and quantify the influence of these different environmental
forces operating on the time series into a number of different components. This is achieved through a
process known as decomposition of the time series.
Once identified and quantified, these components are used to estimate future values of the time
series. An important assumption in time series analysis is the continuation of past patterns into the
future ( i.e the environment in which the time series occurs is stable.)
Notation:

The time series is denoted by xt , t  T where T is the index.

If T is continuous,we have a continuous time series.

If T is discrete,we have a discrete time series,and T   ,the set of all integers. The time
series is sometimes written as ..., x2 , x1 , x0 , x1 x2 ,...
9|Page
TIME SERIES ANALYSIS LECTURE NOTES

2019
For simplicity, we will drop the index set and write xt  or x1 , x2 , x3 ,... to indicate that they are
observation.

In practice, the time series interval for collection of time series could be
seconds,minutes,hours,days,weeks,months,years or any reasonable regular time intervals.
1.3 Time Series plots
The most important step in time series analysis is to plot the observations against time. This graph
should show up important features of the series such as a trend, seasonality, outliers and
discontinuities. The plot is vital, both to describe the data and to help in formulating a sensible model.
This is basically a plot of the response or variable of interest  x  against time t  , denoted xt .
1.4
General Approach to Time Series Modeling

Plot the series and examine the main features: This is usually done with the aid of some
computer package e.g. SPSS, SAS, etc.

Reform a transformation of the data if necessary.

Remove the components to get stationary residuals by differencing the data.(i.e Replacing the
original series xt by yt  xt  xt 1 .

Choose a model to fit the residuals.

Do the forecasting
1.5
Component of a time series
One way to examine a time series is to break it into components. A standard approach is to find
components corresponding to a long –term trend, any cyclic behavior, seasonal behavior and a
residual, irregular part.
1.5.1 Trend (T)
A smooth or regular underlying movement of a series over a fairly long period of time. A gradual and
consistent pattern of changes.
10 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
2019
Example: 1.5.1 (Trend component)
y = -0.0864x + 381.6
Clothing and softgoods sales
$(m illion)
500
Clothing
1999
dollars
Estimated
trend
450
400
350
Linear
(Estimated
trend)
300
250
Mar
1991
Mar
1993
Mar
1995
Mar
1997
Mar
1999
Mar
2001
Mar
2003
1.5.2 Seasonal variation (S)
Movement in a time series which recur year after in some months or quarters with more less the
same intensity.
Example: 1.5.2 (Seasonal component)
1.5.3 Cyclical variation (C)
Period variations extending over a long period of time, caused by different factors such as cycles,
recession, depression, recovery, etc.
1.5.4 Irregular variation (I)
Variations caused by readily identifiable special events such as elections, wars, floods, earthquakes,
strikes, etc.
11 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
2019
Example: 1.5.3 (Irregular component)
1.6 Decomposition of a time series
The main time series analysis is to isolate the influence of each of the four components of the actual
series. The multiplicative time series model is used to analyze the influence of each of these four
components. The multiplicative model is based on the idea that the actual values of a time series xt
can be found by multiplying the trend component T  by cyclical component C  , by seasonal index
S  and by irregular component I 
. Thus, the multiplicative time series is defined as: x  T  C  S  I .
Another model we can use is the additive model given by: x  T  C  S  I .
1.7 Smoothing methods
Smoothing methods are used in attempting to get rid of the irregular, random component of the
series.
1.7.1 Moving averages:
A moving average (ma) of order M is produced by calculating the average value of a variable over a
set of M values of the series.
1.7.2 Running median:
A running median of order M is produced by calculating the median value of a variable over a set of
M values of the series.
1.7.3 Exponential smoothing:
xˆt 1   xt  1    xˆt
Where
12 | P a g e
xˆt 1  the exponential smoothing forecast at time t.
TIME SERIES ANALYSIS LECTURE NOTES
2019
x̂t  the old forecast.
xt  the actual value (observation) at time t.
1.8 Trend Analysis
The trend in a time series can be identified by averaging out the short term fluctuations in the series.
This will result in either a smooth curve or a straight line.
1.8.1 Methods for trend isolation
(a) The moving average method: Produces a smooth curve.
(b) Regression analysis method: Involves fitting a straight line.
1.8.2 Calculating moving average

The three –year moving total for an observation x would be the sum of the observation
immediately before x , x itself and the observation immediately after x .

The three- year moving average would be each of these moving totals divided by 3.

The five-year moving total for an observation x would be the sum of the two observations
immediately before x , x itself and the two observations immediately after x .

The five-year moving average would be each of these moving totals divided by 5.
To illustrate the idea of moving averages, let us consider the observations in example 1.8.1
13 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
2019
Example: 1.8.1
Australia’s official development assistance (ODA) from 1984-85 until 1992-93 is shown (at current
prices, $ million) in Table 1.
Table 1
Year
ODA($
million)
1984-85
1011
1985-86
1031
1986-87
976
1987-88
1020
1988-89
1195
1989-90
1174
1990-91
1261
1991-92
1330
1992-93
1384
(a) Find the three- year moving averages to obtain the trend of the data.
(b) Find the five-year moving averages for the data.
1.8.3 Regression Analysis
A trend line isolates the trend (T) component only. It shows the general direction in which the series
is moving and is therefore best represented by a straight line. The method of least squares from
regression analysis is used to determine the trend line of best fit.
Regression line is defined by: y   0  1 x . If the variable x , are not given, must be coded.
Methods for coding the time variable,
(a) The sequential numbering method.
(b) The zero-sum method.
14 | P a g e
x
TIME SERIES ANALYSIS LECTURE NOTES
2019
Coding x using Zero-sum method


 n 1
To code x when the number of time periods, n is odd, we assign a value of  
 to the first
 2 
time period and for each subsequent period, add one to the previous period’s x value.
To code x when the number of time periods, n is even, we assign a value of  n  1 to the
first time period and for each subsequent period, add two to the previous period’s x value.
Example 1.8.2
Table 2
Year
1977
1978
1979
1980
1981
1982
1983
Y
2
6
1
5
3
7
2
a) Calculate the regression line using sequential numbering method.
b) Calculate the regression line using Zero-sum method.
Exercise: 1.8.1
Consider the monthly earning of a small business.
Table 3
Year
1977
1978
1979
1980
1981
1982
1983
Y
12
14
18
17
13
4
17
a) Find the 3 point moving average.
b) Find the least squares trend line for small business using Zero- sum method.
c) Find the least squares trend line for small business using sequential method.
1.9 Seasonal Analysis
Seasonal analysis isolates the influence of seasonal forces on a time series. The ratio-to-moving
average method is used to measure these influences. The seasonal influence is expressed as an
index number.
15 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
2019
1.9.1The ratio-to-moving average method
Step-1
Identify the trend/cyclical movement.
The moving average approach is used to isolate the trend/cyclical movement in a time series.
Step-2
Find seasonal ratio using the formula:
y
Actual y
 100%  t  100%
Moving average series
MA
T C  S  I

 S  I  100%
T C
seasonal ratio 
Seasonal ratio is an index which expresses the percentage deviation of each actual y(which includes
seasonal influences) from its moving average value(which contains trend and cyclical influences
only).
Step-3
Average the seasonal ratios across corresponding periods within each year.
The averaging of seasonal ratios has the effect of smoothing out irregular component inherent in the
seasonal ratios. Generally, the median is used to find the average of seasonal ratios for correspond
periods.
Step-4
Compute adjusted seasonal indices. The adjusted factor is determined as follows:
Adjusted factor 
k  100
 Median seasonal indices
De-seasonalising the actual time series
Seasonal influences may be removed from a time series by dividing the actual y value for each
period by its corresponding seasonal index.
That is, Deseasonalised y 
16 | P a g e
Actual y
 100
Seasonal index
TIME SERIES ANALYSIS LECTURE NOTES
2019
Example: 1. 9.1
The average daily sales (in litres) of milk at a country store are shown in Table 4 for each of the years
1983 to 1985.
Table 4
Year
Quarter Average daily
sales ( Yt )
1
47.6
2
48.9
1983 3
51.5
4
55.3
1
57.9
1984 2
61.7
3
65.3
4
70.2
1
76.1
1985 2
84.7
3
93.2
4
97.2
(a) Find the four- year moving averages.
(b) Calculate the seasonal index by making use of the ratio-to-moving average method.
Exercise: 1.9.1
Table 5
Year
1993
1994
Quarter
1
2
3
4
1
2
3
4
actuals
10
20
30
16
29
45
25
50
17 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
2019
(a) Find the three- year moving averages.
(b) Calculate an uncentred four –quarter moving average.
(c) Calculate centred moving averages for these data.
(d) Find the adjusted seasonal index for each quarter.
CHAPTER 2
THE MODEL BUILDING STRATEGY
2.1 Introduction
Perhaps the most important question we ask now is “how do we decide on the model to use?”
Finding appropriate models for time series is not an easy task. We will develop a model building
strategy which was developed by Box and Jenkins in 1976.
2.2 The Box-Jenkins Technique
There are three main steps in the Box-Jenkins procedure, each of which may be used several times:
1) Model specification
2) Model fitting.
3) Model diagnostics.
2.2.1 Model specification
In model specification (or identification) we select classes of time series that may be appropriate for a
given observed series. In this step we look at the time plot of the series, compute many different
statistics from the data, and also apply knowledge from the subject area in which the data arise, such
as economics, physics, chemistry, or biology. The model chosen at this point is tentative and may be
revised later in the analysis. In the process of model selection we shall try to adhere to the principle
of parsimony.
Definition 2.2.1 (The principle of parsimony) :The model used should require the smallest possible
number of parameters that will adequately represent the data.
a) Test for white noise
For the data set to be purely random sequence/white noise the sample autocorrelation
18 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
2
̂ k 
2019
(i.e the sample ACF must be within the boundaries).
n
Example 2.2.1
200 observations on a stationary series were analyzed and gave the following sample
autocorrelation:
Table 6
k
1
2
3
4
5
̂ k
0.59
0.56
0.46
0.38 0.31
a) Is the data set a white noise?
2.2.2 Model fitting
Model fitting consists of finding the best possible estimates of those unknown parameters within a
given model. After we have identified the model and estimated the unknown parameters we need to
check if the model is a good model, this is done through diagnostic checking.
2.2.3 Model Diagnostics
Here we are concerned with analyzing the quality of the model that we have specified and estimated.
We ask the following questions to guide us:
1) How well does the model fit the data?
2) Are the assumptions of the model reasonably satisfied?
If no in adequacies are found, the modeling may be assumed to be complete, and the model can be
used, for example, to forecast future values of the series. Otherwise we choose another model in light
of the inadequacies found: that is we return to model specification. In this way we cycle through the
three steps until an acceptable model is found.
2.3 Stationary time series
Definition 2.3.1: A time series is said to be stationary if there is no systematic change in mean (no
trend), if there is no systematic change in variance, and if strictly periodic variations have been
removed.
It should be noted that in real life it is not often the case that a stochastic process is stationary. This
could arise due to, for example,
1) Change of policy on the process.
19 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
2019
2) Slump on the process.
3) Improvement on the process, etc.
2.3.1 Transformations
If the process is not stationary to analyze its series we must transform the series to be stationary.
There are various transformations that we can use to make a time series stationary. Some of them
are:
1) Differencing.
2) Log transformation.
3) Square root transformation.
4) Arcsine transformation.
5) Power transformation.
The three main reasons for making a transformation are as follows:
(a) To stabilize the variance:
If the variance of the series increases with the mean in the presence of trend, then it is advisable to
transform the data. A logarithmic transformation is appropriate if the standard deviation is directly
proportional to the mean.
(b) To make the seasonal effect additive:
If size of the seasonal effect appears to increase with the mean in the presence of the trend, then it is
advisable to transform the data so as to make the seasonal effect ADDITIVE. If the size of the
seasonal effect is directly proportional to the mean, then the seasonal effect is said to be
Multiplicative and the logarithmic transformation is appropriate to make the seasonal effect additive.
(c) To make the data normally distributed:
Often the general class of transformations called the Box-Cox transformation given by



xt  1

Yt  
,  0



, 0
 log xt


For some transformation parameter . the logarithmic and square root transformations are special
cases of this general class.
20 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
2019
2.3.2 Stationarity through differencing
Models that are not stationary when subjected to differencing often yield processes that are
stationary. Thus if we difference a time series we denote it by  xt  xt  xt 1 . We will use the operator
 to denote the difference operation. In some instances, differencing ones may not yield a stationary
process.
In that situation we continue differencing the series until it is stationary. Once the process is
stationary there is no need to continue differencing it otherwise we will over difference it.
Example 2.3.1(Second order differences)
2 xt  xt   xt  2 xt 1  xt  2
Exercise 2.3.1
Suppose we have a process given by xt  5  2t  zt where z t is white noise with mean zero and
variance  z2 .
1) Show that xt is not stationary.
2) Verify that the process is now stationary if we difference it once.
2.4 Analyzing Series Which Contain a Trend
2.4.1 Filtering
Definition 2.4.1: A linear filter is a linear transformation or any operator which converts one time
series, xt called the input or leading indicator series into another series yt called the output series
through the linear operation y t   a j xt  j or schematic:
j
xt  Filter  yt
Note that filter can be in a series,
zt   b j yt  j
j
  b j  a r xt  j  r
j
r
  c k xt  k
k
where c k   a r bk  r 
r
21 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
2019
, are the weights for the overall filter. The weights ck  are obtained by a procedure called
convolution and symbolically expressed as ck   ar * b j 
Example: 2.4.1
1 1 
1 1 
(a) Consider the following two filters: A   ,  , B   ,  Compute A* B where *denotes the
2 2
2 2
convolution operator.
(b) Consider the following two filters: A   1,1, B   1,1Compute A* B where *denotes the
convolution operator.
Exercise: 2.4.1
Calculate the convolution of the following filters:
1 1 1 1 1 1 1 1
a)  , , ,  *  , , , 
4 4 4 4 4 4 4 4
1 1 1 1 1 1 1
b)  , , ,  *  , , 
 4 4 4 4  3 3 3
2.5 Stochastic processes
Definition 2. 5.1: A time series x0 , x1 ,... is a sequence of observation. More technically a time series
is a sample path or realization of a stochastic (random) process xt , t  T where T is an ordered set.
Definition 2.5.2: Let xt , t  T be a stochastic process and let  be a set of all possible realization or
sample of path then  is called the ensemble for the process xt . An individual time series is a
member of ensemble.
Remarks: In time series literature, the terms “time series” and process are used (often)
interchangeably.
2.6 Means, Variances and Covariances
The mean and autocovariance function (ACVF) are given by:
 t  xt  and  t k  xt   t xt k  t k  respectively.
The variance function  t is defined by var xt    t
2
22 | P a g e
2
TIME SERIES ANALYSIS LECTURE NOTES
2019
CHAPTER 3
MODELS FOR STATIONARY TIME SERIES
3.1 Introduction
In order to be able to analyze or make meaningful inference about the data generating process it is
necessary to make some simplifying and yet reasonable assumption about the process. A
characteristic feature of time series data which distinguishes it from other types of data is that the
observations are, in general, correlated or dependent and one principal aim of time series is to study,
investigate, explore and model this unknown correlation structure.
3.1.1 Strictly Stationary Processes
Definition 3.1.1: A time series xt is said to be strictly stationary if the joint density functions
depend only on the relation location of the observations, so that f xt1 h , xt 2 h ,..., xtk h   f xt1 , xt 2 ,..., xtk .
meaning that xt1 h , xt 2 h ,..., xtk  h  and xt1 , xt 2 ,..., xtk  have the same joint distributions for all h and for
all choices of the time points t i .
Example: 3.1.1
Let n=1, the distribution of xt is strictly stationary if  t   and  t   are both constant. And if n=2,
2
the joint distribution of xt 1 and xt 2 depend only on t 2  t1 , which is called the lag. Thus the
autocovariance function  t1 ,t 2  depends only on t 2  t1  and is written as  k  , where
 k   xt   xt  k   
 covxt , xt  k 
And  k  is called the autocovariance coefficient at lag k.
3.1.2 Weakly Stationary Processes
Definition 3.1.2: A stochastic process z t is weakly stationary (or of second order stationary), if both
the mean function and the autocovariance function do not depend on time t .thus,
 t  xt  = (a constant) and  t  k   k = (a constant).Note that  t k  cov( xt , xt k )
Example: 3.1.2
Prove or disprove the following process is covariance stationary:
(a) z t   1 A , where A is a random variable with zero mean and unit variance?
t
23 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
2019
Exercise: 3.1.1
Consider the following time sequence xt  z0 cosct  .
Where z0  is a sequence of independent normal r.v with mean 0 and variance  z2 .
Show that xt  is not a covariance stationary.
3.2 Autocorrelation function of stationary processes
Definition 3.2.1: Suppose a stationary stochastic process xt  has mean  , variance  2 and ACVF
 k  , then the autocorrelation function (ACF) is given by:  k  
 k   k 

 0   2
Note that  0  is the variance of the series given by  2 and  0   1 .
Properties
1. The ACF is an even function, so that  k    k  i.e. the correlation between xt  and xt  k  is the
same as the correlation between xt  and xt k  .
2.  k   1
3. The ac.f does not uniquely identify the underlying model. That is why invertibility has to be checked
in moving average processes.
Exercise: 3.2.1
Prove the following:
a)  k    k 
b)  k   1
3.3 Purely random process
A very important example of a stationary process is the so-called white noise process. One simple
definition is that a white noise is a (univariate or multivariate) discrete-time stochastic process whose
terms are independent and identically distributed (IID), with zero mean. While this definition captures
the spirit of what constitutes a white noise, the IID requirement is often too restrictive for applications.
Typically the IID requirement is replaced with a requirement that terms have constant second
moments, zero autocorrelations and zero means. Let’s formalize this.
24 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
2019
Definition 3.3.1: A discrete time process is called a purely random process (white noise) if it
consists of a sequence of random variable z t which are mutually independent and identically
distributed. It follows that both mean and variance are constants and the acv.f. is:
 k   covz t , z t  k 
 0 for k  1,2...
1 , for k  0
The ac.f.is given by  k   
0 , for k  1,2,...
Definition 3.3.2 (IID Noise Process)
A process xt  is said to be an IID noise with mean 0 and variance  x2 ,written xt ~ IID0,  x2  if the
random variable xt are independent and identically distributed with xt   0 and var xt    x2 .
3.4 Random walk
Let z1 , z 2 ,...be independent identically distributed random variables, each with mean 0 and variance
 z2 . The time series that can be observed xt  is called a random walk if it can be expressed as
follows:
x1  z1
x 2  z1  z 2
x3  z1  z 2  z 3


xt  z1  z 2  z 3    z t
.................
If z' s are interpreted as the size of ‘steps’ taken forward or backward at time t , then xt is the position
of a ‘random walk’ at time t .
From  we obtain the mean function:  t  xt   0 , variance of xt , var xt   t  z2 and the
covariance of xt and xs ,  t , s  covxt , x s   t  z2 .
Backshift operator
The backshift operator is used to express and manipulate time series models. The backshift operator
denoted B on the time index of a series and shifts time back 1 time unit to form a new series i.e:
Bxt   xt 1 .
25 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
2019
3.5 Moving average processes
Mathematically , we express a first order moving average model MA(1) as xt  z t  1 z t 1 ,where

 based on the notation in the text book a first order MA(1) process is given by
  z ,where z ~ N 0,  .Clearly the two model are the same with   1 and   
z t ~ N 0,  z
xt   0 z t
2
2
1 t 1
t
z
0
1
1
.usually in most practical situations  0 is standardized so that it is equal to one.
Schematically, a moving average of order 1 can be expressed as
xt  moving average  z t , z t 1
operator  filter 
3.5.1 First- order moving average [MA (1)] process
The MA (1) for the actual data, as opposed to deviations to deviation from the mean will be written as
xt    zt  1 zt 1 Or xt    zt  1 zt 1 where  is the mean of the series corresponding to the
intercept in the moving average case.
Example: 3.5.1(Backshift operator)
Consider an MA1 model. In terms of B, we can write xt  zt   zt 1  zt   B zt  1   B  zt .
Where  B  is the MA characteristic polynomial “evaluated at B”.
3.5.2 Second –order moving average [MA (2)] process
Invertible conditions
The second order moving average process is defined by xt  zt  1 zt 1   2 zt 2 and is stationary for all
values of  1 and  2 .However , it is invertible only if the roots of the characteristic equation
1  1 B   2 B 2  0 lie outside the unit circle, that is ,
(i)  2   1  1
(ii)  2   1  1
(iii)  1   2  1
Example: 3.5.2
Find the variance and ACF of the following process:
xt  zt  1 zt 1   2 zt 2
26 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
2019
Solution:


(a) The variance of the process is  0   z 2 1   21   2 2 .
(b) The ACF’s are 1 
2 
 1 1   2 
1  1   2
2
2
2
1  1   2
2
2
 3  0 for k  3
Exercise: 3.5.1
Find the ACF of the following first order MA processes:
(a) xt  zt  1 zt 1
(b) xt  z t 
1

z t 1 ,where z t is a white noise.
3.5.3 qth-order moving average [ MA (q) ] process.
Definition 3.5.1: Suppose that z t is a purely random process, such that E z t   0, var z t    z .then
2
a process xt is said to be a moving average process of order q (abbreviated as MA (q)) if
xt  z t  1 z t 1  ...   q z t q .Where  i are constants.
Example: 3.5.3
Consider a MA (q) given by xt  z t  1 z t 1  ...   q z t q .
Find ACVF and ACF of xt .
Exercise: 3.5.2
Find the ACF of the first-order moving average given by xt  z t  1 z t 1 .
27 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
2019
3.6 Partial autocorrelation function
Definition 3.6.1: The partial correlation (PACF) is defined as the correlation between xt and xt  k with
their linear dependency on the intervening variables xt k ,..., xt k 1 removed,
 kk  corr xt , xt  k xt 1 ,..., xt  k 1 .
 0 1 1
 11  1
,
 22
0

 1
0
1
 33 
1
2
1
0
1  0  2
 2 1  3
 0 1  2
1  0 1
 2 1  0
Exercise 3.6.1
Find the 11 ,  22 ,  33 of xt  zt  1 zt 1 .
ˆ kk  1
Definition 3.6.2: Standard error for 
n
3.7 Autoregressive processes
Definition 3.7.1: Let z t be a purely random process with mean zero and variance  z 2 . Then a
process xt is said to be an autoregressive process of order p if xt  1 xt 1  ...   p xt  p  z t .
An autoregressive process of order p will be abbreviated as AR ( p ).
3.7.1 First -order Autoregressive (Markov) process
Example: 3.7.1
Consider a model xt  1 xt 1  zt where xt is a white noise.
a) Find the ACVF and ACF .
Solution:
The first-order autoregressive process is xt  1 xt 1  zt where 1 must satisfy the following condition:
 1  1  1 for the process to be stationary.
28 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
2019
Using the backward shift operation B, the process xt  1 xt 1  zt may be written as 1  1 B xt  zt so


that xt  1  1 B  z t  1  1 B  1 B 2  ... Z t  z t  1 z t 1  1 z t 2  ...
1
2
2
xt   0


var xt    z 1  1  1  ...
2
2
4
Thus the variance is finite provided that 1  1 , in which case var xt    x
2
 z2

.
2
1  1
a

i.e s   1  r , where

1 4 1 2

2
 1
r  2 
1
1

and a  1


The ACVF is given by  k   xt xt  k   

1
i
z t i

1
j

zt k  j   z
1 k  z 2
k
2
 1  x
This converges for 1  1 to  k  
2
1  1


For k  0, we find  k     k  .
The ACF is given by  k   1 k for k  0,1,2,...
 int erger k ,  k   1
k
for k  0  1,2,...
  0  1
 1  1
Exercise: 3.7.1
Consider a model xt  0.7 xt 1  zt where z t is a white noise.
Find : ACV.F and ACF of xt .
3.7.2 Second – order autoregressive process.
Stationary condition
The second-order autoregressive process may be written as:
29 | P a g e
2

1
i
1 k i for k  0
TIME SERIES ANALYSIS LECTURE NOTES
2019
xt  1 xt 1  2 xt 2  z t .
For stationarity, the roots of  B   1  1 B  2 B 2  0 must lie outside the unit circle, which implies that
the parameters 1 and  2 must lie in the triangular region:
(i )  2  1  1
(ii)  2  1  1
(iii)  1   2  1
3.7.3 pth-order autoregressive process


2
p
xt  1 xt 1  2 xt 2  ...   p xt  p  z t or 1  1 B   2 B  ...   p B xt  z t
Stationary conditions:
The roots Bi , i  1,2..., p of the characteristic equation 1  1 B  2 B 2  ...   P B p  0 must lie outside the
unit circle.
3.8 The dual relation between AR & MA processes
Table 7
Process
ACF
PACF
Stationary
Invertible
condition
condition
AR( p )
Damps out
Cuts off after lagp
Roots of
characteristic
equation
outside unit
circle.
Always
invertible
MA( q )
Cuts off after lagq
Damps out
Always
stationary
Roots of
characteristic
equation
outside unit
circle.
ARMA(
p, q )
Damps out
Damps out
Roots of
characteristic
equation
outside unit
circle.
Roots of
characteristic
equation
outside unit
circle.
30 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
Example: 3.7.2
Theoretical behavior of the ACF and PACF for AR(1) and AR(2) models:
AR1: PACF  0 for lag  1 ; ACF  0
AR2: PACF  0 for lag  2 ; ACF  0
In this context………..
 “damps out/die out” means” tend to zero gradually”
 “cuts off” means ”disappear” or “is zero”.
Example: 3.7.3
Theoretical behavior of the ACF and PACF for MA(1) and MA(2) models:
MA1: ACF  0 for lag  1 ; PACF  0
31 | P a g e
MA2: ACF  0 for lag  2 ; PACF  0
2019
TIME SERIES ANALYSIS LECTURE NOTES
2019
3.9 Yule-Walker equation
Yule-Walker equation using the fact that  k     k  for all k is given by
 k   1  k  1  ...   p  k  p  , for all k  0
The general solution is  k   A1 1  ...  A p  p where  i ,are the roots of the auxiliary equation:
k
k
y p  1 y p 1  ...   p
Ai  Are constant,  0  1 and  Ai
1
The first ( p  1) Yule-Walker equations provide ( p  1) further restrictions on the Ai  using
 0  1 and  k     k  .From the general form of  k  , it is clear that  k  tends to zero as k
increases provided that  i  1 for all i , and this is a necessary and sufficient condition for the process
to be stationary.
For stationary, the roots of the equation  B   1  1 B  ...   p B p  0 must lie outside the unit circle.
Example: 3.9.1
Suppose we have AR (2) process, when  1 ,  2 are the roots of the quadratic equation y 2  1 y  2  0 ,
thus  i  1 if 1 

1
2
 4 2
2
  1.
When roots are real, the constants A1 , A2 are found as follows:
32 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
2019
Using Yule-Walker equations for AR (2),  k   1  k  1   2  k  2 we have
 1  1  0   2   1

 1  A1 1  A2 2
1  2
  1  A1 1  1  A1  2
 1 

 2
1   2 

Where A1 
and A2  1  A1
1   2
Example: 3.9.2
Consider the AR (2) process given by xt   xt 1 
1
xt  2  z t
4
Show that xt is stationary and then calculate its ACF.
Exercise: 3.9.1
Consider the AR (2) process given by xt 
1
2
xt 1  xt  2  z t
3
9
Is this process stationary? If so, what is its ACF?
______________________________________________________________________
Note: for real –valued linear different, a complex root of  B   0 must appear in pairs.
That is, if c  di  is a root, then its complex conjugate c  di*  c  di is also a root. A general
complex number can always be written in polar form, i.e
c  di    cos   i sin  
 c  dik   k cosk  i sin k 
1
d 
Where   c 2  d 2  c 2  d 2 2 And   tan 1   .
c
________________________________________________________________________________
3.10 Mixed ARMA models
Definition 3.10: A mixed autoregressive/moving –average process containing p AR terms and q MA
terms is said to be an ARMA process of order (p, q) and is given by:
xt  1 xt 1  ...   p xt  p  z t  1 z t 1  ...   q z t q .
33 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
2019
The backshift operator B, is given by  B xt   B zt where  B ,  B  are polynomials of order p, q
respectively, such that  B   1  1 B  ...   P B p and  B   1  1 B  ...   q B q .
Stationary process
The roots of  B   0 must lie outside the unit circle.
Invertible process
The roots of  B   0 must lie outside the unit circle.
3.10.1 ARMA (1, 1)
xt  1 xt 1  zt  1 zt 1 , using a backshift operator we have 1  1 B xt  1  1 z t .
Stationary and invertibility conditions
The process is stationary if  1  1  1 , and invertible if  1  1  1 .
Example: 3.10.1
Consider the following process: xt 
1
xt 1  z t  2 z t 1 .
2
a) Is the process stationary/invertible?
b) Find the AC.F
3.10.2 Weights ( ,  )
From the relations  1  1 0  1  1  1 and j  1 j 1 for j  1 , we find that the  j weights are
given by  j  1  1 1
j 1
, for j  1, and similarity it is easily seen that  j  1  1 1
j 1
, for j  1 ,for
the stationary and ARMA (1, 1) process.
The  weights or  weights may be obtained directly by division or by equating powers of B in an
equation such as B  B    B  .
Example: 3.10.2
Find the  weights and  weights for the ARMA (1, 1) process given by xt  0.5xt 1  zt  0.3zt 1 .
Exercise: 3.10.1
34 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
Consider the ARMA (1,1) process given by xt 
(a) Is this process stationary/invertible?
2019
1
xt 1  z t  z t 1
2
(b) Calculate the ACF.
Exercise: 3.10.2
Obtain the first 3 -weights and 3 -weights of the following models:
a) 1  1 B  2 B 2  xt  1   B  z t .


b) 1  3B  B 2 xt  z t
3.11 Seasonal ARMA models
In other situations we may have data with a seasonal component. In order to fit a model we need to
take this seasonality component into consideration.
Definition 3.11.1: A process xt  is called a seasonal ARMA process of non seasonal order p, q and
seasonal component P, Q and a seasonality order S if xt  satisfies  B B  xt   B B  zt .
Where,
B   1   B S   2 B 2 S     P B PS
 B   1   B   2 B 2     p B p
 B   1   1 B   2 B 2     q B q
B   1  1 B S   2 B 2 S     Q B QS
CHAPTER 4
MODEL FOR NON STATIONARY SERIES
4.1 ARIMA models
A series xt is said to follow Integrated Autoregressive- Moving Average (ARIMA) model if the d th
difference wt   d xt is stationary ARIMA process. If wt is ARMA p, q  , we say that xt is
ARIMA p, d , q  . For practical purpose we usually take d to be at most 2.
Note: If we want to check whether an ARMA p, q  is stationary we check the AR  p  part only.
Let xt  1 xt 1  2 xt 2     p xt  p  z t  1 z t 1   2 z t 2     q z t q be a non stationary process.
Writing wt   d xt  1  B  xt the general autoregressive integrated moving average process (ARIMA)
d
is of the form: wt  1 wt 1  ...   p wt  p  z t  1 z t 1  ...   q z t q
35 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
2019
Using back shift operator B, we have  B wt   B z t or  B 1  B  xt   B z t .thus we have an
d
ARMA  p, q  model for wt , while the model in equation above, describing the dth differences of xt , is
said to be an ARIMA process of order  p, d , q  .The model for xt is clearly non-stationary, as AR
operator  B 1  B d had d roots on the unit circle. If the value of d is taken to be one, the random
walk can be regarded as an ARIMA 0,1,0  process.
Exercise: 4.1.1
Identify the following models
a) 1  0.8B 1  B  xt  zt .
b) 1  B  xt  1  0.75B zt .
c)
1  0.9B1  B xt  1  0.5B zt
4.2 Non stationary seasonal processes
4.2.1. SARIMA model
Definition 4.2.1: A process xt  is called a seasonal ARIMA process of non-seasonal component
p, d , q and seasonal components P, D, Q if xt satisfies  B  B  d  sD xt   B  B  z t .
B   1   B S   2 B 2 S     P B PS
 B   1   B   2 B 2     p B p
Where:
 B   1   1 B   2 B 2     q B q
B   1  1 B S   2 B 2 S     Q B QS
Example: 4.2.1
Let consider a time series where a period consists of S seasons (for monthly data S  12 , for
quarterly data S  4 ,etc.)
d
Suppose a non-seasonal ARIMA model is fitted to the series i.e  p B 1  B  xt   q B at .......
where at is a white noise. This series can also be represented as ARIMA model
 
P BS 1 BS

D
 
at   Q B S bt ..............  
 
 
where  P B S  1  1 B S   2 B 2 S  ...   P B PS and  Q B S  1  1 B S   2 B 2 S  ...   Q B QS
 
 
It is assumed that the polynomials  P B S and  Q B S have no common roots and that their roots
lie outside the unit circle.
36 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
2019
combining  and   gives the multiplicative seasonal ARIMA model.
 

 P B S  P B 1  B  1  B S
d

D
 
xt   q B  Q B S at .This model is denoted as ARIMA p, d , q   P, D, QS
Example: 4.2.2
Let us consider the ARIMA0,1,1  0,1,112 model. Where Wt  1  B 1  B12  xt  1  B 1  B12  z t .
Find the autocovariance and the autocorrelations of Wt .
CHAPTER 5
PARAMETER ESTIMATION
5.1 Introduction
Having tentatively specified ARIMA p, d , q  the next step is to estimate the parameters of this model.
This chapter focuses on estimation of parameters of an AR and MA models. We shall deal with the
most commonly used method of estimating parameters, these are:
1) Method of moments.
2) Least square method
3) Maximum-likelihood method.
5.2 Method of moments
This method consist of equating sample moments such as the sample mean x ,sample variance  0
and sample autocorrelation function to the theoretical counterparts and solving the resultant
equation(s).
5.2.1 Mean
With only a single realization (of length n ) of the process, a natural estimator of the mean,  is the
sample mean x 
Since E x  
1 n
 xt .where x is the average time average of n observation.
n t 1
1 n
 E xt    , x is an unbiased estimator of 
n t 1
If xt  is a stationary process with autocorrelation function  k  then
37 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
var x  
2019
 0 
n 1

 k
1

2
1    k 


n 
n
k 1 

ESTIMATION OF
k
and
k
Suppose that we have n observations x1, x2 ,, xn then the corresponding sample autocovariance
and autocorrelation functions (or estimates) at lag k are:
nk
1 nk
ˆk   xt  x xt  k  x  and ̂ k 
n t 1
 x
t 1
t
 x xt  k  x 
n
 x
t 1
t
 x
2
n 1
1 n 1
As an example, ˆ1   xt  x xt 1  x  and ̂1 
n t 1
 x
t 1
t
 x  xt 1  x 
n
 x
t 1
t
 x
are used to estimate  1 and 1 .
2
Example: 5.2.1a
Table 8
t
1
2
3
4
xt
4
5
2
5
Estimate 1 ,  2 , 3 , and  4 for the time series given in table 8.
Example: 5.2.1b
Suppose that xt  is a MA1 process defined by xt  z t   z t 1 where  is a fixed parameter and


z t ~ IID 0,  Z . Where IID stand for Independent Identically Distributed noise.
2
Calculate var  x  in terms of  .
Exercise: 5.2.1
Suppose that xt  is moving average process given by xt 
white noise with var z t    z2 . Calculate var  x  .
38 | P a g e
1
z t  z t 1  z t 2  where zt  is zero mean
3
TIME SERIES ANALYSIS LECTURE NOTES
2019
5.2.2 Autoregressive model
Example: 5.2.2
Consider an AR 1 model: xt  1 xt 1  zt Find the estimate of 1 using the method of moments.
Exercise: 5.2.2
Consider an AR2 model: xt  1 xt 1  2 xt 2  zt Find the estimates 1 and  2 using the method of
moments.
5.2.3 Moving average models
Method of moments is not convenient when applied to moving average models. However, for
purposes of illustration, we shall consider MA1 process given in the following example:
Example: 5.2.3
Consider an MA1 model: xt  zt  1 zt 1 Find the estimate of  1 using the method of moments.
5.3 The Least Square Estimation (LSE)
The method of Least Square Estimation is an estimation procedure developed for standard
regression models. In this section we discuss LSE procedure and its associated problems in time
series analysis.
Recall: For sample linear regression model given by: yt   xt  zt
The Least Square Estimate is given by: ˆ 
, t  1,2,..., n
n  xt y t   xt  y t
n xt2   xt 
2
The estimate  is a consistent and best linear unbiased estimator of  . This holds under the
following basic assumptions on the error term z t :
1) Zero mean: z t   0 .
2) Constant variance: z t    z2 .
2
3) Non –autocorrelation: zt z k   0
for t  k .
4) Uncorrelated with explanatory variable: xt z t   0
39 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
2019
In the next subsection we shall apply the LSE method to time series model.
5.3.1 Autoregressive models
Example: 5.3.1
Consider an AR 1 model: xt  1 xt 1  zt
….*
Model *can be viewed as a regression model with predictor variable xt 1 and responded variable xt .
On LSE method we minimize the sum of squares of the difference xt  xt 1 that is
S     z t2    xt   xt 1 
2
Consider the minimization of S   with respect to  we have

d S  
 2 xt   xt 1  xt 1 ...............* *
d  

Setting this equal to zero and solving for  yields  2 xt  ˆ xt 1 xt 1  0  ˆ 
x
x
t 1
xt
2
t 1
.
Example: 5.3.2
Consider an AR 1 model: xt     ( xt 1   )  zt .
Find the estimates  and  using the LSE method.
Exercise: 5.3.1
Consider an AR2 model: xt  x  1 ( xt 1  x )  2 ( xt 2  x )  zt Find the estimates 1 and  2 using
the LSE method .
5.4 Confidence interval for mean
Suppose that xt    z t where  is constant, zt ~ N 0,  0  and z t is a stationary process. Under
 
these assumptions, xt ~ N  ,  0  and xt is stationary , therefore X ~ N  , 0
 n
n 1

 k  
1

2
1    k  


n  
k 1 

If  0 and the  k ' s are known then a 1001    percent confidence interval for  is
X  z
2
0 

 k 
1  2 1    k  . Where z  is upper quantile from the standard normal distribution.
n 
2
 n 
2
Note that if  k  0 for all k ,then this confidence interval formula reduces to X  z 
2
40 | P a g e
0
n
.
TIME SERIES ANALYSIS LECTURE NOTES
2019
Example 5.4.1
Suppose that in a sample of size 100 from AR 1 : xt  1 xt 1  z t process with mean  ;   0.6 and
 2  2 we obtain X 100  0.271
(a) Construct an approximate 95% confidence interval for  .
(b) Are the data computable with the hypothesis that   0 ?
Exercise 5.4.1
Suppose that in a sample of size 100 from MA1 process with mean  ;   0.6 and  2  1 we obtain
X 100  0.157 .
a) Construct an approximate 95% confidence interval for  .
b) Are the data computable with the hypothesis that   0 ?
CHAPTER 6
MODEL DIAGNOSTICS
6.1 Introduction
Model diagnostics is primarily concerned with testing the goodness of fit of a tentative model. Two
complementary approaches are: Analysis of residuals from fitted models and analysis of over
parameterized model will be considered in this chapter.
6.1.1 Residual Analysis
Before a model can be used for inference the assumptions of the model should be assessed using
residuals. Recall from regression analysis, residuals are given by :
Residual=Actual Value-Predicted Value.
Residual can be used to assess if the ARMA is adequate and if the parameter estimates are close to
the true values. Model adequacy is checked by assessing whether the model assumptions are
satisfied.
The basic assumption is that , z t are white noise .That is they possess the properties of
independence, identically and normally distributed random variables with zero mean and constant
variance  z2 .
A good model is one with residuals that satisfy these properties, that is, it should have residuals
which are:
41 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
2019
1) Independent (uncorrelated errors),
2) Normally distributed and
3) Constant variance.
6.1.2 Test of independence
A test of independence can be performed by:
-Examining ACF: Compute the sample ACF of the residual. Residuals are independent if they do not
form any pattern and are statistically insignificant, that is, they are within Z  standard deviation.
2
6.1.3 Test for normality
Test of normality can be performed by:
-Constructing Histogram: Gross normality can be assessed by plotting histogram of residuals.
Histogram of normally distributed residuals should approximately be symmetric and bell shaped.
6.1.4 Test of constant variance
Test of constant variance can be inspected by plotting the residuals over time. If the model is
adequate we expect the plot to suggest a rectangular scatter around zero horizontal level with no
trends whatsoever.
6.2 Autocorrelation of residuals
The basic idea behind the ARIMA modeling is to account for any autocorrelation pattern in the series
xt  with a parsimonious combination of AR and MA terms, leaving random terms zt as a white
noise. If the residuals are white noise this implies that they are correlated, that is, they are serial
independent. To determine if the residuals ACF are significantly different from zero we use the
following Portmanteau test.
6.2 1 Test for combined residual ACF: Portmanteau test
This test uses the magnitude of residual autocorrelations as a group to check for model adequacy.
The test is as follows:
Hypothesis
H 0 : 1   2  ...   k  0 mod el is correct 
H 1 :  k  0 for atleast one k  1,2,..., K mod el is incorrect
Here we use the modified Box-Pierce Statistic ( Q).
42 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
2019
K
Test statistic: Q  N  ̂ k2
k 1
where N is the number of terms in the differenced series.
K is the number of lags we wish to use in the test.
k denote the autocorrelation coefficient at lag k.
Decision rule: Reject H 0 if Q   k2,1 and conclude that the random term z t  from the estimated
model are correlated and that the estimated model may be inadequate.
Remark: If a correct ARMA p, q  model is fitted, then Q should be approximately distributed as  2
with k  p  q degree of freedom. Where p, q are numbers of AR and MA terms, respectively in the
model.
Note: The maximum lag K is taken large enough so that the weights are negligible for j  K .
Alternative test: Ljung-Box test, Q *  N N  2 ˆ k2 / N  k 
K
k 1
Example: 6.2.1
From example 5.2.1a, calculate the value of Q if k  1 .
Example: 6.2.2
The AR2 model xt  C  1 xt 1   2 xt 2  zt was fitted to a data set of length 121.
24
a) The Box-Pierce statistic value was Q  121 ˆ k2  31.5
k 1
At 95% level of significance, test the hypothesis that AR2 model fit the data.
b) The parameter estimates are ˆ1  1.5630 , ˆ 2  0.6583 and Cˆ  0.4843
The last four values in the series were 7.07, 6.90, 6.63, 6.20 .
Use AR2 model to predict the next observation value.
6.3 Over fitting
Another basic diagnostic tool is that of over fitting .In this diagnostic check we add another coefficient
to see if the model is better. Recall that any ARMA p, q  model can be considered as a special case
of a more general ARMA model with the additional parameters equal to zero. Thus, after specifying
43 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
2019
and fitting our tentative model, we fit a more general model that contains the original model as a
special case.
There is no unique way to over fit a model, but one should be careful not to add coefficients to both
sides of the model. Over fitting both AR and MA terms at the same time leads to estimation problems
because of the parameter redundancy as well as violating the principle of parsimony.
If our tentative model is AR2 we might over fit with AR 3 . The original AR2 will be confirmed if :
1) The estimates of the additional parameter 3 is not significantly different from zero.
2) The estimates for the parameters 1 and  2 do not change significantly from the original
states.
CHAPTER 7
FORECASTING
7.1 Minimum Mean Square Error
The minimum mean square error (MMSE) forecasts xˆ n l  of z n l at the forecast origin n is given by
the conditional expectation: xˆ n l   E xn l z n , z n1 ,... .
Example: 7.1.1

Let  B    B Wt   B 1  B   1   1 B  ...   p  q B p  d
d

7.1.1 ARMA model forecast equation in infinite MA form
The ARMA model for xt is xt   B zt  xt   1 zt 1   2 zt 2  ...  zt
Assume that we have x1 , x2 ,..., xn and we want to forecast xnl (i.e l step-ahead forecast from origin,
the actual value is: xnl   1 z nl 1   2 z nl 2  ...  z nl .
The “minimum mean square error” forecast for xnl is xˆ n l    1 z nl 1    2 z nl 2   ...
This form is not very useful for computing forecasts, but is useful in finding the forecast error.
7.2 Computation of Forecasts
7.2.1 Forecast error and forecast error variances
(a) One step –ahead ( l  1)
en 1  xn1  xˆ n 1  z n1
44 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
var en 1  var z n1    z
2019
2
(b) Two steps –ahead ( l  2 )
en 2  xn2  xˆ n 2  z n2   1 z n1

var en 2  var z n  2    1 var z n1    z   1 z z   z 1   1
2
2
2
2
2
2

(c) Three steps-ahead ( l  3 )
en 3  xn3  xˆ n 3  z n3   1 z n 2   2 z n1

var en 3  var z n3    1 var z n 2    2 var z n1    z   1  z   2  z   z 1   1   2
2
2
2
2
2
2
2
2
2
(d) In general, l steps-ahead
en l   xnl  xˆ n l   z nl   1 z nl 1  ...   l 1 z n1


var en l    z 1   1   2  ...   l 1   z
2
2
2
2
2
l 1

i 0
2
i
where  0  1 .
To forecast l steps-ahead from origin n :
The actual value is xnl   1 z nl 1   2 z nl 2  ...  z nl
The minimum mean square error forecast for xnl is xˆ n l    1 z nl 1    2 z nl 2   ...
7.3 Prediction Interval
A 95% prediction interval for :
a) l step ahead forecast is xˆ n l   Z 0.025  var en l  .
b) One step ahead is xˆ n 1  Z 0.025  var en 1 .
c) Two steps ahead is xˆ n 2  Z 0.025  var en 2 .
d) Three steps ahead is xˆ n 3  Z 0.025  var en 3 .
7.4 Forecasting
AR  p 
and
Example: 7.4.1
For each of the following models,
45 | P a g e
MAq 
models.
2

TIME SERIES ANALYSIS LECTURE NOTES
AR 1 process: xt     xt 1     zt .
MA1 process: xt  zt   zt 1 .
Find: a) xˆ n 1
b) xˆ n 2
c) xˆ n 1
Exercise: 7.4.1
For each of the following models,
AR2 process: xt  1 xt 1  2 xt 2  zt .
MA1 process: xt  zt  1 zt 1   2 zt 2 .
Find: a) xˆ n 1
b) xˆ n 2
7.5 Forecasting
c) xˆ n 1
ARMA p, q 
models
Example: 7.5.1
For 1   B xt     1   B  zt
a) find the first, 2nd and l -step ahead forecast.
b) Calculate:
i)   weight ii) forecast error variance
iii) 95% forecast limits for xˆ n l .
Exercise: 7.5.1
Consider the model IMA 1,1 : 1  B  xt  1   B  z t .
Calculate:
a) the l  step ahead forecast xˆ n l  of xnl .
b)   weight of xt .
c) 95% forecast limits for xˆ n l , xˆ n 1 and
7.6 Forecasting
ARIMA p, d , q 
xˆ n 2 .
models
Consider the ARIMA p, d , q  model at time t  n  l ,  p B 1  B  xt   q B  z t
d
46 | P a g e
2019
TIME SERIES ANALYSIS LECTURE NOTES
2019
APPENDIX A
Table 7A: The chi-squared distribution (χ²), right tail probability.
The table gives the values of  2 ,v where P  2   2 ,v   with v degree of freedom.


0.10
0.05
0.025
0.01
0.005
1
2.706
3.841
5.024
6.635
7.879
2
4.605
5.991
7.378
9.210
10.597
3
6.251
7.815
9.348
11.345
12.838
4
7.779
9.488
11.143
13.277
14.860
5
9.236
11.070
12.833
15.086
16.750
6
10.645
12.592
14.449
16.812
18.548
7
12.017
14.067
16.013
18.475
20.278
8
13.362
15.507
17.535
20.090
21.955
9
14.684
16.919
19.023
21.666
23.589
10
15.987
18.307
20.483
23.209
25.188
11
17.275
19.675
21.920
24.725
26.757
12
18.549
21.026
23.337
26.217
28.300
13
19.812
22.362
24.736
27.688
29.819
14
21.064
23.685
26.119
29.141
31.319
15
22.307
24.996
27.488
30.578
32.801
16
23.542
26.296
28.845
32.000
34.267
17
24.769
27.587
30.191
33.409
35.718
18
25.989
28.869
31.526
34.805
37.156
19
27.204
30.144
32.852
36.191
38.582
20
28.412
31.410
34.170
37.566
39.997
21
29.615
32.671
35.479
38.932
41.401
22
30.813
33.924
36.781
40.289
42.796

47 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
2019
23
32.007
35.172
38.076
41.638
44.181
24
33.196
36.415
39.364
42.980
45.559
25
34.382
37.652
40.646
44.314
46.928
26
35.563
38.885
41.923
45.642
48.290
27
36.741
40.113
43.195
46.963
49.645
28
37.916
41.337
44.461
48.278
50.993
29
39.087
42.557
45.722
49.588
52.336
30
40.256
43.773
46.979
50.892
53.672
40
51.805
55.758
59.342
63.691
66.766
50
63.167
67.505
71.420
76.154
79.490
60
74.397
79.082
83.298
88.379
91.952
70
85.527
90.531
95.023
100.425
104.215
80
96.578
101.879
106.629
112.329
116.321
90
107.565
113.145
118.136
124.116
128.299
100
118.498
124.342
129.561
135.807
140.169
110
129.385
135.480
140.917
147.414
151.948
120
140.233
146.567
152.211
158.950
163.648
48 | P a g e
TIME SERIES ANALYSIS LECTURE NOTES
2019
Table 8A gives the values of t , where P  t  t ,    with  degree of freedom.
Table 8A: The student’s t-distribution


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
40
60
120

0.1
0.05
0.025
0.01
0.005
0.001
0.0005
3.078
1.886
1.638
1.533
1.476
1.440
1.415
1.397
1.383
1.372
1.363
1.356
1.350
1.345
1.341
1.337
1.333
1.330
1.328
1.325
1.323
1.321
1.319
1.318
1.316
1.315
1.314
1.313
1.311
1.310
1.303
1.296
1.289
1.282
6.314
2.920
2.353
2.132
2.015
1.943
1.895
1.860
1.833
1.812
1.796
1.782
1.771
1.761
1.753
1.746
1.740
1.734
1.729
1.725
1.721
1.717
1.714
1.711
1.708
1.706
1.703
1.701
1.699
1.697
1.684
1.671
1.658
1.645
12.076
4.303
3.182
2.776
2.571
2.447
2.365
2.306
2.262
2.228
2.201
2.179
2.160
2.145
2.131
2.120
2.110
2.101
2.093
2.086
2.080
2.074
2.069
2.064
2.060
2.056
2.052
2.048
2.045
2.042
2.021
2.000
1.980
1.960
31.821
6.965
4.541
3.747
3.365
3.143
2.998
2.896
2.821
2.764
2.718
2.681
2.650
2.624
2.602
2.583
2.567
2.552
2.539
2.528
2.518
2.508
2.500
2.492
2.485
2.479
2.473
2.467
2.462
2.457
2.423
2.390
2.358
2.326
63.657
9.925
5.841
4.604
4.032
3.707
3.499
3.355
3.250
3.169
3.106
3.055
3.012
2.977
2.947
2.921
2.898
2.878
2.861
2.845
2.831
2.819
2.807
2.797
2.787
2.779
2.771
2.763
2.756
2.750
2.704
2.660
2.617
2.576
318.310
22.326
10.213
7.173
5.893
5.208
4.785
4.501
4.297
4.144
4.025
3.930
3.852
3.787
3.733
3.686
3.646
3.610
3.579
3.552
3.527
3.505
3.485
3.467
3.450
3.435
3.421
3.408
3.396
3.385
3.307
3.232
3.160
3.090
636.620
31.598
12.924
8.610
6.869
5.959
5.408
5.041
4.781
4.587
4.437
4.318
4.221
4.140
4.073
4.015
3.965
3.922
3.883
3.850
3.819
3.792
3.767
3.745
3.725
3.707
3.690
3.674
3.659
3.646
3.551
3.460
3.373
3.291
________________________________________________________________________________________
49 | P a g e