Time Series Analysis and Classification
Tarek Medkour
January 31, 2024
Part 1: Time Series Analysis
Introduction
Time Series Models
Spectral Analysis
Time-Frequency Representation
Multivariate Time Series
Part 2: Time Series Classification
Pattern Recognition and Detection
Feature Extraction and Selection
Models and Representation Learning
Data Enhancement and Preprocessings
Change-Point and Anomaly Detection
Introduction and Examples
0.4
●
●
●
0.2
●
0.0
●
●
●
●
●
●
●●●●
●
●
●
●●
−0.2
●
●
● ●
●
● ● ●
●
●
●
●
●
●
●
●●
●●●
●●
●
●
●
● ●
● ● ●
1860
1880
●
●●
●
●
●●
●
●●
●
● ●●●
●
1900
●
●
●
●
●● ●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●● ●
●
●
● ●
●
●
●
●
●●●●
−0.4
Global temperature deviations
●
● ●●
●
●
●
1920
Year
1940
1960
1980
2000
Introduction and Examples
1700
●
●●
●
●
1600
●
●
●
●
●
●
● ●
●
1500
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
● ●
●
●
●
●
● ●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●● ●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
● ●
●●
●
● ●
● ●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
1300
Amount of milk produced
●●
1400
●
●●
●
●
●
●
●
●
●
●
1994
1996
1998
2000
Year
2002
2004
2006
Introduction and Examples
●
●
●
●
200
190
180
170
CREF stock values
210
220
●●
●
●●
●
●●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●● ●
●
● ●●
●●
●●●● ●
●
●
●
●
● ●●
●
●
●
●●●●
●
●● ●
●
●
●
●
●
●● ● ●
●●●●
● ●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●●
●
●
● ●●
●●
●
● ●
●●
●●
●
●● ●●
●
●●
●
●●
●● ●●
●●
●
●●
●●
●●
●
●
●
●
●●
●
● ●
●
●
●●
●
●●
●
●
●
●
●
●●● ●
●
●
●
●● ●
●●
●
●
●●
●
●
●
● ●
●
●
●●●●
●
●
●
●
●
●
●
● ●
● ●
●
●
●●
●
●
● ●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●●●
●●
●●
●
●
●● ●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●●
●●
●
●
● ●
●
●
●
●
●
●
●● ●
●
●●
● ●●
●
●
● ● ●●
●●
●
●
●
●●●
●
●
●●
●
●●
●●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
● ●
●●●
●
●
● ●● ●
●
●●
●
●
●●
●
●●
●
●
●●
●
●
●●
●●
●
●
●
●●
●●
●●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●●
●●●
●● ●
●
● ●
●
●
●●●
●
●
●
● ●●
●
● ●
● ●
●
●
●
●
0
100
200
300
Time
400
500
Introduction and Examples
●
●
200
●
●
●
●
●
●
●
150
●
●
●
●
●
●
●●
●
100
●●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
● ●
●
● ●
●
●
●
●
●●
●
●
●
50
Number of Home Runs
●
●●
●
●
●
●
●
●
●
●
● ●
● ●
● ●●
●
●● ● ●
●●●
1920
1940
1960
Time
1980
●
●
●
●
●
●
●
●●
●
2000
●
●
Introduction and Examples
40
●
●
●
●
●
● ●
30
●
●
●
●
●
● ●
25
●
●
●
●
●
●
20
●
●
● ●
● ●
●●
●
●
●
●
●
●●
●
●
●
●●
●●
●
● ● ●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
15
●
●
●
●
●
●●
10
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
5
Number of earthquakes (7.0 or greater)
35
●
1900
1920
1940
1960
Time
1980
2000
Introduction and Examples
●
●
●
●●
25000
●
●
●
●●
●
●
●
●
●
●
●●
●●●
●●
●
●
●
●
●
20000
●
●
15000
●
●
●
10000
●
●
●
●
●
●
5000
USC Columbia fall enrollment
●
●●●
●
●
●
●
●●
●
●●●●
1900
1910
1920
1930
Time
1940
1950
●●●
30
35
Introduction and Examples
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●●
25
20
Star brightness
15
10
5
●
●
●
●
●
●●
●
●
●● ●
●●
●
● ●
●● ●
●
●
●●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
● ●●
●● ●● ●
●
●●
●
●
●
●
● ●
● ●●
●
●●
●
●●
● ●●
●● ●
●
●
●
●
●
●
● ●●
●
● ●
●●
●●
●
● ●●
● ●
●●
●
● ●
●
● ●
●●
●
●
●
●
●
● ●●
●
●
● ●●
●
●
●
●
● ●
●●
●● ●
● ●●
●
●
●
● ●
●
●●
●
● ●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
● ● ●● ●
●
●
● ●
●●
●●
●
●● ●
● ● ●
●
●
●●
●
● ●
● ●●
●
●
● ●●
●
● ●
● ●
●● ●
●●
●● ●
●
●
● ● ●
●
●
● ● ●
● ● ●
●
●
● ● ●
●
●
●
●
● ●●
● ●
● ●
●
● ●
●
●● ●
● ● ●● ●
●
●● ●
●
●●
●
● ●●
● ●●
●
●
●●
●
●
●
● ●
● ●●
●●
● ●
●
●
●
●
●
●
●
●
● ●● ●
● ● ●●
●
●
●●
●
●
●
●● ●
●
●
●
●
●
●●
●●
● ●
●● ●
●
●● ●
●
●
●
● ●
●
●●
●● ●
●
● ● ●●
●
●●
●
●
●● ●
●
●
●
●
● ●
●
●
●
● ●● ●●
●● ●
●●
●
●● ●●
●
●
●
●
●
●
●
●●
● ●
●
●
●●
●●
●
● ●● ●●
●● ●
●
●
●
●
●
●
●
●
●
●
●●
● ●●
●●
● ●
●
●
● ●
●●
●
●
●●
●● ●
●●
●
●● ●
●
●
●
●
●
●●
●
●● ●
●●
●
●
●
● ●●
●● ●
●
●
●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
●
●
●●
0
●
●
●●
●
●
0
100
200
300
Time
400
500
600
Introduction and Examples
Introduction and Examples
1500
●
●
●
●
●●
●
●
● ●●●
● ●
●●
1450
●
●
1350
1400
●●
●
●●
●
●
● ●
●
●
●
●
●
●
● ●
●
●
●
● ●●
1300
●
●
●●
●
●
1250
SP500 Index
●
1900
●
●
●
●
●●
●
●
●● ●
●
● ●
●
●
●● ● ●
●
●
●
●
●● ●
●
●
●
● ●
●
●
●●
●
●
● ●
●●
● ●●●●●
● ●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●●
●
●
●
●●
●
● ●●
●
●
●●
●
●
●●
●
●
●
●● ●
●
● ●
●
● ●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●●
● ●
●●
●
●●
●
●
●
●
●
●
●
1950
●
●●
●
●●
●
●●
●
●
●
● ●
●
● ●
●
● ●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●● ●
●●
●
●
●
●
●
●
2000
2050
Year
2100
2150
Introduction and Examples
●
●●
60
●
●●
●
●
●
●●
●
●
●
●
40
●●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●●
●●
●
●
●
●
●
●
●
●●
●●
●● ●
●●
●●● ●●
●
●
●● ● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
30
●
●
●●
●
●
●●
● ●●
●
●
●
● ● ●
●● ●
●●
●
●
●
●●●
●●
● ●
●●
●
●
● ●●●●
●
●
●
● ● ●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●●
● ●
●●
●
●
●
●
●
●● ● ●
● ●
●●
●
●●●
●
●●
●●
● ●●
●●
● ● ●
●
●
●
● ●●
●
● ●●
● ●
●●
●
●
●● ●
●
●
●● ●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●●
●
●● ●
●
●
●●
●
●
●
●
●
●
●
20
●
10
Oil prices
50
●
● ●
1990
1995
Year
2000
2005
3000000
Introduction and Examples
●
●
●
●
●
2500000
●
●
●
2000000
●
●
●
●
1500000
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
1000000
Algerian Wheat Production
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
1960
●
1970
1980
1990
Date
●
●
●
2000
2010
Introduction and Examples
Terminology
A time series is a sequence of ordered data. The ordering refers
generally to time, but other orderings could be envisioned (e.g.,
over space, etc.). In this class, we will be concerned exclusively
with time series that are
measured on a single continuous random variable Y
equally spaced in discrete time; that is, we will have a
single realization of Y at each second, hour, day, month,
year, etc.
Introduction and Examples
Ubiquity
Time series data arise in a variety of fields. Here are just a few
examples.
In business, we observe daily stock prices, weekly interest
rates, quarterly sales, monthly supply figures, annual
earnings, etc.
In agriculture, we observe annual yields (e.g., crop
production), daily crop prices, annual herd sizes, etc.
In engineering, we observe electric signals, voltage
measurements, etc.
In natural sciences, we observe chemical yields, turbulence
in ocean waves, earth tectonic plate positions, etc.
Introduction and Examples
In medicine, we observe ECG or EEG measurements on
patients, drug concentrations, blood pressure readings, etc.
In epidemiology, we observe the number of flu cases per
day, the number of health-care clinic visits per week,
annual tuberculosis counts, etc.
In meteorology, we observe daily high temperatures, annual
rainfall, hourly wind speeds, earthquake frequency, etc.
In social sciences, we observe annual birth and death rates,
accident frequencies, crime rates, school enrollments, etc.
Introduction and Examples
Justification
The purpose of time series analysis is twofold:
1
to model the stochastic (random) mechanism that gives
rise to the series of data
2
to predict (forecast) the future values of the series based on
the previous history
Introduction and Examples
Notes
The analysis of time series data calls for a new way of thinking
when compared to other statistical methods courses.
Essentially, we get to see only a single measurement from a
population (at time t) instead of a sample of measurements at a
fixed point in time (cross-sectional data).
Introduction and Examples
The special feature of time series data is that they are not
independent! Instead, observations are correlated through
time.
Correlated data are generally more difficult to analyze.
Statistical theory in the absence of independence becomes
markedly more difficult.
Most classical statistical methods (e.g., regression, analysis
of variance, etc.) assume that observations are statistically
independent.
Introduction and Examples
For example, in the simple linear regression model
Yi = β0 + β1 xi + ϵi ,
we typically assume that the ϵ error terms are independent and
identically distributed (iid) normal random variables with mean
0 and constant variance.
Introduction and Examples
There can be additional trends or seasonal variation
patterns (seasonality) that may be difficult to identify and
model.
The data may be highly non-normal in appearance and be
possibly contaminated by outliers
Introduction and Examples
Modeling I
Our goal in this part of the course is to build (and use)
uni-variate time series models for data. This breaks down into
different parts.
1 Model specification (identification)
Consider different classes of time series models for
stationary processes.
Use descriptive statistics, graphical displays, subject matter
knowledge, etc. to make sensible candidate selections.
Abide by the Principle of Parsimony.
2
Model fitting
Introduction and Examples
Modeling II
Once a candidate model is chosen, estimate the parameters
in the model.
We will use least squares and/or maximum likelihood to do
this.
3
Model diagnostics
Use statistical inference and graphical displays to check how
well the model fits the data.
This part of the analysis may suggest the candidate model
is inadequate and may point to more appropriate models.
Time series and stochastic processes
Terminology
The sequence of random variables {Yt : t = 0, 1, 2, . . . }, or
more simply denoted by {Yt }, is called a stochastic process. It
is a collection of random variables indexed by time t; that is,
Y0 = value of the process at time t = 0
Y1 = value of the process at time t = 1
Y2 = value of the process at time t = 2
..
.
Yn = value of the process at time t = n.
The subscripts are important because they refer to which time
period the value of Y is being measured.
Time series and stochastic processes
A stochastic process can be described as a statistical
phenomenon that evolves through time according to a set of
probabilistic laws.
A complete probabilistic time series model for {Yt } would
specify all of the joint distributions of random vectors
Y = {Y1 , Y2 , . . . , Yn }, for all n = 1, 2, . . . , :
P(Y1 ≤ y1 , Y2 ≤ y2 , . . . , Yn ≤ yn ),
for all y = (y1 , y2 , . . . , yn ) and n = 1, 2, . . . .
Time series and stochastic processes
This specification is not generally needed in practice. In
this course, we specify only the first and second-order
moments; i.e., expectations of the form E(Yt ) and
E(Yt Yt−k ), for k = 0, 1, 2, . . . , and t = 0, 1, 2, . . . .
Much of the important information in most time series
processes is captured in these first and second moments
(or, equivalently, in the means, variances, and covariances).
Means, Variances, and Covariances
Terminology
For the stochastic process {Yt : t = 0, 1, 2, . . . ,}, the mean
function is defined as
µt = E(Yt ),
for t = 0, 1, 2, . . . ,. That is, µt is the theoretical (or
population) mean for the series at time t.
Means, Variances, and Covariances
The autocovariance function is defined as
λt,s = cov(Yt , Ys ) = E(Yt Ys ) − E(Yt )E(Ys ),
for t, s = 0, 1, 2, . . .. The autocorrelation function is given by
p
p
ρt,s = corr(Yt , Ys ) = cov(Yt , Ys )/ var(Yt )var(Ys ) = λt,s / λt,t λs,s .
Means, Variances, and Covariances
Values of ρt,s near ± 1 ⇒ strong linear dependence
between Yt and Ys .
Values of ρt,s near 0 ⇒ weak linear dependence between Yt
and Ys .
Values of ρt,s = 0 ⇒ Yt and Ys are uncorrelated.
Some examples of Stochastic Processes
White Noise
Example 1
A stochastic process {et : t = 0, 1, 2, . . . ,} is called a white
noise process if it is a sequence of independent and identically
distributed (iid) random variables with
E(et ) = µe , var(et ) = σe2 .
Some examples of Stochastic Processes
3
White Noise
●
●
●
●
●
2
●
●
●
●
●
●
●
1
●
●●
●
●
●
●
●
●
●
●
−1
●
● ●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
0
●
●
●
●
●
●
●
●
●
●
●
●
●
−2
Simulated white noise process
●
●
●●
●
0
50
100
Time
●
150
Some examples of Stochastic Processes
White Noise
Both µe and σe2 are constant (free of t). It is often assumed that
µe = 0; that is, {et } is a zero mean process. A slightly less
restrictive definition would require that the et are uncorrelated
(not independent). However, under normality; i.e., et˜
iid N(0, σe2 ), this distinction becomes irrelevant (for linear time
series models).
Some examples of Stochastic Processes
White Noise
Auto-covariance Function
For t = s,
cov(et , es ) = cov(et , et ) = var(et ) = σe2 .
For t ̸= s,
cov(et , es ) = 0,
because the et ’s are independent. Thus, the autocovariance
function of et is
(
σe2 , for |t − s| = 0
λt,s =
0, for |t − s| =
̸ 0
Some examples of Stochastic Processes
White Noise
Autocorrelation Function
For t = s,
ρt,s = corr(et , es ) = corr(et , et ) = λt,t /
p
λt,t λt,t = 1.
for t ̸= s,
p
ρt,s = corr(et , es ) = λt,s / λt,t λs,s = 0.
Thus, the autocorrelation function is
(
1, for |t − s| = 0
ρt,s =
0, for |t − s| =
̸ 0
Some examples of Stochastic Processes
White Noise
Remark
A white noise process, by itself, is rather uninteresting for
modeling real data. However, white noise processes still play a
crucial role in the analysis of time series data!
Some examples of Stochastic Processes
White Noise
Time series processes Yt generally contain two different types of
variation:
systematic variation (that we would like to capture and
model; e.g., trends, seasonal components, etc.)
random variation (that is just inherent background noise in
the process).
Some examples of Stochastic Processes
White Noise
Our goal as data analysts is to extract the systematic part of
the variation in the data (and incorporate this into our model).
If we do an adequate job of extracting the systematic part, then
the only part left over should be random variation, which can
be modeled as white noise.
Some examples of Stochastic Processes
Random Walk
Example 2
Suppose that et is a zero mean white noise process with
var(et ) = σe2 . Define
Y1 = e1
Y2 = e1 + e2
···
Yn = e1 + e2 + . . . + en .
Some examples of Stochastic Processes
Random Walk
By this definition, note that we can write, for t > 1,
Yt = Yt−1 + et ,
where E(et ) = 0 and var(et ) = σe2 .
The process Yt is called a random walk process.
Random walk processes are used to model stock prices,
movements of molecules in gases and liquids, animal locations,
etc.
Some examples of Stochastic Processes
Random Walk
●
●●
●
0
●●●
● ●
●
●
●
●
●●
● ●
●
●
●
●●
●●
●
●
●
●●
● ●
●
●
●
●
●
●
●
●
●●●
●●●
● ●
●●
●
●
●
● ●
●●
●
● ●
●
● ●
●
●
●
●
●●
● ●
●● ●
●● ●
●●
●
●
●
●
●
●●
●
●
●●●
●●●
●
●●●●
●
●
●
●
●●●
●
●
●●
●●
−5
●
●
●
●●●
● ●
●
●
●
●
●
●
●● ●
●
●
●●●●
●
●
●
●
−10
Simulated random walk process
5
●
●
●● ●
●
●
●
●
●
●
●
0
50
100
Time
150
Some examples of Stochastic Processes
Random Walk
Mean Function
The mean of Yt is
µt = E(Yt )
= E(e1 + e2 + . . . + et )
= E(e1 ) + E(e2 ) + . . . + E(et ) = 0.
That is, Yt is a zero mean process.
Some examples of Stochastic Processes
Random Walk
Variance Function
The variance of Yt is
var(Yt ) = var(e1 + e2 + . . . + et )
= var(e1 ) + var(e2 ) + . . . + var(et ) = tσe2 ,
because var(e1 ) = var(e2 ) = . . .= var(et ) =σe2 , and cov(et , es ) =
0 for all t ̸= s.
Some examples of Stochastic Processes
Random Walk
Autocovariance Function
For t ≤ s, the autocovariance of Yt and Ys is
λt,s = cov(Yt , Ys ) = cov(e1 + . . . + et , e1 + . . . + et + et+1 + . . . + es )
= cov(e1 + e2 + . . . + et , e1 + e2 + . . . + et )
+̇cov(e1 + e2 + . . . + et , et+1 + . . . + es )
Xt
XX
cov(ei , ej )
=
cov(ei , ei ) +
i=1
1≤i̸=j≤t
Xt
=
var(ei ) = σe2 + σe2 + . . . + σe2 = σe2 .
i=1
Because λt,s = λs,t , the autocovariance function for a random
walk process is
λt,s = tσe2 , for 1 ≤ t ≤ s.
Some examples of Stochastic Processes
Random Walk
Autocorrelation Function
For 1 ≤ t ≤ s, the autocorrelation function for a random walk
process is
r
λt,s
t
tσe2
.
= 2 2 =
ρt,s = corr(Yt , Ys ) = p
tσe sσe
s
λt,t λs, s
Note that
when t is closer to s, the autocorrelation ρt,s is closer to 1.
That is, two observations Yt and Ys close together in time
are likely to be close together, especially when t and s are
both large (later on in the series).
On the other hand, when t is far away from s (that is, for
two points Yt and Ys far apart in time), the autocorrelation
is closer to 0.
Some examples of Stochastic Processes
Random Walk
Example 3
Suppose that et is a zero mean white noise process with var(et )
= σe2 . Define
1
Yt = (et + et−1 + et−2 ),
3
that is, Yt is a running (or moving) average of the white noise
process (averaged across the most recent 3 time periods).
Some examples of Stochastic Processes
Random Walk
1.5
●
●
●
1.0
●
●
●
●
0.5
● ●
●
●
●
●
●
●●
●
●
●
●
●
0.0
●
●
●
●
●
●● ●
●
●
●
●●
●
● ●
●
●
●
●
●
●
−0.5
●
●
●
●
●●
●
●● ●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
0
●
●● ●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
−1.0
Simulated moving average process
●
●
50
100
Time
150
Some examples of Stochastic Processes
Random Walk
Mean Function
The mean of Yt is
1
µt = E(Yt ) = E[ (et + et−1 + et−2 )]
3
1
= [E(et ) + E(et−1 ) + E(et−2 )] = 0,
3
because et is a zero-mean process. Yt is a zero mean process.
Some examples of Stochastic Processes
Random Walk
Variance Function
The variance of Yt is
1
var(Yt ) = var[ (et + et−1 + et−2 )]
3
1
= var(et + et−1 + et−2 )
9
1
= [var(et ) + var(et−1 ) + var(et−2 )]
9
1
1
= (3var(et )) = σe2 ,
9
3
because var(et ) = σe2 for all t and because et , et−1 , and et−2 are
independent (all covariance terms are zero).
Some examples of Stochastic Processes
Random Walk
Autocovariance Funcion
We need to consider different cases.
Case 1: If s = t, then
1
λt,s = λt,t = cov(Yt , Yt ) = var(Yt ) = σe2 .
3
Case 2: If s = t + 1, then
λt,s = λt,t+1 = cov(Yt , Yt+1 )
1
1
= cov[ (et + et−1 + et−2 ), (et+1 + et + et−1 )]
3
3
1
= [cov(et , et ) + cov(et−1 , et−1 )]
9
1
2
= [var(et ) + var(et−1 )] = σe2 .
9
9
Some examples of Stochastic Processes
Random Walk
Case 3: If s = t + 2, then
λt,s = λt,t+2 = cov(Yt , Yt+2 )
1
1
= cov[ (et + et−1 + et−2 ), (et+2 + et+1 + et )]
3
3
1
= cov(et , et )
9
1
1
= var(et ) = σe2 .
9
9
Case 4: If s > t + 2, then λt,s = 0 because Yt and Ys will have
no common white noise error terms. Because λt,s = λs,t , the
autocovariance function can be written as
 2
σe /3, |t − s| = 0




2σ 2 /9, |t − s| = 1
e
λt,s =

σe2 /9, |t − s| = 2




0, |t − s| > 2.
Some examples of Stochastic Processes
Random Walk
Autocorrelation Function
Recall that the autocorrelation function is
ρt,s = corr(Yt , Ys ) =
λt,s
.
λt,t λs,s
Because λt,t = λs,s = σe2 /3, the autocorrelation function for this
process is

1, |t − s| = 0



2/3, |t − s| = 1
ρt,s =

1/3, |t − s| = 2



0, |t − s| > 2.
Some examples of Stochastic Processes
Random Walk
Notes
Observations Yt and Ys that are 1 unit apart in time have
the same autocorrelation regardless of the values of t and s.
Observations Yt and Ys that are 2 units apart in time have
the same autocorrelation regardless of the values of t and s.
Observations Yt and Ys that are more than 2 units apart
in time are uncorrelated
Some examples of Stochastic Processes
Autoregression
Example 4
Suppose that et is a zero mean white noise process with
var(et ) = σe2 . Consider the stochastic process defined by
Yt = 0.75Yt−1 + et ,
that is, Yt is directly related to the (downweighted) previous
value of the process Yt−1 and the random error et (a shock or
innovation that occurs at time t).
This is called an autoregressive model. Autoregression
means regression on itself. Essentially, we can envision
regressing Yt on Yt−1 .
Note: We will postpone mean, variance, autocovariance, and
autocorrelation calculations for this process until later when we
discuss autoregressive models in more detail.
Some examples of Stochastic Processes
Autoregression
Some examples of Stochastic Processes
Seasonal
Example 5
Many time series exhibit seasonal patterns that correspond to
different weeks, months, years, etc. One way to describe
seasonal patterns is to use models with deterministic parts
which are trigonometric in nature. Suppose that et is a zero
mean white noise process with var(et ) = σe2 . Consider the
process defined by
Yt = a sin(2πωt + ϕ) + et .
Some examples of Stochastic Processes
Seasonal
In this model,a is the amplitude, ω is the frequency of
oscillation, and ϕ controls the phase shift. With a = 2,
ω = 1/52 (one cycle/52 time points), and ϕ = 0.6π, note that
E(Yt ) = 2 sin(2πt/52 + 0.6π),
since E(et ) = 0. Also, var(Yt ) = var(et ) = σe2 .
Some examples of Stochastic Processes
Seasonal
2
●
●
●
●
●
●
●
●●
● ●
●
● ●
●●
●
●
●●●
●● ●●●
●
●
● ●●
●
●
●
●
● ●●
●
● ●
●
● ● ●
●●● ●●
●
●
●
●
● ●
● ●
●● ●
●●
●●
●
●
●
● ●
●●
●
●
●
●
●
●● ●●
● ●
●
●
●
● ●
●
●●●●
●
●●● ●
●
●●
●●●
●
●●●● ●●
●
● ● ●●
●●
●
● ●●
●
●
●
●●
●
●●● ●●
●
●
●
●●
●
●
●
−2
−2
−1
0
0
1
2
●
●
●
●●
●
●●●
●
●●
●
●
0
50
50
100
150
●
●
●
● ●
●
●
●
●
●●
●●
●
●● ● ●
●
●●●
●
●● ● ● ●
● ●●
● ● ●●
●
●
●
●● ● ●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
● ●●●
●● ● ● ● ●
●●
●
●
● ●
●
●● ●
●
●●
● ● ● ●
●
●
●
● ●● ● ● ●
●●
● ●● ● ●
● ●
●
●
●
● ●
●
● ● ●
●
●
● ●
●
●
●
●
●
●
● ●● ●
●
●
● ●
●
●
● ● ●●
●●
●
●
●
●
●
●
50
100
Time
150
●
5
●
0
●
−5
4
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●●●
● ● ●● ●
●
●
●●
●●
● ●●
● ●● ●
● ●
● ●●
●
●
●
●
● ●
● ●●
●
●●●● ●
●
●●●
●● ●
● ● ● ●●● ●
●
●
●●
●
●●
●● ●
●
● ●● ● ●
● ●
●
●
●●
●
●
●● ● ●
●
●
● ●
●
●
●● ●
●
●
● ● ●
● ●●
●
●
● ●
●
●
●
● ●●
●
●
●
●
●● ●
●
●
●●
●
●
●
●
2
0
−2
0
Time
●
−4
150
Time
● ●●
●
●●
●● ●
●
●
●
0
100
●
●●
●
●●
●
● ●
●●
0
●
50
100
Time
150