Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Time Series II – Frequency Domain Methods
John Fricks
Astrostatistics Summer School
Penn State University
University Park, PA 16802
June 8, 2007
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Introduction
Stationarity
The Periodogram and the Spectral Density
Periodogram and Regression
Spectral Density
Smoothing and Tapering
Smoothing
Tapering
Example
Extensions and References
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Introduction
I
In the previous tutorial, we discussed rules for evolution of a
time series in the time domain.
I
In this tutorial, we will discuss time series from a different
perspective; we will look at the frequency components of the
data.
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Stationarity
I
The assumption of stationarity imposes regularity on a time
series model.
I
We will need repeated observations with the same or similar
relationship to one another in order to estimate the underlying
relationships between observations.
I
There are other ways to do this, but stationarity is the most
common and perhaps most basic.
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Strictly Stationary
A time series ..., x−1 , x0 , x1 , x2 , ... is strictly stationary if for a
sequence of times t1 , t2 , ..., tk
{xt1 , ..., xtk }
has the same distributions as
{xt1 +h , ..., xtk +h }
for every integer h. In other words,
P{xt1 ≤ c1 , ..., xtk ≤ ck } = P{xt1 +h ≤ c1 , ..., xtk +h ≤ ck }.
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Weak Stationarity
An important measure of dependency in time series is
autocovariance. This is defined as
γ(t, s) = E (xt − µt )(xs − µs )
where µt = Ext .
The time series xt is weakly stationary if µt is constant and γ(s, t)
depends only on the distance |s − t|.
In the case of Gaussian time series, these two concepts of
stationarity overlap.
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Autocovariance Notation
For a weakly stationary time series, the notation used for
autocovariance uses only lag:
γ(h) = E (xt − µ)(xt−h − µ)
where µ is the constant variance.
We also have a concept of the autocorrelation function which we
saw in the first section in the ACF plot. The autocorrelation
function is defined as
γ(h)
ρ(h) =
γ(0)
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Periodogram and Regression
Spectral Density
Discrete Fourier Transform of the Time Series
What if we are less interested in how our underlying process
evolves in time and are more interested in the variance of the time
series at certain frequencies?
We may attempt to apply a Fourier transform to the data. For our
time series, x1 , ..., xn , the discrete Fourier transform would be
d(ωj ) = n−1/2
n
X
xt exp(−2πitωj )
t=1
where ωj = 0, 1/n, ..., (n − 1)/n.
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Periodogram and Regression
Spectral Density
An Alternate Representation
Note that we can break up d(ωj ) into two parts
d(ωj ) = n−1/2
n
X
xt cos(2πiωj t) − in−1/2
t=1
n
X
xt sin(2πiωj t)
t=1
which we could write as as a cosine component and a sine
component
d(ωj ) = dc (ωj ) − ids (ωj )
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Periodogram and Regression
Spectral Density
We may use an inverse Fourier transform to rewrite the data as
xt
= n−1/2
n
X
d(ωj )e 2πiωj t
j=1
= n−1/2
n
X
d(ωj )e 2πiωj t
j=1
= a0 + n−1/2
m
X
d(ωj )e 2πiωj t + n−1/2
j=1
= a0 +
d(ωj )e 2πiωj t
j=m+1
m
X
2dc (ωj )
j=1
n
X
n−1/2
cos(2πiωj t) +
m
X
2ds (ωj )
j=1
n−1/2
sin(2πiωj t)
where m = b n2 c
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
I
I
I
Periodogram and Regression
Spectral Density
We can think of the Fourier transform as a regression of xt on
sines and cosines.
√
The coefficients of this regression are equal to 2/ n times the
sine part and the cosine part of the Fourier transforms
respectively.
Also note that if our time series is Gaussian, dc (ωj ) and
ds (ωj ) are Gaussian random variables.
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Periodogram and Regression
Spectral Density
The Periodogram
I
The periodogram is defined as
I (ωj ) = |d(ωj )|2 = dc2 (ωj ) + dc2 (ωj )
I
If there is no periodic trend in the data, then Ed(ωj ) = 0, and
the periodogram expresses the variance of xt at frequency ωj .
I
If a periodic trend exists in the data, then Ed(ωj ) will be the
contribution to the periodic trend at the frequency ωj .
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Periodogram and Regression
Spectral Density
The Periodogram
I
What are we trying to estimate with the periodogram?
I
We can use the periodogram to find periodic trends in the
data.
I
Is there information left in the periodogram after the trend is
removed?
I
Assuming that we have a stationary time series, what does the
periodogram estimate?
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Periodogram and Regression
Spectral Density
The Spectral Density
The spectral density is the Fourier transform of the autocovariance
function
h=∞
X
e −2πiωh γ(h)
f (ω) =
h=−∞
for ω ∈ (−0.5, 0.5). Note that this is a population quantity. (i.e.
This is a constant quantity defined by the model.)
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Periodogram and Regression
Spectral Density
Why is the periodogram an estimate for the spectral density? Let
m be the sample mean of our data.
I (ωj ) = |d(ωj )|2 = n−1
n
X
xt e −2πiωj t
xt e −2πiωj t
t=1
t=1
= |d(ωj )|2 = n−1
n
X
n X
n
X
(xt − m)(xs − m)e −2πiωj (t−s)
t=1 s=1
(n−1)
n−|h|
X
X
= n−1
(xt+|h| − m)(xt − m)e −2πiωj (h)
h=−(n−1) t=1
(n−1)
=
X
γ̂(h)e −2πiωj (h) ≈ f (ωj )
h=−(n−1)
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Smoothing
Tapering
I
Is the periodogram a good estimator for the spectral density?
Not really!
I
The periodogram, I (ω1 ), ..., I (ωm ), attempt to estimate
paramters f (ω1 ), ..., f (ωm ). We have nearly the same number
of parameters as we have data.
I
Moreover, the number of parameters grow as a constant
proportion of the data. Therefore, the periodogram is NOT a
consistent estimator of the spectral density.
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Smoothing
Tapering
Moving Average
I
A simple way to improve our estimates is to use a moving
average smoothing technique
m
X
1
fˆ(ωj ) =
I (ωj−k )
2m + 1
I
We can also iterate this procedure of uniform weighting to be
more weight on closer observations.
1
1
1
ût = ut−1 + ut + ut+1
3
3
3
Then, we iterate.
1
1
1
ûˆt = ût−1 + ût + ût+1
3
3
3
Then, substitute to obtain better weights.
k=−m
John Fricks
Time Series II – Frequency Domain Methods
6000
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Smoothing
Tapering
●
5000
●
●
●
4000
● ●
●
●
●
●
●
●
3000
2000
●
1000
●
● ●
●
●
●
●
0
var
●
●
●
●
0.1
0.2
John Fricks
●
● ●
●
●
●
●
0.3
●
● ●
● ●
● ●
●
0.0
●
●
● ●
●
● ● ● ●
●
●
0.4
freq Series II – Frequency Domain Methods
Time
0.5
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Smoothing
Tapering
Smoothing Summary
I
Smoothing decreases variance by averaging over the
periodogram of neighboring frequencies.
I
Smoothing introduces bias because the expectation of
neighboring periodogram values are similar but not identical
to the frequency of interest.
I
Beware of oversmoothing!
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Smoothing
Tapering
Tapering
I
Tapering corrects bias introduced from the finiteness of the
data.
I
The expected value of the periodogram at a certain frequency
is not quite equal to the spectral density.
I
It is affected by the spectral density at neighboring frequency
points.
I
For a spectral density which is more dynamic, more tapering is
required.
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Smoothing
Tapering
Why do we need to taper?
Our theoretical model ..., x−1 , x0 , x1 , ... consists of a doubly infinite
time series. We could think of our data, yt as the following
transformation of the model
yt = ht xt
where ht = 1 for t = 1, ..., n and zero otherwise. This has
repercussions on the expectation of the periodogram of our data.
Z 0.5
E [Iy (ωj )] =
Wn (ωj − ω)fx (ω)dω
−0.5
where Wn (ω) = |Hn (ω)|2 and Hn (ω) is the Fourier transform of
the sequence ht .
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Smoothing
Tapering
The Taper
Specifically,
n
1 X
Hn (ω) = √
ht e −2πiωt
n t=1
When we put in the ht above, we obtain a spectral window of
Wn (ω) =
sin2 (n2πω)
.
sin2 (πω)
We set Wn (0) = n.
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Smoothing
Tapering
There are problems with this spectral window, namely there is too
much weight on neighboring frequencies (sidelobes).
400
200
0
spectrum
Fejer window, n=480
−0.10
−0.05
0.00
0.05
0.10
freq
−20
−60
spectrum
Fejer window (log), n=480
−0.10
−0.05
0.00
0.05
0.10
freq
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Smoothing
Tapering
One way to fix this is to use a Cosine taper. We select a transform
ht to be
2π(t − t
n
ht = 0.5 1 + cos
Fejer window(log), n=480, L=9
0
−20
0
−60
−40
spectrum
30
10
spectrum
50
Fejer window, n=480, L=9
−0.10
0.00
0.05
0.10
−0.10
freq
0.00
0.05
0.10
freq
−5
−15
spectrum
15
10
−25
5
0
spectrum
20
Full Tapering Window, n=480, L=9 Full Tapering Window(log), n=480, L=9
−0.10
0.00
0.05
0.10
freq
John Fricks
−0.10
0.00
0.05
0.10
freq
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Smoothing
Tapering
0.0
0.4
0.8
Full Tapering, n=480, transformation in time domain
0
100
200
300
400
time
15
5
0
spectrum
Full Tapering Window, n=480, L=9
−0.10
−0.05
0.00
0.05
0.10
freq
−10
−25
spectrum
0
Full Tapering Window(log), n=480, L=9
−0.10
−0.05
0.00
0.05
0.10
freq
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Smoothing
Tapering
0.0
0.4
0.8
50% Tapering, n=480, transformation in time domain
0
100
200
300
400
time
30
0 10
spectrum
50% Tapering Window, n=480, L=9
−0.10
−0.05
0.00
0.05
0.10
freq
−5
−15
spectrum
50% Tapering Window(log), n=480, L=9
−0.10
−0.05
0.00
0.05
0.10
freq
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Smoothing
Tapering
Smoothing and Tapering
I
Smoothing introduces bias, but reduces variance.
I
Smoothing tries to solve the problem of too many
“parameters”.
I
Tapering decreases bias and introducese variance.
I
Tapering attempts to diminish the influence of sidelobes that
are introduced via the spectral window.
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
0
50
sun
100
150
Wolfer sunspots 1770-1869
0
20
40
60
80
100
Time
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Raw Periodogram
8000
6000
4000
2000
0
spectrum
12000
Series: sun
Raw Periodogram
0.0
0.1
0.2
0.3
0.4
0.5
frequency
bandwidth = 0.00289
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Periodogram with Smoothing Window of 3
6000
4000
2000
0
spectrum
8000
10000
Series: sun
Smoothed Periodogram
0.0
0.1
0.2
0.3
0.4
0.5
frequency
bandwidth = 0.00764
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Periodogram with Smoothing Window of 5
4000
2000
0
spectrum
6000
8000
Series: sun
Smoothed Periodogram
0.0
0.1
0.2
0.3
0.4
0.5
frequency
bandwidth = 0.0126
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Periodogram with Smoothing Window of 3 with Some
Tapering
6000
4000
2000
0
spectrum
8000
10000
Series: sun
Smoothed Periodogram
0.0
0.1
0.2
0.3
0.4
0.5
frequency
bandwidth = 0.00764
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Periodogram with Smoothing Window of 3 with More
Tapering
6000
4000
2000
0
spectrum
8000
10000
12000
Series: sun
Smoothed Periodogram
0.0
0.1
0.2
0.3
0.4
0.5
frequency
bandwidth = 0.00764
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Smoothed Periodogram with ARMA Spectral Density
0
2000
4000
var
6000
8000
10000
12000
The smoothed periodogram of the sun spot data with the spectral
density of the AR(3) model overlayed.
0.0
0.1
0.2
0.3
0.4
0.5
freq
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Dynamic Fourier Analysis
I
What can be done for non-stationary data?
I
One approach is to decompose our time series as a sum of a
non-constant (deterministic) trend plus a stationary “noise”
term:
xt = µt + yt
I
What if our data instead appears as a stationary model locally,
but globally the model appears to shift? One approach is to
divide the data into shorter sections (perhaps overlapping) and
I
This approach is developed in Shumway and Stoffer. One
essentially looks at how the spectral density changes over
time.
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
Wavelets
I
We have been using Fourier components as a basis to
represent stationary processes and seasonal trends.
I
Since we are dealing with finite data, we must use a finite
number of terms, and perhaps one could use an alternative
basis.
I
Wavlets are one option to accomplish this goal. They are
particularly well suited to the same situation as dynamic
Fourier analysis.
John Fricks
Time Series II – Frequency Domain Methods
Outline
Introduction
Stationarity
The Periodogram and the Spectral Density
Smoothing and Tapering
Example
Extensions and References
References
I
I
I
I
I
I
Robert Shumway and David Stoffer. Time Series Analysis and
Its Applications. Springer NY, 2006.
Peter Brockwell and Richard Davis. Time Series: Theory and
Methods, Second Ed. Springer NY, 1991.
David Brillinger. Time Series: Data Analysis and Theory.
SIAM, 2001.
Donald Percival and Andrew Walden. Spectral Analysis for
Physical Applications. Cambridge University Press, 1993.
Donald Percival and Andrew Walden. Wavelet Methods for
Time Series Analysis. Cambridge University Press, 2000.
Stéphane Mallat. A Wavelet Tour of Signal Processing.
Academic Press, 1998.
John Fricks
Time Series II – Frequency Domain Methods