Environmental Data Analysis with MatLab
2nd Edition
Lecture 19:
Smoothing, Correlation and Spectra
SYLLABUS
Lecture 01
Lecture 02
Lecture 03
Lecture 04
Lecture 05
Lecture 06
Lecture 07
Lecture 08
Lecture 09
Lecture 10
Lecture 11
Lecture 12
Lecture 13
Lecture 14
Lecture 15
Lecture 16
Lecture 17
Lecture 18
Lecture 19
Lecture 20
Lecture 21
Lecture 22
Lecture 23
Lecture 24
Lecture 25
Lecture 26
Using MatLab
Looking At Data
Probability and Measurement Error
Multivariate Distributions
Linear Models
The Principle of Least Squares
Prior Information
Solving Generalized Least Squares Problems
Fourier Series
Complex Fourier Series
Lessons Learned from the Fourier Transform
Power Spectra
Filter Theory
Applications of Filters
Factor Analysis
Orthogonal functions
Covariance and Autocorrelation
Cross-correlation
Smoothing, Correlation and Spectra
Coherence; Tapering and Spectral Analysis
Interpolation
Linear Approximations and Non Linear Least Squares
Adaptable Approximations with Neural Networks
Hypothesis testing
Hypothesis Testing continued; F-Tests
Confidence Limits of Spectra, Bootstraps
Goals of the lecture
examine interrelationships between
smoothing, correlation and power spectral density
review
Autocorrelation
and
Cross-correlation
Autocorrelation
Measure of correlation in time series
at different lags
u(t)
t
Autocorrelation
Measure of correlation in time series
at different lags
t
t
lag, multiply and sum area
no lag
a(t)
0
lag, t
Autocorrelation
Measure of correlation in time series
at different lags
t
t
lag, multiply and sum area
small lag
a(t)
0
lag, t
Autocorrelation
Measure of correlation in time series
at different lags
t
t
lag, multiply and sum area
large lag
a(t)
0
lag, t
Autocorrelation
Measure of correlation in time series
at different lags
t
t
lag, multiply and sum area
a(t)
0
lag, t
Autocorrelation
Measure of correlation in time series
at different lags
t
t
lag, multiply and sum area
a(t)
0
lag, t
a(t)=u(t)⋆u(t)
crooss-correlation
Measure of correlation between two time series
at different lags
u(t)
v(t)
t
t
crooss-correlation
Measure of correlation between two time series
at different lags
u(t)
v(t)
t
t
c(t)=u(t)⋆v(t)
important relationships
c(t) = u(t)⋆v(t) = u(-t)*v(t)
c(ω) = u*(ω) v(ω)
a(ω)= |u(ω)|2
rough time series
u(t)
t
sharp autocorrelation
a(t)
0
lag, t
wide spectrum
|u(ω)|2
0
frequency, ω
smooth time series
u(t)
t
wide autocorrelation
a(t)
0
lag, t
narrow spectrum
|u(ω)|2
0
frequency, ω
v(t)
t
rough
timeseries
a(t)
a(t)
0
lag, t
0
v(t)
lag, t
t
a(t)
0
lag, t
Part 1
Smoothing a Time Series
smoothing as filtering
(example of 3-point smoothing)
smoothing as filtering
(example of 3-point smoothing)
non-causal
fix-up
allow for a delay
fix-up
allow for a delay
dsmoothed and delayed = s * dobs
causal filter, s
triangular smoothing filters
3 points
si
index, i
si
21 points
index, i
smoothing if Neuse River Hydrograph
4
x 10
d-obs
2
1
0
50
100
150
200
250
300
time, days
350
400
450
500
50
100
150
200
250
300
time, days
350
400
450
500
50
100
150
200
250
300
time, days
350
400
450
500
4
x 10
3-point
2
1
0
4
x 10
21-point
2
1
0
question
how does smoothing effect the
the autocorrelation of d
answer
the autocorrelation of s
acts as a smoothing filter on
the autocorrelation of d
effect of smoothing on autocorrelation
effect of smoothing on autocorrelation
autocorrelation of
smoothed time series
effect of smoothing on autocorrelation
autocorrelation of
smoothed time series
everything written as
convolution
effect of smoothing on autocorrelation
autocorrelation of
smoothed time series
everything written as
convolution
regrouped
effect of smoothing on autocorrelation
autocorrelation of
smoothing filter
convolved
with
*
autocorrelation
of time series
answer
the autocorrelation of s
acts as a smoothing filter on
the autocorrelation of d
Part 2
What Makes a Good Smoothing Filter?
dsmoothed(t) = s(t) * dobs(t)
then by the convolution theorem
dsmoothed(t) = s(t) * dobs(t)
then by the convolution theorem
so what’s this look like?
example of a
uniform or “boxcar” smoothing filter
s(t)
1/T
0 T
time, t
take Fourier Transform
where
sinc(x) = sin(πx) / (πx)
A) T=3
|s| for L=3
1
0.5
0
0
10
15
20
25
30
frequency, Hz
35
40
45
50
15
20
25
30
frequency, Hz
35
40
45
50
B) T=21
1
|s| for L=21
5
0.5
0
0
5
10
|s| for L=3
0.5
0
0
10
20
25
30
frequency, Hz
35
40
45
50
falls off with frequency (good)
0.5
0
15
B) T=21
1
|s| for L=21
5
0
5
10
15
20
25
30
frequency, Hz
35
40
45
50
|s| for L=3
0.5
0
0
10
15
B) T=21
1
|s| for L=21
5
20
25
30
frequency, Hz
35
40
45
50
bumpy side lobes (bad)
0.5
0
0
5
10
15
20
25
30
frequency, Hz
35
40
45
50
a box car filter does not suppress high
frequencies evenly
the challenge
find a filter that suppresses high
frequencies evenly
Normal Function
Fourier Transform of a Normal Function
is a
Normal Function
(which has no side lobes)
B) T=3
1
|s| for L=3
A) L=3
0.5
0
0
10
15
20
25
30
frequency, Hz
35
40
45
50
15
20
25
30
frequency, Hz
35
40
45
50
B) T=21
1
|s| for L=21
5
0.5
0
0
5
10
but a Normal Function
is non-causal
(unless you truncate it, in which case it is not
exactly a Normal Function)
Box Car
Normal Function
simplicity
sidelobes
Triangle
Part 3
Designing a Filter that Suppresses
Specific Frequencies
General form of the IIR Filter, f
z-transform of the IIR filter
General form of the IIR Filter
z-transform
General form of the IIR Filter
z-transform
ratio of
polynomials
z-transform of the IIR filter
u(z) as a
product of its
factors
roots of u(z)
roots of
v(z) as a v(z)
z-transform ratio of
polynomials product of its
factors
so designing a filter
is equivalent to
specifying the roots
of the two polynomails
u(z) and v(z)
at this point we need to explore the
relationship between the
Fourier Transform
and the
z-transform
Answer
the Fourier Transform
is the
z-transform
evaluated at a specific set of z’s
Relationship between Fourier
Transform and Z-transform
since
Relationship between Fourier
Transform and Z-transform
Fourier Transform
since
Relationship between Fourier
Transform and Z-transform
Fourier Transform
discrete times and frequencies
since
Relationship between Fourier
Transform and Z-transform
Fourier Transform
discrete times and frequencies
z-transform
since
Relationship between Fourier
Transform and Z-transform
Fourier Transform
discrete times and frequencies
z-transform
since
specific choice of z’s
in words
the Fourier Transform
is the
z-transform
evaluated at a specific set of z’s
there are N specific z’s
zk
or
with θ
they plot as equally-spaced points around
a “unit circle” in the complex z-plane
unit circle,
|z|2=1
q
real z
zero
frequency
Nyquist
frequency
imag z
Back to the IIR Filter
roots of
u(z)
roots of
v(z)
Back to the IIR Filter
(z-zju) is zero at z=zju
produces a low amplitude
region near z=zju
called a “zero”
Back to the IIR Filter
1/(z-zkv) is infinite at z=zku
produces a high amplitude
region near z=zkv
called a “pole”
so build a filter by
placing the poles and zeros at
strategic points in the complex z-plane
Rules
zeros suppress frequencies
poles amplify frequencies
all poles must be outside the unit circle
(so vinv converges)
all poles, zeros must be in complex conjugate
pairs
(so filter is real)
|S|2
A)
-2
4
-1.5
3.5
-1
3
2.5
|S|2
imag z
-0.5
0
0.5
1.5
1
1
1.5
0.5
2
-2
-1
0
real z
1
2
0
-2
0.1
0.2
0.3
frequency, Hz
0.4
0.5
4
-1.5
3.5
-1
3
-0.5
|S|2
2.5
0
0.5
2
1.5
1
1
1.5
2
-2
0
|S|2
B)
imag z
2
0.5
-1
0
real z
1
2
0
0
0.1
0.2
0.3
frequency, Hz
0.4
0.5
|S|2
A)
-2
4
-1.5
-1
3
2.5
|S|2
imag z
-0.5
0
0.5
2
1.5
1
1
1.5
0.5
2
-2
-1
0
real z
1
2
0
-2
0.1
0.2
0.3
frequency, Hz
4
-1.5
0.4
0.5
zero near the
Nyquist frequency
suppresses high
frequencies
“low pass filter”
3.5
-1
3
-0.5
|S|2
2.5
0
0.5
2
1.5
1
1
1.5
2
-2
0
|S|2
B)
imag z
zero near zero
frequency
suppresses low
frequencies
“high pass filter”
3.5
0.5
-1
0
real z
1
2
0
0
0.1
0.2
0.3
frequency, Hz
0.4
0.5
-2
|S|2
A)
35
30
-1
|S|2
imag z
25
0
20
15
1
10
5
2
-2
-1
0
real z
1
2
0
|S|2
B)
0
0.1
0.2
0.3
frequency, Hz
0.4
0.5
0
0.1
0.2
0.3
frequency, Hz
0.4
0.5
1.2
-2
1
-1
|S|2
imag z
0.8
0
0.6
0.4
1
0.2
2
-2
-1
0
real z
1
2
0
-2
|S|2
A)
35
poles near ± a
given frequency
amplify that
frequency
“band pass filter”
30
-1
|S|2
imag z
25
0
20
15
1
10
5
2
-2
-1
0
real z
1
2
0
|S|2
B)
0
0.1
0.2
0.3
frequency, Hz
0.4
1.2
-2
poles and zeros
near ± a given
frequency
attenuate that
frequency
“notch filter”
1
-1
|S|2
imag z
0.8
0
0.6
0.4
1
0.2
2
-2
-1
0
real z
0.5
1
2
0
0
0.1
0.2
0.3
frequency, Hz
0.4
0.5
something useful
a tunable band pass filter
|f(ω)|2
-fny -f2 -f1
0
f1 f2 +fny
frequency, f
Chebychev band-pass filter: 4 zeros, 4 poles
-2
-1.5
-1
imag z
-0.5
0
0.5
1
1.5
2
-2
-1.5
-1
-0.5
0
real z
0.5
1
1.5
2
Chebychev band-pass filter: 4 zeros, 4 poles
-2
-1.5
pole
-1
2 zeros
imag z
-0.5
pole
0
2 zeros
pole
0.5
pole
1
1.5
2
-2
-1.5
-1
-0.5
0
real z
0.5
1
1.5
2
2
output spectrum
1
0.5
0
0
10
20
30
40
frequency, Hz
50
2
output spectrum
1
not quite
as boxy as
one might
hope …
0.5
0
0
10
20
30
40
frequency, Hz
50
1
input spectrum
1
input
0.5
0
-0.5
-1
0.8
0.9
1
1.1
time, s
0.5
0
1.2
0
10
20
3
frequency,
0
10
20
3
frequency,
1
output spectrum
1
output
0.5
0
-0.5
-1
0.8
0.9
1
1.1
time, s
1.2
0.5
0
In MatLab
Ground velocity at Palisades NY
Ground velocity at Palisades NY
Low pass filter
Ground velocity at Palisades NY
high pass filter