Environmental Data Analysis with MatLab
2nd Edition
Lecture 20:
Coherence; Tapering and Spectral Analysis
SYLLABUS
Lecture 01
Lecture 02
Lecture 03
Lecture 04
Lecture 05
Lecture 06
Lecture 07
Lecture 08
Lecture 09
Lecture 10
Lecture 11
Lecture 12
Lecture 13
Lecture 14
Lecture 15
Lecture 16
Lecture 17
Lecture 18
Lecture 19
Lecture 20
Lecture 21
Lecture 22
Lecture 23
Lecture 24
Lecture 25
Lecture 26
Using MatLab
Looking At Data
Probability and Measurement Error
Multivariate Distributions
Linear Models
The Principle of Least Squares
Prior Information
Solving Generalized Least Squares Problems
Fourier Series
Complex Fourier Series
Lessons Learned from the Fourier Transform
Power Spectra
Filter Theory
Applications of Filters
Factor Analysis
Orthogonal functions
Covariance and Autocorrelation
Cross-correlation
Smoothing, Correlation and Spectra
Coherence; Tapering and Spectral Analysis
Interpolation
Linear Approximations and Non Linear Least Squares
Adaptable Approximations with Neural Networks
Hypothesis testing
Hypothesis Testing continued; F-Tests
Confidence Limits of Spectra, Bootstraps
Goals of the lecture
Part 1
Finish up the discussion of
correlations between time series
Part 2
Examine how the finite observation time affects estimates
of the power spectral density of time series
Part 1
“Coherence”
frequency-dependent correlations between time series
Scenario A
in a hypothetical region
windiness and temperature correlate at periods
of a year, because of large scale climate patterns
but they do not correlate at periods of a few days
wind speed
1
2
3
temperature
time, years
1
2
time, years
3
summer hot
and windy
wind speed
winters cool
and calm
1
2
3
temperature
time, years
1
2
time, years
3
wind speed
heat wave not
especially
windy
1
cold snap not
especially calm
2
3
temperature
time, years
1
2
time, years
3
in this case
times series correlated at long periods
but not at short periods
Scenario B
in a hypothetical region
plankton growth rate and precipitation correlate
at periods of a few weeks
but they do not correlate seasonally
growth rate
1
2
3
precipitation
time, years
1
2
time, years
3
plant growth rate
growth rate has no
seasonal signal
1
summer drier
than winter
2
3
precipitation
time, years
1
2
time, years
3
plant growth rate
growth rate high at times of peak precipitation
1
2
3
precipitation
time, years
1
2
time, years
3
in this case
times series correlated at short periods
but not at long periods
Coherence
a way to quantify
frequency-dependent correlation
strategy
band pass filter the two time series, u(t) and v(t)
around frequency, ω0
compute their zero-lag cross correlation
(large when time series are similar in shape)
repeat for many ω0’s to create a function c(ω0)
band pass filter f(t) has this p.s.d.
2Δω
|f(ω)|2
2Δω
ω
-ω0
0
ω0
evaluate at
zero lag t=0 and at many ω0’s
Short Cut
Fact 1
A function evaluates at time t=0 is equal to the
integral of its Fourier Transform
Fact 2
the Fourier Transform of a convolution is the product
of the transforms
integral over frequency
integral over frequency
assume ideal band pass filter that is either 0 or 1
negative
frequencies
positive
frequencies
integral over frequency
assume ideal band pass filter that is either 0 or 1
negative
frequencies
positive
frequencies
c is real
so real part is symmetric, adds
imag part is antisymmetric,
cancels
integral over frequency
assume ideal band pass filter that is either 0 or 1
negative
frequencies
positive
frequencies
c is real
so real part is symmetric, adds
imag part is antisymmetric,
cancels
interpret intergral as an
average over frequency band
integral over frequency can be viewed
as an average over frequency
(indicated with the overbar)
Two final steps
1. Omit taking of real part in formula
(simplifying approximation)
2. Normalize by the amplitude of the two time series and
square, so that result varies between 0 and 1
the final result is called
Coherence
precipitation
A)
5
0
0
200
400
600
800
1000
1200
1400
1600
T-air
B)
20
10
0
new dataset:
0
200
400
600
800
1000
1200
1400
1600
T-water
C)
10
salinity
0
0
turbidity
D)
200
400
600
800
1000
1200
1400
1600
200
400
600
800
1000
time, days
1200
1400
1600
200
400
600
800
1000
time, days
1200
1400
1600
200
400
600
800
1000
time, days
1200
1400
1600
34
32
30
28
0
chlorophyll
Water Quality
20
E)
20
0
0
F)
20
0
0
Reynolds
Channel,
Coastal Long
Island, New
York
precipitation
precipitation
A)
5
0
0
200
400
600
800
1000
1200
1400
1600
air temperature
T-air
B)
20
10
0
new dataset:
0
200
400
600
800
1000
1200
1400
1600
water temperature
T-water
C)
10
salinity
0
0
turbidity
D)
E)
400
600
800
1000
1200
1400
1600
200
400
600
800
1000
time, days
1200
1400
1600
600
800
1000
time, days
1200
1400
1600
800
1000
time, days
1200
1400
1600
turbidity
20
0
200
salinity
34
32
30
28
0
chlorophyll
Water Quality
20
0
F)
200
400
chlorophyl
20
0
0
200
400
600
Reynolds
Channel,
Coastal Long
Island, New
York
B) periods near 5 days
800 1000 1200 1400 1600
1
0
-1
-2
400
450
500
550
800 1000 1200 1400 1600
4
2
0
-2
-4
400
450
500
550
1
0
-1
400
450
500
550
0.6
0.4
0.2
0
-0.2
-0.4
400
450
500
time, days
550
450
500
time, days
550
450
500
time, days
550
0.05
0
-0.05
precip
precip
A) periods near 1 year
0
200
400
600
T-air
T-air
10
0
-10
-10
200
400
600
800 1000 1200 1400 1600
10
salinity
salinity
600
0
0
-10
0
200
400
600
800 1000 1200 1400 1600
time, days
1
0
-1
-2
-3
turbidity
turbidity
400
10
0
0
200
400
600
4
2
0
-2
-4
-6
0
200
400
600
800 1000 1200 1400 1600
time, days
2
0
-2
-4
400
800 1000 1200 1400 1600
time, days
chlorophyl
chlorophyl
200
T-water
T-water
0
2
0
-2
400
Fig, 9.18. Band-pass filtered water quality measurements from Reynolds Channel (New York) for several years
starting January 1, 2006. A) Periods near one year; and B) periods near 5 days. MatLab script eda09_16.
B) periods near 5 days
800 1000 1200 1400 1600
1
0
-1
-2
400
450
500
550
800 1000 1200 1400 1600
4
2
0
-2
-4
400
450
500
550
1
0
-1
400
450
500
550
0.6
0.4
0.2
0
-0.2
-0.4
400
450
500
time, days
550
450
500
time, days
550
450
500
time, days
550
0.05
0
-0.05
precip
precip
A) periods near 1 year
0
200
400
600
T-air
T-air
10
0
-10
-10
200
400
600
800 1000 1200 1400 1600
10
salinity
salinity
600
0
0
-10
0
200
400
600
800 1000 1200 1400 1600
time, days
1
0
-1
-2
-3
turbidity
turbidity
400
10
0
0
200
400
600
4
2
0
-2
-4
-6
0
200
400
600
800 1000 1200 1400 1600
time, days
2
0
-2
-4
400
800 1000 1200 1400 1600
time, days
chlorophyl
chlorophyl
200
T-water
T-water
0
2
0
-2
400
Fig, 9.18. Band-pass filtered water quality measurements from Reynolds Channel (New York) for several years
starting January 1, 2006. A) Periods near one year; and B) periods near 5 days. MatLab script eda09_16.
0.5
one year
0
C)
precipitation and salinity
1
one year
coherence
coherence
1
0
B)
air-temp and water-temp
water-temp and chlorophyll
1
coherence
A)
0.5
0.5
one week
0.05 0.1 0.15 0.2
frequency, cycles per day
0
0
0.05 0.1 0.15 0.2
frequency, cycles per day
0
0
0.05 0.1 0.15 0.2
frequency, cycles per day
high coherence at
periods of 1 year
0.5
one year
0
C)
precipitation and salinity
1
one year
coherence
coherence
1
0
B)
air-temp and water-temp
water-temp and chlorophyll
1
coherence
A)
0.5
0.5
one week
0.05 0.1 0.15 0.2
frequency, cycles per day
0
0
0.05 0.1 0.15 0.2
frequency, cycles per day
moderate
coherence at
periods of about a
month
0
0
0.05 0.1 0.15 0.2
frequency, cycles per day
very low
coherence at
periods of months
to a few days
Part 2
windowing time series before computing powerspectral density
scenario:
-1
0
0 indefinitely
200 long phenomenon
400
you are studying an
… 0
0.1
time t, s
ASD of d
ASD of W
0
0
-1
-10
0
but you only observe
a
1
200
400
200
400
time t, s
time t, s of it …
short portion
W(t)
W(t)*d(t)
1
0
0
-1
-10
0
200
200
time t, s
time t, s
400
400
ASD
of W
ASD
of Wd)
d(t)
W(t)
1
1
10
5
5
0
00
0
0.1
0.1
4
5
2
0
00
0
0.1
0.1
how does the power spectral
density of the short piece
differ from the p.s.d. of the
indefinitely long phenomenon
(assuming stationary time series)
-1
0
200
time t, s
00
0
200
200
200
time
t, ss
time
time t,
t, s
400
400
400
200
200
200
time
time
time t,t,
t, ss
s
400
400
400
200
200
time
time t,t, ss
400
400
00
0
0.2 0.3
0.4
frequency f, Hz
00
000
0
0.1
0.1
0.1
0.2
0.3
0.4
0.2
0.4
0.2 0.3
0.3
0.4
frequency
f,
Hz
frequency
frequency f,
f, Hz
Hz
00
00
0.1
0.1
0.2 0.3
0.4
frequency f, Hz
55
5
We might suspect that the
4
difference
will be
55
increasingly
significant
as
2
the window of observation
00
00
becomes
so0.2short
that
it
0.1
0.3
0
0.1
0
0.1 0.2
0.3 0.4
0.4
frequency
f,f, Hz
frequency
Hz
includes just
a few
44
oscillations of the period of
22
interest.
ASD of
of Wd)
Wd)
ASD
W(t)*d(t)
W(t)*d(t)
11
00
-1
-1
00
0.1
ASD
of
W
ASD
ASD
ofof
Wd)
W
W(t)
W(t)*d(t)
W(t)
11
1
-1
-1
-10
0
0
0
10
10
ASDofofdd
ASD
ASD of W
d(t)
W(t)
d(t)
11
1
-1
-1
-100
0
0
400
starting point
short piece
is
the indefinitely long time series
multiplied by a
window function, W(t)
ASD of d
d(t)
1
0
-1
0
200
time t, s
10
5
0
400
0
0.1
0.2 0.3
frequency f,
0
0.1
0.2 0.3
frequency f,
0
0.1
0.2 0.3
frequency f,
ASD of W
W(t)
1
0
-1
0
200
time t, s
ASD of Wd)
W(t)*d(t)
0
-1
0
400
1
0
200
time t, s
400
5
4
2
0
by the convolution theorem
Fourier Transform of short piece
is
Fourier Transform of indefinitely long time series
convolved with
Fourier Transform of window function
so
Fourier Transform of short piece
exactly
Fourier Transform of indefinitely long time series
when
Fourier Transform of window function
is a spike
ASD of d
d(t)
1
0
-1
10
5
0
0
200
400
boxcar
window
function
0
time t, s
0.2 0.3
0.4
its0.1Fourier
Transform
frequency f, Hz
ASD of W
W(t)
1
0
-1
0
200
time t, s
ASD of Wd)
W(t)*d(t)
0
-1
0
400
1
0
200
time t, s
400
5
0
0.1
0.2 0.3
0.4
frequency f, Hz
0
0.1
0.2 0.3
0.4
frequency f, Hz
4
2
0
ASD of d
d(t)
1
0
-1
10
5
0
0
200
400
boxcar
window
function
0
time t, s
ASD of W
W(t)
1
0
-1
0
200
time t, s
ASD of Wd)
W(t)*d(t)
0
-1
0
400
1
0
200
time t, s
400
5
0.2 0.3
0.4
its0.1Fourier
Transform
frequency f, Hz
sinc() function
sort of spiky
but has side lobes
0
0.1
0.2 0.3
0.4
frequency f, Hz
0
0.1
0.2 0.3
0.4
frequency f, Hz
4
2
0
ASD of d
d(t)
1
0
-1
0
200
time t, s
10
5
0
400
0
0.1
0.2 0.3
0.4
frequency f, Hz
0
0.1
0.2 0.3
0.4
frequency f, Hz
0
0.1
0.2 0.3
0.4
frequency f, Hz
ASD of W
W(t)
1
0
-1
0
200
time t, s
ASD of Wd)
W(t)*d(t)
0
-1
0
400
1
0
200
time t, s
400
5
4
2
0
Effect 1: broadening of spectral peaks
ASD of d
d(t)
1
0
-1
0
200
time t, s
5
0
400
narrow spectral
peak
10
0
ASD of W
W(t)
1
0
-1
0
200
time t, s
ASD of Wd)
W(t)*d(t)
1
0
-1
0
200
time t, s
400
0.2 0.3
0.4
frequency f, Hz
wide central spike
5
0
400
0.1
0
0.1
4
0.2 0.3
0.4
frequency f, Hz
wide spectral peak
2
0
0
0.1
0.2 0.3
0.4
frequency f, Hz
Effect 2: spurious side lobes
ASD of d
d(t)
1
0
-1
0
200
time t, s
5
0
400
only one spectral
peak
10
0
ASD of W
W(t)
1
0
-1
0
200
time t, s
ASD of Wd)
W(t)*d(t)
1
0
-1
0
200
time t, s
400
0.2 0.3
0.4
frequency f, Hz
side lobes
5
0
400
0.1
0
0.1
4
spurious spectral
peaks
2
0
0.2 0.3
0.4
frequency f, Hz
0
0.1
0.2 0.3
0.4
frequency f, Hz
Q: Can the situation be improved?
A: Yes, by choosing a smoother
window function
more like a Normal Function
(which has no side lobes)
but still zero outside of interval of observation
-1
5
0
-1
0
0
200
400
boxcar
window
function
200
time t, s
ASD of Wd)
W(t)*d(t)
W(t)*d(t)
0
-1
0
200
time t, s
400
5
0
0-1
00
400
1
1
ASD of W
W(t)
ASD of W
W(t)
0
5
frequency f, Hz
time t, s
1
-1
10
0.1
0.2window
0.3
0.4function 0
Hamming
0
200
400
0
time t, s
0
0
0.1
41
20
0-1
00
0.1
200 0.3 400
0.2
0.4
time t,f,sHz
frequency
4
2
0
0.2
0.4
200 0.3 400
frequency
time t,f,sHz
ASD of Wd)
0
ASD of d
10 1
ASD of d
d(t)
d(t)
1
0
2
1
0
0
ASD of d
d(t)
1
0
-1
0
200
time t, s
ASD of W
W(t)
1
-1
0
200
time t, s
ASD of Wd)
W(t)*d(t)
0
-1
0
200
time t, s
400
0
0.1
0.2 0.3
0.4
frequency f, Hz
0
0.1
0.2 0.3
0.4
frequency f, Hz
0
0.1
0.2 0.3
0.4
frequency f, Hz
4
2
0
400
1
5
0
400
0
10
2
1
0
ASD of d
d(t)
1
0
-1
0
200
time t, s
ASD of W
W(t)
1
-1
0
200
time t, s
ASD of Wd)
W(t)*d(t)
0
-1
0
200
time t, s
400
0
0.1
0.2 0.3
0.4
frequency f, Hz
0
0.1
0.2 0.3
0.4
frequency no
f, Hzside
4
2
0
400
1
5
0
400
0
10
2
1
0
0
0.1
lobes
but
central peak
wider than with
boxcar
0.3
0.4
0.2
frequency f, Hz
Hamming Window Function
Q: Is there a “best” window function?
A: Only if you carefully specify what
you mean by “best”
(notion of best based on prior information)
“optimal”window function
maximize
ratio of
power in central peak
(assumed to lie in range ±ω0 )
to overall power
The parameter, ω0, allows you to choose how
much spectral broadening you can tolerate
Once ω0 is specified, the problem can be solved
by using standard optimization techniques
One finds that there are actually several window
functions, with radically different shapes, that
are “optimal”
(t)
W3(t)
v
w3(t)
W2(t)
v
w2(t)
W1(t)
v
w1(t)
Family of three “optimal” window functions
0.2
0
-0.2
0
10
20
0
10
20
0
10
20
30
timev t, s
40
50
60
30
time, s
time
v t, s
40
50
60
30
timev t, s
40
50
60
time, s
0.2
0
-0.2
0.2
0
-0.2
0.2
time, s
a common strategy is
to compute the power spectral density
with each of these window functions separately
and then average the result
technique called
Multi-taper Spectral Analysis
|d(f)|
-0.5
400
w1(t)d(t)
400
400
0.4
0.2
0
500 0
400
0.2
0
500 0
v
w2(t)d(t)
W1(t)d(t) 0.2
0.1
0
-0.1
0
100
200
300
time t, s v
time t, s
w3(t)d(t)
W2(t)d(t) 0.1
0
-0.1
0
100
200
300
time t, s v
time t, s
W3(t)d(t) 0.1
0
-0.1
0
100
200
300
time t, s
time t, vs
0.1
0.2
0.3
0.4
frequency
v f, Hz
|d0(f)|
200
300
time
t, s t,vs
time
100
50
0
500 0
|d1(f)|
d(t)
0
-0.5
100
0
frequency, Hz
1
B(t)d(t)0.5
0
500
400
0.1
0.1
0.2
0.3
0.4
frequency
v f, Hz
frequency, Hz
0.5
0.2
0.3
0.4
v f, Hz
frequency
frequency, Hz
|d2(f)|
200
300
time
t, s t,vs
time
0.1
0.2
0.3
0.4
frequency
v f, Hz
frequency, Hz
|d3(f)|
100
0.2
0
500 0
|d(f)|avg
0
20
10
0
v
d(t)
1
d(t) 0.5
0
0.2
0.1
0
0
0.1
0.1
0.2
0.3
0.4
frequency
v f, Hz
frequency, Hz
0.2
0.3
0.4
frequency
v f, Hz
frequency, Hz
0.5
v
|d(f)|
-0.5
400
w1(t)d(t)
400
400
0.4
0.2
0
500 0
400
0.2
0
500 0
v
w2(t)d(t)
W1(t)d(t) 0.2
0.1
0
-0.1
0
100
200
300
time t, s v
time t, s
w3(t)d(t)
W2(t)d(t) 0.1
0
-0.1
0
100
200
300
time t, s v
time t, s
W3(t)d(t) 0.1
0
-0.1
0
100
200
300
time t, s
time t, vs
0.1
0.2
0.3
0.4
frequency
v f, Hz
|d0(f)|
200
300
time
t, s t,vs
time
100
50
0
500 0
|d1(f)|
d(t)
0
-0.5
100
0
frequency, Hz
1
B(t)d(t)0.5
0
500
400
0.1
0.1
0.2
0.3
0.4
frequency
v f, Hz
frequency, Hz
0.5
0.2
0.3
0.4
v f, Hz
frequency
frequency, Hz
|d2(f)|
box car tapering
200
300
time
t, s t,vs
time
0.1
0.2
0.3
0.4
frequency
v f, Hz
frequency, Hz
|d3(f)|
100
0.2
0
500 0
|d(f)|avg
0
20
10
0
v
d(t)
1
d(t) 0.5
0
0.2
0.1
0
0
0.1
0.1
0.2
0.3
0.4
frequency
v f, Hz
frequency, Hz
0.2
0.3
0.4
frequency
v f, Hz
frequency, Hz
0.5
v
|d(f)|
-0.5
0
100
200
300
time
t, s t,vs
time
400
d(t)
0
-0.5
100
200
300
time
t, s t,vs
time
0
0.1
0.2
0.3
0.4
frequency
v f, Hz
frequency, Hz
1
B(t)d(t)0.5
0
500
20
10
0
400
|d0(f)|
d(t)
1
d(t) 0.5
0
100
50
0
500 0
0.1
0.2
0.3
0.4
400
0.2
0
500 0
w3(t)d(t)
W2(t)d(t) 0.1
0
-0.1
0
100
200
300
time t, s v
time t, s
W3(t)d(t) 0.1
0
-0.1
0
100
200
300
time t, s
time t, vs
400
0.1
0.2
0.3
0.4
v f, Hz
frequency
frequency, Hz
|d2(f)|
200
300
time t, s v
time t, s
0.5
0.1
0.2
0.3
0.4
frequency
v f, Hz
frequency, Hz
|d3(f)|
100
0.2
0
500 0
|d(f)|avg
0
-0.1
0
|d1(f)|
400
0.4
0.2
0
500 0
v
w2(t)d(t)
W1(t)d(t) 0.2
0.1
v
w1(t)d(t)
Hz
tapering with three “optimal” windowfrequency,
functions
frequency
v f, Hz
0.2
0.1
0
0
0.1
0.1
0.2
0.3
0.4
frequency
v f, Hz
frequency, Hz
0.2
0.3
0.4
frequency
v f, Hz
frequency, Hz
0.5
v
|d(f)|
-0.5
400
w1(t)d(t)
400
400
0.4
0.2
0
500 0
400
0.2
0
500 0
v
w2(t)d(t)
W1(t)d(t) 0.2
0.1
0
-0.1
0
100
200
300
time t, s v
time t, s
w3(t)d(t)
W2(t)d(t) 0.1
0
-0.1
0
100
200
300
time t, s v
time t, s
W3(t)d(t) 0.1
0
-0.1
0
100
200
300
time t, s
time t, vs
0.1
0.2
0.3
0.4
frequency
v f, Hz
|d0(f)|
200
300
time
t, s t,vs
time
100
50
0
500 0
|d1(f)|
d(t)
0
-0.5
100
0
frequency, Hz
1
B(t)d(t)0.5
0
500
400
p.s.d. produced by
averaging
0.1
0.1
0.2
0.3
0.4
frequency
v f, Hz
frequency, Hz
0.5
0.2
0.3
0.4
v f, Hz
frequency
frequency, Hz
|d2(f)|
200
300
time
t, s t,vs
time
0.1
0.2
0.3
0.4
frequency
v f, Hz
frequency, Hz
|d3(f)|
100
0.2
0
500 0
|d(f)|avg
0
20
10
0
v
d(t)
1
d(t) 0.5
0
0.2
0.1
0
0
0.1
0.1
0.2
0.3
0.4
frequency
v f, Hz
frequency, Hz
0.2
0.3
0.4
frequency
v f, Hz
frequency, Hz
0.5
v
Summary
always taper a time series before
computing the p.s.d.
try a simple Hamming taper first
it’s simple
use multi-taper analysis when
higher resolution is needed
e.g. when the time series is very short