Lecture 1
Sampling of Signals
by
Graham C. Goodwin
University of Newcastle
Australia
Lecture 1
Presented at the “Zaborszky Distinguished Lecture Series”
December 3rd, 4th and 5th, 2007
1
Recall Basic Idea of Sampling
and Quantization
6
5
4
Quantization
3
2
1
t
0
t1
t3
t2
t4
Sampling
2

In this lecture we will ignore quantization
issues and focus on the impact of different
sampling patterns for scalar and
multidimensional signals
3
Outline
1.
2.
3.
4.
5.
6.
7.
8.
9.
One Dimensional Sampling
Multidimensional Sampling
Sampling and Reciprocal Lattices
Undersampled Signals
Filter Banks
Generalized Sampling Expansion (GSE)
Recurrent Sampling
Application: Video Compression at Source
Conclusions
4
Sampling:
Assume amplitude quantization
sufficiently fine to be negligible.
Question:
Say we are given
f (ti ) ; i Î Z
Under what conditions can we recover f (t ); t Î ¡
from the samples?
5
A Well Known Result (Shannon’s
Reconstruction Theorem for Uniform
Sampling)
Consider a scalar signal f(t) consisting of
ö
çç- ws , ws ÷
frequency components in the range æ
÷. If
2
2ø
è
this signal is sampled at period D < 2p w , then the
s
signal can be perfectly reconstructed from the
samples using:
¥
y (t ) =
å
k= - ¥
éæws ö
ù
ú
sin êçç ÷
t
k
D
(
)
÷
÷
êëè 2 ø
ú
û
y [k ]
æws ö
çç ÷
(t - kD )
÷
÷
è2ø
6
Proof: Sampling produces folding
Ys (w)
- ws
2
ws
2
ws
Low pass filter recovers
original spectrum
Hence Y (w) = H s (w)Ys (w)
H s (w) = 1
or y (t ) =
=
ò
¥
- ¥
ò
¥
- ¥
s
hs (s )y (t - s )d s
= 0
æ- ws
ö
ws ÷
çç
£ w£
÷
çè 2
ø
2÷
otherwise
¥
hs (s ) å
y [k ]d(t - s - k D )d s
k= - ¥
¥
=
å
k= - ¥
y [k ]hs (t - k D )
7
A Simple (but surprising) Extension
[Recurrent Sampling]
D k = M kD
where
{M k } is a periodic sequence of integers; i.e., M k + N = M k
Let
N
å
Mk = K
k= 1
T = KD
KD
=D
Note that the average sampling period is
e.g.
N
D1 = 9
D2 = 1
D3 = 9
D4 = 1
average 5
8
x x
xx
x x
-1 0
9 10
19 20
x
0
Non-uniform
x
x
x
x
5
10
15
20
Uniform
9
Claim:
Provided the signal is bandlimited to æççè- ws 2 , ws 2 öø÷÷
where
ws = 2p
D
, then the signal can be
perfectly reconstructed from the periodic
sampling pattern.
where D = average sampling period
Proof:
We will defer the proof to later when we
will use it as an illustration of Generalized
Sampling Expansion (GSE) Theorem.
10
Outline
1.
2.
3.
4.
5.
6.
7.
8.
9.
One Dimensional Sampling
Multidimensional Sampling
Sampling and Reciprocal Lattices
Undersampled Signals
Filter Banks
Generalized Sampling Expansion (GSE)
Recurrent Sampling
Application: Video Compression at Source
Conclusions
11
Multidimensional Signals
Digital Photography
x1
x2
Digital Video
x2
x1
x3 (time)
12
Outline
1.
2.
3.
4.
5.
6.
7.
8.
9.
One Dimensional Sampling
Multidimensional Sampling
Sampling and Reciprocal Lattices
Undersampled Signals
Filter Banks
Generalized Sampling Expansion (GSE)
Recurrent Sampling
Application: Video Compression at Source
Conclusions
13
How should we define sampling for multidimensional signals?
Utilize idea of Sampling Lattice
T nonsingular matrix Î ¡
D
´ ¡
D
Sampling Lattice
L = Lat (T ) = {Tn : n Î Z D }
14
Also, need multivariable frequency domain
concepts.
These are captured by two ideas
i.
ii.
Reciprocal Lattice
Unit Cell
15
Reciprocal Lattice
{
L = Lat 2p (T
*
T - 1
)
} {
= 2p (T
T - 1
)
n:nÎ ZD
}
Unit Cell UC (L * ) (Non-unique)
i.
{UC (L )+ 2p (T
*
n1 , n2 Î Z D
ii.
U{
T - 1
)
nÎ Z
T - 1
)
}
n2 = Æ
n1 ¹ n2
UC (L )+ 2p (T
D
}{
n1 Ç UC (L )+ 2p (T
*
T - 1
)
}
n = RD
16
One Dimensional Example
x
x
-20
-10
x
0
D
x
x
10
20
Sampling Lattice
L = {D .n : n Î Z }
17
Reciprocal Lattice and Unit Cell
Unit Cell
w1
0
1
10
2
10
3
10
2p
18
Multidimensional Example
x2
5
4
3
2
1
-4 -3 -2 -1
1 2
3 4 5
x1
-1
-2
-3
-4
é2 1ù
ú
T= ê
êë0 2ú
û
19
Reciprocal Lattice and Unit Cell for
Example
w2
2p
(
- 1
UC 2p (T T )
)
1/2
1/4
-1/4
-1/2
-3/4
-1
1/4 1/2 3/4 1
w1
2p
é1
ê
ê2
T - 1
(T ) = ê 1
êê
ë 4
ù
0ú
ú
ú
1ú
ú
2û
20
Outline
1.
2.
3.
4.
5.
6.
7.
8.
9.
One Dimensional Sampling
Multidimensional Sampling
Sampling and Reciprocal Lattices
Undersampled Signals
Filter Banks
Generalized Sampling Expansion (GSE)
Recurrent Sampling
Application: Video Compression at Source
Conclusions
21
We will be interested here in the situation where
the Sampling Lattice is not a Nyquist Lattice for
the signal (i.e., the signal cannot be perfectly
reconstructed from the original pattern!)
Strategy:
We will generate other samples by ‘filtering’
or ‘shifting’ operations on the original pattern.
22
Consider a bandlimited signal f (x), x Î ¡
D
.
Assume the D-dimension Fourier transform has finite
support, S.
Then for given D-dimensional lattice T, there always
P
exists a finite set {wi } Î L * , such that support
(
fˆ (w) Í S =
)
P
1
(
)
*
UC
L
(
)+ wi .
U
i= 1
Heuristically: The idea of “Tiling” the
area of interest in the frequency domain
23
One Dimensional Example
Our one dimensional example continued.
Sampling Lattice L = {kD ; k Î Z }
Unit Cell
w1
0
fˆ (w)
1
10
2
10
3
10
2p
Bandlimited spectrum
- 1
12
w1 = 0
Use
æ2p ö Support
w2 = - çç ÷
÷
çè 10 ÷
ø
w
1
12
2p
( fˆ (w)) = éêëUC (L )ùúûUéêëUC (L )+ w ùúû
*
*
2
24
Outline
1.
2.
3.
4.
5.
6.
7.
8.
9.
One Dimensional Sampling
Multidimensional Sampling
Sampling and Reciprocal Lattices
Undersampled Signals
Filter Banks
Generalized Sampling Expansion (GSE)
Recurrent Sampling
Application: Video Compression at Source
Conclusions
25
Generation of Extra Samples
Suppose now we generate a data set
Q
as shown in below (Q ³ P)
g q (Tn)
{{
}q= 1}nÎ Z
D
ĥ1 (w)
f (x )
g1 (x)
L
M
gq (x)
hˆq (w)
L
M
gQ (x)
ˆ
hQ (w)
L
Q – Channel Filter Bank
26
Outline
1.
2.
3.
4.
5.
6.
7.
8.
9.
One Dimensional Sampling
Multidimensional Sampling
Sampling and Reciprocal Lattices
Undersampled Signals
Filter Banks
Generalized Sampling Expansion (GSE)
Recurrent Sampling
Application: Video Compression at Source
Conclusions
27
éhˆ (w + w ) hˆ (w + w ) L
1
2
1
ê1
êˆ
êh1 (w + w2 )
Define H (w) = ê
M
ê
ê
êhˆ1 (w + wP )
ë
Let
hˆQ (w + w1 )ù
ú
ú
ú
ú
ú
ú
ˆ
hQ (w + wP )ú
û
éF 1 (w, x )ù
ê
ú
F (w, x ) = êê M úú
êF (w, x )ú
êë Q
úû
ée jw1T x ù
ê
ú
be the solution (if it exists) of H (w)F (w, x ) = êê M úú
ê jwTP x ú
êëe úû
*
for w Î UC (L )
28
Conditions for Perfect
Reconstruction
GSE Theorem:
f (x) can be reconstructed from
Q
f (x) =
å å
g q (Tk )f q (x - Tk )
q= 1 k Î Z D
if and only if H (w) has full row rank for all w in
the Unit Cell
where f
q
(x)=
òF
UC
q
(w, x)e
jwT x
dw
29
Proof:
F q (w, x)e
jwT x
=
å
- jwT Tk
f q (x - Tk )e
; w Î UC (L * )
kÎ Z D
Multiply both sides by hˆq (w + wi ) where wi Î L * (the
Reciprocal Lattice). Then sum over ‘q’
Q
å å
q= 1 k Î Z
hˆq (w + wi )f q (x - Tk )e
- jwT Tk
T
hˆq (w + wi )F q (w, x )e jw x
å
=
D
q= 1
T
=e
Q
j(w+ wi ) x
from the Matrix identity that defines F (w, x )
Note that {w + wi ; w Î L * and i = 1,K , P} “tiles” the entire
support S
Thus, f (x ) =
ò
fˆ (w)e
P
dw =
fˆ (w + wi )e
å ò
T
j(w+ wi ) x
dw
i= 1 UC L *
( )
s
P
=
jwT x
å ò
i= 1 UC L *
( )
Q
fˆ (w + wi )å
å
q= 1 k Î Z D
hˆq (w + wi )f q (x - Tk )e-
jwT Tk
dw
30
éP
ê
f (x) = å å êå ò fˆ (w + wi )hˆq (w + wi )e
q= 1 k Î Z D êi= 1 UC L *
( )
êë
Q
ù
ú
j(w+ wi ) Tk
d wúf q (x - Tk )
ú
ú
û
T
where we have used the fact that wi = 2p (T
T - 1
)
l for l Î Z D .
Since gq (x) is the output of f(x) passing through hˆq (w),
then
[
]= gq (Tk )
Hence, we finally have
Q
f (x) =
å å
q= 1 k Î Z D
g q (Tk )f q (x - Tk )
ÑÑÑ
31
Outline
1.
2.
3.
4.
5.
6.
7.
8.
9.
One Dimensional Sampling
Multidimensional Sampling
Sampling and Reciprocal Lattices
Undersampled Signals
Filter Banks
Generalized Sampling Expansion (GSE)
Recurrent Sampling
Application: Video Compression at Source
Conclusions
32
Special Case: Recurrent Sampling
(where hq is implemented by a “spatial” shift xq )
This amounts to the sampling pattern:
Q
Y=
I {Lat (T )+ x }
q
q= 1
where w.l.o.g.
{xq }Î UC (T )
Now, given the samples
{f (x%)}x%Î Y , our goal is to
perfectly reconstruct f (x).
33
jw x
Here hˆq (w) = e q , and g q (x) = f (x + xq )
T
Thus
{g q (Tn)}nÎ Z
P
=
,q= 1,K Q
{f (x%)}x%Î Y
To apply the GSE Theorem we require
T
j(w+ w1 ) xQ ù
é j(w+ w1 )T x1
L e
êe
ú
H (w) = ê
ú
T
T
j
w
+
w
x
(
)
j
w
+
w
x
(
)
P
Q ú
P
1
êe
L
e
ë
û
ée jw1T x1 L e jw1T xQ ùé jwT x1
ù
0
e
úê
ú
= êê T
O
T
ú
jw xQ ú
jwTP xQ ê
jwP x1
0
e
êëe
ú
ú
L e
û
ûêë
Nonsingular
34
Something to think about
The GSE result depends on inversion of a
particular matrix, H(w). Of course we have
assumed here perfect representation of all
coefficients. An interesting question is
what happens when the representation is
imperfect i.e. coefficients are represented
with finite wordlength (i.e. they are
quantized)
We will not address this here but it is
something to keep in mind.
35
Return to our one-dimensional
example
Recall that we had w1 = 0
2p
w2 = 10
so that
*
éUC (L * )+ w ù
È
support fˆ (w) = éêëUC (L )ù
2ú
ú
û êë
û
(
)
Say we use recurrent sampling with
x1 = 0
x2 = 0.9D ; D = 10
36
x1 = 0
x
x
x
0
10
20
x
x
x
-1 0
9
19
x2 = 0.9D
xx
xx
xx
-1 0
9 10
19 20
37
Condition for Perfect Reconstruction is
ée jw1x1 e jw1x2 ù
ê
ú nonsingular
êe jw2 x1 e jw2 x2 ú
ë
û
é1
ù
1
= ê - j(0.9)(2 p )ú
ê1 e
ú
ë
û
Hence, the original signal can be recovered
from the sampling pattern given in the
previous slide.
38
Summary


We have seen that the well known
Shannon reconstruction theorem can be
extended in several directions; e.g.

Multidimensional signals

Sampling on a lattice

Recurrent sampling
Given specific frequency domain
distributions, these can be matched to
appropriate sampling patterns.
39
Outline
1.
2.
3.
4.
5.
6.
7.
8.
9.
One Dimensional Sampling
Multidimensional Sampling
Sampling and Reciprocal Lattices
Undersampled Signals
Filter Banks
Generalized Sampling Expansion (GSE)
Recurrent Sampling
Application: Video Compression at Source
Conclusions
40
Application: Video Compression
Source
Introduction to video cameras
 Instead of tape, digital cameras use 2D sensor
array (CCD or CMOS)
DVCD
controller
Image
Image
Processing
Processor
Processor
Pipeline
Display
( TV or LCD )
Memory
41
Image Sensor

A 2D array of sensors replaces the traditional
tape

Each sensor records a 'point' of the
continuous image

The whole array records the continuous
image at a particular time instant
42
2D Colours Sensor Array
Data transfer from array is sequential
and has a maximal rate of Q.
43
* Based on http://www.dpreview.com/learn/
Current Technology
Uniform 3D sampling

a sequence of identical frames equally spaced in
time
44
Video Bandwidth
w
x
depends on
spatial resolution
of the frames
x
depends on
the frame rate
The volume of ‘box’ depends on the capacity:
pixel rate = (frame rate) x (spatial resolution)
45
Hard Constraints
1. Data recording on sensor:
• Sensor array density
- for spatial resolution
• Sensor exposure time
- for frame rate
R


 pixels 
 frame 
F


 frames 
 sec. 
2. Data reading from sensor:
• Data readout time
- for pixel rate
Q


 pixels 
 sec. 
46
BUT...
Generally
Need:
s.t.
Q << RF
R1< RF1 < F
R1F1 = Q
Compromise:

spatial resolution

temporal resolution
R1< R
F1 < F
47
Actual Capacity (Data Readout)
wx
volume determined
by Q  R1  F1
wy
wt
 R1
 F1
48
Observation
Most energy of typical video scene
is concentrated around the  w x ,w y 
plane and the w t axis.
wt
wx
49
The Spectrum of this Video Clip
uniform sampling
- no compromise
uniform sampling
- compromise
in frame rate
uniform sampling
- compromise
in spatial resolution
wt
wx
50
Recurrent Non-Uniform Sampling
 2 M 1  1  t
 2 M 2  1  t
t
t
frame type A
x
 2 L  1  x
frame type B
y
 2 N  1  y
52
What Does it Buy?
wx
wt
wy
53
Schematic Implementation
non-uniform data from the sensor
t
Filter
bank
t
t
uniform high def. video
'compression at the source'
54
Recurrent Non-Uniform Sampling
A special case of
Generalized Sampling Expansion Theorem
55
Sampling Pattern
The resulting sampling pattern is given by:

lx  2 M1 
 0 
  LAT (U )  LAT (U )      LAT (U )  

0
m

t
l 1 
   m2 M 2 1 


2L

2 ( L  M ) 1
 LAT (U )  x 
s 1
s
56
Frequency Domain
S
 UC(2U
2 ( L  M ) 1
r 1
where:
T
)  wr



w 


UC (2U T )  w   x  : w x 
, wt 

w
(
2
L

1
)

x
(
2
M

1
)

t
t


1


is the unit cell of the reciprocal lattice


 (2 L  1)x
T
LAT (2U )  

0





2
n : n  Z 



(2M 1  1)t 

0
57
Reciprocal Lattice
wt
2M  1 

( 2 M 1  1) t
Unit cell

(2 M 1  1) t

wx


x
x

(2 L  1)x

2M  1 

( 2 M 1  1) t
58
Apply the GSE Theorem
 1 
 

2

   ( x)
H (w )




 2( L  M )1 
where: H (w) is uniquely defined by H1…H2(…)
 is a set of 2(L+M)+1 constraints
If
H 1 exists,
we can find the reconstruction function
(w , x)  H (w ) ( x)
1
59
Reconstruction Scheme
I(x,t)
H1
1
H2L+1
2L+1
H2(L+M)+1
2(L+M)+1
H r .s  e
jw
T
r

Î(x,t)
 xr 
 
 tr 
The sub-sampled frequency of each filter H is:
Nyquist frequency
2(L  M)  1
60
Reconstruction functions



 sin 

sin  ( x  (r - 1)x)  
x
  (2M  1)t 
 r ( x, t )  (2 L  1)xt 
 ( x  (r  1)x)
t
for r = 2,3,…,2L+1

 

 sin  (t  (r - 2L - 1)t) 
sin 
(2L  1)x   t


 r ( x, t )  (2M  1)xt
x
 (t  (r  2 L  1)t )
for r = 2(L+1),…,2(L+M)+1
Multidimensional ‘sinc like’ functions
61
Demo
Full resolution
sequence
Temporal
decimation
Reconstructed
sequence
Spacial
decimation
62
Outline
1.
2.
3.
4.
5.
6.
7.
8.
9.
One Dimensional Sampling
Multidimensional Sampling
Sampling and Reciprocal Lattices
Undersampled Signals
Filter Banks
Generalized Sampling Expansion (GSE)
Recurrent Sampling
Application: Video Compression at Source
Conclusions
63
Conclusions

Nonuniform sampling of scalar signals

Nonuniform sampling of multidimensional
signals

Generalized sampling expansion

Application to video compression

A remaining problem is that of joint design of
sampling schemes and quantization strategies
to minimize error for a given bit rate
64
References
 One Dimensional Sampling
 A. Feuer and G.C. Goodwin, Sampling in Digital Signal Processing and Control.
Birkhäuser, 1996.
 R.J. Marks II, Ed., Advanced Topics in Shannon Sampling and Interpolation Theory.
New Your: Springer-Verlag, 1993.
 Multidimensional Sampling
 W.K. Pratt, Digital Image Processing, 3rd ed: John Wiley & Sons, 2001.
 B.L. Evans, “Designing commutative cascades of multidimensional upsamplers and
downsamplers,” IEEE Signal Process Letters, Vol4, No.11, pp.313-316, 1997.
 Sampling and Reciprocal Lattices, Undersampled Signals
 A.Feuer, G.C. Goodwin, ‘Reconstruction of Multidimensional Bandlimited Signals for
Uniform and Generalized Samples,’ IEEE Transactions on Signal Processing, Vol.53,
No.11, 2005.
 A.K. Jain, Fundamentals of Digital Image Processing, Englewood Cliffs, NJ: PrenticeHall, 1989.
65
References
 Filter Banks
 Y.C. Eldar and A.V. Oppenheim, ‘Filterbank reconstruction of bandlimited signals
from nonuniform and generalized samples,’ IEEE Transactions on Signal Processing,
Vol.48, No.10, pp.2864-2875, 2000.
 P.P. Vaidyanathan, Multirate Systems and Filter Banks. Englewood Cliffs, NJ:
Prentice-Hall, 1993.
 H. Bölceskei, F. Hlawatsch and H.G. Feichtinger, ‘Frame-theoretic analysis of
oversampled filter banks,’ IEEE Transactions on Signal Processing, Vol.46, No.12,
pp.3256-3268, 1998.
 M. Vetterli and J. Kovaĉević, Wavelets and Subband Coding, Englewood Cliffs, NJ:
Prentice Hall, 1995.
66
References
 Generalized Sampling Expansions, Recurrent Sampling
 A. Papoulis, ‘Generalized sampling expansion,’ IEEE Transaction on Circuits and
Systems, Vol.CAS-24, No.11, pp.652-654, 1977.
 A. Feuer, ‘On the necessity of Papoulis result for multidimensional (GSE),’ IEEE
Signal Processing Letters, Vol.11, No.4, pp.420-422, 2004.
 K.F.Cheung, ‘A multidimensional extension of Papoulis’ generalized sampling
expansion with application in minimum density sampling,’ in Advanced Topics in
Shannon Sampling and Interpolation Theory, R.J. Marks II. Ed., New York: SpringerVerlag, pp.86-119, 1993.
 Video Compression at Source
 E. Shechtman, Y. Caspi and M. Irani, ‘Increasing space-time resolution in video’,
European Conference on Computer Vision (ECCV), 2002.
 N. Maor, A. Feuer and G.C. Goodwin, ‘Compression at the source of digital video,’
To appear EURASIP Journal on Applied Signal Processing.
67
Lecture 1
Sampling of Signals
by
Graham C. Goodwin
University of Newcastle
Australia
Lecture 1
Presented at the “Zaborszky Distinguished Lecture Series”
December 3rd, 4th and 5th, 2007
68