1/C Kelly Nelan
A Study of Shannon’s Sampling Theory
Final Capstone Paper
SM472
26APR2007
In this paper I will explain the components that make up Sampling Theory, the
man behind the theory, and the different types of sampling that make communication
possible. Sampling Theory is known as one of the most important mathematical
techniques used in communication engineering and information theory. It has become
very useful in physics and engineering, such as in signal analysis, image processing,
radar, sonar, acoustics, optics, holography, meteorology, oceanography, crystallography,
physical chemistry, medical imaging and many more. Sampling Theory can be useful in
any manner in which functions need to be reconstructed from sampled data. [1]
Historical Information on C.E. Shannon
The mathematical minds behind sampling theory can be traced back to many great
mathematicians, such as Poisson, E. Borel, Hadamard, and E.T. Whittaker. Although it is
unknown who originally discovered Sampling Theory, we do know that the results from
these older mathematicians were rediscovered by C. E. Shannon in 1940 by using
information theory and communication engineering. [1]
(http://www.windoweb.it/edpstory_new/foto_storia_inventori/ep_shannon_f1.gif)
Claude Elwood Shannon is commonly known as the “founding father of the
electronic communications age.” He was born in Gaylord, Michigan on April 30, 1916.
During the first sixteen years of his life, Shannon was always mechanically inclined,
building model planes, a radio-controlled model boat and a telegraphy system constructed
of two barbed wires around a nearby pasture. He also earned money fixing radios for a
local department store. [3]
In 1932 he entered the University of Michigan and earned the degrees of Bachelor
of Science in Electrical Engineering and Bachelor of Science in Mathematics. His
interest in these two subjects continued throughout the remainder of his life. After
graduation he got the position of research assistant in the Department of Electrical
Engineering at the Massachusetts Institute of Technology. His work at the time
specialized in the Bush differential analyzer, the most advanced calculating machine at
that time. It solved by analog means differential equations of up to the sixth degree. The
machine was so large and complex that at some times, up to four assistants would be
needed to crank in the functions. [3]
Shannon furthered his career by joining AT&T Bell Telephones in New Jersey in
1941 and stayed there until 1972 as a research mathematician. It is here that he made his
many advances in information technology and communication systems. Shannon’s
Information Theory showed that the basic elements of any general communications
system include [4]:
1. a transmitting device that tranforms the information or message into a suitable
form for transmission over a medium
2. the medium over which the message is transmitted
3. a receiving device which decodes the message back into some form of the
original signal
4. the destination or recipient of the message
5. a source of noise from either interference or distortion
Shannon also noted many important quantities yielded from the generalized
communication system, including [4]:
1. the rate at which information is produced at the source
2. the capacity of the channel for handling information
3. the average amount of information in a message
It is from this information that Shannon developed his sampling theory and applied it to
information theory. Shannon was looking at how the quantities stated above could
transform the original message, and he explored in what ways the message would be
impossible to reconstruct. Through this he found out how many samples of the original
signal he would be able to send to a receiver and still reconstruct the original message, as
is stated in the sampling theorem.
The Sampling Theory states that
If a function of time is limited to the band from 0 to W cycles per second, it is completely
determined by giving its ordinates at a series of discrete points spaced 1/2W seconds
apart in the manner indicated by the following result: If f(t) has no frequencies over W
cycles per second, then
∞
 n  sin π (2Wt − n )
[1]
f (t ) = ∑ f 

n = −∞  2W  π (2Wt − n ) )
In other words, this means that it is possible to reconstruct a signal from samples if the
signal is band-limited and the sampling frequency is greater than twice the frequency of
the original signal. Note that one important principle for this theorem is to note that all
the information contained in a signal is also contained in the sampled values taken at
equidistantly spaced instances. The minimum rate at which the signal needs to be
sampled in order to reconstruct the simple depends upon the knowledge of the frequency
bound. This minimum rate is also known as the Nyquist rate, named after H. Nyquist,
who was the first person to point out the importance of having a minimum sampling rate
in connection with telegraphy. [1]
Terminology
Signal – a continuous (analog) signal, mathematically written as a function of time,
distinguishable from a discrete (digital) signal, which is mathematically written as a
sequence of numbers.
Ex: May represent the voltage difference at time t between two points in an
electrical circuit
A communication system consists mainly of a transmitter, communication
channel (medium), and a receiver. The purpose of this system is to send a message,
which may consist of written or spoken words, pictures, sound, or etc, from the
transmitter to the receiver. The transmitter changes the message into a signal that can be
sent through the medium (ex: wires, atmosphere) to the receiver, which will then turn the
signal back into the original message. [1] The following picture is an example of a
general communication system:
(http://alumni.media.mit.edu/~carsonr/phd_proposal/figures/shannon-schematic.jpg)
In the process of transmitting a signal, the signal may aquire some alterations,
such as static, sound, or distortions in shape, by the time it reaches the receiver. These
alterations fall under two categories: distortion and noise. Distortion is a fixed
operation applied to a signal, and therefore, in theory, can be undone by applying the
correct inverse operation. Noise involves statistical and unpredictable perturbation which
sometimes cannot be corrected. [1]
A distortionless transmission of a signal to a receiver means that the exact shape
of the input signal is reproduced at the output, regardless of whether there is a change in
amplitude of a time-delay. If the input signal is denoted by f(t) and the output by g(t), a
distortionless transmission can be represented by
g (t ) = Lf (t ) = Af (t − t 0 )
where L is a linear, time-invariant operator. [1]
Taking the Fourier transform of both sides of this equation, we get
G(ω ) = H (ω )F (ω )
where F, G are the Fourier transforms of f, g respectively and H (ω ) = Ae it0ω . Here
H (ω ) is called the system transfer function, or the system function. It’s inverse Fourier
transform h(t) is called the impulse response to the system. [1]
Processing a signal means that we are operating on it in some fashion, in order to
change its shape, configuration and properties or to extract some useful information.
Usually this operation is required to be invertible. Sometimes for practical and
economical reasons, only some data extracted from the signal are transmitted and are
used at the receiver to reconstruct the original signal. This is why we use sampling
theory. [1]
In electrical engineering a filter is a circuit or system that has some frequency
selective mechanism. Before signal is sampled it must be filtered. Theoretically, the
maximum frequency is half of the sampling frequency, but in practice we must use a
higher sampling rate due to non-ideal filters. The ideal filters usually come in four
categories: low pass, high pass, band pass or band stop. The system transfer functions of
these ideal filters are as follows:
Low pass filter
 Ae it0ω ,
H (ω ) = 
0,
for ω ≤ ω1
otherwise
Where ω1 is the cut-off frequency;
High pass filter
 Ae it0ω ,
H (ω ) = 
0,
for ω ≥ ω 2
otherwise
Band pass filter
 Ae it0ω ,
H (ω ) = 
0,
Band stop filter
0,
H (ω ) =  it ω
 Ae 0 ,
forω 2 ≤ ω ≤ ω1
otherwise
forω 2 ≤ ω ≤ ω1
otherwise
[1]
A function f is called band limited if its Fourier transform fˆ is 0 outside of a
finite interval [-L,L].
Periodic sampling is the process of representing a continuous signal with a
sequence of discrete data values. With regards to sampling, the main concern is to ensure
that the sampling is fast enough to ensure that the information content is preserved. [3]
A normalized sinc function is defined by
sinc(x)=
sin(πx)
πx
Proof
I will now attempt to prove a version of the Sampling Theorem from James S. Walker’s
Fast Fourier Transforms. His theorem states:
Suppose that f is band limited. If fˆ is 0 outside of [-L,L], then
∞
f ( x) =
∑
n = −∞
 n 
f   sinc (2 Lx − n) )
 2L 
Proof: Since fˆ is 0 outside of [-L,L], we can periodically extend it. We will use fˆP to
denote the periodic extension of fˆ with a period equal to 2L.
∞
∑ fˆ (u − 2nL)
fˆP (u ) =
n = −∞
Using Poisson summation, the Fourier series for fˆP is
fˆP (u ) =
∞
1 ˆ n 
∑ 2 L fˆ  2 L e
inπu / L
n = −∞
ˆ
Since fˆ = f ( − x )
fˆP (u ) =
∞
1
−n
∑ 2 L f  2 L e
inπu / L
(*)
n = −∞
We will now multiply the previous equation by the window function in Sampling Theory,
W(u), which satisfies
1,
W (u ) = 
0,
u ≤L
u >L
.
For this window, we have
fˆP (u )W (u ) = fˆ (u )W (u ) = fˆ (u ) .
Multiplying both sides of (*) by W(u), we get
∞
fˆ (u ) =
−n
1
∑ 2 L f  2 L W (u)e
inπu / L
n = −∞
Next we will multiply both sides by e i 2πux and integrate with respect to u from –L to L,
giving the following
L
∫
∞
∑
fˆ (u )e i 2πux du =
n = −∞
−L
 x+n 
L
i 2π 
u
−n 1
 2L 
f
W
(
u
)
e
du

∫
 2L  2L −L
L
Let S ( x) =
1
W (u )e i 2πux du , and substitute this into the previous equation.
2 L −∫L
L
∫
∞
∑
fˆ (u )e i 2πux du =
n = −∞
−L
n 
−n 
f
S  x +

2L 
 2L  
Since fˆ is 0 outside of [-L,L], we have
∞
L
∫
∫ fˆ (u )e
fˆ (u )e i 2πux du =
−L
i 2πux
du = f ( x)
−∞
∞
∫ fˆ ( y)e
by Fourier inversion, which says f ( x) =
2πixy
dy . From this we get
−∞
∞
f ( x) =
∑
n = −∞
n 
−n 
f
S  x +

2L 
 2L  
Subsitute –n in place of n.
∞
f ( x) =
∑
n = −∞
n 
 n  
f  S  x −

2L 
 2L  
This describes a very general sampling theory, where the sampled values are
∞
  n 
 f  
  2 L  n = −∞
Putting the window function back in S, we get
L
1
S ( x) =
e i 2πux du = sinc (2 Lx)
∫
2L −L
This brings us back to our original equation in the theorem.
∞
f ( x) =
∑
n = −∞
 n 
f   sinc (2 Lx − n )
 2L 
q.e.d. [7]
Errors and Aliasing
Several different errors exist and have been intensively studied in Sampling Theory.
Some of these errors are as follows:
1) The truncation error is the error that results when only a finite number N samples
are used instead of the infinitely many samples needed for the signal
reconstruction. The object function cannot be reconstructed exactly because there
is insufficient information. [2] The truncation error (Tn f )(t ) or Tn (t ) , for short,
can be controlled by imposing some extra conditions on f besides being bandlimited. The first estimated value for the truncation error is credited to A. Jerr,
Tsybakov, and V. Iakovlev. This estimation can be written as [1]:
TN (t ) ≤
 πt 
E sin  
π
 ∆t 
T∆t
,
T 2 −t2
2
(
)
T >0
Where − T ≤ t ≤ T , 0 < ∆t , (1 / σ ) , (2 σ is our bandwideth) and E
is the total energy of the signal which is given by
σ
E=
∫σ F (ω )
2
dω
−
2) The amplitude error, Ae f , arises if the exact sampled values f (t k ) are not
~
accurately known, but we know some approximation of the values, such as f (t k ) ,
differing from f (t k ) , by not more than e, are known [1]:
∞
( Ae f )(t ) =
∑ [ f (t
k
k = −∞
sin σ (t − t k )
~
) − f (t k )
σ (t − t k )
]
3) The time-jitter error, J e f , is caused by sampling at instants ~
tk = t k + γ k , which
differ from the Nyquist sampling instants [1]:
∞
( J e f )(t ) =
∑ [ f (t
k = −∞
k
sin σ (t − t k )
) − f (~
tk ) ]
σ (t − t k )
4) The aliasing error, Rσ f , as stated earlier, is the result when the band-limitedness
conditions are violated. Therefore an aliasing error occurs during undersampling. [1]
∞
( Rσ f )(t ) = f (t ) −
∑
k = −∞
 kπ  sin(σt − kπ )
f 
 σ  (σt − kπ )
All four of these error types can be combined into the form [1]:
N
( Ef )(t ) = f (t ) −
∑
k =− N
~
sin(σt − kπ )
f (t k + γ k )
(σt − kπ )
Aliasing
In discrete-time signal samples, there is a frequency ambiguity that does not exist in the
continuous signal world. The following is a graph that represents frequency ambiguity,
taken from Richard Lyon’s Understanding Digital Signal Processing.:
A is a discrete-time sequence of values and B shows two difference sine waves that
could pass through the discrete values.
Suppose you were given the following sequence of values to represent instantaneous
values of a time-domain sine wave:
x(0)=0
x(1)=0.866
x(2)=0.866
x(3)=0
x(4)=-0.866
x(5)=-0.866
x(6)=0
Next you are asked to draw the sine wave from the given values. You would most likely
draw the solid line in (b) whereas someone else may draw the lighter line. The problem
here is that we cannot unambiguously determine the frequency of the sine wave from just
the sampled values. [8]
Sampling Low-Pass Signals
Consider sampling the continuous real signal shown in the following picture, also
from Lyons’ Understanding Digital Signal Processing:
A is the original continuous signal spectrum; B is the spectral replications of the
sampled signal. C also shows replication, but at a sampling frequency less than twice the
original frequency. The spectral amplitude is zero above +B Hz and below –B Hz;
therefore, the signal is band-limited. When the signal is sampled at a rate of f s
samples/s, we can see how the signal is replicated. If we perform a frequency translation
operation or induce a change in sampling rate through decimation or interpolation, the
spectral replications will shift up or down and could cause problems. In figure (c) the
sampling frequency is lowered to f s = 1.5 B Hz, resulting in under-sampling. The
spectral replications are now overlapping causing aliasing errors. For this case, we can
use low-pass filters. In practice one would use a low-pass anti-aliasing filter to attenuate
any unwanted signal energy above +B and below –B Hz. [8]
Non-Uniform Sampling
In Shannon’s Sampling Theory the sampled points are uniformly spaced, and the
sampling procedure associated with it is known as uniform sampling. In practice, nonuniform sampling is often used and can sometimes even give better results, such as when
a signal is varying very rapidly. This non-uniform sampling for reconstructing signals is
used in many areas, such as Radio Astronomy, Computed Tomography, Magnetic
Resonance Imaging, optical and electronic imaging system, as well as many other areas.
Two main techniques are used to reconstruct signals from uniformly and nonuniformly collected data. One technique is based on using the values of the signal and
some of its derivatives, at predetermined instants. The other technique is based on using
instants, where the signal assumes predetermined values, such as zero.
A signal that is band-limited to the interval [a,b] is uniquely determined by its
zeros up to an exponential factor that depends on the spectral end points. If a signal is
band-limited to an interval symmetric about the origin, say to [-a,a], then it is completely
determined by its zeros – up to a constant – and can be reconstructed from these zeros via
the formula
∞

z 
f ( z ) = f (0)∏ 1 − 
zn 
n =1 
where z is a complex number and { z n } are the zeros of f , which are not necessarily
uniformly distributed. This equation assumes that f (0) ≠ 0 .
In order to reconstruct a band-limited signal from its zeros, one needs to know all
of them (real and complex). The real zeros are also called the zero-crossings of the
signal. If they are all detected, they can be used to reconstruct the signal. These signal
types are called real-zero signals, also commonly known as RZ signals. The process of
reconstructing the RZ signal from its zero-crossings is known as real-zero interpolation
(RZI).
Reconstructing signals from their zero crossings has many important practical
applications, especially in Image Processing. For example, when an image is blurred or
distorted, but the zero crossings are still preserved, it is possible to recover the image
from the distorted version. Generally, if a band-limited signal f undergoes a nonlinear
transformation T, such as g (t ) = T { f (t )} , it may be possible to recover f from the
knowledge of the output signal g if T is a zero-crossing preserving transformation. [1]
Conclusion
Sampling Theory is very important in today’s busy world where there are limited
frequencies available for our use. This is why we study the minimum amount of
information that we can send through a medium, while stile being able to receive the
appropriate message. Mathematicians and engineers continue to search for new methods
of sending samples of a message, so as not to crowd our current frequency use, which is
why we continue to study Sampling Theory today.
References
1. Zayed, Ahmed I., Advances in Shannon’s Sampling Theory, CRC Press, 1993.
2. Collins, Graham P. , “Claude E. Shannon: Founder of Information Theory”,
http://www.sciam.com/print_version.cfm?articleID=000745C4-9E66-1DA5815A809EC5880000.
3. Sloane, N.J.A, Wyner, A.D., “Biography of Claude Elwood Shannon”,
4. http://www.research.att.com/~njas/doc/shannonbio.html.
5. http://www2.egr.uh.edu/~glover/applets/Sampling/Sampling.html
6. http://members.aol.com/ajaynejr/nyquist.htm
7. Walker, James S., Fast Fourier Transforms, CRC Press, 1996.
8. Lyons, Richard G., Understanding Digital Signal Processing, Addison Wesley
Longman, Inc, 1997.