Some Historical Remarks
on
Sampling Theorem
Radomir S. Stanković, Jaakko T. Astola, Mark G. Karpovsky
Dept. of Computer Science, Faculty of Electronics, Niš, Serbia
Tampere Int. Center for Signal Processing, Tampere
University of Technology, Tampere, Finland
Dept. of Electrical and Computer Engineering,
Boston University, Boston, USA
Motivation – Why to It Discuss Further?
1. Newton’s law of force
2. The law of universal gravitation
4. Maxwell’s equations
3. The second law of thermodynamics
5. The Navier-Stokes equation
Newton’s second law for fluids.
6. The Stefan-Boltzmann law
7. Relativity - Einstein’s formula
8. The Lorentz transformation
9. The Schrodinger wave equation
10. Shannon’s sampling theorem
IEEE Potentials, December 96-January 97, 39-40
Yet Another Review?
We hope NOT due to analysis of previous Reviews
Butzer, P.L., Stens. L., "Sampling theory for not necessarily band-limited
functions – A historical overview", SIAM Review, Vol. 34, No. 1, 1992, 40-53.
Higgins, J.R., ”Five short stories about the cardinal series”, Bull. Amer.
Math. Soc., Vol. 12, No. 1, 1985, 54-89.
Jerri, A., "The Shannon sampling theorem – Its various extensions and
applications, A tutorial review", Proc. IEEE, Vol. 65, No. 11, 1977, 1565-1596.
Luke, H.D., "The origins of the sampling theory",
IEEE Communications Magazine, Vol. 37, No. 4, 1999, 106-108.
Reviews on The Sampling Theorem
Meijering, E., "A chronology of interpolation: From ancient astronomy to
modern signal and image processing",
Proc. IEEE, Vol. 90, No. 3, 2002, 319-342.
Sangwine, S.J., Whitehouse, J.E., "The sampling theorem – a tutorial",
Proc. of the IEE Colloquium on Mathematical Aspects of
Digital Signal Processing, February 10, 1994, 1/1 - 1/6.
Unser, M., "Sampling-50 years after Shannon",
Proc. IEEE, Vol. 88, No. 4, 2000, 569-587.
Vaidyanathan, P.P., "Generalizations of the sampling theorem - seven
decades after Nyquist",
IEEE Trans. Circuits and Systems, Vol. 48, No. 9, 2001, 1094-1109.
Novelity in the Paper and Outline
The novelity in the paper resides in
Group-theoretic approach to Sampling Theorem
Real line
Dyadic group
Localy Compact Abelian Groups
Finte groups, Abelian, non-Abelian
Derivation of the Sampling theorem on dyadic groups from
the Sampling theorem on LCA groups
Disucssion of Sampling theorem on finite groups
Emphasis on unifrom derivation and unified discussions
Contributors
The history of technology is a continual succession of ideas,
and not just a discrete list of major accomplishments.
Donald R. Mack
Practitioners
Theoreticians not noticing or
emphasizing applications
in Signal processing
Theoreticians working
purposely for applications
Luke, H.D.
Subject of The Sampling Theorem
1. Sampling
A band limited function is completely
determined by its samples.
2. Reconstructing
Recovering function values from samples
for a function fulfilling the sampling conditions.
Prehistory - Lagrange
In 1765, J.L. Lagrange has shown that a periodic function,
fulfilling conditions to be representable by a trigonometric
series, can be expressed as a linear combination of a constant
and n sine and n cosine terms.
The knowledge of function values
at 2n+1 equidistant points
within a period is sufficient to represent
uniquely a periodic function with
assumed properties.
Joseph-Louis, Comte de Lagrange
This statement can be viewed as
a sampling theorem for bandlimited periodic functions.
Prehistory - Cauchy
In 1841, A.L. Cauchy has shown an interpolation formula
for an M-bandlimited function, i.e., a function representable by
a complex Fourier series with no more than |n| < M coefficients.
f ( x) =
2πinx
c
e
∑ n , N = 2n + 1
| n|≤ M
N −1
1
f ( x) = sin(πxn) ∑
m=0 N
(−1) m
m
f 
 N  sin π  x − m 


N

A. Cauchy, “Mémoire sur diverses formules d’analyze,”
Comptes Rendus des Séances de l’Académie des
Sciences, Vol. 12, No. 6, 1841, 283–298.
Augustin Louis Cauchy
H. S. Black, Modulation Theory, New York, Van Nostrand, 1953.
Prehistory - Borel
Sampling part of
the Sampling Theorem
Borel, E., “Sur l’ interpolation,”
Comptes Rendus des Séances
de l’Académie des Sciences,
Vol. 124, No. 13, 1897, 673–676.
E. Borel, “Mémoire sur les séries
divergentes,” Annales Scientifiques
de l’École Normale Supérieure,
Ser. 2, Vol. 16, 1899, 9–131.
Félix Édouard Justin Émile Borel
sin πz
∞
ak
∑
π k = −∞
z−k
solves the interpolation problem
G( z) =
(−1) k
G (n) = an , n ∈ Z , if
∑a
k ≠0
k
/k <∞
Harry Nyquist
Distortionless transmission of
telegraphic (digital) signals
Nyquist, H., "Certain factors affecting telegraph speed",
Bell Syst. Tech. J., Vol. 3, Apr. 1924, 324-346.
Nyquist, H., "Certain topics in telegraph transmission theory",
Trans. AIEEE, Vol. 47, Feb. 1928, 617-664.
This work was presented at the Winter Convention of
the A. I. E. E., New York, NY, February 13–17, 1928.
Nyquist and the Sampling Theorem
Conclusions
Nyquist considered a different problem,
the sampling theorem is a statment dual to that of Nyquist.
There is a sense to consider the sampling theorem
for discrete signals.
An interpretation of the Sampling Theorem
Knowledge of data at the minimum subarea of the domain
of definition sufficient to describe a function completely
under some conditions provided.
Edmund Taylor Whittaker
The problem of determining a function passing through the points
(a+kw,f(a+kw)), where k ∈ Z, w – complex number.
Smoothest possible interpolation without singularities
and rapid oscillations for given tabular values of f(x).
∞
C ( x) = ∑ f (a + kw)
−∞
sin
π
π
w
w
Sinc function
( x − a − kw)
( x − a − kw)
Whittaker, E.T., "On the functions expansions
of the interpolation theory", Proc. Roy. Soc.
Edinburgh, Vol. 35, 1915, 181-194.
E.T. Whittaker
John Macnaughten Whittaker
Whittaker, J.M., "The "Fourier" theory of the cardinal function",
Proc. Edinburgh Math. Soc., 2, 1927-1929, 169-176.
Whittaker, J.M., "On the cardinal function of interpolation theory",
Proc. Edinburgh Math. Soc., Vol. 1, 1929, 412-46.
Weak version of the sampling theorem,
not included in
Whitaker, J.M., Interpolation Function Theory,
Cambridge Tracts in Mathematics and Mathematical Physics,
No. 33, Cambridge University Press, Chapt. IV, 1935.
J.M. Whittaker
Kinnosuke Ogura
Sangaku
Ogura, K., "On a certain transcedental
integral function in the theory of interpolation",
Tohoky Math. J., 17, 1920, 64-72.
Ogura, K., "On some central difference
formulas of interpolation",
Tohoky Math. J., Vol. 17, 1920, 232-241.
The Sampling Theorem and a Proof of It
Ogura was the first who stated
the sampling theorem and traced
the way of a rigorous proof of it.
For a proof Ogura recommended to
use the results from
Lindelof, E., Les calcul des Residues,
Gautier-Villard,
Paris, 1905.
Ogura
Lindelof
Yosan
Wasan
E. Lindelof
Владимир Александрович Котельников
Any function F(t) which consists of frequencies
from 0 to f1 periods per second
may be represented by the following series

k 

sin w1  t −
∞
 2 f1 
F (T ) = ∑ Dk
k
−∞
t−
2 f1
where k-integer, w1=2πf1,
Dk – constant which depends on F(t).
Kotelnikov, V.A., "On the carrying capacity of
the "ether" and wire in telecommunications",
Material for the First All-Union Conference on Questions
of Communications, Izd. Red.Upr. Svyazi RKKA,
Moscow, 1933.
A.V. Kotelnikov
Any function F(t), which consists of frequencies from 0 to f1,
can be transmitted continuously with an arbitrary accuracy,
by means of numbers sent at intervals of 1/2f1 seconds.
Indeed, by measuring of the value F(t) at t=n/2f1 (n integer),
we get
F(n/2f1)=Dnw1.
Since all terms of the series (2) for this value of t tend to zero,
except the term for k=n, as it can be easily established
by calculation of the indefinite point, equals Dnw1.
In this way, after each 1/2f1 we can determine the next Dk.
When these Dk are transmitted in a row at each 1/2f1 sec.,
we can from (2) reconstruct F(t) termwise to any degree of accuracy.
"Электричество",
Энергетический институт Академии Наук СССР
Rejected the paper with comment ”far from engineering needs”.
Claude Elwood Shannon
Theorem 13, page 34
Let f(t) contain no frequencies over W. Then,
∞
f (t ) = ∑ X n
−∞
sin π (2Wt − n)
,
π (2Wt − n)
 n 
.
where X n = f 
 2W 
Shannon
Denis Gabor
Theorem 13 has been given previously in other forms
by mathematicians (Whittaker)
but in spite of its evident importance seems
not to have appeared explicitly in
the literature of communication theory.
Nyquist, (1924, Bennett, 1941) however, and more
recently Gabor, (1946) have pointed out that
approximately 2TW numbers are sufficient,
basing their arguments on a Fourier series
expansion of the function over the time interval T.
Gabor
Herbert P. Raabe
Raabe, H., "Untersuchungen an der
Wechselzeitigen Mehrfachübertragung
(Multiplexübertragung)",
Elektrische Nachrichtentechnik,
Vol. 16, 1939, 213-228.
Benett, W.R., ”Time division multiplex systems”,
Bell. Syst. Tech. J., Vol. 20, 1941, 199.
Royal Techical Highschool in Berlin around 1890
where H. Raabe graduated in 1936 and received
Dr.-Ing. Degree, summa cum laude, in 1939.
Karl Kupfmuller
"The time law allows comparison of the capacity of
each transfer method with various known methods.
On the other hand it indicates the limits that
the development of technology must stay within.
One interesting question for example is where
the lower limit for k lies.
The answer is acquired by at least one power change being
needed to achieve one signal. So the frequency range
must be at least so wide that the settling time becomes
less than the duration of a signal, and from
this comes k=1/2. So we can never get below this value,
no matter how technology develops."
Karl Kupfmuller
Isao Someya
Someya, I., Hakei Denso (Waveform Transmission),
Shykyo,Tokyo, 1949.
The Theorem in Western Litearature
Wetson, J.D., "A note on the theory
of communication",
Philos. Mag., 303, Vol. 40, 1949,
449-453.
Wetson, J.D., "The cardinal series
in Hilbert space",
Proc. Cambridgh. Philos. Soc., Vol.
45, 1949, 335-341.
Sampling Theorem for
Duration-Limited Signals
Bandlimited functions f ( x), x ∈ R
Sf(w) = 0, w > w0
Duration (time) limited functions
f ( x), x ∈ R
f ( x) = 0, x ∉ [a, b]
cannot be band-limited
Functions of bounded variation
f(x), x∈R, f(x) ∈R x ∈ [a, b] sup ∑ f ( xi +1 ) − f ( xi ) - total variation
P
i
supremum over all partitions P = {x0,...,xn} of [a,b]
Bounded variation, if total variation is finite
Charles-Jean Baron de la Vallée Poussin
De la Valléee Poussin, Ch.-J., "Sur la convergence
des formulaes d'interpolation entre ordonees équidistantes",
Bull. Cl. Sci. Acad. Roy. Belg., 4, 1908, 319-410.
Interpolation formula for time-limited functions
Fm ( x) =
αk =
∑
α
k ∈[ a ,b )
sin m( x − α k )
f (α k )
m( x − α k )
kπ
, k ∈ Z = {0,±1,±2,...}, m = n or
m
C.-J. de la Vallée Poussin
m = n + 1 / 2, n ∈ N = {1,2,3,...}
A generalization of the Lagrange interpolation
formula for infinite number of nodes
A counterpart of the Riemman localization principle
or Fourier integrals in the case of Fm.
Extensions by P.L. Butzer & P.L. Stens
Under the additional condition
f(b) = 0 besides f(x) = 0, for x ≠ [a, b], the interpolation
function Fm can be viewed as a discrete version of
the Dirichlet convolution integral, a particular form of the
Fourier inversion integral, and the behaviour of Fm for m → ∞
is similar to that of the Fourier inversion integral for f
Maria Theis
For the convergence of Fm(x) to f(x),
besides continuity of f(x) on [a,b],
f(x) should be of bounded variation.
For convergence of Fm(x)
for any continuous function f(x),
Theis used the kernel function
φ ( x) = (sin πx / πx) 2
- a counterpart of Féjer method
in summation of Fourier series
Theis, M., Über eine Interpolations
formel von de la Vallée Poussin",
Math. Z., 3, 1919, 93-113.
Lipót Fejér
Functions of Bounded Variation
Not necessarily vanishing outside a finite interval [a,b]
Consider a function f that is Riemann
integrable over any finite interval of R and f(x)/x is
of bounded variation in (N, ∞) and (-∞,N) for some
N > 0. If f is continuous at x0∈R and of bounded
variation in a neighborhood of x0, then
f ( x0 ) = lim
m →∞
∞
∑
k = −∞
f (α k )
sin m( x0 − α k )
m( x0 − α k )
Whittaker, J.M., "The "Fourier" theory of the cardinal function",
Proc. Edinburgh Math. Soc., 2, 1927-1929, 169-176.
Sampling of non-band-limited functions
Weiss, P., "An estimate of the error arising from misapplication of the sampling theorem",
Notices Amer. Math. Soc., 10,1963, 351.
Brown, J.L., "On the error in reconstructing a non-limited function by means of
the bandapass sampling theorem", J. Math. Annal. Appl., 18, 1967, 75-84, Erratum,
same journal 21,1968, 699.
Butzer, P.L., Splettstosser, E., "A sampling theorem for duration limited functions
with error estimates", Information and Control, Vol. 34, 1977, 55-65.
Butzer, P.L., Splettstosser, E., "Sampling principle for duration
limited signals and dyadic Walsh analysis",
Information Science, Vol. 14, 1978, 93-106.
Butzer, P.L., Splettstosser, W.,
Index of Papers on Signal Theory, 1972-1989,
Lehrstuhl A fur Mathematik, Aachen University of Technology,
Aachen, Germany, 1990.
Locally Compact Abelian Groups
Notation
G – additive locally compact Abelian group
Γ – dual group of G
H – discrete subgroup of G with the discrete annihilator
Λ = {w | χ ( y, w) = 1, ∀y ∈ H }
Baire measurable subset Ω of Γ
which contains a single element
from each coset of Λ
Integers Dk or Xn in
Kluvanek and Shannon notation
Ω ∩ ( w + Λ ) contains a single point
for each w ∈ Γ
René-Louis Baire
Sampling Theorem on LCA Groups
If f ∈ L2(G) and its Fourier transform Sf (w) =0 for almost all w∉Ω,
then f is almost everywhere equal to a continuous function, and if f is
a continuous function,
f ( x) =
∑ f ( y)φ ( x − y)
y∈H
(1)
where this series converge both uniformly on G and in the norm in
L2(G). Further,
f
2
=
∑
f ( y)
2
y∈H
The function φ in (1) is defined as
φ ( x) = ∫ χ ( x, w)dmΓ ( w),
Ω
where mΓ is the Haar measure on G and χ (x,w) are group characters of G.
Igor Kluvánek
If G = Γ = (−∞, ∞), Ω = (−α , α ) and ,
consequently, H = {...,−2h,−h,0, h,2h,...},
with hα = π , we get φ ( x) = (sin α x) /(α x).
If f ∈ L2 (−∞, ∞) and S f ( w) = 0 for w > α
we get the sampling theorem on R.
Kluvánek, I., "Sampling theorem in
abstract harmonic analysis",
Mat. Fiz. Časopis Sloven. Akad.,
Vied. 15, 1965, 43-48.
Igor Kluvánek
Walsh-Fourier Analysis
x ∈ R+ = [0, ∞ )
x=
∞
∑x 2
i =− N ( x )
i
−i
, xi ∈ {0,1}, N ( x) ∈ Z = {0,±1,±2,...}
{
D+ = x ∈ R+ | x = p / 2 q , p ∈ P = {0,1,2,...}, q ∈ Z
}
dyadic rational numbers
x ∈ D+ select finite expansion
If
x ∉ D+ unique representation
ψ ( y, x), x, y ∈ R+
- generalized Walsh functions
y - sequency
r
J ( x, r ) = ∫ψ ( x, s )ds - Walsh-Dirichlet kernel
0
r=2
n
2 n , 0 ≤ x < 2 − n ,
J ( x, 2 ) = 
 0, otherwise
n
[
)
1, x ∈ 2 − n s, 2 − n ( s + 1) ,
J (1,2 x ⊕ s ) = 
0, otherwise
n
Dyadic Sampling Theorem
If f and its Walsh-Fourier spectrum Sf(w) belong to L1(R+)
f continuous on R+\D+
f continuous from the right on D+
Sf (w) = 0 for w ≥ 2n, n ∈ Z, Z – the set of integers, i.e.,
f is sequency limited, then
∞
 s 
f ( x) = ∑ f  n J (1,2 n x ⊕ s ), x ∈ R+
2 
s =0
Pichler, F.R., ”Sampling theorem with respect to WalshFourier analaysis”, Appendix B in Reports Walsh Functions
and Linear System Theory, Elec. Eng., Dept., Univ.
of Maryland, College Park, May 1970
F. Pichler
S. Kak
P.L. Butzer
Dyadic and LCA Groups
Walsh-Fourier analysis
domain group the additve group G’ of the dyadic field
the set of generalized Walsh functions Ψ(λ,x)
dual group
A discrete subgroup H = s/2n, s = 0,1,..., n ∈ Z
The annihilator Λ for H is isomorphic to the sequences
Λ = ..., λ− k ,0,...,0,..., λi ∈ {0,1}
For every x ∈ H
∑λ
n
Ω=[0,2n)
x = 0 and Ψ(λ,x)=1
1− n n
Dyadic Sampling Theorem – A Proof
1, if x ∈ (0,2 − n ),
Define the reconstruction function as φ ( x) = 2 ∫ψ ( w, x)dw = 
0, otherwise
0
n
−n
2n
2
From Kluvánek theorem
f ( x) = ∫ S f ( w)φ ( w, x)dw
0
For every
x ∈ [2 − n k , 2 − n (k + 1))
2 − n ( k +1)
f ( x) = 2 n
∫ f (u )du,
k ∈ P ={1,2,...}
2 −n k
Thus, f is constant on all the intervals [2-nk, 2-n(k+1)), k ∈ P
∞
Then,
the characteristic function of [a,b)
Since for k∈ P
 k 
f ( x) = ∑ f  n ρ[2 − n k ,2 − n (k + 1))( x), x ∈ R+
k =0  2 
1, if x ∈ [a, b),
ρ[a, b)( x) = 
0, if x ∉ [a, b)
1, if x ∈ [2 − n k ,2 − n (k + 1)),
ρ[2 k ,2 (k + 1))( x) = 
= J (1,2 n x ⊕ k )
0, otherwise

−n
−n
Sampling in Finite Walsh Analysis
Important research topics
Characterization of signals by duration and bandwith
Band-limiting of time-limited signals and vice versa
In discrete analysis
Applications dealing with large sets of discrete and finite
signal samples (compression, multiplexing)
Determining minimum number of points to define
a function under some conditions provided
Le Dinh, C. T., Le, P., Goulet, R., "Sampling expansions in discrete and finite WalshFourier analysis", Proc. 1972 Symp. Applic. Walsh Functions,
Washington, D.C., USA, 265-271.
Le Dinh, C. T., Goulet, R., "Time-sequency-limited signals in finite Walsh transforms",
IEEE Trans. Inform. Theory, Vol. 20, No. 2, 1974, 274-276.
Finite Walsh Sampling Theorem
f(x), x ∈ Bn = {0,1,...,2n-1}
f(x) is M-sequency band-limited (MBL) if Sf(w)=0 for w > M
Walsh-Fourier kernel
M −1
R −1
w=0
r =0
d M ( x) = ∑ wal ( w, x) = M ∑ δ ( x ⊕ r ), = NM −1
⊕ componentwise addition modulo 2
An MBL function f(x) can be completely reconstructed from its M values
f ( x) = M
−1
M −1
∑ f ( p)d
p =0
M
( x ⊕ p)
M-sequency limited bandpass functions (MBP)
Sampling on Finite Abelian Groups
m −1
m −1
g = ∏ g i , g 0 ≤ g1 ≤ ... ≤ g m −1
G = × Gi
i =0
i =0
k
p = M = ∏ g m −i −1 , k ∈ {0,1,..., m − 1}
i =0
M −1
R −1
d M ( x) = ∑ χ ( w, x) = M ∑ δ (kM  x −1 ), R = g / M
*
k =0
w=0
° - group operation on G, χ – group character,
* - complex conjugate, δ(x)=δx,0 – Kronecker delta
f is MBL on G if Sf(w) = 0 for w ≥ M, then
f ( x) = M
−1
M −1
∑ f (r )d
r =0
−1
M
(r  x ) = M
−1
M −1
∑
r =0
 R −1

f ( p ) M ∑ δ (kM  (r  x −1 ) −1 
 k =0

Sampling on Finite non-Abelian Groups
m −1
m −1
G = × Gi
i =0
g = ∏ g i , g 0 ≤ g1 ≤ ... ≤ g m −1
i =0
Mi - number of irreducible unitary representations Rw,i of Gi
Γ – dual object of G,
|Γ | = K
k
Rw(x) xy ≠ yx
M = ∏ M m −i −1 , M < K
i =0
f is M-band limited (MBL) on G if Sf(w) = 0 for w ≥ M
M −1
d M ( x) = ∑ rw (TrR w ( x))
w=0
Q −1
f ( x) = Q −1 ∑ f (u )d M ( x  u −1 )
u =0
k
Q = ∏ g m −i −1
u =0
k- fixed when selecting M
Instead of Conclusions
Sampling Theorem on Groups
Signals
Group
Real line R
Band-limited
K. Kupfmuller, 1931
H. Nyquist, 1924
V.A. Kotelnikov, 1933
H. Raabe, 1938
K. Ogura, 1920
Locally
compact Abelian
Dyadic
Finite dyadic
Finite Abelian
Finite non-Abelian
C.E. Shannon, 1948
I. Someya, 1949
J.D. Wetson, 1949
E.T. Whittaker, 1915
J.M. Whittaker, 1924
Time-limited
C.-J. de la Vallée Poussin, 1908
M. Theis, 1919
Weiss, P., 1963
Brown, J.L., 1967
Butzer, P.L., 1977 further
I. Kluvánek, 1965
L.J. Fogel, 1959
F. Pichler, 1970
S. Kak, 1970
P.L. Butzer, 1978
Le Dinh, C. T., Le, P., Goulet, R., 1972
Acknowledgment
Thanks are due to
Mrs. Marju Taavetti
and
Mrs. Pirkko Ruotsalainen
of Tampere Univeristy of Technology
for the help in collecting the literature.