Chapter 6 - Problem
Pattern Recognition
2023. 03. 27
Pham Van Khanh
 Problem
 Problem 2
Fourier’s Theorem can be used to show that a three-layer neural net with sigmoidal hidden units can approximate to
arbitrary accuracy any posterior function. Consider two-dimensional input and a single output, 𝑧 𝑥1 , 𝑥2 . Recall that Fourier’s
Theorem states that, given weak restrictions, any such functions can be written as a possibly infinite sum of cosine
functions, as
𝑧 𝑥1 , 𝑥2 ≈
𝐴𝑓1 𝑓2 cos 𝑓1 𝑥1 cos 𝑓2 𝑥2
𝑓1
𝑓2
with coefficients 𝐴𝑓1 𝑓2
(a) Use the trigonometric identity
to write 𝑧 𝑥1 , 𝑥2
1
1
cos 𝛼 cos 𝛽 = cos 𝛼 + 𝛽 + cos 𝛼 − 𝛽
2
2
as a linear combination of terms cos 𝑓1 𝑥1 + 𝑓2 𝑥2 and cos 𝑓1 𝑥1 − 𝑓2 𝑥2
(b) Show that cos 𝑥 or indeed any continuous function f(x) can be approximated to any accuracy by a
linear combination of sign functions as:
𝑁
1 + 𝑆𝑔𝑛 𝑥 − 𝑥𝑖
𝑓 𝑥 ≈ 𝑓 𝑥0 +
𝑓 𝑥𝑖+1 − 𝑓 𝑥𝑖
2
𝑖=0
where the 𝑥𝑖 are sequential values of 𝑥; the smaller 𝑥𝑖+1 − 𝑥𝑖 , the better the approximation.
(c) Put your results together to show that 𝑧 𝑥1 , 𝑥2 can be expressed as a linear combination of step
functions or sign functions whose arguments are themselves linear combinations of the input variables 𝑥𝑖
and 𝑥𝑖 . Explain, in turn, why this implies that a three-layer network with sigmoidal hidden units and a linear
output unit can implement any function that can be expressed by a Fourier series.
(d) Does your construction guarantee that the derivative d𝑓 𝑥 /𝑑𝑥 can be well approximated too?
 Solution

Problem 2
(a) Use the trigonometric identity
1
1
cos 𝛼 cos 𝛽 = cos 𝛼 + 𝛽 + cos 𝛼 − 𝛽
2
2
To give:
𝑧 𝑥1 , 𝑥2 ≈
𝐴𝑓1 𝑓2 cos 𝑓1 𝑥1 cos 𝑓2 𝑥2
𝑓1
𝑧 𝑥1 , 𝑥2 ≈
𝑓1
𝑓2
𝑓2
𝐴𝑓1 𝑓2
cos 𝑓1 𝑥1 + 𝑓2 𝑥2 + cos 𝑓1 𝑥1 − 𝑓2 𝑥2
2
 𝑧 𝑥1 , 𝑥2 is approximately equal to the sum of cos 𝑓1 𝑥1 + 𝑓2 𝑥2 and cos 𝑓1 𝑥1 − 𝑓2 𝑥2
 Solution

Problem 2
(b) We can approximate f(x) using Fourier series approximation such as:
𝑁
𝑓 𝑥 ≈ 𝑎0 +
𝑎𝑛 cos 𝑛𝑥 + 𝑏𝑛 sin 𝑛𝑥
𝑛=1
Where N is a positive integer, and the coefficients
2
𝑎𝑛 =
𝜋
𝜋
𝑓 𝑥 cos 𝑛𝑥 𝑑𝑥
0
2
𝑏𝑛 =
𝜋
𝜋
𝑓 𝑥 sin 𝑛𝑥 𝑑𝑥
0
To approximate 𝑓 𝑥 using sign functions, we can replace the cos 𝑛𝑥 and sin 𝑛𝑥 terms with sign
functions. Specifically, we can use the following formulas:
cos 𝑛𝑥 ≈
1
𝜋
𝜋
sgn 𝑥 −
+ sgn − 𝑥
2
2
2
sin 𝑛𝑥 ≈
1
𝜋
𝜋
sgn 𝑥 −
− sgn − 𝑥
2𝑖
2
2
 Solution

Problem 2
(c)
𝑁
𝑓 𝑥 ≈ 𝑓 𝑥0 +
𝑓 𝑥𝑖+1 − 𝑓 𝑥𝑖
𝑖=0
cos 𝑓1 𝑥1 + 𝑓2 𝑥2
1 + 𝑆𝑔𝑛 𝑥 − 𝑥𝑖
2
𝑁
≈ cos 𝑓1 𝑥10 + 𝑓2 𝑥20 +
cos 𝑓1 𝑥1𝑖+1 + 𝑓2 𝑥2𝑖+1 − cos 𝑓1 𝑥1𝑖 + 𝑓2 𝑥2𝑖
1 + 𝑆𝑔𝑛 𝑥1 − 𝑥1𝑖 𝑆𝑔𝑛 𝑥2 − 𝑥2𝑖
2
cos 𝑓1 𝑥1𝑖+1 − 𝑓2 𝑥2𝑖+1 − cos 𝑓1 𝑥1𝑖 − 𝑓2 𝑥2𝑖
1 + 𝑆𝑔𝑛 𝑥1 − 𝑥1𝑖 𝑆𝑔𝑛 𝑥2 − 𝑥2𝑖
2
𝑖=0
cos 𝑓1 𝑥1 − 𝑓2 𝑥2
𝑁
≈ cos 𝑓1 𝑥10 − 𝑓2 𝑥20 +
𝑖=0
(d) The construction do not guarantee that the derivative d𝑓 𝑥 /𝑑𝑥 can be well approximated too?
Thank you for your attention