'In104
The method of least squares
$
Method of least squares
Discrete data
3
2.5
P
2
1.5
1
0.5
0
0
0.1
0.2
0.3
0.4
0.5
t
0.6
0.7
0.8
0.9
1
0.7
0.8
0.9
1
Data given by a function
3
2.5
P
2
1.5
1
0.5
0
0
&
0.1
0.2
0.3
0.4
0
0.5
t
0.6
%
'In104
The method of least squares
$
Given n data points:
(ti, pi), i = 0, 1, . . . n − 1.
425
420
415
P
410
405
400
395
390
0
2
4
6
8
10
12
14
16
18
t
Norsk Hydro, stock prizes in the time period 1.8.97 to 28.8.97
Problem:
Find a function approximating these
data.
&
1
%
'In104
The method of least squares
$
We will consider modeling the discrete data
by
&
•
a constant (zero order polynomial)
•
a linear function (rst order polynomial)
•
a second order polynomial
•
a third order polynomial
•
a piecewise linear function
•
a spline function
2
%
'In104
Approx. by a constant
$
Approximation by a constant
Given the data points
(ti, pi), i = 0, 1, . . . , n − 1.
Model of the data:
u(t) = α,
where α is a constant.
p
425
420
415
410
α
u(t)
405
400
395
390
0
1
2
3
4
5
6
7
8
9
t
&
3
%
'In104
Approx. by a constant
$
Least squares approach:
Find α such that
( )=
R α
n−
X1
n−
X1
i=0
i=0
(u(ti) − pi)2 =
is minimized.
(α − pi)2,
Recall from calculus that if R0(α) = 0 and
R00(α) > 0, then α is a minimum of R.
R(α)
2000
1800
1600
1400
1200
1000
800
600
400
400
&
402
404
406
408
410
4
412
414
416
418
420
%
'In104
Approx. by a constant
$
Dierentiation gives
( )=2
R0 α
thus
n−
X1
and
α
i=0
=
n−
X1
i=0
n−
X1
i=0
pi,
(α − pi),
nα
=
n−
X1
i=0
pi,
n−
X1
1
α=
pi, (the arithmetic average).
n
i=0
&
5
%
'In104
Approx. by a constant
$
Example
ti
pi
0
1
2
3
4
398.00 396.00 399.00 413.00 411.00
ti
pi
5
6
7
8
9
411.50 413.50 416.00 411.00 412.00
9
X
1
α=
pi ≈ 408.
10 i=0
P
425
420
415
410
α
405
400
395
390
0
1
2
3
4
5
6
7
8
9
t
&
6
%
'In104
Approx. by a linear function
$
Approximation by a linear function
Given n data points
(ti, pi), i = 0, 1, . . . , n − 1.
Model of the data:
u(t) = α + βt,
where α and β are constants.
p
425
420
415
410
405
400
395
390
&
0
1
2
3
4
7
5
6
7
8
9
t
%
'In104
Approx. by a linear function
$
Least squares approach:
Compute α and β such that
(
R α, β
)=
n−
X1
i=0
(u(ti) − pi) =
2
is minimized.
n−
X1
i=0
(α + βti − pi)2,
A minimum of R is computed by nding
and β such that
and
∂
R α, β
∂α
(
)=0
(1)
(
)=0
(2)
∂
R α, β
∂β
&
α
8
%
'In104
Approx. by a linear function
(
R α, β
)=
Consider (1) rst:
(
∂
R α, β
∂α
)=2
n−
X1
+
i=0

nα
&
i=0
(α + βti − pi)2
(α + βti − pi) = 0,
 i=0

n−
1
n−
1
X
X



α
βti

or
n−
X1
+
n−
X1
i=0
$
i=0

ti β
9
=
=
n−
X1
i=0
n−
X1
i=0
pi.
pi,
(3)
%
'In104
Approx. by a linear function
$
Next we consider (2):
(
∂
R α, β
∂β
)=2

or
i=0


ti α
i=0
&
αti

n−
X1
(α + βti − pi)ti = 0,
 i=0


n−
X1
n−
X1
+
+
n−
X1
i=0


βt2
i


n−
1
X


t2
i β
i=0
10
=
=
n−
X1
i=0
n−
X1
i=0
piti,
piti
(4)
%
'In104
Approx. by a linear function
$
Now, (3) and (4) represent two equations for
the two unknowns α and β .
We want to write the 2 × 2 system in a more
convenient way.
Let
!
α
x=
β
which is the vector we want to compute, and
let b denote the two right hand sides of (3)
and (4) i.e.
b
=


b
 0
b1
=
 P

n−1
p
 Pi=0 i 
n−1
i=0 piti
The coecients of the left hand sides of (3)
and (4) are stored in a 2 × 2 matrix A.
&
11
%
'In104
Approx. by a linear function
Dene the 2 × 2 matrix A by
A
=
where
a00
a10
a00 a01
a10 a11
= n,
=
n−
X1
i=0
ti,
!
a01
=
a11
=
$
(5)
n−
X1
i=0
n−
X1
i=0
ti,
t2
i.
Using these denitions we can write the system (3), (4) on the form
a00α + a01 β = b0
a10α + a11 β = b1
&
12
%
'In104
Approx. by a linear function
$
On a matrix-vector form, this can be written
as
Ax = b.
Here A is given by (5) and
Ax
and
=
a00 a01
a10 a11
a00α
a10α
b
&
!
+ a01β
+ a11β
=
b0
b1
13
α
β
!
!
=
,
!
.
%
'In104
Approx. by a linear function
$
Systems on the form
=b
can be solved in many dierent ways. This is
discussed in Ma104, MaIn127 and MaIn227.
Ax
p
425
420
415
410
405
400
395
390
0
1
2
3
4
5
6
7
8
9
t
Linear approximation of the Norsk Hydro stock prizes in the
time period 1.8.97 to 14.8.97
&
14
%
'In104
Approx. by a 2nd order polynomial
$
Approximation by a second order
polynomial
Given n data points
(ti, pi), i = 0, 1, . . . , n − 1.
Model of the data:
u(t) = α + βt + γt2 ,
where α, β and γ are constants.
p
425
420
415
410
405
400
395
390
&
0
1
2
3
4
15
5
6
7
8
9
t
%
'In104
Approx. by a 2nd order polynomial
$
Least squares approach:
Compute α, β and γ such that
(
R α, β, γ
=
n−
X1
i=0
)=
n−
X1
i=0
(u(ti) − pi)2
(α + βti + γt2i − pi)2
is minimized.
Again, a minimum of R is computed by nding α, β and γ such that
(
∂
R α, β,γ
∂α
&
∂
∂
) = ∂β
R(α, β,γ ) =
R(α, β,γ )=0.
∂γ
16
%
'In104
Approx. by a 2nd order polynomial
(
R α, β, γ
)=
=2
∂R
∂α
implies that

nα
&
+
n−
X1
i=0
$
n−
X1
i=0
n−
X1
i=0
(α + βti + γt2i − pi)2
(α + βti + γt2i − pi) = 0

ti β
+


n−
1
X


t2
i γ
i=0
17
=
n−
X1
i=0
pi
(6)
%
'In104
Approx. by a 2nd order polynomial
$
Similarely,
∂R
∂β


=2
implies that
n−
X1

tiα
i=0
+
n−
X1
i=0
(α + βti + γt2i − pi)ti = 0


n−
1
X


t2
i β
i=0
+
And nally
∂R
∂γ
=2
n−
X1
i=0
&
+
n−
X1
=
piti.
i=0
(7)
(α + βti + γt2i − pi)t2i = 0
which implies that


n−
1
X


t2
i α
i=0


n−
1
X


t3
i γ
i=0


n−
1
X


t3
i β
i=0
+


n−
1
X


t4
i γ
i=0
18
n−
X1
=
pit2
i.
i=0
(8)
%
'In104
Approx. by a 2nd order polynomial
$
As above we write the system dened by (6),
(7) and (8) on the form
Ax = b
(9)
where

x

b
and
=

b
 0 


 b1 


=
b2

A
&
=
=
α

(10)


 β ,
γ
 P
n−1
p
 Pi=0 i

n−1

pt
 Pi=0 i i
n−1
2
i=0 piti
a00 a01 a02





(11)



 a10 a11 a12 
a20 a21 a22
19
%
'In104
Approx. by a 2nd order polynomial
$
The components of A are the coecients of
the system (6), (7) and (8):

nα



tiα
n−
X1
i=0


n−
X1


t2
i α
i=0
&
n−
X1
+

ti β
i=0

n−
1
X


t2
i β
i=0


n−
X1


t3
i β
i=0


n−
X1
+
+
+
+
+


t2
i γ
i=0

n−
1
X


t3
i γ
i=0


n−
X1


t4
i γ
i=0

20
n−
X1
=
pi
(12)
=
piti
(13)
=
pit2
i
(14)
i=0
n−
X1
i=0
n−
X1
i=0
%
'In104
Approx. by a 2nd order polynomial
$
By dening
a00
= n,
a10
=
a20
=
a01
=
ti,
a11
=
t2
i,
a21
=
n−
X1
i=0
n−
X1
i=0
n−
X1
ti,
a02
=
t2
i,
a12
=
t3
i,
a22
=
i=0
n−
X1
i=0
n−
X1
i=0
n−
X1
t2
i
i=0
n−
X1
t3
i
i=0
n−
X1
t4
i,
i=0
we can write the system (6), (7) and (8) on
the form
Ax = b,
where x and b are given by (10) and (11)
respectively.
&
21
%
'In104
Approx. by a 2nd order polynomial
p
$
425
420
415
410
405
400
395
390
0
1
2
3
4
5
6
7
8
9
t
Approximation by a second order polynomial of the Norsk
Hydro stock prizes in the time period 1.8.97 to 14.8.97
&
22
%
'In104
Approx. by a 3rd order polynomial
$
Approximation by a third order
polynomial
Given n data points
(ti, pi), i = 0, 1, . . . , n − 1.
Model of the data:
u(t) = α + βt + γt2 + δt3,
where α, β , γ and δ are constants.
p
425
420
415
410
405
400
395
390
&
0
1
2
3
4
23
5
6
7
8
9
t
%
'In104
Approx. by a 3rd order polynomial
$
Least squares approach:
Compute α, β , γ and δ such that
(
R α, β, γ, δ
=
n−
X1
i=0
)=
n−
X1
i=0
(u(ti) − pi)2
(α + βti + γt2i + δt3i − pi)2
is minimized.
Again, a minimum is attained where all the
partial derivatives are zero. We have to nd
values of α, β , γ and δ such that
∂R
∂α
&
=
∂R
∂β
=
∂R
∂γ
24
=
∂R
∂δ
= 0.
%
'In104
Approx. by a 3rd order polynomial
R(α, β, γ, δ ) =
n−
X1
i=0
(α + βti + γt2i + δt3i − pi)2
n−
X1
∂R
= 2 (α + βti + γt2i + δt3i − pi) = 0,
∂α
i=0
(15)
n−
X1
∂R
= 2 (α + βti + γt2i + δt3i − pi)ti = 0,
∂β
i=0
(16)
n−
X1
∂R
= 2 (α + βti + γt2i + δt3i − pi)t2i = 0,
∂γ
i=0
n−
X1
∂R
= 2 (α + βti + γt2i + δt3i − pi)t3i = 0.
∂δ
i=0
(15) ⇒
n α+
n−
X1
ti
$
!
β+
n−
X1
!
t2i γ +
n−
X1
(17)
(18)
!
t3i
n−
X1
δ=
pi ,
i=0
i=0
i=0
i=0
!
!
!
!
n−
n−
n−
n−
n−
X1
X1
X1
X1
X1
(16) ⇒
ti α +
t2i β +
t3i γ +
t4i δ = piti ,
(17) ⇒
(18) ⇒
&
i=0
n−
X1
!
i=0
n−
X1
!
t2i α +
t3i α +
i=0
i=0
n−
X1
!
i=0
n−
X1
!
t3i β +
t4i β +
i=0
25
i=0
n−
X1
!
i=0
n−
X1
!
t4i γ +
t5i γ +
i=0
i=0
n−
X1
!
i=0
n−
X1
!
t5i
t6i
i=0
i=0
n−
X1
δ=
pit2i ,
i=0
n−
X1
δ=
pit3i .
i=0
%
'In104
Approx. by a 3rd order polynomial
This system can be written on the form
Ax = b,
where


α
x
=




 β 

,


 γ 


b
=









b0 

b1 

δ
a10 =
a20 =
a30 =
&
a01 =
n−
X1
ti,
i=0
n−
X1
a11 =
t2i ,
a21 =
t3i ,
a31 =
i=0
n−
X1
i=0
=
b3
and
a00 = n,

b2 

n−
X1
ti,
i=0
n−
X1
a02 =
t2i ,
a12 =
t3i ,
a22 =
t4i ,
a32 =
i=0
n−
X1
i=0
n−
X1
i=0
26
 P
n−1
p
 Pi=0 i

n−1

pt
 Pi=0 i i

n−1
2

p
t
i
i
 Pi=0
n−1
3
i=0 pi ti
n−
X1
t2i ,
a03 =
t3i ,
a13 =
t4i ,
a23 =
t5i ,
a33 =
i=0
n−
X1
i=0
n−
X1
i=0
n−
X1
i=0
$








n−
X1
t3i ,
i=0
n−
X1
t4i ,
i=0
n−
X1
t5i ,
i=0
n−
X1
t6i ,
i=0
%
'In104
Approx. by a 3rd order polynomial
p
$
425
420
415
410
405
400
395
390
0
1
2
3
4
5
6
7
8
9
t
Approximation by a third order polynomial of the Norsk Hydro
stock prizes in the time period 1.8.98 to 14.8.98
&
27
%
'In104
Approx. of Continuous Functions
$
Approximation of Continuous Functions
Given a function
p
p
= p(t)
5
4.5
p=p(t)
4
3.5
3
u (approximation)
p (data)
2.5
2
1.5
1
0.5
0
1
1.5
2
2.5
3
t
3.5
4
4.5
5
Problem:
Find a function u = u(t) approximating the
data represented by p = p(t).
&
28
%
'In104
Approximation by a constant
$
Approximation by a constant
Given data represented by the function
p = p(t), a ≤ t ≤ b.
Model of the data
u(t) = α, a ≤ t ≤ b,
where α is a constant.
p 7
6
p(t)
5
4
α 3
u(t)
2
1
0
0
1
2
3
a
&
4
5
6
t
b
29
%
'In104
Approximation by a constant
The least squares approach:
Compute α such that
Z b
$
Z b
( ) = a (u(t) −p(t))2 dt = a (α−p(t))2 dt
is minimized.
R α
Dierentiation gives
Z b
( ) = 2 a (α − p(t)) dt = 0
R0 α
⇒
Z b
a
α dt
=
Z b
()
p t dt
a
Z b
(b − a)α = a p(t) dt
Z b
1
α=
p(t) dt
b−a a
&
30
%
'In104
Approximation by a constant
$
Example
1 sin(10t).
= 1, b = 5, p(t) = t + 10
Then,
Z b
Z 5
1
1
1
α=
p(t) dt =
t+
t) dt
sin(10
b−a a
4 1 10
!
5
5
1 cos(10t)
= 14 12 t2 − 100
1
1
1
1
1
1
1
= 4 2 · 25 − 2 − 100 cos(50) + 100 cos(10)
≈ 2.995
a
p 7
6
p(t)
5
4
α 3
u(t)
2
1
&
0
0
1
2
3
a
4
5
b
31
6
t
%
'In104
Approx. by a linear function
$
Approximation by a linear function
Given data represented by the function
p = p(t), a ≤ t ≤ b.
Model of the data
u(t) = α + βt, a ≤ t ≤ b,
where α and β are constants.
p
5
4.5
p=p(t)
4
3.5
3
u (approximation)
p (data)
2.5
2
1.5
1
0.5
0
&
1
1.5
2
2.5
32
3
t
3.5
4
4.5
5
%
'In104
Approx. by a linear function
$
The least squares approach:
Compute α and β such that
Z b
Z b
( )= a (u(t) −p(t))2dt = a (α + βt−p(t))2dt
is minimized.
R α, β
Dierentiation gives
∂R
∂α
∂R
∂β
Z b
= 2 a (α + βt − p(t)) dt = 0,
(2)
Z b
= 2 a (α + βt − p(t))t dt = 0,
(2) ⇒
(3) ⇒
&
(b − a) α +
Z b
a
!
t dt α
+
Z b
a
Z b
a
33
!
β
=
t2dt β
=
t dt
!
Z b
a
Z b
a
(3)
(4)
()
p t dt
()
p t t dt
%
'In104
Approx. by a linear function
$
Example
1 sin(10t).
= 1, b = 5, p(t) = t + 10
This system can be written on the form
Ax = b,
where



  R

b
α 
b0   a p(t) dt 


x=
, b=
= Rb
,
β
b1
a p(t)t dt
and
!
a00 a01
A=
.
a10 a11
a
&
34
%
'In104
Approx. by a linear function
$
First we calculate b0 and b1
Z b
Z 5
1
b0 =
p(t) dt =
t+
sin(10
t) dt
10 !
1
a
5
5
1
1
2
= 2 t − 100 cos(10t) ≈ 11.98.
1
1
Z b
Z 5
1
p(t)t dt =
t+
sin(10t) t dt
b1 =
10
1
a
5 5 5
1
1
t
= 3 t3 + − 100 cos(10t) + 1000 sin(10t)
1
1
1
≈ 41.33 − 0.057 + 0.00028 = 41.273.
The entries in the matrix A are given by
a00 = b − a = 4,
Z 5
a01 = a10 =
t dt = 12,
1
Z 5
= 1 t2 dt = 41.33.
When solving the system Ax = b, we nd
α = −0.0067, β = 1.0006.
a11
&
35
%
'In104
Approx. by a linear function
$
The function u = u(t) is now given by
u(t) = −0.0067 + 1.0006t, 1 ≤ t ≤ 5.
p 7
6
5
4
α 3
u (approximation)
p (data)
2
1
0
0
1
2
3
a
&
4
5
6
t
b
36
%
'In104
Approx. by a linear function
$
Example 2
= 0, b = 1, p(t) = et.
Again, the system can be written on the form
Ax = b,
where



  R

b
α 
b0   a p(t) dt 


x=
, b=
= Rb
,
β
b1
a p(t)t dt
and
!
a00 a01
A=
.
a10 a11
a
&
37
%
'In104
Approx. by a linear function
We calculate b0 and b1
Z b
$
Z 1
= a p(t) dt = 0 et dt
h i1
= et 0 ≈ 1.718
Z b
Z 1
b1 =
p(t)t dt =
ett dt
0
a
b0
h
i1
Z 1
= ett 0 − 0 et dt
=1
The entries in the matrix A are given by
a00 = b − a = 1,
Z 1
1
a01 = a10 =
t dt = ,
2
0
Z 1
1
2
a11 =
t dt =
3
0
When solving the system Ax = b, we nd
α = −0.8368, β = 1.7625.
&
38
%
'In104
Approx. by a linear function
$
The function u = u(t) is then given by
u(t) = 0.8368 + 1.7625t, 0 ≤ t ≤ 1.
p
3
2.8
2.6
2.4
u (approximation)
p (data)
2.2
2
1.8
1.6
1.4
1.2
1
&
0
0.1
0.2
0.3
0.4
39
0.5
0.6
0.7
0.8
0.9
1
t
%