Basis Basics
Selected from presentations by
Jim Ramsay, McGill University,
Hongliang Fei, and Brian Quanz
1. Introduction



Basis: In Linear Algebra, a basis is a set
of vectors satisfying:
Linear combination of the basis can
represent every vector in a given vector
space;
No element of the set can be
represented as a linear combination of
the others.



In Function Space, Basis is degenerated
to a set of basis functions;
Each function in the function space can
be represented as a linear combination
of the basis functions.
Example: Quadratic Polynomial bases
{1,t,t^2}
What are basis functions?



We need flexible method for constructing a function
f(t) that can track local curvature.
We pick a system of K basis functions φk(t), and call
this the basis for f(t).
We express f(t) as a weighted sum of these basis
functions:
f(t) = a1φ1(t) + a2φ2(t) + … + aKφK(t)
The coefficients a1, … , aK determine the shape of the
function.
What do we want from basis
functions?





Fast computation of individual basis functions.
Flexible: can exhibit the required curvature where
needed, but also be nearly linear when appropriate.
Fast computation of coefficients ak: possible if
matrices of values are diagonal, banded or sparse.
Differentiable as required: We make lots of use of
derivatives in functional data analysis.
Constrained as required, such as periodicity,
positivity, monotonicity, asymptotes and etc.
What are some commonly used
basis functions?



Powers: 1, t, t2, and so on. They are the basis
functions for polynomials. These are not very
flexible, and are used only for simple problems.
Fourier series: 1, sin(ωt), cos(ωt), sin(2ωt),
cos(2ωt), and so on for a fixed known frequency ω.
These are used for periodic functions.
Spline functions: These have now more or less
replaced polynomials for non-periodic problems.
More explanation follows.
What is Basis Expansion?
Given data X and transformation
hm ( X ) : p  , m  1,..., M . Then we model

as a linear basis expansion in X, where
hm ( X ) is a basis function.
Why Basis Expansion?



In regression problems, f(X) will
typically nonlinear in X;
Linear model is convenient and easy to
interpret;
When sample size is very small but
attribute size is very large, linear model
is all what we can do to avoid over
fitting.
2. Piecewise Polynomials and
Splines




Spline:
In Mathematics, a spline is a special function defined
piecewise by polynomials;
In Computer Science, the term spline more frequently
refers to a piecewise polynomial (parametric) curve.
Simple construction, ease and accuracy of
evaluation, capacity to approximate complex
shapes through curve fitting and interactive
curve design.

Assume four knots spline (two boundary
knots and two interior knots), also X is
one dimensional.
Piecewise constant basis:

Piecewise Linear Basis:

Piecewise Cubic Polynomial


Basis functions:
Six functions corresponding to a sixdimensional linear space.
Piecewise Cubic Polynomial
Spline Interpolation Method
Slides taken from the lecture by
Authors: Autar Kaw, Jai Paul
http://numericalmethods.eng.usf.edu
14
What is Interpolation ?
Given (x0,y0), (x1,y1), …… (xn,yn), find the
value of ‘y’ at a value of ‘x’ that is not given.
15
http://numericalmethods.eng.usf.edu
Interpolants
Polynomials are the most common
choice of interpolants because they
are easy to:
Evaluate
Differentiate, and
Integrate.
16
http://numericalmethods.eng.usf.edu
Why Splines ?
1
f ( x) 
1  25 x 2
Table : Six equidistantly spaced points in [-1, 1]
x
1
1  25 x 2
-1.0
0.038461
-0.6
0.1
-0.2
0.5
0.2
0.5
0.6
0.1
1.0
17
y
0.038461
Figure : 5th order polynomial vs. exact function
http://numericalmethods.eng.usf.edu
Why Splines ?
1.2
0.8
y
0.4
0
-1
-0.5
0
0.5
1
-0.4
-0.8
x
19th Order Polynomial
f (x)
5th Order Polynomial
Figure : Higher order polynomial interpolation is a bad idea
18
http://numericalmethods.eng.usf.edu
Linear Interpolation
Given  x0 , y0 , x1 , y1 ,......, x n1 , y n 1  x n , y n  , fit linear splines to the data. This simply involves
forming the consecutive data through straight lines. So if the above data is given in an ascending
order, the linear splines are given by  yi  f ( xi ) 
Figure : Linear splines
19
http://numericalmethods.eng.usf.edu
Linear Interpolation (contd)
f ( x )  f ( x0 ) 
f ( x1 )  f ( x 0 )
( x  x 0 ),
x1  x 0
x 0  x  x1
 f ( x1 ) 
f ( x 2 )  f ( x1 )
( x  x1 ),
x2  x1
x1  x  x 2
.
.
.
 f ( x n 1 ) 
f ( x n )  f ( x n 1 )
( x  x n 1 ), x n 1  x  x n
x n  x n 1
Note the terms of
f ( xi )  f ( x i 1 )
xi  x i 1
in the above function are simply slopes between xi 1 and x i .
20
http://numericalmethods.eng.usf.edu
Example
The upward velocity of a rocket is given as a
function of time in Table 1. Find the velocity at
t=16 seconds using linear splines.
Table Velocity as a
function of time
21
t (s)
v(t ) (m/s)
0
10
15
20
22.5
30
0
227.04
362.78
517.35
602.97
901.67
Figure. Velocity vs. time data
for the rocket example
http://numericalmethods.eng.usf.edu
Linear Interpolation
t 0  15,
v (t 0 )  362.78
t1  20,
v (t1 )  517.35
v(t )  v (t 0 )
v (t )  v(t 0 )  1
(t  t 0 )
t1  t 0
517.35
500
ys
f ( range)
 362.78 
517.35  362.78
fx

(t  15) desired
20  15
v (t )  362.78  30.913( t  15)
At t  16,
 393.7
22
450
400
362.78
v (16)  362.78  30.913(16  15)
550
350
10
12
x s  10
0
14
16
18
x s  range x desired
20
22
24
x s  10
m/s
http://numericalmethods.eng.usf.edu
1
Quadratic Interpolation
Given  x0 , y0 ,  x1 , y1 ,......, x n 1 , y n 1 ,  x n , y n  , fit quadratic splines through the data. The splines
are given by
f ( x )  a1 x 2  b1 x  c1 ,
 a 2 x 2  b2 x  c2 ,
x 0  x  x1
x1  x  x 2
.
.
.
 a n x 2  bn x  cn ,
x n 1  x  x n
Find a i , bi , ci , i  1, 2, …, n
23
http://numericalmethods.eng.usf.edu
Quadratic Interpolation (contd)
Each quadratic spline goes through two consecutive data points
a1 x 0  b1 x 0  c1  f ( x0 )
2
a1 x1  b1 x1  c1  f ( x1 )
2
.
.
.
a i xi 1  bi xi 1  ci  f ( xi 1 )
2
a i xi  bi xi  c i  f ( xi )
2
.
.
.
a n x n 1  bn x n 1  c n  f ( xn 1 )
2
a n x n  bn xn  cn  f ( x n )
2
24
This condition gives 2n equations
http://numericalmethods.eng.usf.edu
Quadratic Splines (contd)
The first derivatives of two quadratic splines are continuous at the interior points.
For example, the derivative of the first spline
a1 x 2  b1 x  c1 is
2 a1 x  b1
The derivative of the second spline
a 2 x 2  b2 x  c 2 is
2 a2 x  b2
and the two are equal at x  x1 giving
2 a1 x1  b1  2a 2 x1  b2
2 a1x1  b1  2a 2 x1  b2  0
25
http://numericalmethods.eng.usf.edu
Quadratic Splines (contd)
Similarly at the other interior points,
2a 2 x 2  b2  2a3 x 2  b3  0
.
.
.
2ai xi  bi  2ai 1 xi  bi 1  0
.
.
.
2a n 1 x n 1  bn 1  2a n x n1  bn  0
We have (n-1) such equations. The total number of equations is (2n)  (n  1)  (3n  1) .
We can assume that the first spline is linear, that is a1  0
26
http://numericalmethods.eng.usf.edu
Quadratic Splines (contd)
This gives us ‘3n’ equations and ‘3n’ unknowns. Once we find the ‘3n’ constants,
we can find the function at any value of ‘x’ using the splines,
f ( x)  a1 x 2  b1 x  c1 ,
 a 2 x 2  b2 x  c 2 ,
x0  x  x1
x1  x  x 2
.
.
.
 a n x 2  bn x  c n ,
27
x n 1  x  x n
http://numericalmethods.eng.usf.edu
Quadratic Spline Example
The upward velocity of a rocket is given as a function of time.
Using quadratic splines
a) Find the velocity at t=16 seconds
b) Find the acceleration at t=16 seconds
c) Find the distance covered between t=11 and t=16 seconds
Table Velocity as a
function of time
28
t (s)
v(t ) (m/s)
0
10
15
20
22.5
30
0
227.04
362.78
517.35
602.97
901.67
Figure. Velocity vs. time data
for the rocket example
http://numericalmethods.eng.usf.edu
Solution
2
v(t )  a1t  b1t  c1 , 0  t  10
 a2t  b2t  c2 , 10  t  15
2
 a3t  b3t  c3 , 15  t  20
2
 a4t  b4t  c4 , 20  t  22.5
2
 a5t  b5t  c5 ,
2
22.5  t  30
Let us set up the equations
29
http://numericalmethods.eng.usf.edu
Each Spline Goes Through
Two Consecutive Data Points
v(t )  a1t  b1t  c1 , 0  t  10
2
a1 (0)  b1 (0)  c1  0
2
a1 (10)  b1 (10)  c1  227.04
2
30
http://numericalmethods.eng.usf.edu
Each Spline Goes Through
Two Consecutive Data Points
31
t
s
0
v(t)
m/s
0
10
15
20
22.5
227.04
362.78
517.35
602.97
30
901.67
a2 (10) 2  b2 (10)  c2  227.04
a2 (15) 2  b2 (15)  c2  362.78
2
a3 (15)  b3 (15)  c3  362.78
2
a3 (20)  b3 (20)  c3  517.35
a4 (20)  b4 (20)  c4  517.35
a4 (22.5)2  b4 (22.5)  c4  602.97
2
a5 (22.5)  b5 (22.5)  c5  602.97
a5 (30) 2  b5 (30)  c5  901.67
2
http://numericalmethods.eng.usf.edu
Derivatives are Continuous at
Interior
Data
Points
2
v(t )  a1t  b1t  c1 , 0  t  10
 a2t  b2t  c2 ,10  t  15
2

d
2
a1t  b1t  c1
dt


t 10
d
2

a2t  b2t  c2
dt
2a1t  b1  t 10  2a2t  b2  t 10
2a1 10  b1  2a2 10  b2

t 10
20a1  b1  20a2  b2  0
32
http://numericalmethods.eng.usf.edu
Derivatives are continuous at
Interior Data Points
At t=10
2a1 (10)  b1  2a2 (10)  b2  0
At t=15
2a2 (15)  b2  2a3 (15)  b3  0
At t=20
2a3 (20)  b3  2a4 (20)  b4  0
At t=22.5
2a4 (22.5)  b4  2a5 (22.5)  b5  0
33
http://numericalmethods.eng.usf.edu
Last Equation
a1  0
34
http://numericalmethods.eng.usf.edu
Final Set of Equations
0
100
0

0
0
0

0
0
0

0
 20
0

0
0
 1
35
0
1
0
0
0
0
0
0
0
0
0
10 1
0
0
0
0
0
0
0
0
0
0
0
100 10 1
0
0
0
0
0
0
0
0
225 15 1
0
0
0
0
0
0
0
0
0
0
0
225 15 1
0
0
0
0
0
0
0
0
400 20 1
0
0
0
0
0
0
0
0
0
0
0
400
20
1
0
0
0
0
0
0
0
0 506.25 22.5 1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
0  20  1 0
0
0
0
0
0
0
0
0
30
1
0  30  1 0
0
0
0
0
0
0
0
0
40
1
0
 40
1
0
0
0
0
0
0
0
0
0
45
1
0
0
0
0
0
0
0
0
0
0
0
0
0  a1 
0  b1 
 0 
227.04
0
0
   227.04
0
0
0 c1
  

0
0
0  a 2  362.78
0
0
0  b2  362.78
  517.35
0
0
0 c2
  

0
0
0  a3  517.35
0
0
0  b3   602.97
506.25 22.5 1   c3  602.97
  

900
30 1   a 4  901.67
0
0
0  b4   0 
0
0
0  c4   0 
  

0
0
0  a5 
0



 45
 1 0  b5   0 
0
0
0  c5   0 
0
0
http://numericalmethods.eng.usf.edu
Coefficients of Spline
36
i
ai
bi
ci
1
0
22.704
0
2
0.8888
4.928
88.88
3
−0.1356
35.66
−141.61
4
1.6048
5
0.20889
−33.956 554.55
28.86
−152.13
http://numericalmethods.eng.usf.edu
Quadratic Spline Interpolation
Part 2 of 2
http://numericalmethods.eng.usf.edu
37
http://numericalmethods.eng.usf.edu
v(t )  22.704t ,
38
Final Solution
0  t  10
 0.8888t 2  4.928t  88.88,
10  t  15
 0.1356t 2  35.66t  141.61,
 1.6048t 2  33.956t  554.55,
15  t  20
20  t  22.5
 0.20889t 2  28.86t  152.13,
22.5  t  30
http://numericalmethods.eng.usf.edu
Velocity at a Particular Point
a) Velocity at t=16
0  t  10
v(t )  22.704t ,
 0.8888t 2  4.928t  88.88,
10  t  15
 0.1356t 2  35.66t  141.61,
 1.6048t 2  33.956t  554.55,
15  t  20
20  t  22.5
 0.20889t 2  28.86t  152.13,
22.5  t  30
v16  0.135616  35.6616  141.61
2
 394.24 m/s
39
http://numericalmethods.eng.usf.edu
Quadratic Spline Graph
t=a:2:b;
Quadratic Spline Graph
t=a:0.5:b;
Natural Cubic Spline Interpolation
SPLINE OF DEGREE k = 3



The domain of S is an interval [a,b].
S, S’, S’’ are all continuous functions on [a,b].
There are points ti (the knots of S) such that a = t0 < t1 < .. tn = b and
such that S is a polynomial of degree at most k on each subinterval [ti,
ti+1].
x
t0
t1
…
tn
y
y0
y1
…
yn
ti are knots
Natural Cubic Spline Interpolation
Si(x) is a cubic polynomial that will be used on
the subinterval [ xi, xi+1 ].
Natural Cubic Spline Interpolation

Si(x) = aix3 + bix2 + cix + di
•
•
4 Coefficients with n subintervals = 4n equations
There are 4n-2 conditions
•
Interpolation conditions
Continuity conditions
•
Natural Conditions
•
•
•
S’’(x0) = 0
S’’(xn) = 0