Chapter 13 Objectives
• Familiarizing yourself with some basic
descriptive statistics and the normal distribution.
• Knowing how to compute the slope and intercept
of a best fit straight line with linear regression.
• Knowing how to compute and understand the
meaning of the coefficient of determination and
the standard error of the estimate.
• Understanding how to use transformations to
linearize nonlinear equations so that they can be
fit with linear regression.
• Knowing how to implement linear regression
with MATLAB.
Statistics Review
Measure of Location
• Arithmetic mean: the sum of the individual
data points (yi) divided by the number of
yi

points n:
y
n
• Median: the midpoint of a group of data.
• Mode: thevalue that occurs most
frequently in a group of data.
Statistics Review
Measures of Spread
• Standard deviation:
St
sy 
n 1
where St is the sum of the squares
of the data residuals:
2
St  yi  y 

and n-1 is referred to as the degrees of freedom.
2
2
2
• Variance:
yi  y   yi   yi / n


2

sy 

n 1
n 1
 
• Coefficient of variation: sy
c.v.  100%
y

Descriptive Statistics in
MATLAB
• MATLAB has several built-in commands to
compute and display descriptive statistics.
Assuming some column vector s:
– mean(s), median(s), mode(s)
• Calculate the mean, median, and mode of s. mode is a part
of the statistics toolbox.
– min(s), max(s)
• Calculate the minimum and maximum value in s.
– var(s), std(s)
• Calculate the variance and standard deviation of s
• Note - if a matrix is given, the statistics will be
returned for each column.
Histograms in MATLAB
• [n, x] = hist(s, x)
– Determine the number of elements in each bin of data
in s. x is a vector containing the center values of the
bins.
• [n, x] = hist(s, m)
– Determine the number of elements in each bin of data
in s using m bins. x will contain the centers of the
bins. The default case is m=10
• hist(s, x) or hist(s, m) or hist(s)
– With no output arguments, hist will actually produce
a histogram.
Linear Least-Squares
Regression
• Linear least-squares regression is a method to
determine the “best” coefficients in a linear
model for given data set.
• “Best” for least-squares regression means
minimizing the sum of the squares of the
n
estimate residuals.
Forn a straight line
model,
2
2
S

e
 i   yi  a0  a1xi 
r
this gives:
i1
i1
• This method
will yield a unique line for a given

set of data.
Least-Squares Fit of a Straight
Line
• Using the model:
y  a0  a1x
the slope and intercept producing the best

fit can be found
using:
a1 
n xi yi   xi  yi
n x 
a0  y  a1 x
2
i
 x 
2
i
Example
V
(m/s)
F
(N)
a1 
i
xi
yi
(xi)2
x iy i
1
10
25
100
250
2
20
70
400
3
30
380
900
1400

11400
4
40
550
1600
22000
5
50
610
2500
30500
6
60
1220
3600
73200
7
70
830
4900
58100
8
80
1450
6400
116000

360
5135
20400
312850

n xi yi   xi  yi
n x 
2
i
 x 
2
i

8312850  3605135
820400  360
2
 19.47024
a0  y  a1 x  641.875 19.47024 45  234.2857
Fest  234.2857 19.47024v
Quantification of Error
• Recall for a straight line, the sum of the
squares of the estimate residuals:
n
n
Sr   e   yi  a0  a1 xi 
2
i
i1
i1

• Standard error of theSestimate:
s y/ x 
r
n2
2
Standard Error of the Estimate
• Regression data showing (a) the spread of data around the
mean of the dependent data and (b) the spread of the data
around the best fit line:
• The reduction in spread represents the improvement due to
linear regression.
Coefficient of Determination
• The coefficient of determination r2 is the difference
between the sum of the squares of the data residuals
and the sum of the squares of the estimate residuals,
normalized by the sum of the squares of the data
St  Sr
2
residuals:
r 
St
• r2 represents the percentage of the original uncertainty
explained 
by the model.
• For a perfect fit, Sr=0 and r2=1.
• If r2=0, there is no improvement over simply picking the
mean.
• If r2<0, the model is worse than simply picking the mean!
Example
V
(m/s)
F
(N)
i
xi
yi
a0+a1xi
1
10
25
-39.58
380535
4171
2
20
70
155.12
327041
7245
Fest  234.2857 19.47024v
(yi- ȳ)2 (yi-a0-a1xi)2
3
30
380
349.82
68579
911
4
40
550
544.52
8441
30
5
50
610
739.23
1016
16699
6
60
1220
933.93
334229
81837
7
70
830
1128.63
35391
89180
8
80
1450
1323.33
653066
16044

360
5135
1808297
216118

St   yi  y   1808297
2
Sr   yi  a0  a1 xi   216118
2
sy 
1808297
 508.26
8 1
216118
 189.79
82
1808297  216118
r2 
 0.8805
1808297
s y/ x 
88.05% of the original uncertainty
has been explained by the
linear model
Nonlinear Relationships
• Linear regression is predicated on the fact
that the relationship between the
dependent and independent variables is
linear - this is not always the case.
• Three common examples are:  x
exponential :
power :
y  1e
1
y  2 x 2
x
saturation - growth - rate : y   3
3  x
Linearization of Nonlinear
Relationships
• One option for finding the coefficients for a
nonlinear fit is to linearize it. For the three
common models, this may involve taking
logarithms or inversion:
Model
Nonlinear
Linearized
exponential :
y  1e1 x
ln y  ln 1  1 x
power :
y  2 x 2
log y  log  2  2 log x
saturation - growth - rate : y   3
x
3  x
1 1 3 1
 
y 3 3 x
Transformation Examples
MATLAB Functions
• MATLAB has a built-in function polyfit that fits a
least-squares nth order polynomial to data:
– p = polyfit(x, y, n)
• x: independent data
• y: dependent data
• n: order of polynomial to fit
• p: coefficients of polynomial
f(x)=p1xn+p2xn-1+…+pnx+pn+1
• MATLAB’s polyval command can be used to
compute a value using the coefficients.
– y = polyval(p, x)
Chapter 14 Objectives
• Knowing how to implement polynomial
regression.
• Knowing how to implement multiple linear
regression.
• Understanding the formulation of the general
linear least-squares model.
• Understanding how the general linear leastsquares model can be solved with MATLAB
using either the normal equations or left division.
• Understanding how to implement nonlinear
regression with optimization techniques.
Polynomial Regression
•
•
The least-squares
procedure from Chapter
13 can be readily
extended to fit data to a
higher-order polynomial.
Again, the idea is to
minimize the sum of the
squares of the estimate
residuals.
The figure shows the
same data fit with:
a) A first order polynomial
b) A second order
Process and Measures of Fit
• For a second order
polynomial,
the best fit would mean
n
n
minimizing:
2
2 2
S  e 
y a a x a x
r

i1
i

i
0
1 i
2 i

i1
n this would
n
• In general,
mean minimizing:
2
2
m 2
Sr   ei   yi  a0  a1 xi  a2 xi   am xi 
i1
i1

• The standard error for fitting an mth order polynomial to n
Sr
data points is:
s y/ x 
n  m 1

because the mth order polynomial has (m+1) coefficients.
2 is still found using:
• The coefficient of determination
r
S

S
2
r
 t r

St
Multiple Linear Regression
• Another useful extension of
linear regression is the case
where y is a linear function of
two or more independent
variables:
y  a0  a1x1  a2 x2  am xm
• Again, the best fit is obtained by
minimizing the sum of the
squares of the estimate
residuals:
n
n
i1
i1
Sr   ei2   yi  a0  a1 x1,i  a2 x2,i 
am xm,i 
2
General Linear Least Squares
• Linear, polynomial, and multiple linear
regression all belong to the general linear
least-squares model:
y  a0 z0  a1z1  a2 z2 
am zm  e
where z0, z1, …, zm are a set of m+1 basis
functions
and e is the error of the fit.

• The basis functions can be any function
data but cannot contain any of the
coefficients a0, a1, etc.
Solving General Linear Least
Squares Coefficients
• The equation: y  a0 z0  a1z1  a2 z2 
am zm  e
can be re-written for 
each
point
y  data
Z a
 eas a matrix
equation:

where {y} contains the dependent data, {a} contains the
z01 z11
zm1 
coefficients
of the equation,
the error at
z02 z12{e} contains

zm2
Z
each point, and [Z]
is:


zmn 
z0n z1n

with zji representing the the value of the jth basis function
calculated
 at the ith point.
Solving General Linear Least
Squares Coefficients
• Generally, [Z] is not a square matrix, so
simple inversion cannot be used to solve
for {a}. Instead the sum of the squares of
the estimate residuals is minimized:
m

2
Sr   ei2  
yi   a j z ji 

i1
i1 
j0

n
n
• The outcome of this minimization yields:






Z  Z  a  Z  y
T
T
MATLAB Example
• Given x and y data in columns, solve for the
coefficients of the best fit line for y=a0+a1x+a2x2
Z = [ones(size(x) x x.^2]
a = (Z’*Z)\(Z’*y)
– Note also that MATLAB’s left-divide will automatically
include the [Z]T terms if the matrix is not square, so
a = Z\y
would work as well
• To calculate measures of fit:
St = sum((y-mean(y)).^2)
Sr = sum((y-Z*a).^2)
r2 = 1-Sr/St
syx = sqrt(Sr/(length(x)-length(a)))
Nonlinear Regression
• As seen in the previous chapter, not all fits are
linear equations of coefficients and basis
functions.
• One method to handle this is to transform the
variables and solve for the best fit of the
transformed variables. There are two problems
with this method:
– Not all equations can be transformed easily or at all
– The best fit line represents the best fit for the
transformed variables, not the original variables.
• Another method is to perform nonlinear
regression to directly determine the leastsquares fit.
Nonlinear Regression in
MATLAB
• To perform nonlinear regression in
MATLAB, write a function that returns the
sum of the squares of the estimate
residuals for a fit and then use MATLAB’s
fminsearch function to find the values of
the coefficients where a minimum occurs.
• The arguments to the function to compute
Sr should be the coefficients, the
independent variables, and the dependent
variables.
Nonlinear Regression in
MATLAB Example
• Given dependent force data F for independent
velocity data v, determinea the coefficients for the
F  a0 v
fit:
1
• First - write a function called fSSR.m containing

the following:
function f = fSSR(a, xm, ym)
yp = a(1)*xm.^a(2);
f = sum((ym-yp).^2);
• Then, use fminsearch in the command window
to obtain the values of a that minimize fSSR:
a = fminsearch(@fSSR, [1, 1], [], v, F)
where [1, 1] is an initial guess for the [a0, a1]
Nonlinear Regression Results
• The resulting coefficients will produce the
largest r2 for the data and may be different
from the coefficients produced by a
transformation:
Chapter 15 Objectives
• Recognizing that evaluating polynomial coefficients with
simultaneous equations is an ill-conditioned problem.
• Knowing how to evaluate polynomial coefficients and
interpolate with MATLAB’s polyfit and polyval functions.
• Knowing how to perform an interpolation with Newton’s
polynomial.
• Knowing how to perform an interpolation with a
Lagrange polynomial.
• Knowing how to solve an inverse interpolation problem
by recasting it as a roots problem.
• Appreciating the dangers of extrapolation.
• Recognizing that higher-order polynomials can manifest
large oscillations.
Polynomial Interpolation
• You will frequently have occasions to estimate
intermediate values between precise data
points.
• The function you use to interpolate must pass
through the actual data points - this makes
interpolation more restrictive than fitting.
• The most common method for this purpose is
th order
polynomial interpolation,2 where an
(n-1)
f (x)  a1  a2 x  a3 x   an x n1
polynomial is solved that passes through n data
points: MATLAB version :
f (x)  p1 x n1  p2 x n2 
 pn1 x  pn
Determining Coefficients
• Since polynomial interpolation provides as
many basis functions as there are data
points (n), the polynomial coefficients can
be found exactly using linear algebra.
• MATLAB’s built in polyfit and polyval
commands can also be used - all that is
required is making sure the order of the fit
for n data points is n-1.
Polynomial Interpolation
Problems
• One problem that can occur with solving for the
coefficients ofn1a polynomial
is that the system to be
x1
x1n2
x1 1 p1   f x1  
inverted is inx n1
thexform:
 f x 
n2
x
1 p 
 
 2
 n1
xn1
n1

xn
2
n2
xn1
xnn2
2
xn1
xn
 2   2 

  
1pn1  f xn1 
  

1
 pn   f xn  
• Matrices such as that on the left are known as

Vandermonde
matrices, and they are very ill-conditioned
- meaning their solutions are very sensitive to round-off
errors.
• The issue can be minimized by scaling and shifting the
data.
Newton Interpolating
Polynomials
• Another way to express a polynomial
interpolation is to use Newton’s
interpolating polynomial.
• The differences between a simple
polynomial and Newton’s interpolating
polynomial for first and second order
interpolations are:
Order
1st
2nd
Simple
f1 (x)  a1  a2 x
f2 (x)  a1  a2 x  a3 x 2
Newton
f1 (x)  b1  b2 (x  x1 )
f2 (x)  b1  b2 (x  x1 ) b3(x  x1 )(x  x2 )
Newton Interpolating
Polynomials (cont)
• The first-order Newton
interpolating polynomial
may be obtained from
linear interpolation and
similar triangles, as
shown.
• The resulting formula
based on known points x1
and x2 and the values of
the dependent function at
f xis:
2   f x1 
those
points
f1 x  f x1  
x  x1 
x2  x1
Newton Interpolating
Polynomials (cont)
• The second-order Newton
interpolating polynomial
introduces some curvature to
the line connecting the
points, but still goes through
the first two points.
• The resulting formula based
on known points x1, x2, and
x3 and the values of the
dependent function at those
f x3   f x2  f x2   f x1 
points is:

f2 x  f x1 
f x2   f x1 
x  x1 
x2  x1
x 3  x2
x3  x1
x2  x1
x  x1 x  x2 
Newton Interpolating
Polynomials
(cont)
th
• In general, an (n-1) Newton interpolating
polynomial has all the terms of the (n-2)th
polynomial plus one extra.
• The general formula is:
fn1 x  b1  b2 x  x1  
where
 bn x  x1 x  x2 
x  xn1 
b1  f x1 
b2  f x2 , x1 

b3  f x3 , x2 , x1 
bn  f xn , xn1 ,
, x2 , x1 
and the f[…] represent divided differences.

Divided Differences
• Divided difference
are calculated as follows:
f x   f x 
f x , x 
x x
i
i
j
j
i
f xi , x j , xk 
j
f xi , x j  f x j , xk 
xi  x k
f xn , xn1, , x2 , x1  
f xn , xn1 , , x2   f xn1 , xn2 , , x1 
xn  x1
• Divided differences are calculated using divided

difference
of a smaller number of terms:
Lagrange Interpolating
Polynomials
• Another method that uses shifted value to express
an interpolating polynomial is the Lagrange
interpolating polynomial.
• The differences between a simply polynomial and
Lagrange interpolating polynomials for first and
second order polynomials is:
Order
1st
2nd
Simple
f1 (x)  a1  a2 x
f2 (x)  a1  a2 x  a3 x 2
Lagrange
f1 (x)  L1 f x1  L2 f x2 
f2 (x)  L1 f x1  L2 f x2  L3 f x3 
where the Li are weighting coefficients that are
functions of x.
Lagrange Interpolating
Polynomials (cont)
• The first-order Lagrange
interpolating polynomial
may be obtained from a
weighted combination of
two linear interpolations, as
shown.
• The resulting formula based
on known points x1 and x2
and the values of the
dependent function at those
f1 (x)
 L1 f x1  L2 f x2 
points is:
x  x2
x  x1
L1 
, L2 
x1  x2
x2  x1
x  x2
x  x1
f1 (x) 
f x1 
f x2 
x1  x2
x2  x1
Lagrange Interpolating
Polynomials (cont)
• In general, the Lagrange polynomial
interpolation for n points
is:
n
fn1 xi    Li x  f xi 
i1
where Li is given by:
x  xj
Li x  
x  xj
j1 i

n
ji

Inverse Interpolation
• Interpolation general means finding some value
f(x) for some x that is between given
independent data points.
• Sometimes, it will be useful to find the x for
which f(x) is a certain value - this is inverse
interpolation.
• Rather than finding an interpolation of x as a
function of f(x), it may be useful to find an
equation for f(x) as a function of x using
interpolation and then solve the corresponding
roots problem:
f(x)-f
=0 for x.
Extrapolation
• Extrapolation is the
process of estimating a
value of f(x) that lies
outside the range of the
known base points x1,
x2, …, xn.
• Extrapolation
represents a step into
the unknown, and
extreme care should be
exercised when
extrapolating!
Extrapolation Hazards
• The following shows the results of
extrapolating a seventh-order population
data set:
Oscillations
• Higher-order polynomials can not only lead to round-off
errors due to ill-conditioning, but can also introduce
oscillations to an interpolation or fit where they should
not be.
• In the figures below, the dashed line represents an
function, the circles represent samples of the function,
and the solid line represents the results of a polynomial
interpolation: