Programming for LSPR:
1. Write a program to implement Cramer’s Rule
The SCIPY library
“skippy” not to be confused with the “Skippy the super virus”
(Thanks to Travis Oliphant!)
SciPy is a collection of mathematical algorithms and convenience functions built on
the Numeric extension for Python. It adds significant power to the interactive
Python session by exposing the user to high-level commands and classes for the
manipulation and visualization of data. To use SciPy you need to import all of the
names defined in the SciPy namespace using the command
>>> from scipy.linalg import det
>>> from scipy import mat
Matrices in Python.
Matrix Class
The matrix class is initialized with the SciPy command mat. If you are going to be doing a lot of
matrix-math, it is convenient to convert arrays into matrices using this command. One convenience
of using the mat command is that you can enter two-dimensional matrices with commas or spaces
separating columns and semicolons separating rows as long as the matrix is placed in a string
passed to mat.
For example, let
This matrix could be entered
>>> A = mat(’[1 3 5; 2 5 1; 2 3 8]’)
>>> A
Matrix([[1, 3, 5],
[2, 5, 1],
[2, 3, 8]])
Finding Determinant
The determinant of a square matrix A is often denoted |A| and is a quantity often used in linear
algebra. Suppose aij are the elements of the matrix A and let Mij = |Aij| be the determinant of the
matrix left by removing the ith row and jthcolumn from A. Then for any row i,
This is a recursive way to define the determinant where the base case is defined by accepting that
the determinant of a 1 × 1 matrix is the only matrix element. In SciPy the determinant can be
calculated with linalg.det. For example, the determinant of
is
In SciPy this is computed as shown in this example:
>>> A = mat(’[1 3 5; 2 5 1; 2 3 8]’)
>>> linalg.det(A)
-25.000000000000004
Alternately, you can define the matrix as follows:
>>> equation1='2,1,1;'
>>> equation2='1,-1,-1;'
>>> equation3='1,2,1'
>>> A=mat(equation1+equation2+equation3)
>>> print A
[[ 2 1 1]
[ 1 -1 -1]
[ 1 2 1]]
>>> print det(A)
3.0
Of finally we could define it as we have all along as a list of lists:
>>> equation1=[2,1,1]
>>> equation2=[1,-1,-1]
>>> equation3=[1,2,1]
>>> A = [equation1,equation2,equation3]
>>> print A
[[2, 1, 1], [1, -1, -1], [1, 2, 1]]
>>> print det(A)
3.0
Exercise:
Write a function that takes as input a matrix or list of coefficients and calculates
the values of x,y and z that satisfy the three equations using Cramer’s rule.
Here is the sample data we have been using:
2x + y + z = 3
x– y–z=0
x + 2y + z = 0
And we recall from class that the solutions are:
x = 1, y = –2, and z = 3
Linear Regression:
Recall that in LSPR, you have to solve a system of linear equations that look like
this:

 – x = ax + by + c
 – y = dx + ey + f
We did an example in class where we hand calculated how to reduce a system of
five equations with three unknowns down to three equations.
7x1
3x1
2x1
4x1
9x1
–
+
–
+
-
6x2
5x2
2x2
2x2
8x2
+
+
–
+
8x3
2x3
7x3
5x3
7x3
–
–
–
–
–
15
27
20
2
5
=
=
=
=
=
0
0
0
0
0
At this point you should think about how you would translate this into a program
to find those coefficients. Think about ways to use nested loops and list
structures to simplify this process.
Setup the normal equations:
A11 = ∑ai12 A12 = ∑ai1ai2
A22 = ∑ai22
A13 = ∑ai1ai32
B1 = ∑ai1bi
A23 = ∑ai2ai32
B2 = ∑ai2bi
A33 = ∑ai32
B3 = ∑ai3bi
Exercise:
Write a function that takes as its input a matrix or list of the coefficients of n
equations and return a list or matrix of three equations using the method linear
regression outlined in class. The output that your program should have when
passed the sample data is given below.
159x1 – 95x2 + 107x3 – 279 = 0
-95x1 + 133x2 – 138x3 + 31 = 0
107x1 – 138x2 + 191x3 – 231 = 0
Functional programming:
You may ignore this section at your discretion. It is strictly for those that are
interested in programming and would like to see a nicer way to approach certain
programming problems. (Thanks to Scott Moonen!)
Introduction
Imagine you have a list of numbers and want to filter out all even numbers:
The filter function applies the specified function to each member of the list or
array that is passed to it.
For Example:
q = [1,2,5,6,8,12,15,17,20,23,24]
def is_even(x) :
return x % 2 == 0
result = filter(is_even, q)
Would put [2,6,8,12,20,24] into result.
(Remember that the “%” operator is the “modulus” or “remainder” operator. If the
remainder when a number is divided by two is zero, then the number must be
even.)
If we only use is_even once, it’s kind of annoying that we have to define it.
Wouldn’t it be nice if we could just define it inside of the call to filter? We want to
do something like “q = filter(x % 2 == 0, q)”, but that won’t quite work, since the
“x % 2 == 0” is evaluated before the call to filter. We want to pass a function to
filter, not an expression.
We can do this using the lambda operator. lambda allows you to create an
unnamed throw-away function. Try this:
q = [1,2,5,6,8,12,15,17,20,23,24]
result = filter(lambda x : x % 2 == 0, q)
lambda tells Python that a function follows. Before the colon (:) you list the
parameters of the function (in this case, only one, x). After the colon you list what
the function returns. This function doesn’t have a name, and Python disposes of
it as soon as filter is done with it.
You can, however, assign the function to a variable to give it a name. The
following two statements are identical:
def is_odd(x) :
return x % 2 == 1
is_odd = lambda x : x % 2 == 1
map
Here’s another example. The map function applies a function to every item in a
list, and returns the result. This example adds 1 to every element in the list:
result = map(lambda x : x + 1, [1,2,3,4,5])
(At this point, result holds [2,3,4,5,6].)
The map function can also process more than one list at a time, provided they are
the same length. This allows you to do things like add all of the elements in a pair
of lists. Note that our lambda-function here has two parameters:
result = map(lambda x,y : x+y, [1,2,3,4,5], [6,7,8,9,10])
(At this point, result holds [7, 9, 11, 13, 15].)
reduce
Python also provides the reduce function. This is a bit more complicated; you
provide it a list and a function, and it reduces that list by applying the function to
pairs of elements in the list. An example is the best way to understand this. Let’s
say we want to find the sum of all the elements in a list:
sum = reduce(lambda x,y : x+y, [1,2,3,4,5])
reduce will first apply the function to 1,2, yielding 3. It will then apply the
function to 3,3 (the first 3 is the result of adding 1+2), yielding 6. It will then
apply the function to 6,4 (the 6 is the result of adding 3+3), yielding 10. Finally, it
will apply the function to 10,5, yielding 15. The result stored in sum is 15.
Similarly, we can find the cumulative product of all items in a list. The following
example stores 120 (=1*2*3*4*5) in product:
product = reduce(lambda x,y : x*y, [1,2,3,4,5])
Background
One of the advantages of lambda-functions is that they allow you to write very
concise code. A result of this, however, is that the code is literally “dense” with
meaning, and it can be hard to read if you are not accustomed to functional
programming. In our first example, “filter(is_even, q)” was fairly easy to
understand (fortunately, we chose a descriptive function name), while
“filter(lambda x : x % 2 == 0, q)” takes a little longer to comprehend.
If you are a masochistic mathematical geek, you’ll probably enjoy this (I do). If
not, you might still prefer to break things into smaller pieces by defining all of
your functions first and then using them by name. That’s ok! Functional
programming isn’t for everyone.
Another advantage of functional programming is
that it corresponds very closely to the mathematical
notion of functions. Note that our lambda-functions
didn’t store any results into variables, or have any
other sort of side effect. This is not simply “nifty”.
Functions without side effects cause fewer bugs,
because their behavior is deterministic (this is
related to the dictum that you aren’t supposed to
use global variables). It is also much easier to
mathematically prove that such functions do what
you think they are doing.
Here is an example of a uber geeky(See side note)
use of lambda functions to calculate the sample
standard deviation of a list of residuals:
print "Sigma", sqrt(reduce(lambda x,y : x + y ,map(lambda x : (x reduce(lambda x,y : x + y, residuals)/len(residuals))**2,
residuals))/(len(residuals) -1))
Exercise:
Now combine your two functions(Linear Regression and Cramer) to create a
program that can take a set of the coefficients of n equations and return the best
solutions to those equations. You do NOT need to use functional programming to
do this!
The solutions that your program should give for this data are:
x1 = 2.474 x2 = 5.397 x3 = 3.723
Determining the Quality of your fit:
Recall that the residuals are defined as the result when the best fit solutions are
plugged back in to the original equations. For example if we plug our solutions
into the first equation:
7x1
–
6x2
+
8x3
–
15 =
0
We get:
7 (2.474)
–
6(5.397) +
8(3.723) –
15
=
-0.28
Therefore the residual is -0.28
Once we have calculated the residuals we would like to find the standard
deviation of the residuals. Recall that the equation for the sample standard
deviation is:
s
1 N
2
 xi  x 

N  1 i 1
*
Or written in words, the standard deviation is square root of the sum of the
square of the deviations from the mean divided by N -1.
Remember your will need to import the sqrt function from the math library:
from math import sqrt
Exercise:
Add another function to your program to calculate each of your residuals and the
find the standard deviation of the residuals.
Your results for the equations given in the beginning should be:
Residuals are: -0.281,-0.321,-0.109,-0.033,0.118
Standard Deviation 0.180
.
*
Eric has suggested that we use s for sample standard deviation and  for population standard deviation.