Linear Algebra and Simplex
Method
Preliminary Content
1
Vectors
• An n-vector is a row or a column array of n numbers.
1
𝒂=
−1
Column vector of size or dimension n = 2
𝒃= 3
−1 2
4
Row vector of size or dimension n = 4
2
Vectors
3
1
-3
1
2
-2
-1
3
Vectors
• Zero Vector:
𝟎= 0
0 0 0… 0
• ith Unit Vector
𝐮𝑖 = 0
0… 1 … 0
0
• Sum Vector:
𝟏 = 1 1 1 1… 1
4
Vectors
• Vector Addition:
• Scalar Multiplication:
5
Vectors
6
Vectors
7
Vectors
• Inner Product:
Given any two n-vectors a, b:
• Example:
𝐚= 1
−2
−1 , 𝐛 =
1
𝐚𝐛 = −3
• Norm of a Vector:
Given a vector a, ԡ𝒂ԡ is called the norm of a vector
• Euclidean Norm: ԡ𝒂ԡ =
σ𝑛𝑗=1 𝑎𝑗2
8
Matrices
• A matrix is a rectangular array of numbers. If the matrix has m rows
and n columns, it is called an m×n matrix
𝑎12
𝑎31
3 × 2 matrix
• The entry in row i and column j is denoted by 𝑎𝑖𝑗 .
• An m×n matrix A can be represented by its columns or by its rows.
• Every vector is a matrix, but every matrix is not necessarily a vector.
9
Matrices
• Addition of Matrices:
• Matrices must be of the same dimension.
+
=
0 2 4
3 1 0
3
3
1 3 3 + 0 3 2 = 1
6
1 6 5
9 5 3
10 11
+
=
A
B
C
4
5
8
• 𝑐𝑖𝑗 = 𝑎𝑖𝑗 + 𝑏𝑖𝑗 for 𝑖 = 1 … 𝑚, 𝑗 = 1 … 𝑛
10
Matrices
• Multiplication by a Scalar:
𝑎11 𝑎12 𝑎13
𝑘𝑎11
𝑘 ∙ 𝑎21 𝑎22 𝑎23 = 𝑘𝑎21
𝑎31 𝑎32 𝑎33
𝑘𝑎31
𝑘𝑎12
𝑘𝑎22
𝑘𝑎32
𝑘𝑎13
𝑘𝑎23
𝑘𝑎33
×
• Matrix Multiplication:
+
0 2 4 × −1 3
12 −14
−1 3 −3 × 0
1 = −8 12
1 6 5
3 −4
2 −11
× × B
C
A
𝑚×𝑝
𝑚 × 𝑛Must be equal 𝑛 × 𝑝
3×2
3×2
3×3
11
Matrices
• Multiplication by a Scalar:
𝑎11 𝑎12 𝑎13
𝑘𝑎11
𝑘 ∙ 𝑎21 𝑎22 𝑎23 = 𝑘𝑎21
𝑎31 𝑎32 𝑎33
𝑘𝑎31
𝑘𝑎12
𝑘𝑎22
𝑘𝑎32
𝑘𝑎13
𝑘𝑎23
𝑘𝑎33
×
• Matrix Multiplication:
+
0 2 4 × −1 3
12 −14
−1 3 −3 × 0
1 = −8 12
1 6 5
3 −4
2 −11
× × B
C
A
𝑚×𝑝
𝑚 × 𝑛Must be equal 𝑛 × 𝑝
3×2
3×2
3×3
12
Matrices
• Zero Matrix:
0 0
0 0
0 0
𝑚×𝑛
• Identity Matrix:
1 0 0
𝐼3 = 0 1 0
0 0 1
𝑛×𝑛
• If A is m×n, then A𝐼𝑛 = A and 𝐼𝑚 A = A
13
Matrices
• Triangular Matrix:
1 −1
0 3
0 0
2
1
2
Upper Triangular
−3
1
−1
𝑛×𝑛
0 0
7 0
2 1
Lower Triangular
• Transposition:
1 3𝑡
1 0 2
0 1 =
3 1 4
2 4
𝑛
𝑚
×𝑚
×𝑛
14
Matrices
• Triangular Matrix:
1 −1
0 3
0 0
2
1
2
Upper Triangular
−3
1
−1
𝑛×𝑛
0 0
7 0
2 1
Lower Triangular
• Transposition:
1 3𝑡
1 0 2
0 1 =
3 1 4
2 4
𝑛
𝑚
×𝑚
×𝑛
15
Matrices
• Triangular Matrix:
1 −1
0 3
0 0
2
1
2
Upper Triangular
• Transposition:
1 3𝑡
1 0 2
0 1 =
3 1 4
2 4
𝑛
𝑚
×𝑚
×𝑛
−3
1
−1
𝑛×𝑛
0 0
7 0
2 1
Lower Triangular
Results
16
Matrices
• Solving a System of Linear Equations by Elementary Matrix
Operations:
• Example:
17
Matrices
2
1
−1 2
1 −1
1 10
1 8
2 2
1
× 𝑅𝑜𝑤 1
2
1
−1
1
1/2 1/2 5
8
2
1
2
−1
2
Add
𝑅𝑜𝑤 1
to 𝑅𝑜𝑤 2
1 1/2 1/2 5
0 2.5 1.5 13
2
1 −1
2
Subtract
𝑅𝑜𝑤 1
From 𝑅𝑜𝑤 3
3
Add 2 ×
𝑅𝑜𝑤 2 to
𝑅𝑜𝑤 3
1 1/2 1/2
1
5 2
0
1
3/5 26/5 5 × 𝑅𝑜𝑤 2 0
0 −3/2 3/2 −3
0
1 1/2
1/2
5
26/5
0
1
3/5
0
0
24/10 24/5
10
24
× 𝑅𝑜𝑤 3
1 1/2 1/2
5
0
1
3/5 26/5
2
0
0
1
Upper Triangular
1/2 1/2 5
5/2 3/2 13
−3/2 3/2 −3
𝑥1 = 2
𝑥2 = 4
𝑥3 = 2
18
Matrices
2
1
−1 2
1 −1
1 10
1 8
2 2
1
× 𝑅𝑜𝑤 1
2
1
−1
1
1/2 1/2 5
8
2
1
2
−1
2
Add
𝑅𝑜𝑤 1
to 𝑅𝑜𝑤 2
1 1/2 1/2 5
0 2.5 1.5 13
2
1 −1
2
Subtract
𝑅𝑜𝑤 1
From 𝑅𝑜𝑤 3
The process of reducing A into an upper triangular
1with1/2
1/2
1 1/2 1/2 5
5 2
matrix
ones
on
the
diagonal
is
called
3
× 𝑅𝑜𝑤 2 0
Add 2 ×
0
1
3/5
5/2 3/2 13
26/5
5
Gaussian
reduction
of
the
system.
𝑅𝑜𝑤 2 to
𝑅𝑜𝑤 3
0 −3/2 3/2
1 1/2
1/2
5
26/5
0
1
3/5
0
0
24/10 24/5
10
24
× 𝑅𝑜𝑤 3
−3
1 1/2 1/2
5
0
1
3/5 26/5
2
0
0
1
Upper Triangular
0 −3/2 3/2 −3
𝑥1 = 2
𝑥2 = 4
𝑥3 = 2
19
Matrices
• The system can be further reduced as follows:
1 1/2 1/2
5
0
1
3/5 26/5
2
0
0
1
3
Subtract 5 ×
1
𝑅𝑜𝑤 3
0
from𝑅𝑜𝑤 2
0
1/2 1/2 5
4
1
0
2
0
1
𝑅𝑜𝑤 1 −
1
− × 𝑅𝑜𝑤 2
2
1
− × 𝑅𝑜𝑤 3
2
1 0 0 2
0 1 0 4
0 0 1 2
The process of reducing A into an Identity matrix is
called a Gauss-Jordan reduction of the system
20
Matrices
• Matrix Inversion:
• Let A be a square n×n matrix. If B is n×n matrix such that AB = I
and BA = I, then B is called the inverse of A.
• If A has an inverse, A is called nonsingular; otherwise, A is called
singular.
• Condition for Existence of the Inverse:
• Given an n×n matrix A, it has an inverse if and only if the rows/
columns of A are linearly independent.
21
Matrices
• Calculation of the Inverse (𝐀−1 𝐄𝐱𝐢𝐬𝐭):
2
−1
1
1 1 1 0
2 1 ቮ0 1
−1 2 0 0
0
0
1
1 1/2 1/2 1/2 0
0 5/2 3/2 ተ 1/2 1
0 −3/2 3/2 −1/2 0
0
0
1
Objective: Reduce this to the
Identity matrix using row
operations
1
0
0
0 1/5 2/5 −1/5 0
1 3/5 ተ 1/5
2/5 0
0 12/5 −1/5 3/5 1
1 0
0 1
0 0
0 5/12
0 ተ 3/12
1 −1/12
−3/12
3/12
3/12
−1/12
−3/12
5/12
22
Matrices
• Calculation of the Inverse (𝐀−1 𝐄𝐱𝐢𝐬𝐭):
2
−1
1
1 1 1 0
2 1 ቮ0 1
−1 2 0 0
0
0
1
Divide the first row by
2. Add the new first
row to the second row
and subtract it from
the third row
1 1/2 1/2 1/2 0
0 5/2 3/2 ተ 1/2 1
0 −3/2 3/2 −1/2 0
0
0
1
Multiply the second row by 2/5. Multiply the new
second row by -1/2 and add to the first row. Multiply
the new second row by 3/2 and add to the third row
1
0
0
0 1/5 2/5 −1/5 0
1 3/5 ተ 1/5
2/5 0
0 12/5 −1/5 3/5 1
Multiply the third row by
5/12. Multiply the new third
row by - 3/5 and add to the
second row. Multiply the new
third row by -1/5 and add to
the first row.
𝐀−1
1 0
0 1
0 0
0 5/12
0 ተ 3/12
1 −1/12
−3/12
3/12
3/12
−1/12
−3/12
5/12
23
Matrices
• Calculation of the Inverse (𝐀−1 𝐃𝐨𝐞𝐬 𝐍𝐨𝐭 𝐄𝐱𝐢𝐬𝐭):
1
2
1
1 2 1 0
−1 1 ቮ0 1
2 3 0 0
the first row by -2 1
0 Multiply
1
and add to the second row.
0 Multiply the first row 0 −3
by -1 and add to the third
1
0 1
row
2 1 0
−3 ቮ−2 1
1 −1 0
0
0
1
Multiply the second row by -1/3. Then multiply the
new second row by -1 and add to the first row.
Multiply the new second row by -1 and add to the
third row
Column 3 is
dependent on
Columns 1 and 2
(Column 3 = Column
1 + Column 2). OR:
Row 3 is dependent
on Rows 1 and 2
(Row 3 = 5/3Row 1
– 1/3Row 2)
1
0
0
1/3 0
0 1 1/3
1 1 ተ 2/3 −1/3 0
0 0 −5/3 1/3 1
24
Matrices
• Matrix Inversion Fact:
• If A and B are both n×n nonsingular matrices, then AB is nonsingular
and (𝐀𝐁)−1 = 𝐁 −1 𝐀−1
• Determinant of a Matrix:
• Associated with any square matrix A is a number called the
determinant of A (det of A or |A|).
• For a 1 × 1 matrix A = [a11], det A = a11
2
• For a 2 × 2 matrix A =
3
4
, det A = 2 × 5 – 3 × 4 = −2
5
25
Matrices
• Definition:
• If A is an 𝑚 × 𝑚 matrix, then for any values of i and j, the ijth minor
of A (written Aij) is the (𝑚 − 1) × (𝑚 − 1) submatrix of A obtained
by deleting row i and column j of A.
26
Matrices
• Let A be an 𝑚 × 𝑚 matrix:
for any value i.
27
Matrices
• Example:
•
𝑎11
𝑎12
𝑎13
Choose i = 1
det A11 = 5(9) - 8(6) = -3
det A12 = 4(9) - 7(6) = -6
28
Matrices
• Example:
det A13 = 4(8) - 7(5) = -3
29
The Simplex Method
• The development of the simplex method computations is facilitated
by imposing two requirements on the LP model:
1. All the constraints are equations with nonnegative right-hand side.
2. All the variables are nonnegative.
30
The Simplex Method
• Converting (≤) inequalities into equations with nonnegative
right-hand side:
• Add a nonnegative slack (unused) amount variable s to the left-hand
side.
6x + 4x ≤ 24
1
2
6x + 4x + s = 24,
1
2
1
s ≥0
1
31
The Simplex Method
• Converting (≥) inequalities into equations with nonnegative
right-hand side:
• Subtract a nonnegative surplus (excess) amount variable s to the
left-hand side.
x + x ≥ 800
1
2
x + x - s = 800, s ≥ 0
1
2
1
1
32
The Simplex Method
• Computational details of the simplex Algorithm:
• Example:
33
The Simplex Method
34
The Simplex Method
• First step is to rewrite the objective function as follows:
z - 5x1 - 4x2 = 0
• Next, start at the solution (x1, x2) = (0,0)
Identity Matrix
• Iteration# 1:
Non-Basic
Basic
Basic
z
x1
x2
s1
s2
s3
s4
Solution
z
1
-5
-4
0
0
0
0
0
s1
0
6
4
1
0
0
0
24
s2
0
1
2
0
1
0
0
6
s3
0
-1
1
0
0
1
0
1
s4
0
0
1
0
0
0
1
2
35
The Simplex Method
36
The Simplex Method
• The second row in the table shows that increasing x1 by one unit will
improve the objective function by 5 and increasing x2 by one unit
will improve by 4. Hence, x1 should enter the basic solutions
(increased from zero) because it has the most negative value (If the
problem was to minimize, we select the most positive value).
Pivot column
Basic
z
x1
x2
s1
s2
s3
s4
Solution
z
1
-5
-4
0
0
0
0
0
s1
0
6
4
1
0
0
0
24
s2
0
1
2
0
1
0
0
6
s3
0
-1
1
0
0
1
0
1
s4
0
0
1
0
0
0
1
2
37
The Simplex Method
• Consequently, one variable from the basic solution should leave to
the non-basic solution (set to equal zero).
• This variable is decided using the simplex feasibility condition.
Pivot column
Pivot
Row
Pivot Element
Basic
z
x1
x2
s1
s2
s3
s4
Solution
z
1
-5
-4
0
0
0
0
0
s1
0
6
4
1
0
0
0
24
x1 = 6 = 4
s2
0
1
2
0
1
0
0
6
x1 = 1 = 6
s3
0
-1
1
0
0
1
0
1
x1 = −1 = −1
s4
0
0
1
0
0
0
1
2
x1 = 0
Ratio
Test
24
6
1
2
38
×
The Simplex Method
39
The Simplex Method
• Swap x1 with s1:
• Basic Solution: (x1, s2, s3, s4)
• Non-Basic Solution: (x2, s1)
• Update the table using Gauss-Jordan row operations as follows:
40
The Simplex Method
Basic
z
x1
x2
s1
s2
s3
s4
Solution
z
1
-5
-4
0
0
0
0
0
s1
0
6
4
1
0
0
0
24
s2
0
1
2
0
1
0
0
6
s3
0
-1
1
0
0
1
0
1
s4
0
0
1
0
0
0
1
2
• Iteration# 2:
41
The Simplex Method
42
The Simplex Method
Basic
z
x1
x2
s1
s2
s3
s4
Solution
z
1
0
-2/3
5/6
0
0
0
20
x1
0
1
2/3
1/6
0
0
0
4
s2
0
0
4/3
-1/6
1
0
0
2
s3
0
0
5/3
1/6
0
1
0
5
s4
0
0
1
0
0
0
1
2
Ratio
Test
43
The Simplex Method
Basic
z
x1
x2
s1
s2
s3
s4
Solution
z
1
0
-2/3
5/6
0
0
0
20
X1
0
1
2/3
1/6
0
0
0
4
s2
0
0
4/3
-1/6
1
0
0
2
s3
0
0
5/3
1/6
0
1
0
5
s4
0
0
1
0
0
0
1
2
x1
x2
s1
s2
s3
s4
Solution
• Iteration# 3:
Basic
z
z
1
x1
0
x2
0
s3
0
s4
0
44
The Simplex Method
Optimal
45
The Simplex Method
• If any of the simplex-ready equations do not have a slack (or a
variable that can play the role of a slack), an artificial variable, Ri, is
added to form a starting solution at the origin.
• The artificial variables are not part of the original problem
• A modeling “trick” is needed to force them to zero value by
assigning a penalty to the objective function:
• Artificial variable objective function coefficient =
−𝑀
in a maximization problem
ቊ
+𝑀
in a minimization problem
46
The Simplex Method
• M is a sufficiently large positive value (mathematically, M → ∞)
47
The Simplex Method
48
The Simplex Method
• What value of M should we use?
• Answer: depends on the data.
• M must be sufficiently large relative to the original objective
coefficients to force the artificial variables to be zero (which
happens only if a feasible solution exists).
• In the given example, M = 100 is a reasonable number to use.
49
The Simplex Method
Basic
x1
x2
x3
R1
R2
x4
Solution
z
-4
-1
0
-100
-100
0
0
R1
3
1
0
1
0
0
3
R2
4
3
-1
0
1
0
6
x4
1
2
0
0
0
1
4
• Before proceeding with the simplex method computations, the z-row
must be made consistent with the rest of the tableau as follows:
New z-row = Old z-row + (100 * R -row + 100 * R -row)
1
2
Basic
x1
x2
x3
R1
R2
x4
Solution
z
696
399
-100
0
0
0
900
R1
3
1
0
1
0
0
3
R2
4
3
-1
0
1
0
6
x4
1
2
0
0
0
1
4
50
The Simplex Method
• Iteration 1:
Basic
x1
x2
x3
R1
R2
x4
Solution
z
696
399
-100
0
0
0
900
R1
3
1
0
1
0
0
3
R2
4
3
-1
0
1
0
6
x4
1
2
0
0
0
1
4
Basic
x1
x2
x3
R1
R2
x4
Solution
z
0
167
-100
-232
0
0
204
x1
1
1/3
0
1/3
0
0
1
R2
0
5/3
-1
-4/3
1
0
2
x4
0
5/3
0
-1/3
0
1
3
• Iteration 2:
51
The Simplex Method
• After a couple more iterations:
• Optimal Solution:
2
9
17
x= , x= , z=
5
5
5
1
2
52
The Simplex Method
• Using Matrices to Solve Linear Programs:
• Consider this middle iteration:
𝑥𝐵𝑉
Basic
z
x1
x2
s1
s2
s3
s4
Solution
z
1
0
-2/3
5/6
0
0
0
20
X1
0
1
2/3
1/6
0
0
0
4
s2
0
0
4/3
-1/6
1
0
0
2
s3
0
0
5/3
1/6
0
1
0
5
s4
0
0
1
0
0
0
1
2
𝑋1
𝑠2
𝑥𝐵𝑉 = 𝑠
3
𝑠4
Variables must be added in the same
sequance as found in the LHS of the
current iteration
53
The Simplex Method
• Using Matrices to Solve Linear Programs:
𝑥𝑁𝐵𝑉
Basic
z
x1
x2
s1
s2
s3
s4
Solution
z
1
0
-2/3
5/6
0
0
0
20
X1
0
1
2/3
1/6
0
0
0
4
s2
0
0
4/3
-1/6
1
0
0
2
s3
0
0
5/3
1/6
0
1
0
5
s4
0
0
1
0
0
0
1
2
𝑥2
𝑥𝑁𝐵𝑉 = 𝑠
1
54
The Simplex Method
• Using Matrices to Solve Linear Programs:
𝑐1
𝑐2
𝑐3
𝑐4
𝑐5
𝑐6
5
0
𝑐𝐵𝑉 =
0
0
Values must be added in the same
sequance as found in the LHS of the
current iteration
55
The Simplex Method
• Using Matrices to Solve Linear Programs:
6
𝐴= 1
−1
0
4
2
1
1
1
0
0
0
0
1
0
0
0
0
1
056
0
0
0
1
The Simplex Method
• Using Matrices to Solve Linear Programs:
𝑎1
𝑎2
𝑎3 𝑎4
𝑎5
𝑎6
• Another representation of A:
𝐴 = 𝑎1
𝑎2
𝑎3
𝑎4
𝑎5
𝑎6
57
The Simplex Method
• Using Matrices to Solve Linear Programs:
𝑎1
6
𝐵= 1
−1
0
𝑎4
0
1
0
0
0
0
1
0
0
0
0
1
𝑎5
𝑎6
Columns must be added in the same
sequance as found in the LHS of the
current iteration
58
The Simplex Method
• Using Matrices to Solve Linear Programs:
𝑎2
4
𝑁= 2
1
1
1
0
0
0
𝑎3
59
The Simplex Method
• Using Matrices to Solve Linear Programs:
Basic
z
x1
x2
s1
s2
s3
s4
Solution
z
1
0
-2/3
5/6
0
0
0
20
X1
0
1
2/3
1/6
0
0
0
4
s2
0
0
4/3
-1/6
1
0
0
2
s3
0
0
5/3
1/6
0
1
0
5
s4
0
0
1
0
0
0
1
2
1/6
−1/6
−1
𝐵 =
1/6
0
0
1
0
0
0
0
1
0
0
0
0
1
𝐵 −1 will always consist of the columns
of the current iteration found under
the basic variables of iteration 0.
60
The Simplex Method
• Using Matrices to Solve Linear Programs:
• In case the columns of the basic variables in iteration 0 were not
showing the typical arrangement of identity matrix, you will have to
rearrange 𝐵−1 columns to match the correct arrangement of the
original identity Matrix of iteration 0.
• An example is shown next:
61
The Simplex Method
• Iteration 0:
Since the columns of the basic variables are not in the typical arrangement of an
identity matrix, we must be careful when constructing 𝐵 −1 in subsequent iterations
Basic
x1
x2
x3
x4
x5
x6
x7
Solution
z
-1
202
-100
-100
0
0
0
300
x6
1
1
-1
0
0
1
0
3
x7
-1
1
0
-1
0
0
1
1
x5
0
1
0
0
1
0
0
2
• Iteration 1:
Basic
x1
x2
x3
x4
x5
x6
x7
Solution
z
201
0
-200
102
0
0
-402
98
x6
2
0
-1
1
0
1
-1
1
x2
-1
1
0
-1
0
0
1
1
x5
1
0
0
1
1
0
-1
2
These are the columns of 𝐵 −1
, but
62
they are not arranged!
The Simplex Method
• Iteration 0:
Since the columns of the basic variables are not in the typical arrangement of an
identity matrix, we must be careful when constructing 𝐵 −1 in subsequent iterations
Basic
x1
x2
x3
x4
x5
x6
x7
Solution
z
-1
202
-100
-100
0
0
0
300
x6
1
1
-1
0
0
1
0
3
x7
-1
1
0
-1
0
0
1
1
x5
0
1
0
0
1
0
0
2
• Iteration 1:
Hence, the first column of 𝐵 −1 will be the column under 𝑥6 , the scond column is the
one under 𝑥7 , and the third column is under 𝑥5
Basic
x1
x2
x3
x4
x5
x6
x7
Solution
z
201
0
-200
102
0
0
-402
98
x6
2
0
-1
1
0
1
-1
1
x2
-1
1
0
-1
0
0
1
1
x5
1
0
0
1
1
0
-1
2
These are the columns of 𝐵 −1
, but
63
they are not arranged!
The Simplex Method
• Iteration 0:
Since the columns of the basic variables are not in the typical arrangement of an
identity matrix, we must be careful when constructing 𝐵 −1 in subsequent iterations
Basic
x1
x2
x3
x4
x5
x6
x7
Solution
z
-1
202
-100
-100
0
0
0
300
x6
1
1
-1
0
0
1
0
3
x7
-1
1
0
-1
0
0
1
1
x5
0
1
0
0
1
0
0
2
z
201
0
-200
102
0
0
-402
98
x6
2
0
-1
1
0
1
-1
1
x2
-1
1
0
-1
0
0
1
1
x5
1
0
0
1
1
0
-1
2
1
𝐵−1 =1: 0
• Iteration
Basic
x1 0
−1 0
Hence, the first column of 𝐵 −1 will be the column under 𝑥6 , the scond column is the
1 0
one under 𝑥7 , and the third column is under 𝑥5
−1
1 x3
x2
x4
x5
x6
x7
Solution
These are the columns of 𝐵 −1
, but
64
they are not arranged!
The Simplex Method
• Using Matrices to Solve Linear Programs:
24
𝐛= 6
1
2
65