Introduction
Linear constraints
Conclusion
Elimination of variables
Presentation for Nonlinear optimisation, equations and least
squares
Anders Märak Leffler
April 18, 2012
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
Outline
1
Introduction
On inequality constraints
A warning re nonlinear constraints
2
Linear constraints
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
3
Conclusion
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
On inequality constraints
A warning re nonlinear constraints
Introduction
Talk on Nocedal & Wright ch 15.2-3
Motivation: reduce number of free variables
Simpler problem.
Complete characterisation (and all eq. constraints) gives
unconstrained problem.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
On inequality constraints
A warning re nonlinear constraints
Introduction
Talk on Nocedal & Wright ch 15.2-3
Motivation: reduce number of free variables
Simpler problem.
Complete characterisation (and all eq. constraints) gives
unconstrained problem.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
On inequality constraints
A warning re nonlinear constraints
Introduction
Talk on Nocedal & Wright ch 15.2-3
Motivation: reduce number of free variables
Simpler problem.
Complete characterisation (and all eq. constraints) gives
unconstrained problem.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
On inequality constraints
A warning re nonlinear constraints
Introduction
Talk on Nocedal & Wright ch 15.2-3
Motivation: reduce number of free variables
Simpler problem.
Complete characterisation (and all eq. constraints) gives
unconstrained problem.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
On inequality constraints
A warning re nonlinear constraints
Introduction
Talk on Nocedal & Wright ch 15.2-3
Motivation: reduce number of free variables
Simpler problem.
Complete characterisation (and all eq. constraints) gives
unconstrained problem.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
On inequality constraints
A warning re nonlinear constraints
Outline
1
Introduction
On inequality constraints
A warning re nonlinear constraints
2
Linear constraints
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
3
Conclusion
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
On inequality constraints
A warning re nonlinear constraints
Complications from inequality constraints
Searching for candidates using necessary conditions looks
”simple”.
Test out a working set W ⊆ I of active ineq-constraints and look
at results.
 
ci1
 
Ie find x: ∇f (x) = λT ∇c(x), c =  ... , ij ∈ W . λ corresponding.
cik
Check that λ ≥ 0.
Problem: which constraints are active?
At most 2|I| such subsets.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
On inequality constraints
A warning re nonlinear constraints
Complications from inequality constraints
Searching for candidates using necessary conditions looks
”simple”.
Test out a working set W ⊆ I of active ineq-constraints and look
at results.
 
ci1
 
Ie find x: ∇f (x) = λT ∇c(x), c =  ... , ij ∈ W . λ corresponding.
cik
Check that λ ≥ 0.
Problem: which constraints are active?
At most 2|I| such subsets.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
On inequality constraints
A warning re nonlinear constraints
Complications from inequality constraints
Searching for candidates using necessary conditions looks
”simple”.
Test out a working set W ⊆ I of active ineq-constraints and look
at results.
 
ci1
 
Ie find x: ∇f (x) = λT ∇c(x), c =  ... , ij ∈ W . λ corresponding.
cik
Check that λ ≥ 0.
Problem: which constraints are active?
At most 2|I| such subsets.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
On inequality constraints
A warning re nonlinear constraints
Complications from inequality constraints
Searching for candidates using necessary conditions looks
”simple”.
Test out a working set W ⊆ I of active ineq-constraints and look
at results.
 
ci1
 
Ie find x: ∇f (x) = λT ∇c(x), c =  ... , ij ∈ W . λ corresponding.
cik
Check that λ ≥ 0.
Problem: which constraints are active?
At most 2|I| such subsets.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
On inequality constraints
A warning re nonlinear constraints
Complications from inequality constraints
Searching for candidates using necessary conditions looks
”simple”.
Test out a working set W ⊆ I of active ineq-constraints and look
at results.
 
ci1
 
Ie find x: ∇f (x) = λT ∇c(x), c =  ... , ij ∈ W . λ corresponding.
cik
Check that λ ≥ 0.
Problem: which constraints are active?
At most 2|I| such subsets.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
On inequality constraints
A warning re nonlinear constraints
Complications from inequality constraints
Searching for candidates using necessary conditions looks
”simple”.
Test out a working set W ⊆ I of active ineq-constraints and look
at results.
 
ci1
 
Ie find x: ∇f (x) = λT ∇c(x), c =  ... , ij ∈ W . λ corresponding.
cik
Check that λ ≥ 0.
Problem: which constraints are active?
At most 2|I| such subsets.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
On inequality constraints
A warning re nonlinear constraints
Outline
1
Introduction
On inequality constraints
A warning re nonlinear constraints
2
Linear constraints
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
3
Conclusion
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
On inequality constraints
A warning re nonlinear constraints
A warning on nonlinear constraints
One can reduce problems such as
minx f (x) = f (x1 , x2 , x3 , x4 )
x1 + x32 − x4 x3 = 0,
st
−x2 + x4 + x32 = 0
to a two-variable problem by x1 = x4 x3 − x32 , x2 = x4 + x32 .
In general: much harder to do mechanically.
Consider
min x 2 + y 2 , st(x − 1)3 = y 2 .
[xy ]
y 2 ≥ 0 and so x − 1 ≥ 0 is obvious only on inspection.
Rest of talk (and ch 15.3): consider linear approximations.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
On inequality constraints
A warning re nonlinear constraints
A warning on nonlinear constraints
One can reduce problems such as
minx f (x) = f (x1 , x2 , x3 , x4 )
x1 + x32 − x4 x3 = 0,
st
−x2 + x4 + x32 = 0
to a two-variable problem by x1 = x4 x3 − x32 , x2 = x4 + x32 .
In general: much harder to do mechanically.
Consider
min x 2 + y 2 , st(x − 1)3 = y 2 .
[xy ]
y 2 ≥ 0 and so x − 1 ≥ 0 is obvious only on inspection.
Rest of talk (and ch 15.3): consider linear approximations.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
On inequality constraints
A warning re nonlinear constraints
A warning on nonlinear constraints
One can reduce problems such as
minx f (x) = f (x1 , x2 , x3 , x4 )
x1 + x32 − x4 x3 = 0,
st
−x2 + x4 + x32 = 0
to a two-variable problem by x1 = x4 x3 − x32 , x2 = x4 + x32 .
In general: much harder to do mechanically.
Consider
min x 2 + y 2 , st(x − 1)3 = y 2 .
[xy ]
y 2 ≥ 0 and so x − 1 ≥ 0 is obvious only on inspection.
Rest of talk (and ch 15.3): consider linear approximations.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
On inequality constraints
A warning re nonlinear constraints
A warning on nonlinear constraints
One can reduce problems such as
minx f (x) = f (x1 , x2 , x3 , x4 )
x1 + x32 − x4 x3 = 0,
st
−x2 + x4 + x32 = 0
to a two-variable problem by x1 = x4 x3 − x32 , x2 = x4 + x32 .
In general: much harder to do mechanically.
Consider
min x 2 + y 2 , st(x − 1)3 = y 2 .
[xy ]
y 2 ≥ 0 and so x − 1 ≥ 0 is obvious only on inspection.
Rest of talk (and ch 15.3): consider linear approximations.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
On inequality constraints
A warning re nonlinear constraints
A warning on nonlinear constraints
One can reduce problems such as
minx f (x) = f (x1 , x2 , x3 , x4 )
x1 + x32 − x4 x3 = 0,
st
−x2 + x4 + x32 = 0
to a two-variable problem by x1 = x4 x3 − x32 , x2 = x4 + x32 .
In general: much harder to do mechanically.
Consider
min x 2 + y 2 , st(x − 1)3 = y 2 .
[xy ]
y 2 ≥ 0 and so x − 1 ≥ 0 is obvious only on inspection.
Rest of talk (and ch 15.3): consider linear approximations.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
On inequality constraints
A warning re nonlinear constraints
A warning on nonlinear constraints
One can reduce problems such as
minx f (x) = f (x1 , x2 , x3 , x4 )
x1 + x32 − x4 x3 = 0,
st
−x2 + x4 + x32 = 0
to a two-variable problem by x1 = x4 x3 − x32 , x2 = x4 + x32 .
In general: much harder to do mechanically.
Consider
min x 2 + y 2 , st(x − 1)3 = y 2 .
[xy ]
y 2 ≥ 0 and so x − 1 ≥ 0 is obvious only on inspection.
Rest of talk (and ch 15.3): consider linear approximations.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Outline
1
Introduction
On inequality constraints
A warning re nonlinear constraints
2
Linear constraints
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
3
Conclusion
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
The problem
We start with
st
min f (x)
cj (x) = 0, j ∈ E
Approximate with m linear constraints
st
min f (x)
aiT x = bi , i = 1, . . . , m
T
With A = a1 . . . am , b = [bi ]
min
f (x),
Anders Märak Leffler
st
Ax = b
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
The problem
We start with
st
min f (x)
cj (x) = 0, j ∈ E
Approximate with m linear constraints
st
min f (x)
aiT x = bi , i = 1, . . . , m
T
With A = a1 . . . am , b = [bi ]
min
f (x),
Anders Märak Leffler
st
Ax = b
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
The problem
We start with
st
min f (x)
cj (x) = 0, j ∈ E
Approximate with m linear constraints
st
min f (x)
aiT x = bi , i = 1, . . . , m
T
With A = a1 . . . am , b = [bi ]
min
f (x),
Anders Märak Leffler
st
Ax = b
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Outline
1
Introduction
On inequality constraints
A warning re nonlinear constraints
2
Linear constraints
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
3
Conclusion
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Simple elimination
Assume A is m × n, full row rank (ie check feasibility, remove
redundant constraints).
Select basis variables, permute to front.
AP = B|N , B basis for Rm .
Similarly for variables. [xB
T
xN ] = P T x
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Simple elimination
Assume A is m × n, full row rank (ie check feasibility, remove
redundant constraints).
Select basis variables, permute to front.
AP = B|N , B basis for Rm .
Similarly for variables. [xB
T
xN ] = P T x
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Simple elimination
Assume A is m × n, full row rank (ie check feasibility, remove
redundant constraints).
Select basis variables, permute to front.
AP = B|N , B basis for Rm .
Similarly for variables. [xB
T
xN ] = P T x
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Simple elimination
Assume A is m × n, full row rank (ie check feasibility, remove
redundant constraints).
Select basis variables, permute to front.
AP = B|N , B basis for Rm .
Similarly for variables. [xB
T
xN ] = P T x
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Simple elimination
Assume A is m × n, full row rank (ie check feasibility, remove
redundant constraints).
Select basis variables, permute to front.
AP = B|N , B basis for Rm .
Similarly for variables. [xB
T
xN ] = P T x
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Then
x
b = Ax = A(PP T )x = (AP)(P T x) = [|{z}
B | |{z}
N ] B = BxB +NxN
xN
m×m m×(m−n)
RHS first part has unique solution, b ∈ Rm .
Thus xB = B −1 b − B −1 NxN
Indirectly: parameter solution of Ax = b, select parameters.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Then
x
B | |{z}
N ] B = BxB +NxN
b = Ax = A(PP T )x = (AP)(P T x) = [|{z}
xN
m×m m×(m−n)
RHS first part has unique solution, b ∈ Rm .
Thus xB = B −1 b − B −1 NxN
Indirectly: parameter solution of Ax = b, select parameters.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Then
x
B | |{z}
N ] B = BxB +NxN
b = Ax = A(PP T )x = (AP)(P T x) = [|{z}
xN
m×m m×(m−n)
RHS first part has unique solution, b ∈ Rm .
Thus xB = B −1 b − B −1 NxN
Indirectly: parameter solution of Ax = b, select parameters.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Then
x
B | |{z}
N ] B = BxB +NxN
b = Ax = A(PP T )x = (AP)(P T x) = [|{z}
xN
m×m m×(m−n)
RHS first part has unique solution, b ∈ Rm .
Thus xB = B −1 b − B −1 NxN
Indirectly: parameter solution of Ax = b, select parameters.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Then
x
B | |{z}
N ] B = BxB +NxN
b = Ax = A(PP T )x = (AP)(P T x) = [|{z}
xN
m×m m×(m−n)
RHS first part has unique solution, b ∈ Rm .
Thus xB = B −1 b − B −1 NxN
Indirectly: parameter solution of Ax = b, select parameters.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Then
x
B | |{z}
N ] B = BxB +NxN
b = Ax = A(PP T )x = (AP)(P T x) = [|{z}
xN
m×m m×(m−n)
RHS first part has unique solution, b ∈ Rm .
Thus xB = B −1 b − B −1 NxN
Indirectly: parameter solution of Ax = b, select parameters.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Then
x
B | |{z}
N ] B = BxB +NxN
b = Ax = A(PP T )x = (AP)(P T x) = [|{z}
xN
m×m m×(m−n)
RHS first part has unique solution, b ∈ Rm .
Thus xB = B −1 b − B −1 NxN
Indirectly: parameter solution of Ax = b, select parameters.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
We had:
min
Now
min
f (x), s t Ax = b, x ∈ Rn
−1
xB
B b − B −1 NxN
f P
=f P
xN
xN
Fewer variables (xN ∈ Rn−m ).
Unconstrained problem.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
We had:
min
Now
min
f (x), s t Ax = b, x ∈ Rn
−1
xB
B b − B −1 NxN
f P
=f P
xN
xN
Fewer variables (xN ∈ Rn−m ).
Unconstrained problem.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
We had:
min
Now
min
f (x), s t Ax = b, x ∈ Rn
−1
xB
B b − B −1 NxN
f P
=f P
xN
xN
Fewer variables (xN ∈ Rn−m ).
Unconstrained problem.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
We had:
min
Now
min
f (x), s t Ax = b, x ∈ Rn
−1
xB
B b − B −1 NxN
f P
=f P
xN
xN
Fewer variables (xN ∈ Rn−m ).
Unconstrained problem.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
We had:
min
Now
min
f (x), s t Ax = b, x ∈ Rn
−1
xB
B b − B −1 NxN
f P
=f P
xN
xN
Fewer variables (xN ∈ Rn−m ).
Unconstrained problem.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
An example
8
3
|
min f (x1 , x2 , x3 , x4 , x5 , x6 ),
st −6 1 9 4 0
6
x=
, x ∈ R6
2 0 −1 6 4
−4
{z
}
A
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
An example
Pick x3 , x6 as basis variables. Permute to front.
1 0 | 8 −6 9 4
6
Px =
0 4 | 3 2 −1 6
−4
In old notation:
1 0
1 0
−1
B=
,B =
,
0 4
0 14
8 −6 9 4
N=
3 2 −1 6
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
An example
Pick x3 , x6 as basis variables. Permute to front.
1 0 | 8 −6 9 4
6
Px =
0 4 | 3 2 −1 6
−4
In old notation:
1 0
1 0
−1
B=
,B =
,
0 4
0 14
8 −6 9 4
N=
3 2 −1 6
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
An example
Pick x3 , x6 as basis variables. Permute to front.
1 0 | 8 −6 9 4
6
Px =
0 4 | 3 2 −1 6
−4
In old notation:
1 0
1 0
−1
B=
,B =
,
0 4
0 14
8 −6 9 4
N=
3 2 −1 6
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Thus
 
x1

8 6 9 4 
0
4
x2 
− 3 1
1
3  
1
−6
− 4 2 x4
4
4
2
{z
}
x5
B −1 b
{z
}
|
B −1 Nx
N
6 − 8x1 − 6x2 − 9x4 − 4x5
=
−1 − 34 x1 − 12 x2 + 14 x4 − 32 x5
1
x
xB = 3 =
x6
0
|
Yielding
min
[x1 ,x2 ,x4 ,x5 ]
f
x1 , x2 , 6 − 8x1 . . . , x4 , x5 , −1 −
Anders Märak Leffler
3
x1 . . .
4
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Thus
 
x1

0
8 6 9 4 
4
x2 
− 3 1
1
1
3  
−6
− 4 2 x4
4
4
2
{z
}
x5
B −1 b
{z
}
|
B −1 Nx
N
6 − 8x1 − 6x2 − 9x4 − 4x5
=
−1 − 34 x1 − 12 x2 + 14 x4 − 32 x5
1
x
xB = 3 =
x6
0
|
Yielding
min
[x1 ,x2 ,x4 ,x5 ]
f
x1 , x2 , 6 − 8x1 . . . , x4 , x5 , −1 −
Anders Märak Leffler
3
x1 . . .
4
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Thus
 
x1

0
8 6 9 4 
4
x2 
− 3 1
1
1
3  
−6
− 4 2 x4
4
4
2
{z
}
x5
B −1 b
{z
}
|
B −1 Nx
N
6 − 8x1 − 6x2 − 9x4 − 4x5
=
−1 − 34 x1 − 12 x2 + 14 x4 − 32 x5
1
x
xB = 3 =
x6
0
|
Yielding
min
[x1 ,x2 ,x4 ,x5 ]
f
x1 , x2 , 6 − 8x1 . . . , x4 , x5 , −1 −
Anders Märak Leffler
3
x1 . . .
4
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Thus
 
x1

0
8 6 9 4 
4
x2 
− 3 1
1
1
3  
−6
− 4 2 x4
4
4
2
{z
}
x5
B −1 b
{z
}
|
B −1 Nx
N
6 − 8x1 − 6x2 − 9x4 − 4x5
=
−1 − 34 x1 − 12 x2 + 14 x4 − 32 x5
1
x
xB = 3 =
x6
0
|
Yielding
min
[x1 ,x2 ,x4 ,x5 ]
f
x1 , x2 , 6 − 8x1 . . . , x4 , x5 , −1 −
Anders Märak Leffler
3
x1 . . .
4
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Outline
1
Introduction
On inequality constraints
A warning re nonlinear constraints
2
Linear constraints
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
3
Conclusion
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Linear algebra review
Given a transformation x → Ax, A is m × n, rank (A) = p we say that
Ran(A) = span{A1 , A2 , . . . , An } = span{Ai1 , Ai2 , . . . , Aip } ⊆ Rm
where Aij linear independent.
rank (A) = #linear independent cols/rows of A = dim Ran(A).
dim Ran(A) + dim Null(A) = n
(dimension theorem, transformation Rn → Rm )
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Linear algebra review
Given a transformation x → Ax, A is m × n, rank (A) = p we say that
Ran(A) = span{A1 , A2 , . . . , An } = span{Ai1 , Ai2 , . . . , Aip } ⊆ Rm
where Aij linear independent.
rank (A) = #linear independent cols/rows of A = dim Ran(A).
dim Ran(A) + dim Null(A) = n
(dimension theorem, transformation Rn → Rm )
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Linear algebra review
Given a transformation x → Ax, A is m × n, rank (A) = p we say that
Ran(A) = span{A1 , A2 , . . . , An } = span{Ai1 , Ai2 , . . . , Aip } ⊆ Rm
where Aij linear independent.
rank (A) = #linear independent cols/rows of A = dim Ran(A).
dim Ran(A) + dim Null(A) = n
(dimension theorem, transformation Rn → Rm )
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Linear algebra review
Given a transformation x → Ax, A is m × n, rank (A) = p we say that
Ran(A) = span{A1 , A2 , . . . , An } = span{Ai1 , Ai2 , . . . , Aip } ⊆ Rm
where Aij linear independent.
rank (A) = #linear independent cols/rows of A = dim Ran(A).
dim Ran(A) + dim Null(A) = n
(dimension theorem, transformation Rn → Rm )
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Linear algebra review
Given a transformation x → Ax, A is m × n, rank (A) = p we say that
Ran(A) = span{A1 , A2 , . . . , An } = span{Ai1 , Ai2 , . . . , Aip } ⊆ Rm
where Aij linear independent.
rank (A) = #linear independent cols/rows of A = dim Ran(A).
dim Ran(A) + dim Null(A) = n
(dimension theorem, transformation Rn → Rm )
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Null(A) = {x : Ax = 0}, a subspace with some basis
{N1 , N2 , . . . N(n−p) } ⊆ Rn .
Let Z = N1 N2 . . . N(n−p)
AZ α = N1 N2 . . . N(n−p) α = 0 0 . . . 0 α = 0, α ∈ R(n−p) .
Geometrically Z α ∈ Null(A), ∀α ∈ Rn−p , coordinate
interpretation:


α1


AZ α = AZ  ...  = A(α1 N1 + . . . + αn N(n−p) ) = 0
α(n−p)
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Null(A) = {x : Ax = 0}, a subspace with some basis
{N1 , N2 , . . . N(n−p) } ⊆ Rn .
Let Z = N1 N2 . . . N(n−p)
AZ α = N1 N2 . . . N(n−p) α = 0 0 . . . 0 α = 0, α ∈ R(n−p) .
Geometrically Z α ∈ Null(A), ∀α ∈ Rn−p , coordinate
interpretation:


α1


AZ α = AZ  ...  = A(α1 N1 + . . . + αn N(n−p) ) = 0
α(n−p)
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Null(A) = {x : Ax = 0}, a subspace with some basis
{N1 , N2 , . . . N(n−p) } ⊆ Rn .
Let Z = N1 N2 . . . N(n−p)
AZ α = N1 N2 . . . N(n−p) α = 0 0 . . . 0 α = 0, α ∈ R(n−p) .
Geometrically Z α ∈ Null(A), ∀α ∈ Rn−p , coordinate
interpretation:


α1


AZ α = AZ  ...  = A(α1 N1 + . . . + αn N(n−p) ) = 0
α(n−p)
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Null(A) = {x : Ax = 0}, a subspace with some basis
{N1 , N2 , . . . N(n−p) } ⊆ Rn .
Let Z = N1 N2 . . . N(n−p)
AZ α = N1 N2 . . . N(n−p) α = 0 0 . . . 0 α = 0, α ∈ R(n−p) .
Geometrically Z α ∈ Null(A), ∀α ∈ Rn−p , coordinate
interpretation:


α1


AZ α = AZ  ...  = A(α1 N1 + . . . + αn N(n−p) ) = 0
α(n−p)
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Null(A) = {x : Ax = 0}, a subspace with some basis
{N1 , N2 , . . . N(n−p) } ⊆ Rn .
Let Z = N1 N2 . . . N(n−p)
AZ α = N1 N2 . . . N(n−p) α = 0 0 . . . 0 α = 0, α ∈ R(n−p) .
Geometrically Z α ∈ Null(A), ∀α ∈ Rn−p , coordinate
interpretation:


α1


AZ α = AZ  ...  = A(α1 N1 + . . . + αn N(n−p) ) = 0
α(n−p)
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Null(A) = {x : Ax = 0}, a subspace with some basis
{N1 , N2 , . . . N(n−p) } ⊆ Rn .
Let Z = N1 N2 . . . N(n−p)
AZ α = N1 N2 . . . N(n−p) α = 0 0 . . . 0 α = 0, α ∈ R(n−p) .
Geometrically Z α ∈ Null(A), ∀α ∈ Rn−p , coordinate
interpretation:


α1


AZ α = AZ  ...  = A(α1 N1 + . . . + αn N(n−p) ) = 0
α(n−p)
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Null(A) = {x : Ax = 0}, a subspace with some basis
{N1 , N2 , . . . N(n−p) } ⊆ Rn .
Let Z = N1 N2 . . . N(n−p)
AZ α = N1 N2 . . . N(n−p) α = 0 0 . . . 0 α = 0, α ∈ R(n−p) .
Geometrically Z α ∈ Null(A), ∀α ∈ Rn−p , coordinate
interpretation:


α1


AZ α = AZ  ...  = A(α1 N1 + . . . + αn N(n−p) ) = 0
α(n−p)
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Outline
1
Introduction
On inequality constraints
A warning re nonlinear constraints
2
Linear constraints
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
3
Conclusion
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Generalised strategy
A as in simple elimination.
Pick Y , n × m, Z , n × (n − m) such that
[Y |Z ] (n × n) has full rank. All columns are linearly independent.
AZ = 0, ie Ran(Z ) ⊆ Null(A)
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Generalised strategy
A as in simple elimination.
Pick Y , n × m, Z , n × (n − m) such that
[Y |Z ] (n × n) has full rank. All columns are linearly independent.
AZ = 0, ie Ran(Z ) ⊆ Null(A)
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Generalised strategy
A as in simple elimination.
Pick Y , n × m, Z , n × (n − m) such that
[Y |Z ] (n × n) has full rank. All columns are linearly independent.
AZ = 0, ie Ran(Z ) ⊆ Null(A)
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Generalised strategy
A as in simple elimination.
Pick Y , n × m, Z , n × (n − m) such that
[Y |Z ] (n × n) has full rank. All columns are linearly independent.
AZ = 0, ie Ran(Z ) ⊆ Null(A)
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Generalised strategy
A as in simple elimination.
Pick Y , n × m, Z , n × (n − m) such that
[Y |Z ] (n × n) has full rank. All columns are linearly independent.
AZ = 0, ie Ran(Z ) ⊆ Null(A)
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
This has implications for Y and Z:
rank (A) = m, full row rank, and Y has independent columns.
rank (A[Y |Z ]) = rank ([AY |0]) = rank (AY ) = m
⇒ Ran(AY ) = Rm = Ran(A).
Recall: A is m × n, So dim Null(A) + dim Ran(A) = n
dim Null(A) = dim Rn − dim Ran(A) = n − rank (A) = n − m, so
Null(A) = Ran(Z )
(Z has n − m linearly independent columns, all in Null(A))
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
This has implications for Y and Z:
rank (A) = m, full row rank, and Y has independent columns.
rank (A[Y |Z ]) = rank ([AY |0]) = rank (AY ) = m
⇒ Ran(AY ) = Rm = Ran(A).
Recall: A is m × n, So dim Null(A) + dim Ran(A) = n
dim Null(A) = dim Rn − dim Ran(A) = n − rank (A) = n − m, so
Null(A) = Ran(Z )
(Z has n − m linearly independent columns, all in Null(A))
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
This has implications for Y and Z:
rank (A) = m, full row rank, and Y has independent columns.
rank (A[Y |Z ]) = rank ([AY |0]) = rank (AY ) = m
⇒ Ran(AY ) = Rm = Ran(A).
Recall: A is m × n, So dim Null(A) + dim Ran(A) = n
dim Null(A) = dim Rn − dim Ran(A) = n − rank (A) = n − m, so
Null(A) = Ran(Z )
(Z has n − m linearly independent columns, all in Null(A))
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
This has implications for Y and Z:
rank (A) = m, full row rank, and Y has independent columns.
rank (A[Y |Z ]) = rank ([AY |0]) = rank (AY ) = m
⇒ Ran(AY ) = Rm = Ran(A).
Recall: A is m × n, So dim Null(A) + dim Ran(A) = n
dim Null(A) = dim Rn − dim Ran(A) = n − rank (A) = n − m, so
Null(A) = Ran(Z )
(Z has n − m linearly independent columns, all in Null(A))
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
This has implications for Y and Z:
rank (A) = m, full row rank, and Y has independent columns.
rank (A[Y |Z ]) = rank ([AY |0]) = rank (AY ) = m
⇒ Ran(AY ) = Rm = Ran(A).
Recall: A is m × n, So dim Null(A) + dim Ran(A) = n
dim Null(A) = dim Rn − dim Ran(A) = n − rank (A) = n − m, so
Null(A) = Ran(Z )
(Z has n − m linearly independent columns, all in Null(A))
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
This has implications for Y and Z:
rank (A) = m, full row rank, and Y has independent columns.
rank (A[Y |Z ]) = rank ([AY |0]) = rank (AY ) = m
⇒ Ran(AY ) = Rm = Ran(A).
Recall: A is m × n, So dim Null(A) + dim Ran(A) = n
dim Null(A) = dim Rn − dim Ran(A) = n − rank (A) = n − m, so
Null(A) = Ran(Z )
(Z has n − m linearly independent columns, all in Null(A))
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
This has implications for Y and Z:
rank (A) = m, full row rank, and Y has independent columns.
rank (A[Y |Z ]) = rank ([AY |0]) = rank (AY ) = m
⇒ Ran(AY ) = Rm = Ran(A).
Recall: A is m × n, So dim Null(A) + dim Ran(A) = n
dim Null(A) = dim Rn − dim Ran(A) = n − rank (A) = n − m, so
Null(A) = Ran(Z )
(Z has n − m linearly independent columns, all in Null(A))
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Express feasible x as x = YxY + ZxZ (ok, as Y has rank m).
Ax = A (YxY + ZxZ ) = AYxY + 0 = b, xY = (AY )−1 b.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Express feasible x as x = YxY + ZxZ (ok, as Y has rank m).
Ax = A (YxY + ZxZ ) = AYxY + 0 = b, xY = (AY )−1 b.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Express feasible x as x = YxY + ZxZ (ok, as Y has rank m).
Ax = A (YxY + ZxZ ) = AYxY + 0 = b, xY = (AY )−1 b.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Express feasible x as x = YxY + ZxZ (ok, as Y has rank m).
Ax = A (YxY + ZxZ ) = AYxY + 0 = b, xY = (AY )−1 b.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Transformed problem
We had:
minx f (x)
s t Ax = b
x ∈ Rn
Now
minxZ
f Y (AY )−1 b + ZxZ
xZ ∈ Rn−m
where Y (AY )−1 b is constant; displacement.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Transformed problem
We had:
minx f (x)
s t Ax = b
x ∈ Rn
Now
minxZ
f Y (AY )−1 b + ZxZ
xZ ∈ Rn−m
where Y (AY )−1 b is constant; displacement.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Transformed problem
We had:
minx f (x)
s t Ax = b
x ∈ Rn
Now
minxZ
f Y (AY )−1 b + ZxZ
xZ ∈ Rn−m
where Y (AY )−1 b is constant; displacement.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Outline
1
Introduction
On inequality constraints
A warning re nonlinear constraints
2
Linear constraints
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
3
Conclusion
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Motivation
There are many ways to select Y , Z .
We want well-conditioned problem (robust w r t perturbations)
QR factorisation of AT (n × m) is one way.
Bad for sparse A (may be expensive), otherwise nice.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Motivation
There are many ways to select Y , Z .
We want well-conditioned problem (robust w r t perturbations)
QR factorisation of AT (n × m) is one way.
Bad for sparse A (may be expensive), otherwise nice.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Motivation
There are many ways to select Y , Z .
We want well-conditioned problem (robust w r t perturbations)
QR factorisation of AT (n × m) is one way.
Bad for sparse A (may be expensive), otherwise nice.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Motivation
There are many ways to select Y , Z .
We want well-conditioned problem (robust w r t perturbations)
QR factorisation of AT (n × m) is one way.
Bad for sparse A (may be expensive), otherwise nice.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Motivation
There are many ways to select Y , Z .
We want well-conditioned problem (robust w r t perturbations)
QR factorisation of AT (n × m) is one way.
Bad for sparse A (may be expensive), otherwise nice.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Motivation
There are many ways to select Y , Z .
We want well-conditioned problem (robust w r t perturbations)
QR factorisation of AT (n × m) is one way.
Bad for sparse A (may be expensive), otherwise nice.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Factorisation:
AT Π = [Q1
Q2 ]
R
0
Where
Π - permutation matrix (n × n)
[Q1 Q2 ] is (n × n), orthogonal.
Q1 (n × m) and Q2 (n × (n − m)) have orthogonal columns.
R is (m × m), upper triangular. pause
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Factorisation:
AT Π = [Q1
Q2 ]
R
0
Where
Π - permutation matrix (n × n)
[Q1 Q2 ] is (n × n), orthogonal.
Q1 (n × m) and Q2 (n × (n − m)) have orthogonal columns.
R is (m × m), upper triangular. pause
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Factorisation:
AT Π = [Q1
Q2 ]
R
0
Where
Π - permutation matrix (n × n)
[Q1 Q2 ] is (n × n), orthogonal.
Q1 (n × m) and Q2 (n × (n − m)) have orthogonal columns.
R is (m × m), upper triangular. pause
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Factorisation:
AT Π = [Q1
Q2 ]
R
0
Where
Π - permutation matrix (n × n)
[Q1 Q2 ] is (n × n), orthogonal.
Q1 (n × m) and Q2 (n × (n − m)) have orthogonal columns.
R is (m × m), upper triangular. pause
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Factorisation:
AT Π = [Q1
Q2 ]
R
0
Where
Π - permutation matrix (n × n)
[Q1 Q2 ] is (n × n), orthogonal.
Q1 (n × m) and Q2 (n × (n − m)) have orthogonal columns.
R is (m × m), upper triangular. pause
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Let Y = Q1 , Z = Q2 .
Then AY = ΠR T , AZ = 0, follows earlier criteria for Y , Z .
Explicitly: x = Q1 R −T ΠT b + Q2 xZ where displacement is easy to
calculate.
This means that x = AT (AAT )−1 b + Q2 xZ . Looks suspiciously
similar to least-squares eq.
Indeed, x solves min kxk22 subject to Ax = b. Why?
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Let Y = Q1 , Z = Q2 .
Then AY = ΠR T , AZ = 0, follows earlier criteria for Y , Z .
Explicitly: x = Q1 R −T ΠT b + Q2 xZ where displacement is easy to
calculate.
This means that x = AT (AAT )−1 b + Q2 xZ . Looks suspiciously
similar to least-squares eq.
Indeed, x solves min kxk22 subject to Ax = b. Why?
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Let Y = Q1 , Z = Q2 .
Then AY = ΠR T , AZ = 0, follows earlier criteria for Y , Z .
Explicitly: x = Q1 R −T ΠT b + Q2 xZ where displacement is easy to
calculate.
This means that x = AT (AAT )−1 b + Q2 xZ . Looks suspiciously
similar to least-squares eq.
Indeed, x solves min kxk22 subject to Ax = b. Why?
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Let Y = Q1 , Z = Q2 .
Then AY = ΠR T , AZ = 0, follows earlier criteria for Y , Z .
Explicitly: x = Q1 R −T ΠT b + Q2 xZ where displacement is easy to
calculate.
This means that x = AT (AAT )−1 b + Q2 xZ . Looks suspiciously
similar to least-squares eq.
Indeed, x solves min kxk22 subject to Ax = b. Why?
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Let Y = Q1 , Z = Q2 .
Then AY = ΠR T , AZ = 0, follows earlier criteria for Y , Z .
Explicitly: x = Q1 R −T ΠT b + Q2 xZ where displacement is easy to
calculate.
This means that x = AT (AAT )−1 b + Q2 xZ . Looks suspiciously
similar to least-squares eq.
Indeed, x solves min kxk22 subject to Ax = b. Why?
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Let Y = Q1 , Z = Q2 .
Then AY = ΠR T , AZ = 0, follows earlier criteria for Y , Z .
Explicitly: x = Q1 R −T ΠT b + Q2 xZ where displacement is easy to
calculate.
This means that x = AT (AAT )−1 b + Q2 xZ . Looks suspiciously
similar to least-squares eq.
Indeed, x solves min kxk22 subject to Ax = b. Why?
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Let Y = Q1 , Z = Q2 .
Then AY = ΠR T , AZ = 0, follows earlier criteria for Y , Z .
Explicitly: x = Q1 R −T ΠT b + Q2 xZ where displacement is easy to
calculate.
This means that x = AT (AAT )−1 b + Q2 xZ . Looks suspiciously
similar to least-squares eq.
Indeed, x solves min kxk22 subject to Ax = b. Why?
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Let Y = Q1 , Z = Q2 .
Then AY = ΠR T , AZ = 0, follows earlier criteria for Y , Z .
Explicitly: x = Q1 R −T ΠT b + Q2 xZ where displacement is easy to
calculate.
This means that x = AT (AAT )−1 b + Q2 xZ . Looks suspiciously
similar to least-squares eq.
Indeed, x solves min kxk22 subject to Ax = b. Why?
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Let Y = Q1 , Z = Q2 .
Then AY = ΠR T , AZ = 0, follows earlier criteria for Y , Z .
Explicitly: x = Q1 R −T ΠT b + Q2 xZ where displacement is easy to
calculate.
This means that x = AT (AAT )−1 b + Q2 xZ . Looks suspiciously
similar to least-squares eq.
Indeed, x solves min kxk22 subject to Ax = b. Why?
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Nocedal & Wright mainly relate QR to Cholesky.
Naive algorithm for A=QR more intuitive.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Nocedal & Wright mainly relate QR to Cholesky.
Naive algorithm for A=QR more intuitive.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Nocedal & Wright mainly relate QR to Cholesky.
Naive algorithm for A=QR more intuitive.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Constructing Q
For A n × m: If column a1 = 0, set Q:s column q1 = 0, otherwise
q1 = a1 ka11 k .
for k = 2, 3,. . . , m
Pk −1
yk = ak − i=1 (qiT ak )qi
if yk = 0 then
qk = 0
else
qk = T 1 1 yk
(yk yk ) 2
R is upper-triangular.
Adapted from Horn & Johnson - Matrix analysis (Cambridge, 1985)
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Constructing Q
For A n × m: If column a1 = 0, set Q:s column q1 = 0, otherwise
q1 = a1 ka11 k .
for k = 2, 3,. . . , m
Pk −1
yk = ak − i=1 (qiT ak )qi
if yk = 0 then
qk = 0
else
qk = T 1 1 yk
(yk yk ) 2
R is upper-triangular.
Adapted from Horn & Johnson - Matrix analysis (Cambridge, 1985)
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Constructing Q
For A n × m: If column a1 = 0, set Q:s column q1 = 0, otherwise
q1 = a1 ka11 k .
for k = 2, 3,. . . , m
Pk −1
yk = ak − i=1 (qiT ak )qi
if yk = 0 then
qk = 0
else
qk = T 1 1 yk
(yk yk ) 2
R is upper-triangular.
Adapted from Horn & Johnson - Matrix analysis (Cambridge, 1985)
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Constructing Q
For A n × m: If column a1 = 0, set Q:s column q1 = 0, otherwise
q1 = a1 ka11 k .
for k = 2, 3,. . . , m
Pk −1
yk = ak − i=1 (qiT ak )qi
if yk = 0 then
qk = 0
else
qk = T 1 1 yk
(yk yk ) 2
R is upper-triangular.
Adapted from Horn & Johnson - Matrix analysis (Cambridge, 1985)
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Constructing Q
For A n × m: If column a1 = 0, set Q:s column q1 = 0, otherwise
q1 = a1 ka11 k .
for k = 2, 3,. . . , m
Pk −1
yk = ak − i=1 (qiT ak )qi
if yk = 0 then
qk = 0
else
qk = T 1 1 yk
(yk yk ) 2
R is upper-triangular.
Adapted from Horn & Johnson - Matrix analysis (Cambridge, 1985)
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Constructing Q
For A n × m: If column a1 = 0, set Q:s column q1 = 0, otherwise
q1 = a1 ka11 k .
for k = 2, 3,. . . , m
Pk −1
yk = ak − i=1 (qiT ak )qi
if yk = 0 then
qk = 0
else
qk = T 1 1 yk
(yk yk ) 2
R is upper-triangular.
Adapted from Horn & Johnson - Matrix analysis (Cambridge, 1985)
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Constructing Q
For A n × m: If column a1 = 0, set Q:s column q1 = 0, otherwise
q1 = a1 ka11 k .
for k = 2, 3,. . . , m
Pk −1
yk = ak − i=1 (qiT ak )qi
if yk = 0 then
qk = 0
else
qk = T 1 1 yk
(yk yk ) 2
R is upper-triangular.
Adapted from Horn & Johnson - Matrix analysis (Cambridge, 1985)
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Geometry
Gram-Schmidt orthonormalisation.
Geometry (figure 15.4) of Nocedal & Wright.
Cf projection onto Ax=0 and displacement.
Ax = 0 define subspace, Ax = b affine.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Geometry
Gram-Schmidt orthonormalisation.
Geometry (figure 15.4) of Nocedal & Wright.
Cf projection onto Ax=0 and displacement.
Ax = 0 define subspace, Ax = b affine.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Geometry
Gram-Schmidt orthonormalisation.
Geometry (figure 15.4) of Nocedal & Wright.
Cf projection onto Ax=0 and displacement.
Ax = 0 define subspace, Ax = b affine.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
The problem
Basis- and nonbasis variables
Linear algebra review
Generalised strategy
QR factorisation
Geometry
Gram-Schmidt orthonormalisation.
Geometry (figure 15.4) of Nocedal & Wright.
Cf projection onto Ax=0 and displacement.
Ax = 0 define subspace, Ax = b affine.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
Conclusion
(Undergraduate) linear algebra is useful.
Anders Märak Leffler
Fundamentals - Elimination of variables
Introduction
Linear constraints
Conclusion
Conclusion
(Undergraduate) linear algebra is useful.
Anders Märak Leffler
Fundamentals - Elimination of variables