The problem
Non-CG methods
CG methods
On inequality-constraints
Quadratic programming
Presentation for Nonlinear optimisation, equations and least
squares
Anders Märak Leffler
April 25, 2012
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Outline
1
The problem
2
Non-CG methods
3
CG methods
4
On inequality-constraints
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
The problem
minx q(x) = 12 x T Gx + x T c
s t c(x) = 0
where G is n × n symmetric, c is n × 1.
Linearise constraints, consider


−∇c1 (x)T −


..
J(x) = 

.
−∇cp (x)T −
In point x, we get aiT x = bi for each constraint i = 1, . . . , p.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
The problem
minx q(x) = 12 x T Gx + x T c
s t c(x) = 0
where G is n × n symmetric, c is n × 1.
Linearise constraints, consider


−∇c1 (x)T −


..
J(x) = 

.
−∇cp (x)T −
In point x, we get aiT x = bi for each constraint i = 1, . . . , p.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
The problem
minx q(x) = 12 x T Gx + x T c
s t c(x) = 0
where G is n × n symmetric, c is n × 1.
Linearise constraints, consider


−∇c1 (x)T −


..
J(x) = 

.
−∇cp (x)T −
In point x, we get aiT x = bi for each constraint i = 1, . . . , p.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Reducing to m independent constraints gives
minx q(x) = 21 x T Gx + x T c
s t Ax = b
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Necessary conditions
Lagrange function:
L(x, λ) = q(x) − λT (Ax − b) =
1 T
x Gx + x T c − x T AT λ + bT λ
2
If (x ∗ , λ∗ ) optimal, then ∇x L(x ∗ , λ∗ ) = 0 and Ax ∗ = b. (λ∗
unconstrained).
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Necessary conditions
Lagrange function:
L(x, λ) = q(x) − λT (Ax − b) =
1 T
x Gx + x T c − x T AT λ + bT λ
2
If (x ∗ , λ∗ ) optimal, then ∇x L(x ∗ , λ∗ ) = 0 and Ax ∗ = b. (λ∗
unconstrained).
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Necessary conditions
Lagrange function:
L(x, λ) = q(x) − λT (Ax − b) =
1 T
x Gx + x T c − x T AT λ + bT λ
2
If (x ∗ , λ∗ ) optimal, then ∇x L(x ∗ , λ∗ ) = 0 and Ax ∗ = b. (λ∗
unconstrained).
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Necessary conditions
Lagrange function:
L(x, λ) = q(x) − λT (Ax − b) =
1 T
x Gx + x T c − x T AT λ + bT λ
2
If (x ∗ , λ∗ ) optimal, then ∇x L(x ∗ , λ∗ ) = 0 and Ax ∗ = b. (λ∗
unconstrained).
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Necessary conditions
Lagrange function:
L(x, λ) = q(x) − λT (Ax − b) =
1 T
x Gx + x T c − x T AT λ + bT λ
2
If (x ∗ , λ∗ ) optimal, then ∇x L(x ∗ , λ∗ ) = 0 and Ax ∗ = b. (λ∗
unconstrained).
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Condition (i): ∇x L(x ∗ , λ∗ ) = Gx ∗ + c − AT λ∗ = 0
⇔ Gx ∗ − AT λ∗ = −c
Condition (ii): Ax ∗ + 0T λ∗ = b
Together:
G
A
−AT
0
Anders Märak Leffler
∗ x
−c
=
λ∗
b
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Condition (i): ∇x L(x ∗ , λ∗ ) = Gx ∗ + c − AT λ∗ = 0
⇔ Gx ∗ − AT λ∗ = −c
Condition (ii): Ax ∗ + 0T λ∗ = b
Together:
G
A
−AT
0
Anders Märak Leffler
∗ x
−c
=
λ∗
b
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Condition (i): ∇x L(x ∗ , λ∗ ) = Gx ∗ + c − AT λ∗ = 0
⇔ Gx ∗ − AT λ∗ = −c
Condition (ii): Ax ∗ + 0T λ∗ = b
Together:
G
A
−AT
0
Anders Märak Leffler
∗ x
−c
=
λ∗
b
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Condition (i): ∇x L(x ∗ , λ∗ ) = Gx ∗ + c − AT λ∗ = 0
⇔ Gx ∗ − AT λ∗ = −c
Condition (ii): Ax ∗ + 0T λ∗ = b
Together:
G
A
−AT
0
Anders Märak Leffler
∗ x
−c
=
λ∗
b
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Restating with h = Ax − b, g = c + Gx, p = x ∗ − x:
g
G AT −p
=
λ∗
h
A 0
p difference between current point and optimum
h constraint violation
g
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Restating with h = Ax − b, g = c + Gx, p = x ∗ − x:
g
G AT −p
=
λ∗
h
A 0
p difference between current point and optimum
h constraint violation
g
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Restating with h = Ax − b, g = c + Gx, p = x ∗ − x:
g
G AT −p
=
λ∗
h
A 0
p difference between current point and optimum
h constraint violation
g
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Restating with h = Ax − b, g = c + Gx, p = x ∗ − x:
g
G AT −p
=
λ∗
h
A 0
p difference between current point and optimum
h constraint violation
g
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Restating with h = Ax − b, g = c + Gx, p = x ∗ − x:
g
G AT −p
=
λ∗
h
A 0
p difference between current point and optimum
h constraint violation
g
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Sufficient conditions
Let Z be n × (n − m), rank (Z ) = n − m, AZ = 0 (null space of A)
Theorem (N & W Lemma 16.1)
rank (A) = m, Z T GZ > 0 ⇒ K nonsingular.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Sufficient conditions
Let Z be n × (n − m), rank (Z ) = n − m, AZ = 0 (null space of A)
Theorem (N & W Lemma 16.1)
rank (A) = m, Z T GZ > 0 ⇒ K nonsingular.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Sufficient conditions
Let Z be n × (n − m), rank (Z ) = n − m, AZ = 0 (null space of A)
Theorem (N & W Lemma 16.1)
rank (A) = m, Z T GZ > 0 ⇒ K nonsingular.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Theorem (N & W Theorem 16.2)
rank (A) = m, Z T GZ > 0 ⇒ (x ∗ , λ∗ ) unique global solution.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
With Z and Z T GZ > 0 we have unique global min.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
With Z and Z T GZ > 0 we have unique global min.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Factoring
Note:
GT
K =
(AT )T
T
AT
0
T
=K
Cholesky? No, K indefinite. (thm 16.3)
LU? Perhaps. Ignores symmetry of K.
Symmetric indefinite factorisation, Schur-complements,
null-space-based (like last week)
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Factoring
Note:
GT
K =
(AT )T
T
AT
0
T
=K
Cholesky? No, K indefinite. (thm 16.3)
LU? Perhaps. Ignores symmetry of K.
Symmetric indefinite factorisation, Schur-complements,
null-space-based (like last week)
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Factoring
Note:
GT
K =
(AT )T
T
AT
0
T
=K
Cholesky? No, K indefinite. (thm 16.3)
LU? Perhaps. Ignores symmetry of K.
Symmetric indefinite factorisation, Schur-complements,
null-space-based (like last week)
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Factoring
Note:
GT
K =
(AT )T
T
AT
0
T
=K
Cholesky? No, K indefinite. (thm 16.3)
LU? Perhaps. Ignores symmetry of K.
Symmetric indefinite factorisation, Schur-complements,
null-space-based (like last week)
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Factoring
Note:
GT
K =
(AT )T
T
AT
0
T
=K
Cholesky? No, K indefinite. (thm 16.3)
LU? Perhaps. Ignores symmetry of K.
Symmetric indefinite factorisation, Schur-complements,
null-space-based (like last week)
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Factoring
Note:
GT
K =
(AT )T
T
AT
0
T
=K
Cholesky? No, K indefinite. (thm 16.3)
LU? Perhaps. Ignores symmetry of K.
Symmetric indefinite factorisation, Schur-complements,
null-space-based (like last week)
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Factoring
Note:
GT
K =
(AT )T
T
AT
0
T
=K
Cholesky? No, K indefinite. (thm 16.3)
LU? Perhaps. Ignores symmetry of K.
Symmetric indefinite factorisation, Schur-complements,
null-space-based (like last week)
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Symmetric indefinite factorisation
Additional assumptions: none. (K symmetric)
P T KP = LBLT , where
P permutation
L lower triangular
B block diagonal ((1 × 1) or (2 × 2) blocks)
Method:
1
Factor
2
solve Lz = P T
3
4
5
g
h
solve By = z
T
solve
L z̄ = y
−p
let
= P z̄
λ∗
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Symmetric indefinite factorisation
Additional assumptions: none. (K symmetric)
P T KP = LBLT , where
P permutation
L lower triangular
B block diagonal ((1 × 1) or (2 × 2) blocks)
Method:
1
Factor
2
solve Lz = P T
3
4
5
g
h
solve By = z
T
solve
L z̄ = y
−p
let
= P z̄
λ∗
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Symmetric indefinite factorisation
Additional assumptions: none. (K symmetric)
P T KP = LBLT , where
P permutation
L lower triangular
B block diagonal ((1 × 1) or (2 × 2) blocks)
Method:
1
Factor
2
solve Lz = P T
3
4
5
g
h
solve By = z
T
solve
L z̄ = y
−p
let
= P z̄
λ∗
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Symmetric indefinite factorisation
Additional assumptions: none. (K symmetric)
P T KP = LBLT , where
P permutation
L lower triangular
B block diagonal ((1 × 1) or (2 × 2) blocks)
Method:
1
Factor
2
solve Lz = P T
3
4
5
g
h
solve By = z
T
solve
L z̄ = y
−p
let
= P z̄
λ∗
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Symmetric indefinite factorisation
Additional assumptions: none. (K symmetric)
P T KP = LBLT , where
P permutation
L lower triangular
B block diagonal ((1 × 1) or (2 × 2) blocks)
Method:
1
Factor
2
solve Lz = P T
3
4
5
g
h
solve By = z
T
solve
L z̄ = y
−p
let
= P z̄
λ∗
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Symmetric indefinite factorisation
Additional assumptions: none. (K symmetric)
P T KP = LBLT , where
P permutation
L lower triangular
B block diagonal ((1 × 1) or (2 × 2) blocks)
Method:
1
Factor
2
solve Lz = P T
3
4
5
g
h
solve By = z
T
solve
L z̄ = y
−p
let
= P z̄
λ∗
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Symmetric indefinite factorisation
Additional assumptions: none. (K symmetric)
P T KP = LBLT , where
P permutation
L lower triangular
B block diagonal ((1 × 1) or (2 × 2) blocks)
Method:
1
Factor
2
solve Lz = P T
3
4
5
g
h
solve By = z
T
solve
L z̄ = y
−p
let
= P z̄
λ∗
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Symmetric indefinite factorisation
Additional assumptions: none. (K symmetric)
P T KP = LBLT , where
P permutation
L lower triangular
B block diagonal ((1 × 1) or (2 × 2) blocks)
Method:
1
Factor
2
solve Lz = P T
3
4
5
g
h
solve By = z
T
solve
L z̄ = y
−p
let
= P z̄
λ∗
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Symmetric indefinite factorisation
Additional assumptions: none. (K symmetric)
P T KP = LBLT , where
P permutation
L lower triangular
B block diagonal ((1 × 1) or (2 × 2) blocks)
Method:
1
Factor
2
solve Lz = P T
3
4
5
g
h
solve By = z
T
solve
L z̄ = y
−p
let
= P z̄
λ∗
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Symmetric indefinite factorisation
Additional assumptions: none. (K symmetric)
P T KP = LBLT , where
P permutation
L lower triangular
B block diagonal ((1 × 1) or (2 × 2) blocks)
Method:
1
Factor
2
solve Lz = P T
3
4
5
g
h
solve By = z
T
solve
L z̄ = y
−p
let
= P z̄
λ∗
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Schur-Complement method
Additional assumption: G > 0.
AG−1 −Gp + AT λ∗ = AG−1 g ⇒ AG−1 AT λ∗ = AG−1 g
(p ∈ Null(A))
AG−1 Gp = 0
⇒ (AG−1 AT )λx = (AG−1 g − h)
Gp = AT λ∗ − g
Good if G well-conditioned, easily inverted, AG−1 AT small
(rank(A) = m small)
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Schur-Complement method
Additional assumption: G > 0.
AG−1 −Gp + AT λ∗ = AG−1 g ⇒ AG−1 AT λ∗ = AG−1 g
(p ∈ Null(A))
AG−1 Gp = 0
⇒ (AG−1 AT )λx = (AG−1 g − h)
Gp = AT λ∗ − g
Good if G well-conditioned, easily inverted, AG−1 AT small
(rank(A) = m small)
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Schur-Complement method
Additional assumption: G > 0.
AG−1 −Gp + AT λ∗ = AG−1 g ⇒ AG−1 AT λ∗ = AG−1 g
(p ∈ Null(A))
AG−1 Gp = 0
⇒ (AG−1 AT )λx = (AG−1 g − h)
Gp = AT λ∗ − g
Good if G well-conditioned, easily inverted, AG−1 AT small
(rank(A) = m small)
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Schur-Complement method
Additional assumption: G > 0.
AG−1 −Gp + AT λ∗ = AG−1 g ⇒ AG−1 AT λ∗ = AG−1 g
(p ∈ Null(A))
AG−1 Gp = 0
⇒ (AG−1 AT )λx = (AG−1 g − h)
Gp = AT λ∗ − g
Good if G well-conditioned, easily inverted, AG−1 AT small
(rank(A) = m small)
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Schur-Complement method
Additional assumption: G > 0.
AG−1 −Gp + AT λ∗ = AG−1 g ⇒ AG−1 AT λ∗ = AG−1 g
(p ∈ Null(A))
AG−1 Gp = 0
⇒ (AG−1 AT )λx = (AG−1 g − h)
Gp = AT λ∗ − g
Good if G well-conditioned, easily inverted, AG−1 AT small
(rank(A) = m small)
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Schur-Complement method
Additional assumption: G > 0.
AG−1 −Gp + AT λ∗ = AG−1 g ⇒ AG−1 AT λ∗ = AG−1 g
(p ∈ Null(A))
AG−1 Gp = 0
⇒ (AG−1 AT )λx = (AG−1 g − h)
Gp = AT λ∗ − g
Good if G well-conditioned, easily inverted, AG−1 AT small
(rank(A) = m small)
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Schur-Complement method
Additional assumption: G > 0.
AG−1 −Gp + AT λ∗ = AG−1 g ⇒ AG−1 AT λ∗ = AG−1 g
(p ∈ Null(A))
AG−1 Gp = 0
⇒ (AG−1 AT )λx = (AG−1 g − h)
Gp = AT λ∗ − g
Good if G well-conditioned, easily inverted, AG−1 AT small
(rank(A) = m small)
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Schur-Complement method
Additional assumption: G > 0.
AG−1 −Gp + AT λ∗ = AG−1 g ⇒ AG−1 AT λ∗ = AG−1 g
(p ∈ Null(A))
AG−1 Gp = 0
⇒ (AG−1 AT )λx = (AG−1 g − h)
Gp = AT λ∗ − g
Good if G well-conditioned, easily inverted, AG−1 AT small
(rank(A) = m small)
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Schur-Complement method
Additional assumption: G > 0.
AG−1 −Gp + AT λ∗ = AG−1 g ⇒ AG−1 AT λ∗ = AG−1 g
(p ∈ Null(A))
AG−1 Gp = 0
⇒ (AG−1 AT )λx = (AG−1 g − h)
Gp = AT λ∗ − g
Good if G well-conditioned, easily inverted, AG−1 AT small
(rank(A) = m small)
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Null-space method
Additional assumption: none. (recall, though: rank(A)=m,
Z T GZ > 0)
Z as above, Y ∈ Rn×m : rank ([Y |Z ]) = n.
Partition p = YpY + ZpZ .
(as last week)
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Null-space method
Additional assumption: none. (recall, though: rank(A)=m,
Z T GZ > 0)
Z as above, Y ∈ Rn×m : rank ([Y |Z ]) = n.
Partition p = YpY + ZpZ .
(as last week)
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Null-space method
Additional assumption: none. (recall, though: rank(A)=m,
Z T GZ > 0)
Z as above, Y ∈ Rn×m : rank ([Y |Z ]) = n.
Partition p = YpY + ZpZ .
(as last week)
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Null-space method
Additional assumption: none. (recall, though: rank(A)=m,
Z T GZ > 0)
Z as above, Y ∈ Rn×m : rank ([Y |Z ]) = n.
Partition p = YpY + ZpZ .
(as last week)
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Null-space method
Additional assumption: none. (recall, though: rank(A)=m,
Z T GZ > 0)
Z as above, Y ∈ Rn×m : rank ([Y |Z ]) = n.
Partition p = YpY + ZpZ .
(as last week)
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Ap = −h ⇒ AYpY + AZpZ = −h ⇒ AYpY = −h
rank (AY ) = m and ran(AY ) ⊆ Rm so pY unique.
−G (YpY + ZpZ ) + AT λ∗ = g.
⇒ −Z T GYpY − Z T GZpZ + Z T AT λ∗ = Z T g
⇒ (Z T GZ )pZ = −Z T GYpY − Z T g
Then (AY )T λ∗ = Y T (g + Gp) easily found.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Ap = −h ⇒ AYpY + AZpZ = −h ⇒ AYpY = −h
rank (AY ) = m and ran(AY ) ⊆ Rm so pY unique.
−G (YpY + ZpZ ) + AT λ∗ = g.
⇒ −Z T GYpY − Z T GZpZ + Z T AT λ∗ = Z T g
⇒ (Z T GZ )pZ = −Z T GYpY − Z T g
Then (AY )T λ∗ = Y T (g + Gp) easily found.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Ap = −h ⇒ AYpY + AZpZ = −h ⇒ AYpY = −h
rank (AY ) = m and ran(AY ) ⊆ Rm so pY unique.
−G (YpY + ZpZ ) + AT λ∗ = g.
⇒ −Z T GYpY − Z T GZpZ + Z T AT λ∗ = Z T g
⇒ (Z T GZ )pZ = −Z T GYpY − Z T g
Then (AY )T λ∗ = Y T (g + Gp) easily found.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Ap = −h ⇒ AYpY + AZpZ = −h ⇒ AYpY = −h
rank (AY ) = m and ran(AY ) ⊆ Rm so pY unique.
−G (YpY + ZpZ ) + AT λ∗ = g.
⇒ −Z T GYpY − Z T GZpZ + Z T AT λ∗ = Z T g
⇒ (Z T GZ )pZ = −Z T GYpY − Z T g
Then (AY )T λ∗ = Y T (g + Gp) easily found.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Ap = −h ⇒ AYpY + AZpZ = −h ⇒ AYpY = −h
rank (AY ) = m and ran(AY ) ⊆ Rm so pY unique.
−G (YpY + ZpZ ) + AT λ∗ = g.
⇒ −Z T GYpY − Z T GZpZ + Z T AT λ∗ = Z T g
⇒ (Z T GZ )pZ = −Z T GYpY − Z T g
Then (AY )T λ∗ = Y T (g + Gp) easily found.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Ap = −h ⇒ AYpY + AZpZ = −h ⇒ AYpY = −h
rank (AY ) = m and ran(AY ) ⊆ Rm so pY unique.
−G (YpY + ZpZ ) + AT λ∗ = g.
⇒ −Z T GYpY − Z T GZpZ + Z T AT λ∗ = Z T g
⇒ (Z T GZ )pZ = −Z T GYpY − Z T g
Then (AY )T λ∗ = Y T (g + Gp) easily found.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Ap = −h ⇒ AYpY + AZpZ = −h ⇒ AYpY = −h
rank (AY ) = m and ran(AY ) ⊆ Rm so pY unique.
−G (YpY + ZpZ ) + AT λ∗ = g.
⇒ −Z T GYpY − Z T GZpZ + Z T AT λ∗ = Z T g
⇒ (Z T GZ )pZ = −Z T GYpY − Z T g
Then (AY )T λ∗ = Y T (g + Gp) easily found.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Ap = −h ⇒ AYpY + AZpZ = −h ⇒ AYpY = −h
rank (AY ) = m and ran(AY ) ⊆ Rm so pY unique.
−G (YpY + ZpZ ) + AT λ∗ = g.
⇒ −Z T GYpY − Z T GZpZ + Z T AT λ∗ = Z T g
⇒ (Z T GZ )pZ = −Z T GYpY − Z T g
Then (AY )T λ∗ = Y T (g + Gp) easily found.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Ap = −h ⇒ AYpY + AZpZ = −h ⇒ AYpY = −h
rank (AY ) = m and ran(AY ) ⊆ Rm so pY unique.
−G (YpY + ZpZ ) + AT λ∗ = g.
⇒ −Z T GYpY − Z T GZpZ + Z T AT λ∗ = Z T g
⇒ (Z T GZ )pZ = −Z T GYpY − Z T g
Then (AY )T λ∗ = Y T (g + Gp) easily found.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Advantage: (Z T GZ ) > 0 and symmetric (G is). ⇒ Cholesky
factorisation works.
Problem: Z not unique. QR factorisation, Z has orthonormal
columns, gives well-conditioned problem. Might be expensive.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Advantage: (Z T GZ ) > 0 and symmetric (G is). ⇒ Cholesky
factorisation works.
Problem: Z not unique. QR factorisation, Z has orthonormal
columns, gives well-conditioned problem. Might be expensive.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Advantage: (Z T GZ ) > 0 and symmetric (G is). ⇒ Cholesky
factorisation works.
Problem: Z not unique. QR factorisation, Z has orthonormal
columns, gives well-conditioned problem. Might be expensive.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Advantage: (Z T GZ ) > 0 and symmetric (G is). ⇒ Cholesky
factorisation works.
Problem: Z not unique. QR factorisation, Z has orthonormal
columns, gives well-conditioned problem. Might be expensive.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
N & W recommendations
Schur-complement - if G > 0, AG−1 AT easy to compute.
Null-space method otherwise.(if Z T GZ ∈ R(n−m)×(n−m) easier to
factorise than G ∈ Rn×n ).
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
N & W recommendations
Schur-complement - if G > 0, AG−1 AT easy to compute.
Null-space method otherwise.(if Z T GZ ∈ R(n−m)×(n−m) easier to
factorise than G ∈ Rn×n ).
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
N & W recommendations
Schur-complement - if G > 0, AG−1 AT easy to compute.
Null-space method otherwise.(if Z T GZ ∈ R(n−m)×(n−m) easier to
factorise than G ∈ Rn×n ).
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Linear conjugate gradient methods
Recall: linear CG method for solving Ax = b, equivalently
min Φ(x) =
x
1 T
x Ax − bT x
2
Minimize over expanding (affine) subspaces
Memory efficient.
Works for A > 0
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Linear conjugate gradient methods
Recall: linear CG method for solving Ax = b, equivalently
min Φ(x) =
x
1 T
x Ax − bT x
2
Minimize over expanding (affine) subspaces
Memory efficient.
Works for A > 0
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Linear conjugate gradient methods
Recall: linear CG method for solving Ax = b, equivalently
min Φ(x) =
x
1 T
x Ax − bT x
2
Minimize over expanding (affine) subspaces
Memory efficient.
Works for A > 0
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Linear conjugate gradient methods
Recall: linear CG method for solving Ax = b, equivalently
min Φ(x) =
x
1 T
x Ax − bT x
2
Minimize over expanding (affine) subspaces
Memory efficient.
Works for A > 0
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Linear conjugate gradient methods
Recall: linear CG method for solving Ax = b, equivalently
min Φ(x) =
x
1 T
x Ax − bT x
2
Minimize over expanding (affine) subspaces
Memory efficient.
Works for A > 0
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
CG in context
Here: x ∗ = YxY + ZxZ , Y , Z as above.
AYxY = b yields displacement.
Unconstrained problem:
1
min xZT (Z T GZ )xZ + xZT cZ
xZ 2
(cZ = Z T GYxY + Z T c constant)
CG works if Z T GZ > 0.
Preconditioning with Wzz .
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
CG in context
Here: x ∗ = YxY + ZxZ , Y , Z as above.
AYxY = b yields displacement.
Unconstrained problem:
1
min xZT (Z T GZ )xZ + xZT cZ
xZ 2
(cZ = Z T GYxY + Z T c constant)
CG works if Z T GZ > 0.
Preconditioning with Wzz .
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
CG in context
Here: x ∗ = YxY + ZxZ , Y , Z as above.
AYxY = b yields displacement.
Unconstrained problem:
1
min xZT (Z T GZ )xZ + xZT cZ
xZ 2
(cZ = Z T GYxY + Z T c constant)
CG works if Z T GZ > 0.
Preconditioning with Wzz .
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
CG in context
Here: x ∗ = YxY + ZxZ , Y , Z as above.
AYxY = b yields displacement.
Unconstrained problem:
1
min xZT (Z T GZ )xZ + xZT cZ
xZ 2
(cZ = Z T GYxY + Z T c constant)
CG works if Z T GZ > 0.
Preconditioning with Wzz .
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
CG in context
Here: x ∗ = YxY + ZxZ , Y , Z as above.
AYxY = b yields displacement.
Unconstrained problem:
1
min xZT (Z T GZ )xZ + xZT cZ
xZ 2
(cZ = Z T GYxY + Z T c constant)
CG works if Z T GZ > 0.
Preconditioning with Wzz .
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
CG in context
Here: x ∗ = YxY + ZxZ , Y , Z as above.
AYxY = b yields displacement.
Unconstrained problem:
1
min xZT (Z T GZ )xZ + xZT cZ
xZ 2
(cZ = Z T GYxY + Z T c constant)
CG works if Z T GZ > 0.
Preconditioning with Wzz .
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
CG in context
Here: x ∗ = YxY + ZxZ , Y , Z as above.
AYxY = b yields displacement.
Unconstrained problem:
1
min xZT (Z T GZ )xZ + xZT cZ
xZ 2
(cZ = Z T GYxY + Z T c constant)
CG works if Z T GZ > 0.
Preconditioning with Wzz .
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
The general CG algorithm
Given x0 , preconditioner M;
Set r0 ← Ax0 − b, solve My0 = r0 for y0 , set p0 = −y0 , k ← 0;
while rk 6= 0
αk ← −
rkT yk
;
pkT Apk
xk +1 ← xk + αk pk ;
rk +1 ← rk + αk Apk ;
Solve Myk +1 = rk +1 ;
βk +1 ←
rkT+1 yk +1
;
rkT yk
pk +1 ← −yk +1 + βk +1 pk ;
k ← k + 1;
end (while)
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
The general CG algorithm
Given x0 , preconditioner M;
Set r0 ← Ax0 − b, solve My0 = r0 for y0 , set p0 = −y0 , k ← 0;
while rk 6= 0
αk ← −
rkT yk
;
pkT Apk
xk +1 ← xk + αk pk ;
rk +1 ← rk + αk Apk ;
Solve Myk +1 = rk +1 ;
βk +1 ←
rkT+1 yk +1
;
rkT yk
pk +1 ← −yk +1 + βk +1 pk ;
k ← k + 1;
end (while)
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
The general CG algorithm
Given x0 , preconditioner M;
Set r0 ← Ax0 − b, solve My0 = r0 for y0 , set p0 = −y0 , k ← 0;
while rk 6= 0
αk ← −
rkT yk
;
pkT Apk
xk +1 ← xk + αk pk ;
rk +1 ← rk + αk Apk ;
Solve Myk +1 = rk +1 ;
βk +1 ←
rkT+1 yk +1
;
rkT yk
pk +1 ← −yk +1 + βk +1 pk ;
k ← k + 1;
end (while)
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
The general CG algorithm
Given x0 , preconditioner M;
Set r0 ← Ax0 − b, solve My0 = r0 for y0 , set p0 = −y0 , k ← 0;
while rk 6= 0
αk ← −
rkT yk
;
pkT Apk
xk +1 ← xk + αk pk ;
rk +1 ← rk + αk Apk ;
Solve Myk +1 = rk +1 ;
βk +1 ←
rkT+1 yk +1
;
rkT yk
pk +1 ← −yk +1 + βk +1 pk ;
k ← k + 1;
end (while)
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
The general CG algorithm
Given x0 , preconditioner M;
Set r0 ← Ax0 − b, solve My0 = r0 for y0 , set p0 = −y0 , k ← 0;
while rk 6= 0
αk ← −
rkT yk
;
pkT Apk
xk +1 ← xk + αk pk ;
rk +1 ← rk + αk Apk ;
Solve Myk +1 = rk +1 ;
βk +1 ←
rkT+1 yk +1
;
rkT yk
pk +1 ← −yk +1 + βk +1 pk ;
k ← k + 1;
end (while)
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Here
Given starting point xZ .
−1
Set rZ = Z T GZxZ + cZ , gz = Wzz
rz , dz = −gZ
repeat
αk ← rzT gz /(dzT (Z T GZ )dz )
xz ← xz + αd;
rz+
← rz + αk (Z T GZ )Adz ;
−1 +
Solve Wzz
gz = rz+ ;
(rz+ )T gz+
;
rzT gz
dz ← −gz+ + βdz ;
β←
gz ← gz +; rz ← rz+ ;
−1
until termination test satisfied (eg rzT Wzz
rz < )
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Here
Given starting point xZ .
−1
Set rZ = Z T GZxZ + cZ , gz = Wzz
rz , dz = −gZ
repeat
αk ← rzT gz /(dzT (Z T GZ )dz )
xz ← xz + αd;
rz+
← rz + αk (Z T GZ )Adz ;
−1 +
Solve Wzz
gz = rz+ ;
(rz+ )T gz+
;
rzT gz
dz ← −gz+ + βdz ;
β←
gz ← gz +; rz ← rz+ ;
−1
until termination test satisfied (eg rzT Wzz
rz < )
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Here
Given starting point xZ .
−1
Set rZ = Z T GZxZ + cZ , gz = Wzz
rz , dz = −gZ
repeat
αk ← rzT gz /(dzT (Z T GZ )dz )
xz ← xz + αd;
rz+
← rz + αk (Z T GZ )Adz ;
−1 +
Solve Wzz
gz = rz+ ;
(rz+ )T gz+
;
rzT gz
dz ← −gz+ + βdz ;
β←
gz ← gz +; rz ← rz+ ;
−1
until termination test satisfied (eg rzT Wzz
rz < )
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Here
Given starting point xZ .
−1
Set rZ = Z T GZxZ + cZ , gz = Wzz
rz , dz = −gZ
repeat
αk ← rzT gz /(dzT (Z T GZ )dz )
xz ← xz + αd;
rz+
← rz + αk (Z T GZ )Adz ;
−1 +
Solve Wzz
gz = rz+ ;
(rz+ )T gz+
;
rzT gz
dz ← −gz+ + βdz ;
β←
gz ← gz +; rz ← rz+ ;
−1
until termination test satisfied (eg rzT Wzz
rz < )
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Here
Given starting point xZ .
−1
Set rZ = Z T GZxZ + cZ , gz = Wzz
rz , dz = −gZ
repeat
αk ← rzT gz /(dzT (Z T GZ )dz )
xz ← xz + αd;
rz+
← rz + αk (Z T GZ )Adz ;
−1 +
Solve Wzz
gz = rz+ ;
(rz+ )T gz+
;
rzT gz
dz ← −gz+ + βdz ;
β←
gz ← gz +; rz ← rz+ ;
−1
until termination test satisfied (eg rzT Wzz
rz < )
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Projected CG
Given starting point xZ , conditioner H, projection P = Z (Z T HZ )−1 Z T
Find initial x s t Ax = b (feasible). Set r = Gx + c, g = Pr , d = −g.
repeat
αk ← r T g/(d T Zd)
x ← x + αd;
r + ← r + αGd;
(r + )T g +
;
rT g
d ← −g + + βd;
g + ← Pr + ; β ←
g ← g+; r ← r +;
until termination test satisfied (eg r T g = r T Pr < )
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Projected CG
Given starting point xZ , conditioner H, projection P = Z (Z T HZ )−1 Z T
Find initial x s t Ax = b (feasible). Set r = Gx + c, g = Pr , d = −g.
repeat
αk ← r T g/(d T Zd)
x ← x + αd;
r + ← r + αGd;
(r + )T g +
;
rT g
d ← −g + + βd;
g + ← Pr + ; β ←
g ← g+; r ← r +;
until termination test satisfied (eg r T g = r T Pr < )
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Projected CG
Given starting point xZ , conditioner H, projection P = Z (Z T HZ )−1 Z T
Find initial x s t Ax = b (feasible). Set r = Gx + c, g = Pr , d = −g.
repeat
αk ← r T g/(d T Zd)
x ← x + αd;
r + ← r + αGd;
(r + )T g +
;
rT g
d ← −g + + βd;
g + ← Pr + ; β ←
g ← g+; r ← r +;
until termination test satisfied (eg r T g = r T Pr < )
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Projected CG
Given starting point xZ , conditioner H, projection P = Z (Z T HZ )−1 Z T
Find initial x s t Ax = b (feasible). Set r = Gx + c, g = Pr , d = −g.
repeat
αk ← r T g/(d T Zd)
x ← x + αd;
r + ← r + αGd;
(r + )T g +
;
rT g
d ← −g + + βd;
g + ← Pr + ; β ←
g ← g+; r ← r +;
until termination test satisfied (eg r T g = r T Pr < )
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
Projected CG
Given starting point xZ , conditioner H, projection P = Z (Z T HZ )−1 Z T
Find initial x s t Ax = b (feasible). Set r = Gx + c, g = Pr , d = −g.
repeat
αk ← r T g/(d T Zd)
x ← x + αd;
r + ← r + αGd;
(r + )T g +
;
rT g
d ← −g + + βd;
g + ← Pr + ; β ←
g ← g+; r ← r +;
until termination test satisfied (eg r T g = r T Pr < )
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
A note on preconditioners
g + is a preconditioned residual. In Null(A).
Old problem remains: choice of Z
For H = I, PI = Z (Z T Z )−1 Z T can be written
PI = I − AT (AAT )−1 A
ie projection onto null space of A.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
A note on preconditioners
g + is a preconditioned residual. In Null(A).
Old problem remains: choice of Z
For H = I, PI = Z (Z T Z )−1 Z T can be written
PI = I − AT (AAT )−1 A
ie projection onto null space of A.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
A note on preconditioners
g + is a preconditioned residual. In Null(A).
Old problem remains: choice of Z
For H = I, PI = Z (Z T Z )−1 Z T can be written
PI = I − AT (AAT )−1 A
ie projection onto null space of A.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
A note on preconditioners
g + is a preconditioned residual. In Null(A).
Old problem remains: choice of Z
For H = I, PI = Z (Z T Z )−1 Z T can be written
PI = I − AT (AAT )−1 A
ie projection onto null space of A.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
A note on preconditioners
g + is a preconditioned residual. In Null(A).
Old problem remains: choice of Z
For H = I, PI = Z (Z T Z )−1 Z T can be written
PI = I − AT (AAT )−1 A
ie projection onto null space of A.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
General H, rank(H)=n: P = H −1 (I − AT (AH −1 AT )−1 AH −1 .
Perfect preconditioner: H = G, but all H : Z T HZ > 0 ok.
Note: all without explicit Z .
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
General H, rank(H)=n: P = H −1 (I − AT (AH −1 AT )−1 AH −1 .
Perfect preconditioner: H = G, but all H : Z T HZ > 0 ok.
Note: all without explicit Z .
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
General H, rank(H)=n: P = H −1 (I − AT (AH −1 AT )−1 AH −1 .
Perfect preconditioner: H = G, but all H : Z T HZ > 0 ok.
Note: all without explicit Z .
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
General H, rank(H)=n: P = H −1 (I − AT (AH −1 AT )−1 AH −1 .
Perfect preconditioner: H = G, but all H : Z T HZ > 0 ok.
Note: all without explicit Z .
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
On inequality constraints
Necessary optimality conditions, A(x ∗ ) = {i ∈ E
P
Gx ∗ + c − i∈A(x ∗ ) λ∗i ai = 0,
aiT x ∗ = bi , i ∈ A(x ∗ ),
T ∗
ai x ≥ bi , i ∈ IT− A(x ∗ ),
λ∗i ≥ 0, i ∈ I A(x ∗ ).
S
I|aiT x ∗ = bi }:
(complementarity implicit)
If G ≥ 0 and the above hold for some λ∗i , i ∈ A(x ∗ ), x ∗ is global
solution.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
On inequality constraints
Necessary optimality conditions, A(x ∗ ) = {i ∈ E
P
Gx ∗ + c − i∈A(x ∗ ) λ∗i ai = 0,
aiT x ∗ = bi , i ∈ A(x ∗ ),
T ∗
ai x ≥ bi , i ∈ IT− A(x ∗ ),
λ∗i ≥ 0, i ∈ I A(x ∗ ).
S
I|aiT x ∗ = bi }:
(complementarity implicit)
If G ≥ 0 and the above hold for some λ∗i , i ∈ A(x ∗ ), x ∗ is global
solution.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
On inequality constraints
Necessary optimality conditions, A(x ∗ ) = {i ∈ E
P
Gx ∗ + c − i∈A(x ∗ ) λ∗i ai = 0,
aiT x ∗ = bi , i ∈ A(x ∗ ),
T ∗
ai x ≥ bi , i ∈ IT− A(x ∗ ),
λ∗i ≥ 0, i ∈ I A(x ∗ ).
S
I|aiT x ∗ = bi }:
(complementarity implicit)
If G ≥ 0 and the above hold for some λ∗i , i ∈ A(x ∗ ), x ∗ is global
solution.
Anders Märak Leffler
Quadratic programming
The problem
Non-CG methods
CG methods
On inequality-constraints
The end
Thank you for your attention.
Anders Märak Leffler
Quadratic programming