15.081J/6.251J Introduction to Mathematical
Programming
Lecture 21: Primal Barrier
Interior Point Algorithm
1
Outline
Slide 1
1. Barrier Methods
2. The Central Path
3. Approximating the Central Path
4. The Primal Barrier Algorithm
5. Correctness and Complexity
2
Barrier methods
Slide 2
min f (x)
s.t. gj (x) ≤ 0,
hi (x) = 0,
j = 1, . . . , p
i = 1, . . . , m
S = {x| gj (x) < 0, j = 1, . . . , p,
hi (x) = 0, i = 1, . . . , m}
2.1
Strategy
Slide 3
• A barrier function G(x) is a continous function with the property that is
approaches ∞ as one of gj (x) approaches 0 from negative values.
• Examples:
G(x) = −
p
�
j=1
log(−gj (x)), G(x) = −
p
�
j=1
1
gj (x)
Slide 4
• Consider a sequence of µk : 0 < µk+1 < µk and µk → 0.
• Consider the problem
�
�
xk = argminx∈S f (x) + µk G(x)
• Theorem Every limit point xk generated by a barrier method is a global
minimum of the original constrained problem.
1
.
.
.
.
.
.
x*
c
x(0.01)
x(0.1)
central p a th
x(1)
x(10)
analy t ic center
2.2 Primal path-following
IPMs for LO
(P ) min
s.t.
Slide 5
c′ x
Ax = b
(D) max
s.t.
x ≥ 0
s≥0
Barrier problem:
min
s.t.
b′ p
A′ p + s = c
Bµ (x) = c′ x − µ
n
�
log xj
j=1
Ax = b
Minimizer: x(µ)
3
Central Path
Slide 6
• As µ varies, minimizers x(µ) form the central path
• limµ→0 x(µ) exists and is an optimal solution x∗ to the initial LP
• For µ = ∞, x(∞) is called the analytic center
min
s.t.
−
n
�
log xj
j=1
Ax = b
Slide 7
3.1 Example
Slide 8
min x2
s.t. x1 + x2 + x3 = 1
x1 , x2 , x3 ≥ 0
2
x3
Q
( 1 / 2 , 0 , 1/ 2 )
the analy t ic
centerof Q
.
( 1 / 3 , 1/ 3 , 1/ 3 )
the analy t ic
centerof P
P
.
x2
the central p a th
x1
�
• Q = x | x = (x1 , 0, x3 ), x1 + x3 = 1, x ≥ 0}, set of optimal solutions to
original LP
• The analytic center of Q is (1/2, 0, 1/2)
min
s.t.
x2 − µ log x1 − µ log x2 − µ log x3
x1 + x2 + x3 = 1
min x2 − µ log x1 − µ log x2 − µ log(1 − x1 − x2 ).
1 − x2 (µ)
2
�
1 + 3µ − 1 + 9µ2 + 2µ
x2 (µ) =
2
1 − x2 (µ)
x3 (µ) =
2
The analytic center: (1/3, 1/3, 1/3)
Slide 9
3.2
Slide 10
x1 (µ) =
Solution of Central Path
• Barrier problem for dual:
max
s.t.
• Solution (KKT):
p′ b + µ
n
�
log sj
j=1
′
p′ A + s = c′
Ax(µ)
x(µ)
A′ p(µ) + s(µ)
s(µ)
X(µ)S(µ)e
3
=
≥
=
≥
=
b
0
c
0
eµ
Slide 11
• Theorem: If x∗ , p∗ , and s∗ satisfy optimality conditions, then they are
optimal solutions to problems primal and dual barrier problems.
• Goal: Solve barrier problem
min Bµ (x) = c′ x − µ
n
�
log xj
j=1
s.t. Ax = b
4
Approximating the central path
Slide 12
∂Bµ (x)
µ
= ci −
∂xi
xi
∂ 2 Bµ (x)
µ
= 2
2
∂xi
xi
∂ 2 Bµ (x)
= 0, i �= j
∂xi ∂xj
Given a vector x > 0:
Slide 13
Bµ (x + d) ≈ Bµ (x) +
n
�
∂Bµ (x)
i=1
∂xi
di +
n
1 � ∂ 2 Bµ (x)
di dj
2 i,j=1 ∂xi ∂xj
1
= Bµ (x) + (c′ − µe′ X −1 )d + µd′ X −2 d
2
X = diag(x1 , . . . , xn )
Approximating problem:
min
s.t.
Slide 14
1
(c′ − µe′ X −1 )d + µd′ X −2 d
2
Ad = 0
Solution (from Lagrange):
c − µX −1 e + µX −2 d − A′ p = 0
Ad = 0
Slide 15
• System of m + n linear equations, with m + n unknowns (dj , j = 1, . . . , n,
and pi , i = 1, . . . , m).
4
.
.
.
.
.
.
x*
c
x(0.01)
central p a th
x(0.1)
x(1)
x(10)
analy t ic center
• Solution:
�
�
��
1
I − X 2 A′ (AX 2 A′ )−1 A xe − X 2 c
µ
2 ′ −1
2
p(µ) = (AX A ) A(X c − µxe)
d(µ) =
4.1
The Newton connection
Slide 16
• d(µ) is the Newton direction; process of calculating this direction is called
a Newton step
• Starting with x, the new primal solution is x + d(µ)
�
�
• The corresponding dual solution becomes (p, s) = p(µ), c − A′ p(µ)
• We then decrease µ to µ = αµ, 0 < α < 1
4.2
Geometric Interpretation
Slide 17
• Take one Newton step so that x would be close to x(µ)
• Measure of closeness
��
��
�� 1
��
�� XSe − e�� ≤ β,
�� µ
��
0 < β < 1, X = diag(x1 , . . . , xn ) S = diag(s1 , . . . , sn )
• As µ → 0, the complementarity slackness condition will be satisfied
5
Slide 18
5
The Primal Barrier Algorithm
Slide 19
Input
(a) (A, b, c); A has full row rank;
(b) x0 > 0, s0 > 0, p0 ;
(c) optimality tolerance ǫ > 0;
(d) µ0 , and α, where 0 < α < 1.
1. (Initialization) Start with some primal and dual feasible x0 > 0, s0 >
0, p0 , and set k = 0.
2. (Optimality test) If (sk )′ xk < ǫ stop; else go to Step 3.
3. Let
Slide 20
X k = diag(xk1 , . . . , xkn ),
µk+1 = αµk
Slide 21
4. (Computation of directions) Solve the linear system
′
k+1
µk+1 X −2
X −1
k d−A p = µ
k e−c
Ad = 0
5. (Update of solutions) Let
xk+1 = xk + d,
pk+1 = p,
sk+1 = c − A′ p.
6. Let k := k + 1 and go to Step 2.
6
Correctness
√
β−β
Theorem Given α = 1 − √
√ , β < 1, (x0 , s0 , p0 ), (x0 > 0, s0 > 0):
β+ n
��
��
�� 1
��
�� X 0 S 0 e − e�� ≤ β.
��
�� µ0
Then, after
� √
�
√
β+ n
(s0 )′ x0 (1 + β)
K= √
log
ǫ(1 − β)
β−β
iterations, (xK , sK , pK ) is found:
(sK )′ xK ≤ ǫ.
6
Slide 22
6.1
Proof
Slide 23
�
�
� 1
�
�
• Claim (by induction): �| µk X k S k e − e��| ≤ β
• For k = 0 we have assumed it
• Assume it holds for k;
�
�
�
�
�
�
�
� 1
�|
�| = �| 1 X k S k e − e �|
X
S
e
−
e
k
k
� µk+1
�
� αµk
�
� �
�
�
� 1 1
1 − α ��
= ��|
X
S
e
−
e
+
e
k
k
�|
α µk
α
�
�
�
1� 1
1−α
≤ ��| k X k S k e − e��| +
||e||
α
α µ
β
1 − α√
n
+
α
α
�
=
β
≤
• We next show that ||X −1
k d|| ≤
√
• d solves
β < 1, where d = xk+1 − xk .
′
k+1
µk+1 X −2
X −1
k d−A p = µ
k e − c,
Ad = 0
• By left-multiplying the first equation by d′
�
′
k+1
µk+1 d′ X −2
X −1
k d = d µ
k e−c
2
||X −1
= d′ X k−2 d
k d||
=
�
=
�
=
�
X −1
k e
1
X−
k e
−
−
X −1
k e−
= −
�
1
µk+1
1
µk+1
1
c
�′
d
k
′
k
(s + A p )
sk
µk+1
�′
1
X kSke − e
µk+1
�
�
�
�′
d
�′
X −1
k d
�
� 1
≤ ��| k+1 X k S k e − e��| ||X −1
k d||
µ
hence, ||X −1
k d|| ≤
√
≤
β < 1.
�
1
β||X −
k d||
7
d
• We next show that xk+1 and (pk+1 , sk+1 ) are primal and dual feasible Since
Ad = 0, we have
Axk+1 = b
1
xk+1 = xk + d = X k (e + X −
k d) > 0,
because ||X −1
k d|| < 1
A′ pk+1 + sk+1 = c,
by construction and
−1
sk+1 = c − A′ pk+1 = µk+1 X −1
k (e − X k d) > 0,
•
because ||X −1
k d|| < 1
�
xk+1
= xjk 1 +
j
dj
xkj
�
,
�
dj
µk+1
1− k
xkj
xj
sk+1
=
j
�
.
Therefore,
�
1
1
dj
xk+1 sk+1
− 1 = k+1 xkj 1 + k
j
µk+1 j
µ
xj
�
dj
= −
xkj
• D = diag(d1 , . . . , dn ), ||u||1 =
�
i
�2
�
µk+1
xkj
�
1−
.
|ui |. Note that ||u|| ≤ ||u||1
�
�
� 1
�
� −2 2 �
�
�
�
�|
� µk+1 X k+1 S k+1 e − e�| = |X k D e |
�
�
2 �
≤ �|X −2
k D e |
1
2
2
= e′ X −
k D e
2
= e′ DX −
k De
= d′ X −2
k d
1 � 2
= �|X −
k d |
�
≤ (
�
β)2
�
= β,
and hence the induction is complete.
• Since at every iteration
�
�
�
�
�| 1 X k S k e − e �| ≤ β
�
� µk
−β ≤
dj
xkj
1 k k
x j sj − 1 ≤ β
µk
nµk (1 − β) ≤ (sk )′ xk ≤ nµk (1 + β)
8
�
−1
•
k
k
0
µ =α µ =
• After
√
�k
√
β−β
−k √
√
β−β
0
β+ n µ0
√
µ ≤e
1−
√
β+ n
�
�√
� � √
�
√
√
(s0 )′ x0 (1 + β)
µ0n(1 + β)
β+ n
β+ n
√
log
≤ √
log
=K
ǫ
ǫ(1 − β)
β−β
β−β
iterations, the primal barrier algorithm finds primal and dual solutions xK ,
(pK , sK ), that have duality gap (sK )′ xK less than or equal to ǫ
7
Complexity
Slide 24
• Work per iteration involves solving a linear system with m + n equations
in m + n unknowns. Given that m ≤ n, the work per iteration is O(n3 ).
• ǫ0 = (s0 )′ x0 : initial duality gap. Algorithm needs
�√
ǫ0 �
O
n log
ǫ
iterations to reduce the duality gap from ǫ0 to ǫ, with O(n3 ) arithmetic
operations per iteration.
9
MIT OpenCourseWare
http://ocw.mit.edu
6.251J / 15.081J Introduction to Mathematical Programming
Fall 2009
For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.