Complementary Slackness Theorem

Complementary Slackness Theorem : Suppose that x = (x1, x2, . . . , xn) is
primal feasible and that y = (y1, y2, . . . , ym) is dual feasible. Let (w1,w2, . . .
,wm) denote the corresponding primal slack variables, and let (z1, z2, . . . , zn)
denote the corresponding dual slack variables. Then x and y are optimal for their
respective problems if and only if:
x j z j  0, for j  1, 2, . . . , n,
wi yi  0, for i  1, 2, . . . , m.

Proof
m
n
m
 m

c j x j     yi aij  x j   yi   aij x j    bi yi

j=1
j=1  i 1
i 1
j=1
i 1

n
n
m
 x j  0,  yi aij  c j
i 1
m
ya
i 1
i ij
m
 c j ; z j   yi aij  c j  0  x j z j  0
i 1
KKT conditions - the karush-kuhn-tucker (kkt) conditions for
constrained optimization

Theorem. Assume that f ( x), g1 ( x), g 2 ( x), . . . , g m ( x) are differentiable
functions satisfying certain regularity conditions, then x*  ( x1*, x2 *, . . . , xn *)
can be an optimal solution for the nonlinear programming problem only if there
exist m numbers u1 , u2 , . . . , um such that all the following KKT conditions are
satisfied:
f ( x*) m gi ( x*)
  ui
0
o
x j
x j
i 1
 f ( x*) m gi ( x*) 
 f ( x*) m gi ( x*) 
xj *
  ui
 0 or 
  ui

  0 if x can
 x
 x
x j 
x j 
i 1
i 1
j
j


be negative
o gi ( x*)  bi  0
o
o Complmentary slackness ui  gi ( x*)  bi   0
o Primal and dual feasibility x j *, ui  0
KKT Example:
f ( x)  ln( x1  1)  x2
2 x1  x2  3
x1.x2  0
KKT solution:
1
 2u1  0
x1  1
 1

x1 
 2u1   0
 x1  1

(1  u1 )  0
x2 (1  u1 )  0
2 x1  x2  3  0
u1  2 x1  x2  3  0
x1 , x2 , u1  0
Conlusions
1  u1 , x1 , x2 , u1  0 
1
 2u1  0  x1  0
x1  1
 2 x1  x2  3  x2  3  u1  1
Quadratic Programming and KKT

Define lagrangian: L( x) 
1 t
x Qx  ct x   t ( Ax  b)
2

d
L( x,  )  0  Qx  c   t A  0
dx
d
x L( x,  )  0  x t  Qx  c   t A   0
dx
d
L( x,  )  0  Ax  b  0
d
d

L( x,  )  0   t  Ax  b   0
d
x,   0
Add slack variables:
Qx  c   t A  y  0
xt  Qx  c   t A   0
Ax  b  v  0
 t  Ax  b   0
x,  , y , v  0

Examples of nonregular optimal solutions.
Consider the problem Min x s.t. x2  0,
or equivalently
–Max –x s.t. x2  0.
The feasible region is {x | x=0} and Min {x | x=0} = 0 at x*=0. The point x* is not
regular, since gE(x*) = [2x]x=0 = [0] which is of rank 0 < |E| = 1.
The KKT conditions do not hold: there exists no   0 such that
(x*) = [- 1] =  g(x*) =  . [0].
Notice that x2  0 is equivalent to x=0, however we must use g(x) = -x2 in the KKT
conditions.
(2) Consider the problem Min x1 s.t. x2  x13, x2 0.
Then (x) = -x1 and g1(x) = x2 - x13  0, g2(x) = -x2  0.
The solution is x1*=x2*=0.
 -3x12
J(x*) = 
 0
1
=

1 x*
0 1 


 0 1
has rank 1<|E|=2, thus x* is not a regular point.
The KKT conditions would require that
(x*) = [-1 0] =  g(x*) = 1 . [0 1] +1 . [0 -1] = [0, 1+2]
which is of course impossible. .
Notice however that for the function (x) = -x2, the KKT conditions hold at x*, even
though it is not a regular point.
One more example of KKT
Consider the problem of minimizing f(x)=4(x-1)2+(y-2)2 with constraints:
x + y ≤ 2; x ≥ - 1; y ≥ - 1;
Solution
m
p
k 1
j 1
L( x,  ,  )  f ( x)   k hk ( x)    j g j ( x) :
L( x,  ,  )  4  x  1   y  2   1  x  y  2   2  x  1  3  y  1
2
2
There are 3 inequality constraints, each can be chosen active/ non active  yield 8
possible combinations. However, 3 constraints together: x+y=2 & x=-1 & y=-1 has no
solution, and combination of any two of them yields a single intersection point.
The general case is:
L( x,  ,  )  4  x  1   y  2   1  x  y  2   2  x  1  3  y  1
2
2
We must consider all the combinations of active / non active constraints:
1.
L  x, y   4  x  1   y  2 
2
2
2. y  1  L  x, y,    4  x  1   y  2     y  1
2
2
3. x  1  L  x, y,    4  x  1   y  2     x  1
2
2
4. x  y  2  L  x, y,    4  x  1   y  2     x  y  2 
2
 x, y   
x  1 and y  1   x, y    1, 1
x  y  2 and y  1   x, y    3, 1
x  y  2 and x  1   x, y    1,3
5. x  y  2 and
6.
7.
8.
2
x  1 and
x  1 
Finally, we compare among the 8 cases we have studied: case (7) resulted was overconstrained and had no solutions, case (8) violated the constraint x+y≤2. Among the
cased (1)-(6), it was case (1) ,(x,y)=(0.8,1.2),f(x,y)=0.8, yielding the lowest value of
f(x,y).