..-,
,I,.~~~~~~~~
I~~~liIII--I--
I :I -
"'-
- ~ '--
~
~~:.
~ ,~-
1.1i
I
.
" I
.
. - ? ~:
~'.
·
:,I.
,- - :,
,
,~ ..
,
II.-:
~~~~~
~ ~~~ ~
Z
-::,. , - ...~,..~-. . . .
-.I
,-
,
-.
I.
- ,.-
:
-
;-,
-~
,
,Z , r ,-. - ,.
I
?.~
,~:;.z
?..i,:
- -: f~-.. .~ .r-,,
. '-.~---. ...'
. ,-
,
I
'
,
,
,
.
.:-m~~~~~~~~~~~~::
...~..
.
. :~ I.
~-?:.:
. _ -
,
,.
--
I, . - -, - - :- -.
I
I '
-,
. I
,.:
-
--
,
,
--
'~ '
. '
.
I
:
.
-
.:
:~
: ..
.
. .r4-...~..
.-, %,
. - ~... .
~ ~~~~"
~ I~:I
,I~~~~~
I .I-
-I~~~~~ ,. I I.,: - ': -,
-~~~~~~~~~~~~~
- ,,-,-,1
, 6.
. . . . .
I ., .17
,
':.- -"
,-i-.-'
'
'" ',','--
,
~
~
. I ,~
.~~.
11~~~F
4~~~
--.
-,
,·':
:
-~ . I
,,
... ,I"-,~~~.I.
-I.
.
,
4 ·,
.~
----
.
:
-:.. ..
.F4.,.
-.~
,.~,
~ ~~
,
'-:.
,-:
~~~~
~ ~~ ~~ ~ ~
-
-Z
-.
'
:,
?,
.
.
,-. -,
-, I-,,
v~~,
-,, -.
'-
I,
-I;I~
~. .
.
-1.
I.
.
: .!
,,,. - ..I,
- ;
'.·',
.~-
-
',
.I
, :, 2,"~I
:-? .~-',7
' ,,~
.
I
.
.
,
I~I
-.. ..
.
'.
:
.,
.
I,
I
.
~
-'l
.,,,,.
I,:..~[
,
, I'.-'
(, : . .,.
.,
., _~-
.
-
,
;
~ ~,
..
,
"Z
I , ;- ,
. . .
.
.
.
.
,?
.
:. I ,~:., . . . . . . .. I''!
--.1 , ... . ... .-,. , ..: ,- '" '
- ., , " I -
.
~~~~~~-.- ,' ~. ~~~-"
~~,,
~~~~~~~.
~'...
~~~ ' -~
, ~~- ~ ,~
.,.",
,.
.:
~
,
..
-o - . < 1 .-- -
. .:
-,
~
. --..
,. /
:
. ,I
..
--~.,,,-~ '~ , I
;., I- -,4 :~ .
.
,
,
:-:
'
.
-
',"
- .. (,,
,
x,-.
,I
...-
,
:;: - :~
" '
~, . · II."I ~.
~~~~~~'~~~~ ~ ~ -;" ~
~~- ~,.~~ :: '. Z:
.1I j:,
.I.~I
-.I...,-i--X,
--,~
. ,,:,,~~~~~~~~;
, -~
-~
.--I:"-~I ~: ~, .~ , --. .~,
,'. :~~~~~~~~~~
.I.1I~.- ~. , -,, .-!. ..I --
'
,.
.q
I
-I
.
...
:
I
-.
..
III I:. I.
.-.~~~~~,
,
-
-
, -J,! -:.
.II, .
.-
-'. .... "
~-. ;..;
:~I~,-:
:- -1 I..I:..""':--, .I
.. . ""
.2 ::.- -.~-~~~,~~~~~~~~~~.
'"1" -· :, 7-....
~ I~~ .,-~. ~ ,;.~
.- , -
,
-.:'
.
"
.
-,
-. . w
; . . ..
.f".
,.
-
-.,
, --
. I. I I
I.;-
--
,-
,.,
~
, .., , - . . ,:.
,
. . _ ,-
,
-
,-.,~~~~~
~ ~~~
'
-.
.; . I-~~~~~~. - ~.,, ? ~,~.
- >. : ·~..?~.
. ..
',~
I,
--
,.
.
mI1.
-
,
.
,-..4.1
--~~~~~'-.. -~~~~~
-- I I
~~
~ ~ ~ ~ ~..',
~II.
,
.~1
-;I.
'I
,:,,
,
...
~ 9 ..:I . 1 - . ... .. . . .. I 1,I . . , .
~.~ ,~Z.
I -::;:"
,
~~~~,--~~~I-~~~.~~-~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~,.;
~~
~r :- · ..-.
1 II~
:
. ...!.o. . I ~ .I:-~, ,- 7
",~ -"-, i'.:
~~~~~~~~~~~~~~. ~~~~~~~~.
~~~~"~~
~ ~~~~~~~~ ~~~~~ ~~~~
.. . ~
I
,
-.
·
--.
~
-.:: - r. -:.
.
-~~
~
I.
.
.
I..
.
...
,
. . ,. .
- .~~~~~~
~
~
. -....
. ..~.-,. ,- ~, , ~.. . . ....-. . . .
~. ~- .....
' :;.-- . -.
- --- -I..~~~I, -~ ~ ~~r
- I :, ,-.::, -- . . - ~ ~ ~ ~~ ~ -I
~I?
.,.
. . .. -~..-'m,1
- ~... ..... ,..,., -...
"",;,~~. ;-.-`:---'
' .. I-~~:
. II--tI,
~ pO
. .~
I - .- :Z-.
.
I -I . . . . .I.
-. - :,~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.,
.: . . .-:...... ...
I ,.-,-I
,
- ~:. .-,-,4-'-'.
.I: .-.... :~ I , ,
,
.
~~~~~~~,~~~~;
: . ~~ .I .-I
- I I . II
-
..-
-
--
.
-"
:
I
- :~
,.,
- . . ,
_
-:
'i,
,"_;
.''
,.-
, '~.
FENCHEL AND LAGRANGE DUALITY
ARE EQUIVALENT
by
T.L. Magnanti
OR 036-74
July 1974
Supported in part by the U.S. Army Research Office (Durham) under
Contract No. DAHC04-73-C-0032.
ABSTRACT
A basic result in ordinary (Lagrange) convex programming is the
saddlepoint duality theorem concerning optimization problems with
convex inequalities and linear-affine equalities satisfying a
Slater condition.
This note shows that this result is equivalent
to the duality theorem of Fenchel.
1
Among the most powerful tools in mathematical programming are the
saddlepoint duality theories for both Fenchel and ordinary (Langrange)
convex programs.
The purpose of this note is to exhibit the equivalence
of these two theories by showing that each can be used to develop the other.
The connection between Fenchel and Lagrange duality is certainly not
surprising since each is based upon separating hyperplane arguments.
Previously, Whinston [7] has shown how the Lagrangian theory can be derived
from Frenchel's results when optimizing over Rn in the absense of equality
Both theories also can be obtained from Rockafellar's. general'
constraints.
The full equivalence between these theories, though,
perturbation theory.
has not been stated in the open literature and seems to be unknown to
most of the mathematical programming community.
In fact, a number of recent
comprehensive books in convex optimization including Luenberger [3] ,
Rockafellar [5] and Stoer and Witzgall [6] do not explicitly mention or
exploit this equivalence, but rather develop the two theories separately.
The Fenchel and Lagrange convex programs can be stated as
Lagrange
Fenchel
v=inf {fo(x):g(x)
v=inf {fl(x)-f 2 (x)
x1CllC
<o,h(x)=o}
X Co
2
Each Cj is a convex subset of Rn; fb(x) and fl(x) are convex functions
defined respectively on C
C2 ; g:C
°
and C1 ;f2 (x) is a concave function defined on
--) Rm is convex and h:Rn-- Rr is linear-affine.
The saddlepoint dual problems are:
Lagrange Dual
Fenchel Dual
d-max {f2(7)
7Rn
2
-
f*(T)}
1n
d=max
7r>
rr0o
a
Rr
r6ERn
inf {L(x;r,a)}
X£CO
xC 0
2
where
fl(n)= - inf {fl(x)-7x} , f2 (7) = inf{7x-f2 (x)
xEC 1
xeC 2
are respectively conjugate convex and conjugate functions and
L(x;7,a)
is the Lagrangian function.
maximizations.
f(x)+7g(x)+ah(x)
=
Note that the dual problems are formulated as
We shall consider conditions to insure that these maxima are
attained.
Recall that
fl(r) = A+
and/or
f2(i) = -
are possibilities. Such values
will be inoperative for the maximization in the Fenchel dual and thus
for
that problem is frequently written with
C·= { cRn:f *()<+-}
and
rrCl*nC2*
instead of
7rERn
where
C2= {ERn:f ()>--}.
Two of the most useful and delicate theorems connecting these primal-dual
pairs are (for any convex
Theorem 1:
If
Theorem 2:
Assume
v>-
C, ri(C)
ri(C 1)n ri(C 2 )# $ , then
and
>--
x°Ori(C o)
denotes its relative interior):
v=d.
and the Slater condition that there exists an
such that
g(x°)<o
and
h(x°)= o.
Then
v=d
The first theorem is Fenchel's original duality theorem [2] and the second is
a consequence of a result due to Fan, Glicksberg and Hoffman [1] (see Chapter
28 of [5] or Chapter 6 of [6]).
When there are no inequality constraints,
theorem 2 remains valid by omitting the reference to
condition.
vd
satisfied:
there exists
g(x)
in the Slater
is equivalent to saying that the Kuhn-Tucker condition is
7>o
and a such that
v<L(x,r,a)
for all
xeC
.
Below we show that either theorem 1 or 2 can be used to derive the other.
The following elementary result concerning relative interiors will be useful.
Lemma 1:
Let C= {y,yly2)R
If
Proof:
l
+m+r : yO>fo(x),ylg(x)
xri(CO), yO>f (x°), yl>g(x°)
and
and
y2=h(x)
y2 =h(xo), then y=(yO,yl,y2)eri(C).
The following condition ([5] theorem 6.4) characterizes the
relative interior of any convex set
._ III~III_
~IIIIII_
~~~~l__·CIYI
CI·_Y
I--·
lll~~ll_ ..-*-~~(
11~------~ ·_1_~~1__
S:
for some xeCo}.
3
p>1
such that
pz+(l-p)z
C
- (-o,yl,y2)
0
°>fo(x),
with
such that
and
xeCo
and
2-h(x), we must find a
1<p<X.
there is a
Since h()
X>1
such that
is linear-affine
(l-)x)
are continuous on
and fo()
and yl shows that
(1)
ri(Co)
py°+(1-p)y°>fo (xO+(1-p)x)
(2)
pyl+(l-p)yl>g (Ix°++(l-p)x)
(3)
for some
Proof;
yl>g(x)
)2 = ph(xO)+(l-p)h(x)=h(pxO+
and since both g()
the definition of y
Theorem 3:
S.
xri(CO)
Px°+(l-p)xECo for all
Py2+(l-_
there is a
py+(l-p)yeCo.
By convexity and
and
S
is a convex set; given any
From our hypothesis
V>1
z
if and only if for each
zeri(S)
1<
<.(l)-(3)
imply that
py+(l-p)sC
so that
yri(C)..
Theorem 1 and Theorem 2 are equivalent.
(Theorem 2
Theorem 1)
Letting
Co=C 1 xC2 xR
and
x=(xl,x 2,x3 )
the Fenchel problem is restated as
v
inf {fl(xl)-f
(x 2
3
2):xl= x,x
2
x3
xeC o
This is an ordinary convex problem with the associations
f2(x 2)
d
and
h(x) = (x2-x3).
fo(x)=fl(xl) -
Its Lagrangian dual problem is
2 )x3}
max inf {fl(xl)-f2(x 2 )+alx l+a2 x 2-(al+a
(4)
.
aieRn X2Co
The hypothesis ri(Cl)nri(C 2 )$0
implies the Slater condition
for this problem and consequently that
x3 eR n,
v=d
the infimum in the dual problem is
by Theorem 2. Since
--
Therefore the dual maximization occurs for some
the Lagrangian dual
giving
_
i
1
1_1·1__^
111_
d=d=v
lillII---·II1_X_
-
whenever
i=
-a1=a 2
l -a2.
and
(4) can be written as the Fenchel dual,
to prove Theorem 1.
4
> Theorem 2).
(Theorem 1
Letting
C1
be the set
C
of
Lemma 1 and letting
yl<o and y2 =o}
C2 = {(y°,yl,y2)SRll*r
the Lagrange problem
is written as
v- inf {yO: y=(yo,yl,y 2 )ClC2} .1
If
x°
is the Slater point of Theorem 1 and
y 0>fo(x 0 ), o>yl>g(x),
y2 =h(x°), Lemma 1 implies that
(yO,yl,y2 )sri(C1 )nri(C 2 )
Identifying
fl(y)=y °
and
f2 (y)=o, the Fenchel dual to this formula-
tion is
d=max
nf(inf
o1,2
and by Theorem 2,
(pyo0 +'rlyl+
v = d .
j=l,...,m
d=max
7r<0o
and is
o
--
w°0 o
if
yEC 1
or
1
'n>o
otherwise, this dual problem reduces to
inf {yO-lyl-r2 y2 } = max inf
so that v = d to prove Theorem 2.
. .
y )}
sC2
Since the second infimum is
for some
22
7r>o
xC
fo(x)+g(x)+ahh(x)}=d
o
5
Observe that the sets
C1
and
C2
used above are those typically
Also,
constructed in a separating hyperplane approach to Lagrange duality.
the approach utilized here can be applied to relate other versions of the
saddlepoint theories, for example when the dual problems do not have
optimizing solutions.
For instance,
Other extensions can be obtained.
the problem
k
fj (AJx) : Ax
v = min {
J=1
where
fj
Cj
are convex sets, Aj
is convex on
Cj
min
xC
are real matrices with
n
columns, and
can be expressed in Lagrange form as
k
{ Z fj(Aj): AxJ= A
' j=l
n
Co= ClxC2 x...xCkxR
where
Gj }
and
x=(x
}
...,x k ,).
Writing the Lagrange
dual and simplifying provides the dual problem
k
k
d= mjx
{ Z inf (f.(x) +
j=l xCj
and
v = d
if there is an x°Rn
Jx) :
TJ A
= o
}
J=1
with AJxO E ri(Cj).
When
r=2
and
is the identity matrix this is the dual problem of Rockafellar [5].
A1l
These
duality correspondences can be carried even further by appending inequality
and equality constraints to the above formulation.
I
___I_
__I___
6
Acknowledgement:
This material is an outgrowth of a section of report [4].
I would like to thank W. Oettli for several comments concerning this report
and for suggesting this note.
References:
1.
Ky Fan, I. Glicksberg and A. J. Hoffman, "Systems of inequalities
involving convex functions, "Proc. Amer. Math. Soc. 8 (1957), 617-622
2.
W. Fenchel, "Convex Cones, Sets and Functions", lecture notes,
Princeton University, 1951
3.
4.
D. Luenberger, Optimization by Vector Space Methods, John Wiley & Sons,
New York, 1969
T. Magnanti, "A linear approximation approach to duality in nonlinear
irogramming, "Tech. Report OR 016-73, Operations Research Center,
M.I.T., April 1973
5.
R. T. Rockafellar, Convex Analysis, Princeton University Press, 1970
6.
J. Stoer and C. Witzgall, Convexity and Optimization in Finite
Dimensions I, Springer-Verlag, 1970
7.
A. Whinston, "Some applications of the conjugate function theory
to duality", in Nonlinear Programming (J. Abadie ed.), NorthHolland Publishing Co., (1967), 77-98.