Hidden Minimum-Norm Problems in
Quadratic Programming
by
Robert M.
Sloan W.P. #1768-86
Revised
Freund
April,
1986
Abstract:
quadratic
This note presents
sufficient conditions
program, or its dual,
(Euclidean) norm problem,
linear transformation
conditions are shown
i.e.
to be transformed into a minimum
a problem of minimizing
of an element of a polytope.
the norm of
sufficient conditions,
These sufficient
As part of the
we characterize when the two linear
Ax > b and Az
< b have simultaneous solutions.
These results are used in conjunction with duality constructions
obtain two equivalent reformulations of a given quadratic
Key Words:
Quadratic program,
norm, minimum norm problem,
program.
Running Header:
a
to be necessary under a suitable restriction on
the class of transformations that are allowed.
inequality systems
for a convex
Hidden Norm Problems.
to
program.
gauge
A minimum
(Euclidean) norm problem over a polytope is a program
of the form:
NP:
UCx
minimize
+ dl
x
subject to:
where
11*11
denotes
the Euclidean
The purpose of
implications
Ex >
f
(12)
this note is
norm.
to explore
when can a convex quadratic
of the following question:
program or its dual
be conveniently
answers to and
transformed into a minimum
(Euclidean) norm program?
The
standard convex quadratic program is given as
QP:
1/2 xtQx + qtx
minimize
x
subject to:
where Q is
assumed to be
standard dual
QD:
Ax > b
symmetric and positive semi-definite.
of QP, see Dorn [1],
is given by
-1/2 ytQy + btX
maximize
y,X
subject
The
to:
-Qy + AtX = q
X > 0
The concern herein lies
data (Q,q,A,b)
in discovering properties of the problem
that allow the objective function
replaced by a norm,
in QP or QD to be
thus transforming QP or QD into an instance of
NP.
The rationale for exploring transformations of
program to the minimum
norm problem is
threefold.
a quadratic
First,
the
minimum norm problem has an immediate geometric interpretation that
may be useful
program.
in
Second,
the solution of a given quadratic
the analysis and
the minimum norm problem is
1
*1__1_______1______
__11
iI_____
a classical
optimization problem, and has received extensive study,
example,
[5].
Luenberger
Third, the author has
see for
recently
investigated an alternate duality theory for the minimum norm
problem
[2],
that
is applicable to quadratic
programs that can be so
transformed.
If the matrix Q is symmetric and positive semi-definite, then Q
can be written as
Q = MtM for some matrix M;
efficient
for constructing M are well-known, see e.g. Gill,
procedures
Murray, and Wright
[3].
Proposition 1.
At
If the system of
Qs
(1.1)
= -q
(1.2)
> 0
(1.3)
+ bt6
btT
> 0,
6 >
has a solution (N,
inequalities
q
+ Qs =
At6 -
linear
O,
6, s),
(1):
(1.4)
then the program QP is equivalent
to the
minimum norm problem
NP1:
{{Mx
minimize
x
subject
to:
+ Msl
Ax > b,
where M is any matrix for which
MtM = Q.
For any x satisfying Ax > b,
Proof:
1/2 xtMtMx + qtx = 1/2 xtMtMx +
-
1/2
and
stMtMs
tAx
similarly, using
1/2 xtQx
btR
+
+ qtx
> -bth,
< 1/2
> 1/2
tAx
IIMx + MsH 2
1/2
xtQx + qtx
+ stMtMx =
-
1/2
1/2 stMtMs
=
UIMx + Msll2
+ btr
,
(1.2),
IIMx
+ MsI1 2
-
1/2 stMtMs
equality is obtained throughout,
2
-
bt6.
But since
and 1/2 xtQx + qtx =
1/2 IIMx + Ms112
increasing in
-
1/2
IMx + Msl,
solution,
If q lies
then
(0,
Remark 2.
(,
particular, any
solves
6,
s)
> b will
index i,
(1).
= (0,
(n,
6,
s)
to
(1),
i < m,
transformed into a minimum norm problem.
tAx
from Q if CtC = Q.
for any feasible points x,
(respectively,
)
(respectively,
>).
Proposition 2.
= 6tb.
In
i > 0 will
for QP to be
Before turning to the
we first
to be monotonically transformable
and any
> b.
introduce some
Given the matrix Q of the program QP,
said to be derived
-bt6,
i > 0 or
for which
the system Ax
question of necessary conditions,
If Q is
(1).
=
bt
= ntb
ttAxand
satisfy
1
Q-lq) solves
Proposition 1 provides sufficient conditions
is
function can
Qs = q has a
i.e.,
In particular,
,
be an always-active constraint of
terminology.
strictly
[X]
For any solution
x that solves Ax
is
This expression
in the column space of Q,
0, s)
positive definite,
+ btn.
and so the quadratic objective
+ MslI.
be replaced by [IMx
Remark 1.
stMtMs
A norm
the norm
Cx + d
Cx + dl
is said
in the objective function of
x of QP.
[lCx
+ dl
>
if and only if 1/2 xtQx + qtx
If QP is feasible,
Cx
QP if
+ dl
> 1/2 xtQx + qtx
then QP can be monotonically
transformed to a minimum norm problem with objective function
llCx + d
(1)
derived from Q only if the system of linear inequalities
has a solution.
Proof:
Suppose that system
of the alternative,
scalar
e
(1) has no solution.
there exist
vectors x,
that satisfies:
3
Then, by a theorem
x and a nonnegative
Ax > be
Ax
> be
Qx = Qx
qtx > qtx
There are two cases to consider, depending on whether e is positive
or zero.
Case 1:
8
e >
.
Without loss of generality, we can assume that
= 1, and so x and x satisfy:
Ax
> b
Ax > b
Qx = Qx
qtx > qtx
If there exists C and d such that IICx + dll is a strictly monotonic
transformation of the objective function of QP and is derived from
Q,
then
Cx = Cx,
11Cx
+ dl
and so
>
Cx
+ dll.
IICx + dl
=
But Qx = Qx and Q = CtC
IlCx + dll,
a contradiction.
implies
Thus no
that
such
norm can be found.
Case 2:
e = 0.
Because QP is feasible, there exists x' that
satisfies Ax' > b.
Therefore (x' + x) and (x' + x) satisfy:
A(x' + x)
A(x'
+ x)
Q(x' +
qt(x'
>
b
> b
) =
+ x)
Q(x + x)
>
qt(x'
+ x)
and the proof follows that of case 1, with x replaced by (x' + x)
and x replaced by (x' +x).
[X]
Turning to the dual quadratic program QD, our first result is:
4
Av > b
(2.1)
Az < b
(2.2)
Qv - Qz = o
(2.3)
qtv
- qtz
has a solution
(v,
= 0
,
(2):
(2.4)
then the dual quadratic program QD is
z),
to the minimum norm problem
equivalent
NP2:
inequalities
If the system of linear
Proposition 3.
IlMy - Mz
minimize
y,X
subject
-Qy + AtX
to:
= q
X > 0O
where M is any matrix for which Q = MtM.
Proof:
-
1/2 ytQy + btX < -
+ vtQy = we
that is feasible
For any (y, X)
obtain -
1/2
HMy -
2
with qtz = qtv, we have
HMy -
Mz,
IlMy -
Mz11.
Remark 3.
applied
ztQZ.
and
1/2 ytMtMy + qtv
1/2
My -
1/2
2
+ qtz
+ 1/2
(2.2),
ztQz.
and combining this relation
ytQy + btX
This expression is
Mzl
Similarly, using
= -
1/2
flMy - Mzll2
strictly decreasing
in
so we can replace the maximand by the minimand
[X]
Proposition 3 is
to the
dual QD in
-
that Mz = Mv,
-
-
+ qtv + 1/2 vtQv.
1/2 ytQy + btX <
However, Qz = Qv implies
+ qtz + 1/2
+ vtAtX =
1/2 ytQy
Mv
for QD,
dual.
structurally the same as Proposition
1,
Proposition 3 is obtained by rewriting the
the format of the primal and then applying Proposition
In this sense, the two propositions are the same.
5
1.
In order
to prove a result about necessary conditions for QD to
be transformed
amended.
into a minimum norm problem,
Given the matrix Q of the dual quadratic program QD,
norm {ICy + EX + dl
E = O.
our notation must be
is said to
Cy + EX + dl
A norm
transformable in
to be monotonically
the objective function of QD
points (y, X),
(y, X)
(respectively,
) if
CtC = Q and
be derived from Q if
is said
of QD,
Cy + EX + d
and only if
-
the
1/2
>
if for any feasible
Cy
+ EX + dl
ytQy + btX
<
1/2 ytQy + btX
<).
(respectively,
Analogous to Proposition 2, we have:
Proposition 4.
transformed
inequalities
derived from Q only if the system of linear
(2)
has a solution.
The proof exactly parallels that of
is feasible,
vectors
feasible, then QD can be monotonically
to a minimum norm problem with objective function
RCy + EX + d
Proof:
If QD is
X,
then if
system (2)
Propostion 2.
has no solution,
If QD
there must exist
X, y that satisfy
-
Qy + AtX
= q
- Qy + AtX =
q
X >
X> 0
btX >
Thus
if
Cy
+ EX + dl
function of QD,
<
Cy + EX + d,
and
btX
is strictly monotonic
is derived
from Q,
in the objective
then ilCy + EX + d{
which is clearly a contradiction,
Thus no such norm can be found.
[X]
6
because E = 0.
Proposition 2 (and in the dual,
Proposition 4) shows
solvability of the system of linear inequalities
(1)
that the
is necessary
for QP to be transformed to the minimum norm program, provided that
the transformation is restricted by the monotonicity condition and
the condition that the norm be derived from Q.
restrictions are relaxed,
the solvability of the system (1)
longer the necessary condition,
Q 0 0
Then
(1)
b [31
0
have no solution.
However,
Cx + d,
C
this instance
1
where
and
1
=
d
-8]
this, note that for any x feasible in QP, qtx > 8,
To see
indeed, the optimal
UCx + d
is
solution is x*
=
=
(3 ,5 )t,
42(qtx -8)(qtx
with qtx*
and
8.
strictly increasing in qtx for qtx
solution.
However,
C is not
8.
derived from Q in
to the above transformation was
Also,
-8)
Thus
Cx + d
monotonically transformable from QP, even though system (1)
key
Let
is monotonically transformable to the minimum norm program
with objective function
which
is no
as the following example shows.
I, A =1
the inequalities
of QP
When these
has no
this example.
the judicious choice of
is
The
d,
based
on a known lower bound on the optimal value of QP.
If
the montonicity condition is relaxed,
of the set of optimal solutions
to QP allows us to write any
instance of QP as a minimum norm program.
optimal
solution x*
of QP
Is
unique
7
II
____l______sll
then prior knowledge
For example,
if the
(which it can be even if Q=O,
i.e.
QP is
just a
linear program),
then QP
is equivalent to
the
minimum norm program:
IIx-x*11
minimize
x
subject
to:
Ax > b
This transformation appears
somewhat pointless
in that the quadratic
program has been solved before the transformation
is even made.
Nevertheless,
the transformation can be accomplished
time, because
linear and convex quadratic programming are solvable
in polynomial time
quadratic
depends
[4].
program to be
on the class of
in polynomial
The question of necessary conditions for a
transformed into a norm program thus clearly
transformations
Note that the pair of
inequalities
that are allowed.
(2.1)
and
(2.2) of
Proposition 3 are described by reflecting the halfspaces defined by
the feasible region of QP.
A curious
Proposition 3 is the simultaneous
and Az
< b.
aff(x) and
This
issue is
rec(x) denote,
recession cone of
a set X,
in the system Ax
constraint
Ax > b,
issue raised in light of
solvability of
treated in the next proposition.
respectively,
the affine hull
see Rockafellar
[6].
b is said to be parallel
is redundant and Ajx = d > bj
i.e.,
the system Ax >
b
Let
and the
A constraint Ajx > bj
redundant
if the
for every x that satisfies
Ajx is constant for every x satisfying Ax >
b.
The
following proposition characterizes when the system Ax > b and Az <
both have a solution.
Proposition 5.
furthermore
Let
assume that
Ax > b is parallel
dim(rec(x))
X =
{xERnAx > b}
X
redundant.
and Z =
{zERnlAz < b),
and
and that no constraint of the system
Then Z
= dim(x).
8
if and
only if
b
Proof:
and Az
0
E
Suppose Z
*.
Then there
Let k = dim(x),
< b.
rec(x),
equality must hold,
dim aff{x,
and
so
xl,...,xk)
{xl-z
But A(xl-z)
...
elements of X,
for which
,xk-,
Conversely,
If
index set M
B, where
If k > 0,
> O,
*,
Z =
rec(X) > k,
x-z}
and hence equal
and assume that dim(rec(x))
such that Aax = b
interior of
k=O, X is a singleton
(x)
for every
satisfies Ax > b
X).
Furthermore,
Therefore,
there
(x is any element
(A.) = n-k.
= b,
and so x
= aff(rec(x))
for which Ajxi
=
< dim(x).
= Aj.
and Aaxi = 0,
{x E RnIAax = b).
= 0,
combination of the
tXA
i=l,...,k,
i=l,...,k,
rows
Thus X ba
of A,
bj,
If k > 0,
If there
< dim(x).
i.e.
and aff(xl,...,xk,o)
exists an index J
6
then Aj must be a nonnegative
i.e.
there exists Xa > 0 for which
and Ajx > bj
is a parallel redundant
constraint, violating the hypothesis of the proposition.
dim(rec(x))
If
Z,
there exists linearly independent vectors x 1 ,...,xk in rec(x),
> 0,
dim(X)
B is a parallel
Thus Ax
dim(rec(X))
=
sets a and
x E X, and
rank
and every index j
redundant constraint, whereby B=0.
that satisfy Axi
to k,
then the constraint
can be partitioned into disjoint
x of X that
contradicting Z=*.
there exist
< dim(x).
= {l,...,m}
the relative
< dim(x),
{xl-z,...,xk-z,
i.e.
the constraint matrix A has m rows,
exists an element
of
Thus dim
suppose
u B = M,
dim(rec(x))
has k linearly independent elements.
> 0,...,A(xk-z) > 0, A(x-z)
because dim(rec(x))
If k=O, then since
Therefore dim aff{x, x,...,xk, z}) > k,
-z)
are elements of rec(X).
= k > 0.
However, since
= dim(X).
= k.
z for which Ax > b
0.
>
dim(rec(x))
i.e.,
vectors xl,...,xk, all
and note that k
> 0 = k.
dim(rec(x))
exists x and
[X]
9
Thus
B
It
characterized in
Proposition 5 can be
the dimension of
terms of
and rec(X),
as opposed to a characterization in terms
multipliers
via a theorem of the alternative.
Q =
In
this example
linear inequalities
satisfy the
both programs will
0
q =
,
6,
(,
minimum norm program.
=
s)
(0,
A =
0
)
0,
solves
neither, or
(1),
(2).
(1) and/or
b
1
0
X
of dual
show, the primal, dual,
As the following examples
Example 1
b in
that the solvability of Ax > b and Az
is curious
3
=
whereby QP is a
However, Az < b has no solution, and
so the
transformation of the dual QD by Proposition 3 cannot be
accomplished.
Example 2
In this
(1,
Q =
example,
z = (-1,
)t,
transformable to
Example 3
1
-
=
s =
solves
(2);
(1,
1
)t solves
(1),
therefore
this
example,
1
and x =
both QP and QD are
norm programs.
Q
0
0 00
11
=
A =
10
1
0
In
1
1
0
0
A =
-
0,
6 =
= 0,
-l)t
I
, q
it is
straightforward to
10
=
0
-1
show that neither
(2) have a solution, and so the transformations
and 3 cannot be applied.
01b
-- I
-1
-
(1) nor
of Propositions
1
in Gauge Duality
An Application of Hidden Norms
One motivation
case.
norm
the author's recent
programming is
quadratic
gauge
for the study of hidden minimum norm programs
[2],
programs
of which
If QP can be converted
problem NP1,
the minimum norm problem is a specific
(through Proposition 1) to the minimum
standard
then either the
see
dual of NP1,
GNPI:
is given
[2],
(Lagrange) dual
to:
Mth
-
(stAt
by
=
AtX
Replacing the objective
(Lagrange)
subject
to:
dual
Af -
0O
function by 1/2 hth and then taking the
yields:
1/2 ftqf
minimize
f,e
0
+ bt)X = 1
>
QP:
or the
IhU 2
minimize
h,X
subject
standard
investigation of dual
of NP1 can be constructed.
gauge dual
The gauge
in
+ estQf + 1/2
stQse 2
-
e
be > 0.
Because QP is obtained from QP by two consecutive duality
constructions
(first
the gauge dual,
should expect QP and QP to
then the Lagrange
be equivalent.
The feasible
dual),
we
region of QP
is obtained by adding the extra variable e that scales the righthand-side b,
system Af
cone
-
and converts the
be > O,
constraints Ax > b to the homogeneous
replacing the polytope of QP by a polyhedral
in one higher dimension.
The sense of equivalence of QP and QP
is shown in the next proposition, which shows how optimal solutions
of QP and QP transform one to the other.
11
Let x be an
s) be a solution to (1).
,
Let (T,
6.
Proposition
solution to QP with optimal Karush-Kuhn-Tucker
optimal
, and define t = stQx + stQs
multipliers
if
(i)
t
O,
if
t = O,
= (
Ax
+ btn + btS.
Then
e) = (x/t, l/t) solves QP with optimal K-K-T
(f,
multipliers
(ii)
(K-K-T)
> b,
+
)/t.
Qx + Qs = 0 has a solution, and QP is
unbounded.
Let
(f, 8)
c.
Then:
(i)
solution to QP with optimal K-K-T multipliers
be an optimal
if e
O,
#
x =
f/e solves QP with optimal K-K-T multipliers
= C/e + S.
(ii)
The
if e
= O,
proof of this
then QP is infeasible.
proposition follows
[X]
the K-K-T
from an examination of
conditions and from direct substitution of the indicated
transformations.
The
program QP was
followed by the
the
(Lagrange)
obtained by taking the gauge dual of QP,
standard (Lagrange)
dual
dual.
If
standard quadratic program dual QD,
QP:
feasible
Aw > 0
to:
region of QP
recession cone of
homogeneous
(2),
then
Starting with the
and converting QD to the minimum
and then taking the gauge
(_qt
The
solves
dual of NP2 yields:
1/2 wtQw
minimize
w
subject
z)
QD is transformable to a minimum norm problem, and
the order of the dualization can be reversed.
norm problem NP2,
(v,
ZtQ)w =
1
(X)
(r)
is composed of the intersection of the
the feasible region of QP
(described by the
constraints Aw > 0) with a hyperplane
12
(described by
-
(_qt
tQ)
w
= 1)
that scales elements of the recession cone.
Analagous to Proposition 6, we have:
Proposition 7.
Let (v, z) be a solution to (2).
Let x be an
optimal solution to QP with optimal K-K-T multipliers A, and define
u =
-qtx + qtz -_
(i)
tQx +
if u # O,
w =
multipliers
(ii)
if u = 0,
X
tQz.
(x = n/u,
Then
)/u solves QP with optimal K-K-T
r = 1/u.
QP is infeasible.
Let w be an optimal solution to QP with optimal K-K-T multipliers
X, r.
Then
(i)
if r # 0,
x = w/r + v solves QP with optimal K-K-T
multipliers
(ii)
if r = 0,
=
t/r.
QP is unbounded.
[X]
It is hoped that the equivalences of QP to the programs QP and QP
given herein will be useful in applications of quadratic programming.
13
References
[1]
W.S. Dorn, "Duality in quadratic
18, 155-162, 1960.
[2]
R.M. Freund, "Dual gauge programs, with applications to quadratic
programming and the minimum-norm problem," to appear in
Mathematical Programming.
[3]
P. Gill, W. Murray, and M. Wright,
Academic Press (New York, 1981).
[4]
M.K. Kozlov, S. P. Tarasov, and L.G. Khachiyan, "Polynomial
solvability of convex quadratic programming," Doklady Academii
English translation, Soviet Mathematics
Nauk SSR 248(5) (1979)
Doklady 20(5) ( 1979)].
[5]
D.G. Luenberger, Optimization by vector space methods, John Wiley
& Sons (New York, 1969).
[6]
R.T. Rockafellar, Convex analysis, Princeton University Press
(Princeton, New Jersey, 1970).
14
programming,"
Quart. Appl. Math.
Practical Optimization,