Chapter 6 Method of
Successive Quadratic
Programming
Professor Shi-Shang Jang
National Tsing-Hua University
Chemical Engineering Department
6-1 Quadratic Programming Problems

Problem:
Min c1 x1  c 2 x 2    c N x N  h11 x1  h12 x1 x 2    h1 N x1 x N
2
 h 22 x 2    h 2 N x 2 x N  h NN x N
2
2
s .t . a 11 x1    a N 1 x N  b1

a M 1 x1    a MN x N  b M
x0
In compact
matrix form :
min f ( x )  cx 
1
2
s .t . Ax  b
x0
x ' Qx
Classical Constrained Regression
Problem
min e ' Ie

s .t .Y  x   e
Where β is an m×1 vector of regression coefficients to
be estimated, e is an n×1 vector of error variables, Y
is an n×1 observations on the dependent variable and
X is an n×m matrix of observations on the
independent variables. It is clear that the classical
regression problem is a quadratic programming
problem.
Example- A numerical problem
f ( x )   6 x1  2 x1  2 x1 x 2  2 x 2
2
2
s .t . x1  x 2  2
x1 , x 2  0
compact
matrix
min f  x    6
s .t .1
 x1 
1   2
 x2 
form
 x1 
0     x1
 x2 
 4
x 2 
 2
 2   x1 
 
4   x2 
Example- A numerical problem
c   6
0
 4
Q  
 2
A  1
b  2 
1
 2

4 
Example-A scientist’s model
A scientist has observed a certain quantity Q
as a function of t. She has a good reason to
believe that there is a physical law relating t
and Q that takes the form:
Q(t)=asint+bcost+c
She wants to have the “best” possible idea of the
value of the coefficients a,b and c using the
results of her n experiments:

Example-A scientist’s model

(t1,q1; t2,q2;…, tn,qn). Taking into account
that her experiments are not perfect and also
perhaps that the model is not rigorous
enough, she does not expect to find a perfect
fit. So defines an error term:
Ei=qi-q(ti)
Her idea is to minimize:
n
Z  min

i 1
2
ei
Example-A scientist’s model
What if a=3;b=2;c=5 with 15
experimental data?
n
min f ( a , b , c , e1 ,  , e n ) 
e
2
i
i 1
s .t .q 1  a sin t1  b cos t1  c  e1  0
q 2  a sin t 2  b cos t 2  c  e 2  0

q n  a sin t n  b cos t n  c  e n  0
Classical Least Square Fitting





Problem: Given a set of data (x1,y1; x2,y2;…,
xn,yn). Find a set of parameters ={a, b,
c,…} such that:
ei2=yi-(axi+bxi2+..)
And iei2 is minimized
The solution of the unconstrained case:
=(XTX)-1XTY=quasi_inverse(X)Y
Example: Curve Fitting
>> x=linspace(0,5);
>> for i=1:100
y(i)=3*x(i)^2+2*x(i)+5;
y(i)=y(i)+randn(1,1);
xx(i,1)=x(i)*x(i);xx(i,2)=x(i);xx(i,3)=1;
end
>> theta=inv(xx'*xx)*xx'*y'
theta =
2.9983
2.1215
4.5839
>> for i=1:100
yy(i)=theta(1)*x(i)^2+theta(2)*x(i)+theta(3);
end
>> plot(x,yy)
Example: Curve Fitting-Continued
100
90
80
70
60
50
40
30
20
10
0
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Program QUADPROG


QUADPROG Quadratic programming.
X=QUADPROG(H,f,A,b) solves the quadratic programming problem:



min 0.5*x'*H*x + f'*x subject to: A*x <= b
x



X=QUADPROG(H,f,A,b,Aeq,beq) solves the problem above while additionally
satisfying the equality constraints Aeq*x = beq.






X=QUADPROG(H,f,A,b,Aeq,beq,LB,UB) defines a set of lower and upper
bounds on the design variables, X, so that the solution is in the
range LB <= X <= UB. Use empty matrices for LB and UB
if no bounds exist. Set LB(i) = -Inf if X(i) is unbounded below;
set UB(i) = Inf if X(i) is unbounded above.
Curve Fitting Using Quadprog
x=linspace(0,5);
for i=1:100
y(i)=3*x(i)^2+2*x(i)+5;
y(i)=y(i)+randn(1);
xx(i,1)=x(i)*x(i);xx(i,2)=x(i);xx(i,3)=1;
end
a=eye(100);
a1=zeros(1,100);
H=[a1; a1; a1; a];
f=zeros(103,1);
HH=[f f f H];
Aeq=[xx eye(100)];beq=y';
A=[];b=[];
X=QUADPROG(HH,f,A,b,Aeq,beq) ;
XX=X';
XX(1)
XX(2)
XX(3)
The Solution Method-Complementary
Pivot Problem

K-T Condition for a QP:
c x
T
Q  Q   u  vA  0
Ax  b
x0
ux  0
u 0
T
The Solution Method-Complementary
Pivot Problem

Find vectors w and z such that
w  Mz  q
w0
z 0
w z 0
T
The Solution Method-Complementary
Pivot Problem

The solution of the complementary pivot
problem is equivalent to find a K-T point of
the original QP problem and hence the
solution is sufficiently found since the QP is a
convex objected (quadratic function) and
qualified constrained (linear constraints).
6-2 The Successive QP Approach

Consider a quadratic function:
q  x ; x 0   f  x 0    f  x 0  x  x 0  

1
2
 x  x 0 T  2 f  x 0  x  x 0 
The objective is to find d=(x-x0) such that the
following problem can be minimized

min  f x


T
d 
t 
t 
k
t
j
T

d  f x
T
2
T
d 0
2
t
j
1
   h x 
x      g x    d
s .t .h k x
g
t 
0
t 
d
Example
1
2
min f ( x )  6 x1 x 2  x 2 x1
s .t .h ( x )  x1 x 2  2  0
g ( x )  x1  x 2  1  0
1
x 0  2
The correspond ing QP can be found :
 23
min 
 4
s .t .1
1
1 T  3/8
47 

d  d 
2
4 
  25 / 4
2 d  0
1 d  2  0
 25 / 4 
d
24 
Example
Algorithm



Step 1: Formulate the QP problem
Step 2: Solve the QP problem, and set
x(t+1)=x(t)+*d
Step 3: Check Convergency
Remarks

Original SQP is subjected to the following
two problems:
(1) The second derivative of the objective
function is generally difficult to obtain.
(2) There is no guarantee that the second
derivative of any function at any point is
positive definite. In case the hessian is not
positive definite, then the QP is failed.
6-3 Quadratic Approximation of the
Lagrangian Function
L ( x )  f ( x )  vh ( x )  ug ( x )
min f ( x )
s .t .h ( x )  0
g (x)  0
can be approximat
min  f  x  d 
T
1
2
ed by :
d  x L  x , u , v d
T




s .t .h x   h x d
T
0
g x    g x  d  0
T
2
Algorithm-The Variable Metric
Method


Given initial estimates x0,u0,v0, and a symmetric
positive definitive matrix H0
Step 1: Solve the problem:

min  f x

t 

T
d 
1
T
d H
t 
d
2
   h x  d  0
 
 
g x    g x  d  0
s .t .h x
t

t 
t 
t
T
T
Step 2: Select the step size α along d(t), and set
x(t+1)=x(t)+ α d(t)
Algorithm-The Variable Metric
Method (VMCON)



Step 3: Check convergence
Step 4: Update H(t) using the gradient difference
BSF update formulation:
z x
t 1 
x

y   xL x
t 
t 1 
,u
t 1 
,v
t 1 
   L x   , u   , v   
t
t
t
x
Define
1

   0.8 z T H  t  z
 z T H t  z  z T y

if
z y  0 .2 z H
T
T
otherwise
then
w   y  1    H
t 
z
Finally
H
t 1 
 H
t 

H
t 
T
T
zz H
z H
t 
z
t 

ww
T
T
z w
t 
z
Line Search on the SQP direction

Given d from QP, instead of using the original
objective function, it is more useful to implement the
following penalty function:
 K
P ( x , R )  f ( x )  R   hk  x  
 k 1
K
 f (x) 

k
hk  x  
k 1
J




min(
0
,
g
x


j
j 1

J
min( 0 , g j  x 
j
j 1
where for the first iteration
 k  vk ,  j  u j
and for all subsequent
t 

t 
 k  max  v k ,


t 
j
1
2
iterationt
 
t 1 
k

 t  1
 max  u j , 
2

 t 1 
j
t 
 vk
t 
 uj
,




Numerical Example
1
2
min f ( x )  6 x1 x 2  x 2 x1
s .t .h ( x )  x1 x 2  2  0
g ( x )  x1  x 2  1  0
 23
1 ;  f  
 4
x 0  2
The first run will
 23
min 
 4
s .t .1
1
d
2 d  0
1 d  2  0
0 
gives :
  4 , 2 
T
2  ;  g  1
take the form by choosing
47 
1 T 1

d  d 
4 
2
0
The solution
T
47 

 ;  h  1
4 
0
d
1
T
H
1 
 I
1
T
MATLAB Program - VMCON
% DEFINE GLOBAL VARIABLES
global mu
global sigma
global vv
global uu
global dd
global X0
%
% INITIALIZE THE PROBLEM
H=eye(2);X0=[2;1];uu=0;vv=0;
dx=100;mu=0;sigma=0;iter=0;
%
% THE WHILE LOOP
%
while abs(dx)>0.000001
iter=iter+1;
%
% FIND THE GRADIENT df
%
[df,dlgn]=sqp_Lagrangian(X0);
%
% DEFINE Aq,beq such that Aq*d=beq
%
A, b such that Ad<b
MATLAB Program – VMCON-Continued
A=[-1 -1];b=(X0(1)+X0(2)-1);
Aeq=[X0(2) X0(1)];
beq=(X0(1)*X0(2)-2);
%
% SOLVE THE SUB-QUADRATIC PROBLEM FOR d
%
[ss,FVAL,EXITFLAG,OUTPUT,LAMBDA]=QUADPROG(H,df,A,b,Aeq,beq);
dd=ss';
%
% GET THE LARAGNGE=MULTIPLIERS
%
uu=LAMBDA.ineqlin;
vv=LAMBDA.eqlin;
%
% PREPARE THE LINE SEARCH
%
mu=max(abs(vv),0.5*(mu+abs(vv)));
sigma=max(abs(uu),0.5*(sigma+abs(uu)));
a0=0.1;
%
% LINE SEARCH ALONG dd WITH THE PENALTY FUNCTION AS THE OBJ
%
alopt = FMINSEARCH('sqp_penalty',a0);
MATLAB Program – VMCON-Continued
%
% GET THE NEW POINT
%
X=X0+alopt*dd';
%
% PREPARE TO UPDATE THE APPROXIMATE HESSIAN
%
z=X-X0;
[df,dlgn_1]=sqp_Lagrangian(X0);
[df,dlgn_2]=sqp_Lagrangian(X);
y=dlgn_2-dlgn_1;
zz=z'*H*z;
zzp=0.2*zz;
zy=z'*y;
if(zy>zzp)
theta=1;
else
theta=0.8*z'*H*z/(zz-zy);
end
w=theta*y+(1-theta)*H*z;
%
% UPDATE THE APPROXIMATE HESSIAN
%
H=H-(H*z*z'*H)/zz+w*w'/(z'*w);
MATLAB Program – VMCON-Continued
%
% CALCULATE THE OBJ AND EQUALITY CONSTRAINT AT THE NEW POINT
%
hh=X(1)*X(2)-2;
ddx=X-X0;
obj=sqp_obj(X);
%
% UPDATE THE NEW POINT
%
X0=X;
%
% CHECK THE CONVERGENCY
%
dx=sqrt(ddx(1)^2+ddx(2)^2);
end
%
% OUTPUT THE RESULT
%
iter
X
obj
hh
MATLAB Program Using Fmincon
> help fmincon
FMINCON Finds a constrained minimum of a function of
several variables.
FMINCON solves problems of the form:
min F(X) subject to: A*X <= B, Aeq*X = Beq (linear
constraints)
X
C(X) <= 0, Ceq(X) = 0 (nonlinear
constraints)
LB <= X <= UB
MATLAB Program Using FminconContinued
X=FMINCON(FUN,X0,A,B,Aeq,Beq,LB,UB,NON
LCON) subjects the minimization to the
constraints defined in NONLCON. The function
NONLCON accepts X and returns
the vectors C and Ceq, representing the nonlinear
inequalities and equalities
respectively. FMINCON minimizes FUN such that
C(X)<=0 and Ceq(X)=0.
MATLAB Program Using FminconExample
X0=[2;1];
A=[-1 -1];
B=1;
Aeq=[];Beq=[];LB=[];UB=[];
X=FMINCON('sqp',X0,A,B,Aeq,Beq,LB,UB,'sqp_nlcon')
function [c,ceq]=sqp_nlcon(x)
c=[];
ceq=x(1)*x(2)-2;