Project Proposal
10-15-2013
Solving minimax problems with feasible sequential quadratic programming
Qianli Deng (Shally)
E-mail: dqianli@umd.edu
Advisor: Dr. Mark A. Austin
Department of Civil and Environmental Engineering
Institute for Systems Research
University of Maryland, College Park, MD 20742
E-mail: austin@isr.umd.edu
Abstract:
As one of the SQP-type programming, FSQP involves the solutions of quadratic programs as
subproblems. This algorithm is particularly suited to the various classes of engineering applications where
the number of variables is not too large but the evaluations of objective or constraint functions and of
their gradients are highly time consuming. This project is intended to implement the Feasible Sequential
Quadratic Programming (FSQP) algorithm as a tool for solving certain nonlinear constrained optimization
problems with feasible iterates.
Keywords: Nonlinear optimization; minimax problem; sequential quadratic programming; feasible
iterates
1
Project Proposal
10-15-2013
1. Introduction
The role of optimization in both engineering analysis and design is continually expanding. As a result, the
faster and more powerful optimization algorithms are in constant demand. Motivated by problems from
engineering analysis and design, feasible sequential quadratic programming (FSQP) are developed as a
dramatic reduction in the amount of computation while still enjoying the same global and fast local
convergence properties.
The application of FSQP includes all branches of engineering, medicine, physics, astronomy, economics
and finances, which abounds of special interest. In particular, the algorithms are particularly appropriate
for problems where the number of variables is not so large, while the function evaluations are expensive
and feasibility of iterates is desirable. But for problems with large numbers of variables, FSQP might not
be a good fit. The minimax problems with large numbers of objective functions or inequality constraints,
such as finely discretized semi-infinite optimization problems, could be handled effectively, for instance,
problems involving time or frequency responses of dynamical systems.
The typical constrained minimax problem is in the following format showing in eq.(1).
fi ( x)}
minimize max{
f
iI
x X
(1)
where f i is smooth. FSQP generates a sequence {xk } such that g j ( xk )  0 for all j and k .
I f  {1,..., n f } where n f stands for the number of objective functions f . If n f  0 , I f   . X is a set
of points x 
n
satisfying the following constraints, as shown in eq.(2).



 g ( x) 
 j


 h j ( x) 

bl  x  bu
g j ( x)  0,
c j  ni , x  d j  ni  0,
h j ( x)  0,
a j  ne , x  b j  ne  0,
j  1,..., ni
j  ni  1,..., ti
j  1,..., ne
(2)
j  ne  1,..., te
where g j and h j are smooth; ni stands for the number of nonlinear inequality constraints; ti stands for
the total number of inequality constraints; ne stands for the number of nonlinear equality constraints, and
te stands for the total number of inequality constraints. bl stands for the constraint of lower boundary
and bu stands for the constraint of upper boundary. a j , b j , c j and d j stand for the parameters of
linear constraints.
2
Project Proposal
10-15-2013
2. Algorithm
To solve the constrained minimax problem using FSQP, five steps are taken as a sequence: 1)
initialization; 2) computation of a search arc; 3) arch search; 4) updates; and 5) stop.
Parameters setting as:   0.1 ,   0.01 ,   0.1 ,   0.5 ,   2.1 , 1   2  2.5 , t  0.1 , 1  1 ,
 s  1 ,   5 , 0  s
Data: x0 
n
1.
,   0 ,  e  0 and p0, j   2 for j  1,..., ne .
2.1 Initialization
FSQP solves the original problem with nonlinear equality constraints by solving a modified optimization
problem with only linear constraints and nonlinear inequality constraints. Set k  0 , H 0  the identity
matrix as the initial Hessian matrix, and an initial guess x00 . If x00 is infeasible for linear constraints, a
point x0 satisfying these constraints is generated by solving the strictly convex quadratic program. If x00
or newly generated initial guess is infeasible for the nonlinear inequality constraints, a point x0 is
generated satisfying all constraints by iterating on the problem of minimizing the maximum of the
nonlinear inequality constraints.
hj  0 ,
j  1, 2,..., ne
(3)
fi ( x)} is replaced by the modified objective function
and the original objective function max{
f
iI
ne
f m ( x, p)  max
 fi ( x)   p j h j ( x)
f
iI
j 1
(4)
where p j , j  1,..., ne , which are positive penalty parameters that are iteratively adjusted. For
j  1,..., ne , replace h j ( x ) by  h j ( x) whenever h j ( x0 )  0 .
Based on a sequential quadratic programming iteration, an Armijo-type line search is used when
minimizing the maximum of the nonlinear inequality constraints to generate an initial feasible point x0 . If
x0 is infeasible for some constraint other than a nonlinear equality constraint, substitute a feasible point.
3
Project Proposal
10-15-2013
2.2 Stop check
If || d k0 ||  and
ne
| h (x ) |  
j 1
j
k
e,
iteration stops and return the value xk .
2.3 Computation of a search arc
From an initial guess x0 , the following steps are repeated as xk converges to the solution. The four steps
are used to compute a search arc: 1) compute d k0 ; 2) compute d k1 ; 3) compute d k ; and 4) compute d k .
d k0 stands for the direction of descent for the objective function and d k1 stands for an arbitrary feasible
descent direction. d k stands for the feasible descent direction between the directions of d k0 and d k1 . And
d k stands for a second order correction which could be deemed as a “bent” of the search direction. Inner
relations among the parameters are displayed in Figure 1.
Figure 1. Calculations of direction d in FSQP
2.3.1 Compute d k0
Compute d k0 , the solution of the quadratic program QP( xk , H k , pk ) . At each iteration k , the quadratic
program QP( xk , H k , pk ) that yields the SQP direction d k0 is defined at xk for H k symmetric positive
definite by
4
Project Proposal
10-15-2013

min
d0

 s.t.







1 0
d , H k d 0  f '( xk , d 0 , pk )
2
bl  xk  d 0  bu
g j ( xk )  g j ( xk ), d 0  0,
j  1,..., ti
h j ( xk )  h j ( xk ), d 0  0,
j  1,..., ne
(5)
j  1,..., te  ne
a j , xk  d 0  b j
Given I  I f , following notation is made.
f I ( x)  max{ f i ( x)}
iI
(6)
ne
f '( x, d , p)  max{
fi ( x)  f i ( x), d }  f I f ( x)   p j hi ( x), d
f
iI
j 1
(7)
2.3.2 Compute d k1
Compute d k1 by solving the strictly convex quadratic program

n
 d1min
, 

 s.t.









 0 1 0 1
dk  d , dk  d  
2
bl  xk  d 1  bu
f '( xk , d 1 , pk )  
g j ( xk )  g j ( xk ), d 1  
j  1,..., ni
c j , xk  d  d j
j  1,..., ti  ni
h j ( xk )  h j ( xk ), d 1  
j  1,..., ne
a j , xk  d 1  b j
j  1,..., te  ne
1
(8)
2.3.3 Compute d k
Set d k  (1   k )d k0   k d k1 with  k || d k0 || /(|| d k0 || vk ) , where vk  max(0.5,|| d k1 ||1 ) .
5
Project Proposal
10-15-2013
2.3.4 Compute d k
In order to avoid the Maratos effect and guarantee a superlinear rate of convergence, a second order
correction d  d ( x, d , H ) 
n
is used to “bend” the search direction. That is an Armijo-type search is
performance along the arc x  td  t 2 d . The Maratos correction d k is taken as the solution of QP.
1

(d k  d ), H k (d k  d )  f I' f ( d ) ( xk  d k , xk , d , pk )
n
min
k
k
d
2

bl  xk  d k  d  bu
 s.t.

g j ( xk  d k )  g j ( xk ), d   min(v || d k ||,|| d k || 2 )



c j  ni , xk  d k  d  d j ni


h j ( xk  d k )  h j ( xk ), d   min(v || d k ||,|| d k || 2 )


a j , xk  d k  d  b j

j  I kg (d k )  { j : j  ni }
j  I kg (d k )  { j : j  ni }
(9)
j  1,..., ne
j  1,..., te  ne
Compute d k by solving the strictly convex quadratic program
ne
f '( x, d , p)  max{ fi ( x  d )  fi ( x), d }  f I ( x)   p j hi ( x), d
iI
j 1
(10)
The set of active constraints by
I kg (d k )  { j {1,..., ti }:| g j ( xk ) | 0.2 d k  g j ( xk ) }
(11)
{ j {1,..., ti }:| k , j  0}
2.4 Arc search
If ni  ne  0 ,  k  f '( xk , d k , pk ) , while if ni  ne  0 ,  k   d k0 , H k d k0 . Compute t k , the first
number t in the sequence {1,  ,  2 ,...} satisfying
fm ( xk  tdk  t 2dk , pk )  f m ( xk , pk )  t k
(12)
g j ( xk  td k  t 2 d k )  0 , j  1,..., ni .
(13)
c j ni , xk  td k  t 2 d k  d j ni , j  ni & j  I kg (d k )
6
(14)
Project Proposal
10-15-2013
h j ( xk  td k  t 2 d k )  0 , j  1,..., ne
(15)
2.5 Updates
The updating scheme for the Hessian estimates H k is also defined in implementation. Using BFGS
formula with Powell’s modification to compute the new approximation H k 1 as the Hessian of the
Lagrangian, where H k 1  H kT1  0.
k 1
xk 1  xk
(16)
 k 1
 x L( xk 1 , ˆk )  x L( xk , ˆk )
(17)
where L( x, ˆ ) stands for the Lagrange function, and ˆk stands for the Lagrange multiplier.
A scalar  k 1  (0,1] is then defined by

1,
kT1 k 1  0.2kT1 H kk 1

 0.8kT1 H kk 1
otherwise
 T H    T 
 k 1 k k 1 k 1 k 1
 k 1
Defining  k 1 
n
(18)
as
 k 1  k 1   k 1  (1   k 1 )  H k  k 1
(19)
The rank two Hessian update is
H k 1  H k 
H kk 1kT1H k  k 1 kT1

kT1H kk 1 kT1 k 1
(20)
Then, the new point is set as
xk 1  xk  tk dk  tk2dk
(21)
Solve the unconstrained quadratic problem in 
nf
ti
j 1
j 1
minne ||   k , j f j ( xk 1 )  k   k , j g j ( xk 1 ) 

te

j  ne 1
For j  1,..., ne , update the penalty parameters as
7
ne
k , j h j ( xk 1 )   j h j ( xk 1 ) ||2
j 1
(22)
Project Proposal
10-15-2013
pk 1, j
pk , j   j  1 ;

pk , j


|| d k0 || min{0.5 , 0.01  m }

otherwise
max{1   j ,  pk , j }
(23)
Set k  k  1.
3. Implementation
Hardware: the program will be developed and implemented on my personal computer.
Software: the program will be developed by Java.
4. Validation
Example. For x 
3
, the objective function min3 f ( x)  ( x1  3x2  x3 )  4( x1  x2 )
2
x
The constraints are
s.t.
xi  0, i  1, 2,3
x  6 x2  4 x3  3  0
1  x1  x2  x3  0
3
1
Feasible initial guess x0  (0.1, 0.7, 0.2)T , f ( x0 )  7.2
Global minimizer x*  (0, 0,1)T , f ( x* )  1
5. Testing
Example. The objective function min6 max fi ( x)
x
s.t.
i 1,...,163
 x1  s  0
x1  x2  s  0
x2  x3  s  0
x3  x4  s  0
x4  x5  s  0
x5  x6  s  0
x6  3.5  s  0
8
2
Project Proposal
Where fi ( x) 
i 
10-15-2013
6

1 2
  cos(7 sin i )   cos(2 x j sin i ) 
15 15 
j 1

 (8.5  0.5i )
180
, i  1, 2,...,163
s  0.425
The feasible initial guess is x0  (0.5,1,1.5, 2, 2.5,3)T
With the corresponding value of the objective function max fi ( x0 )  0.2205
i 1,...,163
Other practical problems such as the wind turbine setting could also be applied if time permits.
6. Project Schedule
Time
October
November
December
February
Tasks

Literature review;

Specify the implementation module details;

Structure the implementation;

Develop the quadratic programming module;

Unconstrained quadratic program;

Strictly convex quadratic program;

Validate the quadratic programming module;

Develop the feasible initial point module;

Validate the feasible initial point module;

Develop the Gradient and Hessian matrix calculation module;

Validate the Gradient and Hessian matrix calculation module;

Midterm project report and presentation;

Develop Step II - the computation of a search arc;

Validate Step II;

Develop Step IV - the Hessian matrix and the penalty parameters updating;

Validate Step IV;
9
Project Proposal
March
April
May
10-15-2013

Develop Step III - the arc search;

Integrate the program;

Debug and well document the program;

Validate and test the program;

Develop the user interface if time available;

Final project report and presentation;
7. Deliverables

Project proposal;

Algorithm description;

Java code;

Validation results;

Test database;

Test results;

Project reports;

Presentations;
8. Bibliography
Charalambous, Conn and AR Conn. "An Efficient Method to Solve the Minimax Problem Directly."
SIAM Journal on Numerical Analysis 15, no. 1 (1978): 162-187.
Goldfarb, Donald and Ashok Idnani. "A Numerically Stable Dual Method for Solving Strictly Convex
Quadratic Programs." Mathematical programming 27, no. 1 (1983): 1-33.
Lawrence, Craig T and André L Tits. "Nonlinear Equality Constraints in Feasible Sequential Quadratic
Programming∗." Optimization Methods and Software 6, no. 4 (1996): 265-282.
Lawrence, Craig T and André L Tits. "A Computationally Efficient Feasible Sequential Quadratic
Programming Algorithm." Siam Journal on optimization 11, no. 4 (2001): 1092-1118.
Li, Wu and John Swetits. "A New Algorithm for Solving Strictly Convex Quadratic Programs." SIAM
Journal on Optimization 7, no. 3 (1997): 595-619.
10
Project Proposal
10-15-2013
Madsen, Kaj and Hans Schjaer-Jacobsen. "Linearly Constrained Minimax Optimization." Mathematical
Programming 14, no. 1 (1978): 208-223.
Mayne, DQ and E Polak. "Feasible Directions Algorithms for Optimization Problems with Equality and
Inequality Constraints." Mathematical Programming 11, no. 1 (1976): 67-80.
Watson, GA. "The Minimax Solution of an Overdetermined System of Non-Linear Equations." IMA
Journal of Applied Mathematics 23, no. 2 (1979): 167-180.
11