SMU Course EMIS 8381
Nonlinear Programming
January 14, 2012
Hossam Zaki
1
Outline
• Introductions
– 5 minutes each
• Course Scope
– What is an optimization
problem?
– Examplea
– Classification of
Optimization Problems
• Course Syllabus
– Description & Pre
requisites
– Text & References
– Calendar &Grading
• On Line Resources
• Course Topics
• Course Context
– DSS Example & Paradigm
– Career Roles & Questions
2
Scope
3
What is an Optimization
Problem?
• Optimization problems involve the
selection of values of a number of
interrelated variables in such a way to
optimize on one or more selection criteria
designed to measure the quality of each
selection
4
Example: Price Optimization
• A distributor sells 100K products to 20K
customers needs to determine every month the
set of prices that will maximize margin while
satisfying product and customer pricing rules
such as
– GbB (Good, better, Best),
– PGS (Platinum, Gold, Silver)
• Sources of nonlinearity
– Margin = (price – cost) * quantity
– Quantity = a – b * price
5
High Level Price Optimization
Volume
Market
Response
Model
V*
Price
Margin $
MG*
Price
Optimizer
Price
Increase
Zone
Price
Decrease
Zone
Pm*
Price
6
Example: Portfolio Optimization
• Markowitz Model
– Given stocks returns and covariances, find a
minimum variance portfolio that will generate
a pre specified average return. That is,
determine how much to invest in each stock
to minimize risk at a certain level of return
– Sources of nonlinearity
• Risk = w(i) * w(j) * cov(i,j)
7
Example: Regression
• Given N(1000) observations, find the
values of the (linear or nonlinear) model
parameters that will minimize the distance
between model and observed
• Sources of nonlinearity
– Minimum [ Y (mode) – Y (observed)]2
– Y (model ) = a0 + sum ( aj * xj)
8
Optimization Problem Statement
• Select the values of a vector x in Rn
• From a set of feasible vectors X in Rn
– that satisfy a group of algebraic constraints
• In such a way that will optimize the value
of a real-valued function f(x)
9
Terminology
• Decision Variables = x
• Objective Function(s) = f(x)
• Feasible Region = X
– Constraints defining X
• Inequality constraints : g(x)<=0
• Equality constraints : h(x)=0
10
Optimization Problems
Three Classifiers
Decision
Variables
Optimization
Problems
Constraints
Objective
11
DV Attributes
Continuous
Sequential
Discrete
DV
Infinite
Dims
Finite
Dims
12
Objective Attributes
One
Non Linear
Linear
Many
Objective
Non
Differentiable
From
Simulation
Closed form
Differentiable
13
Constraints Attributes
Un
Constrained
Stochastic
Linear
Simple
Bounds
Deterministic
Constraints
GUB
Nonlinear
Block
Diagonal
Network
14
Optimization Problems
Finite DV
Continuous DV
Decision Variables (DV)
• Discrete
• Continuous
• Finite
• Infinite
Objective
• One
One Obj
• Many
Differentiable• Obj
Differentiable
Deterministic• Obj
Non Differentiable
• Closed Form
• From Simulation
• Linear
• Non linear
• Deterministic
• Stochastic
Constraints
Unconstrained
• Unconstrained
Linear Constraints
• Constrained
• Linear
• Non linear
• Simple bounds
• GUB
• Network
• Block diagonal
15
© 2006, 2007 Zilliant, Inc. -- CONFIDENTIAL
Course Scope
• Decision Variable
– Continuous
– Finite dimensions
• Objective
–
–
–
–
Single
Closed form or From Simulation
Linear or Nonlinear
Deterministic
• Constraints
– Deterministic
16
Exercise # 1
• Identify the attributes that define the following
optimization problems and provide one simple
example. Display the results in table format
1.
2.
3.
4.
5.
Linear Programming
Goal Programming
Network Programming
Integer Programming
Non differentiable
Optimization
6. Global Optimization
7. Dynamic Programming
8. Stochastic Programming
9. Quadratic Programming
10. Fractional Programming
11. Geometric Programming
12. Multi Objective
Optimization
17
Course Syllabus
18
Course Description
• This course discusses, presents and
explains the most important methods and
results used to model and solve nonlinear
optimization problems.
19
Prerequisites
• Advanced calculus (partial derivatives)
• Linear algebra (vectors and matrices).
• Knowledge on linear programming is
helpful but not required (we will cover what
we need).
20
Text & References
• Text Book
– "Nonlinear Programming: Theory and Algorithms" by, M. Bazaraa,
H. Sherali and C Shetty, 3rd Edition, John Wiley, ISBN
• References
– “Handbooks in OR & MS, Volume 1, Optimization”, Nemhauser et al
editors, North Holand, Chapters I and III, 1989
– Dennis & Schnabel, “Numerical Methods for Unconstrained
Optimization”, Prentice-Hall, 1983
– Gill, Murray and Wright, “Practical Optimization”, Academic Press,
1981
– D. Luenberger, “Linear and Nonlinear Programming”, 2nd Edition,
Addison Wesely, 1984
– D. Bertsekas, “Nonlinear Programming”, 2nd Edition, Athena
Scientific, 1999
21
Class Calendar
• Class (Section001)
– Saturday 9 AM-11:50 PM
• 10 Minute break every 50 minutes
– Meets in 205 Junkins
• 14 Class Periods
– First Class Period: January 14
• No Class: 3/10 (SB) & 4/7 (GF is on 4/6)
– Last Class Period: April 28
22
Grading
• In-class Exams (70%)
– Mid Term on 3/2 = (30%)
– Final on 4/28 = (40%)
• Term Paper & Presentation (20%)
– A nonlinear optimization topic, e.g. algorithm,
application and /or software demo not covered in
class
– Due on 3/31
• Homework (10%)
23
On Line Resources
24
Mathematical Programming
Glossary
http://glossary.computing.society.informs.org/index.php?page=N.html
• General Information - A list of dictionaries, suggested methods of
citation, and contribution instructions.
• The Nature of Mathematical Programming - See this for basic terms
and a standard form of a mathematical program that is used
throughout this glossary.
• Notation - Read this to clarify notation.
• Supplements - A list of supplements that are cited by entries.
• Myths and Counter Examples - Some common and uncommon
misconceptions.
• Tours - Collections of Glossary entries for a particular subject.
• Biographies - Some notes on famous mathematicians.
• Please remember this is a glossary, not a survey, so no attempt is
made to cite credits.
25
1998 Nonlinear Programming
Software Survey
• http://www.lionhrtpub.com/orms/surveys/nlp/nlp.
html
• The information in this survey was provided by
the vendors in response to a questionnaire
developed by Stephen Nash. The survey should
not be considered as comprehensive, but rather
as a representation of available NLP packages.
The listings are limited to products that fit the
parameters of the survey as outlined in the
accompanying article.
26
Nonlinear Programming
Frequently Asked Questions
http://wwwunix.mcs.anl.gov/otc/Guide/faq/nonlinearprogramming-faq.html
• Q1. "What is Nonlinear Programming?"
• Q2. "What software is there for nonlinear
optimization?"
• Q4. "What references are there in this
field?"
• Q5. "What's available online in this field?"
27
NEOS Guide
http://www-fp.mcs.anl.gov/OTC/Guide/
• The Optimization Tree. Our thumbnail sketch of
optimization (also known as numerical optimization or
mathematical programming) and its various sub
disciplines.
• The Optimization Software Guide. Information on
software packages from the book by Moré and Wright,
updated for the NEOS Guide.
• Frequently Asked Questions on Linear and Nonlinear
Programming. These are the FAQs initiated by John
Gregory, now maintained by Bob Fourer as part of the
NEOS Guide.
28
MIT Nonlinear Course
• Prof. Dimitri Bertsekas
http://ocw.mit.edu/OcwWeb/ElectricalEngineering-and-Computer-Science/6252JNon-linearProgrammingSpring2003/CourseHome/
29
UCLA Nonlinear EE Course
• Prof. Lieven Vandenberghe
http://www.ee.ucla.edu/ee236b/
30
Course Topics
31
Course Topics
• Chapter 1
– Intro
• Appendix A
– Math Review
• Chapter 2
– Convex Sets
• Chapter 3
– Convex Functions
• Chapter 4
• Chapter 6
– Lagragian Duality
• Chapter 8
– Unconstrained Opt
• Chapter 9
– Penalty and Barrier
• Chapter 10
– Methods of Feasible
Directions
– Optimality Conditions
32
Exercise # 2
Plot and solve graphically:
Maximize
x1
Subject to
h1(x1,x2) =
x12 – x2 + a = 0
h2(x1,x2) = – x1 + x22 + a = 0
For the following values of a:
a = 1, 0.25, 0 and -1
33
Course Context
34
DSS Example
Cost of new
Resources
Aggregate
Demand
Profile
Timing
Constraints
Optimization Model
Business Rules
Capacity
Configuration
Alternative
Resources
Existing
Capacity
Feedback
Stochastic
Daily
Demand
Stochastic
Show-up rates
Simulation Model
Daily
Resources
Stochastic
Success Rates
What if
Operations
Characteristics
35
Decision Support System Development Methodology
12. Estimate Lift
(Improvement)
1. Understand Business Context and
Formulate Problem Statement in English
11. Perform What-if
2. Develop DSS High Level Design
10. Prepare & analyze output.
Validate results
9. Solve Real Problems
3. Formulate Mathematical Model
DV, Objective, Constraints, Goals
ITERATE
4. Receive, Clean and
Synthesize Business Data.
Create DB
8. Refine Formulation:
Aggregate, Decompose,
Transform
5. Estimate and/or
Forecast needed
data, e.g. demand
7. Prepare & Solve Small
Sample Problems
6. Develop or
Select Solver,
prepare input
files, connect to
DB
DV = Decision Variable
DB = Database
36
How will the course help u with
career questions?
Which problems to invest $ in?
What is the expected lift?
Develop in house or buy?
Service or license?
How to gain tech diff over competitors?
P
Should we productize?
SVP
How many scientists do we need?
VP
Director
What is the impact of a
data change?
How to validate results?
Reformulate?
How to formulate?
Senior Manager
Manager
Senior Scientist
Scientist, OR Specialist, Consultant
How many person-months?
How to prioritize tasks?
Which solver to use?
Which parameters?
Which algorithm?
37
Questions?
38