ISE 5406: Optimization II
Lecture 1
Manish Bansal
Industrial and Systems Engineering
Virginia Tech
Reading for this lecture
Chapter 1 and Section 2.1 of Nocedal and Wright (2006)
Chapter 1 of Bazaraa et al. (2006)
ISE 5406: Lecture 1
1/17
Mathematical Formulation or Modeling
Keywords:
Objective: For example, profit, time, cost, etc.
Variables (or unknowns)
Constraints (or restrictions)
1
If a model is too simplistic, it might not give useful insight
into the practical problem,
2
But if it is too complex, it may become too difficult to solve.
ISE 5406: Lecture 1
2/17
Mathematical Modeling
Age Problem
Find age of Justin and Brad, given that sum of their ages is equal to 76
and after 7 years, age of Brad will be twice the age of Justin.
Solution:
ISE 5406: Lecture 1
3/17
Mathematical Modeling
Age Problem
Find age of Justin and Brad, given that sum of their ages is equal to 76
and after 7 years, age of Brad will be twice the age of Justin.
Solution: Let x and y denote the age of Justin and Brad, respectively.
Then
x + y = 76
(1)
2(x + 7) = y + 7.
(2)
On solving the foregoing equations we get
x = 23 and y = 53.
ISE 5406: Lecture 1
3/17
Mathematical Modeling
Age Problem
Find age of Justin and Brad, given that sum of their ages is equal to 76
and after 7 years, age of Brad will be twice the age of Justin.
Solution: Let x and y denote the age of Justin and Brad, respectively.
Then
x + y = 76
(1)
2(x + 7) = y + 7.
(2)
On solving the foregoing equations we get
x = 23 and y = 53.
Geometric Interpretation. Intersection of two lines.
ISE 5406: Lecture 1
3/17
Mathematical Optimization
Production Planning Problem
In a manufacturing firm, given the demand for two days along with per
unit production and inventory costs, decide the production schedule for
both days.
Production
Inventory s0
Demand
x1
1
10
x2
s1
2
s2
4
ISE 5406: Lecture 1
4/17
Mathematical Optimization
Production Planning Problem
In a manufacturing firm, given the demand for two days along with per
unit production and inventory costs, decide the production schedule for
both days.
Production
Inventory s0
x1
1
x2
s1
2
Decision Variables:
s2
x1 : Production on Day 1
x2 : Production on Day 2
s0 : Initial Inventory
Demand
10
4
s1 : Inventory at the end of Day 1
s2 : Inventory at the end of Day 2
ISE 5406: Lecture 1
4/17
Mathematical Optimization
Production Planning Problem
In a manufacturing firm, given the demand for two days along with per
unit production and inventory costs, decide the production schedule for
both days.
Production
Inventory s0
x1
1
x2
s1
2
Decision Variables:
s2
x1 : Production on Day 1
x2 : Production on Day 2
s0 : Initial Inventory
Demand
10
4
s1 : Inventory at the end of Day 1
s2 : Inventory at the end of Day 2
Data:
Per unit production cost on Day 1 and Day 2: $10 and $16, respectively
Per unit inventory cost = $5
ISE 5406: Lecture 1
4/17
ISE 5406: Lecture 1
5/17
Mathematical Optimization: Numerical Algorithms
There is no universal optimization algorithm.
There are numerous algorithms, each of which is tailored to a
particular type of optimization problem:
1
Deterministic Optimization
Linear Programming [ISE 5405]
Nonlinear Programming [ISE 5406]
Integer Programming [ISE 6414]
..
.
2
Optimization under Uncertainty:
Robust Optimization
Stochastic Optimization [ISE 5984]
..
.
ISE 5406: Lecture 1
6/17
Mathematical Optimization: Optimality Conditions
After an optimization algorithm has been applied to the
model, we must be able to recognize whether it has succeeded
in its task of finding a solution.
In many cases, there are elegant mathematical expressions,
known as Optimality Conditions, for checking that the current
set of variables is indeed the solution of the problem.
If the optimality conditions are not satisfied, they may give
useful information on how the current estimate of the solution
can be improved.
Sensitivity Analysis. Reveals the sensitivity of the solution to
changes in the model and data.
ISE 5406: Lecture 1
7/17
Mathematical Optimization
Definition (in words): Optimization is the maximization or
minimization of a function subject to constraints on its
variables.
Mathematically, optimization problem can be written as
Minimize f (x)
subject to gi (x) = 0,
i ∈ E,
hi (x) ≥ 0,
i ∈ I,
x ∈ Rn ,
where f , gi , and hi are scalar-valued functions of the variables
x, and I, E are sets of indices.
ISE 5406: Lecture 1
8/17
Mathematical Optimization
Linear Program:
max {cx : Ax ≤ b, x ≥ 0}
x2
(1)
LP solution space
(2)
ISE 5406: Lecture 1
x1
9/17
Mathematical Optimization
Integer Linear Program:
max {cx : Ax ≤ b, x integral}
x2
(1)
IP solution space
(2)
ISE 5406: Lecture 1
x1
9/17
Mathematical Optimization
Integer Linear Program:
max {cx : Ax ≤ b, x integral}
x2
Cutting Plane Algorithm:
Cutting planes: Cut off part
(1)
of LP relaxed region
(2)
solution space
ISE 5406: Lecture 1
x1
9/17
Mathematical Optimization
Integer Linear Program:
max {cx : Ax ≤ b, x integral}
x2
Cutting Plane Algorithm:
Cutting planes: Cut off part
of LP relaxed region
Strongest Cuts: Facets
(1)
solution space
(2)
ISE 5406: Lecture 1
x1
9/17
Nonlinear Programming
min (x1 − 2)2 + (x2 − 1)2
s.t. x21 − x2 ≤ 0,
x1 + x2 ≤ 2.
ISE 5406: Lecture 1
10/17
Nonlinear Programming
Constrained Optimization Problem:
min (x1 − 2)2 + (x2 − 1)2
s.t. x21 − x2 ≤ 0,
x1 + x2 ≤ 2.
ISE 5406: Lecture 1
10/17
Nonlinear Programming
Constrained Optimization Problem:
min (x1 − 2)2 + (x2 − 1)2
s.t. x21 − x2 ≤ 0,
x1 + x2 ≤ 2.
Unconstrained Optimization Problem:
min x1 + x1 x2 + x22
s.t. (x1 , x2 ) ∈ R2 .
ISE 5406: Lecture 1
10/17
Nonlinear Programming: Facility Location Problem
Suppose that there are n markets with known demands and
locations. These demands are to be met from m warehouses of
known capacities.
The problem is to determine the locations of the warehouses so that
the total distance weighted by the shipment from the warehouses to
the markets is minimized.
Data Parameters:
1
ci : capacity of warehouse i for i = 1, . . . , m
2
(aj , bj ): known location of market j for j = 1, . . . , n
3
rj : known demand at market j for j = 1, . . . , n
ISE 5406: Lecture 1
11/17
Decision Variables:
1
(xi , yi ): unknown location of warehouse i for i = 1, . . . , m.
2
wij and dij : unit shipped and distance, respectively, from
warehouse i to market area j for i = 1, . . . , m; j = 1, . . . , n
m X
n
X
Objective Function: Minimize
wij dij
i=1 j=1
Constraints:
1
Known capacity of warehouse i:
n
X
wij ≤ ci ,
for i = 1, . . . , m
j=1
2
Known demand of market j:
m
X
wij = rj ,
for j = 1, . . . , n
i=1
3
Non-negativity constraints: wij ≥ 0 for all i, j.
ISE 5406: Lecture 1
12/17
Decision Variables:
1
2
(xi , yi ): unknown location of warehouse i for i = 1, . . . , m.
wij and dij : unit shipped and distance, respectively, from
warehouse i to market area j for i = 1, . . . , m; j = 1, . . . , n
Objective Function: Minimize
m X
n
X
wij dij
i=1 j=1
Rectilinear Distance: dij = |xi − aj | + |yi − bj |
Euclidean Distance: dij = [(xi − aj )2 + (yi − bj )2 ]1/2
lp Norm Metrics: dij = [|xi − aj |p + |yi − bj |p ]1/p
Note: If the location of warehouse are fixed, i.e. the dij values are
known, and this problem reduces to a linear program known as
transportation problem.
ISE 5406: Lecture 1
12/17
Fundamentals of Unconstrained Optimization
Minimize f (x)
where x ∈ Rn is a real vector with n ≥ 1 components and f : Rn → R is
a smooth function.
Usually, we lack a global perspective on the function f .
All we know are the values of f and maybe some of its derivatives
at a set of points x0 , x1 , x2 , . . ..
Design algorithm(s) which chooses these points without using too
much computer time or storage.
Often, the information about f or derivative does not come cheaply,
so we usually prefer algorithms that do not call for this information
unnecessarily.
ISE 5406: Lecture 1
13/17
Least-Square Problem
Given the plots measurements y1 , y2 , ..., ym of a signal taken at
times t1 , t2 , ..., tm , find a curve that fits this data.
We choose to model it by the function:
2
φ(t; x) = x1 + x2 e−(x3 −t)
/x4
+ x5 cos(x6 t)
where real number xi , i = 1, . . . , 6 are the parameters of the model.
ISE 5406: Lecture 1
14/17
Least-Square Problem
We would like to choose x1 , x2 , . . . , x6 to make the model values
φ(tj ; x) fit the observed data yj , j = 1, . . . , m, as closely as
possible.
Variables: x = (x1 , . . . , x6 ) ∈ R6
Objective function: Minimize the residuals which measure the
discrepancy between the model and the observed data, i.e.
2
min f (x) = r12 (x) + . . . + rm
(x)
where rj (x) = yj − φ(tj , x) for j = 1, . . . , m.
ISE 5406: Lecture 1
15/17
Local and Global Minimizers
Global Minimizer. A point x∗ is a global minimizer if
f (x∗ ) ≤ f (x) for all x.
Local Minimizer. A point x∗ is a local minimizer if there is a
neighborhood N of x∗ such that
f (x∗ ) ≤ f (x) for all x ∈ N .
Note that a neighborhood of x∗ is simply an open set that contains x∗ .
More precisely, -neighborhood given a point x ∈ Rn is the set
N (x) := {y : ||y − x||2 < }
Strict Local Minimizer. A point x∗ is a strict (or strong) local
minimizer if there is a neighborhood N of x∗ such that
f (x∗ ) < f (x) for all x ∈ N .
ISE 5406: Lecture 1
16/17
Isolated Local Minimizer. A point x∗ is an isolated local minimizer if
there is a neighborhood N of x∗ such that x∗ is the only local minimizer
in N .
While strict local minimizers are not always isolated, it is true that
all isolated local minimizers are strict.
Readings for Next Lecture
Chapter 2 of Bazaraa et al. (2006)
ISE 5406: Lecture 1
17/17