Non-Linear Optimization
Distinguishing Features
Common Examples
EOQ
Balancing Risks
Minimizing Risk
15.057 Spring 03 Vande Vate
1
Hierarchy of Models
Network Flows
Linear Programs
Mixed Integer Linear Programs
15.057 Spring 03 Vande Vate
2
A More Academic View
Mixed Integer
Linear
Programs
Non-Convex Optimization
Network Flows
Linear Programs
Convex Optimization
15.057 Spring 03 Vande Vate
3
A More Academic View
Integer Models
Non-Convex Optimization
Networks & Linear Models
Convex Optimization
15.057 Spring 03 Vande Vate
4
Convexity
The Distinguishing Feature
Separates Hard from Easy
Convex Combination
Weighted Average
Non-negative weights
Weights sum to 1
15.057 Spring 03 Vande Vate
5
Convex Functions
Convex Function
The
function
lies
below
the line
800
Examples?
700
600
Convex
combinations of the
values
500
400
300
200
100
0
0
5
10
15
15.057 Spring 03 Vande Vate
20
25
30
6
What’s “Easy”
Find the minimum of a Convex Function
A local minimum is a global minimum
15.057 Spring 03 Vande Vate
7
Convex Set
A set S is CONVEX if every convex
combination of points in S is also in S
The set of points above a convex function
Convex Function
800
700
600
500
400
300
200
100
0
0
5
15.057 Spring 03 Vande Vate
10
15
20
25
30
8
What’s “Easy”
Find the minimum of a Convex
Function over (subject to) a Convex
Set
15.057 Spring 03 Vande Vate
9
Concave Function
Concave Function
250
The
function
lies
ABOVE
the line
Examples?
200
150
100
50
0
5
10
15
15.057 Spring 03 Vande Vate
20
25
30
10
What’s “Easy”
Find the maximum of a Concave
Function over (subject to) a Convex
Set.
15.057 Spring 03 Vande Vate
11
Academic Questions
Is a linear function convex or concave?
Do the feasible solutions of a linear
program form a convex set?
Do the feasible solutions of an integer
program form a convex set?
15.057 Spring 03 Vande Vate
12
Cost
Ugly - Hard
Volume
15.057 Spring 03 Vande Vate
13
Integer Programming is “Hard”
3
Why?
Coils Limit
2
Production Capacity
1
Bands Limit
Coils
1
2
3
15.057 Spring 03 Vande Vate
4
Bands
14
Review
Convex Optimization
Convex (min) or Concave (max) objective
Convex feasible region
Non-Convex Optimization
Stochastic Optimization
Incorporates Randomness
15.057 Spring 03 Vande Vate
15
Agenda
Convex Optimization
Unconstrained Optimization
Constrained Optimization
Non-Convex Optimization
Convexification
Heuristics
15.057 Spring 03 Vande Vate
16
Convex Optimization
Unconstrained Optimization
If the partial derivatives exist (smooth)
find a point where the gradient is 0
Otherwise (not smooth)
find point where 0 is a subgradient
15.057 Spring 03 Vande Vate
17
Unconstrained Convex
Optimization
Smooth
Find a point where the Gradient is 0
Find a solution to ∇f(x) = 0
Analytically (when possible)
Iteratively otherwise
15.057 Spring 03 Vande Vate
18
Solving ∇f(x) = 0
Newton’s Method
Approximate using gradient
∇f(y) ≈ ∇f(x) + ½(y-x)tHx(y-x)
Computing next iterate involves inverting Hx
Quasi-Newton Methods
Approximate H and update the approximation
so we can easily update the inverse
(BFGS) Broyden, Fletcher, Goldfarb, Shanno
15.057 Spring 03 Vande Vate
19
Line Search
Newton/Quasi-Newton Methods yield
direction to next iterate
1-dimensional search in this direction
Several methods
15.057 Spring 03 Vande Vate
20
Unconstrained Convex Optimization
Non-smooth
Subgradient Optimization
Find a point where 0 is a subgradient
15.057 Spring 03 Vande Vate
21
What’s a Subgradient
Like a gradient
f(y) ≥ f(x) +γx(y-x)
f(x) is a minimum point
f(y) = f(x) - 2(y-x)
x
f(y) = f(x) + (y-x)
1 ≥ γx ≥ -2
a 0 is a subgradient if and only if ...
15.057 Spring 03 Vande Vate
22
Steepest Descent
If 0 is not a subgradient at x, subgradient
indicates where to go
Direction of steepest descent
Find the best point in that direction
line search
15.057 Spring 03 Vande Vate
23
Examples
EOQ Model
Balancing Risk
Minimizing Risk
15.057 Spring 03 Vande Vate
24
EOQ
How large should each order be
Trade-off
Cost of Inventory (known)
Cost of transactions (what?)
Larger orders
Higher Inventory Cost
Lower Ordering Costs
15.057 Spring 03 Vande Vate
25
The Idea
Increase the order size until the
incremental cost of holding the last item
equals the incremental savings in ordering
costs
If the costs exceed the savings?
If the savings exceed the costs?
15.057 Spring 03 Vande Vate
26
Modeling Costs
Q is the order quantity
Average inventory level is
Q/2
h*c is the Inv. Cost. in $/unit/year
Total Inventory Cost
h*c*Q/2
Last item contributes what to inventory
cost?
h*c/2
15.057 Spring 03 Vande Vate
27
Modeling Costs
D is the annual demand
How many orders do we place?
D/Q
Transaction cost is A per transaction
Total Transaction Cost
AD/Q
15.057 Spring 03 Vande Vate
28
Total Cost
Total Cost = h*cQ/2 + AD/Q
Total Cost
120
What kind of function?
100
80
60
40
20
0
0
10
20
30
40
50
60
70
Order Quantity
15.057 Spring 03 Vande Vate
29
Incremental Savings
What does the last item save?
Savings of Last Item
AD/(Q-1) - AD/Q
[ADQ - AD(Q-1)]/[Q(Q-1)] ~ AD/Q2
Order up to the point that extra carrying
costs match incremental savings
h*c/2 = AD/Q2
Q2 = 2AD/(h*c)
Q = √2AD/(h*c)
15.057 Spring 03 Vande Vate
30
Key Assumptions?
Known constant rate of demand
15.057 Spring 03 Vande Vate
31
Value?
No one can agree on the ordering cost
Each value of the ordering cost implies
A value of Q from which we get
An inventory investment c*Q/2
A number of orders per year: D/Q
Trace the balance for each value of
ordering costs
15.057 Spring 03 Vande Vate
32
The EOQ Trade off
Known values
Annual Demand D
Product value c
Inventory carrying percentage h
Unknown transaction cost A
For each value of A
Calculate Q = √2AD/(h*c)
Calculate Inventory Investment cQ/2
Calculate Annual Orders D/Q
15.057 Spring 03 Vande Vate
33
The Tradeoff Benchmark
EOQ Trade off
$3,000
$2,500
Where are you?
Inventory Investment
$2,000
Where can you be?
$1,500
But wait…!
What prevents
getting there?
$1,000
$500
$0
0
50
100
150
200
250
300
Orders Per Year
15.057 Spring 03 Vande Vate
34
Balancing Risks
15.057 Spring 03 Vande Vate
35
Variability
Some events are inherently variable
When customers arrive
How many customers arrive
Transit times
Daily usage Stock Prices
...
Hard to predict exactly
Dice
Lotteries
15.057 Spring 03 Vande Vate
36
Random Variables
Examples
Outcome of rolling a dice
Closing Stock price
Daily usage
Time between customer arrivals
Transit time
Seasonal Demand
15.057 Spring 03 Vande Vate
37
Distribution
The values of a random variable and their
frequencies
Example: Rolling 2 Fair Die
34
21
Number of Outcomes
Fraction of Outcomes
Value
33
43
44
32
42
52
53
54
22
23
24
25
35
45
55
31
41
51
61
62
63
64
65
11
12
13
14
15
16
26
36
46
56
66
1
2
3
4
5
6
5
4
3
2
1
0.028 0.056 0.083 0.111 0.139 0.167 0.139 0.111 0.083 0.056 0.028
2
3
4
5
6
7
8
9
10
11
12
15.057 Spring 03 Vande Vate
38
Theoretical vs Empirical
Empirical Distribution
Based on observations
Value
Number of Outcomes
Fraction of Outcomes
2
1
0.03
3
2
0.06
4
1
0.03
5
5
0.14
6
3
0.08
7
9
0.25
8
8
0.22
9
3
0.08
10
3
0.08
11
1
0.03
12
-
Theoretical Distribution
Based on a model
Value
Fraction of Outcomes
2
0.03
3
0.06
4
0.08
5
0.11
6
0.14
15.057 Spring 03 Vande Vate
7
0.17
8
0.14
9
0.11
10
0.08
11
0.06
12
0.03
39
Empirical vs Theoretical
One Perspective: If the die are fair
and we roll many many times,
empirical should match theoretical.
Another Perspective: If the die are
reasonably fair, the theoretical is close
and saves the trouble of rolling.
15.057 Spring 03 Vande Vate
40
Empirical vs Theoretical
The Empirical Distribution is flawed
because it relies on limited
observations
The Theoretical Distribution is
flawed because it necessarily
ignores details about reality
Exactitude? It’s random.
15.057 Spring 03 Vande Vate
41
Continuous vs Discrete
Discrete
Value of dice
Number of units sold
…
Continuous
Essentially, if we measure it, it’s discrete
Theoretical convenience
15.057 Spring 03 Vande Vate
42
Probability
Discrete: What’s the probability we roll a
12 with two fair die:
1/36
Continuous: What’s the probability the
temperature will be exactly 72.00o F
tomorrow at noon EST?
Zero!
Events: What’s the probability that the
temperature will be at least 72o F
tomorrow at noon EST?
15.057 Spring 03 Vande Vate
43
Continuous Distribution
Standard Normal Distribution
0.45
0.4
0.35
0.3
Probability the random variable
is greater than 2 is the area
under the curve above 2
0.25
0.2
0.15
0.1
0.05
0
-10
-8
-6
-4
-2
0
15.057 Spring 03 Vande Vate
2
4
6
8
10
44
Total Probability
Empirical, Theoretical, Continuous,
Discrete, …
Probability is between 0 and 1
Total Probability (over all possible outcomes) is 1
15.057 Spring 03 Vande Vate
45
Summary Stats
The Mean
Weights each outcome by its
probability
AKA
Expected Value
Average
May not even be possible
Example:
Win $1 on Heads, nothing on Tails
15.057 Spring 03 Vande Vate
46
Summary Stats
The Variance
Measures spread about the mean
How unpredictable is the thing
Nomal Distributions with Different Variances
0.45
0.4
Variance 1
0.35
0.3
Which would you rather manage?
0.25
0.2
0.15
Variance 9
0.1
0.05
0
-10
-8
-6
-4
-2
15.057 Spring 03 Vande Vate
0
2
4
6
8
10
47
Variance
Nomal Distributions with Different Variances
0.45
0.4
0.35
0.3
Variance 1
0.25
0.2
0.15
Variance 9
0.1
0.05
0
-10
-8
-6
-4
-2
0
15.057 Spring 03 Vande Vate
2
4
6
8
10
48
Std. Deviation
Variance is measured in units squared
Think sum of squared errors
Standard Deviation is the square root
It’s measured in the same units as the
random variable
The two rise and fall together
Coefficient of Variation
Standard Deviation/Mean
Spread relative to the Average
15.057 Spring 03 Vande Vate
49
Balancing Risk
Basic Insight
Bet on the outcome of a variable process
Choose a value
You pay $0.5/unit for the amount your bet
exceeds the outcome
You earn the smaller of your value and the
outcome
Question: What value do you choose?
15.057 Spring 03 Vande Vate
50
Similar to...
Anything you are familiar with?
15.057 Spring 03 Vande Vate
51
The Distribution
Distribution
0.45
Mean
5
Std. Dev. 1
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
0
2
4
6
15.057 Spring 03 Vande Vate
8
10
12
52
The Idea
Balance the risks
Look at the last item
What did it promise?
What risk did it pose?
If Promise is greater than the risk?
If the Risk is greater than the
promise?
15.057 Spring 03 Vande Vate
53
Measuring Risk and Return
Revenue from the last item
$1 if the Outcome is greater,$0 otherwise
Expected Revenue
$1*Probability Outcome is greater than our choice
Risk posed by last item
$0.5 if the Outcome is smaller, $0 otherwise
Expected Risk
$0.5*Probability Outcome is smaller than our choice
15.057 Spring 03 Vande Vate
54
Balancing Risk and Reward
Expected Revenue
$1*Probability Outcome is greater
than our choice
Expected Risk
$0.5*Probability Outcome is smaller
than our choice
How are probabilities Related?
15.057 Spring 03 Vande Vate
55
Risk & Reward
Distribution
0.45
Prob. Outcome is smaller
0.4
Our choice
0.35
How are they
related?
Prob. Outcome is larger
0.2
0.15
0.1
0.05
0
0
2
4
6
15.057 Spring 03 Vande Vate
8
10
12
56
Balance
Expected Revenue
$1*(1- Probability Outcome is smaller than our
choice)
Expected Risk
$0.5*Probability Outcome is smaller than our choice
Set these equal
1*(1-P) = 0.5*P
1 = 1.5*P
2/3 = P = Probability Outcome is smaller than our choice
15.057 Spring 03 Vande Vate
57
Making the Choice
Distribution
0.45
0.4
Prob. Outcome is smaller
Our choice
0.35
0.3
Cumulative
Probability
0.25
0.2
2/3
0.15
0.1
0.05
0
0
2
4
6
15.057 Spring 03 Vande Vate
8
10
12
58
Constrained Optimization
Feasible Direction techniques
Eliminating constraints
Implicit Function
Penalty Methods
Duality
15.057 Spring 03 Vande Vate
59
Feasible Directions
Unconstrained
Optimization
X
Start at a point: x0
Identify an^ improving direction: d
Feasible
Find a best ^ solution in direction d: x + εd
Feasible
Repeat
A Feasible direction: one you can move in
A Feasible solution: don’t move too far.
Typically for Convex feasible region
15.057 Spring 03 Vande Vate
60
Constrained Optimization
Penalty Methods
Move constraints to objective with penalties or barriers
As solution approaches the constraint the penalty
increases
Example:
min f(x)
s.t. x2 ≤ 3x
=> min f(x) + t/(3x - x2)
as x2 approaches 3x, penalty increases rapidly
15.057 Spring 03 Vande Vate
61
Relatively reliable tools for
Quadratic objective
Linear constraints
Continuous variables
15.057 Spring 03 Vande Vate
62
Summary
“Easy Problems”
Convex Minimization
Concave Maximization
Unconstrained Optimization
Local gradient information
Constrained problems
Tricks for reducing to unconstrained or simply
constrained problems
NLP tools practical only for “smaller” problems
15.057 Spring 03 Vande Vate
63