Bayesian Optimization
with
Experimental Constraints
Javad Azimi
Advisor: Dr. Xiaoli Fern
PhD Proposal Exam
April 2012
1
Outline
• Introduction to Bayesian Optimization
• Completed Works
– Constrained Bayesian Optimization
– Batch Bayesian Optimization
– Scheduling Methods for Bayesian Optimization
• Future Works
– Hybrid Bayesian optimization
• Timeline
2
Bayesian Optimization
• We have a black box function and
we don’t know anything about its
distribution
• We are able to sample the
function but it is very expensive
• We are interested to find the
maximizer (minimizer) of the
function
• Assumption:
– lipschitz continuity
Introduction to BO
Constrained BO
Batch BO
Scheduling
3
Future Work
Big Picture
Current Experiments
Posterior Model
Select Experiment(s)
Run Experiment(s)
Introduction to BO
Constrained BO
Batch BO
Scheduling
4
Future Work
Posterior Model (1):
Regression approaches
• Simulates the unknown function distribution based on
the prior
– Deterministic (Classical Linear Regression,…)
• There is a deterministic prediction for each point x in the
input space
– Stochastic (Bayesian regression, Gaussian Process,…)
• There is a distribution over the prediction for each point x in
the input space. (i.e. Normal distribution)
– Example
• Deterministic: f(x1)=y1, f(x2)=y2
• Stochastic: f(x1)=N(y1,0.2) f(x2)=N(y2,5)
Introduction to BO
Constrained BO
Batch BO
Scheduling
5
Future Work
Posterior Model (2):
Gaussian Process
• Gaussian Process is used to
build the posterior model
Points with high
output expectation
– The prediction output at any
point is a normal random
variable
– Variance is independent from
observation y
Points with high
output variance
– The mean is a linear combination
of observation y
6
Selection Criterion
• Goal: Which point should be selected next to
get to the maximizer of the function faster.
MM
MUI
MPI
MEI
• Maximum Mean (MM)
– Selects the points which has the highest output mean
– Purely exploitative
• Maximum Upper bound Interval (MUI)
– Select point with highest 95% upper confidence bound
– Purely explorative approach
• Maximum Probability of Improvement (MPI)
– It computes the probability that the output is more than
(1+m) times of the best current observation , m>0.
– Explorative and Exploitative
• Maximum Expected of Improvement (MEI)
– Similar to MPI but parameter free
– It simply computes the expected amount of
improvement after sampling at any point
Introduction to BO
Constrained BO
Batch BO
Scheduling
7
Future Work
Motivating Application:
Fuel Cell
This is how an MFC works
Nano-structure of
anode significantly
impact the electricity
production.
e-
e-
Oxidation
products (CO2)
bacteria
Fuel (organic matter)
SEM image of bacteria sp. on
Ni nanoparticle enhanced
carbon fibers.
O2
H+
H2 O
Cathode
Anode
We should optimize anode nano-structure to maximize power by selecting a set of experiment.
8
Introduction to BO
Constrained BO
Batch BO
Scheduling
Future Work
Other Applications
•
•
•
•
•
Financial Investment
Reinforcement Learning
Drug test
Destructive tests
And …
Introduction to BO
Constrained BO
Batch BO
Scheduling
9
Future Work
Constrained Bayesian optimization
(AAAI 2010, to be submitted Journal)
Introduction to BO
Constrained BO
Batch BO
Scheduling
10
Future Work
Problem Definition(1)
• BO assumes that we can ask for specific
experiment
• This is unreasonable assumption in many
applications
– In Fuel Cell it takes many trials to create a nanostructure with specific requested properties.
– Costly to fulfill
Introduction to BO
Constrained BO
Batch BO
Scheduling
11
Future Work
Problem Definition(2)
• It is less costly to fulfill a request that specifies ranges for the
nanostructure properties
• E.g. run an experiment with Averaged Area in range r1 and
Average Circularity in range r2
• We will call such requests “constrained experiments”
Averaged Area
Space of Experiments
Constrained Experiment 1
• large ranges
• low cost
• high uncertainty about which
experiment will be run
Constrained Experiment 2
• small ranges
• high cost
• low uncertainty about which
experiment will be run
Average Circularity
Introduction to BO
Constrained BO
Batch BO
Scheduling
12
Future Work
Proposed Approach
• We introduced two different formulation
• Non Sequential
– Select all experiments at the same time
• Sequential
– Only one constraint experiment is selected at each iteration
• Two challenges:
– How to compute heuristics for constrained experiment?
– How to take experimental cost into account?(which has been
ignored by most of the approaches in BO)
Introduction to BO
Constrained BO
Batch BO
Scheduling
13
Future Work
Non-Sequential
• All experiments must be chosen at the same
time
• Objective function:
– A sub set of experiments (with cost B) which jointly
have the highest expected maximum is selected, i.e.
E[Max(.)]
Introduction to BO
Constrained BO
Batch BO
Scheduling
14
Future Work
Submodularity
• It simply means adding an element to the smaller set provides us
with more improvement than adding an element to the larger set
• Example: We show that max (.) is submodular
– S1={1, 2, 4}, S2={1, 2, 4, 8}, (S1 is a subset of S2), g=max(.) and x=6
– g(S1, x) - g(S1)=2, g(S2,x)-g(S2)=0
• E[max(.)] over a set of jointly normal random variable is a
submodular function
• Greedy algorithm provides us with a “constant” approximation
bound
Introduction to BO
Constrained BO
Batch BO
Scheduling
15
Future Work
Greedy Algorithm
Introduction to BO
Constrained BO
Batch BO
Scheduling
16
Future Work
Sequential Policies
• Having the posterior distribution of p(y|x,D) and
px(.|D) we can calculate the posterior of the output
of each constrained experiment which has a closed
form solution
Discretization
Level
Input space
• Therefore we can compute standard BO heuristics for
constrained experiments
– There are closed form solution for these heuristics
Introduction to BO
Constrained BO
Batch BO
Scheduling
17
Future Work
Budgeted Constrained
• We are limited with Budget B.
• Unfortunately heuristics will typically select the smallest and most
costly constrained experiments which is not a good use of budget
-Low
uncertainty
-High
uncertainty
-Better
heuristic value
-Lower
heuristic value
-Expensive
-Cheap
• How can we consider the cost of each constrained experiment in
making the decision?
– Cost Normalized Policy (CN)
– Constraint Minimum Cost Policy(CMC)
Introduction to BO
Constrained BO
Batch BO
Scheduling
18
Future Work
Cost Normalized Policy
• It selects the constrained experiment
achieving the highest expected improvement
per unit cost
• We report this approach for MEI policy only
Introduction to BO
Constrained BO
Batch BO
Scheduling
19
Future Work
Constraint Minimum Cost Policy (CMC)
• Motivation:
1. Approximately maximizes the heuristic value
2. Has expected improvement at least as great as spending
the same amount of budget on random experiments
•
Example:
Cost=10 random
Cost=4 random
Poor heuristic value: not
select due to 1st
condition
Introduction to BO
Very expensive: 10 random
experiments likely to be
better
Constrained BO
Batch BO
Cost=5 random
Selected
Constrained experiment
Scheduling
20
Future Work
Results (1)
Real
Fuel Cell
CMC-MEI
Cosines
Rosenbrock
Introduction to BO
Constrained BO
Batch BO
Scheduling
21
Future Work
Results (2)
Real
Rosenbrock
Introduction to BO
Constrained BO
Fuel Cell
NS
Cosines
Batch BO
Scheduling
22
Future Work
Batch Bayesian Optimization
(NIPS 2010)
Sometimes it is better to select batch.
(Javad Azimi)
23
Motivation
• Traditional BO approach request a single experiment at each
iteration
• This is not time efficient when running an experiment is very
time consuming and there is enough facilities to run up to k
experiments concurrently
• We would like to improve performance per unit time by
selecting/running k experiments in parallel
• A good batch approach can speedup the experimental
procedure without degrading the performance
Introduction to BO
Constrained BO
Batch BO
Scheduling
24
Future Work
Main Idea
• We Use Monte Carlo
simulation to select a batch
of k experiments that
closely match what a good
sequential policy selection
in k steps
Given a sequential Policy and
batch size k
x21
x31
xn1
x12
x22
x32
xn2
x13
...
..
x23
...
..
x33
...
..
xn3
...
..
x1k
x2k
x3k
...
...
..
x11
xnk
Return B*={x1,x2,…,xk}
Introduction to BO
Constrained BO
Batch BO
Scheduling
25
Future Work
Objective Function(1)
• Simulated Matching:
– Having n different trajectories with length k from a given
sequential policy
– We want to select a batch of k experiments that best matches
the behavior of the sequential policy
• This objective can be viewed as minimizing an upper bound on the
expected performance difference between the sequential policy
and the selected batch.
• This objective is similar to weighted k-medoid
Introduction to BO
Constrained BO
Batch BO
Scheduling
26
Future Work
Supermodularity
• Example: Min(.) is a supermodual function
– B1={1, 2, 4}, B2={1, 2, 4, -2}, f=min(.) and x = 0
– f(B1) -f(B1, x)=1, f(B2)-f(B2, x)=0
• Quiz: What is the difference between submodular and
supermodular function?
– If the inequality is changed then we have submodular function
• The proposed objective function is a supermodular function
• The greedy algorithm provides us with an approximation bound
Introduction to BO
Constrained BO
Batch BO
Scheduling
27
Future Work
Algorithm
Introduction to BO
Constrained BO
Batch BO
Scheduling
28
Future Work
Results (5)
Greedy
Introduction to BO
Constrained BO
Batch BO
Scheduling
29
Future Work
Scheduling Methods for Bayesian
Optimization (NIPS 2011(spotlight))
30
Extended BO Model
Time Horizon h
Lab 1
x1
Lab 2
x2
x4
x5
x8
x3
Lab 3
xn-1
x7
x6
Lab l
Problem:
xn
We consider the following:
• Concurrent experiments
(up to l exp. at any time)
•
Stochastic exp. durations
(known distribution p)
•
Experiment budget
(total of n experiments)
•
Experimental time
horizon h
Stochastic Experiment
Durations
Schedule when to start new experiments and which ones to start
Introduction to BO
Constrained BO
Batch BO
Scheduling
31
Future Work
Challenges
Objective 2: maximize info. used in
selecting each experiments
(favors minimizing concurrency)
Lab 1
x1
x4
Lab 2
x2
x5
x3
x6
Lab 3
Lab 4
x4
x1
x2
⋯
xn
x7
We present online and offline approaches that effectively
trade off these two conflicting objectives
Introduction to BO
Constrained BO
Batch BO
Scheduling
32
Future Work
Objective Function
• Cumulative prior experiments (CPE) of E is measured as follows:
• Example: Suppose n1=1, n2=5, n3=5, n4=2, Then
CPE=(1*0)+(5*1)+(5*6)+(2*11)=57
• We found a non trivial correlation between CPE and regret
Introduction to BO
Constrained BO
Batch BO
Scheduling
33
Future Work
Offline Scheduling
• Assign start times to all n experiments before
the experimental process begins
• The experiment selection is done online
• Two class of schedules are presented
– Staged Schedules
– Independent Labs
Introduction to BO
Constrained BO
Batch BO
Scheduling
34
Future Work
Staged Schedules
• There are N stage and each stage is represent as <ni,di>
such that
– CPE is calculated as:
– We call an schedule uniform if |ni-nj|<2
n1=4
n2=3
n3=4
n4=3
x1
x2
x3
x4
x5
x6
x7
x8
x9
x10
x11
x12
x13
x14
d1
d2
h
d3
d4
• Goal: finding a p-safe uniform schedule with
maximum number of stages.
Introduction to BO
Constrained BO
Batch BO
Scheduling
35
Future Work
Staged Schedules: Schedule
Introduction to BO
Constrained BO
Batch BO
Scheduling
36
Future Work
Independent Lab (IL)
•
•
•
•
Assigns mi experiment to each lab i such that
Experiments are distributed uniformly within the labs
Start times of different labs are decoupled
The experiments in each lab have equal duration to maximize the
finishing probability within horizon h
• Mainly designed for policy switching schedule
Lab1
x11
x12
x13
x14
Lab2
x21
x22
x23
x24
Lab3
x31
x32
x33
Lab4
x41
x42
x43
h
Introduction to BO
Constrained BO
Batch BO
Scheduling
37
Future Work
Online Schedules
• p-safe guarantee is fairly pessimistic and we can
decrease the parallelization degree in practice
• Selects the start time of experiments online
rather than offline
• More flexible than offline schedule
Introduction to BO
Constrained BO
Batch BO
Scheduling
38
Future Work
Baseline online Algorithms
• Online Fastest Completion policy (OnFCP)
– Finish all of the n experiments as quickly as possible
– Keeps all l labs busy as long as there are experiments left
to run
– Achieves the lowest possible CPE
• Online Minimum Eager Lab Policy (OnMEL)
– OnFCP does not attempt to use the full time horizon
– use only k labs, where k is the minimum number of labs
required to finish n experiments with probability p
Introduction to BO
Constrained BO
Batch BO
Scheduling
39
Future Work
Policy Switching (PS)
•
PS decides about the number of new experiments at each decision step
•
Assume a set of policies or a policy generator is given
•
The goal is defining a new policy which performs as well as or better than
the best given policy at any state s
•
The i-th policy waits to finish i experiments and then call offIL algorithm
to reschedule
•
The policy which achieves the maximum CPE is returned
• The CPE of the switching policy will not be much worse than the best of
the policies produced by our generator
Introduction to BO
Constrained BO
Batch BO
Scheduling
40
Future Work
Experimental Results
Setting: h=4,5,6; pd=Truncated normal distribution, n=20 and L=10
Best Performance
Best CPE in each setting
Introduction to BO
Constrained BO
Batch BO
Scheduling
41
Future Work
Future Work
Introduction to BO
Constrained BO
Batch BO
Scheduling
42
Future Work
Traditional Approaches
•Sequential:
– Only one experiment is selected at each iteration
– Pros: Performance is optimized
– Cons: Can be very costly when running one experiment takes long time
• Batch:
–
k>1 experiments are selected at each iteration
– Pros: k times speed-up comparing to sequential approaches
– Cons: Can not performs as well as sequential algorithms
Introduction to BO
Constrained BO
Batch BO
Scheduling
43
Future Work
Batch Performance (Azimi et.al NIPS 2010)
k=5
k=10
Introduction to BO
Constrained BO
Batch BO
Scheduling
44
Future Work
Hybrid Batch
• Sometimes, the selected points by a given sequential
policy at a few consequent steps are independent
from each other
• Size of the batch can change at each time step
(Hybrid batch size)
Introduction to BO
Constrained BO
Batch BO
Scheduling
45
Future Work
First Idea
(NIPS Workshop 2011)
•
Based on a given prior (blue circles) and
an objective function (MEI), x1 is
selected
•
To select the next experiment, x2 , we
need, y1=f(x1) which is not available
•
The statistics of the samples inside the
red circle are expected to change after
observing at actual y1
•
We set y1 =M and then EI of the next
x2
x3
x1
step is upper bounded
•
If the next selected experiment is outside
of the red circle, we claim it is
independent from x1
Introduction to BO
Constrained BO
Batch BO
Scheduling
46
Future Work
Next
• Very pessimistic to set Y=M and then the speedup
is small
• Can we select the next point based on any
estimation without degrading the performance?
• What is the distance of selected experiments in
batch and the actual selected experiments by
sequential policy?
Introduction to BO
Constrained BO
Batch BO
Scheduling
47
Future Work
TimeLine
• Spring 2012: Finishing the Hybrid batch
approach
• Summer 2012: Finding a job and final defend
(hopefully  )
Introduction to BO
Constrained BO
Batch BO
Scheduling
48
Future Work
Publications
49
And
• I would like to thank Dr. Xaioli Fern and
Dr. Alan Fern
Introduction to BO
Constrained BO
Batch BO
Scheduling
50
Future Work
Introduction to BO
Constrained BO
Batch BO
Scheduling
51
Future Work
Results (1)
Random
Introduction to BO
Constrained BO
Batch BO
Scheduling
52
Future Work
Results (2)
Sequential
Introduction to BO
Constrained BO
Batch BO
Scheduling
53
Future Work
Results (3)
EMAX
Introduction to BO
Constrained BO
Batch BO
Scheduling
54
Future Work
Results (4)
K-means
Introduction to BO
Constrained BO
Batch BO
Scheduling
55
Future Work
Constrained BO: Results
Real
Fuel Cell
CMC-MUI
Random
Cosines
Rosenbrock
Introduction to BO
Constrained BO
Batch BO
Scheduling
Future Work
56
Constrained BO: Results
Real
Fuel Cell
CN-MEI
Cosines
Rosenbrock
Introduction to BO
Constrained BO
Batch BO
Scheduling
Future Work
57
Constrained BO: Results
Real
Fuel Cell
CMC-MPI(0.2)
Cosines
Rosenbrock
Introduction to BO
Constrained BO
Batch BO
Scheduling
Future Work
58
PS Performance Bound
•  ( s , t ) is our policy generator at each time step t and state s
• State s is the current running experiments with their starting time and
completed experiments.
•  ( s , t ,  ) denotes is the policy switching result whereis the base policy
selected in the last step
• The decision by is returned by N independent simulations.
•
C

t
(s)
is the CPE of policy  with  error
Introduction to BO
Constrained BO
Batch BO
Scheduling
59
Future Work