Optimization Problems
Greg Stitt
ECE Department
University of Florida
Introduction


Why are we studying optimization
problems?
Just about every RC-tool related issue
is an optimization problem

This lecture should provide abstract
solutions to many RC tool problems
Time Complexity

Time Complexity - Informal definition:


Defines how execution time increases as
input size increases
We are interested in worst case execution
time

Big O Notation



O(n)
 Execution time increases linearly with input size
O(n2)
 Quadratic growth
O(2n)
 Exponential growth
Problem Classes

P

A problem is “in P” if it has a polynomial time solution


Undecidable

Provably impossible to solve



Does not mean “Not-Polynomial”!!!!
A NP problem has a non-deterministic polynomial time solution
NP-hard

A problem is NP-hard if every problem in NP is reducible to it


Example: halting problem
NP


O(n), O(n2), O(n3), etc.
Basically means that NP-hard problems are at least as hard as the hardest
problems in NP
NP-complete


NP + NP-hard
Most interesting problems are NP-complete!!!!


Traveling salesman, Subset sum, 0-1 knapsack, graph coloring, vertex cover
Place and route, logic minimization, minimum resource scheduling, hw/sw
partitioning, etc.
The Big Question

Does P = NP?



Been studied for a long time
Never been proven either true or false
Why do we care?

Currently, best known solutions for NP-complete
problems typically have complexity of O(2n), O(n!)


If one NP-complete problem can be solved in polynomial
time (is in P), then all NP-complete problems can be
solved in polynomial time


Known as intractable – takes more than your lifetime to
solve
Remember, many interesting problems are NP-complete
If you can find a polynomial time solution to an NPcomplete RC problem:

I’ll give you an A
And, I’ll get a million dollars (www.claymath.org/millennium/P_vs_NP/ )
Problem Classes
Optimization Problems

Informal definition

Problem of finding the best solution from all possible solutions


Possible solution: exhaustive search



Check every possible solution, save best one
Works, but
Many optimization problems are NP-complete


Not feasible to generate all possible solutions
Example: O(2n)


Typically involves finding best solution for millions, billions, or even
more possibilities
n = 5 => 32, n = 10 => 1024, n = 100 => 1.3 * 1030
Problem: If most optimization problems are intractable, how do
we solve them?

Could use solution to any NP-complete problem!


But, this doesn’t really help


Remember: any NP-complete problem can be reduced to any other
NP-complete problem
There is no known polynomial solution to any NP-complete problem
So, what can we do?
Branch and Bound

Another solution:

Branch and bound


Idea: Try to eliminate many possibilities without evaluating them
How it works:


Imagine a tree representing all possible solutions
Algorithm progressively builds tree


When considering a branch in the tree, determines best possible
solution represented by branch



Maintains best found solution so far
If branch can’t possibly be better than current best, don’t both to
explore possible solutions
“Prunes” solution space
Result

+ Very often, can eliminate large percentage of possible solution



+ Usually, much faster than exhaustive search
- But, still has exponential complexity 


+ Still finds optimal solution
O(2n)
- Still not feasible for large input sizes
Heuristics

Another solution:


Forget about trying to find the best solution
Focus on finding a “good” solution quickly


Known as heuristics
Good candidates for heuristics

Map problem to other NP-complete problem, use heuristic
for that problem


Source of way too many publications
Use generalized optimization problem heuristic


Due to large number of interesting optimization problems,
research has introduced general heuristics that apply to all
optimization problems
Examples:



Hill Climbing
Simulated Annealing
Genetic Algorithms
Hill Climbing

Background: Graph of solution space


x axis = possible solutions
y axis = quality of solution



Informal description:



1) Choose a solution s (usually randomly)
2) Find neighboring solution n (i.e. change 1 thing)
3) If n better than s (climbs the hill)


s = n, Repeat from 2)
4) If all neighboring solutions worse than s


Height of solution represents goodness
Peak represents best solution
s is answer (s is a peak)
Example: Traveling Salesman



1) Find a solution
2) Swap 2 cities, if improves solution keep, if not reject
3) repeat 2) until no improvement can be found
Hill Climbing

Advantages:


Very fast, works well for certain problems
Disadvantages:

What if there are multiple peaks?



Hill climbing gets stuck at all peaks, known as
local maxima
Optimal solution is highest peak – global
maximum
May result in extremely suboptimal solution
if many peaks
Simulated Annealing

Problem: Hill climbing suffered from local peaks



Need way of escaping local peaks
=> Have to accept some bad moves
Solution: Simulated annealing


Annealing is process of heating and cooling metals in order to
improve strength
Idea: Controlled heating and cooling of metal


When hot, atoms move around
When cooled, atoms find configuration with lower internal energy


i.e. makes metal stronger
Analogy:

Temperature = probability of accepting worse neighboring solution


When temperature is high, likely to accept worse neighboring solutions (but
may lead to better overall solution)

Analogous to atoms wandering around
Cooling represents shrinking probability of accepting worse solutions
Simulated Annealing

Informal description




1) Set initial temperature and probability p
2) Find initial solution s
3) Find neighboring solution n
4) If n better than s


5) If n worse than s




s = n (always accept better solutions)
Accept n with probability p
6) Reduce “temperature” and p by some amount
7) If final temperature not reached, repeat from 3)
8) If final temperature reached, report best
solution found
Simulated Annealing

Advantages


Can find near optimal solutions – escapes local
maxima
Disadvantages

Takes a long time to find near optimal solution


But, still fast compared to algorithms
Very sensitive to input parameters – must be
configured well to get good results

How long to run?



Initial temperature/final temperature
What should initial probability be?
Cooling schedule

How much to reduce temperature and probability at each
step?
Genetic Algorithms

Another possible solution:


Imitate evolution – survival of the fittest
Assumption: evolution produces “better” humans/animals


Implies genetic processes must work well
Genetic Algorithms


Generates a “population” of solutions
Selection chooses members of population (solutions) to
survive by probability based on fitness function (i.e. natural
selection)


Reproduction combines attributes of selected population



Bad solutions less likely to “survive”
Crossover/Inheritence – Combines traits for each parent
(solution)
Mutation – Random change to characteristic
Repeat until solution has “evolved” to acceptable lavel
Genetic Algorithms

Advantages


Can find near optimal solutions
Disadvantages

Takes a long time to find near optimal solution


Very sensitive to input parameters – must be
configured well to get good results





But, still fast compared to algorithms
How large should population be?
How does reproduction occur?
How many generations should be considered?
How much of each generations should be killed off?
Same advantages/disadvantages as
simulated annealing
Other Heuristics




Ant colony optimization
Tabu search
Stochastic optimization
Many others
Summary


Optimizations problems search huge solution space for best
solution
Many optimization problems are NP-complete



Heuristics must be used


Find a “good” solution in reasonable time
General optimization problem heuristics




No known efficient solution
Branch and bound helps a little
Hill climbing – fast, but gets stuck at local maxima
Simulated annealing – works well but sensitive to parameters
Genetic algorithms - work well but sensitive to parameters
Why should you care about any of this?


Most RC tool problems are NP-complete optimization
problems
You now know how to solve almost all tool problems

You can use branch and bound, hill climbing, simulated
annealing, genetic algorithms, ant colony optimization, tabu
search, etc.