EECS 336: Introduction to Algorithms
Philosophy, Tractibility, Big-Oh
Reading: Chapters 2 & 3.
Announcements:
• Lecture notes on Canvas.
• Prerequisites:
• EECS 212: Discrete Math.
• EECS 214: Data Structures.
• Homework:
• work with lab partner (meet up after class)
• must communicate solution well.
• peer review (can you tell if a solution is good)
• automatic late policy for 25% of
grade.
Last Time:
• fibonacci numbers
Today:
• philosophy
• computational tractability
• runtime analysis & big-oh
• graphs & graph traversals
1
Lecture 2
Algorithms
Analysis
Design
and Computational
ity
Tractabil-
gives rigorous mathematical framework for “is a problem solvable by a computer?”
thinking about and solving problems in CS
Def: problem is tractable if worst-case runand other fields.
time to compute solution is polynomial in size
of input.
Goals
• quickly compute solutions to problems.
Question: What is “a problem”?
• understand the essence of problem.
Answer: worst cases instances of a given
size.
• identify general algorithm design and
Question: Other possibilities?
analysis approaches.
• every instance?
Three Steps
• typical instances?
• random instances?
1. problem modeling: abstract problem to
essential details.
Question: Benefits?
2. algorithm design
• usually doable.
3. algorithm analysis
• often tight for typical or random instances.
• efficiency,
• analyses “compose”
• correctness, and
• (sometimes) “quality”.
Def: T (n) = worst case runtime of instances
Note: design and analysis of good algoof size n.
rithms requires deep understanding of problem.
• size n measured in bits, or
• number of “components”.
Example: Fibonacci Numbers
fib(k) has n = log k bits.
n
• recursive: T (n) ≈ 22 .
2
Efficient vs. Brute-force
• dynamic program / iterative alg:
T (n) ≈ 2n .
• brute-force: “try all solutions, output
best one”
• repeated squaring: T (n) ≈ n.
Question:
puter”?
• often runtime of brute-force ≥ exponential time
What is “solvable by a com-
• efficient algorithms require exploiting
structure of problem.
Answer: T (n) = polynomial.
• want to solve “large” instances.
• want to scale well.
Main goals for algorithm design
i.e., T (cn) ≤ dT (n).
1. show problem is tractable
exists algorithm with polynomial runtime.
⇒ T (n) should be polynomial.
Example:
T (n) = nk
k
k
k
2. show problem is intractable
for all algorithms, runtime is superpolynomial.
k
T (cn) = (cn) = |{z}
c n = dn .
d
Question: Which is easier?
Answer: showing tractable.
3
Runtime Analysis
⇒ f + g = O(n).
QED
“bound T (n) for algorithm”
Note:
Big-Oh Notation
• be succinct: do not write O(n2 + n),
O(5n), etc.
Def: T (n) is O(f (n)) if ∃n0 , c > 0 such that
∀n > n0 , T (n) < cf (n).
• be tight: if T (n) is n2 do not say T (n) is
O(n3).
Question: why?
Answer:
• exact analysis is too detailed.
• constants vary from machine to machine.
Logarithms and Big-Oh
Example:
T (n) = an2 + bn + d
= O(n)? O(n2 )? O(n3 )?
T (n) ≤ an2 + bn2 + dn2
= (a + b + d) n2
| {z }
Def: logb n = ℓ ⇔ bℓ = n
• log10 n = number of digits to represent
n.
c
3
≤ cn
• log2 n = number of bits to represent n.
Fact 4: ∀b, c, logb n = O(logc n)
Fact 1: f = O(g)&g = O(h) ⇒ f = O(h).
Fact 2:
O(h).
Fact 5: ∀b, x, logb n = O(nx ).
f = O(h)&g = O(h) ⇒ f + g =
Proof: (of Fact 4)
Fact 3: g = O(f ) ⇒ g + f = O(f ).
logc n = ℓ ⇒ n = cℓ
Proof: (of Fact 2)
logb n = logb (cℓ )
= ℓ logb c
= logc n logb c
| {z }
f = O(h) ⇒ ∃c, n0 such that ∀n >
n0 , f (n) < ch(n)
d
g = O(h) ⇒ ∃c′ , n′0 such that ∀n >
n′0 , g(n) < c′ h(n)
= O(logc n)
⇒ ∀n > max(n0 , n′0 ), f (n) + g(n) ≤ (c′ +
c)h(n)
4
Common Runtimes
O(log n) – logarithmic
O(n) – linear
O(n log n)
O(n2 ) – quadratic
O(n3 ) – cubic
O(nk ) – polynomial
O(2n ) – exponential
O(n!)
Lower bounds
Def: T (n) is Ω(f (n)) if ∃n0 , c > 0 such that
∀n > n0 , T (n) > cf (n).
Exact bounds
Def: T (n) is Θ(f (n)) if T (n) is O(f (n)) and
Ω(f (n)).
5
Graphs
1
2
3
4
“encode pair-wise relationships”
Examples: computer networks, social networks, travel networks, dependencies.
BFS from 1: 1, 2, 3, 4 or 1, 3, 2, 4.
vertices
G = (V , E )
• Depth First Search (DFS).
edges
Example: DFS from 1: 1, 2, 4, 3 or 1,
3, 4, 2.
Example:
3
1
2
4
• V = {1, 2, 3, 4}
• E = {(1, 2), (2, 3), (2, 4), (3, 4)}
Concepts
• degree
• neighbors
• paths, path length
• distance
• connectivity, connected components
• directed graphs.
Graph Traversals
“visit all the vertices in a connected component of graph”
• Breadth First Search (BFS).
Example:
6