CS 332: Algorithms
Single-Source Shortest Path
S-S Shortest Path: Dijkstra’s Algorithm
Disjoint-Set Union
Amortized Analysis
David Luebke
1
7/1/2016
Single-Source Shortest Path
● Problem: given a weighted directed graph G,
find the minimum-weight path from a given
source vertex s to another vertex v
■ “Shortest-path” = minimum weight
■ Weight of path is sum of edges
■ E.g., a road map: what is the shortest path from
Chapel Hill to Charlottesville?
David Luebke
2
7/1/2016
Shortest Path Properties
● Again, we have optimal substructure: the
shortest path consists of shortest subpaths:
■ Proof: suppose some subpath is not a shortest path
○ There must then exist a shorter subpath
○ Could substitute the shorter subpath for a shorter path
○ But then overall path is not shortest path. Contradiction
David Luebke
3
7/1/2016
Shortest Path Properties
● Define (u,v) to be the weight of the shortest
path from u to v
● Shortest paths satisfy the triangle inequality:
(u,v)  (u,x) + (x,v)
● “Proof”:
x
u
v
This path is no longer than any other path
David Luebke
4
7/1/2016
Shortest Path Properties
● In graphs with negative weight cycles, some
shortest paths will not exist (Why?):
<0
David Luebke
5
7/1/2016
Relaxation
● A key technique in shortest path algorithms is
relaxation
■ Idea: for all v, maintain upper bound d[v] on (s,v)
Relax(u,v,w) {
if (d[v] > d[u]+w) then d[v]=d[u]+w;
}
5
2
9
5
2
Relax
5
David Luebke
2
6
Relax
7
5
6
2
6
7/1/2016
Bellman-Ford Algorithm
BellmanFord()
for each v  V
d[v] = ;
d[s] = 0;
for i=1 to |V|-1
for each edge (u,v)  E
Relax(u,v, w(u,v));
for each edge (u,v)  E
if (d[v] > d[u] + w(u,v))
return “no solution”;
Initialize d[], which
will converge to
shortest-path value 
Relaxation:
Make |V|-1 passes,
relaxing each edge
Test for solution
Under what condition
do we get a solution?
Relax(u,v,w): if (d[v] > d[u]+w) then d[v]=d[u]+w
David Luebke
7
7/1/2016
Bellman-Ford Algorithm
BellmanFord()
for each v  V
d[v] = ;
d[s] = 0;
for i=1 to |V|-1
for each edge (u,v)  E
Relax(u,v, w(u,v));
for each edge (u,v)  E
if (d[v] > d[u] + w(u,v))
return “no solution”;
What will be the
running time?
Relax(u,v,w): if (d[v] > d[u]+w) then d[v]=d[u]+w
David Luebke
8
7/1/2016
Bellman-Ford Algorithm
BellmanFord()
for each v  V
d[v] = ;
d[s] = 0;
for i=1 to |V|-1
for each edge (u,v)  E
Relax(u,v, w(u,v));
for each edge (u,v)  E
if (d[v] > d[u] + w(u,v))
return “no solution”;
What will be the
running time?
A: O(VE)
Relax(u,v,w): if (d[v] > d[u]+w) then d[v]=d[u]+w
David Luebke
9
7/1/2016
Bellman-Ford Algorithm
BellmanFord()
for each v  V
s
d[v] = ;
d[s] = 0;
for i=1 to |V|-1
for each edge (u,v)  E
Relax(u,v, w(u,v));
for each edge (u,v)  E
if (d[v] > d[u] + w(u,v))
return “no solution”;
B
-1
A
2
E
2
3
1
4
C
5
-3
D
Ex: work on board
Relax(u,v,w): if (d[v] > d[u]+w) then d[v]=d[u]+w
David Luebke
10
7/1/2016
Bellman-Ford
● Note that order in which edges are processed affects
how quickly it converges
● Correctness: show d[v] = (s,v) after |V|-1 passes
■ Lemma: d[v]  (s,v) always
○ Initially true
○ Let v be first vertex for which d[v] < (s,v)
○ Let u be the vertex that caused d[v] to change:
d[v] = d[u] + w(u,v)
○ Then d[v] < (s,v)
(s,v)  (s,u) + w(u,v) (Why?)
(s,u) + w(u,v)  d[u] + w(u,v) (Why?)
○ So d[v] < d[u] + w(u,v). Contradiction.
David Luebke
11
7/1/2016
Bellman-Ford
● Prove: after |V|-1 passes, all d values correct
■ Consider shortest path from s to v:
s  v1  v 2  v3  v4  v
○ Initially, d[s] = 0 is correct, and doesn’t change (Why?)
○ After 1 pass through edges, d[v1] is correct (Why?) and
doesn’t change
○ After 2 passes, d[v2] is correct and doesn’t change
○…
○ Terminates in |V| - 1 passes: (Why?)
○ What if it doesn’t?
David Luebke
12
7/1/2016
DAG Shortest Paths
● Problem: finding shortest paths in DAG
■ Bellman-Ford takes O(VE) time.
■ How can we do better?
■ Idea: use topological sort
○ If were lucky and processes vertices on each shortest
path from left to right, would be done in one pass
○ Every path in a dag is subsequence of topologically
sorted vertex order, so processing verts in that order, we
will do each path in forward order (will never relax
edges out of vert before doing all edges into vert).
○ Thus: just one pass. What will be the running time?
David Luebke
13
7/1/2016
Dijkstra’s Algorithm
● If no negative edge weights, we can beat BF
● Similar to breadth-first search
■ Grow a tree gradually, advancing from vertices
taken from a queue
● Also similar to Prim’s algorithm for MST
■ Use a priority queue keyed on d[v]
David Luebke
14
7/1/2016
Dijkstra’s Algorithm
Dijkstra(G)
B
2
10
for each v  V
4
3
A
D
d[v] = ;
d[s] = 0; S = ; Q = V;
5
1
C
while (Q  )
u = ExtractMin(Q); Ex: run the algorithm
S = S U {u};
for each v  u->Adj[]
if (d[v] > d[u]+w(u,v))
Relaxation
Note: this
d[v] = d[u]+w(u,v);
Step
is really a
call to Q->DecreaseKey()
David Luebke
15
7/1/2016
Dijkstra’s Algorithm
Dijkstra(G)
How many times is
for each v  V
ExtractMin() called?
d[v] = ;
d[s] = 0; S = ; Q = V;
How many times is
while (Q  )
u = ExtractMin(Q); DecraseKey() called?
S = S U {u};
for each v  u->Adj[]
if (d[v] > d[u]+w(u,v))
d[v] = d[u]+w(u,v);
What will be the total running time?
David Luebke
16
7/1/2016
Dijkstra’s Algorithm
Dijkstra(G)
How many times is
for each v  V
ExtractMin() called?
d[v] = ;
d[s] = 0; S = ; Q = V;
How many times is
while (Q  )
u = ExtractMin(Q); DecraseKey() called?
S = S U {u};
for each v  u->Adj[]
if (d[v] > d[u]+w(u,v))
d[v] = d[u]+w(u,v);
A: O(E lg V) using binary heap for Q
Can acheive O(V lg V + E) with Fibonacci heaps
David Luebke
17
7/1/2016
Dijkstra’s Algorithm
Dijkstra(G)
for each v  V
d[v] = ;
d[s] = 0; S = ; Q = V;
while (Q  )
u = ExtractMin(Q);
S = S U{u};
for each v  u->Adj[]
if (d[v] > d[u]+w(u,v))
d[v] = d[u]+w(u,v);
Correctness: we must show that when u is
removed from Q, it has already converged
David Luebke
18
7/1/2016
Correctness Of Dijkstra's Algorithm
p2
u
s
p2
x
y
Note that d[v]  (s,v) v
● Let u be first vertex picked s.t.  shorter path than d[u]
● Let y be first vertex V-S on actual shortest path from su
●
d[u] > (s,u)
 d[y] = (s,y)
Because d[x] is set correctly for y's predecessor x  S on the shortest path, and
■ When we put x into S, we relaxed (x,y), giving d[y] the correct value
■
David Luebke
19
7/1/2016
Correctness Of Dijkstra's Algorithm
p2
u
s
p2
x
y
Note that d[v]  (s,v) v
● Let u be first vertex picked s.t.  shorter path than d[u]
d[u] > (s,u)
● Let y be first vertex V-S on actual shortest path from su
 d[y] = (s,y)
● d[u] > (s,u)
= (s,y) + (y,u) (Why?)
= d[y] + (y,u)
 d[y]
But if d[u] > d[y],
wouldn't have chosen u. Contradiction.
David Luebke
20
7/1/2016
●
Disjoint-Set Union Problem
● Want a data structure to support disjoint sets
■ Collection of disjoint sets S = {Si}, Si ∩ Sj = 
● Need to support following operations:
■ MakeSet(x): S = S U {{x}}
■ Union(Si, Sj): S = S - {Si, Sj} U {Si U Sj}
■ FindSet(X): return Si  S such that x  Si
● Before discussing implementation details, we
look at example application: MSTs
David Luebke
21
7/1/2016
Kruskal’s Algorithm
Kruskal()
{
T = ;
for each v  V
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T U {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
22
7/1/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14
17
T = ;
8
25
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T U {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
23
9
1
7/1/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14
17
T = ;
8
25
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T U {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
24
9
1
7/1/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14
17
T = ;
8
25
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T U {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
25
9
1
7/1/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
9
14
17
T = ;
8
25
5
for each v  V
21
13
1?
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T U {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
26
7/1/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14
17
T = ;
8
25
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T U {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
27
9
1
7/1/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2?
19
{
14
17
T = ;
8
25
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T U {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
28
9
1
7/1/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14
17
T = ;
8
25
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T U {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
29
9
1
7/1/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
9
14
17
T = ;
8
25
5?
for each v  V
21
13
1
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T U {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
30
7/1/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14
17
T = ;
8
25
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T U {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
31
9
1
7/1/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14
17
T = ;
8?
25
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T U {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
32
9
1
7/1/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14
17
T = ;
8
25
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T U {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
33
9
1
7/1/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
9?
14
17
T = ;
8
25
5
for each v  V
21
13
1
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T U {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
34
7/1/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14
17
T = ;
8
25
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T U {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
35
9
1
7/1/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14
17
T = ;
8
25
5
for each v  V
21
13?
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T U {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
36
9
1
7/1/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14
17
T = ;
8
25
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T U {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
37
9
1
7/1/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14?
17
T = ;
8
25
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T U {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
38
9
1
7/1/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14
17
T = ;
8
25
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T U {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
39
9
1
7/1/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14
17?
T = ;
8
25
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T U {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
40
9
1
7/1/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19?
{
14
17
T = ;
8
25
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T U {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
41
9
1
7/1/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14
17
T = ;
8
25
5
for each v  V
21?
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T U {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
42
9
1
7/1/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14
17
T = ;
8
25?
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T U {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
43
9
1
7/1/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14
17
T = ;
8
25
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T U {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
44
9
1
7/1/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14
17
T = ;
8
25
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T U {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
45
9
1
7/1/2016
Correctness Of Kruskal’s Algorithm
● Sketch of a proof that this algorithm produces
an MST for T:
■ Assume algorithm is wrong: result is not an MST
■ Then algorithm adds a wrong edge at some point
■ If it adds a wrong edge, there must be a lower
weight edge (cut and paste argument)
■ But algorithm chooses lowest weight edge at each
step. Contradiction
● Again, important to be comfortable with cut
and paste arguments
David Luebke
46
7/1/2016
Kruskal’s Algorithm
What will affect the running time?
Kruskal()
{
T = ;
for each v  V
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T U {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
47
7/1/2016
Kruskal’s Algorithm
What will affect the running time?
Kruskal()
1 Sort
{
O(V) MakeSet() calls
T = ;
O(E) FindSet() calls
for each v  V
O(V) Union() calls
(Exactly how many Union()s?)
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T U {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
48
7/1/2016
Kruskal’s Algorithm: Running Time
● To summarize:
■ Sort edges: O(E lg E)
■ O(V) MakeSet()’s
■ O(E) FindSet()’s
■ O(V) Union()’s
● Upshot:
■ Best disjoint-set union algorithm makes above 3
operations take O(E(E,V)),  almost constant
■ Overall thus O(E lg E), almost linear w/o sorting
David Luebke
49
7/1/2016
Disjoint Set Union
● So how do we implement disjoint-set union?
■ Naïve implementation: use a linked list to
represent each set:
○ MakeSet(): ??? time
○ FindSet(): ??? time
○ Union(A,B): “copy” elements of A into B: ??? time
David Luebke
50
7/1/2016
Disjoint Set Union
● So how do we implement disjoint-set union?
■ Naïve implementation: use a linked list to
represent each set:
○ MakeSet(): O(1) time
○ FindSet(): O(1) time
○ Union(A,B): “copy” elements of A into B: O(A) time
■ How long can a single Union() take?
■ How long will n Union()’s take?
David Luebke
51
7/1/2016
Disjoint Set Union: Analysis
● Worst-case analysis: O(n2) time for n Union’s
Union(S1, S2)
Union(S2, S3)
…
Union(Sn-1, Sn)
“copy”
“copy”
1 element
2 elements
“copy”
n-1 elements
O(n2)
● Improvement: always copy smaller into larger
■ Why will this make things better?
■ What is the worst-case time of Union()?
● But now n Union’s take only O(n lg n) time!
David Luebke
52
7/1/2016
Amortized Analysis of Disjoint Sets
● Amortized analysis computes average times
without using probability
● With our new Union(), any individual element is
copied at most lg n times when forming the
complete set from 1-element sets
■ Worst case: Each time copied, element in smaller set
1st time
resulting set size
2
2nd time
4
…
(lg n)th time
n
David Luebke
53
7/1/2016
Amortized Analysis of Disjoint Sets
● Since we have n elements each copied at most
lg n times, n Union()’s takes O(n lg n) time
● We say that each Union() takes O(lg n)
amortized time
■ Financial term: imagine paying $(lg n) per Union
■ At first we are overpaying; initial Union $O(1)
■ But we accumulate enough $ in bank to pay for
later expensive O(n) operation.
■ Important: amount in bank never goes negative
David Luebke
54
7/1/2016
Amortized Analysis:
Accounting Method
● Accounting method
■ Charge each operation an amortized cost
■ Amount not used stored in “bank”
■ Later operations can used stored money
■ Balance must not go negative
● Book also discusses potential method
■ But we won’t worry about it here
David Luebke
55
7/1/2016
The End
David Luebke
56
7/1/2016