Algorithms
BITS, PILANI K. K. BIRLA GOA CAMPUS
Design & Analysis of Algorithms
(CS F364)
Lecture No. 1
Seen many algorithms
Sorting
- Insertion Sort
- Bubble Sort
- Merge Sort
- Quick Sort
- Heap Sort
- Radix Sort
- Counting Sort
Graph Searching
Graph Algorithms
- Breadth First Search
- Shortest Path
- Depth First Search
- Minimum Spanning Tree
- Tree Traversal
Searching
- Linear Search
- Binary Search
And Many More
1
Key Questions
Given a problem:
1st Question
Does Solution/Algorithm exist?
Do we know any such problem?
2nd Question
If solution exists, is there alternate better solution?
3rd Question
What is the least time required to solve the problem?
- lower bound results
4th Question
Does there exist algorithm solving the problem taking the least
time?
Key Questions
5th Question
Is the known solution polynomial time?
What about fractional knapsack?
What about 0-1 knapsack?
6th Question
If the known solution is not polynomial time, does/will
there exist a polynomial time solution?
7th Question
Can we prove that no polynomial time solution will
ever exist?
8th Question
answer to 7th Question is no, then what?
Course Objective 1:
Algorithm Design Techniques
We already know one popular strategy
Divide & Conquer
Consider the Coin Change Problem with coins of
denomination 1, 5, 10 & 25
Solution is Easy
What is the guarantee that solution works!
Introduce two more popular & widely applicable problem
solving strategies:
Dynamic Programming
Greedy Algorithms
BITS, PILANI K. K. BIRLA GOA CAMPUS
Design & Analysis of Algorithms
(CS F364)
Lecture No. 2
1
Re Cap - Key Questions
Given a problem:
1st Question
Does Solution/Algorithm exist?
Do we know any such problem?
2nd Question
If solution exists, is there alternate better solution?
3rd Question
What is the least time required to solve the problem?
- lower bound results
4th Question
Does there exist algorithm solving the problem taking the least
time?
Re Cap - Key Questions
5th Question
Is the known solution polynomial time?
What about fractional knapsack?
What about 0-1 knapsack?
6th Question
If the known solution is not polynomial time, does/will
there exist a polynomial time solution?
7th Question
Can we prove that no polynomial time solution will
ever exist?
8th Question
answer to 7th Question is no, then what?
Re Cap
Course Objective 1:
Algorithm Design Techniques
We already know one popular strategy
Divide & Conquer
Consider the Coin Change Problem with coins of
denomination 1, 5, 10 & 25
Solution is Easy
What is the guarantee that solution works!
Introduce two more popular & widely applicable problem
solving strategies:
Dynamic Programming
Greedy Algorithms
Course Objective - 3
How to deal with the class of problems for
which we strongly believe that no polynomial
time algorithm will exist?
This class consists of important practical
problems
Example: Traveling Salesman Problem, 0-1
Knapsack Problem, Bin Packing Problem
And Many more
Key Questions
Course Objective 2:
One of the objectives of this course is to look at
Question 5 to Question 8 in detail for a class of
problems
Understand famous P vs NP problem
We strongly believe that certain important class
of problems will not have polynomial time
solution.
Course Objective - 3
1st Alternative
Try to get polynomial time solution for an important
particular instance of the problem
2nd Alternative
Backtracking Algorithms
With good heuristics this works well for some important
particular instance of the problem
3rd Alternative
Approximation Algorithms
As name indicates, these algorithms will give approximate
solutions but will be polynomial in time.
Many Other Alternatives
Backtracking Algorithms
n - Queens Problem
State Space Tree
Backtracking Algorithm
Subset Sum Problem
Instance S = (3, 5, 6, 7) and d = 15
The state-space tree can be constructed as a binary tree.
The complete recursion tree
for the 4 queens problem.
Course Objective - 3
Course Objective 3
For a certain class of problems to study, develop
Binary Search Tree
Binary Search Tree
Well known, important data structure for efficient
search
If T is Binary tree with n nodes, then
min height of the tree is
logn
max height of the tree is
n-1
Disadvantage
Tree can be skewed making search inefficient
Binary Search Tree
How to fix the skewness
Balanced trees
- AVL Trees
- Red-Black Trees
- Multi-way Search Trees, (2, 4) Trees
- Few More
Technique used for balancing
- Rotations
Left Rotation or Right Rotation
Binary Search Tree
Disadvantage with Balanced Trees
Number of rotations needed to maintain balanced
structure is
O(logn)
Question
Are there type of binary search trees for which
number of rotations needed for balancing is
constant
(independent of n)
Rotations
Rotations
Note : The rotation does not violate any of the properties of binary search trees.
Course Objective - 4
Course Objective 4
Study type of binary search trees called Treap for
which expected number of rotations needed for
balancing is constant
We will be proving that expected number of
rotations needed for balancing in Treap is 2
(something really strong)
Treap is an Randomized Data Structure
Algorithm Design Strategy - Review
Divide & Conquer
- binary search
- merge sort
- quick sort
- Matrix Multiplication (Strassen Algorithm)
Many More
Many standard iterative algorithms can be written
as Divide & Conquer Algorithms.
Example: Sum/Max of n numbers
Question: Any advantage in doing so?
Divide & Conquer
How to analyze Divide & Conquer Algorithms
Generally, using Recurrence Relations
Let T(n) be the number of operations required to
solve the problem of size n.
Then T(n) = a T(n/b) + c(n) where
- each sub-problem is of size n/b
- There are a such sub-problems
- c(n) extra operations are required to combine the
solutions of sub-problems into a solution of the
original problem
Divide & Conquer
Divide-and-conquer algorithms:
1. Dividing the problem into smaller subproblems
(independent sub-problems)
2. Solving those sub-problems
3. Combining the solutions for those smaller
sub-problems to solve the original problem
Divide & Conquer
How to solve Recurrence Relations
- Substitution Method
- works in simple relations
- Masters Theorem
- See Text Book for details
Text Book
Text Book
Thomas H. Cormen, Charles E. Leiserson,
Ronald L. Rivest & Clifford Stein
Introduction to Algorithms
Third Edition, PHI Learning Private Limited, 2015
BITS, PILANI K. K. BIRLA GOA CAMPUS
Design & Analysis of Algorithms
(CS F364)
Lecture No. 3
1
Red Black Trees
Red Black Trees
Binary search tree with an additional attribute for
its nodes: color which can be red or black
Constrains the way nodes can be colored on any
path from the root to a leaf
red-black trees ensure that no path is more than
twice as long as any other
Ensures tree is approximately balanced.
Red Black Trees
Convention
Each node of the tree now contains the fields color,
key, left, right, and p.
For convenience we use a sentinel NIL[T] to
represent all the NIL nodes at the leafs
NIL[T] has the same fields as an ordinary node
Color[NIL[T]] = BLACK
Red Black Tree
Height of a node
Red Black Tree
A binary search tree is a red-black tree if it satisfies
the following red-black properties:
1. Every node is either red or black.
2. The root of the tree is always black.
3. Every leaf (NIL) is black.
4. If a node is red, then both its children are black.
Definitions
Height of a node: the number of edges in the longest path to
a leaf
Black-height of a node x: bh(x) is the number of black nodes
(including NIL) on the path from x to a leaf, not counting x
26
h=1
bh = 1
17
NIL
41
NIL
No two consecutive red nodes on a simple path from the root to a leaf
5. Every simple path from a node to a descendant
leaf contains the same number of black nodes.
30
h=2
bh = 1
NIL
NIL
Important property of Red-Black-Trees
Lemma
A red-black tree with n internal nodes has height at
most 2log(n + 1).
Need to prove two claims first
Claim 1:
Any node x with height h(x) has bh
Proof:
By property 4, at most h/2 red nodes on the path
from the node to a leaf
Hence at least h/2 are black
h=4
bh = 2
h=3
bh = 2
h=1
bh = 1
38
NIL
47
NIL
h=2
bh = 1
50
NIL
h=1
bh = 1
NIL
Claim 2
Claim 2:
The subtree rooted at any node x contains at least
2bh(x) - 1 internal nodes
Proof:
We prove this claim by induction on the height of x.
Base case:
h[x] = 0
x is a leaf (NIL[T])
bh(x) = 0
Number of internal nodes: 20 - 1 = 0
Inductive Hypothesis: assume it is true for h[x]=h-1
Claim 2 Continued
Inductive step:
Prove it for h[x]=h
Let bh(x) = b, then any child y of x has
bh(y) = b (if the child is red or),
bh(y) = b-1 (if the child is black),
Using inductive hypothesis, the number of internal
nodes for each child of x is at least: 2bh(x) - 1 1
The subtree rooted at x contains at least:
(2bh(x) - 1
1) + (2bh(x) - 1
1) + 1 =
2 · (2bh(x) - 1 - 1) + 1 = 2bh(x) - 1 internal nodes
Operations on Red-Black-Trees
The non-modifying binary-search-tree operations
MINIMUM, MAXIMUM, SUCCESSOR, PREDECESSOR,
and SEARCH run in O(h) time
They take O(log n) time on red-black trees
Question
What about TREE-INSERT and TREE-DELETE?
Height of Red-Black-Trees
Lemma: A red-black tree with n internal nodes
has height at most 2lg(n + 1).
Proof:
Let height(root) = h & bh(root) = b
Then n 2b 1 2h/2 - 1
Add 1 to both sides and then take logs:
b
lg
h/2
h
lg(n + 1)
Rotations
Operations for re-structuring the tree after insert and
delete operations on red-black trees
Rotations take a red-black-tree and a node within
the tree and:
Together with some node re-coloring they help
restore the red-black-tree property
Change some of the pointer structure
Do not change the binary-search tree property
Two types of rotations:
Left & right rotations
Rotations
Rotations
Left Rotate - Example
Left Rotate
Assumptions for a left
rotation on a node x:
The right child of x (y) is not
NIL
Idea:
Pivots around the link from x to y
Makes y the new root of the
subtree
x becomes y
child
y left child becomes x
child
RB Tree - Insert
Aim:
Insert a new node z into a red-black-tree
Idea:
Insert node z into the tree as for an ordinary binary
search tree
Color the node red
Restore the red-black-tree properties
Use an auxiliary procedure RB-INSERT-FIXUP
RB Properties Affected by Insert
Every node is either red or black OK
z is the root
The root is black If not
OK
Every leaf (NIL) is black OK
If a node is red, then both its children are
black If p(z) is red not OK
z and p(z) are both red
For each node, all paths from the node to
descendant leaves contain the same number
of black nodes OK
RB-INSERT-FIXUP Case 3
RB-INSERT-FIXUP Case 1
Case 1:
Case 3:
z
black
z is a left child
red
Idea: (z is a right child)
p[p[z]]
Idea
Color p[z] Black
Color p[p[z]] Red
Right Rotate (T, p[p[z]])
z and p[z] are both red
No longer have 2 reds in a row
p[z] is now black
Color p[z] Black
Color y Black
Case 3
Color p[p[z]] Red
z = p[p[z]]
Push the
violation up the tree
RB-INSERT-FIXUP(T, z)
RB-INSERT-FIXUP Case 2
Case 2:
z
y) is black
z is a right child
Idea:
z p[z]
Left Rotate (T, z)
now z is a left child, and both z and p[z] are red
1. while color[p[z]] = RED
2.
3.
4.
5.
do if p[z] = left[p[p[z]]]
then y
right[p[p[z]]]
if color[y] = RED
then Case1
6.
else if z = right[p[z]]
then Case2
7.
Case3
8.
9.
Case2 is converted to Case 3
The while loop repeats only when
case1 is executed: O(lgn) times
else (same as then
10. color[root[T]]
BLACK
We just inserted the root, or
The red violation reached the root
Example
Insert 4
Case 1
11
2
14
z
15
7
1
5
5
8 y
z and p[z] are both red
4
14 y
2
7
1
Case 2
11
4
z
15
8
z and p[z] are both red
BITS, PILANI K. K. BIRLA GOA CAMPUS
Design & Analysis of Algorithms
(CS F364)
z is a right child
11
14 y
7
z
8
2
5
1
4
15
z and p[z] are red
z is a left child
7
z
2
Case 3
1
Lecture No. 4
11
5
8
14
4
15
20
Re Cap - Red Black Tree
Red Black Tree
A binary search tree is a red-black tree if it satisfies
the following red-black properties:
1. Every node is either red or black.
2. The root of the tree is always black.
3. Every leaf (NIL) is black.
4. If a node is red, then both its children are black.
No two consecutive red nodes on a simple path from the root to a leaf
5. Every simple path from a node to a descendant
leaf contains the same number of black nodes.
1
Re Cap - Important property of Red-Black-Trees
Lemma
A red-black tree with n internal nodes has height at
most 2log(n + 1).
Need to prove two claims first
Claim 1:
Any node x with height h(x) has bh
2
Claim 2:
The subtree rooted at any node x contains at least
2bh(x) - 1 internal nodes
Re Cap- Rotations
Rotations
Re Cap - RB Tree - Insert
Aim:
Insert a new node z into a red-black-tree
Idea:
Idea:
Pivots around the link from x to y
Makes y the new root of the
subtree
x becomes y
child
y left child becomes x
child
Assumptions for a left
rotation on a node x:
The right child of x (y) is not
NIL
RB-INSERT-FIXUP(T, z)
1. while color[p[z]] = RED
2.
3.
4.
5.
do if p[z] = left[p[p[z]]]
then y
right[p[p[z]]]
if color[y] = RED
then Case1
6.
else if z = right[p[z]]
then Case2
7.
Case3
8.
9.
The while loop repeats only when
case1 is executed: O(lgn) times
else (same as then
10. color[root[T]]
BLACK
We just inserted the root, or
The red violation reached the root
Insert node z into the tree as for an ordinary binary
search tree
Color the node red
Restore the red-black-tree properties
Use an auxiliary procedure RB-INSERT-FIXUP
Analysis of RB-INSERT
Inserting the new element into the tree O(logn)
RB-INSERT-FIXUP
The while loop repeats only if CASE 1 is executed
The number of times the while loop can be executed is
O(logn)
Total running time of RB-INSERT: O(logn)
RB Tree Delete
We first apply standard BST-Delete
Rules for BST deletion
1. If vertex to be deleted is a leaf, just delete it.
2. If vertex to be deleted has just one child,
replace it with that child
3. If vertex to be deleted has two children,
replace the value
-order
-order
predecessor
Red-Black Trees - Summary
Operations on red-black-trees:
SEARCH
PREDECESSOR
SUCCESOR
MINIMUM
MAXIMUM
INSERT
DELETE
O(h)
O(h)
O(h)
O(h)
O(h)
O(h)
O(h)
Red-black-trees guarantee that the height of the
tree will be O(logn)
RB Tree Delete
What can go wrong?
If the delete node is red?
Not a problem no RB properties violated
If the deleted node is black?
Two adjacent red vertices.
Black height property might be violated.
For Detailed RB Tree Delete Fix Up
See CLRS
Multi-Way Search Tree
A multi-way search tree is an ordered tree such that
Each internal node has at least two children and stores d 1
key-element items (ki, oi), where d is the number of children
For a node with children v1 v2
vd storing keys k1 k2 kd 1
where k1 2
kd-1
keys in the subtree of v1 are less than k1
keys in the subtree of vi are between ki 1 and ki
(i
d 1)
keys in the subtree of vd are greater than kd 1
The leaves store no items and serve as placeholders
Multi-Way Inorder Traversal
We can extend the notion of inorder traversal from binary
trees to multi-way search trees
Namely, we visit item (ki, oi) of node v between the
recursive traversals of the subtrees of v rooted at
children vi and vi 1
An inorder traversal of a multi-way search tree visits the
keys in increasing order
(2,4) Trees
A (2,4) tree (also called 2-4 tree or 2-3-4 tree) is a multi-way
search with the following properties
Node-Size Property: every internal node has at most four
children
Depth Property: all the external nodes have the same
depth
Depending on the number of children, an internal node of a
(2,4) tree is called a 2-node, 3-node or 4-node
Example
Multi-Way Searching
Similar to search in a binary search tree
A each internal node with children v1 v2 vd and keys k1 k2
kd 1
k ki (i
d 1): the search terminates successfully
k k1: we continue the search in child v1
ki 1 k ki (i
d 1): we continue the search in
child vi
k kd 1: we continue the search in child vd
Reaching an external node terminates the search
unsuccessfully
Example: Search for 30
Height of a (2,4) Tree
Theorem: A (2,4) tree storing n items has height O(log n)
Proof:
Upper bound:
The tallest possible tree for a fixed n is when all internal nodes
are 2-nodes, each storing 1 item; i.e., the tree is equivalent to
binary tree.
By the depth property, it is a complete binary tree.
Let h be the height of a (2,4) tree with n items
Since there are at least 2i items at depth i
h 1 and no
items at depth h,
we have n 1 2 4
2h 1 2h 1
Thus, h log (n 1)
h is of O(log n)
Height of a (2,4) Tree
Lower bound:
The shortest possible tree for a fixed n is when all
internal nodes are 4-nodes, each storing 3 items (by
the size property).
By the depth property, all levels are filled out.
Let h be the height of a (2,4) tree with n items,
We have n 1 4 16
4h 1 4h 1
log4(n +1) h
h is of (log n)
Therefore h is of (log n)
2-4 Tree Insertion
We insert a new item (k, o) at the parent v of the leaf reached by
searching for k and adding an empty leaf node below.
We preserve the depth property
But We may cause an overflow (i.e., node v may become a 5-node)
Example: Inserting key 30
First we find its position between 27 and 32.
However inserting here causes overflow.
Overflow and Split
Overflow and Split
We handle Overflow with a split operation
let v1 v5 be the children of v and k1
k4 be the keys of
v
node v is split into two nodes v' and v"
v' is a 3-node with keys k1 k2 and children v1 v2 v3
v" is a 2-node with key k4 and children v4 v5
key k3 is inserted into the parent u of v (a new root may
be created)
The overflow may propagate to the parent node u
A new root may be created if the root overflows.
Example
Analysis of Insertion
Algorithm insert(k, o)
1. We search for key k to locate the
insertion node v
2. We add the new entry (k, o) at node v
3. while overflow(v)
if isRoot(v)
create a new empty root above v
v split(v)
Let T be a (2,4) tree with n
items
Tree T has O(log n)
height
Step 1 takes O(log n)
time because we visit
O(log n) nodes
Step 2 takes O(1) time
Step 3 takes O(log n)
time because each split
takes O(1) time and we
at most perform O(log n)
splits
2-4 Tree Deletion
If the entry to be deleted is in a node that has internal nodes as
children,
We replace the entry to be deleted with its inorder successor
and delete the latter entry.
Example: to delete key 24, we replace it with 27 (inorder
successor):
This reduces deletion of an
entry to the case where the
item is at the node with leaf
children.
Thus, an insertion in a (2,4)
tree takes O(log n) time
Underflow and Fusion
Deleting an entry from a node v may cause an underflow,
where node v becomes a 1-node with one child and no keys
To handle an underflow at node v with parent u, we
consider two cases
Case 1: An adjacent sibling of v is a 2-node.
Perform a fusion operation, merging v with the adjacent 2node sibling w and moving a key from u to the merged
node v'.
Underflow and Transfer
Case 2: An adjacent sibling w of v is a 3-node or a 4-node.
Perform a transfer operation:
1. Move a child of w to v
2. Move an item from u to v
3. Move an item from w to u
Note: With Transfer Operation the underflow is eliminated.
Which keys are moved depends on the configuration. In this example we move
the largest key out of each node, but it could be different if underflow was in a
non-rightmost child or it was a right sibling that was a 3-node or 4-node.)
Analysis of Deletion
Let T be a (2,4) tree with n items
Tree T has O(log n) height
In a deletion operation
We visit O(log n) nodes to locate the node from which
to delete the entry
We handle an underflow with a series of O(log n)
fusions, followed by at most one transfer
Each fusion and transfer takes O(1) time
Thus, deleting an item from a (2,4) tree takes
O(log n) time
BITS, PILANI K. K. BIRLA GOA CAMPUS
Design & Analysis of Algorithms
(CS F364)
Lecture No. 5
1
Multi-Way Search Tree
A multi-way search tree is an ordered tree such that
Each internal node has at least two children and stores d 1
key-element items (ki, oi), where d is the number of children
For a node with children v1 v2
vd storing keys k1 k2 kd 1
where k1 2
kd-1
keys in the subtree of v1 are less than k1
keys in the subtree of vi are between ki 1 and ki
(i
d 1)
keys in the subtree of vd are greater than kd 1
The leaves store no items and serve as placeholders
Multi-Way Searching
Similar to search in a binary search tree
A each internal node with children v1 v2 vd and keys k1 k2
kd 1
k ki (i
d 1): the search terminates successfully
k k1: we continue the search in child v1
ki 1 k ki (i
d 1): we continue the search in
child vi
k kd 1: we continue the search in child vd
Reaching an external node terminates the search
unsuccessfully
Example: Search for 30
2-4 Tree Insertion
(2,4) Trees
A (2,4) tree (also called 2-4 tree or 2-3-4 tree) is a multi-way
search with the following properties
Node-Size Property: every internal node has at most four
children
Depth Property: all the external nodes have the same
depth
Depending on the number of children, an internal node of a
(2,4) tree is called a 2-node, 3-node or 4-node
Example
We insert a new item (k, o) at the parent v of the leaf reached by
searching for k and adding an empty leaf node below.
We preserve the depth property
But We may cause an overflow (i.e., node v may become a 5-node)
Example: Inserting key 30
First we find its position between 27 and 32.
However inserting here causes overflow.
Theorem: A (2,4) tree storing n items has height O(log n)
Overflow and Split
Overflow and Split
We handle Overflow with a split operation
let v1 v5 be the children of v and k1
k4 be the keys of
v
node v is split into two nodes v' and v"
v' is a 3-node with keys k1 k2 and children v1 v2 v3
v" is a 2-node with key k4 and children v4 v5
key k3 is inserted into the parent u of v (a new root may
be created)
The overflow may propagate to the parent node u
A new root may be created if the root overflows.
Example
2-4 Tree Deletion
If the entry to be deleted is in a node that has internal nodes as
children,
We replace the entry to be deleted with its inorder successor
and delete the latter entry.
Example: to delete key 24, we replace it with 27 (inorder
successor):
This reduces deletion of an
entry to the case where the
item is at the node with leaf
children.
Underflow and Transfer
Case 2: An adjacent sibling w of v is a 3-node or a 4-node.
Perform a transfer operation:
1. Move a child of w to v
2. Move an item from u to v
3. Move an item from w to u
Note: With Transfer Operation the underflow is eliminated.
Which keys are moved depends on the configuration. In this example we move
the largest key out of each node, but it could be different if underflow was in a
non-rightmost child or it was a right sibling that was a 3-node or 4-node.)
Underflow and Fusion
Deleting an entry from a node v may cause an underflow,
where node v becomes a 1-node with one child and no keys
To handle an underflow at node v with parent u, we
consider two cases
Case 1: An adjacent sibling or both immediate sibling of v is
a 2-node.
Perform a fusion operation, merging v with the adjacent 2node sibling w and moving a key from u to the merged
node v'.
Red-Black trees into (2, 4) trees
Mapping Red-Black trees into (2, 4) trees
Given a red-black tree, we can construct a corresponding
(2,4) tree by merging every red node y into its parent and
storing the item from y at its parent.
(2, 4) trees into Red-Black trees
Mapping (2, 4) trees into Red-Black trees
2 - Node
3 - Node
4 - Node
Randomly built binary search trees
Randomly built binary search trees
As with quicksort, we can show that the behavior of
the average case is much closer to the best case
than to the worst case.
Unfortunately, little is known about the average
height of a binary search tree when both insertion
and deletion are used to create it.
When the tree is created by insertion alone, the
analysis becomes more tractable
Randomly built binary search tree on n keys is the
one that arises from inserting the keys in random
order into an initially empty tree, where each of the
n! permutations of the input keys is equally likely.
Convex Functions
Some useful definitions and results
Convex Functions
Geometrical Interpretation
If X is a random variable and f is a convex function,
then
says
Height of a randomly built binary search tree
Theorem
The expected height of a randomly built binary search
tree on n distinct keys is O(logn)
Proof
We define three random variables that help measure
the height of a randomly built binary search tree.
Xn : height of a randomly built binary search on n
keys
: exponential height
Random BST- Height Contd.
When we build a binary search tree on n keys, we
choose one key as that of the root
Rn : Denote the random variable that holds root
rank within the set of n keys
Rn holds the position that this key (root) would occupy
if the set of keys were sorted
The value of Rn is equally likely to be any element of the
Random BST- Height Contd.
Random BST- Height Contd.
If Rn = i,
then the left subtree of the root is a randomly built
binary search tree on i - 1 keys,
and the right subtree is a randomly built binary search
tree on n - i keys
Height of a binary tree is 1 more than the larger of the
heights of the two subtrees of the root
Therefore, the exponential height of a binary tree is
twice the larger of the exponential heights of the two
subtrees of the root
If we know that Rn = i,
It follows that Yn = 2max (Yi-1, Yn-i)
We have Y1 = 1,
Because the exponential height of a tree with 1 node is
20 = 1
And we define Y0 = 0
Next, we define indicator random variables Zn, 1, Zn, 2,
n, n
where Zn, i = I {Rn = i}
Random BST- Height Contd.
Random BST- Height Contd.
Now Rn is equally likely to be any element of {1, 2,..., n}
So, we have that Pr{Rn = i} = 1/n for i = 1, 2,..., n
Therefore, E[Zn, i] = 1/n for i = 1, 2,..., n
Since exactly one value of Zn, i is 1 and all others are 0,
we have
Observe
Having chosen Rn = i, the left subtree (whose
exponential height is Yi-1) is randomly built on the
i-1 keys whose ranks are less than i.
This subtree is just like any other randomly built
binary search tree on i-1 keys.
Other than the number of keys it contains, this
choice of Rn = i
Hence the random variables Yi-1 and Zn, i are
independent.
Likewise, random variables Yn i and Zn, i are
independent.
Claim:
E [Yn] is polynomial in n
Therefore, E [Xn] = O(log n)
Random BST- Height Contd.
Random BST- Height Contd.
Using the substitution method, we shall show that for
all positive integers n, the recurrence (*) has the
solution
Hence we have
Proof is by induction (see CLRS)
Observe, f(x) = 2x is a convex function
Therefore,
inequality
Therefore
Since each term E[Y ], E[Y
1
n-1
(*) appears twice in the0 last summation
]
(**)
Random BST- Height Contd.
Solving (**) and taking log on both sides,
we get E[Xn] = O(logn)
BITS, PILANI K. K. BIRLA GOA CAMPUS
Design & Analysis of Algorithms
(CS F364)
Lecture No. 6
1
Re Cap - Red-Black trees into (2, 4) trees
Mapping Red-Black trees into (2, 4) trees
Given a red-black tree, we can construct a corresponding
(2,4) tree by merging every red node y into its parent and
storing the item from y at its parent.
Re Cap -(2, 4) trees into Red-Black trees
Mapping (2, 4) trees into Red-Black trees
2 - Node
3 - Node
4 - Node
Re Cap - Randomly built binary search trees
Randomly built binary search trees
As with quicksort, we can show that the behavior of
the average case is much closer to the best case
than to the worst case.
Unfortunately, little is known about the average
height of a binary search tree when both insertion
and deletion are used to create it.
When the tree is created by insertion alone, the
analysis becomes more tractable
Randomly built binary search tree on n keys is the
one that arises from inserting the keys in random
order into an initially empty tree, where each of the
n! permutations of the input keys is equally likely.
Treap
Our new data structure is a binary search tree called a
Treap, which is a combination of the words "Tree" and
"Heap"
Every node of Treap maintains two values.
Key Follows standard BST ordering (left is smaller
and right is greater)
Priority Randomly assigned value that follows MaxHeap property.
Re Cap - Height of a randomly built binary search tree
Theorem
The expected height of a randomly built binary search
tree on n distinct keys is O(logn)
Treap
Example: S = {(k1, p1), (k2, p2
kn, pn)}
S = {(2, 13), (4, 26), (6,19), (7, 30), (9,14), (11, 27), (12, 22)}
Treap - Structure
Theorem:
Let S = {(k1, p1), (k2, p2
kn, pn)} be any set of keypriority pairs such that the keys and the priorities are
distinct. Then, there exists a unique treap T(S) for it.
Proof:
Proof by induction
Theorem is true for n = 0 and for n = 1.
Suppose now that n 2, and assume that {(k1, p1),
has the highest priority in S.
Then, a treap for S can be constructed by putting
item 1 at the root of T(S).
Shape of Treap
Observe
The shape of the tree underlying the treap is
determined by the relative priorities of the key
values
Any particular shape can be obtained by
choosing the priorities suitably.
To solve the data structuring problem, we
must somehow pick a good set of priorities for
the items being stored
Proof Contd.
A treap for the items in S of key value smaller
(larger) than k1 can be constructed recursively,
and this is stored as the left (right) sub-tree of
item 1.
Any treap for S must have this decomposition at
the root.
Proof is completed by the inductive hypothesis.
Treap
Insert (k, T)
Insert like BST
Fix the heap property (using left/right rotation)
without violating BST property
Rotations
INSERT - Example
Insert (3, 26) in
Treap - Delete
Delete (k, T)
Exactly the reverse of an insertion :
Rotate the node containing k downward until both its
children are leaves, and then simply discard the node.
Note
The choice of the rotation (left/right) at each stage
depends on the relative order of the priorities of the
children of the node being deleted.
The priority of an item is chosen at random when the
item is first inserted into a set, and the priority for
this item remains fixed until it is deleted. Moreover,
if the item is re-inserted after a deletion, a
completely new random priority is assigned to it.
Probabilistic Games
For proving the main result we introduce probabilistic
games called Mulmuley Games.
Mulmuley games are useful abstractions of processes
underlying the behavior of certain geometric algorithms.
The cast of characters in these games is:
A set P = {P1, P2
p} of players
A set S = {S1, S2
Ss} of stoppers
A set T = {T1, T2
t} of triggers
A set B = {B1, B2
b} of bystanders
The set P S is drawn from a totally ordered universe.
All players are smaller than all stoppers:
for all i and j, Pi Sj
Game A
Game A
Initial set of characters X = P B
The game proceeds by repeatedly sampling from X
without replacement, until the set X becomes
empty.
Random variable V: the number of samples in
which a player Pi is chosen such that Pi is larger
than all previously chosen players.
Value of the game Ap = E[V]
Value of Game A
Lemma:
Here
For all
Ap = Hp
Hk lnk
Proof:
Assume that the set of players is ordered as P1>P2 > ..>Pp
Note:
Bystanders are irrelevant to the game : the value of the
game is not influenced by the number of bystanders. Thus,
we can assume that the initial number of bystanders b = 0.
If the 1st chosen player is Pi the expected value of the game is
1 + Ai-1
This is because the players Pi+1
p cannot contribute
to the game any more and are effectively reduced to being
bystanders.
Value of Game A Contd.
Note :
i is uniformly distributed over the set {1, . . . , p}
Therefore, we obtain the following recurrence.
Upon rearrangement, using the fact that A0 = 0,
Can be shown that Harmonic numbers are the solution to
the above equation.
Analysis of Treaps
Memory less property
Since the random priorities for the elements of S are
chosen independently, we can assume that the priorities
are chosen before the insertion process is initiated
Once the priorities have been fixed, we know that the
treap T is uniquely determined.
This implies that the order in which the elements are
inserted does not affect the structure of the tree.
Without loss of generality, we can assume that the
elements of set S are inserted into T in the order of
decreasing priority.
An advantage of this view is that it implies that all
insertions take place at the leaves and no rotations are
required to ensure the heap order on the priorities.
BITS, PILANI K. K. BIRLA GOA CAMPUS
Design & Analysis of Algorithms
(CS F364)
Lecture No. 7
1
Treap
Our new data structure is a binary search tree called a
Treap, which is a combination of the words "Tree" and
"Heap"
Every node of Treap maintains two values.
Key Follows standard BST ordering (left is smaller
and right is greater)
Priority Randomly assigned value that follows MaxHeap property.
Treap
Insert (k, T)
Insert like BST
Fix the heap property (using left/right rotation)
without violating BST property
Delete (k, T)
Exactly the reverse of an insertion :
Rotate the node containing k downward until both its
children are leaves, and then simply discard the node.
Treap
Example: S = {(2, 13),(4, 26), (6,19),(7, 30), (9,14),(11, 27),(12, 22)}
Theorem:
Let S = {(k1, p1), (k2, p2
kn, pn)} be any set of key-priority pairs
such that the keys and the priorities are distinct. Then, there exists a
unique treap T(S) for it.
Note: Any particular shape can be obtained by choosing the priorities
suitably.
Probabilistic Games
For proving the main result we introduce probabilistic
games called Mulmuley Games.
Mulmuley games are useful abstractions of processes
underlying the behavior of certain geometric algorithms.
The cast of characters in these games is:
A set P = {P1, P2
p} of players
A set S = {S1, S2
Ss} of stoppers
A set T = {T1, T2
t} of triggers
A set B = {B1, B2
b} of bystanders
The set P S is drawn from a totally ordered universe.
All players are smaller than all stoppers:
for all i and j, Pi Sj
Game A
Analysis of Treaps
Game A
Initial set of characters X = P B
The game proceeds by repeatedly sampling from X
without replacement, until the set X becomes empty.
Random variable V: the number of samples in which a
player Pi is chosen such that Pi is larger than all
previously chosen players.
Value of the game Ap = E[V]
Lemma:
For all
Ap = H p
Memory less property
Since the random priorities for the elements of S are
chosen independently, we can assume that the priorities
are chosen before the insertion process is initiated
Once the priorities have been fixed, we know that the
treap T is uniquely determined.
This implies that the order in which the elements are
inserted does not affect the structure of the tree.
Without loss of generality, we can assume that the
elements of set S are inserted into T in the order of
decreasing priority.
An advantage of this view is that it implies that all
insertions take place at the leaves and no rotations are
required to ensure the heap order on the priorities.
Analysis of Treaps
Analysis of Treaps
Lemma:
Let T be a random treap for a set S of size n.
For an element x S having rank k,
E[depth(x)] = Hk + Hn-k+1 1
Proof:
Define the sets S- = {y S
and S+ = { y S
Since x has rank k, it follows that |S-| = k
and |S+| = n k +1
Denote by Qx S the ancestors of x
Let
and
Claim 1:
E[ ] = Hk
By symmetry, it follows that E[ ] = Hn-k+1
Consider any ancestor y
of the node x.
By the memoryless assumption, y must have been
inserted prior to x and py > px .
Since y < x, it must be the case that x lies in the right
sub-tree of y (*)
Claim 2:
An element y S- is an ancestor of x, or a member of
if and only if it was the largest element of S- in the
treap at the time of its insertion. (Like Game A of Mulmuley Games)
Proof of Claim 2
Proof of Claim 2:
Suppose y was the largest element of S- in the treap at
the time of its insertion.
That means x was inserted later and since y < x, y is an
ancestor of x.
Conversely, suppose y S- is an ancestor of x
Suppose y was not the largest element of S- in the
treap at the time of its insertion.
Then there exists z S- such that y < z < x and z was
inserted before y.
In this case y when inserted will be in left subtree of z
whereas x will be in right subtree of z contradicting (*).
Left/Right Spine
Some Definitions
Left spine of a tree:
The path obtained by starting at the root and repeatedly
moving to the left child until a leaf is reached.
Right spine is defined similarly.
Analysis of Treaps
Therefore, distribution of | | is the same as
that of the value of Game A when P = S- and
B = S\SSince |S-| = k, the expected size of | | is Hk
Left/Right Spine
Define
Rx the right spine of the left subtree of x.
Lx the left spine of the right subtree of x.
Lemma:
The number of rotations needed during update
operations is Rx + Lx
Left/Right Spine
Observe:
Let y be the parent of x, and suppose x is a left child of y.
After performing a right rotation, y becomes the right child of x,
and the previous right child of x becomes the left child of y.
Note that the left spine of the right subtree of x before the
rotation is tacked on to y, so the length of that spine
increases by one.
The left subtree of x is unaffected by a right rotation.
The case of a left rotation is symmetric.
Analysis of Treaps
Lemma:
Let T be a random treap for a set S of size n.
For an element x S of rank k,
and
Proof:
As before let S- = {y
and S+ = { y
Game D
Game D
Game D is similar to Game A, But
In Game D, X = P B T
The role of the triggers is that the counting process
begins only after the first trigger has been chosen.
i.e. a player contributes to V only if it is sampled
after a trigger (and of course it is larger than all
previously chosen players).
Lemma:
We will assume the lemma
Analysis of Treaps
Claim 1:
Distribution of Rx is the same as that of the value of
Game D with the choice of characters
P = S- \ {x}, T = {x} and B = S+ \ {x}
Since we have p = k - 1, t = 1, and b = n - k,
It remains to relate the length of Rx to Game D
Claim 2:
An element z < x lies on the right spine of the left subtree of x if and only if z is inserted after x, and all
elements whose values lie between z and x are inserted
after z
Analysis of Treaps
Analysis of Treaps
Proof of Claim 2:
Suppose z < x lies on the right spine of the left sub-tree
of x.
Then it must have been inserted after x is of value
smaller than x.
Suppose y such that z < y < x and y was inserted
before z.
In such a case z
on the right spine of the left
sub-tree of x.
Therefore, all elements whose values lie between z and
x are inserted after z
Conversely, suppose z < x, z is inserted after x, and all
elements whose values lie between z and x are inserted
after z
To prove z < x lies on the right spine of the left sub-tree of x
Since z < x and inserted after x, it must lie in the left subtree of x.
Suppose z dose not lie on the right spine of the left subtree of x.
Then y such that z < y < x and y was inserted before z.
A contradiction.
Therefore, z lies on the right spine of the left sub-tree of
x.
Analysis of Treaps
Treaps
Similarly one can prove that
Main Theorem:
Let T be a random treap for a set S of size n.
1. The expected time for a FIND, INSERT, or DELETE
operation on T is O(log n).
2. The expected number of rotations required during
an INSERT or DELETE operation is at most 2.
Reference :
Rajeev Motwani & Prabhakar Raghavan
Randomized Algorithms
Cambridge University Press, 2000
For Alternate Proof:
CLRS Third Edition
Exercise 13-4 (a to j)
Re Cap - Analysis of Treaps
BITS, PILANI K. K. BIRLA GOA CAMPUS
Design & Analysis of Algorithms
(CS F364)
Lecture No. 8
Lemma:
Let T be a random treap for a set S of size n.
For an element x S having rank k,
E[depth(x)] = Hk + Hn-k+1 1
Lemma:
Let T be a random treap for a set S of size n.
For an element x S of rank k,
and
1
Re Cap - Analysis of Treaps
Main Theorem:
Let T be a random treap for a set S of size n.
1. The expected time for a FIND, INSERT, or DELETE
operation on T is O(log n).
2. The expected number of rotations required during
an INSERT or DELETE operation is at most 2.
Algorithm Design Strategies
Divide & Conquer
- binary search
- merge sort
- quick sort
- Matrix Multiplication (Strassen Algorithm)
Many More
Many standard iterative algorithms can be written
as Divide & Conquer Algorithms.
Example: Sum/Max of n numbers
Question: Any advantage in doing so?
Divide & Conquer
Divide-and-conquer algorithms:
1. Dividing the problem into smaller subproblems
(independent sub-problems)
2. Solving those sub-problems
3. Combining the solutions for those smaller
sub-problems to solve the original problem
Divide & Conquer
How to solve Recurrence Relations
- Substitution Method
- works in simple relations
- Masters Theorem
- See Text Book for details
Divide & Conquer
How to analyze Divide & Conquer Algorithms
Generally, using Recurrence Relations
Let T(n) be the number of operations required to
solve the problem of size n.
Then T(n) = a T(n/b) + c(n) where
- each sub-problem is of size n/b
- There are a such sub-problems
- c(n) extra operations are required to combine the
solutions of sub-problems into a solution of the
original problem
Dynamic Programming
Dynamic Programming is a general algorithm
design technique for solving problems defined by or
formulated as recurrences with overlapping sub
instances
i.e, Applicable when sub problems are not
independent
i.e., Sub problems share subsubproblems
Invented by American mathematician Richard Bellman
in the 1950s to solve optimization problems
Dynamic Programming
Main idea:
- set up a recurrence relating a solution to a
larger instance to solutions of some smaller
instances
- solve smaller instances once
- record solutions in a table
- extract solution to the initial instance from
that table
Example: Fibonacci numbers
Fibonacci numbers:
F(n) = F(n-1) + F(n-2)
F(0) = 0 & F(1) = 1
Computing the nth Fibonacci number recursively
(top-down):
F(n)
F(n-1)
F(n-2)
+
+
F(n-3)
...
F(n-2)
F(n-3)
+
F(n-4)
Subproblems share subsubproblems
Computing binomial coefficient
Recurrence:
C(n, k) = C(n-1, k) + C(n-1, k-1) for n > k > 0
C(n, 0) = 1, C(n, n) = 1 for n 0
Computing binomial coefficient
Value of C(n,k) can be computed by filling a table:
0 0 1 2 . . . k-1
k
0 1
1 1 1
.
.
.
n-1
C(n-1,k-1) C(n-1,k)
n
C(n,k)
Time Complexity
O(nk)
Coin Change Problem
Coin Change Problem
Given a value N, if we want to make change for
N paisa, and we have infinite supply of each of C
= { 1, 5, 10, 25} valued coins, what is the
minimum number of coins to make the change?
Solution Easy (We all know this)
Greedy Solution
Coin Change Problem
Suppose we want to compute the minimum
number of coins with values
& where coin of denomination i has value d[i]
Let c[i][j] be minimum number of coins required
to pay an amount of j units 0<=j<=N using only
coins of denomination s 1 to i, 1<=i<=n
C[n][N] is the solution to the problem
Coin Change Problem
Coin Change Problem Recurrence
In calculating c[i][j], notice that:
Suppose we do not use the coin with value
d[i] in the solution of the (i,j)-problem,
then c[i][j] = c[i-1][j]
Suppose we use the coin with value d[i] in the
solution of the (i,j)-problem,
then c[i][j] = 1 + c[i][ j-d[i]]
Since we want to minimize the number of coins,
we choose whichever is the better alternative
Therefore
c[i][j] = min{c[i-1][j], 1 + c[i][ j-d[i]]}
&
c[i][0] = 0 for every i
Alternative 1
Recursive algorithm
When a recursive algorithm revisits the same problem over and
over again, we say that the optimization problem has overlapping
subproblems.
How to observe/prove that problem has overlapping
subproblems.
Answer Draw Computation tree and observe
Overlapping Subproblems
Computation Tree
Coin Change Problem
Example
We have to pay 8 units with coins worth 1,4 & 6 units
For example c[2][6] is obtained in this case as the smaller of
c[1][6] and 1+ c[2][6-d[2]] = 1+c[2][2]
The other entries of the table are obtained similarly
Dynamic-programming algorithms typically take advantage of overlapping
subproblems by solving each subproblem once and then storing the solution in a
table where it can be looked up when needed, using constant time per lookup.
Analysis
Time Complexity
We have to compute n(N+1) entries
Each entry takes constant time to compute
Running time O(nN)
0
1
2
3
4
5
6
7
8
d[1]=1
0
1
2
3
4
5
6
7
8
d[2]=4
0
1
2
3
1
2
3
4
2
d[3]=6
0
1
2
3
1
2
1
2
2
Important Observation
The table gives the solution to our problem for all the instances
involving a payment of 8 units or less
Question
How can you modify the algorithm to actually
compute the change (i.e., the multiplicities of
the coins)?
Re Cap - Dynamic Programming
BITS, PILANI K. K. BIRLA GOA CAMPUS
Design & Analysis of Algorithms
(CS F364)
Lecture No. 9
Main idea:
- set up a recurrence relating a solution to a
larger instance to solutions of some smaller
instances
- solve smaller instances once
- record solutions in a table
- extract solution to the initial instance from
that table
1
Re Cap - Coin Change Problem
Suppose we want to compute the minimum
number of coins with values
& where coin of denomination i has value d[i]
Let c[i][j] be minimum number of coins required
to pay an amount of j units 0<=j<=N using only
coins of denomination s 1 to i, 1<=i<=n
C[n][N] is the solution to the problem
Re Cap - Coin Change Problem
In calculating c[i][j], notice that:
Suppose we do not use the coin with value d[i] in the
solution of the (i,j)-problem,
then c[i][j] = c[i-1][j]
Suppose we use the coin with value d[i] in the
solution of the (i,j)-problem,
then c[i][j] = 1 + c[i][ j-d[i]]
Therefore
c[i][j] = min{c[i-1][j], 1 + c[i][ j-d[i]]}
&
c[i][0] = 0 for every i
Re Cap - Coin Change Problem
When a recursive algorithm revisits the same problem over and
over again, we say that the optimization problem has overlapping
subproblems.
Re cap - Coin Change Problem
Example
We have to pay 8 units with coins worth 1,4 & 6 units
For example c[2][6] is obtained in this case as the smaller of
c[1][6] and 1+ c[2][6-d[2]] = 1+c[2][2]
The other entries of the table are obtained similarly
0
1
2
3
4
5
6
7
8
d[1]=1
0
1
2
3
4
5
6
7
8
d[2]=4
0
1
2
3
1
2
3
4
2
d[3]=6
0
1
2
3
1
2
1
2
2
Important Observation
The table gives the solution to our problem for all the instances
involving a payment of 8 units or less
Question
How can you modify the algorithm to actually
compute the change (i.e., the multiplicities of
the coins)?
Optimal Substructure Property
Why does the solution work?
Optimal Substructure Property/ Principle of
Optimality
The optimal solution to the original problem
incorporates optimal solutions to the subproblems.
In an optimal sequence of decisions or choices, each
subsequence must also be optimal
This is a hallmark of problems amenable to dynamic
programming.
Not all problems have this property.
Optimal Substructure Property
In our example though we are interested only
in c[n][N], we took it granted that all the other
entries in the table must also represent
optimal choices.
If c[i][j] is the optimal way of making change
for j units using coins of denominations 1 to I,
then c[i-1][j] & c[i][j-d[i]] must also give the
optimal solutions to the instances they
represent
Dynamic Programming Algorithm
The dynamic-programming algorithm can be broken
into a sequence of four steps.
1. Characterize the structure of an optimal solution.
Optimal Substructure Property
2. Recursively define the value of an optimal solution.
3. Compute the value of an optimal solution in a
bottom-up fashion.
Overlapping subproblems
4. Construct an optimal solution from computed
information.
(not always necessary)
Optimal Substructure Property
How to prove Optimal Substructure Property?
Generally by Cut-Paste Argument or By
Contradiction
Note
Optimal Substructure Property looks obvious
But it does not apply to every problem.
Exercise:
Give an problem which does not exhibit Optimal
Substructure Property.
Matrix Chain Multiplication
Recalling Matrix Multiplication
If A is p x q matrix and B is q x r matrix
then the product C = AB is a p x r matrix given by
where
OR
Dot product of ith row of A with jth column of B
Properties
Properties of Matrix Multiplication
If A is p x q matrix and B is q x r matrix then the
product C = AB is a p x r matrix
If AB is defined, BA may not be defined
except for square matrices
i.e. Matrix Multiplication is not commutative
Each element of the product requires q multiplications,
And there are pr elements in the product
Therefore, Multiplying an p×q and a q×r matrix requires
pqr multiplications
Example
Let us use the following example:
Let A be a 2x10 matrix
Let B be a 10x50 matrix
Let C be a 50x20 matrix
Consider computing A(BC):
Total multiplications = 10000 + 400 = 10400
Consider computing (AB)C:
Total multiplications = 1000 + 2000 = 3000
Substantial difference in the cost for computing
Properties
Matrix multiplication is associative
i.e., A1(A2A3) = (A1A2 )A3
So parenthesization does not change result
It may appear that the amount of work done
parenthesization of the expression
But that is not the case!
Matrix Chain Multiplication
Thus, our goal today is:
Given a chain of matrices to multiply,
determine the fewest number of multiplications
necessary to compute the product.
Let dixdi+1 denote the dimensions of matrix Ai.
Let A = A0 A1 ... An-1
Let Ni,j denote the minimal number of
multiplications necessary to find the product: Ai Ai+1
... Aj.
To determine the minimal number of multiplications
necessary N0,n-1 to find A,
That is, determine how to parenthisize the
multiplications
Optimal Substructure Property
BITS, PILANI K. K. BIRLA GOA CAMPUS
Design & Analysis of Algorithms
Optimal Substructure Property/ Principle of
Optimality
The optimal solution to the original problem
incorporates optimal solutions to the subproblems.
(CS F364)
This is a hallmark of problems amenable to dynamic
programming.
Not all problems have this property.
Lecture No. 10
1
Optimal Substructure Property
How to prove Optimal Substructure Property?
Generally by Cut-Paste Argument or By
Contradiction
Note
Optimal Substructure Property looks obvious
But it does not apply to every problem.
Exercise:
Give an problem which does not exhibit Optimal
Substructure Property.
Dynamic Programming Algorithm
The dynamic-programming algorithm can be broken
into a sequence of four steps.
1. Characterize the structure of an optimal solution.
Optimal Substructure Property
2. Recursively define the value of an optimal solution.
3. Compute the value of an optimal solution in a
bottom-up fashion.
Overlapping subproblems
4. Construct an optimal solution from computed
information.
(not always necessary)
Matrix Chain Multiplication
Properties
Recalling Matrix Multiplication
If A is p x q matrix and B is q x r matrix
then the product C = AB is a p x r matrix given by
Matrix multiplication is associative
i.e., A1(A2A3) = (A1A2 )A3
So parenthesization does not change result
It may appear that the amount of work done
where
OR
Dot product of ith row of A with jth column of B
parenthesization of the expression
But that is not the case!
Example
Let us use the following example:
Let A be a 2x10 matrix
Let B be a 10x50 matrix
Let C be a 50x20 matrix
Consider computing A(BC):
Total multiplications = 10000 + 400 = 10400
Consider computing (AB)C:
Total multiplications = 1000 + 2000 = 3000
Substantial difference in the cost for computing
Matrix Chain Multiplication
Thus, our goal today is:
Given a chain of matrices to multiply,
determine the fewest number of multiplications
necessary to compute the product.
Let dixdi+1 denote the dimensions of matrix Ai.
Let A = A0 A1 ... An-1
Let Ni,j denote the minimal number of
multiplications necessary to find the product: Ai Ai+1
... Aj.
To determine the minimal number of multiplications
necessary N0,n-1 to find A,
That is, determine how to parenthisize the
multiplications
Matrix Chain Multiplication
1st Approach
Brute Force
Given the matrices A1,A2,A3,A4 Assume the
dimensions of A1=d0×d1 etc
Five possible parenthesizations of these arrays, along
with the number of multiplications:
(A1A2)(A3A4):d0d1d2+d2d3d4+d0d2d4
((A1A2)A3)A4:d0d1d2+d0d2d3+d0d3d4
(A1(A2A3))A4:d1d2d3+d0d1d3+d0d3d4
A1((A2A3)A4):d1d2d3+d1d3d4+d0d1d4
A1(A2(A3A4)):d2d3d4+d1d2d4+d0d1d4
Matrix Chain Multiplication
Solution to the recurrence is the famous Catalan
Numbers
n/3n/2)
Question : Any better approach?
Matrix Chain Multiplication
Questions?
How many possible parenthesization?
At least lower bound?
n)
The number of parenthesizations is atleast
Exercise: Prove
The exact number is given by the recurrence relation
Because, the original product can be split into two parts
In (n-1) places.
Each split is to be parenthesized optimally
Matrix Chain Multiplication
Step1:
Optimal Substructure Property
If a particular parenthesization of the whole product is
optimal,
then any sub-parenthesization in that product is optimal as
well.
What does it mean?
Yes
Dynamic Programming
If (A (B ((CD) (EF)) ) ) is optimal
Then (B ((CD) (EF)) ) is optimal as well
How to Prove?
Matrix Chain Multiplication
Cut - Paste Argument
Because if it wasn't,
and say ( ((BC) (DE)) F) was better,
then it would also follow that
(A ( ((BC) (DE)) F) ) was better than
(A (B ((CD) (EF)) ) ),
contradicting its optimality!
Matrix Chain Multiplication
Thus,
M[i,j]=M[i,k]+M[k+1,j]+ di-1dkdj
Thus the problem of determining the optimal
sequence of multiplications is broken down to the
following question?
How do we decide where to split the chain?
OR (what is k)?
Answer:
Search all possible values of k & take the minimum
of it.
Matrix Chain Multiplication
Step 2:
Recursive Formulation
Let M[i,j] represent the minimum number of
multiplications required for matrix product Ai × ×
Aj, For
j<n
High-Level Parenthesization for Ai..j
Notation: Ai..j = Ai
Aj
For any optimal multiplication sequence,
at the last step we are multiplying two matrices
Ai..k and Ak+1..j for some k, i.e.,
Ai..j = (Ai
Ak) (Ak+1
Aj) = Ai..k Ak+1..j
Matrix Chain Multiplication
Therefore,
Step3:
Compute the value of an optimal solution in a
bottom-up fashion
Overlapping Subproblem
Matrix Chain Multiplication
Which sub-problems are necessary to solve first?
By Definition M[i,i] = 0
Clearly it's necessary to solve the smaller problems
before the larger ones.
In particular, we need to know M[i, i+1], the number of
multiplications to multiply any adjacent pair of
matrices before we move onto larger tasks.
Chains of length 1
The next task we want to solve is finding all the values
of the form M[i, i+2], then M[i, i+3], etc.
Chains of length 2 & then chains of length 3 & so on
Matrix Chain Multiplication
That is, we calculate in the order
Matrix Chain Multiplication
This tells us the order in which to build the
table:
By diagonals
Diagonal indices:
On diagonal 0, j=i
On diagonal 1, j=i+1
On diagonal q, j=i+q
On diagonal n , j=i+n
Matrix Chain Multiplication
Example
Array dimensions:
A1: 2 x 3 , A2: 3 x 5 , A3: 5 x 2
A4: 2 x 4 , A5: 4 x 3
Matrix Chain Multiplication
Optimal locations for parentheses:
Idea: Maintain an array
where s[i, j] denotes k
for the optimal splitting in computing Ai..j = Ai..k Ak+1..j
Array
n] can be used recursively to compute
the multiplication sequence
Matrix Chain Multiplication
Table for M[i, j]
Matrix Chain Multiplication
Table for s[i, j]
The multiplication sequence is recovered
as follows.
s[1, 5] = 3
(A1 A2 A3 ) ( A4 A5)
s[1, 3] = 1
(A1 (A2 A3 ) )
Hence the final multiplication sequence is
(A1 (A2 A3 ) ) ( A4 A5)
Matrix Chain Multiplication
Pseudo code
Observations
Question 1
How many subproblems are used in an optimal solution
for the original problem?
Coin Change Problem
Two Which?
Matrix Chain Multiplication
Two Which?
Question 2
How many choices we have in determining which
subproblems to use in an optimal solution?
Coin Change Problem
Two Why?
Matrix Chain Multiplication
j - i choices for k (splitting the product)
DP Time Complexity
Intuitively, the running time of a dynamic
programming algorithm depends on two factors:
1. Number of subproblems overall
2. How many choices we have for each subproblem
BITS, PILANI K. K. BIRLA GOA CAMPUS
Design & Analysis of Algorithms
(CS F364)
Matrix multiplication:
O(n2) subproblems
At most n-1 choices
Therefore, Time complexity is O(n3) overall
Lecture No. 11
1
Observations
Question 1
How many subproblems are used in an optimal solution
for the original problem?
Coin Change Problem
Two Which?
Matrix Chain Multiplication
Two Which?
Question 2
How many choices we have in determining which
subproblems to use in an optimal solution?
Coin Change Problem
Two Why?
Matrix Chain Multiplication
DP Time Complexity
Intuitively, the running time of a dynamic
programming algorithm depends on two factors:
1. Number of subproblems overall
2. How many choices we have for each subproblem
Matrix multiplication:
O(n2) subproblems
At most n-1 choices
Therefore, Time complexity is O(n3) overall
j - i choices for k (splitting the product)
Longest Common Subsequence (LCS)
Given two sequences
X = x1, x2
xm
Y = y1, y2
yn
find a maximum length common subsequence
(LCS) of X and Y
Example
X = A, B, C, B, D, A, B
Subsequences of X:
A subset of elements in the sequence taken in order
For example, A, B, D , B, C, D, B , etc.
Example
Example
X = A, B, C, B, D, A, B
Y = B, D, C, A, B, A
B, C, B, A is a longest common subsequence of
X and Y (length = 4)
B, D, A, B is also a longest common
subsequence of X and Y (length = 4)
B, D, A , however is not a LCS of X and Y
Making the choice
Brute Force Solution
Let length of X be m & length of Y be n
Brute Force
For every subsequence of X,
check
Y
Question? How many subsequences are there of X?
There are 2m subsequences of X
Question?
What is time required to check for each subsequence?
Each subsequence takes O(n) time to check
Scan Y for first letter, then scan for second, & so on
Therefore, Running time: O(n2m) Exponential
Notations
Given a sequence X = x1, x2
The i-th prefix of X
Xi = x1, x2 , xi
X = A, B, Z, D
Y = Z, B, D
Choice: include one element into the common
sequence (D) and solve the resulting subproblem
X = A, B, E, Z
Y = Z, B, E
Choice: exclude an element from a string and solve
the resulting subproblem
Recursive Solution
xm
m is
c[i, j] = the length of a LCS of the sequences
Xi = x1, x2
yj
i and Yj = y1, y2
Case 1: xi = yj (Last characters match)
Example
Xi = <D, B, Z,, E>
Yj = <Z,, B, E>
c[i, j] = c[i - 1, j - 1] + 1
Append xi = yj to the LCS of Xi-1 and Yj-1
Must find a LCS of Xi-1 and Yj-1
Recursive Solution
Case 2: xi
yj (Last characters do not match)
In this case xi and yj cannot both be in the LCS
Thus either xi is not part of the LCS, or yj is not part of the LCS
(and possibly both are not part of the LCS).
Recursive Solution
0
if i = 0 or j = 0
c[i, j] = c[i-1, j-1] + 1
if xi = yj
max(c[i, j-1], c[i-1, j]) if xi yj
In the 1st case LCS of Xi and Yj is the LCS of Xi-1 and Yj
In the 2nd case LCS of Xi and Yj is the LCS of Xi and Yj -1
Example
Xi = A, B, Z, G & Yj = A, D, Z
1.
LCS of Xi-1 = A, B, Z and Yj = A, D, Z
2.
LCS of Xi = A, B, Z, G and Yj-1 = A, D
Therefore, c[i, j] =max { c[i - 1, j], c[i, j-1] }
Optimal Substructure
Overlapping Subproblems
Optimal Substructure Property
Easy to prove that in both the cases
Optimal solution to a problem includes optimal
solutions to subproblems
Cut-Paste Argument
To find a LCS of X and Y
We may need to find the LCS between
X and Yn-1 and that of Xm-1 and Y
Both the above subproblems has the subproblem
of finding the following:
LCS of Xm-1 and Yn-1
Subproblems share subsubproblems
Computing the Length of the LCS
c[i, j] =
0
c[i-1, j-1] + 1
max(c[i, j-1], c[i-1, j])
0
yj:
1
y1
2
y2
0 xi:
0
0
0
1 x1
0
0
2 x2
Computing the table
if i = 0 or j = 0
if xi = yj
if xi yj
c[i, j] =
n
yn
0
0
0
first
If xi = yj
b[i
second
Else, if c[i b[i
c[m, n]
-1]
Else
b[i
Pseudo Code for LCS
1. for i
1 to m
2.
do c[i, 0]
0
3. for j
0 to n
4.
do c[0, j]
0
5. for i
1 to m
6.
do for j
1 to n
7.
do if xi = yj
8.
then c[i, j]
c[i - 1, j - 1] + 1
9.
b[i, j ]
10.
else if c[i - 1]
11.
then c[i, j]
c[i - 1, j]
12.
b[i, j]
13.
else c[i, j]
c[i, j - 1]
14.
b[i, j]
Running time: O(mn)
15.return c and b
if i = 0 or j = 0
if xi = yj
if xi yj
Along with c[i, j] we also compute and record b[i, j] which
tells us what choice was made to obtain the optimal value
0
0
0
m xm
0
c[i-1, j-1] + 1
max(c[i, j-1], c[i-1, j])
Example
X = A, B, C, B, D, A, B
Y = B, D, C, A, B, A
If xi = yj
Else if c[i else
0
if i = 0 or j = 0
c[i, j] = c[i-1, j-1] + 1
if xi = yj
max(c[i, j-1], c[i-1, j]) if xi yj
0
yj
1
B
2
D
3
C
4
A
0 xi
-1] 1 A
0
0
0
0
0
0
0
2 B
0
1
1
1 1
3 C
0
1
2
4 B
0
5 D
0
6 A
0
7 B
0
1
1
1
0
2
5
B
0
1
2
0
2
2
2
1
2
2
3
1
2
2
3
0
1
1
2
2
2
2
1
6
A
2
3
3
3
3
3
4
4
4
Constructing a LCS
The 0-1 Knapsack Problem
Given: A set S of n items, with each item i having
Start at b[m, n] and
follow the arrows
yj
D
B
C
A
0 xi
1 A
0
0
0
0
0
0
0
b[i, j]
xi = yj is an
element of the LCS
2 B
0
1
1
1 1
3 C
0
1
2
4 B
0
LCS is BCBA
5 D
0
6 A
0
7 B
0
When we
1
1
1
0
2
B
0
1
2
0
2
2
2
1
2
2
3
1
2
2
3
0
wi - a positive
vi - a positive
1
1
Goal:
2
2
Choose items with maximum total benefit but with weight
at most W.
And we are not allowed to take fractional amounts
In this case, we let T denote the set of items we take
2
2
1
A
2
3
3
3
3
3
4
4
4
Objective : maximize
Constraint :
Greedy approach
Possible greedy approach:
Approach1: Pick item with largest value first
Approach2: Pick item with least weight first
Approach3: Pick item with largest value per
weight first
We know that none of the above approaches work
Exercise:
Prove by giving counterexamples.
Brute Force Approach
Brute Force
The naive way to solve this problem is to go
through all 2n subsets of the n items and pick
the subset with a legal weight that maximizes
the value of the knapsack.
The 0-1 Knapsack Problem
BITS, PILANI K. K. BIRLA GOA CAMPUS
Given: A set S of n items, with each item i having
Design & Analysis of Algorithms
(CS F364)
wi - a positive
vi - a positive
Goal:
Choose items with maximum total benefit but with weight
at most W.
And we are not allowed to take fractional amounts
In this case, we let T denote the set of items we take
Lecture No. 12
Objective : maximize
1
Greedy approach
Possible greedy approach:
Approach1: Pick item with largest value first
Approach2: Pick item with least weight first
Approach3: Pick item with largest value per
weight first
We know that none of the above approaches work
Exercise:
Prove by giving counterexamples.
Constraint :
Brute Force Approach
Brute Force
The naive way to solve this problem is to go
through all 2n subsets of the n items and pick
the subset with a legal weight that maximizes
the value of the knapsack.
Optimal Substructure
As we did before we are going to solve the problem in
terms of sub-problems.
Our first attempt might be to characterize a subproblem as follows:
Let Ok be the optimal subset of elements from {I0, I1
Ik}.
Observe
Optimal subset from the elements {I0, I1
k+1} may
not correspond to the optimal subset of elements from
{I0, I1
Ik}
That means, the solution to the optimization problem for
Sk+1 might NOT contain the optimal solution from
problem Sk.
where Sk: Set of items numbered 1 to k.
Optimal Substructure
Example: Let W=20
Item
I0
I1
I2
I3
B[k , w]
B[k 1, w]
if wk w
max{B[ k 1, w], B[k 1, w wk ] bk }
else
Our goal is to find B[n, W], where n is the total number of
items and W is the maximal weight the knapsack can carry.
This does have Optimal Substructure property.
Exercise Prove Optimal Substructure property.
Value
10
4
9
11
The best set of items from {I0, I1, I2} is {I0, I1, I2}
BUT the best set of items from {I0, I1, I2, I3} is {I0, I2, I3}.
Note
1. Optimal solution, {I0, I2, I3} of S3 for does NOT contain the
optimal solution, {I0, I1, I2} of S2
2. {I0, I2, I3} build's upon the solution, {I0, I2}, which is really the
optimal subset of {I0, I1, I2} with weight 12 or less.
(We incorporates this idea in our 2nd attempt)
Recursive Formulation
Sk: Set of items numbered 1 to k.
Define B[k,w] to be the best selection from Sk with weight
at most w
Weight
3
8
9
8
Optimal Substructure
Consider a most valuable load L where WL W
Suppose we remove item j from this optimal load L
The remaining load
must be a most valuable load weighing at most
pounds that the thief can take from
That is
should be an optimal solution to the
0-1 Knapsack Problem(
,
)
The 0-1 Knapsack Problem
Exercise
Overlapping Subproblems
Analysis
The 0-1 Knapsack Problem
How to find actual Knapsack Items?
if B[i,k]
1,k] then
mark the ith item as in the knapsack
i
kelse
i
// Assume the ith item is not in the
knapsack
Pseudo- code
for w = 0 to W
B[0,w] = 0
for i = 1 to n
B[i,0] = 0
for i = 1 to n
for w = 0 to W
if <= w
// item i can be part of the solution
if + B[i-1,w- ] > B[i-1,w]
B[i,w] = + B[i-1,w- ]
else
B[i,w] = B[i-1,w]
else B[i,w] = B[i-1,w]
// > w
All-Pairs Shortest Path
G = (V, E) be a graph
G has no negative weight cycles
Vertices are labeled 1 to n
If (i, j) is an edge its weight is denoted by wij
Optimal Substructure Property
Easy to Prove
Recursive formulation - 1
Recursive formulation - 2
be the minimum weight of any path from
vertex i to vertex j that contains atmost m edges.
Then,
be the weight of the shortest path from
vertex i to vertex j,
for which all the intermediate vertices are in the
set
Then
wij if k = 0
min (
1
We have tacitly used the fact that an optimal
path through k does not visit k twice.
minimum over all predecessors k of j
Optimal Binary Search Tree
BITS, PILANI K. K. BIRLA GOA CAMPUS
Design & Analysis of Algorithms
(CS F364)
Lecture No. 13
1
We start with a problem regarding binary search trees
in an environment in which the probabilities of
accessing elements and gaps between elements is
known.
Goal:
We want to find the binary search tree that minimizes
the expected number of nodes probed on a search.
Optimal Binary Search Tree
Optimal Binary Search Tree
Formally
We have n keys (represented as k1,k2 kn) in sorted order (so
that k1<k2
kn),
and we wish to build a binary search tree from these keys.
For each ki ,we have a probability pi that a search will be for ki.
In contrast, some searches may be for values not in ki , and so we
also have n+1
d0,d1 dn
In particular, d0 represents all values less than k1, and
dn represents all values greater than kn, and for i = 1,2
-1, the
dummy key di represents all values between ki and ki+1.
For each di ,we have a probability qi that a search will be for di
The dummy keys are leaves (external nodes), and the data keys
mean internal nodes.
Every search is either successful or unsuccessful and
so we have
Because we have probabilities of searches for each key
and each dummy key, we can determine the expected
cost of a search in a given binary search tree T.
Let us assume that the actual cost of a search is the
number of nodes examined,
i.e., the depth of the node found by the search in T,
plus1.
k2
Example
Optimal Binary Search Tree
k1
Then the expected cost of a search in T is
d0
E[ search cost in T]
k4
k3
d2
k5
d1
d3
d4
d5
d4
Figure (a)
(1)
i
0
T
qi
0.05
d5
k4
k5
k3
pi
Where depthT denotes
k1
d0
d1
=
=1+
k2
1
2
3
4
5
0.15
0.10
0.05
0.10
0.20
0.10
0.05
0.05
0.05
0.10
d2
d3
Figure (b)
Optimal Binary Search Tree
Observe:
Figure (a) costs 2.80 ,on another,
the Figure (b) costs 2.75,
and that tree is really optimal.
We can see the height of (b) is more than (a)
So optimal BST is not necessarily a tree whose height
is smallest.
Optimal Binary Search Tree
Brute Force Approach:
Exhaustive checking of all possibilities.
Question
What is the number of binary search trees on n keys?
Answer
The number of BST is atleast
Dynamic Programming Solution
Step1 : Optimal Substructure Property
Exercise
Step2 : Recursive Formulation
We pick our subproblem domain as finding an Optimal
BST containing the keys ki kj , where
, j , and
i-1. (It is when j = i-1 that there are no actual keys;
we have just the dummy key di-1.)
Let us define e[i, j] as the expected cost of searching an
Optimal BST containing the keys ki
kj
Ultimately, we wish to compute e[1,n].
n)
Optimal Binary Search Tree
When j = i-1
Then we have just the dummy key di-1
The expected search cost is e[i, i-1] = qi-1
When
, we need to select a root kr from among
ki kj
and then make an Optimal BST with keys ki kr-1 as its
left subtree
and an Optimal BST with keys kr+1 kj as its right
subtree.
Optimal Binary Search Tree
Optimal Binary Search Tree
What happens to the expected search cost of a subtree
when it becomes a subtree of a node?
The depth of each node in the subtree increases by 1.
So, by equation (1) the excepted search cost of this
subtree increases by the sum of all the probabilities in
the subtree.
For a subtree with keys ki kj let us denote this sum of
probabilities as
Thus, if kr is the root of an optimal subtree containing
keys ki kj , we have
e[i, j]= pr + (e[i, r-1]+w(i, r-1)) + (e[r+1, j]+w(r+1, j))
Noting that w (i, j) = w(i,r-1)+ pr +w(r+1,j)
We rewrite e[i, j] as
e[i, j]= e[i, r-1] + e[r+1, j] + w(i, j)
The recursive equation as above assumes that we know
which node kr to use as the root.
We choose the root that gives the lowest expected
search cost
Optimal Binary Search Tree
Subset Sum
Final recursive formulation:
The e[i, j] values give the expected search costs in
Optimal BST.
To help us keep track of the structure of Optimal BST, we
define root[i, j], for
, to be the index r for
which kr is
the root of an Optimal BST containing keys ki kj
Problem:
Given a set of positive integers wi where i
and an integer t.
Is there some subset of wi that sums to exactly t?
Note:
Not an optimization problem.
Exercise
1. Write a corresponding optimization problem.
2. Solve using Dynamic Programming Formulation.
Subset Sum - Backtracking
Example
(w1, w2, w3, w4) = (11, 13, 24, 7), and t = 31
Then the desired subsets are (11, 13, 7) and (24, 7)
Formulation 1:
We represent the solution vector by giving the indices
of these wi
The two solutions are described by the vectors (1, 2, 4)
and (3, 4).
In general, all solutions are k-tuples (x1, x2, xk)
k n
Different solutions may have different-sized tuple
Subset Sum - Backtracking
Formulation 2:
Each solution subset is represented by an n-tuple
(x1, x2, xn) such that xi {0, 1}
where xi = 0 if wi is not chosen and xi = 0 if wi is chosen
The solutions to the above instance are (1, 1, 0, 1) and
(0, 0, 1, 1)
This formulation expresses all solutions using a
fixed-sized tuple.
Several ways to formulate a problem so that all
solutions are tuples that satisfy some constraints
Subset Sum - Backtracking
Backtracking algorithms determine problem
solutions by systematically searching the
solution space for the given problem instance.
This search is facilitated by using a tree
organization for the solution space.
For a given solution space many tree
organizations may be possible
Formulation - 1
Formulation 1
The edges are labeled such that an edge
from a level i node to a level i +1 node
represents a value for xi
Nodes are numbered as in breadthfirst search
The solution space is defined by all
paths from the root node to any node in
the tree, since any such path
corresponds to a subset satisfying the
explicit constraint
Formulation - 2
Exercise
Exercise:
1. Independent Set: Given a graph G = (V, E), a
subset of vertices S is an independent set if
there are no edges between them
Max Independent Set Problem: Given a graph G
= (V, E), find the largest independent set in G
Solve for the restricted case, when G is a tree
2. Travelling Salesman Problem
Formulation 2
Edges from level i nodes to level i + 1
nodes are labeled with the value of xi
which is either 0 or 1.
The left subtree of the root defines all
subsets containing w1
the right subtree defines all subsets not
containing w1
Activity Selection Problem
BITS, PILANI K. K. BIRLA GOA CAMPUS
Design & Analysis of Algorithms
(CS F364)
Lecture No. 14
1
Input: A set of activities S = {a1 an}
Each activity ai has start time si and a finish
time fi such that
si < fi
An activity ai takes place in the half-open
interval [si , fi )
Two activities are compatible if and only if
their interval does not overlap
i.e., ai and aj are compatible if [si , fi ) and
[sj , fj ) do not overlap (i.e. si fj or sj fi )
Output: A maximum-size subset of mutually
compatible activities
Activity Selection Problem
Example
Recipe for Dynamic Programming
Step 1: Identify optimal substructure.
Step 2: Find a recursive formulation for the
value of the optimal solution.
Step 3: Use dynamic programming to find the
value of the optimal solution.
(bottom up manner)
A = {1,4,8,11} is an optimal solution.
A = {2,4,9,11} is also an optimal solution
Notation
Notation
= subset of activities that start after activity ai finishes
and finish before Activity aj starts
Optimal substructure
Let Aij to be an optimal solution, i.e., a maximal set of mutually
compatible activities in Sij.
At some point we will need to make a choice to include some
activity ak with start time sk and finishing by fk in this solution.
This choice will leave two sets of compatible candidates after ak is
taken out:
Sik : activities that start after ai finishes, and finish before ak
starts
Skj : activities that start after ak finishes, and finish before aj
starts
Observe
Aik = Aij Sik is the optimal solution to Sik
Akj = Aij Skj is the optimal solution to Skj
Recursive Formulation
We Show this is a Greedy Problem
Let c[i,j] be the number of activities in a
maximum-size subset of mutually compatible
activities in Sij.
Theorem:
Consider any nonempty subproblem Sij, and let am
be the activity in Sij with earliest finish time: fm=
min{fk : ak Sij}, then
1. Activity am is used in some maximum-size subset
of mutually compatible activities of Sij.
2. The subproblem Sim is empty, so that choosing am
leaves Smj as the only one that may be nonempty.
So the solution is c[0,n+1]=S0,n+1.
Recurrence Relation for c[i, j] is :
We can solve this using dynamic programming
But there is important property which we can
exploit.
Proof
Proof:
Let Aij be an optimal solution to Sij
And let ak Aij have the earliest finish time in Aij
If ak = am we are done.
Otherwise, let ij = (Aij - {ak}) {am}
(substitute am for ak)
Claim: Activities in ij are disjoint.
Proof of Claim:
Activities in Aij are disjoint because it was a solution.
Since aj is the first activity in Aij to finish, and fm j (am is the
earliest in Sij), am cannot overlap with any other activities in
Since | ij| = |Aij| we can conclude that ij is also an optimal
solution to Sij and it includes am
Solution
Greedy Solution
Repeatedly choose the activity that finishes
first
Remove activities that are incompatible with
it, and
Repeat on the remaining activities until no
activities remain.
ij
We Show this is a Greedy Problem
Example
Consequences of the Theorem?
1. Normally we have to inspect all subproblems,
here we only have to choose one subproblem
2. What this theorem says is that we only have to find
the first subproblem with the smallest finishing time
3. This means we can solve the problem top down by
selecting the optimal solution to the local subproblem
rather than bottom-up as in DP
Elements of greedy strategy
Determine the optimal substructure
Develop the recursive solution
Prove one of the optimal choices is the greedy choice yet safe
Show that all but one of subproblems are empty after greedy
choice
Develop a recursive algorithm that implements the greedy
strategy
Convert the recursive algorithm to an iterative one.
This was designed to show the similarities and differences
between dynamic programming and the Greedy approach;
These steps can be simplified if we apply the Greedy approach
directly
To solve the Si,j:
1. Choose the activity am with the earliest finish time.
2. Solution of Si,j = {am} U Solution of subproblem Sm,j
To solve S0,12, we select a1 that will finish earliest, and solve for
S1,12.
To solve S1,12, we select a4 that will finish earliest, and solve for
S4,12.
To solve S4,12, we select a8 that will finish earliest, and solve for
S8,12
And so on (Solve the problem in a top-down fashion)
Applying Greedy Strategy
Steps in designing a greedy algorithm
The optimal substructure property holds
(same as dynamic programming)
Other Greedy Strategies
BITS, PILANI K. K. BIRLA GOA CAMPUS
Other Greedy Strategies:
Greedy 1: Pick the shortest activity, eliminate all
activities that conflict with it, and recurse.
Greedy 2: Pick the activity that starts first, eliminate all
the activities that conflict with it, and recurse.
Greedy 3: Pick the activity that ends first, eliminate all
the activities that conflict with it, and recurse.
Greedy 4: Pick the activity that has the fewest conflicts,
eliminate all the activities that conflict with it, and
recurse.
Any More???
Design & Analysis of Algorithms
(CS F364)
Lecture No. 15
1
Elements of greedy strategy
Determine the optimal substructure
Develop the recursive solution
Prove one of the optimal choices is the greedy choice yet safe
Show that all but one of subproblems are empty after greedy
choice
Develop a recursive algorithm that implements the greedy
strategy
Convert the recursive algorithm to an iterative one.
This was designed to show the similarities and differences
between dynamic programming and the Greedy approach;
These steps can be simplified if we apply the Greedy approach
directly
The Fractional Knapsack Problem
Given: A set S of n items, with each item i having
wi - a positive
vi - a positive
Goal:
Choose items with maximum total benefit but with weight
at most W.
And we are allowed to take fractional amounts
The Fractional Knapsack Problem
Possible Greedy Strategies:
Pick the items in increasing order of weights
Pick the items in decreasing order of benefits
Pick the items by decreasing order of value per
pound
Note:
1st two strategies do not give optimal solution
Counterexamples - Exercise
Optimal Substructure
Optimal Substructure
Given the problem with an optimal solution
with value
Suppose X has fraction fi of the i-th item.
Let X = X {fraction of the i-th item}
Claim:
X is the optimal solution to the subproblem:
S = S - {fraction of the i-th item}
& the Knapsack capacity W fiwi
Proof: Cut-Paste Argument
Greedy Algorithm
We can solve the fractional knapsack problem with a
greedy algorithm:
Compute the value per pound (vi/wi) for each item
Sort (decreasing) the items by value per pound
Greedy strategy of always taking as much as possible
of the item remaining which has highest value per
pound
Time Complexity:
If there are n items, this greedy algorithm takes
O(nlogn) time
Greedy Choice Property
Theorem
Consider a knapsack instance P, and let item 1
be item of highest value density
Then there exists an optimal solution to P that
uses as much of item 1 as possible (that is,
min(w1, W)).
Proof:
Suppose we have a solution Q that uses weight
w <min(w1, W) of item 1.
min(w1,
Greedy Choice Property
Q must contain at least
some other
item(s), since it never pays to leave the knapsack
partly empty.
Construct Q* from Q by
of
other items and
of item
1
Because item 1 has max value per weight, So
Q* has total value at least as big as Q.
Alternate Proof - 1
Alternate Proof - 1
Alternate Proof:
Assume the objects are sorted in order of cost
per pound.
Let vi be the value for item i and let wi be its
weight.
Let xi be the fraction of object i selected by
greedy and let V(X) be the total value obtained
by greedy
Alternate Proof
Alternate Proof - 1
Alternate Proof - 1
Alternate proof (2) - Proof by contradiction
We can also prove by contradiction.
We start by assuming that there is an optimal solution where we did
not take as much of item i as possible and we also assume that our
knapsack is full (If it is not full, just add more of item i).
Since item i has the highest value to weight ratio, there must exist an
item j in our knapsack such that
We can take item j of weight x from our knapsack and we can add item
i of weight x to our knapsack (Since we take out x weight and put in x
weight, we are still within capacity).
BITS, PILANI K. K. BIRLA GOA CAMPUS
Design & Analysis of Algorithms
(CS F364)
Lecture No. 16
The change in value of our knapsack is x
Therefore
solution in our starting assumption, can in fact be improved by taking
out some of item j and adding more of item i. Hence, it is not optimal.
1
Coding
Fixed Length coding
Suppose that we have a 100000 character data file that
we wish to store. The file contains only 6 characters,
appearing with the following frequencies:
Frequency in thousands
a
b
c
d
e
f
45
13
12
16
9
5
A binary code encodes each character as a binary string
or code word.
Goal: We would like to find a binary code that encodes
the file using as few bits as possible
In a fixed-length code each code word has the same
length.
For our example, a fixed-length code must have at least
3 bits per code word. One possible coding is as follows:
Frequency in thousands
Fixed length coding
a
45
000
b
13
001
c
12
010
d
16
011
e
9
100
f
5
101
The fixed length code requires 300000 bits to store the
file
Key Property - Encoding
variable-length code
In a variable-length code, code words may have
different lengths.
One possible coding for our example is as follows:
a
b
c
d
e
f
Frequency in thousands
45
13
12
16
9
5
Variable length coding
0
101
100
111
1101
1100
The variable-length code uses only
(45x1+13x3+12x3+16x3+9x4+5x4)x1000= 224000 bits
Much better than fixed length code
Can we do better?
Key Property
Given an encoded message, decoding is the process of
turning it back into the original message. A message is
uniquely decodable, if it can only be decoded in one way.
Encoding should be done so that message is uniquely
decodable.
Note:
1. Fixed-length codes are always uniquely decodable
2. May not be so for variable length encoding
Example : C = {a= 1; b= 110; c= 10; d= 111}
1101111 is not uniquely decipherable since it could have
encoded either bad or acad.
Variable-length codes may not be uniquely decodable
Prefix Codes
Prefix Code: A code is called a prefix code if no code word
is a prefix of another one.
Example:
{a= 0; b= 110; c= 10; d= 111} is a prefix code.
Important Fact:
Every message encoded by a prefix code is uniquely
decipherable.
Since no code-word is a prefix of any other we can always
find the first code word in a message, peel it off, and
continue decoding.
Example:
0110100 = 0110100 = abca
We are therefore interested in finding good prefix codes.
Representation Binary Tree
Fixed-length prefix code from the example
earlier represented as a binary tree (the
numbers inside the nodes are the sum of
frequencies for characters in each subtree):
Representation Binary Encoding
Representation
Prefix codes (in general any binary encoding) can be
represented as a binary tree.
Left edge is labeled 0 and the right edge is labeled 1
Label of leaf is frequency of character.
Path from root to leaf is code word associated with
character.
Cost of Tree
For each character c in the set C let freq.c denote the
frequency of c in the file
Given a tree T corresponding to a prefix code
dT(c) denote the depth of leaf in the tree
dT(c) is also the length of the codeword for character c
The number of bits required to encode a file is
An
prefix code from the
example earlier represented as a binary
tree
which is defined as cost of the tree T
Full Binary Tree
Key Idea:
An optimal code for a file is always represented by a
full binary tree.
Full binary tree is a binary tree in which every non-leaf
node has two children
Proof: If some internal node had only one child then
we could simply get rid of this node and replace it with
its unique child. This would decrease the total cost of
the encoding.
Greedy Choice Property
Let T be an optimum prefix code tree, and let b and c
be two siblings at the maximum depth of the tree
(must exist because T is full).
Since x and y have the two smallest frequencies it
f(b) and
Now switch the positions of x and b in the tree
Greedy Choice Property
Lemma:
Consider the two letters, x and y with the
smallest frequencies. Then there exists an
optimal code tree in which these two letters are
sibling leaves in the tree at the lowest level
Proof (Idea)
Take a tree T representing arbitrary optimal
prefix code and modify it to make a tree
representing another prefix code such that the
resulting tree has the required greedy property.
Greedy Choice Property
Since T is optimum
Huffman Coding
Optimal Substructure Property
Optimal Substructure Property
Lemma:
Let T be a full binary tree representing an optimal
prefix code over an alphabet C, where frequency
f[c] is define for each character c belongs to set C.
Consider any two characters x and y that appear as
sibling leaves in the tree T and let z be their parent.
Then, considering character z with frequency f[z] =
f[x] + f[y], tree T` = T - {x, y} represents an optimal
code for the alphabet C` = C - {x, y}U{z}.
Proof : See CLRS
Huffman Code Algorithm
Huffman Code Algorithm
Step1:
Pick two letters x; y from alphabet C with the smallest
frequencies and create a subtree that has these two
characters as leaves.
Label the root of this subtree as z
Step2:
Set frequency f(z) = f(x) +f(y).
Remove x; y and add z creating new alphabet
A* =A
{z} {x, y}, Note that |A*|= |A| - 1
Repeat this procedure, called merge, with new alphabet A* until an alphabet with only one symbol is left.
Huffman Code - Example
Example:
Huffman Code - Example
Example Continued
Minimum Spanning Tree (MST)
Optimal substructure for MST
Lemma:
-{e}, then T' {e} is an MST of
G.
Proof:
Exercise
Huffman Code - Example
Example Continued
Minimum Spanning Tree
Prims Algorithm
Choose some v V and let S = {v}
Let T = Ø
While
V
Choose a least-cost edge e with one
endpoint in S and one endpoint in V S
Add e to T
Add both endpoints of e to S
Minimum Spanning Tree
Greedy Choice Property
Theorem:
Let (S, V S)be a nontrivial cut in G
(
If (u, v) is the lowest-cost edge crossing(S, V S),
then (u, v) is in every MST of G.
Proof:
Exercise
BITS, PILANI K. K. BIRLA GOA CAMPUS
Design & Analysis of Algorithms
(CS F364)
Lecture No. 17
1
Minimize time in the system
Problem Statement:
A single server with N customers to serve
Customer i will take time ti
Goal: Minimize average time that a customer spends in
the system.
where time in system for customer i = total waiting time + ti
Since N is fixed, we try to minimize time spend by all
customers to reach our goal
Minimize T =
Minimize time in the system
Example:
Assume that we have 3 jobs with t1 = 5, t2 = 3, t3 = 7
Order
Total Time In System
1, 2, 3
5 + (5 + 3) + (5 + 3 + 7) = 28
1, 3, 2
5 + (5 + 7) + (5 + 7 + 3) = 32
2, 3, 1
3 + (3 + 7) + (3 + 7 + 5) = 28
2, 1, 3
3 + (3 + 5) + (3 + 5 + 7) = 23
3, 1, 2
7 + (7 + 5) + (7 + 5 + 3) = 34
3, 2, 1
7 + (7 + 3) + (7 + 3 + 5) = 32
optimal
Minimize time in the system
Brute Force Solution:
Time Complexity: N!
Which is exponential!!
Optimal Substructure Property:
Exercise
Minimize time in the system
Observe
Let P = p1p2 pN be any permutation of the
integers 1 to N and let si =
If customers are served in the order corresponding
to P, then
Total time passed in the system by all the
customers is
T(P) = s1 + (s1 + s2) + (s1+ s2 + s3) + .....
= ns1 + (n 1)s2 + (n 2)s3
=
Minimize time in the system
Greedy Strategy
At each step, add to the end of the schedule, the
customer requiring the least service time
among those who remain.
So serve least time consuming customer first.
Minimize time in the system
Theorem:
Greedy strategy is optimal.
Proof:
Suppose strategy P does not arrange the customers in
increasing service time.
Then we can find two integers a & b with a < b and sa > sb
i.e, the a-th customer is served before the b-th
customer even though the former needs more service
time than the latter.
Minimize time in the system
Scheduling with deadlines
Now, we exchange the position of these two customers to
obtain a new order of service O
Then
And T(P) T(O)
= (n a + 1)(sa sb) + (n b + 1)(sb sa)
= (b a)(sa sb)
>0
i.e., the new schedule O is better than the old schedule P
Scheduling with deadlines
Greedy Algorithm
Example,
i
1
2
3
4
gi
50
10
15
30
di
2
1
2
1
Possible schedules & corresponding profits are
Sequence
Profit
Sequence
Profit
1
50
2, 1
60
2
10
2, 3
25
3
15
3, 1
65
4
30
4, 1
80
1, 3
65
4, 3
45
Problem Statement:
We have set of n jobs to execute
Each job takes unit time to execute
At any time T=1,2,.. we can execute exactly one
job
Job i earns us profit gi > 0 if and only if it is
executed no later than time di
Goal: Find a schedule that maximizes the profit
optimal
The sequence 2, 3, 1 is not considered, as job 1 would be executed at time t=3,
after its deadline
Definition
A set of jobs is feasible if there exits at least one
sequence (called feasible sequence) that allows all the
jobs in the set to be executed no later than their
respective deadlines.
Greedy Algorithm
Beginning with empty schedule, at each step,
Add the job with the highest profit among those not
yet considered,
provided that the chosen set of jobs remains feasible.
Greedy Algorithm
Greedy Proof
For our example,
We first choose job 1.
Next we choose job 4, the set {1, 4} is feasible because
it can be executed in the order 4, 1
Next we try the set {1, 3, 4} which is not feasible & so
job 3 is rejected
Finally we try set {1, 2, 4} which is not feasible & so job
2 is rejected
Our optimal solution is the set of jobs {1, 4}
Theorem:
The greedy algorithm always finds an optimal solution.
Proof:
Suppose greedy algorithm returns set of jobs I
And suppose the set J is optimal.
Let SI and SJ be the corresponding feasible sequences
Claim:
By rearranging the jobs in SI and those in SJ we can obtain
two feasible sequences and (which may include gaps)
such that every job common to I and J is scheduled at the
same time in both the sequences.
Example
Proof of the claim
For example, Suppose
p
y
q
x
r
r
s
t
p
u
SI
v
q
w
After reorganization
x
y
p
r
p
r
q
gap
u
s
t
v
q
w
SJ
Proof of the claim:
Suppose that some job a occurs in both the feasible
sequences SI and SJ where it is scheduled at times
tI and tJ respectively.
Case 1: If tI = tJ there is nothing to prove
Case 2: If tI < tJ
Note, since sequence is SJ feasible, it follows that the
deadline for job a is no earlier than tJ
Modify sequence SI as follows:
Case i : Suppose there is a gap in SI at time tJ move job
a from time tI into the gap at tJ
Proof of the claim
Case ii :
Suppose there is some job b scheduled in SI at time tJ ,
then exchange jobs a and b in SI
The resulting sequence is still feasible, since in either
case job a will be executed by its deadline, and in the
second case job b is moved to an earlier time and so
can be executed.
So job a is executed at the same time tJ in both the
modified sequences SI and SJ
Case 3: If tI > tJ
Similar argument works, except in this case SJ is
modified
Greedy Proof
Once job a has been treated in this way, we never need
to move it again.
Therefore, if SI and SJ have m jobs in common, after at
most m modifications of either SI or SJ we can ensure
that all the jobs common to I and J are scheduled at the
same time in both sequences.
and
be the resulting sequences.
Suppose there is a time when the job scheduled in
is different from that scheduled in
Greedy Proof
Greedy Proof
Case1:
If some job a is scheduled in opposite a gap in .
Then a does not belong to J & the set J {a} is feasible.
So we have a feasible solution profitable than J.
This is impossible since J is optimal by assumption.
Case2:
If some job b is scheduled in opposite a gap in .
Then b does not belong to I & the set I {b} is feasible.
So the greedy algorithm would have included b in I.
This is impossible since it did not do so.
Case 3:
Some job a is scheduled in opposite a different job b in
In this case a does not appear in J and b does not appear in I.
Case i : If ga > gb
Then we could replace a for b in J and get a better solution.
This is impossible because J is optimal.
Case ii : If ga < gb
The greedy algorithm would have chosen b before considering a
since (I \ {a}) {b} would be feasible.
This is impossible because the algorithm did not include b in I.
Case iii : The only remaining possibility is ga= gb
The total profit from I is therefore equal to the profit from the
optimal set J, and so I is optimal to.