4/11/2020
CS 583 Fall 2006
Analysis of Algorithms
CS583 Fall'06: Introduction 1

• Class discussion
• Algorithms
– Definition
– Examples
– Efficiency
• Insertion sort
– Pseudocode
– Proof of correctness
– Performance analysis
4/11/2020 CS583 Fall'06: Introduction 2

• Text book
– Introduction to Algorithms, Second Edition by T. Cormen, C.
Leiserson, R. Rivest, and C. Stein, MIT Press, 2001.
• Course
– This course covers concepts associated with design and analysis of algorithms.
– Algorithms capture fundamental ideas of computational systems and often determine the quality and efficiency of software systems.
– Designing efficient algorithms in many cases depends on choosing proper data structures; hence the attention will be paid to both concepts.
– The main topics include: complexity analysis, data structures, sorting, and graph algorithms.
4/11/2020 CS583 Fall'06: Introduction 3

• 3 homeworks – 20%
• Midterm exam – 30%
• Final exam – 40%
• Class participation – 10%
• Both exams are comprehensive, i.e. based on all material covered to date.
• Class participation grade will be based on being active during the class: answering questions, suggesting solutions, etc.
– Active participation is encouraged, but no participation will not be penalized.
4/11/2020 CS583 Fall'06: Introduction 4

August 29
Ch.1, 2: Introduction, Insertion sort, Analyzing algorithms
September 5 Ch.2-5: Merge sort, Asymptotic notation, Randomized
Algorithms
September 12 Ch.6: Heapsort
HW1 assigned
September 19 Ch.7: Quicksort
HW1 due
September 26 Ch 8-9: Sorting in Linear Time, Medians and Order
Statistics
October 3 Ch. 10,13: Basic Data Structures, Red-Black Trees
October 17 Midterm Exam
4/11/2020 CS583 Fall'06: Introduction 5

Ch. 14: Augmenting Data Structures October 24
October 31 Ch.18: B-Trees
November 7 Ch. 19: Binomial Heaps
HW2 assigned
November 14 Ch.20: Fibonacci Heaps
HW2 due
November 21 Ch. 22: Basic Graph Algorithms
November 28 Ch. 23: Minimum Spanning Trees
HW3 assigned
December 5 Ch. 34: NP-Completeness
HW3 due
4/11/2020 CS583 Fall'06: Introduction 6

• Definitions
– An algorithm is a well-defined computational procedure that takes a set of values as input and produces a set of values as output .
– An algorithm can also be thought of as a tool for solving a computational problem that specifies in general terms the desired input/output relationship. For example, the sorting problem:
• Input: A sequence of n numbers < a
1
, ... , a n
>.
• Output: Reordering of the input sequence < a
1
', ... , a n
' > such that:
– a
1
' <= a
2
' <= ... <= a n
'
4/11/2020 CS583 Fall'06: Introduction 7

• Definitions
– An algorithm is correct if for every input instance, it finishes with the correct output. We say that a correct algorithm solves the given computational problem.
– A data structure is a way to store and organize data in order to facilitate access and modifications.
– There are problems for which no efficient solution is known. An interesting subset of these problems is called
NP-complete . These problems have a remarkable property that if an efficient algorithm exists for any one of them, then efficient algorithms exist for all of them.
4/11/2020 CS583 Fall'06: Introduction 8

• Internet related problems:
– Finding good routs on which data will travel.
– Implementing search engines to quickly find pages on which particular information resides.
• Financial problems.
– Find a price of a complex financial instrument using simulated environments.
– Value a large portfolio of securities.
• Information processing.
– Data base related problems such as records search, insertion, deletion.
– XML based processing.
4/11/2020 CS583 Fall'06: Introduction 9

• Two algorithms that are designed to solve the same problem may differ dramatically in their efficiency.
– For example, compare an insertion sort algorithm with merge sort algorithm.
• While the difference may be small for small input size, it becomes very pronounced for large size inputs, where time may really matter.
10 4/11/2020 CS583 Fall'06: Introduction

• The procedure takes as a parameter an array A[1..n] containing a sequence of n elements to be sorted.
The input numbers are sorted in place.
• The algorithm works by finding a correct position for each number among the numbers sorted so far.
The number slides into the right position by comparing with the sorted numbers. There are two approaches:
– Compare from the beginning of the array. In this case all array would need to be shifted forward.
– Compare from the end of an array. This requires shifting one number at a time.
4/11/2020 CS583 Fall'06: Introduction 11

INSERTION-SORT (A, n)
1 for j=2 to n
2 key=A[j]
3 // insert A[j] into sorted A[1..j-1]
4 i=j-1
5 while i>0 and A[i]>key
6 A[i+1]=A[i]
7 i=i-1
8 A[i+1] = key
4/11/2020 CS583 Fall'06: Introduction 12

• Loop invariants
– At the start of the for loop of lines 1-8, the subarray
A[1..j-1] consists of the elements originally in A[1..j-1] but in sorted order. Need to show the following:
– Initialization: It is true prior to the first iteration of the loop.
• One element is always sorted.
– Maintenance: It remains true before the next iteration
• An array is moved to find the right place for A[j] , hence it remains sorted.
– Termination: When the loop terminates, the invariant gives us a useful property that help to show that the algorithm is correct.
• When the loop ends, A[1..n] array is sorted.
4/11/2020 CS583 Fall'06: Introduction 13

• Analyzing an algorithm means predicting the resources that the algorithm requires. Most often we want to measure computational time.
• We assume a generic one-processor random-access machine ( RAM ) model of computation. In the RAM model, instructions are executed one after another, with no concurrent operations.
– The RAM model contains arithmetic, data movement, and control instructions. Each such instruction takes a constant amount of time.
– The data types in the RAM model are integer and floating point.
4/11/2020 CS583 Fall'06: Introduction 14

• Note that the loop statement is executed one extra time to exit from it.
• Let t_j be the number of time the while loop is executed for that value of j .
INSERTION-SORT (A, n) cost times
1 for j=2 to n c1 n
2 key=A[j] c2 n-1
3 // insert A[j] into sorted A[1..j-1]
4 i=j-1
5 while i>0 and A[i]>key
6 A[i+1] = A[i]
7 i = i-1
8 A[i+1] = key c4 c5 c6 c7 c8 n-1
 j=2..n
 j=2..n
 j=2..n
n-1 t j
(t
(t j
-1) j
-1)
4/11/2020 CS583 Fall'06: Introduction 15

Total cost T(n) is:
T(n) = c1*n +c2*(n-1) + c4*(n-1) + c5*  c7*  j=2..n
t j=2..n
(t j
+c6*  j=2..n
(t j j
-1) + c8(n-1)
-1) +
The best case occurs when the array is already sorted:
T(n) = c1*n +c2*(n-1) + c4*(n-1) + c8(n-1) =
(c1+c2+c4+c5+c8)n – (c2+c4+c5+c8)
The running time can be expressed as an+b, i.e. a linear function of n .
4/11/2020 CS583 Fall'06: Introduction 16

If the array is in reverse order – the worst case results.
In this case t j
= j, which means
 j=2..n
 j=2..n
t j
(t
= n(n+1)/2 -1 j
-1) = n(n-1)/2
T(n) = (c5/2+c6/2+c7/2)n 2 +
(c1+c2+c4+c5/2-c6/2-c7/2+c8)n –
(c2+c4+c5+c8)
The worst-case running time can be expressed as an^2+bn+c; it is thus a quadratic function of n.
4/11/2020 CS583 Fall'06: Introduction 17

• It is the rate of growth of the running time that really interests us. We therefore consider only a leading term of a formula in the running time and ignore the constant.
• We write the insertion sort worst-case running time as

(n 2 ).
4/11/2020 CS583 Fall'06: Introduction 18