CISC 621
Algorithm Design and Analysis
Dr. Rui Zhang
Course Information
• Instructor: Rui (Ray) Zhang
• Office: 316A FinTech
• Email: ruizhang@udel.edu
• Meeting time
• Section 010: Tuesday/Thursday 2:20PM--3:40PM
• Section 011: Tuesday/Thursday 3:55PM--5:15PM
• Meeting location
• Section 010: Gore Hall Room 315
• Section 011: Smith Hall Room 130
Course Information
• Office hour
• 11:00AM--3:00PM Wednesday
• Appointment via email
• TA
• Xihan Qin (xihan@udel.edu)
• Jiahao Xie (jiahaox@udel.edu)
Course Objectives
• Upon completion of this course, you will be able to do the
following
• Demonstrate a familiarity with popular algorithmic problems, e.g.,
sorting, shortest path, and minimum spanning tree
• Understand important algorithmic design paradigms such as
dynamic programming, greedy algorithm, and incremental methods
• Analyze the (asymptotic) performance of algorithms
• Synthesize efficient algorithms in common engineering design
situations
• Understand the concept of problem hardness, and be able to
identify NP-hard problems
Course Outline
• Asymptotic analysis
• Randomized algorithm and probabilistic analysis
• Divide-and-conquer paradigm
• Dynamic-programming paradigm
• Greedy paradigm
• Incremental method
• Major graph algorithms
• Advanced data structures
• NP-completeness
• Approximation algorithms
Required Textbook
• Cormen, Leiserson, Rivest and Stein, Introduction to
Algorithms, McGraw-Hill & MIT Press, 4th edition, ISBN-10:
026204630X, ISBN-13: 978-0262046305
Reference Textbook
• Jon Kleinberg and Eva Tardos, Algorithm Design, Pearson; 1st
edition, 2017, ISBN-10: 9780321295354, ISBN-13: 9780321295354
Homework
• There will be 5 homework assignments
• You can discuss homework assignments with others but are
required to write up each problem solution yourself and
identify your collaborator, if any.
• You are required to type homework assignments using a
word processor (e.g., MS Word or Latex). Handwriting will
not be accepted
• You are required to turn in via Canvas by 11:59PM on the
specified due date.
Policies on Late Submissions
• Homework and assignment deadlines will be hard
• Late submission will NOT be graded
Exam
• There will be two midterm exams
• Midterm I is scheduled for October 9th
• Midterm II is scheduled for November 13th
• Both midterms will consist of several problems similar to
homework problems
• Both are close-books/notes
• There will be one final exam
• The final exam will follow the university schedule
• Close-books/notes
Generative AI Tools
• Students are not allowed to use advanced automated tools
(artificial intelligence or machine learning tools such as
ChatGPT or Dall-E 2) on assignments in this course. Each
student is expected to complete each assignment without
substantive assistance from others, including automated
tools.
Grading
• Final course grades will be based on homework assignments,
midterm exams, and the final exam
• Homework assignments . . . . . . . . . . . . . . . . . 25%
• Midterm I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25%
• Midterm II . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20%
• Final exam . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28%
• Course evaluation .. . . . . . . . . . . . . . . . . . . . . . . . 2%
Grading
• A:>=93
• A-: >=90
• B+:>=87
• B: >=84
• B-: >=80
• C+: >=77
• C: >=74
• C-: >=71
• D+:>=68
• D:>=64
• D-: >=60
• F:<60
Lecture 1
Analysis of Algorithm, Insertion Sort,
Merge Sort
Sorting Problem
• Input:
• Output: a permutation of input sequence
such that
Insertion Sort
• Good for sorting a small number of elements
• Intuition: mimic the way to sort a hand of playing cards
How to Express Algorithms?
• Express algorithms in whatever way is the clearest and most
concise
• When issues of control need to be made perfectly clear, we
often use pseudocode
• Pseudocode is similar to C, C++, Pascal, and Java
• The goal is to express algorithms to humans
• We sometimes embed English statements into pseudocode
• Ignore error handling
Pseudocode for Insertion Sort
Example
Proof of Correctness using Loop Invariant
• Loop invariant
• A property of a program loop that is true before each iteration
• Initialization
• It is true prior to the first iteration of the loop
• Maintenance
• If it is true before an iteration of the loop, it remains true before the
next iteration
• Termination
• When the loop terminates, the invariant—usually along with the
reason that the loop terminated—gives us a useful property that
helps show that the algorithm is correct
Loop Invariant of Insertion Sort
• At the start of each iteration of the “outer” for loop
consists of the elements originally in
but in sorted order
Correctness of Insertion Sort
• Initialization
• Just before the first iteration,
. The subarray
the single element
, which is trivially sorted
is
• Maintenance
• Insert
s
into the right position of
is sorted.
such that
• Termination
• The outer loop ends, when
invariant, we get
is sorted
. Plug
into loop
How to Analyze Algorithms?
• Random-access machine (RAM) model
• Instructions executed one after another
• Instructions commonly found in real computers
• Arithmetic: add, subtract, multiply, divide, remainder, floor, ceiling, etc.
• Data movement: load, store, copy
• Control: conditional/unconditional branch, subroutine call and return
• Each of these instructions takes constant time
• Limit on the word size.
• It takes
bits to store a number
How to Analyze an Algorithm’s Running Time
• Running time depends on the input
• Input size matters
• Ex: sorting 10 numbers vs. sorting 10000 numbers
• Different inputs of the same size may have different running time
• Ex: sorting 1000 random numbers vs. sorting 1000 almost sorted
numbers
• Input size
• Usually, the number of items in the input
• Could be something else
• Multiplying two integers: the total number of bits of two integers
• Graph algorithm: the number of vertexes and the number of edges
• etc.
Running Time
• The number of primitive operations (steps) executed
• Define steps to be machine-independent
• Assume that each line of pseudocode requires a constant
amount of time
• One line may take a different amount of time than another
• Line takes constant time
operations
if it consists only of primitive
Analysis of Insertion Sort
: the number of times that the while loop test is executed for that value of
Analysis of Insertion Sort
• Let
be the running time of insertion sort
• Running time
depends on
Best Case
• The array is already sorted
• Always find that
test (when
)
•
for all
• Running time is given by
•
can be expressed as
upon the first time the while loop
, i.e.,
is a linear function of
Worst Case
• The array is in reverse sorted order
• Always find that
• Have to compare
•
for all
• It follows that
in while loop test
with all elements to the left (
elements)
Worst Case
• Running time is given by
•
is a quadratic function of
Which One Shall We Care About?
• Worst-case running time
• Running time guarantee for any input of size
• Average-case running time
• Expected running time for any input of size
• Require assumptions about probability distribution of inputs
• What if the input distribution is not known?
• Best-case running time
Other Types of Analysis
• Probabilistic analysis
• Expected running time of a randomized algorithm for any input of
size
• Amortized analysis
• Worst-case running time for any sequence of
operations
• Competitive analysis
• The performance of an online algorithm is compared to the
performance of an optimal offline algorithm
• etc.
Order of Growth
• Important abstraction to ease analysis and focus on the
important features
• Drop lower-order terms
• Ignore the constant coefficient in the leading term
• Example
• The worst-case running time of insertion sort is
• We say that the running time is
to capture the notion that
that the order of growth is
• More on asymptotic notation later
Divide and Conquer Paradigm
• Divide the problem into a number of subproblems that are
smaller instances of the same problem
• Conquer the subproblems by solving them recursively
• Base case: If the subproblems are small enough, just solve them by
brute force
• Combine the subproblem solutions to give a solution to the
original problem
Merge Sort
• A sorting algorithm based on divide and conquer
• Its worst-case running time has a lower order of growth
than insertion sort
• Subproblem
• Sorting an array
, initially
• Divide by splitting into two subarrays
• Conquer by recursively sorting the two subarrays
• Combine by merging the two sorted subarrays
Pseudocode of Merge Sort
Example
Example
Merging:
• Input: Array
and indices
•
• Subarrays
sorted
and
such that
are both non-empty and
• Output: A single sorted subarray in
• Design goal
• Implement it so that it takes
• What is here?
• The size of the subproblem (
time
)
Idea behind Linear Time Merging
Two sorted pile of cards
At each step, choose the smaller of the two top cards
How many steps in total?
Pseudocode
Analyzing Divide-and-Conquer Algorithms
• Use a recurrence equation (more commonly, a recurrence) to
describe the running time of a divide-and-conquer algorithm
• Let
be the running time on a problem of size
(base case)
subproblems each of size
time to combine solutions
time to divide a size- problem
Analyzing Merge Sort
• For simplicity, assume that
is a power of 2
• Each divide step yields two subproblems, both of size exactly
• Base case: when
• Divide: just compute
, so
• Conquer: recursively solve 2 subproblems, each of size
exactly
, so
• Combine: MERGE on an -element subarray takes
• So we have
Solving the Merge-Sort Recurrence
• By the Master Theorem (Ch. 4), we can show that this
recurrence has the solution
• Compared to insertion sort (
running time), merge sort is faster
worst-case
• Actually a big deal!
• Note that insertion sort may be faster on small input
Understand the Solution Without Master
Theorem
• Rewrite the recurrence as (simplified)
• We can draw a recursion tree
• For the original problem
Understand the Solution Without Master
Theorem
• For each of the size-
subproblems
Understand the Solution Without Master
Theorem
• Continue expanding until the problem sizes get down to 1
Summary of Recursion Tree
• Each level has cost
• Top level has cost
• The next down has 2 subproblems, each contributing cost
• The next level has 4 subproblems, each contributing cost
• …
• Total
• Total cost is
levels
, so
Reading Assignment
• Chapters 2