Lecture 1
Preliminaries
COMP 523: Advanced Algorithmic Techniques
Lecturer: Dariusz Kowalski
Lecture 1: Preliminaries
1
Announcements
• Textbook: J. Kleinberg, E. Tardos. Algorithm
Design. Addison-Wesley, 2005
• Lectures: Wednesday 9-11, Friday 15-16 in 3.10
• Tutorials: Thursday 11-12 in 1.01 Ashton Bldg
– Eleni Akrida (akridel@hotmail com)
• Office hours (room 3.11, Ashton Bldg):
– Wednesday 12-13
– Friday 14-15
• Info and contact:
– http://www.csc.liv.ac.uk/~darek/comp523.html
– darek@liv.ac.uk
Lecture 1: Preliminaries
2
Assessment and feedback
• Assessment: Exam (75%) and 2 tasks (25%
total)
• Feedback:
– Presentation and discussion of solutions after
assignment deadline
– In case of specific questions, individual
appointments
Lecture 1: Preliminaries
3
Algorithms - what does it mean?
Algorithm - a list/structure of instructions, which are
carried out in a fixed order
– to find the answer to a question
– to calculate
Al-Khwarizmi (Muhammed Ibn Musa) Arabian mathematician who introduced
the decimal positional number system to
the Western world, the author of Algebra
This module: techniques to design algorithms for
problems concerning data commonly processed by
computers (written in hard discs, propagated in
networks, entered by user or external device, etc.)
Lecture 1: Preliminaries
4
Types of algorithms
• Constructive vs. Non-constructive
• Discrete vs. Numerical
• Deterministic vs. non-deterministic (e.g.,
Randomized, Quantum)
• Sequential vs. Concurrent vs. DNA vs. …
• Exact vs. Approximate
• and many others
Lecture 1: Preliminaries
5
Example
Problem: Find if a given word occurs in a given
text
Input: Text and word represented as lists
Output: Word found or not
o
n
c
o
n
a
e
u
p
Lecture 1: Preliminaries
o
n
a
t
6
Example - algorithm
Naïve solution (exhaustive search):
Intuition:
• Check letter after letter, starting from the beginning of the lists, if the
corresponding letters are equal
• If some corresponding letters are not equal, re-start comparing from the second
letter of the text (and the first letter of the word)
• Etc. from the 3rd letter, 4th letter, until the end of text or successful comparison
Implementation:
• Initiate three pointers:
– blue_pointer and black_pointer at the beginning of blue_list
– yellow_pointer at the beginning of yellow_list
• Initiate Boolean variable successful into false
Lecture 1: Preliminaries
7
Naïve solution cont.
• Repeat
–
–
–
–
–
stop := false
Move blue_pointer to the next element
Set black_pointer to blue_pointer
Move yellow_pointer to the first element of yellow_list
Repeat
• Move black_pointer and yellow_pointer to the next elements in
corresponding lists
• If values pointed by black_pointer and yellow_pointer are different
then stop := true
until yellow_pointer or black_pointer is at the end of its list or stop
– If yellow_pointer is at the end of yellow_list and not stop then
successful := true
until successful or blue_pointer is at the end of blue_list
Output: successful
Lecture 1: Preliminaries
8
Example - how it works
o
n
c
o
n
a
e
u
p
o
n
a
t
• Repeat
•
•
•
–
stop := false,
move blue_pointer to the next element, set black_pointer to blue_pointer,
move yellow_pointer to the first element of yellow_list
Repeat
• Move black_pointer and yellow_pointer to the next elements in corresponding lists
• If values pointed by black_pointer and yellow_pointer are different then stop := true
–
until yellow_pointer or black_pointer is at the end of its list or stop
If yellow_pointer is at the end of yellow_list and not stop then successful := true
until successful or blue_pointer is at the end of blue_list
Lecture 1: Preliminaries
9
What we expect from algorithms?
• Correctness: computes desired output
• Termination: stops eventually (or with high probability)
• Efficiency: with respect to
– Performance measure
•
•
•
•
Time (or total number of computation steps)
Size of memory used
Number of messages sent
…
– Methods of measuring
Analysis
depends on
Model!
• Worst-case
• Average-case
• Smoothed (a subset of possible inputs, specific distribution of inputs,
etc.)
• Competitive (comparing to the best solution for particular input)
• Expected
• …
Lecture 1: Preliminaries
10
Example - correctness
• Successful is set to true iff after
some execution of the internal loop
the yellow_pointer is at the end of
the yellow_list
• yellow_pointer is at the end of the
yellow_list iff it came through the
whole yellow_list without coming
back to the beginning
• it happens if the values pointed by
black_pointer and yellow_pointer
have been the same during all
checks of internal loop in the
current run (thus a copy of the word
exists in the text)
Repeat
•
•
•
–
stop := false,
move blue_pointer to the next element,
set black_pointer to blue_pointer,
move yellow_pointer to the first
element of yellow_list
Repeat
• Move black_pointer and yellow_pointer
to the next elements in corresponding lists
• If values pointed by black_pointer and
yellow_pointer are different then stop :=
true
–
until yellow_pointer or black_pointer
is at the end of its list or stop
If yellow_pointer is at the end of
yellow_list and not stop then successful
:= true
until successful or blue_pointer is at the
end of blue_list
Lecture 1: Preliminaries
11
Example - termination
• External Loop Invariant:
At the beginning of the external loop,
blue_pointer advances, and it has a
finite number of advances to take
before reaching the end of the text
• Internal Loop Invariant:
Each run of the internal loop finishes
eventually;
It happens because yellow_pointer
keeps advancing every iteration (first
line of the loop), unless stop
condition becomes true or it reaches
the end of the word (or black_
pointer reaches the end of the text)
Repeat
•
•
•
–
stop := false,
move blue_pointer to the next element,
set black_pointer to blue_pointer,
move yellow_pointer to the first
element of yellow_list
Repeat
• Move black_pointer and yellow_pointer
to the next elements in corresponding lists
• If values pointed by black_pointer and
yellow_pointer are different then stop :=
true
–
until yellow_pointer or black_pointer
is at the end of its list or stop
If yellow_pointer is at the end of
yellow_list and not stop then successful
:= true
until successful or blue_pointer is at the
end of blue_list
Lecture 1: Preliminaries
12
Example - efficiency
• Complexity measures: time, size of additional memory
• Size of additional memory:
– 3 single pointers and 2 binary variables
• Time complexity:
– Worst-case: if the sequence is not included in the text then
time is (almost) proportional to the multiplication of sizes of
blue and yellow lists, e.g.,
text: ‘aab’ repeated n times
word: aaa
1, 2 or 3 internal loop runs, 3n times, gives about 6n runs
Lecture 1: Preliminaries
13
Example - efficiency cont.
• Time complexity (cont.):
– Average case: text or sequence are randomly selected;
average short word should be find quickly in the
beginning of an average long text, in time proportional
to the squared length of the word (exercise)
– Smoothed analysis: text must be randomly selected
from “reasonable” set of texts (usually complicated
analysis, depends on the family of texts - dictionary)
Lecture 1: Preliminaries
14
Efficiency - asymptotic notation
• n - integer, size of data, f() - function over integers
• Complexity O(f(n)) : there is a positive constant c such
that for every positive integer n complexity is at most
cf(n)
• Complexity (f(n)) : there is a positive constant c such
that for every positive integer n complexity is at least
cf(n)
• Complexity o(f(n)) : complexity divided by f(n) comes
to 0 with n going to infinity (strictly smaller than f(n))
• Complexity (f(n)) : complexity divided by f(n) comes
to  with n going to infinity (strictly bigger than f(n))
Lecture 1: Preliminaries
15
Example - asymptotic complexity
• Worst case time:
– If size of text is n and size of word is n/2 :
O((n/2)(n/2)) = O(n2)
– If size of text is m and size of sequence is k :
O((m-k)k)
Lecture 1: Preliminaries
16
Examples
5n3+100 = O(n3) , 5n3+100 ≠ O(n2)
5n3+100 = (n3) , 5n3+100 ≠ (n4)
log n = o(na)
for any positive constant a
na = o(cn)
for any positive constants a,c
log (4n) = log n + log 4 = O(log n)
log (n4)= 4 log n = O(log n)
(4n)3 = 64n3 = O(n3)
(n4)3 = n12 = (n4)
(3n)4 = 81n = (3n)
Logarithms are to the base 2
Lecture 1: Preliminaries
17
Symmetric properties and tight bound
If f(n) = O(g(n)) then g(n) = (f(n)) and vice versa
If f(n) = o(g(n)) then g(n) = (f(n)) and vice versa
Definition:
If f(n) = O(g(n)) and f(n) = (g(n)) then
f(n) = (g(n)) and g(n) = (f(n))
Example:
9n2 + n +7 = (3n2) = (n2)
Lecture 1: Preliminaries
18
Transitive properties (order)
If f(n) = O(g(n)) and g(n) = O(h(n)) then
f(n) = O(h(n))
If f(n) = (g(n)) and g(n) = (h(n)) then
f(n) = (h(n))
If f(n) = (g(n)) and g(n) = (h(n)) then
f(n) = (h(n))
Lecture 1: Preliminaries
19
Sum and maximum
f1(n) + … + fa(n) = (max(f1(n), … , fa(n)))
for any constant positive integer a
Example:
• 3n2 + n +7 = (3n2) = (n2)
If the range of the summation index is not
constant:
• in i = n(n+1)/2 = (n2) 
(max{1, … , n}) = (n)
Lecture 1: Preliminaries
20
Logarithmic time O(log n)
• Reduce input to any fraction of it in
constant time
• Example: searching if a given element x is
in a sorted array
Lecture 1: Preliminaries
21
Linear time O(n)
• It is sufficient to look at each element of the
input constant number of times
• Example: finding maximum in an (unsorted)
array
Lecture 1: Preliminaries
22
Time O(n log n)
• Split the input into two pieces of the similar
size and merge both solutions in linear time
• Example: sorting an array (split into halves,
sort each part separately, merge sorted parts
in linear time)
Lecture 1: Preliminaries
23
Quadratic time
2
O(n )
• It is enough to consider pairs of elements
• Examples:
– finding the closest pair of points located in a
plane
– sorting by comparison of subsequent elements
(insertion sort)
Lecture 1: Preliminaries
24
Polynomial time
a
O(n )
• Algorithm has many nested loops
• Example: are there any two disjoint sets
among given family of n sets, each of at
most n elements (time O(n3))
Lecture 1: Preliminaries
25
Exponential time
n
O(c )
• Algorithm considers many subsets of the
input
• Example: exhaustive search to find the
largest clique contained in a given graph
(time O(2n))
Lecture 1: Preliminaries
26
Beyond exponential time:
O(n!), O(nn), …
• Algorithm searching in a large space
• Example: search performed in the set of all
permutations (time O(n!))
Lecture 1: Preliminaries
27
Graphs
• Set of nodes |V| = n
• Set of edges |E| = m
– Undirected edges/graph: pairs of nodes {v,w}
– Directed edges/graph: pairs of nodes (v,w)
• Set of neighbors of v : set of nodes connected by an edge
with v (directed: in-neighbors, out-neighbors)
• Degree of a node: number of its neighbors
• Path: a sequence of (non-repeating) nodes such that every
two consecutive nodes constitute an edge
• Length of a path: number of nodes minus 1
• Distance between two nodes: the length of the shortest
path between these nodes
• Diameter: the longest distance in the graph
Lecture 1: Preliminaries
28
Lines, cycles, trees, cliques
Line
Cycle
Tree
Clique
Lecture 1: Preliminaries
29
Conclusions
• Algorithm: list/structure of instructions
• Algorithmic methods for problems arising
from computer and communication
applications
• Guarantee correctness, termination and
efficiency
– Model is important!
• Asymptotic analysis of efficiency
Lecture 1: Preliminaries
30
Textbook and exercises
READING:
• Chapter 2, up to section 2.4
OBLIGATORY EXERCISES:
• Exercises 1,2,3 page 67
• Solved exercises 1,2 pages 65,66
OPTIONAL:
• Exercises 3,4,5,6 pages 67,68,69
• Exercises 7,8 pages 69,70
Lecture 1: Preliminaries
31