ALGORITHM ANALYSIS
1
ANALYSIS OF ALGORITHMS
• Analysis of Algorithms is the area of computer science that
provides tools to analyze the efficiency of different methods of
solutions.
• How do we compare the time efficiency of two algorithms that
solve the same problem?
2
Running Time
– Easier to analyze
– Crucial to applications such as
games, finance and robotics
best case
average case
worst case
120
100
Running Time
• Most algorithms transform
input objects into output
objects.
• The running time of an
algorithm typically grows with
the input size.
• Average case time is often
difficult to determine.
• We focus on the worst case
running time.
80
60
40
20
0
1000
2000
3000
4000
Input Size
3
Experimental Approach
• Write a program implementing the
9000
8000
algorithm
7000
varying size and composition
• Use a method like clock() to get an
accurate measure of the actual
running time
Time (ms)
• Run the program with inputs of
6000
5000
4000
3000
2000
1000
0
• Plot the results
0
50
100
Input Size
4
Limitations of Experiments
– How are the algorithms coded?
• Comparing running times means comparing the
implementations.
• It is necessary to implement the algorithm, which
may be difficult
• implementations are sensitive to programming style
that may cloud the issue of which algorithm is
inherently more efficient.
– What computer should we use?
• We should compare the efficiency of the algorithms
independently of a particular computer.
5
Limitations of Experiments (cont.)
– What data should the program use?
• Results may not be indicative of the running time
on other inputs not included in the experiment.
• Any analysis must be independent of specific
data.
• As noted above, experimental analysis is valuable, but
it has its limitations. If we wish to analyze a particular
algorithm without performing experiments on its
running time, we can perform an analysis directly on
the high-level pseudo-code instead.
6
Pseudocode
Example: find max
• High-level description of
element of an array
an algorithm
• More structured than
Algorithm arrayMax(A, n)
English prose
Input array A of n integers
• Less detailed than a
Output maximum element of A
program
currentMax  A[0]
• Preferred notation for
for i  1 to n  1 do
describing algorithms
if A[i]  currentMax then
• Hides program design
currentMax  A[i]
issues
return currentMax
7
Theoretical Analysis
• Uses a high-level description of the
algorithm instead of an implementation
• Characterizes running time as a function of
the input size, n.
• Takes into account all possible inputs
• Allows us to evaluate the speed of an
algorithm independent of the
hardware/software environment
8
Theoretical analysis of time efficiency
Time efficiency is analyzed by determining the
number of repetitions of the primitive(basic)
operation as a function of input size
• Basic operation: the operation that contributes
most towards the running time of the algorithm
9
Primitive operations
We define a set of primitive operations as the following:
• Basic computations performed by an algorithm
• Identifiable in pseudo-code
• Largely independent from the programming language
• Exact definition not important
• Assumed to take a constant amount of time in the RAM
model
10
Primitive operations (cont.)
Examples:
• Assigning a value to a variable
• Calling a function
• Performing an arithmetic operation (for example,
adding two numbers)
• Comparing two numbers
• Indexing into an array
• Following an object reference
• Returning from a function
11
Primitive operations (cont.)
• When we analyze algorithms, we should employ
mathematical techniques that analyze algorithms
independently of specific implementations,
computers, or data.
• To analyze algorithms:
– First, we start to count the number of significant
operations (primitive operations) in a particular solution
to assess its efficiency.
– Then, we will express the efficiency of algorithms
using growth functions.
12
The Execution Time of Algorithms
• Each operation in an algorithm (or a program) has a cost.
 Each operation takes a certain amount of time.
count = count + 1; 
take a certain amount of time, but it is constant
A sequence of operations:
count = count + 1;
sum = sum + count;
T1
T2
 Total Time = T1 + T2
13
Counting Primitive Operations
• By inspecting the pseudocode, we can determine the maximum
number of primitive operations executed by an algorithm, as a
function of the input size
Algorithm arrayMax(A, n)
currentMax  A[0]
for i  1 to n  1 do
if A[i]  currentMax then
currentMax  A[i]
{ increment counter i }
return currentMax
Times
2
2n
2(n  1)
2(n  1)
2(n  1)
1
Total
8n  3
14
Counting Primitive Operations
Example: Simple Loop
i = 1;
sum = 0;
while (i <= n) {
i = i + 1;
sum = sum + i;
}
Times
1
1
n+1
2n
2n
Total Time = 1 + 1 + (n+1) + 2n + 2n
 The time required for this algorithm is proportional to n
15
Algorithm Growth Rates
• We measure an algorithm’s time requirement as a function of the
problem size.
– Problem size depends on the application: e.g. number of elements in a list for a
sorting algorithm, the number disks.
• So, for instance, we say that (if the problem size is n)
– Algorithm A requires 5*n2 time units to solve a problem of size n.
– Algorithm B requires 7*n time units to solve a problem of size n.
• The most important thing to learn is how quickly the algorithm’s
time requirement grows as a function of the problem size.
– Algorithm A requires time proportional to n2.
– Algorithm B requires time proportional to n.
• The change in the behavior of an algorithm as the input size
increases is known as growth rate.
• We can compare the efficiency of two algorithms by comparing
their growth rates.
16
Algorithm Growth Rates (cont.)
Time requirements as a function
of the problem size n
17
Common Growth Rates
Function
c
log N
log2N
N
N2
N3
2N
Growth Rate Name
Constant
Logarithmic
Log-squared
Linear
Quadratic
Cubic
Exponential
18
Figure 6.1
Running times for small inputs
19
Asymptotic order of growth
A way of comparing functions that ignores constant
factors and small input sizes
• Big-O: O(g(n)): class of functions f(n) that grow
no faster than g(n)
• Big- theta Θ(g(n)): class of functions f(n) that
grow at same rate as g(n)
• Big-omega: Ω(g(n)): class of functions f(n) that
20
grow at least as fast as g(n)
Order-of-Magnitude Analysis and Big O
Notation
• If Algorithm A requires time proportional to g(n), Algorithm A
is said to be order g(n), and it is denoted as O(g(n)).
• The function g(n) is called the algorithm’s growth-rate
function.
• Since the capital O is used in the notation, this notation is called
the Big O notation.
• If Algorithm A requires time proportional to n2, it is O(n2).
• If Algorithm A requires time proportional to n, it is O(n).
21
Definition of the Order of an Algorithm
Definition:
We say that the function f(n) is O(g(n)) if there is a
real constant c> 0 and an integer constant n0  1
such that :
f(n)< c*g(n) for all n  n0.
• The requirement of n  n0 in the definition formalizes the notion
of sufficiently large problems.
– In general, many values of c and n can satisfy this definition.
22
Order of an Algorithm
• If an algorithm requires n2–3*n+10 seconds to solve a problem
size n. If constants c and n0 exist such that
c*n2 > n2–3*n+10 for all n  n0 .
the algorithm is order n2 (In fact, c is 3 and n0 is 2)
3*n2 > n2–3*n+10 for all n  2 .
Thus, the algorithm requires no more than *n2 time units for n 
n0 ,
So it is O(n2)
23
Big-Oh Examples
7n-2
7n-2 is O(n)
need c > 0 and n0  1 such that 7n-2  c•n for n  n0
this is true for c = 7 and n0 = 1
 3n3 + 20n2 + 5
3n3 + 20n2 + 5 is O(n3)
need c > 0 and n0  1 such that 3n3 + 20n2 + 5  c•n3 for n  n0
this is true for c = 4 and n0 = 21
24
Order of an Algorithm (cont.)
25
A Comparison of Growth-Rate Functions
26
Growth-Rate Functions
• If an algorithm takes 1 second to run with the problem size 8,
what is the time requirement (approximately) for that algorithm
with the problem size 16?
• If its order is:
O(1)
 T(n) = 1 second
O(log2n)  T(n) = (1*log216) / log28 = 4/3 seconds
O(n)
 T(n) = (1*16) / 8 = 2 seconds
O(n*log2n)  T(n) = (1*16*log216) / 8*log28 = 8/3 seconds
O(n2)
 T(n) = (1*162) / 82 = 4 seconds
O(n3)
 T(n) = (1*163) / 83 = 8 seconds
O(2n)
 T(n) = (1*216) / 28 = 28 seconds = 256 seconds
27
Properties of Growth-Rate Functions
1. We can ignore low-order terms in an algorithm’s growth-rate
function.
–
–
If an algorithm is O(n3+4n2+3n), it is also O(n3).
We only use the higher-order term as algorithm’s growth-rate function.
2. We can ignore a multiplicative constant in the higher-order term
of an algorithm’s growth-rate function.
–
If an algorithm is O(5n3), it is also O(n3).
3. O(f(n)) + O(g(n)) = O(f(n)+g(n))
–
–
–
We can combine growth-rate functions.
If an algorithm is O(n3) + O(4n2), it is also O(n3 +4n2)  So, it is O(n3).
Similar rules hold for multiplication.
28
What to Analyze
• An algorithm can require different times to solve different
problems of the same size.
– Eg. Searching an item in a list of n elements using sequential search.  Cost:
1,2,...,n
• Worst-Case Analysis –The maximum amount of time that an
algorithm require to solve a problem of size n.
– This gives an upper bound for the time complexity of an algorithm.
– Normally, we try to find worst-case behavior of an algorithm.
• Best-Case Analysis –The minimum amount of time that an
algorithm require to solve a problem of size n.
– The best case behavior of an algorithm is NOT so useful.
• Average-Case Analysis –The average amount of time that an
algorithm require to solve a problem of size n.
29
What is Important?
• An array-based list retrieve operation is O(1), a linked-listbased list retrieve operation is O(n).
• But insert and delete operations are much easier on a linked-listbased list implementation.
 When selecting the implementation of an Abstract Data Type
(ADT), we have to consider how frequently particular ADT
operations occur in a given application.
• If the problem size is always small, we can probably ignore the
algorithm’s efficiency.
– In this case, we should choose the simplest algorithm.
30