Part one
Introduction
Algorithm
A finite set of rules which give a sequence of operations
for solving a specific type of problem.
It has the following features:
1. Finiteness: An algorithm must always terminate
after a finite number of steps.
2. Definiteness: Each step of an algorithm must be
precisely defined; the actions to be carried out
must be rigorously and unambiguously specified
for each case.
3. Input: An algorithm has zero or more inputs
(externally supplied) i.e. quantities which are given
to it initially before the algorithm begins.
4. Output: An algorithm has one or more outputs
produced. i.e. quantities which have a specified
relation to the inputs.
5. Effectiveness: i.e. the operations to be performed
in the algorithm must be sufficiently basic that
they can in principle be done exactly and in a finite
length of time by a man using pencil and paper.
Analysis of algorithms
Is the name used to describe investigations such as this:
"The general idea is to take a particular algorithm and
to determine its average behavior, and whether optimal
or not".
Theory of algorithms:
Is dealing primarily with the existence or nonexistence
of effective algorithms to compute particular quantities.
1 of 19
Part one
Introduction
Calculating The Running Time of a program
Suppose we have the following segment of a program:
A simple assignment statement to an integer variable.
a=b
Since the assignment statement takes constant time, so its
complexity is O(1).
Consider a simple for loop
sum  0
for i  1 to n
sum  sum  n
The cost of the entire code fragment (complexity) is
n
O(c1   c 2 )  O(n)
i 1
Consider a code fragment with several for loops, some of which
are nested
sum  0
for j 1to n
for i 1to j
sum  sum  1
for k  0 to n
a(k )  k
2 of 19
Part one
Introduction
The total cost is
n
j
n
 O(c1   c2   c3 )
j 1 i 1
 O( n 2 )
k 0
Compare the asymptotic analysis for the following two code
fragments:
1)
sum  0
for i 1to n
for j  1to n
sum  sum  1
2)
sum  0
for i 1to n
for j  1to i
sum  sum  1
2
Thus, both double loops cost O(n ) , although the 2nd requires
about ½ the time of the 1st.
Note:
2
Not all the doubly nested loops are O(n ) . Consider the
following nested loops:
3 of 19
Part one
Introduction
sum  0
k 1
while k  n
{
for j  1to n
sum  sum  1
k  k 2
}
Thus the time is
O (c1  c2 
lg n 1
n
 ( c
k 1
j 1
3
 c4 ))
 O (c1  c2  c3 ( n  lg n  1))
 O ( n  lg n)
4 of 19
Part one
Introduction
While the following loops:
sum  0
k 1
while k  n
{
for j 1to k
sum  sum  1
k  k 2
}
Gives the following time
Suppose the n is a power of 2 (i.e.
 O(c1  c2 
lg n 1 2 k
 c
k 1 j 1
3
n  2k )
 c4 )
lg n
 O(c1  c2  c3  2 k  c4 )
k 0
 O( 2 lg n 1  2 0 )
 O ( n)
 O( 2n 1)
What about the cost of the selection statements (ex: if statement,
or switch).
The cost of an if statement in the worst case is the greater of the
costs for the then and else clauses.
It is also true for the average case, assuming that the size of (n)
does not affect the probability of executing one of the clauses
(which is usually, but not necessarily true).
5 of 19
Part one
Introduction
The Case (switch) statements. The worst-case cost is that of the
most expensive branch.
For subprograms (subroutine calls): Add the cost of executing
the subroutine.
Determining the execution time of a recursive subroutine can
be particularly difficult.
The running time for a recursive subroutine is typically best
expressed by recurrence relation.
Re cursive factorial function
fact (n)
if n  1then
fact  1
else
fact  n  fact (n  1))
The cost of the factorial function is a constant + the time to
execute the recursive call on the smaller input. The running time
can be expressed as
T (n)  T (n  1)  c
The closed form solution for this recurrence relation is O (n) .
While the recurrence relation for the Fibonacci numbers is
Re cursive Fibonacci function
fib(n)
if (n  1) then
return1
else
return( fib(n  1)  fib(n  2))
6 of 19
Part one
Introduction
The relation is
T (n)  T (n  1)  T (n  2)  1
with
for n  2,
T (0)  0, T (1) 1
n
O
(
2
)
Gives an exponential time
We can compute the problem in a liner time
fib(n)
f [0]  0; f [1]  1
for i  2 to n
f [i ]  f [i  1]  f [i  2]
return f [n]
The formula is
n
 O( c1 )  O(n)
i2
Basic Recurrences
1) This recurrence arises for a recursive algorithm that loops
through the input to eliminate one item.
C n  C n 1  n for n  2
with C1  1
2
n
C n is about
7 of 19
2
Part one
Introduction
2) This recurrence arises for a recursive algorithm that halves
the input in one step.
Cn  C n 1
for n  2
2
with C1  0
The solution of C n is about
lg n
3) This recurrence arises for a recursive algorithm that halves
the input, but must examine every item in the input.
Cn  Cn  n
for n  2
2
with C1  0
C n is about 2n
4) This recurrence arises for a recursive algorithm that has to
make a linear pass through the input, before, during, or
after it is split into two halves.
C n  2C n  n for n  2
2
with C1  0
C n is about n lg n
8 of 19
Part one
Introduction
Note:
This is most widely solution, because it is the prototype of
many standard divide and conquers algorithms.
5) This recurrence arises for a recursive algorithm that splits
the input into two halves with one step using two recursive
calls, each operating on about ½ the input.
C n  2C n  1
for n  2
2
with C1  0
C n is about 2n
Summation Formulas
1) Sum of consecutive integers
n(n  1)
i

2
i 1
n
1  2  3    (n  2)  (n  1)
if n is odd, there are
in the middle, thus
 n
( n  1)
pairs and extra number (n  1)
2
2
(n  1)
(n  1) n
(n  1) 
 (n  1)
2
2
2
If n is even, there are
n
2
pairs, hence the sum is
9 of 19
n
( )( n  1) .
2
Part one
Introduction
3
2
2
n

3
n
n
2
i 

6
i 1
n
2)
k
2
3) Power of 2
i
 2 k 1  1
i 0
2 i as a 1-bit in binary number
Think of each term
k
i
2
 1111 , There are (k+1) 1s.
i 0
If we add (1) to this number we have
1000  2 k 1
k 1
2
1
Subtract the added (1) we have
k
1
1
2 k
4) Geometric sums  2 i
2
i 0
k
5)
 i2
i
 (k  1)2 k 1  2
i 1
Mathematical Induction
Induction plays a major role in algorithm design. The
mathematical induction is a very powerful proof technique.
It works as follows:
10 of 19
Part one
Introduction
Let (T) be a theorem that we want to prove. Suppose
that (T) includes a parameter (n) whose value can be
any natural number (positive integer).
Instead of proving (T) holds for all values of (n), we
prove the following two conditions:
1) (T) holds for n=1
2) For every n>1, if (T) holds for (n-1), then (T) holds
for (n).
Consider the following series
p(n)  1  3  5  7  ...  (2n  1)  n 2
n
( (2i  1)  n 2 )
i 1
We wish to prove that p(n) is true for all positive (n).
1) p(1) is true since 1=12
2) if all of p(1), p(2),…, p(n) are true, then we want to
prove p(n+1) is true.
3) Add (2n+1) to both sides
1  3  5  ...  (2n  1)  (2n  1)  n2  2n  1
 (n  1) 2
Which proves that p(n+1) is true also.
i.e. p(n+1)=(n+1)2
11 of 19
Part one
Introduction
Theorem
The sum of the first n natural numbers is n(n+1)/2
S(n)=1+2+…+n = n(n+1)/2
Proof: the proof by induction on n.
1) For n=1, the claim is true s (1)=1=1. (1+1)/2
2) Assume the sum of the 1st n natural numbers is
n(n+1)/2
3) We need to prove that the sum of the 1st (n+1) natural
numbers is S(n+1)=(n+1)(n+2)/2
From the definition of S(n), we have
S(n+1)= S(n) +( n+1)
By assumption S(n)=n(n+1)/2 , therefore
S(n+1)= n(n+1)/2 + (n+1)
= (n+2)(n+1)/2
Which what we want to prove.
Home Work
Compute the sum of the following series
T (n)  8  13  18  23    (3  5n)
We define the Fibonacci sequence
f 0 , f1 , f 2 ,
by the rule that f 0  0, f1 1 , and every further
term is the sum of the preceding two terms. Thus the
sequence is 0,1,1, 2, 3, 5, 8,13, 
1 5 


We have to prove that if
is the number  2 


n 1
We have n
for all positive integers n

f 
12 of 19
Part one
Introduction
Proof
0
n 1
f

1




1) If n=1 then 1
is true
1
21
2) P(2) is true since f 2  1  1.6    
3) Assume P(1), P(2), ….. P(n) are true, and n>1. We
have that P(n-1), P(n) are true. So
f n 1   n  2 , f n   n 1
Adding these inequalities we get
f n 1  f n 1  f n   n  2   n 1
  n  2 1   
Since
 2   1
Then we have
f n 1   n
which is P(n+1).
Prove the following inequality
1 1 1
1
     n 1 for all n  1
2 4 8
2
Proof
1
1) For n=1 is true 2  1
2) We assume the theorem is true for n
1
th
3) Consider n  1 term. Adding 2 n 1 to the left
hand side may increase the sum to more than 1
1 1 1
1
1
     n  n 1
2 4 8
2
2
13 of 19
Part one
Introduction
We take the last n terms
1 1
1
1
    n  n 1
4 8
2
2
11 1 1
1  1
     n 
22 4 8
2  2
1
We can add 2 to both sides we get the expression
for (n+1).
Counting Regions in the Plane
Theorem:
The number of regions in the plane formed by n
n ( n  1)
1.
lines in general position is
2
A set of lines in the plane is said to be in general
position if no two lines are parallel and no three
lines intersect at a common points.
We try to compute the number of regions in the plane
formed by n lines in general position
When n=1 there are 2 regions.
When n=2 there are 4 regions.
When n=3 there are 7 regions.
i
th
Thus for i  3 , the
line adds (i) regions.
So, we guess adding one more line to (n-1) lines in
general position in the plane increases the number of
regions by (n).
14 of 19
Part one
Introduction
Proof
th
We have proved that the n line adds (n)
more regions. 1st line introduced 2 regions.
Hence, the total number of regions for n>1 is
2  2  3  4  5   n
n(n  1)
Since 1  2  3  4    n 
2
n(n  1)
Therefore, the total number of regions is 2  1 .
For all natural numbers X and n, Xn-1 is divisible by
X-1
Proof:
1) For n=1, it is true.
2) We assume it is true for (n-1) {induction hypothesis}
i.e. Xn-1 –1 is divisible by X-1 for all natural numbers X.
3) We need to prove that (Xn -1) is divisible by X-1
X n  1  X ( X n1  1)  ( X  1)
The left term is divisible by X-1, by the induction
hypothesis and the right term is just X-1.
If an algorithm is composed of several parts, then its
complexity is the sum of the complexities of its parts.
The algorithm may consist of a loop executed many
times, each time with a different complexity. We
need techniques for summing expressions in order to
analyze such cases.
15 of 19
Part one
Introduction
th
i
Consider a case of executing a loop in which
2
i
step requires
operations:
n
S 2 ( n)   i 2
i 1
3
S
(
n
)

n
It is clear that 2
, we guess that the
3
S
(
n
)
and
n
differences between 2
are with
constant.
We need to prove our guess and find the constants, by
induction. We guess that
S 2 (n)  P(n)  an 3  bn 2  cn  d
P(n) must satisfy P(1) =1, and the induction step
P(n  1)  P(n)  (n  1) 2



a(n  1)3  b(n  1) 2  c(n  1)  d  an3  bn 2  cn  d  n 2  2n  1
Since coefficients of the same power of n must be equal
Then it implies that
3a + b – b = 1
3a +2b + c - c = 2
a+b+c+d–d=1
1
1
1
a

,
b

,
c

, d 0
Imply that:
3
2
6
n3 n 2 n
S 2 ( n)  

3
2 6
n(n  1)( 2n  1)

6
16 of 19
Part one
Introduction
There is another way to arrive at the above expression. It
is a general technique.
If we guess that S 2 (n) is a 3rd degree polynomial, then
we can try to express S 2 (n) as a combination of such
polynomials.
n
S 3 ( n)   i 3
Consider the sum
i 1
, we can write it
as
n
n
S3 (n)   i   i  1  1
3
3
i 1
i 1
n 1
  i  1
3
i 0
n 1


  i 3  3i 2  3i  1
i 0
Equate two sides
n 1
n


3
2
i

i

3
i
 3i  1
 
3
i 1
i 0
i 3 terms for (i) ranging from 1 to n-1 are common to
both sides and can canceled. Then write an equation for
the rest of terms from both sides
n 1


n  0   3i 2  3i  1
3
3
i 0
17 of 19
Part one
Introduction
 3n(n  1) 
n  3 S 2 ( n)  n  
 n
2


Solve for S 2 (n)

3
Hence,
2

 3n(n  1) 
2
n3  
  n  3 S 2 ( n)  n
2




This implies that
 n(n  1) 
n  3
n
 2 
S 2 ( n) 
 n2
3
n 3 3n 2 n n(n  1)( 2n  1)


 
3
6
6
6
3
This is the same expression.
Compute the following:
n
f ( n)   2 i  1  2  4  8    2 n
i 0
The difference between consecutive terms in f (n) is a
factor of 2. Multiply the whole expression by 2.
2 f (n)  2  4  8   2 n  2 n1
We can get expression involving f (n)
2 f (n)  f (n)  2n 1  1
f (n)  2n 1 1
18 of 19
which implies that
Part one
Introduction
Consider the following sum
n
G (n)   i 2i  1 21  2  22  3  23    n  2n
i 1
We can apply the same technique
2G(n)  1  2 2  2  2 3  3  2 4    n  2 n1
Subtract the two expressions, we eliminate the effect of
the (i) factor:
G(n)  2G(n)  G(n)  n  2 n1  (1  21  1  2 2    n  2 n )
 n  2 n1  (2 n1  2)  (n  1)  2 n1  2
Home Work
Compute the following:
n
G ( n)   i  2 n i
i 1
19 of 19