2.2 Comparison of Algorithms
Suppose we have two algorithms to do the same thing. Let n be the amount of data that
the algorithms are working on. For example, these might be two algorithms that sort the
elements in an array and n is the number of items to be sorted.
For each algorithm we count the number of statements, Sn and Tn, executed as a function
of n. Thus
Algorithm 1:
Algorithm 2:
Example 1. Suppose
Sn = 10n
Tn = 20n
Sn = number of statements executed
Tn = number of statements executed
80
Tn = 20n
60
If possible, we would like to draw
40
some conclusion from Sn and Tn about
which algorithm might be faster.
20
Sometimes we can do this and
Sn = 10n
sometimes we can't. The problem is
that different statements take different
1
2
3
amounts of time to execute. Suppose
in Example 1, the statements in algorithm 1 take 1 microsecond to execute and the
statements in algorithm 2 take ¼ microsecond to execute. Then the times for the two
algorithms would be.
4
Time1 = (10n statements)(1 sec/statement) = 10n sec
Time2 = (20n statements)(¼ sec/statement) = 5n sec
In this case even though Sn < Tn for all n we would have Time2 < Time1 for all n. So in
Example 1, we can't draw a conclusion which algorithm is faster without some
information about the relative execution times of the two statements.
Example 2. Suppose
Sn = 10n
Tn = n2
Things are different from Example 1. Suppose for example, the statements in algorithm 1
take 1 microsecond to execute and the statements in algorithm 2 take 1/100 microsecond
to execute. Then the times for the two algorithms would be.
Time1 = (10n statements)(1 sec/statement) = 10n sec
Time2 = (n2 statements)(1/100 sec/statement) = n2/100 sec
2.2 - 1
In this case Time1 < Time2 if n > 1000.
In a similar fashion one can see that if
the statements in algorithm 1 take an
amount of time t to execute and the
statements in algorithm 2 take ct, then
Time1 = 10tn
Time2 = ctn2
40 000
30 000
Time2 =
n2
100
20 000
10 000
Time1 = 10n
and Time1 < Time2 if n > 10/c. No
500
1000
1500
2000
matter how small c is, algorithm 1 is
faster than algorithm 2 for large amounts of data. However, how large the amount of data
must be for algorithm 1 to be facter than algorithm 2 depends on c.
We want to generalize Example 2. Suppose we haven't been given the formulas for Sn
and Tn yet. In this case if the statements in algorithm 1 take an amount of time t to
execute and the statements in algorithm 2 take ct, then
Time1 = tSn
Time2 = ctTn
S
We will have Time1 < Time2 for large n if tSn < ctTn for large n, i.e. if Tnn < c for large n.
S
Thus algorithm 1 will be faster for large n if for any small number c one has Tn < c for
n
S
large n. It turns out that this occurs if lim Tnn = 0. This corresponds to the following
n
defintion.
Definition 1. Suppose Sn and Tn are two formulas. We say Sn is "little o" of Tn and write
S
Sn = o(Tn) if lim Tnn = 0.
n
Thus Sn = o(Tn) means that no matter what
small constant c you multiply Tn by you
will still have cTn > Sn if n is sufficiently
large. See the graph at the right.
c Tn
Sn
Intuitively, Sn = o(Tn) means Sn is much
smaller than Tn if n is sufficiently large. It
S
turns out that the the condition lim Tnn = 0
n
is a relatively simple way to compare the two formulas Sn and Tn to see if one algorithm
will be faster for large amounts of data.
2.2 - 2
Example 3. Suppose, as in Example 2, one has Sn = 10n and Tn = n2. Then
S
10n
lim Tnn = lim n2
n
n
10
= lim n = 0
n
So 10n = o(n2) we conclude that algorithm 1 will be faster than algorithm 2 for large n.
This was the same conclusion we reached in Example 2.
Example 4. Suppose, as in Example 1, one has Sn = 10n and Tn = 20n. Then
S
10n
1
1
lim Tnn = lim 20n = lim 2 = 2
n
n
n
S
Since lim Tnn  0 we have 10n  o(20n) and can't conclude that algorithm 1 will be faster
n
than algorithm 2 for large n. We would have to have more information on the relative
execution times of the statements in the two algorithms before we could reach a
conclusion as to which was faster for a particular n.
Example 5. Suppose one has Sn = 4n3 and Tn = 3n2. Then
4n3
S
4n
lim n = lim 3n2 = lim 3 = 
n   Tn
n
n
S
Since lim Tnn  0 we have 4n3  o(3n2) and can't conclude that algorithm 1 will be faster
n
than algorithm 2 for large n. However, if we reverse the roles of 4n3 and 3n2 and
consider
3n2
T
3
lim n = lim 4n3 = lim 4n = 0
n   Sn
n
n
So 3n2 = o(4n3) we conclude that algorithm 2 will be faster than algorithm 1 for large n.
This illustrates the following general principle.
S
T
Proposition 1. If lim Tnn =  then lim Snn = 0 and Tn = o(Sn).
n
n
S
S
Note that lim Tnn =  means that for any large number C one has Tnn > C if n is large
n
enough.
L'Hospital's rule is often useful for evaluating the limits that arise when one is comparing
two algorithms.
2.2 - 3
Example 6. Suppose Sn = 10n2 + 7n + 4 and Tn = 5n3 + 6n + 8. Is either one o of the
other.
S
lim Tnn
n
=
=
=
d
(10n2 + 7n + 4)
dn
10n2 + 7n + 4 using l'Hospital's rule
lim
=
lim
3
n   5n + 6n + 8
n   d (5n3 + 6n + 8)
dn
d
(20n + 7)
dn
20n + 7 using l'Hospital's rule
lim 15n2 + 6
=
lim d
n
n
(15n2 + 6)
dn
20
lim
= 0
n   30n
So 10n2 + 7n + 4 = o(5n3 + 6n + 8).
When working with o it is useful to have a Table of Growth. In a Table of Growth, each
expression is o of the ones above it. Usually the Table of Growth only includes simple
formulas. Here is a typical Table of Growth.
Table of Growth
n!
an
bn
2n
nk
nj
n3
n2
n3/2
n log2n
n
n = n1/2
log2n
1
fastest
assuming a > b > 2
assuming k > j > 3
slowest
In order to verify that each one is o of the ones above it one can either show directly that
the limit of the ratio of the lower one to the higher on is zero or one can use l'Hospital's
rule. Here are some examples.
2.2 - 4
Example 7. Show that n = o(n).
n
lim n
= lim
n
n
1
n
= 0
Example 8. Show that n = o(en).
d
(n)
dn
n using l'Hospital's rule
lim en
=
lim d
n
n
(en)
dn
1
= lim en = 0
n
Example 9. Show that n2 = o(en).
n2
lim en
n
=
d 2
(n )
dn
lim d
n
(en)
dn
2n
n
using Example 8
= lim en = 2 lim en
n
n
=
(2)(0) = 0
Example 10. Show that n = o(2n).
d
(n)
dn
n using l'Hospital's rule
lim 2n
=
lim d
n
n
(2n)
dn
1
= lim (ln 2) 2n = 0
n
d
Here we have used the fact that dn (an) = (ln a) an if a is a constant.
Example 11. Show that ln n = o(n).
ln n
using l'Hospital's rule
lim n
=
n
d
(ln n)
dn
lim d
n
(n)
dn
=
1
n
lim
n 1
1
= lim n = 0
n
Example 12. Show that log2n = o(n).
ln n
Recall that log2n = ln 2. So
log2n
lim
n n
=
ln n
ln 2
lim
n n
1
ln n using Example 11
= ln 2 lim n
n
2.2 - 5
=
 1 (0) = 0
ln 2
The above table illustrates that o is transitive, i.e. one has the following
Proposition 2. If Sn = o(Tn) and Tn = o(Un) then Sn = o(Un).
S
S T
S
T
Proof. lim Unn = lim Tnn Unn = lim Tnn lim Unn = (0)(0) = 0
n
n
n
n
2.2 - 6