CST3011 – Algorithms and Data Structures Lecture II: Algorithm Analysis Objectives At the end of this lesson, students are expected to be able to Use the mathematical framework that more rigorously describe the running time of an algorithm Estimate the time required for a program for a given input size N Apply techniques that drastically reduce the running time of an algorithm Outline Introduction Why Analyse Algorithms? Experimental studies Asymptotic Analysis Model Expressing Algorithms Running Time Calculations Asymptotic Notation Introduction An algorithm is a clearly specified set of simple instructions to be followed to solve a problem Criteria for judging an algorithm are Correctness: testing, verification Efficiency: usage of resources, the main subject of the course Solves the problem but requires a year Solves the problem takes over the entire resources Simplicity: design and implementation Once the algorithm is determined to be correct, we determine the amount of resources required: Algorithm Analysis Algorithm Analysis The running time of an algorithm, is a function of the input Sorting 10000 elements require more time than sorting 10 The exact values of these functions depend on several factors Speed of the host machine Quality of the compiler Quality of the program etc. Analysis can be done by experimental studies Theoretical analysis Experimental Studies Write a program implementing the algorithm Run the program with inputs of varying size Get the actual running time Use System.currentTimeMillis() for instance Plot the results Limitations The necessity to implement the algorithm Results may not be indicative of the running time on other inputs not included in the experiment To compare two algorithms, the same hardware must be used Theoretical Analysis Use a high level description of the algorithm instead of implementation Takes into account all possible inputs Allows evaluation of the speed of the algorithm independent of the hardware/software environment Why Analyse? To discover characteristics for evaluation or comparing In this course, we care most about asymptotic performance How does the algorithm behave as the problem size gets very large? *Running time Other characteristics Memory/storage requirements Bandwidth/power requirements/logic gates/etc. We define two functions Tavg(N) as average running time and Tworst(N) as the worst running time Clearly Tavg(N) ≤ Tworst(N) We focus on the worst case Easier to analyse Provides an upper bound An absolute guarantee, crucial to applications like games, finance and robotics Model ◻ Analysis is performed with respect to a computational model ◻ We will usually use a generic uniprocessor random-access machine (RAM) All memory equally expensive to access No concurrent operations All reasonable instructions take unit time ■ Except, of course, function calls Algorithm description Algorithms can be expressed in one of the following medium, depending on the level of expressiveness that is intended Plain English: most natural but least precise language Pseudocode: programming language which never complains about syntax errors Programming language e.g. C, Java: most precise but difficult Pseudocode is a high-level description of an algorithm More structure than English prose Less detailed than programming language Preferred notation for describing algorithms, hides program design issues There is no single standard for Pseudocode. We, for consistency sake, are going to define our own pseudocode, CSC3011CODE based on Pascal/Basic syntax CSC3011CODE Definition in BNF <algorithm> ::= Algorithm <id> (<arglist>) <body> End. <arg_list>::= <letter> , <arg_list> | <empty> <body> ::= Input : <input_desc> Output: <output desc> <stmt_list> <stmt_list> ::= <var_decl>; <stmt_list> |<assign>; <stmt_list> |<empty> <var_decl>::=<letter> <letters>:<type>; <letters>::=, <letter><letters>|<empty> <assign>::=<letter> := <exp> <exp>::=<letter>+<exp>|<letter>*<exp>|<empty> , (<exp>)etc CST3011CODE Definition Loops For . . Do . . Next statement Repeat . . Until . . Selection If . . Then . . Else statement Case statement Array indexing starts at 0 Indentation represents the block structures N2 superscripts and other mathematical formatting allowed Other operators ← for assignment, ≠ not equal, = comparison etc Example Algorithm sumOfSquares(M, N) Input: integers M and N Output: the sum of M2 and N2 P := M*M; Q:= N*N; Return P + Q; End. 2 2 2 6 operations Primitive operations Basic computation performed by an algorithm Identifiable in pseudocode Largely independent from the programming language Exact definition not important Examples: Evaluating an expression Assigning a value to a variable Indexing into an array *Calling a method Return from a method Rules for counting operators Rule 1 – For loops: The running time of the for loop is at most the running time of the statements inside the for loop (including tests) times the number of iterations. E.g: For i ← 1 to n – 1 Do 1+ n – 1 = n s ← s + A[i] 3(n – 1) = 3n – 3, Next n - 1 total 5n – 4 ops Rule 2 – Nested loops: Analyse these inside out. The total running time of a statement inside a group of nested loops is the running time of the statement multiplied by the product of the sizes of all the loops: E.g. For i ← 0 to n – 1 1+n For j ← 0 to m – 1 (1 + m)(n) A[i][j] ← i * j 4 (m)(n) Next (m)(n) Next (n) Rules for counting operators cont’d Rule 3: Consecutive statements These just add Rule 4: If . . Then . . Else Consider If <cond> Then <S1> Else <S2> The running time of this statement is never more than the running time of the test plus the larger of the running times of S1 and s2: E.g. Algorithm f(n) If n ≤ 1 Then Return 1 Else Return n * f(n-1) End. Running time is O(n) (Recurrence problem) Counting Primitive Operations Inspect the Pseudocode to determine the maximum number of primitive operations executed by an algorithm, as a function of the input size Example: # of operations Algorithm arrayMax(A, n) Input: A, an array of size n Output: x, the largest value in A current Max ← A[0] 2 For i ← 1 to n – 1 do n-1+1 = n If A[i]> currentMax then 2(n – 1) currentMax ← A[i] 2(n – 1) End For Return currentMax 1 End. Total operations = 5n-1 Estimating Running Time Algorithm arrayMax executes 5n-1 primitive operations in the worst case scenario Define: T(N) worst case time function dependant on N the input A, Time taken by the fastest operation B ,Time taken by the slowest operation We have A(6N – 3) ≤T(N)≤B(6N – 3) Therefore, T(N) is bound by two linear functions Growth Rate of Running Time Changing the hardware/software environment Affects T(N) by a constant factor Does not alter the growth rate of T(N) The Linear growth rate of running time T(N) is a intrinsic property of the algorithm arrayMax Growth Rates Popular growth rates of functions in order: Logarithmic Linear Quadratic Cubic Others include LogLogN Nlog(N) Exponential Asymptotic Notation The analysis required to estimate resource use is a theoretical issue We use the asymptotic (big-O) notation: What does O(n) running time mean? O(n2)? O(n lg n)? Our first task is to define this notation more formally and completely Asymptotic Notation We will use the following four definitions T(N) = Ο (f(N)) if ∃ c,n0 ∈ Ζ+, : T(N) ≤ cf(N) ∀ N ≥ n0 T(N) = Ω (f(N)) if ∃ c,n0 ∈ Ζ+, : T(N) ≥ cf(N) ∀ N ≥ n0 T(N) = Θ (f(N)) ⇔T(N) = Ο (f(N)) & T(N) = Ω (f(N)) T(N) = ο (f(N)) if ∃ c,n0 ∈ Ζ+, : T(N) < cf(N) ∀ N ≥ n0 These definitions establish a relation order among functions: i.e. the relative rates of growth of functions Example 1000N = Ο(N2) i.e. N2 grows more rapidly than 1000N and will eventually be larger than 1000N. If g(N) = 100N2 + 20N + 200 then g(N) = Ο(N2) = Ο(N3) = Ο(N4) 2N + 10 = Ο(N) Since: 2N + 10 ≤ cN (c – 2)N ≥ 10 N ≥ 10/(c – 2) ∴for c = 3 and n0 = 10, it is true Exercise: Show that g(N) = Ο (f(N)) ⇔ f(N) = Ω (g(N)) Big-Oh Rules and Growth Rates The Big-Oh notation gives a upper bound on the growth rate of the function The statement “f(n) is O(g(n))”, simply means that the growth rate of f(n) is no more than the growth rate of g(n) We can use the Big-Oh notation to rank functions according to their growth rate Big-Oh Rules Rule 1: If T1(n) = O(f(n)) and T2(N) = O(g(n)), then T1(n) + T2(n) = max(O(f(n), O(g(n)) Rule II If f(n) is a polynomial of order k, then f(n) is Θ(nk) i.e. Rule III Logk(n) = O(n) for any constant k Rule of Thumb: Drop the lower-order terms Drop constant factors Say 2n + 4 is O(n) not (O(2n) or O(2n + 4) Use the smallest possible class of functions Say “2n is O(n)” instead of “2n is O(n2) Asymptotic Algorithm Analysis The Asymptotic analysis of an algorithm determines the running time in Big-Oh notation To perform the Asymptotic analysis: We find the worst-case number of primitive operations executed as a function of the input size We express this function with the Big-Oh notation Example: We determine that arrayMax executes at most n – 1 operations We say arrayMax runs O(n) time Since the constant factors and lower-order terms are eventually dropped anyway, we can disregard them when counting primitive operations – but don’t!!! Example: Prefix averages We illustrate asymptotic analysis with two algorithms for prefix averages The i-th prefix average is the average of the array X is the average of the first i elements of X A[i] = [X[0]+X[1]+ . . +X[i]]/(i+1) Computing the array A of prefix averages of another array X has applications to financial analysis Solution I Pseudo code Algorithm prefixAverages(A, n) Input: an array A of size n Output: X, an array of prefix averages of X X ← new array of size n n For i ← 0 to n – 1 do 1 + n S ← A[0] 2n For j = 1 to i do 1 + 2 + .. + (n-1) S ← S + A[j] 3(1 + 2 + .. + (n-1)) End For X[i] ← S/(i + 1) 4n End For Return X 1 End. Solution I: Digest Arithmetic Progression Quadratic order Solution II Pseudocode Algorithm prefixAverages2(A, n) Input Output X ← new array of size n s←0 For i ← 0 to n – 1 do s ← s + A[i] X[i] ← s/(i + 1) Next Return A End. Growth rate by limit To determine the growth rate of two function f(N) and g(N), compute This limit has four possible values Limit = 0 ⇒ f(N) = ο(g(N) Limit = c ≠ 0 ⇒ f(N) = Θ(g(N)) Limit = ∞ ⇒ g(N) = ο(f(N)) Limit oscillates: There is no relation L’Hôpitals Rule In determining the limit of a quotient of f(N) and g(N) as n tends to infinity, you can use the L’Hôpitals Rule, which states Exercise: Show that log(N) = o(N), using the L’Hôpitals Rule. Example Show that log(n) = o(N) Soln: ??QUESTIONS??
