COT4400–Design and Analysis of Algorithms: Course Information and Introduction Asai Asaithambi Spring 2023 General Information Instructor: Asai Asaithambi, Ph. D. Professor, School of Computing Class Meetings: Tue, Thu: 03:05 pm – 04:20 pm Office Hours: Tue, Thu: 10:45 am – 12:00 pm Tue, Thu: 01:45 pm – 03:00 pm Email: asai.asaithambi@unf.edu Textbook: No textbook required Material provided on Canvas What is an Algorithm? Why Study Algorithms? • What is an algorithm? ◦ It is a sequence of well-defined steps to solve a problem. ◦ Named after the Persian mathematician Al-Khwarismi • Why study algorithms? – Our normal daily activities are examples of algorithms ◦ brushing teeth; working out; driving to work; ... – Humans can carry out not-so-well-defined steps, but computers cannot – An algorithm for solving a problem on the computer is characterized by the amount of time and space needed – To come up with well-defined steps is called design – To determine the time/space needed is called analysis – Studying and analyzing algorithms will help design efficient algorithms Application of Algorithms • Subject Areas – Artificial Intelligence; Machine Learning – Robotics; Game Development; Animation – Big data; Cybersecurity; Networks – Biology; Chemistry; Physics – Compilers; Databases; Hardware Design – Anything else that needs computations • Commonly Used Modern Apps – Global Positioning Systems (GPS) – Recommender Systems ◦ product recommendation on e-commerce websites ◦ movie/music recommendation on streaming services – Search engines – Speech to text translation – Face/object detection in camera apps Course Overview • What will I learn in this course? ◦ Methods for mathematically analyzing a given algorithm ◦ Methods for designing algorithms for a variety of problems • What background knowledge is assumed? ◦ Programming (although you will not code anything) ◦ Basic data structures (from COP3530) ◦ Discrete mathematics (from COT3100) • Do I need any other special skills? ◦ Willingness to trace algorithms by hand ◦ Patience with algebraic manipulations How to Succeed in this Course • Make use of all the resources provided by the instructor Instructor’s Office Hours Instructor’s Lecture Notes and Slides • Study everything as the course builds on basic concepts introduced earlier and increases in difficulty as the course progresses • Trace through algorithms by hand • Attend class regularly • Do your homework • Do not memorize problems/solutions • Listen carefully in class • Write down anything for which you need clarification • Make sure to use correct notation • Ask questions · · · Ask questions · · · Ask questions Are There Other Resources Available? The following are free resources available on the Internet: • If you want to review data structures: ◦ https://opendatastructures.org/ • if you want to review discrete mathematics: ◦ https://courses.csail.mit.edu/6.042/spring18/mcs.pdf • if you want a textbook on Algorithms: ◦ http://jeffe.cs.illinois.edu/teaching/algorithms/ • Note of caution: Most books on Algorithms tend to be ◦ Aimed at the researcher or practitioner ◦ Tend to be too terse, thick, or advanced Learning Objectives • Describe fundamental concepts in the design and analysis of computer algorithms; • Describe some modern algorithms for solving standard computing problems; • Analyze some commonly used computer algorithms; • Decide on the suitability of a specific algorithm design technique for a given problem; • Apply algorithm design and analysis techniques learned to solve problems; and • Describe complexity classes. Topics List This course is aimed at introducing some fundamental techniques for the analysis and design of algorithms. The following topics will be covered: • Preliminaries • Algorithms with Numbers • Applied Combinatorics • Divide-and-Conquer Algorithms • Graph Algorithms • Dynamic Programming Algorithms • Greedy Algorithms • Intractable Problems Describing Algorithms • We will use pseudocode to describe an algorithm • We will use the function notation in programming languages Each algorithm will have a header containing its name followed by the inputs as arguments, and end with a statement that returns the output of the algorithm • Each line in the algorithm will be numbered for ease of reference when analyzing the algorithm • An example algorithm: 1: 2: 3: 4: 5: 6: procedure Max(A, n) max ← A[1] for i ← 2 to n do if A[i] > max then max← A[i] return max Why Analyze Algorithms? We analyze algorithms to answer the following questions: • Is my algorithm going to run forever? ◦ time complexity • How much computer memory will my algorithm need? ◦ space complexity • Since the running time of an algorithm grows with the input size, What happens if the ”size” of the problem is doubled? quadrupled? • Can I improve upon my algorithm? ◦ Can I solve my problem faster? ◦ Can I solve my problem with less memory? • Do I know that my algorithm will produce the correct result? ◦ correctness • What if it is not possible to determine the exact amount of computing time or memory? • In such instances, could we estimate as close as possible? ◦ tight bound/estimate • Why should the analysis be performed prior to using it in real life? Why Can’t I “Run” the Algorithm to Analyze? We don’t want to ”run” an algorithm to analyze it because • The performance of the algorithm will be: ◦ machine dependent ◦ programming language and compiler dependent ◦ operating system and implementation dependent • There is no guarantee that the algorithm has been tested on all possible inputs • Experimentation can be expensive and time consuming • Comparing several algorithms for the same problem will take even longer and be more expensive Conclusion: Analyze algorithms theoretically Mathematical Preliminaries and Tools • Algebra/Pre-Calculus Review ◦ Functions in Algorithm Analysis ◦ Laws of Exponents ◦ Polynomials ◦ Logarithms ◦ Floor and Ceiling functions ◦ Factorial • Analysis of Simple Loops • Tools for Algorithm Analysis ◦ Summations ◦ Asymptotic Notations ◦ Combinatorics – Counting – Recurrence Relations – Generating Functions What is a Function? Given two sets A, B, a function f from A to B is a rule that associates every element of A to an element of B. f x 1 y 5 z 7 A B We say f maps A to B, and write f : A 7→ B. Also, since x is paired with 1, we write f (x ) = 1. Similarly, f (y ) = 7 and f (z) = 1. Note: A and B can be infinite sets. Functions (cont.) • We are particularly interested in functions f : N 7→ R • N is the set of natural numbers {0, 1, 2, · · · } • R is the set of real numbers • The rules for f are given as formulas, e.g., f (n) = 2n + 7 f 1 49 17 41 21 9 N R Some Useful Functions Polynomial: A polynomial p of degree d (where d is a nonnegative integer) is the function defined by the rule p(n) = a0 + a1 n + a2 n2 + · · · + ad nd . Example: p(n) = 1 + 2n + 3n2 Floor: For any real number x , we define the floor of x as ⌊x ⌋ = the greatest integer ≤ x . Example: ⌊3.2⌋ = 3, ⌊3.8⌋ = 3 Ceiling: For any real number x , we define the ceiling of x as ⌈x ⌉ = the smallest integer ≥ x . Example: ⌈2.2⌉ = 3, ⌈2.8⌉ = 3 Factorial: For any integer n ≥ 0, we define the factorial of n [denoted by n!] as n! = 1, n · (n − 1)!, if n = 0; otherwise. Example: 5! = 5·4! = 5·4·3! = 5·4·3·2! = 5·4·3·2·1! = 5·4·3·2·1·0! = 120 Things to Remember (Properties of Exponents) Let a, b, p, and q be real numbers. Then: ap · aq = ap+q ap /aq = ap−q (ap )q = apq (ab)p = ap b p (a/b)p = ap /b p (where b ̸= 0) a−p = 1/ap (where a ̸= 0) 1/a−p = ap a0 = 1 √ q ap/q = ap (where a ̸= 0) (where a ̸= 0) (where q ̸= 0) Things to Remember (Properties of Logarithms) If ax = n, then we say that x is the logarithm of n to the base a, and write x = loga n. Also, when the base is the number e ≈ 2.718, we call log2 x as the natural logarithm of x , and write ln x . For f (x ) = ln x , we have f ′ (x ) = 1/x . For all real numbers x , y , z > 0: logz (x · y ) = logz x + logz y logz (x /y ) = logz x − logz y logz (x y ) = y · logz x x = z logz x x logz y = y logz x logz 1 = 0 logz (1/x ) = − logz x logy x logy z 1 logz x = logx z logz x = Analysis of Simple Loops How many times is Hello printed? for (i = 1; i <= n; i++) print "Hello" Represent the loop as a summation: n X 1=n i=1 Hello is printed n times. Consider another example: for (i = 1; i <= n; i++) for (j = 1; j <= n; j++) print "Hello" n X n X i=1 j=1 Hello is printed n2 times. 1 = n2 Analysis of Simple Loops (cont.) How many times is Hello printed? for (i = 1; i <= n; i++) for (j = 1; j <= i; j++) print "Hello" n X i X i=1 j=1 1= n X i=1 i= n(n + 1) 1 1 = n2 + n 2 2 2 Hello is printed (n2 + n)/2 times. Need to work with summations. Some basic ideas related to simplifying sums may be helpful. Evaluating Sums • A function a : N 7→ A is called a sequence, and the value of a(n) for any n ∈ N is denoted by an . Frequently, such a function (sequence) is denoted {an }. • With A = Z or A = R, for the sequence {an }, expressions such as a3 + a4 + · · · + a10 is called a finite series, and this is written in shorthand notation as a summation in the form 10 X ak . k=3 • In the expression 10 X ak : k=3 • The symbol Σ stands for sum; • The index k is an integer that takes on consecutive values from 3 through 10; and • The quantities represented by ak in this range of values of k are summed. An equivalent way in which the same summation can be written is shown below: X 3≤k≤10 ak Evaluating Sums (cont.) • Arithmetic Series: ◦ The sequence a, a + d, a + 2d, · · · is called an arithmetic progression. ◦ a is the first term, and d the common difference ◦ The nth -term is given by a + (n − 1)d for n ≥ 1 ◦ An = a + (a + d) + (a + 2d) + · · · + (a + (n − 1)d) is called an arithmetic series ◦ An = n [2a + (n − 1)d] /2 • Geometric Series: ◦ The sequence a, ar , ar 2 , · · · is called a geometric progression. ◦ a is the first term, and r the common ratio ◦ The nth -term is given by ar n−1 for n ≥ 1 ◦ Gn = a + ar + ar 2 + · · · + ar n−1 is called a geometric series, ◦ Gn = rn − 1 a · r − 1 , if |r | > 1; n 1−r a· , if |r | < 1; 1−r na, if r = 1. Some More Known Summation Results n X k=1 n X 1=n k= k=1 n X k=1 n X k=1 n X k=1 n(n + 1) 2 k2 = n(n + 1)(2n + 1) 6 k3 = n2 (n + 1)2 4 (ak − ak+1 ) = a1 − an+1 Analysis of Simple Loops (cont.) How many times is Hello printed? for (i = 1; i <= n; i = i * 2) print "Hello" ⌊log2 n⌋ X 1≤2k ≤n 1= X 1 = ⌊log2 n⌋ + 1 k=0 Hello is printed ⌊log2 n⌋ + 1 times. Now, mimic the same argument and determine the number of times Hello is printed by the following code: for ( i = n; i > 0; i = ⌊i/2⌋ ) print "Hello" Analysis of Simple Loops (cont.) Assume that n is a power of 2, like n = 2m for some integer m. for ( i = n; i > 0; i = ⌊i/2⌋ ) for (j = 1; j <= i; j = j+1) print "Hello" Note that when n is an integer power of 2, the floor function may be removed in the loop modification. Therefore, in summation notation, the required answer is obtained as X X 1≤i=⌊n/2k ⌋≤n 1≤j≤i X 1= 1≤i=⌊n/2k ⌋≤n i= m X n k=0 2k 1 1 + 2 + ··· + 2 2 1 = n 1 − m+1 1− 2 1 = n 2 − m = 2n − 1. 2 =n 1+ , 1 , 2m 1 , 2 Analysis of Simple Loops (cont.) How many times is Hello printed? for (i = 1; i <= n; i++) for (j = i+20; j <= i+39; j++) print "Hello" n X i+39 X i=1 j=i+20 Hello is printed 20n times. 1= n X i=1 20 = 20n Analyzing a Simple Algorithm Algorithm. Max 1: max ← A[1] 2: for i ← 2 to n do 3: if A[i] > max then 4: max ← A[i] 5: endif 6: end for 7: return max Cost d1 d2 d3 d4 How many times 1 n−1 n−1 ≤n−1 d7 1 Let T (n) denote the time required by this algorithm to find the maximum element in an n-element array. Then: T (n) ≤ d1 + d2 (n − 1) + d3 (n − 1) + d4 (n − 1) + d7 ≤ (d2 + d3 + d4 )n + (d1 + d7 ) − (d2 + d3 + d4 ) ≤ (d2 + d3 + d4 )n + (d1 + d7 ) = K1 n + K2 What we have done is the worst-case analysis. Note that K1 and K2 may vary from machine to machine, or we might say they are hardware dependent. We generally analyze algorithms in a machine independent manner. For this purpose, we use asymptotic analysis. The Big-Oh Notation • We wish to analyze algorithms to determine how their runtime requirements grow as the problem size grows. • We use n to denote the size of the problem. • We use T (n) to denote the running time of an algorithm. • To express how T (n) grows as n grows, several asymptotic notations are used. • The Big-Oh Notation: Provides an asymptotic upper bound • For any function g(n), O(g(n)) is defined as the set of functions f (n) for which there exist positive constants c and n0 satisfying 0 ≤ f (n) ≤ c · g(n) for all n ≥ n0 . • For example, if g(n) = n2 , then O(n2 ) consists of any function f (n) for which there exist positive constants c and n0 satisfying f (n) ≤ c · n2 for all n ≥ n0 . • Thus, by definition O(n2 ) is an infinite set. • We list some members of O(n2 ) below: 10 1000, 257 log2 n, 2n2 , n1.59 , 1010 n2 , 7n2 + 2022n + 1700, · · · Big-Oh Notation Illustration A Big-Oh Example • f (n) = 4n3 + 3n2 + 7n + 10 ∈ O(n3 ) because, for all n ≥ 1, f (n) = 4n3 + 3n2 + 7n + 10 ≤ (4 + 3 + 7 + 10)n3 ≤ 24n3 • • • • In other words, f (n) ≤ c · n3 for all n ≥ n0 , with c = 24 and n0 = 1. Note that we may also say f (n) ∈ O(n4 ) for the same c and n0 because n3 ≤ n4 for all n ≥ 1 as well. In fact, f (n) ∈ O(nk ) for k ≥ 3, but we will say n3 is the tightest bound and nk for k > 3 are all not as tight as n4 . When concluding that f (n) = 4n3 + 3n2 + 7n + 10 as a member of O(n3 ) we have suppressed the constants and lower order terms. Finally, note also that f (n) ̸∈ O(n2 ). Analyzing a Simple Algorithm (cont.) • To avoid worrying about machine-dependent constants, we use a model known as the Random Access Machine (RAM) model.1 • In this model, memory is divided into cells • Each cell can hold a word. • There is no limit on the size of the word. • Any location within the memory can be accessed in constant time. • The cost of every step in an algorithm is assumed to be 1 (instead of constants like d1 , d2 , etc.) • Thus, we often count the number of operations instead of the time required. • This is often called the operation count. • Sometimes we distinguish between different operations that we may count within the same algorithm. 1 not to be confused with Random Access Memory Analyzing a Simple Algorithm (cont.) Algorithm. Max 1: max ← A[1] 2: for i ← 2 to n do 3: if A[i] > max then 4: max ← A[i] 5: endif 6: end for 7: return max Cost d\1 1 d\2 1 d\3 1 d\4 1 How many times 1 n−1 n−1 ≤n−1 d\7 1 1 Let T (n) denote the number of steps carried out by this algorithm to find the maximum element in an n-element array. Then: T (n) ≤ 1 + (n − 1) + (n − 1) + (n − 1) + 1 ≤ 3n − 1 ≤ 3n ∈ O(n) Note that we have translated T (n) ≤ 3n as T (n) ∈ O(n) by letting c = 3 and n0 = 1 in the definition of the Big-Oh notation we saw earlier. Also, T (n) ≤ 3n would then tell us the actual wall-clock time required by this algorithm will not exceed m · 3n for some constant m that depends on the hardware used. Another Big-Oh Example √ We show that log2 n ∈ O( n). We have 1 log2 n 2 = 2 · log2 n1/2 (loga x b = b loga x ) ≤ 2 · n1/2 (log2 x < x for x ≥ 1) log2 n = 2 · Therefore, log2 n ≤ 2 · √ n for n ≥ 1. The definition of Big-Oh holds with c = 2 and n0 = 1. √ Thus, log2 n ∈ O( n). Yet Another Big-Oh Example We show that log2 (n!) ∈ O(n log2 n). We have log2 (n!) = log2 (n · (n − 1) · · · 1) = log2 n + log2 (n − 1) + · · · + log2 1 | {z } n terms ≤ log2 n + log2 n + · · · + log2 n | {z n terms } Therefore, log2 (n!) ≤ n log2 n, which holds for all n ≥ 1. The definition of Big-Oh holds with c = 1 and n0 = 1. Thus, log2 (n!) ∈ O(n log n) Note that the base of the logarithm may be dropped. Growth of Functions in Algorithm Analysis T (n) Class Constant Logarithmic Linear Linearithmic Quadratic Cubic Exponential∗ ∗k > 1 T (n) in O(·) O(1) O(log n) O(n) O(n log n) O(n2 ) O(n3 ) O(k n ) Real Time best/fastest super fast fast Acceptable Slow Slower Bad On a problem of size n = 1 million, when a O(n2 ) algorithm takes roughly 70 hours, a O(n log n) algorithm takes roughly 40 seconds. Suppose the machine being used on both algorithms runs 100 times faster. Then the O(n2 ) algorithm would need roughly 40 minutes whereas the O(n log n) algorithm would need half a second. Conclusion: Algorithm analysis worth the effort! Ω) Notation The Big-Omega (Ω • The Big-Omega (Ω) Notation: Provides an asymptotic lower bound • For any function g(n), Ω(g(n)) is defined as the set of functions f (n) for which there exist positive constants c and n0 satisfying 0 ≤ c · g(n) ≤ f (n) for all n ≥ n0 . • For example, if g(n) = n2 , then Ω(n2 ) consists of any function f (n) for which there exist positive constants c and n0 satisfying 0 ≤ c · n2 ≤ f (n) for all n ≥ n0 . • Thus, by definition Ω(n2 ) is an infinite set. • We list some members of Ω(n2 ) below: 10 2n2 , 17n2.83 , 1010 n2 , 7n3 + 2022n2 + 1700, 9000n4 , 3n , n!, · · · Big-Omega Notation Illustration Analyzing a Simple Algorithm (cont.) Algorithm. Max 1: 2: 3: 4: 5: 6: 7: Cost max ← A[1] for i ← 2 to n do if A[i] > max then max ← A[i] endif end for return max 1 1 1 1 At least How many times 1 n−1 n−1 0 1 1 If T (n) denote the number of steps carried out by this algorithm, then: T (n) ≥ 1 + (n − 1) + (n − 1) + 0 + 1 ≥ 2n ≥ 2n Note that we may now conclude from T (n) ≥ 2n that T (n) ∈ Ω(n) by letting c = 2 and n0 = 1 in the definition of the Big-Omega notation we saw earlier. Θ) Notation The Big-Theta (Θ • The Big-Theta (Θ) Notation: Provides an asymptotic tight bound • For any function g(n), Θ(g(n)) is defined as the set of functions f (n) for which there exist positive constants c1 , c2 , and n0 satisfying 0 ≤ c1 · g(n) ≤ f (n) ≤ c2 · g(n) for all n ≥ n0 . • For example, if g(n) = n2 , then Θ(n2 ) consists of any function f (n) for which there exist positive constants c1 , c2 , and n0 satisfying 0 ≤ c1 · n2 ≤ f (n) ≤ c2 · n2 , for all n ≥ n0 . • Thus, by definition Θ(n2 ) is an infinite set. • We list some members of Θ(n2 ) below: 10 2n2 , 1010 n2 , 7n2 + 2022n + 1700, 9000n2 , · · · Big-Theta Notation Illustration For the Max algorithm, for n ≥ 1, T (n) ≤ 3n and T (n) ≥ 2n. Equivalently, 2n ≤ T (n) ≤ 3n for n ≥ 1. Thus, we conclude that T (n) = Θ(n). For any two functions f (n) and g(n), it is true that f (n) ∈ Θ(g(n)) if an only if f (n) ∈ O(g(n)) and f (n) ∈ Ω(g(n)). Further Insights Into Asymptotic Notation • The Big-Oh notation is helpful in understanding the worst-case complexity. • It will be best if the complexity of an algorithm can be characterized using the Θ notation. However ... • It is not always easy to obtain a tight bound (Θ) on T (n). • An asymptotic upper bound analysis (Big-Oh) works well most of the time. • Remember multiplying constants are dropped when the Big-Oh notation is used. • Asymptotic analysis with the Big-Oh makes sense for comparing performances of different algorithms as the size of the problem increases. • The Big-Oh also provides insights on what happens when the same algorithm is used on a problem with increasing sizes. Insights (cont.) Use a O(n2 ) algorithm to solve a problem of size n and 2n. ◦ Then, since T (n) ≤ k · n2 and T (2n) ≤ k · (2n)2 , we may conclude that T (2n) ≈ 4T (n). ◦ When the size of the problem is doubled, the time requirement for a O(n2 ) algorithm is quadrupled. Use a O(n) algorithm to solve a problem of size n and 2n. ◦ Then, since T (n) ≤ k · n and T (2n) ≤ k · (2n), we may conclude that T (2n) ≈ 2T (n). ◦ When the size of the problem is doubled, the time requirement for a O(n) algorithm is doubled as well. Use a O(log n) algorithm to solve a problem of size n and 2n. ◦ Then, since T (n) ≤ k · log n and T (2n) ≤ k · log(2n), we may conclude that2 T (2n) ≈ T (n) + k. ◦ When the size of the problem is doubled, the time requirement for a O(log n) algorithm just goes up by k, a constant. 2 Assuming the base of the logarithm is 2. More on Asymptotic Analysis If f (n) ∈ O(g(n)), then kf (n) ∈ O(g(n)) for any constant k. If f1 (n) ∈ O(g(n)) and f2 (n) ∈ O(g(n)), then f1 (n) + f2 (n) ∈ O(g(n)). If f1 (n) ∈ O(g(n)) and f2 (n) ∈ O(h(n)), then f1 (n) + f2 (n) ∈ O(g(n) + h(n)). If f1 (n) ∈ O(g(n)) and f2 (n) ∈ O(h(n)), then f1 (n) · f2 (n) ∈ O(g(n) · h(n)). If f (n) ∈ O(g(n)) and g(n) ∈ O(h(n)), then f (n) ∈ O(h(n)). Suppose after theoretical analysis, you conclude that the runtime of an algorithm is O(n2 ), but when you actually run it, it is much faster than O(n2 ). You may wish to look deeper into the analysis and see if a tighter bound such as O(n log n) can be obtained. A reason to study analysis of algorithms. In another scenario, you may have a O(n2 ) algorithm and proved that it is a tight bound. However, a competitor may claim that they have an algorithm that performs better. You may then wish to rethink your algorithm and see if its design can be improved. Another reason to study design and analysis of algorithms. The Little-oh (o) Notation For any function g(n), o(g(n)) is defined as the set of functions f (n) for which there exists a positive constant n0 satisfying 0 ≤ f (n) < c · g(n) for all n ≥ n0 , and any c > 0 We say f (n) ∈ o(g(n)) if lim n→∞ f (n) = 0. g(n) f (n) ∈ O(g(n)) means that f (n) grows no faster than g(n). f (n) ∈ o(g(n)) means that f (n) grows slower than g(n). Note that 1000n ∈ o(n2 ), but 1000n2 ̸∈ o(n2 ). On the other hand, 1000n ∈ O(n2 ), and 1000n2 ∈ O(n2 ). The Little-omega (ω) Notation For any function g(n), ω(g(n)) is defined as the set of functions f (n) for which there exists a positive constant n0 satisfying 0 ≤ c · g(n) < f (n) for all n ≥ n0 , and any c > 0 We say f (n) ∈ ω(g(n)) if lim n→∞ f (n) = ∞. g(n) f (n) ∈ Ω(g(n)) means that f (n) grows no slower than g(n). f (n) ∈ ω(g(n)) means that f (n) grows faster than g(n). Note that 1000n2 ∈ ω(n), but 1000n ̸∈ ω(n). On the other hand, 1000n2 ∈ Ω(n), and 1000n ∈ Ω(n). Problems with Algorithms of Known Time Complexities • Constant or O(1): ◦ Solve a quadratic equation ◦ Given a sequence of numbers in increasing order, find the minimum and maximum elements ◦ Add the first 10 positive odd integers • Logarithmic or O(log n): ◦ Binary search on a sorted array of size n • Linear or O(n): ◦ Find the smallest element of an integer array of size n ◦ Linear search on an (unsorted) array of size n • Linearithmic or O(n log n): ◦ Merge-sort an n-element array of comparable items ◦ The Fast Fourier Transform (FFT) of an n-vector • Quadratic or O(n2 ): ◦ Bubble-sort an n-element array of comparable items ◦ Grade-school multiplication of two n-digit numbers • Cubic or O(n3 ): ◦ Naive matrix multiplication ◦ Gaussian elimination for solving an n × n system of linear equations Exponential vs. Polynomial Complexities Exponential or O(k n ) with k > 1: ◦ List all subsets of an n-element set ◦ Given a Boolean function f of n variables, determine if there is an assignment of truth values to the variables such that f evaluates to TRUE ◦ List all permutations of n objects Polynomial Time Algorithm ◦ An algorithm whose time complexity is O(nα ) for some real number α > 0