CS420 lecture four Recurrence Relations wim bohm, cs.colostate Complexity • Divide a program in basic blocks and count the order of magnitude number of basic block executions – basic block: code fragment that takes constant time to execute – careful: some language constructs may take more than constant time • contains in Java • + in APL / C++ • pattern matching in SNOBOL / Python Recurrence Relations • The complexity of an algorithm can often be expressed in a recurrence relation with the input size as parameter. Eg (merge sort) T(n) = 2T(n/2)+n T(1) = 1 Repeated substitution • Simple recurrence relations (one recurrent term in the rhs) can sometimes be solved using repeated substitution • Two types: Linear and DivCo – Linear F(n) = aF(n-d)+g(n), base: F(1)=v1 – Divco F(n)= aF(n/d)+g(n), base: F(1)=v1 • Two question: – what is the pattern – how often is it applied until we hit the base case Linear Example Hanoi: M(n)=2M(n-1)+1, M(1)=1 M(n) = 2M(n-1)+1 = 2(2M(n-2)+1)+1 = 4M(n-2)+2+1 = 4(2M(n-3)+1)+2+1= 8M(n-3)+4+2+1= ... 2kM(n-k)+2k-1+2k-2+...+2+1= hit base for k = n-1: = 2n-1M(1)+2n-1+2n-2+...+2+1 = 2n-1 DivCo example Merge sort: T(n) = 2T(n/2) + n, T(1)=1 n = 2k T(n)=2(2(T(n/4)+n/2)+n = 4T(n/4) + 2n = 8T(n/8) + 3n ... = 2kT(n/2k)+kn hit base for k = logn = 2kT(n/2k)+kn = n+kn = O(nlogn) Another one: binary search Let n = 2k and f(1)=1 and f(n) = f(n/2)+c f(n)=f(n/2)+c = f(n/4)+2c = f(n/8)+3c = f(n/2k)+kc = hit base for k=lg n: f(1)+lgn.c = O(lg n) DivCo repeated substitution pattern n = bk f(n)= af(n/b)+g(n) = a(af(n/b2)+g(n/b))+g(n) = a2f(n/b2)+ag(n)+g(n) = a2(af(n/b2)+g(n)+ag(n)+g(n) = a3f(n/b3)+a2g(n/b2)+ag(n/b)+g(n) = ... k 1 f ( n ) a f ( n / b ) a g( n / b ) k k i i i 0 n b , so k k 1 f ( n ) a f (1) a g( n / b ) k i i 0 i Linear repeated substitution pattern n = kb+1 f(n)= af(n-b)+g(n) = a(f(n-2b)+g(n-b)+g(n)= a2f(n-2b)+ag(n-b)+g(n) =a2(af(n-3b)+g(n))+ag(nb)+g(n)= a3f(n-3b)+a2g(n-2b)+ag(n-b)+g(n) = ... k 1 f ( n ) a f ( n kb ) a g( n ib ) k i i 0 k 1 f ( n ) a f (1) a g( n ib ) k i i 0 Master Method Cookbook approach to solution, recognizing the form. Cormen et.al.'s method is more complex than Rosen: An = C An/d+knp – An = O(np) if C < dp eg An = 3 An/2+n2 – An = O(nplog(n)) if C = dp eg An = 2An/2+n – An = O(nlogdc) if C > dp eg An = 3 An/2+n Proof based on repeated substitution plus induction Let's prove a simpler version Let f(n)=af(n/b)+c, n=bk, then proof: f (n) O(n log b a f ( n ) O (log n ) ) if a 1 if a 1 k 1 f ( n ) a f (1) k a j c from repeated subst i 0 case k 1 f ( n ) f (1) c 1 a 1: i 0 k lg b f ( n ) f (1) c log n so b n k 1 case a 1 : geometric progression a a 1 k i a 1 i 0 a 1 k f ( n ) a f (1) c k because a a k a 1 log b n n a ( f (1) k log b a c a 1 : f ( n ) C1n ) c a 1 log b a C2 C1a C 2 k App: multiplying two n digit numbers A x B = an-1an-2..a1a0 X bn-1bn-2..b1b0 • high school method: O(n2) • divide and conquer approach A = A110n/2+A2 B = B110n/2+B2 AxB = A1B110n+(A1B2+A2B1)10n/2+ A2B2 A1B2+A2B1 = (A1+A2)(B1+B2)-A1B1-A2B2 so we only need 3 n/2 digit multiplications: A1B2, A2B1 and (A1+A2)(B1+B2) Complexity of DivCo n digit multiply Tn=3Tn/2+O(n) Look up in master table: Complexity of DivCo n digit multiply Tn=3Tn/2+O(n) Look up in master table: Tn=O(nlg3)~O(n1.6) Similar trick for matrix multiply n C i, j A i,k * B k , j C = AxB, k 1 3 simple O(n ) code for i=1 to n for j =1 to n C[i,j]=0 for k = 1 to n C[i,j]+=A[i,k]*B[k,j] Block matrix multiply A1,1 A1,2 B1,1 B1,2 * A2,1 A2,2 = B2,1 B2,2 A1,1 B1,1 + A1,2 B2,1 A1,1 B1,2 + A1,2 B2,2 A2,1 B1,1 + A2,2 B2,1 A2,1 B1,2 + A2,2 B2,2 naive block matmult T(n) = 8T(n/2) + O(n2) master table: naive block matrix multiply T(n) = 8T(n/2) + O(n2) master table: T(n) = O(nlog8)=O(n3) Strassen Strassen shows that we only need 7 block multiplies (plus O(n2) work: matrix adds/subtracts ) see notes T(n) = 7T(n/2) + O(n2) master table: T(n) = O(nlg7)~O(n2.81) Linear homogeneous recurrence relations (source: Rosen chapter 7) • an = c1.an-1 + c2.an-2 + .. + cr.an-r a0 = v0, .. , ar-1 = vr-1 : initial values • Guess individual solutions of form n – give rise to characteristic equation – solve to obtain characteristic roots • General solution – linear combination of roots • Specific solution – find coefficients of general solution using initial values Example: Fibonacci Fn=Fn-1+Fn-2, F0=F1=1 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, ... Exponential growth, guess n and substitute n =n-1 +n-2 2 = +1 characteristic eq: 2 - - 1 = 0 quadratic equation: ax2+bx+c = 0 solution: x1,2=(-b±sqrt(b2-4ac))/2a General Solution • Roots of 2 - - 1 = 0 1,2= (1±sqrt(1+4))/2 = (1±sqrt(5))/2 general solution: linear combination of the roots F n A1 ( 1 5 2 ) A2 ( n 1 5 2 ) n A1 and A2 for specific Solution • Specific solution uses the initial values F0=F1=1 F0 1 : F1 1 : A1 A 2 1 A1 ( 1 5 2 ) A2 ( 1 5 2 ) 1 • two equations two unknowns, solve to find As A1 1 A2 ( 1 2 5 1 5 5 ( 1 ) 5 2 ) Specific Solution Fn 1 ( 1 5 2 1 2 5) n 1 1 ( 1 5 2 1 2 5) n 1 Does it always work? Characteristic equation roots r1 and r2. The general solution: A1r1n+A2r2n Initial values: f0=C0 and f1=C1 specific solution: f0= C0 = A1+A2 A2=C0-A1 f1= C1 = A1r1+A2r2 C1=A1r1+(C0-A1)r2= A1(r1-r2)+C0r2 A1 A1 C 1 C 0 r2 r1 r2 A 2 C 0 A1 C 0 C 1 C 0 r2 r1 r2 C 0 r1 C 1 r1 r2 only works if r1 ≠ r 2 Is it a problem? Not really. Equal roots r come from characteristic equation: (x-r)(x-r)=x2-2rx+r2=0 come from recurrence relation: fn=2rfn-1 - r2fn-2 If our recurrence relation has only positive coefficients we cannot get equal roots. For Analysis of Algorithms we can restrict the recurrences to having only positive coefficients. We ADD the complexity of sub solutions together First order Linear Homogeneous Recurrence Relations Simple form Fn=cFn-1 F0=a Use repeated substitution to solve First order Linear Homogeneous Recurrence Relations Simple form Fn=cFn-1 F0=a Use repeated substitution to solve Fn=cFn-1 =c2Fn-2=c3Fn-3= ... = cnF0=cna 20% yearly interest on $1000 What is the value in year n Vn=1.2Vn-1 V0=1000 20% yearly interest on $1000 What is the value in year n Vn=1.2Vn-1 V0=1000 Vn=(1.2)n.1000 Exercise • Determine the general and specific solution of Fn= Fn-1+2Fn-2 with F0=2 and F1=7 Exercise 2 • Fn= Fn-1+2Fn-2 with F0=2 and F1=7 Characteristic equation? Exercise 3 • Fn= Fn-1+2Fn-2 with F0=2 and F1=7 Characteristic equation: x2-x-2=0 Exercise 4 • Fn= Fn-1+2Fn-2 with F0=2 and F1=7 Characteristic equation: x2-x-2=0 Roots? x1,2=(1±sqrt(1-(-8)))/2 Exercise 5 • Fn= Fn-1+2Fn-2 with F0=2 and F1=7 Characteristic equation: x2-x-2=0 Roots: r1=2 r2=-1 Exercise 6 • Fn= Fn-1+2Fn-2 with F0=2 and F1=7 General Solution? Exercise 7 • Fn= Fn-1+2Fn-2 with F0=2 and F1=7 General Solution: Fn=A12n+A2(-1)n Exercise 8 • Fn= Fn-1+2Fn-2 with F0=2 and F1=7 Specific Solution? Find A1 and A2 using initial values Exercise 9 • Fn= Fn-1+2Fn-2 with F0=2 and F1=7 Specific Solution: F0=2=A1+A2 F1=7=A1.2+A2.(-1) A1=3 A2=-1 Fn=3.2n-(-1)n Exercise 10 • Fn= Fn-1+2Fn-2 with F0=2 and F1=7 Specific Solution: F0=2=A1+A2 F1=7=A1.2+A2.(-1) Fn=3.2n-(-1)n A1=3 A2=-1 DONE??? Exercise 11 • Fn= Fn-1+2Fn-2 with F0=2 and F1=7 Specific Solution: F0=2=A1+A2 F1=7=A1.2+A2.(-1) Fn=3.2n-(-1)n A1=3 A2=-1 NO! Check your answer Exercise 11 • Fn= Fn-1+2Fn-2 with F0=2 and F1=7 Specific Solution: F0=2=A1+A2 F1=7=A1.2+A2.(-1) Fn=3.2n-(-1)n A1=3 A2=-1 Check F0 F1 F2 F3 Linear INhomogeneous recurrences Find the homogeneous solution Suppose pn is a particular solution Then the general solution is constant*homogeneous solution + particular solution Linear Inhomogeneous Simple case an = c1.an-1 + f(n) happens most often – Very simple, special case: c1 = 1 • – an = an-1 + f(n) = ... = a0 + f(1) +..+ f(n) General case: • • Solve homogeneous part: hn = A.c1n if *n is a particular solution then an = A.c1n + *n is the general solution • Particular solution “mimics” f(n) f(n): c, d.n , d.n2 , dn *n : b, a.n + b, a.n2+b.n+c, b.dn or b.n.dn (dc1) or (d=c1) Some particular solutions for An = C An-1 + f(n) F(n) Particular(n) a (constant) b (constant) an bn+c a n2 b n2 + c n + d an b an linear → linear, quadratic → quadratic, etc. Let’s try it on Mn = 2 Mn-1 +1, M1 = 1 Example: Hanoi Mn=2Mn-1+1 M1=1 Homogeneous solution: A.2n Particular solution: B B obeys the recurrence relation so B=2B+1 -> B=-1 General solution: Mn=A.2n-1 M1=1 so A.2-1=1 -> A=1 Specific solution: Mn=2n-1 Another example An=2An-1+n A0=0 A1=1 A2=4 A3=11 ... 1. homogeneous solution? Another example 1 An=2An-1+n A0=0 A1=1 A2=4 A3=11 A4=26 1. homogeneous solution: An=2An-1 An=2n Another example 2 An=2An-1+n A0=0 A1=1 A2=4 A3=11 A4=26 1. homogeneous solution: An=2An-1 An=2n 2. particular solution? Another example 3 An=2An-1+n A0=0 A1=1 A2=4 A3=11 A4=26 1. homogeneous solution: An=2An-1 An=2n 2. particular solution .... has form a.n+b and fits the recurrence Another example 4 An=2An-1+n A0=0 A1=1 A2=4 A3=11 A4=26 2. particular solution has form a.n+b and fits the recurrence a.n+b=2(a.(n-1)+b)+n = 2a.n-2a+2b+n Another example 5 An=2An-1+n A0=0 A1=1 A2=4 A3=11 A4=26 2. particular solution has form a.n+b and fits the recurrence a.n+b=2(a.(n-1)+b)+n = 2a.n-2a+2b+n 0 = a.n-2a+b+n Another example 6 An=2An-1+n A0=0 A1=1 A2=4 A3=11 A4=26 2. particular solution has form a.n+b and fits the recurrence a.n+b=2(a.(n-1)+b)+n = 2a.n-2a+2b+n 0 = a.n-2a+b+n (a+1).n=2a-b Another example 7 An=2An-1+n A0=0 A1=1 A2=4 A3=11 A4=26 2. particular solution has form a.n+b and fits the recurrence a.n+b=2(a.(n-1)+b)+n = 2a.n-2a+2b+n 0 = a.n-2a+b+n (a+1).n=2a-b n=0: 0=2a-b -> 2a=b n=1: a+1=0 -> a=-1 -> b=-2 particular solution: -n-2 Another example 8 An=2An-1+n A0=0 A1=1 A2=4 A3=11 A4=26 2. particular solution has form a.n+b and fits the recurrence a.n+b=2(a.(n-1)+b)+n = 2a.n-2a+2b+n 0 = a.n-2a+b+n (a+1).n=2a-b n=0: 0=2a-b -> 2a=b n=1: a+1=0 -> a=-1 -> b=-2 particular solution: -n-2 CHECK IT! Another example 9 An=2An-1+n A0=0 A1=1 A2=4 A3=11 A4=26 2. particular solution has form a.n+b and fits the recurrence particular solution: -n-2 CHECKING whether –n-2 fits: An=2An-1+n -n-2 = 2(-(n-1)-2) +n = -2n+2-4+n = -n-2 CHECK Another example 10 An=2An-1+n A0=0 A1=1 A2=4 A3=11 A4=26 2. particular solution has form a.n+b and fits the recurrence particular solution: -n-2 CHECKING whether –n-2 fits: An=2An-1+n -n-2 = 2(-(n-1)-2) +n = -2n+2-4+n = -n-2 CHECK 3. General Solution? Another example 11 An=2An-1+n A0=0 A1=1 A2=4 A3=11 A4=26 2. particular solution has form a.n+b and fits the recurrence particular solution: -n-2 CHECKING whether –n-2 fits: An=2An-1+n -n-2 = 2(-(n-1)-2) +n = -2n+2-4+n = -n-2 CHECK 3. General Solution: constant*homogeneous + particular Another example 12 An=2An-1+n A0=0 A1=1 A2=4 A3=11 A4=26 ... looks exponential 2. particular solution: -n-2 3. general Solution: constant*homogeneous+particular An=C.2n+(-n-2) 4. specific solution? Another example 13 An=2An-1+n A0=0 A1=1 A2=4 A3=11 A4=26 ... looks exponential 2. particular solution: -n-2 3. general Solution: An=C.2n+(-n-2) 4. specific solution: A0=0 -> C-2=0 An=2n+1-n-2 CHECK IT!! -> C=2 Another example 14 An=2An-1+n A0=0 A1=1 A2=4 A3=11 A4=26 ... 2. particular solution: -n-2 3. general Solution: An=C.2n+(-n-2) looks exponential 4. specific solution: A0=0 -> C-2=0 -> C=2 An=2n+1-n-2 CHECKING: A0=2-0-2=0 A1=4-1-2=1 A2=8-2-2=4 A3=16-3-2=11 check Assignment1: Ackermann Ackermann invented a recurrence relation that: • has only increments and decrements, • grows insanely fast, and • makes even more recursive calls when implemented as a recursive function Ackermann int ack (int m, int n) { c = c+1; if (m==0) return(n+1); else if (n==0) return(ack(m-1,1)); else return(ack(m-1,ack(m,n-1))); } c counts the number of calls Ackermann Play with it, run it for m=1,2,3,4 (careful with m=4) Analyse it: 1. What are ack(1,n), ack(2,n), ack(3,n)? 2. How many calls for ack(1,n), ack(2,n), ack(3,n)? Solving Ackermann • Don't infer solution by fitting the answer • Create recurrences and solve them, eg A(1,n)=A(0,A(1,n-1))=A(1,n-1)+1 A(1,0)=A(0,1)=2 Solve it. Then go on to A(2,n), then A(3,n) Solving Ackermann • part deux: derive the number of calls for m=1,2,3 Create recurrences for the number of function calls. They are all inhomogeneous recurrences of the kind we discussed. Solve them. Case m=3 is complex. You can leave the answer in terms of sums another exercise fn=2fn-1+3fn-2 f0=f1=1 characteristic eq? another exercise 1 fn=2fn-1+3fn-2 f0=f1=1 characteristic eq: x2-2x-3=0 solve it another exercise 2 fn=2fn-1+3fn-2 f0=f1=1 characteristic eq: x2-2x-3=0 x 1,2 2 x 1 1 general solution? 4 12 2 x2 3 another exercise 3 fn=2fn-1+3fn-2 f0=f1=1 x1=-1 x2=3 general solution: A13n+A2(-1)n specific solution? another exercise 4 fn=2fn-1+3fn-2 f0=f1=1 general solution: fn = A13n+A2(-1)n specific solution: A1+A2=1 3A1-A2=1 A1=1/2 A2=1/2 another exercise 5 fn=2fn-1+3fn-2 f0=f1=1 general solution: fn = A13n+A2(-1)n specific solution: fn = (1/2)3n+(1/2)(-1)n A1+A2=1 3A1-A2=1 A1=1/2 A2=1/2 CHECK IT another exercise 6 fn=2fn-1+3fn-2 f0=f1=1 general solution: fn = A13n+A2(-1)n specific solution: fn = (1/2)3n+(1/2)(-1)n 1/2+1/2=1 3/2-1/2=1 9/2+1/2=5 27/2-1/2=13 check inhomogeneous fn=2fn-1+3fn-2+5 f0=f1=0 homogeneous solution: hn = A13n+A2(-1)n particular solution? inhomogeneous 1 fn=2fn-1+3fn-2+5 f0=f1=0 homogeneous solution: hn = A13n+A2(-1)n particular solution: B B=2B+3B+5 4B=-5 B=-5/4 CHECK IT inhomogeneous 2 fn=2fn-1+3fn-2+5 f0=f1=0 homogeneous solution: hn = A13n+A2(-1)n particular solution: B B=2B+3B+5 4B=-5 B=-5/4 -5/4=-10/4-15/4+20/4 check inhomogeneous 3 fn=2fn-1+3fn-2+5 f0=f1=0 homogeneous solution: hn = A13n+A2(-1)n particular solution: -5/4 general solution: fn= A13n+A2(-1)n-5/4 specific solution? inhomogeneous 4 fn=2fn-1+3fn-2+5 f0=f1=0 general solution: fn= A13n+A2(-1)n-5/4 specific solution: f0=0= A1+A2-5/4 f1=0= 3A1-A2-5/4 A1=A2=5/8 inhomogeneous 5 fn=2fn-1+3fn-2+5 f0=f1=0 general solution: fn= A13n+A2(-1)n-5/4 specific solution: fn= (5/8)3n+(5/8)(-1)n-5/4 check it inhomogeneous 6 fn=2fn-1+3fn-2+5 f0=f1=0 general solution: fn= A13n+A2(-1)n-5/4 specific solution: fn= (5/8)3n+(5/8)(-1)n-5/4 f0=0= 5/8+5/8-5/4 f1=0= 3(5/8)-5/8-5/4 f2=5= 9(5/8)+5/8-5/4=50/8 – 10/8 = 40/8 f3=15= 27(5/8)-5/8-5/4=120/8 check one more fn=3fn-1+n f0=1 homogeneous hn= A13n particular solution? one more 1 fn=3fn-1+n f0=1 homogeneous hn= A13n particular solution: shape an+b an+b = 3(a(n-1)+b)+n = 3an-3a+3b+n = (3a+1)n-3a+3b 0 = (2a+1)n-3a+2b n=0 3a=2b a=(2/3)b n=1 2a+1 – 3a+2b=0 a=2b+1 (2/3)b=2b+1 b=-3/4 a=(2/3)b= -1/2 one more 2 fn=3fn-1+n f0=1 homogeneous hn= A13n particular solution: Pn=-(1/2)n-3/4 check it! one more 3 fn=3fn-1+n f0=1 homogeneous hn= A13n particular solution: Pn=-(1/2)n-3/4 -(1/2)n-3/4 = 3((-1/2)(n-1)-3/4)+n = -(3/2)n+3/2+9/4+n = -1/2n-3/4 check general solution? one more 4 fn=3fn-1+n f0=1 homogeneous hn= A13n particular solution: Pn=-(1/2)n-3/4 general solution: A3n-(1/2)n-3/4 specific solution? one more 5 fn=3fn-1+n f0=1 homogeneous hn= A13n particular solution: Pn=-(1/2)n-3/4 general solution: A3n-(1/2)n-3/4 specific solution: A-3/4 = 1 A=7/4 fn=(7/4)3n-(1/2)n-3/4 one more 6 fn=3fn-1+n f0=1 specific solution: A-3/4 = 1 A=7/4 fn=(7/4)3n-(1/2)n-3/4 one more 7 fn=3fn-1+n f0=1 specific solution: A-3/4 = 1 A=7/4 fn=(7/4)3n-(1/2)n-3/4 7/4-3/4 = 1 21/4-1/2-3/4=16/4=4 (7/4)9-1-3/4=63/4-7/4=56/4=14 check