What is a NP problem? Given an instance of the problem, V, and a ‘certificate’, C, we can verify V is in the language in polynomial time All problems in P are NP problems Why? What is NP-Complete? A problem is NP-Complete if: It is in NP Every other NP problem has a polynomial time reduction to this problem NP-Complete problems: 3-SAT VERTEX-COVER CLIQUE HAMILTONIAN-PATH (HAMPATH) Dilemma NP problems need solutions in real-life We only know exponential algorithms What do we do? A Solution There are many important NP-Complete problems There is no fast solution ! But we want the answer … If the input is small use backtrack. Isolate the problem into P-problems ! Find the Near-Optimal solution in polynomial time. 5 Accuracy NP problems are often optimization problems It’s hard to find the EXACT answer Maybe we just want to know our answer is close to the exact answer? Approximation Algorithms Can be created for optimization problems The exact answer for an instance is OPT The approximate answer will never be far from OPT We CANNOT approximate decision problems Performance ratios We are going to find a Near-Optimal solution for a given problem. We assume two hypothesis : Each potential solution has a positive cost. The problem may be either a maximization or a minimization problem on the cost. 8 Performance ratios … If for any input of size n, the cost C of the solution produced by the algorithm is within a factor of ρ(n) of the cost C* of an optimal solution: Max ( C/C* , C*/C ) ≤ ρ(n) We call this algorithm as an ρ(n)approximation algorithm. 9 Performance ratios … In Maximization problems: C*/ρ(n) ≤ C ≤ C* In Minimization Problems: C* ≤ C ≤ ρ(n)C* ρ(n) is never less than 1. A 1-approximation algorithm is the optimal solution. The goal is to find a polynomial-time approximation algorithm with small constant approximation ratios. Approximation scheme Approximation scheme is an approximation algorithm that takes Є>0 as an input such that for any fixed Є>0 the scheme is (1+Є)-approximation algorithm. Polynomial-time approximation scheme is such algorithm that runs in time polynomial in the size of input. As the Є decreases the running time of the algorithm can increase rapidly: For example it might be O(n2/Є) 11 Approximation scheme We have Fully Polynomial-time approximation scheme when its running time is polynomial not only in n but also in 1/Є For example it could be O((1/Є)3n2) 12 Some examples: Vertex cover problem. Traveling salesman problem. Set cover problem. 13 VERTEX-COVER Given a graph, G, return the smallest set of vertices such that all edges have an end point in the set The vertex-cover problem A vertex-cover of an undirected graph G is a subset of its vertices such that it includes at least one end of each edge. The problem is to find minimum size of vertex-cover of the given graph. This problem is an NP-Complete problem. 15 The vertex-cover problem … Finding the optimal solution is hard (it’s NP!) but finding a near-optimal solution is easy. There is an 2-approximation algorithm: It returns a vertex-cover not more than twice of the size optimal solution. 16 The vertex-cover problem … APPROX-VERTEX-COVER(G) 1 2 3 4 5 6 7 C←Ø E′ ← E[G] while E′ ≠ Ø do let (u, v) be an arbitrary edge of E′ C ← C U {u, v} remove every edge in E′ incident on u or v return C 17 The vertex-cover problem … Near Optimal size=6 Optimal Size=3 18 The vertex-cover problem … This is a polynomial-time 2-aproximation algorithm. (Why?) Because: Optimal APPROX-VERTEX-COVER is O(V+E) |C*| ≥ |A| Selected Edges |C| = 2|A| |C| ≤ 2|C*| Selected Vertices 19 Minimum Spanning Tree Given a graph, G, a Spanning Tree of G is a subgraph with no cycles that connects every vertex together A MST is a Spanning Tree with minimal weight Finding a MST Finding a MST can be done in polynomial time using PRIM’S ALGORITHM or KRUSKAL’S ALGORITHM Both are greedy algorithms HAMILTONIAN CYCLE Given a graph, G, find a cycle that visits every vertex exactly once TSP version: Find the path with the minimum weight MST vs HAM-CYCLE Any HAM-CYCLE becomes a Spanning Tree by removing an edge cost(MST) ≤ cost(min-HAM-CYCLE) Traveling salesman problem Given an undirected complete weighted graph G we are to find a minimum cost Hamiltonian cycle. Satisfying triangle inequality or not this problem is NP-Complete. The problem is called Euclidean TSP. 24 Traveling salesman problem Near Optimal solution Faster Easier to implement. 25 Euclidian Traveling Salesman Problem APPROX-TSP-TOUR(G, W) 1 2 3 4 select a vertex r Є V[G] to be root. compute a MST for G from root r using Prim Alg. L=list of vertices in preorder walk of that MST. return the Hamiltonian cycle H in the order L. 26 Euclidian Traveling Salesman Problem MST root Pre-Order walk Hamiltonian Cycle 27 Traveling salesman problem This is polynomial-time 2-approximation algorithm. (Why?) Because: Pre-order Solution APPROX-TSP-TOUR is O(V2) C(MST) ≤ C(H*) C(W)=2C(MST) C(W)≤2C(H*) C(H)≤C(W) C(H)≤2C(H*) Optimal H*: optimal soln W: Preorder walk H: approx soln & triangle inequality 28 EULER CYCLE 1. 2. Given a graph, G, find a cycle that visits every edge exactly once Necessary & Sufficient Conditions: G is connected and every vertex is even degree. Algorithm (O(n2)): Repeatedly use DFS to find and remove a cycle from G Merge all the cycles into one cycle. Min-Weight Matching Given a complete weighted graph of even nodes, G, find a perfect matching of minimum total weight Algorithm (O(n3)): Formulated as a linear programming problem, but solved using a special algorithm. Euclidian Traveling Salesman Problem APPROX-TSP-TOUR2(G, c) 1 Select a vertex r Є V[G] to be root. 2 Compute a MST T for G from root r using Prim Alg. 3 Find a minimal-weight matching M for vertices of odd degree in T. 4 Find an Euler cycle C in G’ = (V, T U M). 5 L=list of vertices in preorder walk of C. 6 return the Hamiltonian cycle H in the order L. 31 Euclidian Traveling Salesman Problem MST Min Matching Euler Cycle HAM-Cycle 32 Time Complexity APPROX-TSP-TOUR2(G, c) 1 2 3 4 5 6 Select a vertex r Є V[G] to be root. Compute a MST T for G from root r using Prim Alg. Find a minimal-weight matching M for vertices of odd degree in T. Find an Euler cycle A in G’ = (V, T U M). L=list of vertices in preorder walk of A. return the Hamiltonian cycle H in order L. O(1) O(n lg n) O(n3) O(n2) O(n) 33 Traveling salesman problem This is polynomial-time 3/2-approximation algorithm. (Why?) Because: Min match Euler cycle Solution APPROX-TSP-TOUR2 is O(n3) C(MST) ≤ C(H*) Optimal C(M) ≤ 0.5C(H*) C(A) = C(MST)+C(M) C(H) ≤ C(A) C(H) ≤ 1.5C(H*) H*: optimal soln M: min matching A: Euler cycle H: approx soln & triangle inequality 34 Proof of C(M)≤ 0.5C(H*) Let optimal tour be H*: j1…i1j2…i2j3…i2m {i1,i2,…,i2m}: the set of odd degree vertices in T. Define 2 matchings: M1={[i1,i2],[i3,i4],…,[i2m-1,i2m]} M2={[i2,i3],[i4,i5],…,[i2m,i1]} M is min matching: C(M) C(M1) and C(M) C(M2) By triangle inequality: C(H*) C(M1) + C(M2) 2 C(M) C(M) 1/2 C(H*) TSP In General Theorem: If P ≠ NP, then for any constant ρ>1, there is no polynomial time ρ-approximation algorithm. Proof: If we have a polynomial time ρ-approximation algorithm for TSP, we can find a tour of cost ρH*. c(u,w) = if ((u,w) in E) then 1 else ρ|V|+1 The optimal cost H* of a TSP tour is |V|. G has a Ham-cycle iff TSP has a tour of cost ρ|V|. If a TSP tour has one edge of cost ρ|V|+1, then the total cost is (ρ|V|+1)+|V|-1>ρ|V| Selected edge not in E Rest of edges 36 The Set-Cover Problem Instance (X, F) : X : a finite set of elements. F : family of subsets of X. Solution C : subset of F that includes all the members of X. Set-Cover is in NP Set-Cover is NP-hard, as it’s a generalization of vertex-cover problem. 37 An example: |X| = 12, |F| = 6 Minimal Covering set size=3 38 A Greedy Algorithm GREEDY-SET-COVER(X,F) 1M←X 2C←Ø 3 while M ≠ Ø do 4 select S Є F that maximizes |S חM| 5 M←M–S 6 C ← C U {S} 7 return C 39 Not optimal … 1st chose 3rd chose 2nd chose Greedy Covering set size=4 4th chose 40 Set-Cover … This greedy algorithm is polynomialtime ρ(n)-approximation algorithm ρ(n)=lg(n) 41 The bin packing problem n items a1, a2, …, an, 0 ai 1, 1 i n, to determine the minimum number of bins of unit capacity to accommodate all n items. E.g. n = 5, {0.3, 0.5, 0.8, 0.2 0.4} The bin packing problem is NP-hard. APPROXIMATE BIN PACKING Problem: fill in objects each of size<= 1, in minimum number of bins (optimal) each of size=1 (NP-complete). Online problem: do not have access to the full set: incremental; Offline problem: can order the set before starting. Theorem: No online algorithm can do better than 4/3 of the optimal #bins, for any given input set. NEXT-FIT ONLINE BIN-PACKING If the current item fits in the current bin put it there, otherwise move on to the next bin. Linear time with respect to #items - O(n), for n items. Theorem: Suppose, M optimum number of bins are needed for an input. Next-fit never needs more than 2M bins. Proof: Content(Bj) + Content(Bj+1) >1, So, Wastage(Bj) + Wastage(Bj+1)<2-1, Average wastage<0.5, less than half space is wasted, so, should not need more than 2M bins. FIRST-FIT ONLINE BIN-PACKING Scan the existing bins, starting from the first bin, to find the place for the next item, if none exists create a new bin. O(N2) naïve, O(N logN) possible, for N items. Obviously cannot need more than 2M bins! Wastes less than Next-fit. Theorem: Never needs more than Ceiling(1.7M). Proof: too complicated. For random (Gaussian) input sequence, it takes 2% more than optimal, observed empirically. Great! BEST-FIT ONLINE BIN-PACKING Scan to find the tightest spot for each item (reduce wastage even further than the previous algorithms), if none exists create a new bin. Does not improve over First-Fit in worst case in optimality, but does not take more worst-case time either! Easy to code. OFFLINE BIN-PACKING Create a non-increasing order (larger to smaller) of items first and then apply some of the same algorithms as before. Theorem: If M is optimum #bins, then First-fit-offline will not take more than M + (1/3)M #bins. Polynomial-Time Approximation Schemes A problem L has a fully polynomial-time approximation scheme (FPTAS) if it has a polynomial-time (in n and 1/ε) (1+ε)-approximation algorithm, for any fixed ε >0. 0/1 Knapsack has a FPTAS, with a running time that is O(n3 / ε). Knapsack Problem Knapsack problem. Given n objects and a "knapsack." we'll assume wi W Item i has value vi > 0 and weighs wi > 0. Knapsack can carry weight up to W. Goal: fill knapsack so as to maximize total value. Ex: { 3, 4 } has value 40. W = 11 Item Value Weight 1 1 1 2 6 2 3 18 5 4 22 6 5 28 7 49 Knapsack is NP-Complete KNAPSACK: Given a finite set X, nonnegative weights wi, nonnegative values vi, a weight limit W, and a target value V, is there a subset S X such that: wi W iS vi V iS SUBSET-SUM: Given a finite set X, nonnegative values ui, and an integer U, is there a subset S X whose elements sum to exactly U? Claim. SUBSET-SUM P KNAPSACK. Pf. Given instance (u1, …, un, U) of SUBSET-SUM, create KNAPSACK instance: vi wi ui V W U ui U iS ui U iS 50 Knapsack Problem: Dynamic Programming 1 Def. OPT(i, w) = max value subset of items 1,..., i with weight limit w. Case 1: OPT does not select item i. – OPT selects best of 1, …, i–1 using up to weight limit w Case 2: OPT selects item i. – new weight limit = w – wi – OPT selects best of 1, …, i–1 using up to weight limit w – wi 0 if i 0 OPT(i, w) OPT(i 1, w) if w i w max OPT(i 1, w), v OPT(i 1, w w ) otherwise i i Running time. O(n W). W = weight limit. Not polynomial in input size! 51 Knapsack Problem: Dynamic Programming II Def. OPT(i, v) = min weight subset of items 1, …, i that yields value exactly v. Case 1: OPT does not select item i. – OPT selects best of 1, …, i-1 that achieves exactly value v Case 2: OPT selects item i. – consumes weight wi, new value needed = v – vi – OPT selects best of 1, …, i-1 that achieves exactly value v 0 OPT (i, v) OPT (i 1, v) min OPT (i 1, v), wi OPT (i 1, v vi ) if v 0 if i 0, v > 0 if v i v otherwise 52 Knapsack: FPTAS 0 OPT (i, v) OPT (i 1, v) min OPT (i 1, v), wi OPT (i 1, v vi ) if v 0 if i 0, v > 0 if v i v otherwise Item Value Weight 1 1 1 0 1 2 3 4 5 6 7 8 9 1 1 1 1 1 1 0 1 2 3 4 5 2 1 2 3 3 5 0 0 x x x x x x x x x x x x x x x 4 4 6 1 5 6 7 i = 0 or v = 0 0 2 0 3 0 W = 11 4 0 5 0 53 Knapsack: FPTAS 0 OPT (i, v) OPT (i 1, v) min OPT (i 1, v), wi OPT (i 1, v vi ) if v 0 if i 0, v > 0 if v i v otherwise Item Value Weight 1 1 1 0 1 2 3 4 5 6 7 8 9 1 1 1 1 1 1 0 1 2 3 4 5 2 1 2 3 3 5 0 0 x x x x x x x x x x x x x x x 4 4 6 1 5 6 7 i = 1 , v = … 0 1 x x x x x x x x x x x x x x 2 0 3 0 W = 11 4 0 5 0 54 Knapsack: FPTAS 0 OPT (i, v) OPT (i 1, v) min OPT (i 1, v), wi OPT (i 1, v vi ) if v 0 if i 0, v > 0 if v i v otherwise Item Value Weight 1 1 1 0 1 2 3 4 5 6 7 8 9 1 1 1 1 1 1 0 1 2 3 4 5 2 1 2 3 3 5 0 0 x x x x x x x x x x x x x x x 4 4 6 1 5 6 7 i = 2 , v = … 0 1 x x x x x x x x x x x x x x 2 0 1 3 x x x x x x x x x x x x x 3 0 W = 11 4 0 5 0 55 Knapsack: FPTAS 0 OPT (i, v) OPT (i 1, v) min OPT (i 1, v), wi OPT (i 1, v vi ) if v 0 if i 0, v > 0 if v i v otherwise Item Value Weight 1 1 1 0 1 2 3 4 5 6 7 8 9 1 1 1 1 1 1 0 1 2 3 4 5 2 1 2 3 3 5 0 0 x x x x x x x x x x x x x x x 4 4 6 1 5 6 7 i=3,v=… 0 1 x x x x x x x x x x x x x x 2 0 1 3 x x x x x x x x x x x x x 3 0 1 3 5 6 8 x x x x x x x x x x W = 11 4 0 5 0 56 Knapsack: FPTAS 0 OPT (i, v) OPT (i 1, v) min OPT (i 1, v), wi OPT (i 1, v vi ) if v 0 if i 0, v > 0 if v i v otherwise Item Value Weight 1 1 1 0 1 2 3 4 5 6 7 8 9 1 1 1 1 1 1 0 1 2 3 4 5 2 1 2 3 3 5 0 0 x x x x x x x x x x x x x x x 4 4 6 1 5 6 7 i = 4, v = … 0 1 x x x x x x x x x x x x x x 2 0 1 3 x x x x x x x x x x x x x 3 0 1 3 5 6 8 x x x x x x x x x x 4 0 1 3 5 6 7 9 11 12 14 W = 11 x x x x x x 5 0 57 Knapsack: FPTAS 0 OPT (i, v) OPT (i 1, v) min OPT (i 1, v), wi OPT (i 1, v vi ) if v 0 if i 0, v > 0 if v i v otherwise Item Value Weight 1 1 1 0 1 2 3 4 5 6 7 8 8 9 1 1 1 1 1 1 0 1 2 3 4 5 2 1 2 3 3 5 0 0 x x x x x x x x x x x x x x x 4 4 6 1 5 6 7 i = 5, v = … 8 0 1 x x x x x x x x x x x x x x 2 0 1 3 x x x x x x x x x x x x x 3 0 1 3 5 6 8 x x x x x x x x x x 4 0 1 3 5 6 7 9 11 12 14 x x x x x x 5 0 1 3 5 6 7 7 8 10 12 13 14 16 18 19 21 W = 11 S={1, 2, 5} 58 Knapsack: FPTAS Tracing Solution // first call: pick_item(n, v) where M[n,v] <= W and v is max pick_item( i, v) { if (v == 0) return; if (M[i,v] == wi + M[i-1, v-vi]) { print i; pick_item(i-1, v-vi); print i; } else pick_item(i-1, v); } Item Value Weight 1 1 1 0 1 2 3 4 5 6 7 8 8 9 1 1 1 1 1 1 0 1 2 3 4 5 2 1 2 3 3 5 0 0 x x x x x x x x x x x x x x x 4 4 6 1 5 6 7 i = 5, v = … 8 0 1 x x x x x x x x x x x x x x 2 0 1 3 x x x x x x x x x x x x x 3 0 1 3 5 6 8 x x x x x x x x x x 4 0 1 3 5 6 7 9 11 12 14 x x x x x x 5 0 1 3 5 6 7 7 8 10 12 13 14 16 18 19 21 W = 11 S={1, 2, 5} 59 Knapsack Problem: Dynamic Programming II Def. OPT(i, v) = min weight subset of items 1, …, i that yields value exactly v. Case 1: OPT does not select item i. – OPT selects best of 1, …, i-1 that achieves exactly value v Case 2: OPT selects item i. – consumes weight wi, new value needed = v – vi – OPT selects best of 1, …, i-1 that achieves exactly value v 0 OPT (i, v) OPT (i 1, v) min OPT (i 1, v), wi OPT (i 1, v vi ) if v 0 if i 0, v > 0 if v i v otherwise V* n vmax Running time. O(n V*) = O(n2 vmax). V* = optimal value = maximum v such that OPT(n, v) W. Not polynomial in input size! 60 Knapsack: FPTAS Intuition for approximation algorithm. Round all values up to lie in smaller range. Run dynamic programming algorithm II on rounded instance. Return optimal items in rounded instance. Item Value Weight Item Value Weight 1 934,221 1 1 1 1 2 5,956,342 2 2 1 2 3 17,810,013 5 3 3 5 4 21,217,800 6 4 4 6 5 27,343,199 7 5 6 7 W = 11 original instance W = 11 rounded instance S = { 1, 2, 5 } 61 Knapsack: FPTAS Knapsack FPTAS. Round up all values: v v vi i , vˆi i vmax = largest value in original instance – = precision parameter – = scaling factor = vmax / n – Observation. Optimal solution to problems with v or vˆ are equivalent. Intuition. v close to v so optimal solution using v is nearly optimal; vˆ small and integral so dynamic programming algorithm is fast. Running time. O(n3 / ). Dynamic program II running timeis O(n2 vˆmax) , where vmax n vˆmax 62 Knapsack: FPTAS vi vi Knapsack FPTAS. Round up all values: Theorem. If S is solution found by our algorithm and S* is any other feasible solution of the original problem, then (1 ) vi vi iS i S* Pf. Let S* be any feasible solution satisfying weight constraint. vi i S* vi always round up vi solve rounded instance optimally i S* iS (vi ) never round up by more than vi n |S| n iS i S DP alg can take vmax (1 ) vi i S n = vmax, vmax iS vi 63