LIBRARY OF THE MASSACHUSETTS INSTITUTE OF TECHNOLOGY ALFRED P. SLOAN SCHOOL OF MANAGEMENT GENERALIZED LAGRANGE MULTIPLIERS IN INTEGER PROGRAMMING by Jeremy March F. 7, Shapiro 1969 Working Paper 371-69 MASSACHUSETTS INSTITUTE OF TECHNOLOGY 50 MEMORIAL DRIVE CAMBRIDGE, MASSACHUSETTS 02139 GENERALIZED LAGRANGE MULTIPLIERS IN INTEGER PROGRAMMING by Jeremy March F. 7, Shapiro 1969 Working Paper 371-69 RECEIVED JUL 25 1969 M. I. T. Li. 5 f. „, ......a p. 2, line 5 p. 13 — 1 — — Add [2] to ths refsrencss iiatacl "In [iOj. Nemhausar and,.," Add the following footnote to the second sevij-.ence of LEILlA 3. K "1. If E V, k-1 ^ = 0, then division by this sum 1e of csurse not :^ p. 14 — 19 — n _ I a.v,,,. '^ - and „ }1 In the second sum on the i&ft of tha first inequality, read ". "v p. the inequalities are ' liua from the bottom- "example in [9]" - c.v,,,. ABSTRACT The integer programming problem is reformulated using group theory thereby allowing a new Lagrangian optimization problem for integer programming to be constructed. its relationship The properties of this problem and to group theoretic branch and bound and cutting plane algorithms are discussed. Necessary and sufficient conditions for the existence of optimal multipliers are also given. GENERALIZED LAGRANGE MULTIPLIERS IN INTEGER PROGRAMMING 1 . Introduction Several authors ([3], [4], [8], [9], [10], [14]) have proposed generalized Lagrangian methods for finding good or optimal solutions to integer programming problems. The capital budgeting problem of Lorie Savage [9], essentially the 0-1 multi-dimensional Knapsack problem, and has received particular attention in this regard. In [9], Nemhauser and Ullman prove the somewhat negative result that the approach of Everett [4] applied to the capital budgeting problem by Kaplan in [8] can yield an optimal solution only if there is an optimal linear programming solution that is integer. [12], [13]) In this paper, we use group theory ([5], [6], to reformulate the integer programming problem, [7], [11], thereby obtaining a Lagrangian problem which appears to offer greater combinatorial resolution than previous methods. Conversely, the usefulness of the group theoretic approach is enhanced by the Lagrangian problem. In the next section we construct the Lagrangian problem and discuss its properties. The following section is concerned with the relationship of cuts generated by the Lagrangian problem to the strong cuts developed in [6]. Necessary and sufficient conditions for the existence of optimal multipliers are also given. concluding remarks. There follows a numerical example and a few There is one appendix in which some of the previous results are specialized to the zero-one integer programming problem. 6^9G42 2. Construction of the Lagrangian Problem and Its Properties Consider the Integer programming problem In the form s.t. mln cw Aw = b (1) w non-negative integer where lx(m+n) vector of integers, A is an mx(m+n) matrix of is a c integers, and b is an denoted by a., j raxl The columns of A are vector of integers. = 1,..., m+n. Let B be an optimal LP basis for (1) and rearrange the columns of A so that it can be partitioned as (R,B) and c as (c^^, The generic n-vector of non-basic variables is denoted c^). by X and Is called a correction. with addition modulo The columns of B abelian group consisting of D elements (see the group and let Identity element. X , k = 0,1,..., D-I, [ I 5 generate a finite Let G be ]). be its elements where X is the Finally, let g be the function which maps integer m-vectors Into group elements. It can easily be shown that problem (I) is equivalent to min E c.x, (2a) If B is an arbitrary basis, the analysis below is valid with minor modifications. n s.t. Z r..x. < a.x. = B b , = l,...,m, i , (2b) n E (2c) j=l X. where and c. 6 = c. - c B~ a = g(b). j=l,...,n =0,1,2,...; R = (r , . . ) = B~ R, (2d) b = (h = B~ b, a. ) = g(a ) Condition (2b) is equivalent to the requirement that the basic variables (relative to B) be non-negative, and condition (2c) is equivalent to the requirement that they be Integer. If we let X = {x|x satisfies (2c) and (2d)}, Then problem (2) can be rewritten as n min E j s.t. Z 1 r..x. < c.x. =l J J b., 1 = l,...,m, (2 ) =1 xe)C. Problem (2') has the same form as Everett's problem of [2 ] . The [ 4 ] and problem (2) approach suggested by Everett is to weight the constraints (2b) by multipliers and place them in the objective function. 1 " We assume the existence of at least one x such that .Z.a.x. =g. J J Jproblem (1) is infeasible. Otherwise, m In particular, for given u. ^ >_0, i = l,...,m, satisfying c. + T. 1 = j ^ u?r ^J ^ i=l l,...,n, we construct the new Lagrangian optimization problem n rain E 3=1 ,(c m . + J Z — u.r i=l ^ , . )x. ^J J . (4) s.t. xeX Problem (4) is a group optimization problem identical in form to the group optimization problem originally derived in [ 12 (4) ] for a further discussion of this problem. is denoted by x , [ 5 ] See . The requirement that c. + J 11 ] and An optimal solution to or more generally by x(u) when u of multipliers in (4). [ T; i=l is the vector u.r.. ^ ^J > ~ 0, j = l,...,n, is necessary for (4) to have a bounded solution. Notice that we could have constrained explicitly some variables in (1) to be zero or one. The modifications to the construction above in this case are given in Appendix A. 0, We begin our analysis with a restatement in the context of this paper of Everett's theorem 1 4 [ P ={i|u° LEMMA i = An optimal solution x 1: ; p. 401]. > 0} . to First, for given u >^ (5) (4) is optimal in (2) with b^, l,...,m, replaced by n (i) E 3=1 if i e P r. .X? '' ' n (ii) where Proof: q . is any + r. .X? E if i q. non-negative integer, i ^ «! P, P. Suppose the contrary; that is, suppose there exists a non- negative integer vector ycX with the properties that n E n c.y. J j=l < c.x°, E j=l ^ J J and Since u. >^0, i = n _ E r, .y. n < E 0, let _ r^ .x? l,...,m, there obtains for ieP. m n (c. Z j=l + n _ u°r..)y. = Z J 1=1^ (c. + ^J n < J j=l J = u°r..)x° Z icP ^J ^ . + J u°r..)y. Z J icP (E. + ^J ^ m n _ Z (c Z j=l Z J j=l Z J _ u°r..)x° i=l ^ ^J J The last inequality is a contradiction to the optlmality of x When (4) is solved by algorithm of tlie [11 ], K , k = 0,1,..., D-1, replaced by COROLLARY A, K A, 1: X ?^ K Let x° (A e. ) (4). the algorithm also finds optimal solutions to (A) when the groun constant A in B in (/c) is replaced by be the optimal solution when 3 is K. . An optimal solution x (A.) to (4) with group constant is optimal in (2) with b., i = l,...,m, replaced by 1 n (i) Z j=l if ieP r..x°. (A, ) ^J J ^ n (ii) where q. Z ?..x°(A, ) + q. is any non-negative integer, if i ii^P ^ P. Thus, problem (4) is algorithmically useful in several ways. =0 Suppose 5 ] ) fails to solve (i.e., the group optimization problem of n _ Then (4) can be resolved Z r..x. > b.. (2) because for at least one i, ^ j=l ^J J (4) with u [ with new u^ for which some u^ (2b) with X = X is violated. ^ 0; e.g., yl > only if the i constraint in Let x^ be the new optimal solution to (4). . "First, suppose There are three cases to consider. j this case, ex 1 r..x. E =l ^^ - ^b., 1 , i e In P. the cost of an optimal correction to (2)> n _ because of the constraints Z r,.Xj <_ j X e is a lower bound on z*, This lower bound is greater than ex " E r i ^ ^ P, = l ^^ -• implied by (4) which were not in effect when (4) was pre- viously solved with u The improvements of bounds could be very impor- = 0. tant to the algorithm of [ 12 ] in which group theoretic bounds are used to limit the search of the non-basics for an optimal correction. " 1 Z r..x. ^J J 3=1 The second possibility is that - o feasible correction. - b., < - i = l,...,m, or x - . Since ex the non-negative quantity ex z*, < 1 is a ^ an upper bound on the loss from terminating with x - o - ex is 1 . . Finally, if neither of the two cases above obtain, the cut (c is a valid cut + u^R)x > + u^R)x^ (c which could be added to (1). (6) The relationship of the Lagrangian methods to the cutting plane method are discussed in the next section. If the requirements vector b can be relaxed, to q (1) may be derived by replacing b with Rx can be any integer vector satisfying q >_ (A, K + Bq for some and q. = solution to the new (approximate) problem is (x We are now assuming u ) then a suitable (>.,), if i e A K P. solution where The optimal q) is an arbitrary non-negative vector in (4). In order to put this relaxation scheme in better perspective, con- sider the isomorphic representation of G as M(I)/m(B) where m(I) is the group consisting of all integer points in m-space with ordinary addition, and WV(B) is the subgroup consisting of all integer points which can be spanned by integer combinations of the basic activities (2c) is then an equivalence class of m-vectors. . The element Because of the combin- atorial structure of the given problem, it may be that vectors from awkward as hand side B are vectors in (1); e.g., low cost corrections for group right b B in 3 may tend to be long and infeasible in (2). It is tempting to try to construct an algorithm which, starting with u o = 0, solves (4) with a monotonic increasing sequence of u - k so that Rx converges to b k k - from above. k = 0,1,2,..., , Unfortunately, the Rx k vectors are not sufficiently well behaved to make such an algorithm feasible. There - k is, however, a certain degree of regular behavior exhibited by the Rx LEMMA 2: Any optimal solutions x multipliers u and u and x to (4) respectively must satisfy (u^ - u°) R (x° - xS > with non-negative . 10 The proof is immediate from the inequalities Proof. (c + u°R)x° 1 (c + u°R)x\ (c + u^R)x^ (c + u^R)x°. and Thus, if n L — o r..x.. > u, k u, k < and u. = u., = l,...,m, i ^ k, i 1 1 Unfortunately, with u 1 J-1 control the magnitude of the sums then Z r..x. < ..ill— j=l J -^ so defined there is no way to explicitly n _ ^ Y. r..x., j=l ^J i ?* k. Although future research J may lead to algorithms with well behaved properties for using (4) to solve (2), it appears at this time that is as a heuristic. the primary usefulness of (4) in solving (2) In particular, we are currently implementing an adaptive integer programming system based on the ideas in [7], and we plan to experiment with the setting of multipliers in (4) in the near future. Finally, we remark that the multipliers in (4) have an unusual interpretation for the integer programming problem with the formulation (2). In particular, c. + ua J where ij; = uB . . = c. J J - Ila . J + ua . J is adjusted by - -i|; . - i>)a. (IT J J The usual LP interpretation of cost savings to be gained by reducing b Jl. = c. II is that n. by one unit. reflects the In the above expression, which can be interpreted as an added cost because the distance with respect to the i basis activity a from b to the boundary 11 of the cone { H B : £ } is >_ insufficient. This last statement can be made more precise by the following lemma from [13]. Given any set that if b. (1) ^ I ^. t*, i { 1 e , . . I, . ,m} , The lemma states: there exist non-negative integers t* such then (2) with rows i e I omitted in (2b) solves in the sense that there is an optimal solution to the reduced problem which is an optimal correction in (1). the condition of this theorem for some set I. Suppose (2) satisfies Then problem (4) formed from the reduced problem (2) is the same as problem (4) formed from the original problem (2) with u. = 0, i e I. 12 Cutting Plane and Lagrangian Methods 3. In 5 [ Gomory uses the group theoretic approach to develop ], theorems for characterizing and constructing strong cuts for the cutting If we let plane method. [X] of interest are the faces of denote the convex hull of X , then the cuts Our purpose in this section is to [X]. relate problem (4) and the Lagrangian cuts to these faces. Suppose Ax A a and —> —> A o 0. Lagrangian cut kAx >_ >_ (6) A is a face of It is natural this problem is: [ 6 ; 16] p. that to ask if it is possible to obtain equivalent to this face. is an equivalent cut, kA Gomory shows [X]. Since for any k 0, > the correct mathematical statement of Does there exist a u >_ 0, k > 0, such that uR - kA = -c? (7) The system (7) consists of n equations in m+1 variables which does not always possess a solution of the form we seek. Although this inability to produce selected strong cuts appears to be a [X] drawback of the Lagrangian method, we remark that the faces of are generated with respect only to the group identities of the non- basic activities a., l,...,n, and = j the requirements vector b. The Lagrangian approach, on the other hand, incorporates Information about the real space identities of the a. and b. would be to (i) select a boundary for a specified u use the methods of contain x . of [X] by solving problem (4) chosen according to the magnitude of b ?* [ point x A synthesis of the two approaches 6 ] , and (11) to generate cuts beginning with those which 13 Does there Of course, a more fundamental existential question is: exist a u >^ such that (4) solves (2)? A solution needed. implies s t e 2^ To answer this, a definition is is said to be irreducible if s <^ t and Let T be the set of all irreducible solutions. = t. say T = {t easy to demonstrate that T is finite; Suppose X* is an optimal correction. + uR > X e It is • Then it is clear that x* is if and only if there exists a u an optimal solution in (A) c )i^_i s such that >^ (8) 0, and (c Note that each x where (c e t + uR)t'*^ (c > + uR)x*, k = 1,...,K. (9) X can be decomposed in at least one way as x = e T and s is non-negative integer. Thus, (c + uR)x >^ (c + uR)x* and condition (9) is sufficient as well as necessary. t + s + uR)t >_ The following lemma is a direct consequence of Farkas' lemma applied to (8) for u and (9) LEMMA 3: ^ 0. Suppose x* is an optimal correction. Then there exists a u such that X* is optimal in (4) if and only if for every V ^ such that " ^ - k Z Rt k=l + v, K K Z k=l a.v„. J^, v Z ^ k=l . < Rx*, K-^n vector v >_ > and 14 there obtains ct^ I + V, k=l ^ ex*. - K K k=l .5:,c.v \" " ^k " k=l Suppose, however, that This lemma is difficult to interpret as it stands. vector v there exists a and v ^ —> with v = 0, j = l,...,n, and r>.+j also such that K _ L Rt _ v . <_ Rx*, (10) < ex*. (11) ^ k=l K ^ ^k " k=l and •^ I k=l - k ct V, ^ K ^ k=l ""k ^ In this case, there does not exist a u such that x* is an optimal solution in (4). in (A) Conditions (10) and (11) state, of course, that x* is not optimal if there is a convex combination of the and uses no more resource than x*. t which is less costly The interested reader should compare this with Everett's discussion on the source of gaps which can be found on page 408 of reference 4. 15 Lemma 3 was derived without regard to whether or not the given If x* is reducible, then necessary optimal correction x* is irreducible. and sufficient conditions that x* is optimal in (4) is that there exist a u 21 such that for at least one teT for which x* = c ( c^ + uR + uR) t t + s (s >_ 0, s ?< 0) >_ >_ G + uR) k = t 1 , . . . , k (7 + uR)s = Since g(t) = g(x*) = (c + uR)s = B, we have g(s) = 0. Thus, the condition that can be interpreted as the requirement that there be at least one costless circuit in the shortest route network representation of (4). Any such circuit can be built up from costless elementary circuits which can be found by the algorithm of [11]. 16 A. Numerical Example Consider the integer programming problem min + 6x, + Ax^ 21x„ 2 12 + -Ix, + 13x„ s.t. A 5 5x, A + 2xc = (12) 10x„ - Ix. + Ix, + 3xc 3 2 X. Activities a, non-negative integer, A j 16 5 = 8 5 = 1,2, 3, A, 5. Problem (A) derived and a^ constitute an optimal LP basis. with respect to this basis is (all fractions were eliminated by multiplying by 13) min (lA s.t. - 3u + lu )x + 19x -3x ^ + (11 + 19u, + 37u )x + 2x ^ <^ + 2u^ - 5u )x 32 (13) - 5x <_ 2A Ix^ + llx + 8x E 11 X. (8 -* + 37x Ix + (mod 13) non-negative integer, j = 1,2,3 The optimal solution to (13) with u° = 0, u" = 0, is x° = 0, x° x° = 0. = 0, 1, This correction is not feasible in (12) as the second inequality constraint in (13) is violated. X. = Setting u = 0, u„ = 1, yields the solution x„ = 0, x„ = 3, which is the optimal correction we seek. this correction is optimal for any u >^ with u. = 0, 9/13 In fact, ^ u < 8/5. 17 5. Conclusion In this paper we have tried to demonstrate how the Lagrangian problem (4) is useful when trying to solve integer programming problems. We saw that an optimal solution to (4) could be used in a variety of ways to find good or optimal solutions to (1). Unfortunately, the theor- etical results obtained thus far are not sufficiently strong to enable us to algorithmically control the properties of optimal solutions to (4). Future research may lead to an improvement in these procedures. We emphasize once again, however, that we expect to incorporate (4) problem into the adaptive group theoretic algorithm of [7] and experiment with heuristics for setting the multipliers u.. It seems clear that the power of the group problem from [5] can be enhanced by the approach here. Certain results from the literature on generalized Lagrange multipliers were not specialized for our problem although it is possible In particular, Everett's to do so. reference 4 lambda and epsilon theorems in were ignored because they appear to be not directly relevant to our study. Similarly, the linear programming problem in [2] and Brooks theorems in [3] have been ignored. In addition, there may be some connection between the results here and those of Balas in [1] on duality theory in integer programming. 18 Appendix A The Zero-one Integer Programming Problem SC Suppose there is a subset 1, JF.S, = 0, and X 1 ,2, . . . , jtS n _ Z c.x. £ s.t. Then (2) becomes -J J < - b , i = , iJ j=l (14b) _ (-r..)x. Z m 1, i _ n < - J 1-b., i = l,...,m, 1 ' (14c) 1 ' a.x. = B Z X. (lAa) r..x. j=l = x^ = or J J j=l = Suppose further that the first m.- . basic variables are 0-1 variables. min {l,...,m+ti} such that x. (14d) 0,1,2,..., or 1 , i e (14e) S^ e j (14f) S J As before, define Z = For a given u {x|x satisfies (14d), where we now require u the Lagrangian problem (4) >^ (14e) , only if (14f)}. i >_m + 1, we define . 19 n _ ™ _ Z(c.+ Eur..)x. min i=l " ^J J j=l J (15) xeZ Problem (15) is a zero-one group optimization problem which can be solved by the zero-one group algorithm of to lemma 1 2: u? = m, —> 0, i be an optimal solution to (15) with u Let x COROLLARY 1 We have the following corollary [7]. + l,...,m. is optimal in Then x such that (14) with (a) b 1 ' 1 , i = l,...,m, replaced by n (i) r,,x° I j=l ^J if u° > J n (ii) r. .x° Z j=l ^J and (b) 1 - b , i = 1 , . . . ,m , + if u° q, i J -< 1 0; replaced by n (iii) Z (-r. .)x° j=l ^J if u? < J n (iv) where the q. Z (-r. .)x° + q, if u° > 0, can be any non-negative integers. Problems (14) and (15) are the ones applicable to the LorieSavage capital budgeting model. We mention in passing that the zero- one group algorithm solved the example in [8] with u = 0; that is, without resort to explicit use of the multipliers of this paper. 20 REFERENCES (1) Balas, E., "Duality in Discrete Programming," Technical Report No. 67-5, July, 1967, Department of Operations Research, Stanford University. Revised December, 1967. (2) Brooks, R. and A Geoff rion, "Finding Everett's Lagrange Multipliers by Linear Programming," Opns. Res., l±, pp 11A9-1153, (1966). (3) Brooks, R. , "Some Linear Programming Applications to Stockage Problems," The RAND Corporation RM-5422-PR, September, 1967. (4) Everett, H. "Generalized Lagrange Multiplier Method for Solving Problems of Optimum Allocation of Resources," Opns. Res., J[l^, pp 399-417. (5) Gomory, R. , "On the Relation Between Integer and Non-Integer Solutions to Linear Programs," Proceedings of the National Academy of Sciences, 53, pp 250-256, (1965). (6) Gomory, R. , "Some Polyhedra Related to Combinatorial Problems," IBM Research Report RC2145, July, 1968. (7) Gorry, G.A. and J. Shapiro, "An Adaptive Group Theoretic Algorithm for Integer Programming Problems," Technical Report No. 38, Operations Research Center, M.I.T., May, 1968. {^) Kaplan, S., "Solution of the Lorie-Savage and Similan Integer Programming Problems by the Generalized Lagrange Multiplier Method," Opns. Res., pp 1130-1136, (1966). , U (9) Lorie, J. and L.J. Savage, "Three Problems in Capital Rationing," J. of Business, pp 229-239, (1955). (10) Nemhauser, G. and Z. Ullman, "A Note on the Generalized Lagrange Multiplier Solution to an Integer Programming Problem" Opns. Res., 16 pp 450-452, (1968). (11) Shapiro, J., "Dynamic Programming Algorithms for the Integer Programming Problem I: The Integer Programming Problem Viewed as a Knapsack Type Problem," Opns. Res. _16, pp 103-121, (1968). (12) Shapiro, J., "Group Theoretic Algorithms for the Integer Programming Problem II: Extension to a General Algorithm," Opns. Res. _16 pp 928-947. (13) Shapiro, J., "Turnpike Theorems for Integer Programming Problems," Sloan School of Management Working Paper 350-68, M.I.T., (1968). — 21 (14) Weingartner, H.M., "Capital Budgeting of Interrelated Projects: Survey and Synthesis," Mgement Sci., U^, pp A85-516, (1966). (15) Weingartner, H.M. and D. Ness, "Methods for the Solution of the MultiDimensional 0/1 Knapsack Problem," Opns. Res., J[_5, pp 83-103, (1967). BASEMENT: Date Due \%'^ JUL 12 1^85 -- |0V 2 Llb-26-67 ,-6^ ,|||||||||lillllliilillnnillhil""'il TOaO 003 674 bSS 3 MIT LIBRARIES ll||Mll[||| 3 jiljl TDflD III iiiii|iii III 0D3 i||iii |iiiji|ii]ii|| fi7M MbS Mil LlBfiAHlES 3 TOflO 003 fl74 bb3 MIT LiBBAfilES 6'^ TOflO 3 003 T05 24M MIT 3^^ i, L IBWiHIF^ ^^h^ w^-