Multidisciplinary Workshop on Advances in Preference Handling: Papers from the AAAI-14 Workshop Constraint-Based Preferences via Utility Hyper-Graphs Rafik Hadfi and Takayuki Ito Department of Computer Science and Engineering Graduate School of Engineering, Nagoya Institute of Technology Gokiso, Showa-ku, Nagoya 466-8555, Japan rafik@itolab.nitech.ac.jp, ito.takayuki@nitech.ac.jp Abstract shown in (Chajewska and Koller 2000; Bacchus and Grove 1995). Similarly, (Robu, Somefun, and Poutre 2005) used utility graphs for preferences elicitation and negotiation over binary-valued issues. In another work, (Marsa-Maestre et al. 2009) adopts a weighted undirected graph representation of the constraintbased utility space. Particularly, a message passing algorithm is used to find the highest utility bids by finding the set of unconnected nodes which maximizes the sum of the nodes’ weight. However, restricting the graph and the message passing process to constraints’ nodes does not allow the representation to be descriptive enough to exploit any potential hierarchical structure of the utility space through a quantitative evaluation of the interdependencies between both issues and constraints. In (Fujita, Ito, and Klein 2010), issues’ interdependency is captured by means of similar undirected weighted graphs where a node represents an issue. This representation is restricted to binary interdependencies while real-world scenarios involve “bundles” of interdependent issues under one or more constraints. In our approach, we do not restrict the interdependency to lower-order constraints but we allow p ary interdependencies to be defined as an hyper-edge connecting p issues. Adopting such representation with its underlying utility propagation mechanism comes from the intuition that preferences elicitation, when operating on a graphical model, could be seen as some form of Bayesian inference (Kwisthout and van Rooij 2013). Therefore, we chose to adopt a graphical representation that serves more as an adequate framework for any preferencebased space. The advantage of using this representation is its scalability in the sense that the problem becomes harder for a large number of issues and constraints. However, if we can decompose the utility space, we can exploit it more efficiently. That is, the decomposition will isolate independent components, thus, allowing them to be treated independently from the rest of the problem. This could allow a layered, hierarchical view of any given decision making problem. Given such architecture, it is possible to reason over the different layers of the problem according to a top-down approach. Even the idea of issue could be generalized to include sub-issues, located in sub-utility spaces and represented by cliques in the hyper-graph. Consequently, the search process could identify sub-contracts that are subject to improvement at each Real-world decisions involve preferences that are nonlinear and often defined over multiple and interdependent issues. Such scenarios are known to be challenging, especially in strategic encounters between agents having distinct constraints and preferences. In this case, reaching an agreement becomes more difficult as the search space and the complexity of the problem grow. In this paper, we propose a new representation for constraintbased utility spaces that can tackle the scalability problem by efficiently finding the optimal contracts. Particularly, the constraint-based utility space is mapped into an issueconstraint hyper-graph. Exploring the utility space reduces then to a message passing mechanism along the hyper-edges by means of utility propagation. We experimentally evaluate the model using parameterized random nonlinear utility spaces. We show that it can handle a large family of complex utility spaces by finding the optimal contract(s), outperforming previous sampling-based approaches. 1 Introduction Complex decision making problems involve multiple and interdependent issues, which usually yields nonlinear utility spaces. Finding the contract(s) that maximize a given preference profile becomes then challenging as the search cost increases with the increasing complexity of the utility space. In this paper, we propose a new, modular representation for constraint-based utility spaces that can efficiently tackle the complexity of the search process. We address this problem by adopting a representation that allows a modular decomposition of the issues-constraints given the intuition that constraint-based utility spaces are nonlinear with respect to issues, but linear with respect to the constraints. This allows us to rigorously map the utility space into an issueconstraint hyper-graph. Exploring the utility space reduces then to a message passing mechanism along the hyper-edges by means of utility propagation. Adopting a graphical representation while quantitatively reasoning about preferences is not new in the multi-attribute utility literature. In fact, the idea of utility graphs could potentially help decomposing highly nonlinear utility functions into sub-utilities of clusters of inter-related items, as Copyright c 2014, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. 49 sub-level of the problem. This combination of separating the system into layers, then using utility propagation to focus attention and search within independent constrained regions can be very powerful in any decision making problem that involves a large number of issues and constraints. In fact, a similar idea of recursion in the exploration of utility space was introduced by (Marsa-Maestre et al. 2013) although it is region-oriented and does not adopt a graphical representation of the utility space. We experimentally evaluated our representation using parametrized random nonlinear utility spaces, showing that it can handle large and complex spaces by finding the optimal contracts while outperforming previous sampling approaches. The paper is organized as follows. In the next section, we propose the basics of our representation. In section 3, we describe the search mechanism. In section 4, we provide the experimental results. In section 5, we conclude and outline the future work. 2 2.1 Figure 1: 2 dimensional nonlinear utility space {c1 , . . . , cj , . . . , cm }. The utility space is therefore decomposable according to these constraints. This yields a modular representation of the interactions between the issues and how they locally relate to each other. In fact, hyp(cj ) reflects the idea that the underlying contracts are governed by the bounds defined by cj once the contracts are projected according to their issues’ components. In this case, the interdependence is not between issues but between constraints. For instance, two constraints c1 and c2 can have in common one issue ik taking values respectively from an interval Ik,c1 if it is involved in c1 , and values in Ik,c2 if it is involved in c2 , with Ik,c1 6= Ik,c2 . Finding the value that maximizes the utility of ik while satisfying both constraints becomes harder due to fact that changing the value of ik in c1 changes the instance of ik in c2 in a cyclic manner. This nonlinearity gets worse with an increasing number of issues, domains’ sizes, and the non-monotonicity of the constraints. Next, we propose to transform (1) into a modular, graphical representation. Since one constraint can involve one or more multiple issues, we will adopt a hyper-graph representation. Nonlinear Utility Space Representation Formulation We start from the formulation of nonlinear multi-issue utility spaces used in (Ito, Hattori, and Klein 2007). That is, an n dimensional utility space is defined over a finite set of issues I = {i1 , . . . , ik , . . . , in }. The issue k, namely ik , takes its values from a finite set Ik with Ik ⇢ Z. A contract ~c is a vector of issue-values with ~c 2 I and I = ⇥nk=1 Ik . An agent’s utility function is defined in terms of m constraints, making the utility space a constraint-based utility space. That is, a constraint cj is a region of the total n dimensional utility space. We say that the constraint cj has value w(cj , ~c) for contract ~c if cj is satisfied by ~c. That is, when the contract point ~c falls within the hyper-volume defined by constraint cj , namely hyp(cj ). The utility of an agent for a contract ~c is thus defined as in (1). X u(~c) = w(cj , ~c) (1) 2.3 cj2[1,m] , ~ c2hyp(cj ) We assign to each constraint cj a factor j , yielding a factors’ set = { 1 , . . . , j , . . . , m }. The utility hypergraph G is defined as G = (I, ). Nodes in I define the issues and the hyper-edges in are the factors (constraints). The neighbors’ set N ( j ) ⇢ I of factor j represents the issues connected to j (involved in constraint cj ), with |N ( j )| = 'j . In case 'j = 2 8j the problem collapses to a constraints satisfaction problem in a standard graph. To each factor j corresponds a 'j dimensional matrix, M j , where the kth dimension is Ik = [ak , bk ], the domain of issue ik . This matrix contains all the values that could be taken by the issues in N ( j ). Each factor j has a function uj defined as a sub-utility function of the issues in N ( j ), defined as in (2). In the following, we distinguish three types of constraints: Cubic, Bell and Plane constraints. Constraint-based utility spaces are a practical way to represent non-monotonic and nonlinear utility functions (Marsa-Maestre et al. 2009; Lopez-Carmona et al. 2011; Marsa-Maestre et al. 2013). Such representation (1) produces a “bumpy” nonlinear utility space with high points whenever many constraints are satisfied and lower points where few or no constraints are satisfied. Figure 1 shows an example of nonlinear utility space for issues i1 and i2 taking values in I1 = I2 = [0, 100], with m = 500 constraints. 2.2 From Utility Space To Utility Hypergraph New Representation The utility function (1) is nonlinear in the sense that the utility does not have a linear expression against the contract (Ito, Hattori, and Klein 2007). This is true to the extent that the linearity is evaluated with regard to the contract ~c. However, from the same expression (1) we can say that the utility is in fact linear, but in terms of the constraints (2) uj : N ( j ) ' j ! R uj (~x) 7! w(cj , ~x), ~x = (i1 , . . . , ij , . . . , i'j ) As we are dealing with discrete issues, uj is a mapping defined by the matrix M j . That is, uj (i1 , . . . ik , . . . , i'j ) 50 is simply the (1, ..., k, ..., 'j )th entry in M j corresponding to the value w(cj , ~x) described in (2). It is possible to extend the discrete case to the continuous one by allowing continuous issue values and defining uk as a continuous function. Next, we provide few examples about the representation. algebra is changed into an additive algebra to support utility accumulation necessary for the assessment of the contracts. The messages circulating in the hyper-graph are nothing other than the contracts we are attempting to optimize through utility maximization. Next, we develop the message passing (MP) mechanism. Example 1. We illustrate the mapping between a 2 dimensional utility space (Figure 2a) and its corresponding hyper-graph G2 (Figure 2b). The issues’ domains are I1 = I2 = [0, 9]. G2 consists of m = 10 constraints (red squares) where each constraint involves at least 2 issues (white circles). We note 6 cubic constraints, 3 plane constraints and 1 bell constraint. 3.1 Message Passing We consider the issues set I and a contract point ~c = (i1 , . . . , ik , . . . , in ) 2 I. We want to find a contract c~⇤ that maximizes the utility function (1). Assuming that uj is the local sub-utility of constraint j , we distinguish two types of messages: messages sent from issues to constraints, and messages sent from constraints to issues. From issue ik to constraint j : In (5), each message µik ! j coming from ik to j is the sum of the constraints’ messages to ik coming from constraints other than j . X µik ! j (ik ) = µ j0 !ik (ik ) (5) j 0 2N (ik )\ From constraint j to issue ik : Each constraint message is the sum of the messages coming from issues other than ik , plus the sub-utility uj (i1 , . . . , ik , . . . , in ), summed over all the possible values of the issues (connected to the constraint j ) other than the issue ik . (a) Utility space (b) Utility hyper-graph Figure 2: 2 dimensional utility space and its hyper-graph µ Example 2. Now, we consider a 10 dimensional utility space mapped into the hyper-graph G10 = (I, ) with I = {i1 , . . . , i9 } and = { 1 , . . . , S7 }, shown in Figure 3. Each issue ik has a set Ik = ⌫2N (ik ) Ik,⌫ where Ik,⌫ is an edge connecting S ik to its neighbor ⌫ ⇢ N (ik ) 2 . For example, I1 = ⌫2{ 1 , 3 , 6 } I1,⌫ = {[5, 9], [3, 4], [3, 6]}. The constraints are cubic ( 1 , 2 , 3 , 4 ), plane ( 5 , 6 ) and bell ( 7 ). Each constraint cj (resp. factor j ) is assigned a sub-utility function uj to find the utility of a contract if it satisfies cj by being located in hyp(cj ). Depending on its type, each constraint’s sub–utility is defined as in (3). 8 P j < j+ ' if Plane k=1 ↵j,k ik , j , ↵j,k 2 Z (3) uj vj if Cube : V if Bell j (i ) = max . . . max . . . max uj (i1 , . . . , ik , . . . , in ) k j !ik i1 + X ik0 6=k ik0 2N ( j )\ik in µi k 0 ! j (6) (ik ) The MP mechanism starts from the leaves of the hypergraph, i.e. the issues. At t = 0, the content of the initial messages is defined according to (7), with u0j (ik ) being the partial evaluation of ik in the factor j . (7) µik ! j (ik ) = 0 µ j !ik (ik ) = u0j (ik ) At any time, the optimal contract c~⇤ is found by collecting the optimal issue-values as in (8a). c~⇤ = (i⇤1 , . . . , i⇤k , . . . , i⇤n ) X i⇤k = arg max µ That is, the plane constraint j is defined using its 'j dimensional equation, while a cubic constraint is assigned a value vj . The sub-utility Vj of a bell shaped constraints is defined in (4). Herein, is the Euclidean distance from the center of the bell constraint to the contract point. Distances are normalized in [ 1, 1]. 8 < j (1 2 2 ) if < 1/2 j 2Z Vj (4) )2 if <1 2 j (1 j 2Z : 0 else 3 j ik j 2N (ik ) (8a) j !ik (ik ) (8b) In a strategic encounter between agents, it is more common that an agent requires a collection, or bundle, of the optimal contracts rather than one single optimum. In order to find such collection, we should endow (8b) with a caching mechanism allowing each node in the hyper-graph to store the messages that have been sent to it from the other nodes. That is, the cached messages will contain the summed-up utility values of the underlying node’s instance. This is performed every time max is called in (6) so that we can store the settings of the adjacent utility (and contract) that led to the maximum. Once ordered, such data structure could, for instance, be used in a bidding process. In the next section, we algorithmically provide the MP mechanism. Optimal Contracts The exploration of the utility hyper-graph is inspired from the sum-product message passing algorithm for loopy belief propagation (Pearl 1988). However, the multiplicative 51 u6 i0 i1 I1,6 = [3, 6] i3 I3,6 = [1, 4] 6 u5 I0,1 = [0, 4] i6 I1,1 = [5, 9] I6,1 = [5, 9] 1 5 i5 I5,5 = [4, 7] u3 I4,5 = [7, 9] 3 u1 I5,7 = [1, 3] I4,3 = [2, 7] i4 I7,1 = [0, 8] i7 I1,3 = [3, 4] I2,1 = [2, 3] u2 u7 I4,2 = [9] I2,2 = [0, 9] 7 u4 2 I8,2 = [2, 5] i2 4 i8 I9,4 = [1, 3] i9 Figure 3: Issues-Constraints Hypergraph 3.2 Utility propagation algorithm Input: G(I, ), j Output: Updated message µ begin µ [0] ⇥ |Ik | for ⌫ 2 N (ik )\ j do µ µ + ⌫.GetM sg() return µ Main algorithm. Algorithm 1 operates on the hypergraph nodes by triggering the MP process. Despite the fact that we have two types of nodes (issues and constraints), it is possible to treat them abstractly using M sgP ass. The resulting bundle is a collection of optimal contracts with utility greater or equal to the agent’s reservation value rv. Algorithm 2: MsgPass: Issue to constraint Algorithm: Utility Propagation issue. This needs to be performed for each value of the target variable in order to compute the message. If all issues are assigned (@i : ↵[i] = 1), the values of the factor and of all other incoming messages are determined, so that their sum term is compared to the prior maximum, as in Algorithm 4. The resulting messages, stored in bundle, contain the values that maximize the factors’ local utility functions. Input: G = (I, ), rv, mode, ⇢ Output: Optimal contracts (bundle) begin for i = 1 ! (⇢ ⇥ |I [ |) do if mode is Synchronous then foreach ⌫src 2 I [ do foreach ⌫dest 2 ⌫src .N eighbors() do ⌫src .M sgP ass(⌫dest ) Input: G(I, ), ik Output: Updated message µ begin ↵ [ 1] ⇥ 'j ◆ ⇡i k ( j ) if ◆ = ; then µ ik .GetM sg() for i = 1 ! len(µ) do ↵[◆] i µ[i] Sum(↵, ik ) return µ bundle ; foreach i 2 I do bundle[i] ; ◆ [j2i.instances() [j.min, j.max] k⇤ 1 µ⇤ µ i.getmax() foreach k = 1 ! |µ| do if µ⇤ < µ[k] then µ⇤ µ[k] k k⇤ if µ⇤ rv then bundle[i] bundle[i] [ ◆[k⇤ ] Algorithm 3: MsgPass: Constraint to Issue return bundle Algorithm 1: Main Algorithm Optimal issue-values. At any time of the utility propagation process, it is possible to collect the current optimal contract(s) by individual concatenation of all the optimal issuevalues i⇤k , defined in (8b). Particularly, the summation in (8b) is performed as to only include the overlapping evaluations depending on how the issue domains are defined for different factors. For instance, Figure 4 shows how issue ik has three possible evaluations depending on Ik,1 , Ik,j and Ik,m . The maximization objective (8b) will attempt to find the Issue to Constraint. The issue’s message to a factor (or constraint) is the element-wise sum of all the incoming messages from other factors, as shown in Algorithm 2. Constraint to Issue. In Algorithm 3, the factor’s message to a targeted issue is done by recursively enumerating over all variables that the factor references (6), except the targeted 52 Input: G(I, ), ↵, ik Output: max begin ◆ {i | i 2 [1, 'j ] ^ ↵[i] = if ◆ = ; then ⇢ M j [↵] Algorithm: ParamRandHGen Input: n, m, p Output: G(I, ) begin [ min , max ] [1, 100] // constants [↵min , ↵max ] [0, 1] // slopes [0, 9] // bounds [bmin , bmax ] [;] ⇥ m // init constraints set for k = 1 ! m do [k].✓ rand({cube, plane, bell}) if [k].✓ = plane then ↵ [0] ⇥ n ↵[j] rand([↵min , ↵max ]) 8i 2 [1, n] [k].↵ ↵ 1} for ⌫ 2 N ( j )\ik do µ ⌫.GetM sg() ⇢ ⇢ + µ[↵[i]] return ⇢ else max 1 for i = 1 ! dim(M j (◆)) do ↵[◆] i Sum(↵, ik ) if > max then max if [k].✓ 2 {bell, cube} then // refer to (3) or (4) [k]. rand([ min , max ]) µ rand([1, n]) , I ; while |I| 6= µ do ◆ rand([1, p]) if ◆ 2 / I then I I[◆ return max Algorithm 4: Sum: recursive summing combination(s) of v1,i , vk,i and vm,i that maximize the sum. An optimal combination is an optimal issue-value i⇤k . 0 v1,0 i vk,i 0 0 i vm,i 0 0 1 v1,1 1 vk,1 1 0 2 vm,2 2 v1,2 2 vk,2 3 vk,3 3 0 3 vk,3 4 vk,4 4 0 4 vk,4 5 vk,5 5 0 5 0 6 0 7 0 8 0 9 0 6 0 7 0 8 0 9 0 6 vk,6 7 vk,7 8 0 9 0 [k].I ik 2 hyp(cj ) Evaluations of issue ik i v1,i for j = 1 ! µ do I[j].a rand([bmin , bmax ]) I[j].b rand([I[j].a + ✏, bmax ]) Ik,1 = [0, 2] ik Ik,j = [1, 4] Ik,m = [2, 7] Algorithm 5: Utility Hypergraph Generation 1 from Ik,j 8ik 2 N ( j ). Our comparison criteria is based on the utility/duration performed on a set of profiles of the form (n, m, p), with 100 trials for each profile. The first version of the message passing process, SynchMP, is synchronous. That is, all the combinations of issue evaluations are generated deterministically. Figure 5 illustrates the performance of SynchMP for (10, [10, 20, 30], 5). Indeed, the deterministic aspect of SynchMP makes it very slow ( SA << SynchM P ) compared to its SA counterpart which exploits the randomization, allowing it to perform “jumps” in the search space. To avoid the enumeration over local nodes of G, it is possible to add randomization to the way nodes are selected. To introduce an asynchronous mode, AsynchMP, we add a new mode after the synchronous mode condition in Algorithm 1, as follows: j m Figure 4: Finding the optimal issue-values 4 4.1 Experiments Settings Before evaluating the utility propagation algorithm, we identify the criteria that could affect the complexity of the utility space and thus the probability of finding optimal contract(s). Other than n and m, we distinguish p, defined as the maximal number of issues involved in a constraint. p can be unary (p = 1), binary (p = 2), ternary (p = 3), or p-ary in the general case. The parametrized generation of a utility space (or utility hyper-graph) must meet the consistency condition p n m ⇥ p, with n, m, p 2 N+ , to avoid problems like attempting to have an 8 ary constraints in a 5 dimensional utility space. 4.2 I return if mode is Asynchronous then rand2 ([1, |V |]), ⌫dest 6= ⌫src ⌫src , ⌫dest ⌫src .M sgP ass(⌫dest ) For (40, [20, . . . 100], 5), Figure 6 shows the resulting difference in the performance of AsynchMP compared to SA. 5 Discussion We start with the generation of the hyper-graph using Algorithm 5. Then, we will compare the MP mechanism in terms of utility and duration to the Simulated Annealing (SA) approach in (Ito, Hattori, and Klein 2007) for optimal contract(s) search. The SA optimizer will be randomly sampling from the regions that correspond to an overlap of constraints. For instance, generating a random contract satisfying cj is performed backwardly through random generation of values Conclusion We proposed a new representation of constraint-based utility spaces based on hyper-graphs that allows a modular decomposition of the constraints and issues. The exploration and search for optimal contracts is performed based on a message passing mechanism outperforming the sampling based optimizers. As future work, we intend to exploit the structure of the hyper-graphs to induce a hierarchical type of automated bi- 53 (a) Utility (a) Utility (b) Durations (b) Durations SA and SynchM P for m 2 {10, 20, 30} SA and ASynchM P for m 2 [20, 100] Figure 6: AsynchMP vs. SA for (40, [20, . . . 100], 5) Figure 5: SynchMP vs. SA for (10, [10, 20, 30], 5) and Velasco, J. R. 2011. A region-based multi-issue negotiation protocol for nonmonotonic utility spaces. Computational Intelligence 27(2):166–217. Marsa-Maestre, I.; Lopez-Carmona, M. A.; Velasco, J. R.; and de la Hoz, E. 2009. Effective bidding and deal identification for negotiations in highly nonlinear scenarios. In Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems - Volume 2, AAMAS ’09, 1057–1064. Richland, SC: International Foundation for Autonomous Agents and Multiagent Systems. Marsa-Maestre, I.; Lopez-Carmona, M.; Carral, J.; and Ibanez, G. 2013. A recursive protocol for negotiating contracts under non-monotonic preference structures. Group Decision and Negotiation 22(1):1–43. Pearl, J. 1988. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc. Robu, V.; Somefun, D. J. A.; and Poutre, J. L. 2005. Modeling complex multi-issue negotiations using utility graphs. In Proceedings of the 4th International Joint Conference on Autonomous Agents and Multi-Agent Systems (AAMAS 2005), 280–287. lateral negotiation. Additionally, we think about studying the issues’ importance and how they might affect the final outcome of an automated negotiation. References Bacchus, F., and Grove, A. 1995. Graphical models for preference and utility. In Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence, UAI’95, 3– 10. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc. Chajewska, U., and Koller, D. 2000. Utilities as random variables: Density estimation and structure discovery. In In Proceedings of the Sixteenth Annual Conference on Uncertainty in Artificial Intelligence (UAI-00), 63 – 71. Fujita, K.; Ito, T.; and Klein, M. 2010. An approach to scalable multi-issue negotiation: Decomposing the contract space based on issue interdependencies. In Proceedings of the 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology - Volume 02, WI-IAT ’10, 399–406. Washington, DC, USA: IEEE Computer Society. Ito, T.; Hattori, H.; and Klein, M. 2007. Multi-issue negotiation protocol for agents : Exploring nonlinear utility spaces. In Proceedings of the 20th International Joint Conference on Artificial Intelligence (IJCAI-2007), 1347–1352. Kwisthout, J., and van Rooij, I. 2013. Bridging the gap between theory and practice of approximate bayesian inference. Cognitive Systems Research 24(0):2 – 8. Cognitive Systems Research:Special Issue on ICCM2012. Lopez-Carmona, M. A.; Marsa-Maestre, I.; De La Hoz, E.; 54