From: AAAI Technical Report WS-97-03. Compilation copyright © 1997, AAAI (www.aaai.org). All rights reserved. Evolving Claudia Organizations of Agents V. Goldman Jeffrey S. Rosenschein Institute of ComputerScience The Hebrew University Givat Ram, Jerusalem, Israel clag@cs.huji.ac.il, jeff@cs.huji.ac.il Gameof Life (Gardner 1983). This game was invented by John Horton Conwayin 1970. It takes place in a two dimensional grid (assumedto be an infinite plane). Each cell in this board has eight neighboring cells. The game was designed as a one person game, where an initial configuration of the grid is submitted (i.e., specifying which cells are empty and which cells are occupied by a counter). From then on, the rules of the gamewill control and change the configurations of the grid, leading to interesting patterns of organization. There are three rules that define when an occupied cell remains occupied (i.e., it survives), when an occupied cell becomes empty(i.e., it dies), and when an emptycell becomes occupied (i.e., a birth occurs). Every organism with two or three neighboring organisms survives for the next generation. Each organism with four or more neighbors dies (i.e., it is removed) due to overpopulation. Every organism with one neighbor or none dies from isolation. Each empty cell adjacent to exactly three neighbors is a birth cell. An organism is placed on it at the next move. There were three knownbehaviors that could emerge in different configurations with these rules. There were stable populations, that did not change their structure once they got to it. There were configurations that faded away, i.e., the grid remained totally empty after a number of iterations of the game. And, a configuration could be periodic or oscillating (i.e., it is composedof subconfigurations that change amongthem in a definite cycle [larger or equal to one]). These examples stress the unpredictability of the results in Life even though the rules of the game are very simple, and seem predictable. In this paper, we present experimental results, that showinteresting patterns of organization of agents that might evolve, with similar characteristics to patterns evolved in the game of Life. Abstract Weinvestigate howagents can learn to becomeexperts, and eventually organize themselves appropriately for a range of tasks. Ouraim is to look at evolutionary processesthat lead to organizationsof experts. The distributed artificial intelligence (DAI) community has dealt with multiagent systems that organize themselvesin order to achieve a specific shared goal. Various organizations can arise that will effectively balance the load on the agents and improvetheir performance. Wehere look at the process of emergenceof an organization as a step that takes place prior to the execution of a task, and as a general process related to a range of problemsin a domain. To explore the ideas set forward, we designed and implementeda testbed based on the idea of the gameof Life. Wepresent experimental results that showdifferent patterns of organizations that might evolve in a multiagent system. Keywords: organization of agents, ary model, software agents evolution- Introduction The topic of organization and self-organization of agents has been dealt with in the Distributed Artificial Intelligence (DAI) communitywith regard to the division of labor amongagents who are trying to achieve some shared goal. Weare interested in how agents can learn to organize themselves and in particular, how the agents in a multiagent environment can learn to become experts. Wesuggest an algorithm that influences the agent population in the system, and the system evolves expert agents in a domain. The proposed algorithm is performed as a step prior to problem solving in the domain. After the agents have become experts, other applications or the agents themselves can address these experts to get assistance or tasks resolved. Wehave designed and implemented a testbed to experiment with evolving organizations of agents. The implementationof this testbed is along the lines of the The Organization of Agents The organization and self-organization of agents operating in multiagent systems has been studied in the 25 DAIcommunity, in relation to the agents’ tasks. A good organization will balance the load on the agents, and will cause the agents to improve their performance. Most work published on this topic considers the organization of the agents as a meansthat enables the agents to better adapt to their environment. Ishida (Ishida 1992) proposes a model in which agents organize themselves by decomposing the problem they have to solve to achieve a shared goal. He calls this approach organization centered problem solving. Each agent always evaluates its actions based on how each action contributes to the agent’s goal, and on how each action contributes to the other agent’s goal. The process by which agents reorganize themselves is by changing their current goals, i.e., refining their knowledge, or partitioning a goal into subgoals. Another model for self organization was presented by Guichard et al. (Guichard & Ayel 1994). They introduced two primitives of reorganization: the composition (i.e., regrouping several agents into one), and the decomposition (i.e., the creation of several new agents). An agent decomposes itself when it reasons by applying its mechanismof introspection and it concludes that it is unable to perform its tasks adequately. They didn’t explain formally any specific mechanism, but they mention several examples of it based on the number of tasks, the importance of tasks, or the required time for resolving the problem. Huberman and Hogg (Huberman & Hogg 1995) look into communities of practice, i.e., informal networks that generate their own norms and interaclion palterns. They consider a group of individuals that are trying to solve a problem. Individuals can interact with one another, and they might do so with an interaction strength proportional to the frequency with which they exchange information with each other. At each step individuals can choose to work on their own or use information or other help from others in the community. They suggest characterizing the structure of a community by counting the number of neighbors an individual has, weighted by how frequently they interact. They then study the general evolution of such networks and find the conditions that allow the community to achieve the optimal structure purely through adjustments. This requires each agent to learn about the hints it might get from other agents and to learn to use them effectively. Nagendra Prasad et al. (Nagendra Prasad, Lesser, Lander 1996) investigated the usefulness of having heterogeneous agents learning their organizational role in a multiagent parametric design system. The problem the agents are faced with is to find a design, i.e., to find a set of values for the set of parameters of the prob26 lem. The agents can initiate designs, extend or critique them until they get to a final design that is mutually accepted by all of them. In the learning study they considered only the first two roles. Each of these roles is defined as a set of tasks that the agent has to perform in a composite solution that solves a given problem. In (Nagendra Prasad, Lesser, & Lander 1996), is noted that the designer of the system cannot know beforehand the best assignments of roles to the agents because he lacks knowledge about the solution distribution in the space. The agents went through a training period in which they evaluate their possible roles at each state while searching for a solution to specific problems. Then, the agents would select the role that maximizedtheir evaluation. This involves communicating with the other agents in the system to obtain information required to calculate the values of the evaluation function. In addition the actual performance measure of each agent’s choice is only knownwhen the search for the solution is completed. Then, the agents backtrace their evaluations and assigned them to the roles. The agents are trained for all possible states of a search. In all the cases reviewed in this section, the reorganization of a group of agents is strictly related to a specific problem the agents are trying to solve. The study presented in (Nagendra Prasad, Lesser, ~z Lander 1996) is moregeneral, in the sense that a still predefined numberof agents is trained for all the possible states that could occur while searching for a solution for a multiparametric design. The changes produced in these organizational structures are induced by a decrease in performance (i.e., they will reorganize because their performance has decreased, in order to improve it). In summary, the agents search for an internal organization while achieving a shared goal, balancing the load imposed on them as a group, and improving their performance. The Multiagent Testbed Wehave designed and implemented a testbed to experiment with evolving organizations of agents. The rules of our testbed were designed along the lines of the game of Life. The main addition to the game is the consideration of the environment in which the agents grow, die, or live. In our current implementation we choose the information domain, consisting of documents from a given source (e.g., a site on the Internet, files from directory, from an archive, etc.). Therefore, the rules of death, birth, and survival were defined based on the resources (i.e., the information in the documents) that the agents hold. In this testbed, we deal with homogeneous agents coming from a general class of agents. There are two main concepts that need to be defined in the game of Life: the neighborhood relation and the rules of the game. In our case, we compute the nearness among the documents in the set on which the simulation is run. Each agent can access a matrix consisting of the proximity values calculated for any pair of documentsfound in the collection. For our implementation, we download all the documents at the corresponding site. Wethen build an inverted index of all the words that appear in these documents. Westore, for each word, the total number of times the word appeared in all the documents together, the total number of documents in which the word appeared, and for each document that the word appeared in, the total number of appearances in that document. Afterwards, we compute the proximity values amongall the documents collected, and build the neighborhood matrix out of these values. Currently, we have implemented two relations that decide when two documents are considered neighbors. In both cases, the neighborhood relation implies a gradual relation (in contrast to a binary relation, in which we could only say whether two documents are neighbors or not). One neighborhood relation considers the words that appear in the documents, that is, two documents are neighbors when they are semantically similar. The other relation considers the links each documentpoints to. That means that two documents will be considered neighbors if both of them point to mutual links. The matrix is a square matrix with rows and columnsequivalent to the number of documents collected. Each cell is initialized with zero. For each word that appears in the inverted index, and documents Doci, and Docj in whichthe wordappears, the value of the cell [i, j] is incremented by an amount Incr[word, Doci, Docj] based on the Inverse Document Frequency (IDF) weight formula (Salton & McGill 1983). The IDF formula assumes that the word’s importance is proportional to the occurrence frequency of each word in each document and inversely proportional to the total number of documents in which the same word appears. The evolutionary engine program simulates a system populated with agents that gather resources (e.g., documents). The main algorithm for evolving an organization of experts is presented in Fig. 1. C[i] denotes the current set of documents owned by agent i. AgentsOnDocsis the number of agents already holding documents that are neighbors of the documents that agent i is holding. Free is the set of documentsthat are not owned yet by any agent in the current population. Any simulation starts with an initial number of agents set by the user. The user also assigns an initial document to each agent. Then, the agents collect more For each agent Ai in the agents list: Find NumDocsdocuments that are the closest neighbors to the documents already in C[i] Find out the size of AgentsOnDocs Compute PopulationDensity = Aa,nt~OnDoc, NurnDoc* If (MyDocsi < MinDocs) and (PopulationDensity > HighPopDensity) Ai returns its documents in C[i] to Free A~ dies If (MyDocsi >__ MaxDocs) and (PopulationDensity < LowFopDensity) Ai spawns a new agent A~i di divides C[i] into C[i’] and C[i"] If (MyDocsl < MaxDocs) and (PopulationDensity < HighPopDensity) Ai adds to its current C[i], the NumAdded closest documents from NumDocs that are not owned by any other agent. Free is updated accordingly else do nothing Output Ai’s current documents in C[i] Figure 1: The main loop of the evolutionary engine documents that are close enough to their seed document, based on the neighborhood matrix. The number of additional documents is set by the user. From then on, the agents will be involved in a loop in which they might be able to collect more documents from the given collection, they might die when they have very few documents in relation to the other agents, they might split into two agents if they have too manydocuments in relation to the other agents, and they might do nothing. This main loop ends after an iteration in which all the agents do nothing, i.e., no changes were made to the sets of documents the agents hold so far. During the simulation, each agent is described by a personal windowin which the current addresses of the documents it was assigned are presented. At the end of the simulation, all the sets of documents held by each agent, together with the words that characterize them, are copied to HTML files. A set of parameters can be set by the user to test different scenarios, initial conditions of the system, and different thresholds needed by the rules that will influence the behavior of the agents. Experimental Results In the game of Life four main behaviors were reported and described (Gardner 1983): stable, periodic or oscillating, fading, and ever growing patterns of organization. Wewill show similar results in our testbed, considering also the domain we choose. We built a matrix of the neighborhood relation among 166 doc27 uments taken from the Computer Science site at the HebrewUniversity (the URLof the root of the site is http://www.cs.huji.ac.il). The parameters that can be set by the user include NumDocs(i.e., the number of neighbor documents the agent will collect at most at each iteration), NumAdded(i.e., the number of the nearest documents the agent might add to its current cluster [chosen from NumDocs]), HighPopDensity (it determines when the population is overcrowded), MaxDocs(i.e., the ma×imumnumber of documents that each agent might hold in its cluster), MinDocs(i.e., the minimumnumber documents that an agent can have). Stable Patterns The stop condition of our algorithm is a state in which every agent has nothing to do. At this point, the agents got to a stable state, i.e., we found a stable pattern of organization. Each agent holds a defined cluster of documents. The system stabilized in a pattern of two agents for the following parameters: the number of documents added at each iteration was three, there were two agents in the initial population, the threshold for high population was 0.2, the maximal number of documents was 16 and the minimumnumber of documents was set as 10. The same result was gotten for a different initial number of documents that each agent held (i.e., for 2, 5, and 8 initial documents). When the minimal number of documents was changed to 5, a stable pattern of 3 agents was gotten for an initial number of documents ranging from 1 to 8. These three agents in the system were responsible for three topics respectively: material about writing in HTML,pages about a specific project developed in the Distributed Systems Laboratory, and pages about the learning group at the HebrewUniversity. Whenthe threshold for the high population was increased to 0.6, we got two different sets of stable patterns. One was reached with 8 agents when the initial numberof documents was 1, 2, 4, 5, 7. A pattern often agents was reached for 3 or 6 initial documents. The eight agents were responsible for pages on HTML,the programming laboratory, the Data Base group and information for students, the Distributed Systems group, three agents for three different exercises material in a course about learning, material about the learning group and additional material about the Distributed Systems group, respectively. Whenthe threshold for high population density remained at 0.6 and the minimal number of documents was raised to 10, we got three different sets of stable patterns: seven agents composed the final structure starting from 1, 4, or 7 initial number of documents, 28 five agents were the result starting from 2 or 5 initial documents, and six agents resulted when the initial two agents started with 3 or 6 initial numberof documents. In Figures 2,3,4,5, the numberof agents in the organization is presented as a function of the initial number of seed documents given to each of the two agents in the initial population. Wedenote a loop between x agents by a value of-x. For example, in Fig. 2, when there were three documents given to each agent, the organization arrived at a group of one agent and a loop between two other agents. ]n Figures 6, 7, we present the number of agents in the organization that evolved from an initial population of two agents that were given five initial documents. "da~.l" -- ................. l................ i.............. 4 5 [hill nurnbeT ofdocuments Figure 2: HighPopDensity=0.2, MaxDocs=16, MinDocs=10, 2 agents in the initial population * 2 i 3 i t i 4 5 6 Init~lnumber of documents i 7 I B Figure 3: HighPopDensity=0.2, MaxDocs=16, MinDoes=5, 2 agents in the initial population Ever Growing Patterns To achieve a population of agents with an unlimited size, we would need an infinite source of information. For example, for the domain we have chosen, we would need a site or a group of sites that grow all the time. "data,6" -"daq 3"-- o -2 .I ¯2O I I 3 4 6 k~lllalnumber Qfdocuments i I i 2 i 3 i i i 4 5 6 7 initial number ol documents 8 Fading AwayPatterns Wewill consider two cases. In one, we will check whether the whole population of agents can vanish and we will also show cases where agents can die in a system that survives. For the following parameters: threshold for the high population set as 0.2, maximal number of documents as 16, minimal number of documents as 10, three documents added at each iteration, two agents in the initial population with 39 or 40 initial documents, the agents vanished. The same result was achieved for high population threshold set to 0.6 and minimal number of documents set to five, and also for five as the minimal number of documents and the threshold set to 0.2. Wealso got results that show that agents can die, when the appropriate condition is fulfilled. 4 2 C 1 i i i , 15 20 25 documents added ateach Iteration 35 In our current simulation, the site of documents is finite. There is a possibility in which, for example, one agent splits into two agents, one of them dies, and the remaining agent repeats this process of splitting into one agent that dies, and the other agent splits again and so forth, without stopping. Wewill present this example as a blinker pattern. In the specific example, the agent that splits itself (i.e., the father) and the agent that remains alive and will reproduce exchange the clusters of documents they hold. In our implementation, each time an agent reproduces, its two sons get two new names, but we can think of an agent spawning itself into another agent, and the same agent remains alive with part of its original documents. Therefore in our current implementation and given the domain as a finite site of documents, we cannot reach an ever growing pattern of organization. ¯ dala.5" -- 5 3O Figure 7: HighPopDensity=0.2, MaxDocs=16, MinDocs=10,2 agents in the initial population, five initial documents for each agent Figure 5: HighPopDensity=0.6, MaxDocs=16, MinDocs=5, 2 agents in the initial population ,2 0 i i J i to documents 15 20 25 added ateach IteratiO~l I 7 Figure 4: HighPopDensity=0.6, MaxDocs=16, MinDocs=10, 2 agents in the initial population i I i 5 i 3o 35 Figure 6: HighPopDensity=0.6, MaxDocs=16, MinDocs=10,2 agents in the initial population, five initial documents for each agent Oscillating Patterns (Blinkers) Oscillating patterns in multiagent systems are those patterns consist29 ing of agents that pass their documentsfrom one agent to another continuously. For example, when the number of documents added per iteration was three, the threshold for high population was 0.2, the maximal number of documents was 16 and the minimal number of documents was 10, there were two agents in the initial population, we got a blinker pattern for different initial numbers of documents (1,3,4,6,7). The system entered a loop one agent holding documents related to the learning group and another agent holding documents related to a project in the Distributed Systems group. Every three iterations of the simulation, the agent that survived held one of these two clusters, and after three more iterations, the agent that remained alive held the other cluster. Another case was when the threshold for the high population was 0.6, the minimal number of documents was 10, and the initial number of documents was eight. The system evolved into five agents (responsible for the Data Base group and information for the students, an exercise in a course about learning, the programming laboratory course, the learning group, the Distributed Systems group), and a loop between two other agents (one cluster included material about a specific exercise in the learning course, and the other about tools for the learning course). Other behaviors analyzed in Life (Gardner 1983) concern distinct kinds of collisions amongdifferent patterns. This is also relevant in our scenario. There occurs a collision betweenclusters of documents(i.e., between the agents responsible for them) when one agent (or several agents) dies, and its documents are redistributed between the other agents (i.e., other agents take control over these documents like a Life pattern that eats another pattern, or when we get to oscillating behavior (i.e., agents pass their documentsbetween themselves in a cycle). Conclusions Wepropose an algorithm that evolves different patterns of organization in a multiagent system considering the environment of the agents. The algorithm affects the population of agents, i.e., their growth, death and survival. Each agent collects information from the domain, that eventually will becomeits domain of expertise. Weshow different interesting patterns of organization that agents can evolve (i.e., stable, periodic and vanishing behaviors). In our model, the agents get their expertise before they deal with any goal or task. Weare solely interested in agents that learn to become experts, and not with the load balancing or solving of a specific problem. These patterns of organization that evolve in a prior step, are useful as a source of information. Specific agents can be addressed to answer queries on the knowledge they have. For example, the organization can be approached by a user or another agent application to retrieve information related to a given query. Another exampleis to regard this organization as a library of reusable plans (Decker & Lesser 1994). Moreover, these organizations serve as a preparing stage before the agents divide the tasks amongthemselves. Each agent can achieve goals that are more related to its expertise in a more efficient way. Acknowledgments This work has been supported in part by the Israeli Ministry of Science and Technology (Grant 032-8284) and the Israel Science Foundation (Grant 032-7517). Claudia V. Goldmanis supported by the Eshkol Fellowship, Israeli Ministry of Science. We’d like to thank Ofer Margoninski for implementing the system for analyzing the structure of sites on the Internet. References Decker, K., and Lesser, V. 1994. Designing a family of coordination algorithms. Technical Report 94-14, University of Massachusetts, Amherst. Gardner, M. 1983. Wheels, Life and other mathematical amusements. W.H. Freeman and Company. Guichard, F., and Ayel, J. 1994. Logical reorganization of dai systems. ECAI94 Workshop on Agents, Theories, Architectures and Languages LNAI. Huberman, B. A., and Hogg, T. 1995. Communities of practice: Performance and evolution. Computational and Mathematical Organization Theory 1:73-92. Ishida, T. 1992. The tower of babel: Towards organization centered problem solving. In Proceedings of the Eleventh International Workshop on Distributed Artificial Intelligence. Nagendra Prasad, M.; Lesser, V.; and Lander, S. 1996. Learning organizational roles in a heterogeneous multi-agent system. In Proceedings of the Second International Conference on Multiagent Systems. Kyoto, Japan: AAAIPress. Salton, G., and McGill, M. 1983. Introduction to Modern Information Retrieval. McGrawHill International Books.