USING BAYESIAN NETWORKS TO MODEL AGENT RELATIONSHIPS

Transcription

1 Ó Applie ArtiÐcial Intelligence, 14 :867È879, 2000 Copyright 2000 Taylor & Francis /00 $ USING BAYESIAN NETWORKS TO MODEL AGENT RELATIONSHIPS BIKRAMJIT BANERJEE, ANISH BISWAS, MANISHA MUNDHE, SANDIP DEBNATH an SANDIP SEN Mathematical an Computer Science Department, University of Tulsa, Tulsa, Oklahoma, U.S.A. An agent-society of the future is envisione to be as complex as a human society. J ust like human societies, such multiagent systems (MAS) eserve an in-epth stuy of the ynamics, relationships, an interactions of the constituent agents. An agent in a MAS may have only approximate a priori estimates of the trustworthiness of another agent. But it can learn from interactions with other agents, resulting in more accurate moels of these agents an their epenencies together with the inñuences of other environmental factors. Such moels are propose to be represente as Bayesian or belief networks. An objective mechanism is presente to enable an agent elicit crucial information from the environment regaring the true nature of the other agents. T his mechanism allows the moeling agent to choose actions that will prouce guarantee minimal improvement of the moel accuracy. T he working of the propose maximin entropy proceure is emonstrate in a multiagent scenario. Multiagent systems (MAS) may consist of self-intereste agents with iniviual goals. Agents in a MAS often have limite, specialize capabilities an have to epen on other agents to achieve their goals. An agent is usually embee in a complex, ynamic, an uncertain environment teeming with scores of others, some of whom may be past an/or potential interactors. Each agent may be riven by a plethora of objectives, though its resultant behavior can be interprete in the context of a single rational goal of maximizing utility. In the absence of any well-establishe coe-of-conuct for agent relationships, or enforcement of behavioral norms, agents can often Ðn it lucrative to exploit another agent to maximize local utility, whenever the situation permits. Given such a hostile environment, it becomes crucial for an agent to know whom to trust. The eðnition of trust, accoring to This work has been supporte, in part, by an NSF CAREER Awar IIS Aress corresponence to Sanip Sen, Department of Mathematical an Computer Sciences, University of Tulsa, 600 South College Avenue, Tulsa, OK, sanip@kolkata.mcs.utulsa.eu 867

2 868 B. Banerjee et al. Gambetta (1990) stresses that it is funamentally a belief or estimation. Castelfranchi an Falcone (1998) exten this eðnition to inclue the notion of competence along with preictability. One way of ientifying trustworthiness of other agents is by eveloping an eploying mechanisms to moel other agents. The goal is to preict the behavior of other agents. Builing etaile, up-to-ate, an accurate moels, however, is time-consuming an a potential etractor from actual problemsolving. The moel-builing process has three components : aopting as a priori or initial moel engaging or observing the agent in informative interactions upating the initial moel base on such interactions. Each of these components involve signiðcant time an computational cost commitments on the part of the moeling agent. The key is to estimate the true nature of other agents in as few interactions as possible. Recently, agent moeling has receive increasing attention from MAS researchers. Several probabilistic mechanisms have been evelope to moel agents (Sanholm & Crites, 1995 ; Zeng & Sycara, 1997). Some of these moels have been use to explore opponentsï strategies (Carmel & Markovitch, 1998). An agent using such a mechanism moels othersï strategies, which in turn, enables it to choose actions to maximize its payo. Very little work, however, exists on explicitly choosing actions that ai in the moelbuiling process. It is the plan to investigate mechanisms that will allow the moeling agent to choose actions to elicit maximal information from another agent about the latterïs trustworthiness. This shoul provie vital information in ealing with the other agents. The use of bayesian networks is propose to capture the relationships among the agent ispositions an their actions. The moeling agent will use its observations in tanem with its moel to upate its belief about other agents. Some of the agent actions may be such that they can extract more information about other agents, though not necessarily proucing the highest immeiate returns. On the other han, there may be some other actions that are of immeiate beneðt to the agent but tell little about the other agents. Depening on how signiðcant an time-constraine the work at han is, the agent will have to trae o progress in problem-solving with upating its moel of other agents. To illustrate this traeo, a emonstrative example scenario will be use. Consier a situation where an agent A nees some ocuments that agent B has in its possession. A can either irectly ask B to give the ocument to A, or can ask BÏs boss to instruct B to give the ocument to A. The Ðrst action of A will eðnitely provie more information about BÏs ispositions, epening on whether B obliges or not. On the other han, the secon action of A

3 P Moeling Agent Relationships 869 may not reveal BÏs actual cooperativeness, because there is an extra level of uncertainty introuce ue to the meiation by BÏs boss. Hence, if B helps, it may be uner coercion, whereas if it oes not help, it may be because the boss forgot to entertain AÏs request an A may never know that. However, the secon action may be more likely to satisfy AÏs immeiate goal. If A has to choose between these two actions, it has to traeo between the likely immeiate gain by choosing the secon action, an long-term gain from the information extracte from B by virtue of the Ðrst action. In this case, such aitional knowlege is exclusive of high immeiate rewar, while in some other cases it may be a sie e ect of the selecte action. In this paper, the focus is only on how to iscover other agentsï nature (in the sense of trustworthiness) an problem-solving for utility maximization is not consiere. Bayesian networks can be use to moel action. BAYESIAN NETWORKS A bayesian network (Jenson, 1996 ; Charniak, 1991) is a graphical metho of representing relationships, i.e., epenencies an interepenencies among i erent variables that together eðne a moel of a realworl situation. Technically, it is a irecte acyclic graph (DAG) with noes being the variables an each irecte ege representing a epenence between two of them. In aition to its structure, a bayesian network is also speciðe by a set of parameters q that quantify the network. Consier a vector X of variables an an instantiation-vector x (that assigns a value x i to each variable X i in X from its omain D i ). If the imme- iate parents of a variable X i is the vector P, with its instantiation p, Xi xi then Pr[X 5 x q ] 5 P i Pr[X i 5 x i P Xi 5 p xi, q ]. When the instantiation is clear from the context, the above is also written as Pr[X q ] 5 i Pr[X i P Xi, q ]. This eðnes the joint istribution of the variables in X, where each variable X i is conitionally inepenent of its nonescenents given its parents or conitioning variables. Bayesian networks are useful in inference from belief structures an observations. For this purpose, an extension of Bayesian networks calle inñuence iagrams is actually consiere, which incorporate action an ecision noes besies moeling beliefs. Bayesian networks are use for representing belief structures, for the following major reasons :

4 Bayesian networks can reaily hanle incomplete ata sets. This is because bayesian networks o er a way to encoe the correlations among the input variables. Bayesian networks allow one to learn about causal relationships. This is useful to gain an unerstaning about a problem omain. In aition, it allows to make preictions in the presence of interventions. Bayesian networks in conjunction with bayesian statistical techniques facilitate the combination of omain knowlege an ata. Bayesian networks in conjunction with bayesian methos o ers an effi- cient an principle approach to avoiing overðtting of ata. Bayesian networks o er a metho of upating the belief or the probability of occurrence of the particular event for the given causes. 870 B. Banerjee et al. An example bayesian network for a negotiation scenario (Banerjee et al., 1999) is shown in Figure 1 to illustrate how agents can use such a network to moel others. In this particular example, A wants to sell its car to B, an as the negotiation for the price progresses, A upates its moel regaring the factors inñuencing BÏs ecision. In this paper, a similar moeling approach has been assume, albeit for ecision making. FIGURE 1. An example negotiation-scenario to illustrate the use of bayesian networks in moeling others.

5 à Moeling Agent Relationships 871 CHOOSING ACTIONS TO IMPROVE MODELS OF OTHERS The actions of agents in a multiagent environment can reveal their strategies to others. In most omains, agents are strongly couple in the sense that the actions of an agent can inñuence the utility of other agents. In an open environment, a self-intereste agent shoul be aware of the nature, ispositions, an priorities of other agents. Such knowlege can enable an agent to better plan its actions. Hence, in aition to performing its problem-solving tasks, an agents shoul try to elicit accurate knowlege about agents who can a ect its utility an with which it frequently interacts. Actions chosen from a particular subset of available actions, or a particular orer of the same action set, may reveal more information about the true nature of the agents more e ectively. Our goal is to evelop a mechanism for selecting the actions for the moeler so as to form better estimates of the nature of the others. The basic approach of eliciting information from or about an agent is as follows. It is contene that often there are actions that give out more information about an agentïs strategies than other actions. From the moeling agentïs viewpoint, one wants to recognize the scenarios or contexts that will result in the other agentsï choosing actions that reveal more information about their trustworthiness. The moeling agent shoul then, by its own actions, create the corresponing contexts as often as feasible, an to the extent that these o not signiðcantly etract from its regular problemsolving activities. One visualizes an information content in each action of an agent an eðnes it as below. n DeÐnition 1. Suppose an agent A has n available action {ai} 1 represente as noes in the moeling bayesian network. These nee not be istinct noes, but can be the i erent values of the same noe. One consiers the subset of parent-noes of an action noe a i enote by à (a i ) (i.e., à (a i ) k P ai ) which moel the ispositions of an agent like trustworthiness, cooperativeness, etc. T hen information content in action of agent A is given by E i a i E i 5 ^ Pr[à (a i ) a i ] * log 2 (Pr[à (a i ) a i ]). (ai) One notes that this quantity lies between 0 an 2 1, (see Figure 2) with the minimum occurring at maximum uncertainty regaring the possible events. This correspons to the situation which, in oneïs view, provies minimum information about the nature of A that B woul like to know to improve his moel. The moel of agent interaction that one consiers is a two-level game (Luce & Rai a, 1957), where the moeler (B) has to choose from a set of possible actions { ) 1m, which lea to the other agent (A) aopting from its

6 Ã 872 B. Banerjee et al. FIGURE 2. Plot of negative entropy-function, where the case that the probability of all options are equal (here two options) correspons to minimum information content. own set of actions. The jth action of agent A in response to the ith action of agent B,, is enote as a ij, to be chosen from the set {a ij } j 5 n j 5 1. The agent B moels the factors that inñuence AÏs action-choice incluing its own actions, as a bayesian network. The trustworthiness of A is one of the critical factors that guies AÏs response to BÏs actions. A maximin (Luce & Rai a, 1957) mechanism is presente that allows B to select actions that help it to form increasingly better estimates of AÏs trustworthiness, given its response to BÏs actions. The set of actions available to A in response to each action of B are known to B an the latter has prior estimates of the probabilities of factors a ecting each such action of A. Among these, Ã (a ij ) represents those parents of the action noe a ij that reñect the nature of A. Now given the prior probabilities of such factors, B computes the information content of AÏs action as a ij E ij 5 ^ Pr[Ã (a ij ) a ij ] * log 2 (Pr[Ã (a ij ) a ij ]), (aij) where Pr[Ã (a ij ) a ij ]) is compute accoring to Bayes rule as Pr[Ã (a ij ) a ij ]) 5 Ã * Pr[a ij (a ij )] Pr[Ã (a ij ] Pr[a ij ]

7 Moeling Agent Relationships 873 an since, in generalã (a ij ) are all inepenent of, Pr[à (a ij ) a ij ]) 5 à * Pr[a ij (a ij )] Pr[à (a ij )] Pr[a ij ] E ij can also be looke upon as a measure of the i erence between the prior an posterior probabilities of à (a ij ). Now, BÏs goal is to Ðn the action that has the maximum value for minimum information content across all of AÏs responses to the action, i.e., B wants to maximize the minimum guar- antee regaring the information obtaine from AÏs response to. To this en, B Ðrst computes the lower boun on extractable information associate with action as Last, B selects the action e i 5 min j {E ij }. that maximizes this lower boun as. : i 5 arg max k e k. If the prior probabilities are inaccurate, then with progressive interaction, the moeler improves its estimates of the nature (currently uner consieration) of the agent being moele, choosing actions such that convergence is achieve as rapily as possible. Finally, when the prior an posteriors converge, the moeler moves on to explore some other traits of the other, following the same process all over again. In aition to arbitrating between conñicting actions, this proceure also suggests a choice among unrelate actions, as is emonstrate in the following example. MODELING SCENARIO INVOLVING AGENT TRUSTWORTHINESS Now the use of the above-mentione proceure is illustrate with a typical agent-interaction scenario. An example is escribe where agent B has to select action for elicitation of maximal information about agent AÏs nature. In this case, one consiers an agent trustworthy if it respons positively to oneïs request for help. A negative response (refusal to help) will ecrease trust in that agent, in the absence of any efenable reason, e.g., that the agent was busy in something more crucial to its utility. This approach is, however, not limite to a particular eðnition of trust an can be use for other eðnitions as well. One only nees a characterization of the sequence of actions accoring to the eðnition aopte. In this example (see Figure 3), agent B has to choose from the following set of actions:

8 à 874 B. Banerjee et al. FIGURE 3. Bayesian network moel for the example situation. B asks A to help him (b 3 ) B invites A for a treat (b 1 ) B requests AÏs boss to ask A to help B (b 2 ). In this case, the other possible events capable of a ecting AÏs actions are: A may nee help an, hence, may o er help to others hoping for help in return (noe H in Figure 3). To keep matters simple, one oes not count this as one of the noes for the corresponing actions in the calculations. Whether A accepts BÏs invitation to a treat epens on whether A is soci- able. From Figure 3, it can be written in accorance with the notation as à (a 1 ). 1 A helps others with a reasonably high probability an without any com- pulsion if he is trustworthy or epenable. It is assume that this information about AÏs nature is of vital importance to B an so (from Figure 3) it can be written asã (a 3 ) an also asã (a 2 ). One assumes that one has prior probabilities of these events (all events are assume to be binary-value) from omain knowlege. From these prior

9 Moeling Agent Relationships 875 beliefs an conitional probabilities, one estimates the posterior beliefs of B regaring the nature of A, i.e., whether A is trustworthy or not. Illustration of the Action-Selection Proceure The subnetworks have been shown for each of BÏs available actions in Figures 4, 5, 6, incluing the respective conitional-probability tables. One notes that the probability values in the table of Figure 4 have lower values than corresponing elements in Figure 5 wherever the action noe of B has true value. This is because of the aitional uncertainties that were mentione earlier. However, the probability values remain ientical wherever the action noe has false value, because the other inñuencing factors are common an a ect AÏs ecision alike. Base on these probability values, B computes the posterior probability of A being trustworthy, given B selects action b 2 an A selects action a 2 1 as Pr[D a 2 1 b 2 ] 5 Pr[a 2 1 b 2 D] * Pr[D], Pr[a 2 1 b 2 ] where Pr[a 2 1 b 2 D] Pr[H] Pr[a 2 1 Hb 2 D] Pr[Ø H] Pr[a 2 1 Hb 2 D] Pr[a 2 1 b 2 ] Pr[HD] Pr[a 2 1 Hb 2 D] Pr[HØ D] Pr[a 2 1 Hb 2 D] Pr[Ø HD] Pr[a 2 1 Hb 2 D] 1 Pr[Ø HØ D] * Pr[a 2 1 Ø Hb 2 Ø D] Consequently, one has Pr[D a 2 1 b 2 ] Similarly, one calculates the following probabilities : Pr[D a 2 2 b 2 ] 0.166, Pr[D a 3 1 b 3 ] , Pr[D a 3 2 b 3 ] 0.1. Hence, one has E , E , E , E

10 876 B. Banerjee et al. FIGURE 4. Portion of the network of Figure 3 for action b 2. FIGURE 5. Portion of the network of Figure 3 for action b 3.

11 Moeling Agent Relationships 877 FIGURE 6. Portion of the network of Figure 3 for action b 1. The information content, as expecte intuitively, is higher for actions of A which allow B to upate its prior Pr[D] by the maximum amount (either increase or a ecrease), an B shoul choose the action that maximizes the minimal increase in this prior. One sees that e 2 5 min{ , } , e 3 5 min{ , } Clearly, action b 3 is preferre to b 2 for maximal upating of the prior prob- ability Pr[D], as contene earlier. In aition, one has also consiere the action b 1. It can be shown that Pr[S a 1 1 b 1 ] an Pr[S a 1 2 b 1 ] One has again assume the prior probability of A being trustworthy to be 0.5. Here, one Ðns that the action b 1 is the most favore among the actions available to B. With increasing exploration by B into AÏs trustworthiness, its estimates are going to be better. As B evelops more accurate estimates of AÏs trustworthiness, this improve knowlege allows B to be more e ective in its problem-solving activities. B can also ecie to explore other aspects of AÏs nature once an accurate estimate of AÏs trustworthiness has been evelope.

12 878 B. Banerjee et al. CONCLUSION In this paper, a mechanism has been presente to enable bayesian networks-base moelers to select actions that lea to more accurate moels about the nature of another agent. The mechanism involves the use of a maximin proceure for action selection that guarantees a minimum level of improvement in estimation of an agentïs trustworthiness irrespective of whatever action the latter selects. An illustration has been provie of the working of this proceure with a running example. The knowlege of another agentïs nature may be extremely signiðcant in guiing the moeling agentïs problem-solving activities, given the open an competitive environment it is situate in. The progress in problem-solving has been ignore an focus has been solely on exploring the nature of the other agents. An expansion on this moel is planne to incorporate the problem-solving criterion too, an an inication on how the traeo between these two metrics is to be achieve for action selection. This will provie a uniðe framework by which exploratory actions are incorporate as an integral part of routine problem-solving for achieving the goal of maximizing long-term utility. Work on multiple-level ecision-making is also planne where a multilevel tree structure is generate for each action available to an agent. The maximin action-selection metho is conservative in nature. To guarantee a certain improvement in moel estimate it can ignore large improvements. This approach is completely justiðe if the other agent knows that the moeler is trying to improve its moel, an is then eliberately trying to take actions to minimize such increases. When such an assumption is untenable, the moeler can choose the action that prouces the maximum average improvement. An interesting avenue woul be to experimentally evaluate the relative e ectiveness of the maximin an average metrics to select actions. NOTE 1 a i stans for the noe that can take values a ij " j. REFERENCES AI Magazine. Summer Special Issue on Bayesian Techniques, 20(2). Banerjee, B., S. Debnath, an S. Sen Using bayesian networks to ai negotiations among agents. In the working notes of AAAIÏ99, W orkshop on Negotiation : Settling ConÑicts an Ientifying Opportunities (also available as AAAI Technical Report WS-99-12), pp. 44È49, Castelfranchi, C., an R. Falcone Principles of trust for MAS: Cognitive autonomy, social importance, an quantiðcation. In Proceeings of the T hir International Conference on Multiagent Systems, 72È79, Los Alamitos, CA, IEEE Computer Society. Charniak, E., Winter Bayesian networks without tears. AI Magazine 12(4) : 50È63.

13 Moeling Agent Relationships 879 Carmel, D., an S. Markovitch How to explore your opponentïs strategy (almost) optimally. In Proceeings of the T hir International Conference on Multiagent Systems, 64È71, Los Alamitos, CA, IEEE Computer Society. Gambetta, D T rust. Oxfor : Basil Blackwell. Jensen, F. V An introuction to bayesian networks. New York : Springer-Verlag. Luce, R. D., an H. Rai a Games an ecisions : Introuction an critical survey. New York : Dover. Russell, S., an P. Norvig ArtiÐcial intelligence: A moern approach. Englewoo Cli s, NJ: Prentice Hall. Shachter, R Evaluating inñuence iagrams. Operations Research 34(36) :871È882. Sanholm, T. W., an R. H. Crites Multiagent reinforcement learning an iterate prisonerïs ilemma. Biosystems Journal 37: 147È166. Zeng, D., an K. Sycara BeneÐts of learning in negotiation. In Proceeings of the 14th National Conference on ArtiÐcial Intelligence, 36È41, Menlo Park, CA, AAAI Press/MIT Press.