An Algorithm for Probabilistic Least{Commitment Planning 3. Nicholas Kushmerick Steve Hanks Daniel Weld. University ofwashington Seattle, WA 98195

Transcription

1 To aear, AAAI-94 An Algorithm for Probabilistic Least{Commitment Planning 3 Nicholas Kushmerick Steve Hanks Daniel Weld Deartment of Comuter Science and Engineering, FR{35 University ofwashington Seattle, WA fnick, hanks, weldg@cs.washington.edu Abstract We dene the robabilistic lanning roblem in terms of a robability distribution over initial world states, a boolean combination of goal roositions, a robability threshold, and actions whose eects deend on the execution-time state of the world and on random chance. Adoting a robabilistic model comlicates the denition of lan success: instead of demanding a lan that rovably achieves the goal, we seek lans whose robability of success exceeds the threshold. This aer describes a robabilistic semantics for lanning under uncertainty, and resents a fully imlemented algorithm that generates lans that succeed with robability no less than a user-sulied robability threshold. The algorithm is sound (if it terminates then the generated lan is suciently likely to achieve the goal) and comlete (the algorithm will generate a solution if one exists). Introduction Classical lanning algorithms have traditionally adoted stringent certainty assumtions about their domains: the lanning agent is assumed to have comlete and correct information about the initial state of the world and about the eects of its actions. These assumtions allow an algorithm to build a lan that is rovably correct: given an initial world state, a successful lan is a sequence of actions that logically entails the goal. Our research eort is directed toward relaxing the assumtions of comlete information and a deterministic action model, while exloiting existing techniques for symbolic least-commitment lan generation. This aer resents the buridan 1 lanning algorithm. buridan is a sound, comlete, and fully im- 3 We gratefully acknowledge the comments and suggestions of Tony Barrett, Tom Dean, Denise Draer, Mike Erdmann, Keith Golden, Rex Jacobovits, Oren Etzioni, Neal Lesh, Judea Pearl, and Mike Williamson. This research was funded in art by National Science Foundation Grants IRI and IRI , Oce of Naval Research Grant 90-J-1904, and the Xerox Cororation. 1 Jean Buridan (boo redan 0 ), , a French hilosoher and logician, has been credited with originating robability theory. He seems to have toyed with the idea of using his theory to decide among alternative courses of aclemented least-commitment lanner whose underlying semantics is robabilistic: a robability distribution over states catures the agent's uncertainty about the world and a mixed symbolic and robabilistic action reresentation allows the eects of action to vary according to the (modeled) state of the world at execution time as well as unmodeled (random) factors. Our lanner takes as inut a robability distribution over (initial) states, a goal exression, a set of action descritions, and a robability threshold reresenting the minimum accetable success robability. The algorithm roduces a lan such that, given a robability distribution over initial states, the robability that the goal holds after executing the lan is no less than the threshold. We have roved that the algorithm is sound (any lan it returns is a solution) and comlete (if there is an accetable lan the algorithm will nd it). The work reorted here makes several contributions. First, we dene a symbolic action reresentation and its robabilistic semantics. Second, we describe an imlemented algorithm for robabilistic lanning. Third, we briey describe our investigation of alternative lan assessment strategies. The aer concludes with a discussion of related work. This research is described in detail in the long version of this aer (Kushmerick, Hanks, & Weld 1993). Examle. The following examle will be develoed throughout the aer. Suose a robot is given the goal of holding a block (), making sure it is ainted (BP), and simultaneously keeing its grier clean (GC). Initially the grier is clean, but the block is not being held and is not ainted, and the grier is dry () with robability 0.7. Suose further that we will accet any lan that achieves the goal with robability at least 0.8. Finally, the robot has the following actions available: icku: try to ick u the block. If the grier is dry when the icku is attemted, executing this action will make true with robability If the grier is not dry, however, will become true only with robability 0.5. tion: the arable of \Buridan's Ass" is attributed to him, in which an ass that lacked the ability tochoose starved to death when laced between two equidistant iles of hay.

2 aint: aint the block. This action always makes BP true, and if the robot is holding the block when aint is executed, there is a 10% chance that the grier will become dirty (GC). dry: try to dry the grier. This action succeeds in making true 80% of the time. A Semantics for Probabilistic Planning We begin by dening a lanning roblem, and what it means to solve one. First we dening states and exressions, then actions and sequences of actions, then the lanning roblem and its solution. States & exressions. A state is a comlete descrition of the world at a single oint in time, and uncertainty about the world is reresented using a random variable over states. A state is described using a set of roositions in which every roosition aears exactly once, ossibly negated. 2 An exression is a set (imlicit conjunction) of literals. We dene the robability of an exression E with resect to a state s as: P[Ejs]= 1 if E s 0 otherwise For our examle, the world is initially in one of two ossible states: s1 = f; ; GC; BPg and s2 = f; ; GC; BPg, and the robability distribution over these states is characterized by a random variable ~s I as follows: P[~s I = s1] =0:7, P[~s I = s2] =0:3. Actions & action sequences. Our model of action, taken from (Hanks 1990; 1993), combines a symbolic model of the changes the action makes to roositions with robabilistic arameters that reresent chance (unmodeled) inuences. Fig. 1 is our reresentation of the icku action: if the grier is dry ( holds) at execution time, it makes true with robability 0.95, and with robability 0.05 makes no change to the world state. But if is false at execution time, icku makes true only with robability 0.5. Note that the roositions in the boxes refer to changes the action makes, not to world states. For examle, it is not correct to say that the holds with robability 0.95 after executing icku in a state where the grierisdry, since the robability of after icku is executed also deends on the robability of before execution (as well as the robability of before execution). Formally, an action is a set of consequences fht ; ;e i;:::;ht ; ;e ig. Each t is an exression called the consequence's trigger, is a robability, and e is a set of literals called the eects. The reresentation for the icku action is thus fhfg; 0:95; fgi; hfg; 0:05; fgi; hfg; 0:5; fgi; hfg; 0:5; fgig. 2 We use this reresentation for exository uroses only; an imlementation need not maniulate states exlicitly. In fact our lan renement algorithm has no exlicit reresentation of state: it reasons directly about the state's comonent roositions. (1) icku ρ α =0.95 ρ β =0.05 ρ γ =0.5 ρ δ =0.5 α β γ δ Figure 1: The icku action. We require that an action's triggers be mutually exclusive and exhaustive: 8 X s P[t j s] =1 (2) 8s; ; t 6= t ) P[t [ t j s] =0 (3) The notation A i; refers to consequence of action A i, and suerscrits refer to arts of a articular action: A i = f:::;ht i ;i ;eii;:::g. A consequence denes a (deterministic) transition from a state to a state, dened by a function result(e;s), where e is a set of eects and s is a state. This function is similar to add and delete lists in stris; see the long version of this aer for the full denition. An action A induces a change from a state s to a robability distribution over states s 0 : P[s 0 j s; A]= P[t j s] if ht ;;ei2a^s 0 =result(e ;s) 0 otherwise Since an action's triggers P are mutually exclusive and exhaustive, we have that s 0 P[s0 j s; A] = 1 for every action A and state s. We now dene the result of executing actions in sequence. The robability that a state s 0 will hold after executing a sequence of actions ha i i N i=1 (given that the world 2 was initially in state s) is dened as follows: 3 X 2 P s 0 j s; ha ii N i=1 = 00 P[s j s; A1]P s 0 j s 00 ; ha iii=23 N (5) s 00 where P[s 0 j s; hi] =1ifs 0 =sand 0 otherwise. Finally, we dene the robability that an exression E is true after an action sequence is executed beginning in some state s, and the robability of an exression after executing an action sequence given an initial robability distribution over states ~s I : P 2 Ejs; ha ii N i= P Ej~s I;hA ii N i=1 X 2 = P s 0 j s; ha iii=13 N P[Ejs 0] (6) s X 0 = s P 2 Ejs; ha ii N i=13 P[~sI = s] (7) Planning roblems & solutions. A lanning roblem consists of (1) a robability distribution over initial states ~s I, (2) a set of actions fa i g, (3) a goal exression G, and (4) a robability threshold. For the roblem described in this aer, the initial distribution ~s I was dened earlier, the lan can be constructed from the actions ficku; aint; dryg, the goal (4)

3 α is G = f; BP; GCg, and the robability threshold is =0:8. A solution to the roblem i is an action sequence ha i i N i=1 hgj~s if P I ;ha i i N i=1. As we'll see, hdry; aint; ickui is a solution to the examle roblem. The buridan Algorithm We now describe buridan, an algorithm that generates solutions to lanning roblems. buridan, like snl (McAllester & Rosenblitt 1991), searches a sace of artial lans. Each lan consists of a set of actions fa i g where each A i is one of the inut actions annotated with a unique index, a artial temoral ordering relation \<" over fa i g, a set of causal links, and a set of subgoals. The rst two items are straightforward, but there are imortant dierences between the last two items and the analogous snl concets. A link caches buridan's reasoning that a articular consequence of a articular action could make a literal true for a (later) action in the lan. The link A i;!a j records the fact that literal is a member of the trigger of one of action A j 's consequences (A j is the link's consumer), and the eect set of consequence of action A i (the link's roducer) contains. Action A k threatens link A i;!a j if the eect set of some consequence of A k contains, and if A k can be ordered between A i and A j. A lan's subgoals consists of the literals in the lan that buridan is committed to making true. A subgoal is a literal annotated with a articular action, Ai is a subgoal if the lan contains a link q A i;!a j and 2 t i. The set of subgoals is initialized to include all to-level goals. buridan begins searching from the null lan, which contains only two dummy actions A0 and A G, and the ordering constraint A0 < A G. These two actions allow buridan to comactly encode the lanning roblem: A0 and A G encode the robability distribution over initial states and the goal exression, resectively. Fig. 2 shows the null lan for the examle develoed in this aer. The initial action A0 has one consequence for each state in the initial distribution ~s I that has nonzero robability. Its triggers are all emty, and the eect sets and robabilities are the states and robabilities dened by ~s I. The goal action A G has one consequence triggered by the goal exression G. The null lan AG as a subgoal for each 2G. G 0 goal initial ρ α =0.7 ρ β =0.3 {GC,, BP} ρ α =1,, GC, BP β,, GC, BP α SUCCESS Figure 2: A0 and A G encode the initial robability distribution and the goal. Starting from the null lan, buridan erforms two oerations: 1. Plan Assessment: Determine if the robability that the current lan will achieve the goal exceeds, terminating successfully if so. 2. Plan Renement: Otherwise, try to increase the robability of goal satisfaction by rening the current lan. Each renement generates a new node in the sace of artial lans. Signal failure if there are no ossible renements, otherwise nondeterministically choose a new current lan and loo. Rening a lan with conditional and robabilistic actions diers in two ways from classical causal-link renement algorithms (e.g. snl). First, snl establishes a single causal link between a roducing action and a consuming action, and that link alone ensures that the link's literal will be true when the consuming action is executed. Our lanner links one of an action's consequences to a later action. An action can have several consequences, though only one will actually occur. Furthermore, a single link A i;!a j ensures that will be true at action A j only if trigger t i holds with robability one. Therefore multile links may be needed to suort a literal: even if no single link makes the literal suciently likely, their combination might. We lose snl's clean distinction between an \oen condition" (a trigger that is not suorted by a link) and a \suorted condition" that is guaranteed to be true. Causal suort in a robabilistic lan is a cumulative concet: the more links suorting a literal, the more likely it is that the literal will be true. The concet of a threatened link is dierent when actions have conditional eects. Recall that A k threatens A i;!a j if some consequence of A k asserts and if A k can be ordered between A i and A j. buridan resolves threats in the same way that classical lanners do: by ordering the threatening action either before the roducer or after the consumer. But a lan can be suciently likely to succeed even if there is a threat, as long as the threat is suciently unlikely to occur. We can therefore resolve a threat in an additional way, by confrontation: if action A k threatens link A i;!a j, lan for the occurrence of some consequence of A k that does not make false. buridan does so by adoting that consequence's triggers as additional subgoals. Plan Renement buridan's lan renement ste generates all ossible renements of a artial lan. A lan can be rened in two ways: either resolving a threat to a causal link, or adding a link to increase the robability that a subgoal will be true. buridan chooses a threat or subgoal, and then generates ossible renements as follows: 1. If the choice is to add a link to the Aj, buridan considers all new and existing actions A i that have an eect set e i containing, adds a link A i;!a j, and orders A i < A j.

4 2. If the choice is to resolve a threat by action A k to link A i;!a j, buridan resolves the threat in one of three ways: demotion (order A k < A i if it is consistent to do so), romotion (order A j < A k if consistent), and confrontation (lan for the occurrence of a consequence of action A k that does not make false). Link creation, romotion, and demotion are analogous to renement insnl-like lanners, and we will not discuss them further. 3 Confrontation has no analogue in snl, however. The robability that link A i;!a j succeeds in roducing for A j is the robability that executing action A i actually realizes consequence and that each action reresenting a confronted threat realizes a consequence that does not make false. Since the consequences of action are mutually exclusive, making a non-threatening consequence more likely makes the threat less likely. buridan therefore confronts a threat by choosing a non-threatening consequence of the threatening ste, and adots its triggers as subgoals. We exlain confrontation in detail in the longer version of this aer. Examle. We now demonstrate how the lanner constructs a lan that will achieve the goal (holding a ainted block with a clean grier) with robability at least 0.8. We simlify the resentation by resenting a sequence of renement choices that lead directly to a solution lan. Planning starts with the null lan (Fig. 2). The subgoals for the null lan are f@ AG ; BP@ AG ; GC@ AG g. buridan suorts the rst AG,by adding a icku action, A1, and creating a link from its consequence: A1;!A G. Consequence of A1 has as a trigger, so buridan A 1 a subgoal. Suort for this subgoal is then rovided by a link from the initial action: A0;!A1. buridan next suorts the subgoal of having the block ainted, BP@ AG,by adding a new aint action, A2, creating a link A2; BP!A G, and adoting A2;'s A 2 as a subgoal. A link A0;!A2 is added to A 2. Notice that icku and aint are unordered, thus icku threatens the new link: if icku is executed before aint, then the block will be in the grier when aint is executed, making false the trigger of aint's consequence. buridan resolves this threat by romoting icku, adding the ordering constraint A2 < A1. Finally, buridan suorts the subgoal GC@ AG with the link A0; GC!A G, and resolves the threat osed by aint by confrontation: since consequence of A2 does not cause GC, buridan adots its A 2 as a subgoal. The resulting lan is shown in Fig. 3. The assessed robability of Fig. 3's lan is We ignore snl's searation renement, since it alies only to actions with variables. (as described below), which is less than =0:8, so buridan continues to rene the lan. Eventually the lan shown in Fig. 4 is generated; it has success robability 0.804, so buridan terminates. Plan Assessment The lan assessment algorithm decides whether a lan's robability of success exceeds the threshold. The forward assessment algorithm is a straightforward imlementation of the denition of action execution. 4 forward comutes the robability distribution over states roduced by \executing" each action in sequence, runing zero-robability states from the distribution for the sake of eciency. Comlicating assessment is the fact that a solution is dened in terms of a totally ordered sequence of actions, whereas a lan's actions might be only artially ordered. We can still comute a lower bound on the lan's success, however, by considering the minimum over all total orders consistent with the lan's orderings. This olicy is conservative in that it comutes the best robability that can be exected from every total order. The long version of this aer discusses these issues in more detail. Examle. We now illustrate how the forward lan assessment algorithm comutes the success robability for the lan in Fig. 3. forward starts with the inut distribution over initial states, ~s I, and comutes the following distribution over nal states (where A = haint; ickui is the lan's action sequence): P[f; ; GC; BPgj~s I;A] = 0:5985 P 2 f; ; GC; BPgj~s I;A 3 = 0:135 P 2 f; ; GC; BPgj~s I;A 3 = 0:0665 P 2 f; ; GC; BPgj~s I;A 3 = 0:015 P 2 f; ; GC; BPgj~s I;A 3 = 0:0315 P 2 f; ; GC; BPgj~s I;A 3 = 0:135 P 2 f; ; GC; BPgj~s I;A 3 = 0:0035 P 2 f; ; GC; BPgj~s I;A 3 = 0:015 The robability of the goal exression is then comuted by summing over those states in which the goal holds. G = f; GC; BPg is true in the rst two states, so we havep[gj~s I ;A]=0: :135=0:7335. Formal Proerties A least-commitment lanner roduces as outut a artial order over actions. Such a lanner is sound if every consistent total order of these actions is a solution to the inut roblem. The lanner is comlete if it always returns a solution lan if such a lan exists. In the longer aer we rove that buridan is both sound and comlete. 4 In this section we discuss one simle assessment strategy; later we introduce a variety of more comlicated algorithms.

5 1 icku α ρ α =0.95 ρ β =0.05 ρ γ =0.5 ρ δ =0.5 0 α β γ δ G goal initial ρ α =0.7 ρ β =0.3 {GC,, BP},, GC, BP β,, GC, BP 2 ρ α =1 aint α SUCCESS ρ α =0.1 ρ β =0.9 ρ γ =1 α BP, GC β BP γ BP, GC Figure 3: An artial solution to the examle roblem. Gray arrows indicate threats resolved by confrontation. 3 dry ρ α =0.8 ρ β =0.2 1 icku α β ρ α =0.95 ρ β =0.05 ρ γ =0.5 ρ δ =0.5 0 α β γ δ G goal initial ρ α =0.7 ρ β =0.3 {GC,, BP} α,, GC, BP β,, GC, BP 2 ρ α =1 aint α SUCCESS ρ α =0.1 ρ β =0.9 ρ γ =1 α BP, GC β BP γ BP, GC Ecient Plan Assessment Figure 4: A lan that is suciently likely to succeed. The forward assessment strategy, while simle, can be quite inecient, since the number of states with nonzero robability can grow exonentially with the length of the lan. This ineciency motivates a second focus of our research, an exloration of alternative assessment algorithms. Since the general lan assessment roblem is NP-hard (Chaman 1987; Cooer 1990) we cannot exect to roduce an assessment algorithm that runs eciently on every roblem in every domain. However, by exloiting the structure of the actions, goals and state sace, we can sometimes realize tremendous eciency gains. We have imlemented three alternative lan assessment algorithms. While the size of the state distribution may grow exonentially, in general not all of the distinctions between the dierent states will be relevant to whether the goal is true. So one alternative assessment strategy, called query, tries to divide states into subsets based on the truth of the goal. At best query needs to reason about only two subsets of states: those in which the goal is true and those in which it is false. Rather than maniulating states exlicitly, assessment algorithms can reason directly about the roositions that comrise the states. Thus the third assessment algorithm, network, translates the action descritions into a robabilistic network, each nodeof which corresonds to a single roosition at some oint during the execution of the lan. network then solves this network using standard roagation techniques. The resulting network tends to be more comlicated than necessary, however, since it exlicitly includes \ersistence" links encoding the fact that a literal remains unchanged across the execution of an action that neither adds nor deletes it. But recall that ersistence assumtions are already stored in a lan's causallink structures: an unthreatened link is a guarantee that the link's roosition will ersist from the time it is roduced until it is consumed. This motivates a fourth algorithm, reverse, that comutes a lan's success robability by maniulating the lan's causal link structure directly. The longer version of this aer describes these algorithms in detail. Analysis of these algorithms shows no clear winner: in each case we can construct domains and roblems in which one algorithm consistently outerforms the other.

6 Related Work and Conclusions (Mansell 1993) and (Goldman & Boddy 1994) oer alternative aroaches to alying classical lanning algorithms to robabilistic action and state models. (Dean et al. 1993), (Farley 1983), and (Koenig 1992) use fully observable Markov rocesses to model the lanning roblem. These systems generate an execution olicy, a secication of what action the agent should erform in every world state, and assume that the agent isrovided with comlete and accurate information about the world state. buridan roduces a lan, a sequence of stes that is executed without examining the world at execution time, and assumes the agent will be rovided with no additional information at execution time. A recent extension to buridan, (Draer, Hanks, & Weld 1994a; 1994b), strikes a middle ground: the reresentation allows actions that rovide ossibly inaccurate information about the world at execution time, and actions in a lan can be executed contingent on the information rovided by revious stes. Work in decision science has dealt with lanning roblems (see (Dean & Wellman 1991, Cha. 7) for an introduction), but it has focused on solving a given robabilistic model whereas our algorithm interleaves the rocess of constructing and evaluating solutions. But see (Breese 1992) for recent work on model-building issues. (Haddawy & Hanks 1992; 1993) motivate building a lanner like buridan, exloring the connection between building lans that robably satisfy goals and lans that are otimal in the sense of maximizing exected utility. Although lanning with conditional eects is not the rimary focus of our work, buridan also generalizes work on lanning with deterministic conditional effects, e.g. in (Collins & Pryor 1992; Penberthy & Weld 1992). A deterministic form of confrontation is used in uco (Penberthy &Weld 1992). The buridan lanner integrates a robabilistic semantics for action with classical least-commitment lanning techniques. buridan accets robabilistic information about the roblem's initial state, and maniulates actions with conditional and robabilistic eects. Our lanner is fully imlemented in Common Lis and has been tested on many examles. buridan takes about 4:5 seconds to nd a solution to the roblem resented in this aer. Send mail to bug-buridan@cs.washington.edu for information about buridan source code. References Breese, J Construction of belief and decision networks. Comutational Intelligence 8(4). Chaman, D Planning for conjunctive goals. Arti- cial Intelligence 32(3):333{377. Collins, G., and Pryor, L Achieving the functionality of lter conditions in a artial order lanner. In Proc. 10th Nat. Conf. on A.I. Cooer, G The comutational comlexity of robabilistic inference using bayesian belief networks. Articial Intelligence 42. Dean, T., and Wellman, M Planning and Control. Morgan Kaufmann. Dean, T., Kaelbling, L., Kirman, J., and Nicholson, A Planning with deadlines in stochastic domains. In Proc. 11th Nat. Conf. on A.I. Draer, D., Hanks, S., and Weld, D. 1994a. A robabilistic model of action for least-commitment lanning with information gathering. In Proc., Uncertainty inai. Submitted. Draer, D., Hanks, S., and Weld, D. 1994b. Probabilistic lanning with information gathering and contingent execution. In Proc. 2nd Int. Conf. on A.I. Planning Systems. Farley, A A Probabilistic Model for Uncertain Problem Solving. IEEE Transactions on Systems, Man, and Cybernetics 13(4). Goldman, R. P., and Boddy, M. S Esilon-safe lanning. forthcoming. Haddawy, P., and Hanks, S Reresentations for Decision-Theoretic Planning: Utility Functions for Dealine Goals. In Proc. 3rd Int. Conf. on Princiles of Knowledge Reresentation and Reasoning. Haddawy, P., and Hanks, S Utility Models for Goal- Directed Decision-Theoretic Planners. Technical Reort 93{06{04, Univ. of Washington, Det. of Comuter Science and Engineering. Submitted to Articial Intelligence. Available via FTP from ub/ai/ at cs.washington.edu. Hanks, S Practical temoral rojection. In Proc. 8th Nat. Conf. on A.I., 158{163. Hanks, S Modeling a Dynamic and Uncertain World II: Action Reresentation and Plan Evaluation. Technical reort, Univ. of Washington, Det. of Comuter Science and Engineering. Koenig, S Otimal robabilistic and decisiontheoretic lanning using markovian decision theory. UCB/CSD 92/685, Berkeley. Kushmerick, N., Hanks, S., and Weld, D An Algorithm for Probabilistic Planning. Technical Reort , Univ. of Washington, Det. of Comuter Science and Engineering. To aear in Articial Intelligence. Available via FTP from ub/ai/ at cs.washington.edu. Mansell, T A method for lanning given uncertain and incomlete information. In Proc. 9th Conf. on Uncertainty inartical Intelligence. McAllester, D., and Rosenblitt, D Systematic nonlinear lanning. In Proc. 9th Nat. Conf. on A.I., 634{639. Penberthy, J., and Weld, D UCPOP: A sound, comlete, artial order lanner for ADL. In Proc. 3rd Int. Conf. on Princiles of Knowledge Reresentation and Reasoning, 103{114. Available via FTP from ub/ai/ at cs.washington.edu.