From: AAAI-84 Proceedings. Copyright ©1984, AAAI (www.aaai.org). All rights reserved. FOCUSING IN PLAN RECOGNITION Norman Department of Computer and Information The University Amherst, F. Carver, Victor R. Lesser, Daniel L. McCue* *Digital Equipment Science Massachusetts, Merrimack, 01003 future ABSTRACT A plan recognition architecture is presented which exploits application-specific heuristic knowledge to quickly focus the search to a small set of plausible plan interpretations from the very large set of possible The heuristic knowledge is formalized interpretations. maintenance system where for use in a truth heuristic assumptions and their interpretation justifications are recorded. By formalizing this knowledge, the system is able to reason about the behind the current state of the assumptions This makes intelligent backtracking and interpretations. error detection possible. An issue for plan search spaces is how to rapidly number plan based interface has system in recognition in large and accurately recognixe the [3, 6, 7J. POISE the goals actions they By accomplish. in the context intelligent POISE intelligent plan that could of user plans are of user actions and recognizing a assistance recognition is a recognizer may be forced of interpretations plan because of: 1) concurrency complex user‘s of an task active plan since constraints on the temporal among 2) sharing of plan steps among alternative possibility that any partial among of alternative and accurately and reduce in order provide the for most to the user. Complete-Purchase name are specified in a formal and its parameters (Amount, description Items, Vendor) ! A textual DESC When the invoice is received, check with the requester to find out if the goods have been accepted and should be paid for or if they have been returned and so should be canceled. 2 The steps Receive-Invoice (Pay-For-Goods ’ Check-Goods I Cancel-Goods) ! The on the plan COND = qtc%&ved”) (Check-GoodsStatus WILL-EXIST Pay-For-Goods = qwlxrmd”) (Check-GoalsStatus WILL-EXIST Cancel-Goods the involved of the plan IS cmstmints and their relative ordering ’ sttps and objects <=> <=> Receive-Invoice&em = Check-Goods&ems Receive-Invoiceltems = Pay-For-GoodsJtems Cancel-GoaMtems A definition parameters Amount Items Vendor of plan OR = Receive-Invoice Amount = Receive-InvoiceItems = Receive-Invoice.Vendor z~plan is one in a hierarchy of plans *for purchasing in an The plan, Complete-Purchase, consts~ of three steps of which’ the final sten is either Pav-For-Goods or Cancel-goods. (i.e., loose Additional COND clause statements umstmin stateme;lts. the parameters of the subplans (e.g., the item being paid for must be the one referred to in the invoice). WhiIe this plan dots not make use of it, the Is chmse language provides for concurrency with the shufffe operator which leaves the relative ordering of the plan steps unqecified. The steps of concurrent toplevel plans are implicitly shuffled W. clause plan steps); plans; 3) the plan might be continued number in the design of such a interpretations efficient The plan ! consideration in user activities ordering be ! system. to keep a very large number under to PROC to a user be used as part assistant for an office automation Plan of active system assistance detection and management, error agenda (e-g., correction, and plan completion). Figure 1 describes a simple disambiguate is how to rapidly of this model of possible actions, POISE is able to provide to information of a small plan recognixer the number of a small intelligent Hierarchies used to specify typical combinations actions constraint observation A key problem The need for this type of plan arisen 4) insufficient the The descri ‘ens of procedures/plans language [l p” as depicted here: on the observation of plan steps. recognition from 03054 interpretations. the INTRODUCI’ION important the current user New Hampshire, user actions; acquired I Corporation Blvd. (MKOl-2GO9) Continental of Massachusetts by This research was sponsored, in part, by the External Research Program of Digjtal Equipment Corporation and by a grant from Rome Air Development Center. Figure 42 1 - POISE Complete-Purchase plan Our solution a to this problem Plan application-specific the constraint knowledge quickly heuristic is used focus the search from interpretations. The likelihood steps are shared, deal plans, and the set of large heuristics architecture to of with the heuristic for representing heuristic plan recognition systems. about the assumptions interpretation and information to recognize the behind represents heuristic information II they system The plan possible new ability the user has made error led an has to these assumptions, information. This allowing system been can the We believe made, used an to interpretation, to the that this approach the new benefit of why not only applies to heuristic systems knowledge for Adaptability to different applications and environments with changes to the heuristic control knowledge alone (i.e., without requiring changes to the underlying reasoning system). The capability users. backtracking of explaining scheme. its reasoning considered in interpretation basic and semantic i.e., plan by process step providing validity a use context which to be recovered the current 2. temporal It also permits interpretations unlikely invalidates Figure were if new likely interpretations. tracks a user action, by attempting it initiates to integrate the as follows: When the predictions reach the level of the new instantiation, constraint values are checked. If the constraints are satisfied, the monitor integrates the user action instantiation into the higher-level plan instantiations. This is done by “abstracting” from the action instantiation up to the plan level of the predicting instantiation. The uabstraction” creates instantiations and Pro== Plan propagates constraint values. ‘Ihe new and the predicting instantiation structures are copied’ and merged into a single structure which represents a new interpretation. The syntactically and semantically valid continuations of existing interpretations are The use of an intelligent focus-ofcontrol strategy which can reason about context and the relationships between competing and cooperating interpretations when integrating new information. The use of an intelligent previously allows of five If a plan of this type was expected by any interpretation entries in the focus set, the monitor expands the predictions from the higher-level instantiation towards the instantiation of the action. This generates a hierarchy of instantiations on the prediction blackboard. it plan. interpretation This consists An instantiation of the primitive plan which represents the user action is placed on the instantiation blackboard. to for it address issues which must be faced by to exploit the explained user action into existing interpretations any such system: The ability control. the the contraints. focusofcontrol mechanism. the recognition an what how user out a particular but to complex uses to restrict ARCHITECXLJRE syntactic instantiations, When the monitor intelligent decide has the added explain If plan heuristic and implements and attribute information or that error. and how to integrate believes the user is carrying plan recognition, be invalid approach to an error interpretation of of the When made. contradicts the how and architecture depicted which checks orderings to reason state as of possible for recognition architecture machinery system the system can use this reasoning scheme in general A formal for current interpretation the formalized the that which has made assumptions were PLAN RECOGNITION components the current into two recognition shows strategy an has advantages It becomes why backtracking undo assumptions is acquired interpretation, system [4]. III application-specific relative a new plan. use in a reason maintence plan that plan of continuing has been section and the possible the likelihood knowledge discusses focus-f-attention This The II search. plausible set likelihood existing plan versus starting strategy of the paper is organized Section SectiOnS. This heuristic a small very which supplement focusofcontrol to the of alternative can in the plans. by the interpretations in knowledge information The remainder has been to develop architecture recognition ’ This copy permits constraint values to be propagated and then easily retracted if the system decides to backtrack (i.e., revise prevmus decisions about which higher-Ieve structure an mstantiation is part of.) to 43 posted to the focusing system, Continue” facts (see section III). PLAN LIBRARY as “can l The monitor also checks to see if the user action could be the start of a new activity. It scans the plan library to determine whether or not the user action could syntactically start a new high-level plan. l If a new plan could be started, the monitor “abstracts” from the action instantiation to constraints and higher-level plans, checking propagating constraint values until a top-level The plans which the user plan is reached. action can syntactically and semantically start are posted to the focusing system as “can STart” facts (see section III). SEMANTIC DATABASE AGENDA To MONITOR INSTANTIATION BLACKBOARD PREDICI’ION BLACKBOARD l l l . . l 0 consider the interface is monitoring represented The whose step only sub-step Send-Information, where future SEMANTIC DATABASE - maintains a representation the state of objects in the users world. blackboard. current the in progress. mechanism plan, of focus plan interpretation Send-Information which, abstraction The for sequence triggered 44 of other monitor partial monitor is action a that this is the most likely seen so far. the A and its plan instantiation in Figure 3. Send-Information.7, the indicates leading of instantiation blackboard. entry, next AEO9, level of (Send-Information.7). up to this point agenda entries has some of Note that most agenda if the focuser is correctly user actions. The of the of an agenda will not be executed tracking interpretation generate given of actions the creation plan, actions the would which are pictured entries the on the plan, primitive has just occurred the creation if executed, primitive the on set of instantiated This triggered Request-Information. instantiation action is Check-Goods the partial is action Receive-Information. by The is a user Purchase is the plan The has been plan, followed Complete-Purchase Architecture propagation that activity sub-step consists Receive-Information. action constraint plan interact, l), has as its first step, only of Complete Request-Information PLAN LIBRARY - contains data structures describing the plans. The monitor makes use of the temporal specifications of the subpIans to predict the next steps in the plans and ma k esuse of the constraint specifications when propagating attribute values between plan instantiations. the basic Assume a purchase by the primitive next whose FOCUSOF-ATTENTION DATABASE a data structure where the monitor records its interpretations of user actions and the assumptions it made to arrive at those interpretations. The partial pIan instantiations xseg=;he current best interpretations are listed in . of the focusing example. (see Figure Receive-Invoice, MONITOR AGENDA - a prioritized list of possibfe next steps for the monitor to take to expand interpretations. The monitor selects actions from this agenda to expand the interpretations that it is currently %est” interpretations). The pusuing (i.e., its current action of constructing an interpretation on the blackboard triggers the addition of new actions to the monitor agenda. If all agenda actions were executed, ah possible (valid) interpretations would be generated. discussion in [7]. how heuristic following Complete-Purchase Figure 2 - Plan Recognition 2 A full contained and Complete-Purchase INSTANTIATION BLACKBOARD - a data structure where the monitor records potential interpretations of user activities as hierarchically related sets of partially instantiated plans BLACKBOARD - a data structure “simulates” various predictions of understand system The state of the focus set and instantiation blackboard Figure 3. The depicted in pm are as MONITOR - the interpreter which tracks user actions and system events, instantiates pIans corresponding to those actions and events, and propagates constraint information? PREDICTION the monitor actions better recognition compares the instantiation, to the types of actions interpretations considers only in those which were expected entries. Through the monitor detects a the focus interpretations set. for by The the by focus set interpretation use of precomputed match expected between the tables, plan the type, Send-Information, the plan, focus the and Check-Goods, set entry generate agenda. Constraint the prediction the level of values the shown meets predictions representing the user In this case, parameter the partial The action interpretation how focusing interpretations Send-Information of constraint original instantiation Check-Goods. the from the the from been the Focusing Figure integrated 5, the into action, focus set cannot the interpretation. The partial 4, the top structure needs copied Check-Goods24 this revise of was when copy its the previous it was operation decision that interpretation or could be an a valid really error revised will because the existing reuse to pursue interpretations later be by make it less likely. within interface! be should be of from further still might heuristics to is instance Complete-Purchase.6, be interpreted the the be part of Complete-Purchase.6. differently focus entries of noted, (Send-Information.7 user), the focusing which has should and level Figure new to While interpreted set as Send-Information.7 The Receive-Information.1, shows the (Complete-Purchase- system interpretation in instaniations interpretation previously consideration. of that the also eliminates structure, Current best: Complete-Purchase.6 Predictor: CompIete-Purchase.6 Next step: Check-Goods Agenda entry: AEO6 the If the actions interpretation these “discarded” previously thought to be “unlikely.” bhudiation Blackboard Complete-Purchase.6 hi=& AEO6 Predict Check-Goods ... AE09 As Check-Goods24 are considered. In to up Complete-Purchase.6 permits Send-Information.19 with values interpretations values Note in integrated. constraint This example plan prediction. Complete-Purchase down through action, has created Complete-Purchase.6. on the potentially explosive plans such primitive by limiting propagated to meets the expectation in focus. helps control The in Figure 4. When the level of the compatible Send-Information.7. process on actions values are propagated abstraction executed for removing of to actions be predictions are compared. has of could those instantiations instantiation references types next. predictions instantiates blackboard, plan is expected which appropriate monitor prediction which agenda the sub-step also contains monitor’s The starting Abstract from Rcqucst-Infurmation Receiy-Invoice3 Complete-Purchase6 from Send-Information.7 Receive-In&mation.l Send-Lnformatioa.7 Figure 3 Lwtandndon Blackboard I’rdlctlon Gxyylctc-Purchase.6 Agcab ... AE4I9 Abstract Request-Information from Send-Information.7 Rcceiv -Lnvoicc3 2 Receive-Lnformation.1 Black- Check-Goods.16 I Request-Information.18 Send-I&mation.l9 Send-lnformation.7 Figure 4 FIml Focus Iastantlat.lon Set Complete-Purch 25 7 / Check-Goods24 Receive-Lnvoiee3 \ / Request-Information21 Receive-Lnformation.l \ Send-Lnformation.7 gerr;znbcst: Complete-Purchas+ . Request-Information21 Receive-Information Next step: Agenda entry: AEll 4Wh ... AEll Predict Receive-Information Blnckboard from Request-Lnformatioa21 Figure 5 45 Prcdktion BIaekboard aeckiGd-16 Request-Information.18 Send- IJ ormation.19 III Focusing mechanism POISE in which limits interpretations is a heuristic the interpretations considers system the which actions User action k is SHared by plan instantiation Xu and Yv (i.e., the action fulfills parts of two goals). FOCUSING deemed most control Heuristics likely. User action k has GreaTeR likelihood of fulfilling part of plan instantiation Xu than of plan instantiation Yv. of the user those to about the likelihood of user actions are of three types: 1.) Application-specific - Derived from observations of For example, ua single the application environment. action is most likely to be taken to satisfy some part of a single goal.” That is, sharing a single low-level plan among several high-level plans is unlikely. Also, “initiation of a new top-level plan is a less likely reason for executing some action than completing some plan already in progress.” Since these heuristics are derived from a particular application (e.g., office systems), their relative importance (or usefulness) will depend upon the application. Other applications may have different characteristics. of specific actions 2.) Plan-specific - The likelihood being taken to satisfy particular goals (iz., as steps of particular high-level plans) may be specified as part of the plans. For example, “action A is more likely to be part of plan X than plan Y.” 3.) User Goals - The user may explicitly state goals plans with some or all of their (i.e., high-level accomplished. filled in) to be parameters Interpretations which match these explicit user goals may be considered extremely likely. A Most Likely Explanation for user action k in “state” w (i.e., after considering the sequence of actions, w) is that it is part of plan instantiation Xu. The interpretation entries in a focus set are the plan instantiations in the union of the MLEs in a particular state. A formal definition of Most Likely Explanation: IF PE(k,Xu) AND NOT [PE(k,Yv) AND GTR(k,Yv,Xu)] THEN MLm%kw. where k stands for a user action, u, v, and w are sequences of user actions, X and Y are high-level plans, and plan instantiations are represented as Xu for the high-level plan underway (X) and the actions which partially fulfill it (u). The over these heuristics facts application-specific The make focusing strategy uses these about the most assumptions interpretations a form of nonmonotonic by default. A reason recording along to be expanded. and with retracted scheme proposes without having work because assumption undo the effects be unrelated of changing a the heuristic are represented knowledge, the as follows: sT(kJ9 - User action k can STart plan X. WJW - User action k can instantiation Xu. WWW - A Possible Explanation action k is as part instantiation Xu. Continue for of developed for rules of the the office are: Initiation of a new top-level plan is a less likely reason for executing an action than completing some plan already in progress: IF C(k,Xu) AND PE(k,Xuk) AND ST(k,Y) AND PE(k,Yk) AND NOT SH(k,Xuk,Yk) AND CONSISIENT(GTR(k,Xuk,Yk)) THEN GTR(kXuk,Yk). and can be inferred. In order to formal.& facts and assumptions to are backtracking can heuristics Three H2- assumptions assumptions as inference A single user action is most likely to be taken to satisfy some part of a single goal so it is unlikely that the input action be shared by multiple high-level plans. This is implemented by stating that if no explicit assumption has been made that an input is shared by multiple interpretations, assume that it has not been shared: IF PE(k,Xu) AND PE(k,Yv) AND CONSISTENT(NOT SH(k,Xu,Yv)y THEN NOT SH(k,Xu,Yv). is used for Assumptions formalized assumptions. Hl - represent Assumptions incorrect assumptions. application as reasoning system dependency-directed identifies better known interpretation justifications. a which interpretation particular their with retracted reasoning to “reasonable” The heuristics maintenance maintaining heuristics are and plan user plan 3 See [8] for an explanation CONSISTENT. of the nonmonotonic modal upcxator H3 - (fact Of two Possible Explanations for an input action, if one is assumed to be a Most Likely Explanation for a later user action then it has a GReaTer likelihood of being the Most Likely Explanation for this input: IF PE(k,Xukj) AND PE(k,Yvk) AND NOT SH(k,Xukj,Yvk) AND MLE(w,j,Xukj~ AND CONSISTENT(GTR(k,Xukj,Yvk)) THEN GTR(k,Xukj,Yvky 11). assumed This must be done because, to continue l l l l algorithm When there the is no only the focusing given the constraint possible interpretation process Explanation” algorithm Figure 6. a fact The second (facts heuristics interpretation formalized is more likely (H2), results in interpreting of plan X rather and 10). new “Possible than part Explanation” for action partial to plan backtrack, that than starting interpretation 14). User of action c Y the interpretation of action (facts c 22 Hl and I-D. FUTURE RESEARCH system described built. by the focusing office is currently A special-purpose was reason The additional heuristics has been automation We are planning applications the of plan system. system the of as a continuation AND provided existing the heuristic the b (fact the fact b to be reinterpreted POISE in studied. application being to explore its effectiveness which are not as tightly constrained in as the office. We most plan to be invloves whose order to retract scheme retraction explained. reasoning in a more current A about assumptions interpretation interpretations assumption developing The assumption action approach the allows the more the and to intelligent locates intelligent “quality” the identify or to recognize of resulting the “best” that the user has made an error. One of a new currently scheme. recent current only an are backtracking b, can b results 4 Implicitly, w includes k and later user actions. s This is a slightly simplified version of the actuaI of the knowledge a, the continuing a being that a and STATUS effective Another this action as a continuation of action Note from forward by action actions maintenance than the start of plan Y (see facts 9 This interpretation 15-17). some system is more likely The plan recognition actions, user action, starting push is recognized, is no is the 9 generated and c, fully, 3) is the that Zu causes be interpreted N the plan begun by action a above, a, it is b action which it within there and 23) because of heuristics “Most Likely or as the start of a new plan (facts 5 and 7). now example user action, The (see as continuing of plan, causes (i.e., an interpretation Y started can Since there is only one for the first action more The interpretation information. in focus. be interpreted the plan than one “Possible heuristic rules to assumptions. is straightforward. for interpretation in syntactic ignoring focusing a new action, interpret where This assumption continuing user to structure C(c,Zu) retract Post the resulting “Most Likely Explanations” for the user action and propagate the results of this latest interpretation back to earlier interpretations. is third way instantiation). If there are no “Possible Explanations” for the action, either an interpretation error or a user error has occurred. Backtrack through the assumptions which have been made to find one which could result in an alternate interpretation (previously disregarded because it seemed unlikely) which would explain the current action. Retract this assumption, revise the interpretation of the previous actions and repeat the interpretation process for the current action. example considers form Use the “can Continue” and “can STart” facts posted by the monitor during the basic interpretation process (see section II) to post “Possible Explanation” assumptions for the user action. To understand an plan including to be “Shared” by two X plans (i.e., Xab and Xab’ where b’ is another is as follows: If an action has more Explanation,” apply the generate relative likelihood b was has not yet ocurred). interpretation The focusing the partial possible that a is meant although current approach exploit the information in in a the semantic knowledge posted likelihood creating rule. 47 direction about in which we are extending is the use of focusing about database. objects heuristics that the which is contained These heuristics include the use of objects in plans, e.g., the that plans share objects new objects. or the likelihood of ACKNOWLEDGEMENTS Bruce Croft contributions plan and Larry to the design recognition Lefkowitz have as part of Croft, W. B., L. Lefkowitz, V. Lesser, and K. Huff, “POISE: An Intelligent Interface for Profession-Based Systems,” Conference on Artificial Intelligence, Oakland, Michigan, 1983. PI Doyle, J., “A Truth Maintenance Inielligence, 12231-272. PI Gischer, J., “Shuffle Languages, Petri Nets, and Context-Sensitive Grammars,” Communications of the ACM, 24(9)597405. made and implementation architecture PI of the the POISE System,” Artificial project. REFERENCES PI Bates, P., J. Wileden, and V. Lesser, “Event Definition Language: An Aid to Monitoring and Debugging Complex Software Systems,” Technical Report 81-17, Department of Computer and Information Science, University of Massachusetts, Amherst, Massachusetts, 1981. [fd Huff, PI Croft, W. B. and L. Lefkowitz, “An Office Procedure Formalism Used for an Intelligent Interface ,n Technical Report 824, Department of Computer and Information Science, University of Massachusetts, Amherst, Massachusetts, 1982. PI McCue, D. and V. Lesser, “Focusing and Constraint Management in Intelligent Interface Design,” Technical Report 83-36, Department of Computer and Information Science, University of Massachusetts, Amherst, Massachusetts, 1983. PI McDermott, D. and J. Doyle, uNon-monotonic I,” Artificial In!elligence, l3:41-72. Plan Grammar: The sets of facts X = a b d... and assumptions a 1. n(aS) 2. PE(a,Xa) 3. MLE(a,a,Xa) Focus ?&+a} Y = b c... for each User su cewsive K. and V. Lesser, “Knowledge-Based Command Understanding: An Example for the Software Development Environment ,” Technical Report 82-6, Department of Computer and Information Science, University of Massachusetts, Amherst, Massachusetts, 1982. Actions: ab before backtrackina 4. c@Xa) 5. PE ,Xab) 6. ST ,y) 7. PE % ,Yb) 8. -SH@,Xab,Yb) a b c... T&ate” (string of user actions) arc: ab after backtrackina 4. 5. 6. 7. 8. c@JW PE ,Xab) ST ,Y) PE ! ,Yb) -SH(b,Xab,Yb) &7?HlJ 10. MLE(ab,b,Xab) l14. MLE(ab,b,Yb) abc 15. qab) 16. PE(c,Ybc) 17. MLE(abc,c,Ybc) 4. CCbSa) 2 E % %“’ 7: PE :Yb) 18. PE(b,Ybc) 19. --SH@,fiSlab) WV-U 1. 2. 11. 12. GTR(a,Xab,Xa) 20. GTR@,Ybc,Xab) t5,17,lg@31 21. GTR@,Ybe,Yb) t2,ww3~ l3. MLE(ab,a,Xab) Focus Set-b) 13. MLE(ab,a,Xab) Focus t7,17,18J-1 22. MLE(abc,b,Ybc) Sct=jXab,Yb) 23. MLE(abc,a,Xa) Focus Sct=(Xa,Ybc} Facts and assumptions are grouped by the user action to which they refer. only ‘Sn” [4] facts are represented and some obvious -SH assumptions have been ignored. The justifications for assumptions made using the three heuristics formalized in the paper are ’ en in braces ({ )) below the assumptions with the three heuristics denoted as Hl-H3. Thart assumptions cK $ed durin backtracking are denoted with lk C and ST facts arc posted by the monitor as described in stct~on II. t-h e remainder of the facts rcsuIt from the application of various bookeeping ruIes. Figure 6 - Focusing in a Sample Grammar 48 Logic