From: Proceedings, Fourth Bar Ilan Symposium on Foundations of Artificial Intelligence. Copyright © 1995, AAAI (www.aaai.org). All rights reserved. Intelligent talk-and-touch interfaces using multi-modal semantic grammars Bruce Krulwich and Chad Burkey Center for Strategic TechnologyResearch Andersen Consulting LLP 100 South WackerDrive, Chicago, IL 60606 krulwich@andersen.com Abstract Multi-modal interfaces havebeenproposed as a wayto capture theeaseandexpressivity of natural communication. Interfaces of thissortallowusersto communicate with computers through combinations of speech, gesture, touch, expression, etc.A critical problem indeveloping suchan interface is integrating these different inputs (e.g., spoken sentences, pointing gestures, etc.) intoa single interpretation. Forexample, incombining speech andgesture, a system mustrelate eachgesture to theappropriate partof the sentence. We areinvestigating thisproblem asit arises in ourtalkandtouch interfaces, whichcombine full-sentence speech andscreen-touching. Oursolution, whichhasbeen implementedin two completed prototypes, uses multi-modal semantic grammarsto match screen touches to speech utterances. Through this mechanism,our systems can easily support wide variations in the speech patterns used to indicate touch references. Additionally,they can ask specific focusedquestions to the user in the event of an inability to understand the input. Theyearl also incorporate other semantic information, such as contextual references or references to previous sentence referents, through this single unified approach. Ourtwo prototypes appear effective in providing a straightforward and powerfulinterface to novice computerusers. 1. Talk and touch interfaces Natural humandialogue is a lot morethan sequences of words. Context, gestures, expressions, and implicit knowledgeall play a considerable role in conveyingcomplexthoughts and ideas in a discussion. Thesefactors clarify the language, fill in the gaps, and disambiguatethe spokenwords themselves. CSTaR’stalk and touch project is attempting to incorporate information of this sort into an ~ intelligent interface based on full-sentence speech recognition and touch-screen sensing [Krulwich and Burkey, 1994]. Consider the following utterances, in the context of a typical electronic mail system: 1. Readthis message(with a corresponding touch to a messageheader) t Ourscreentouchescanbe easilyreplacedwithmouse clicks, etc. Krulwich 103 From: Proceedings, Fourth Bar Ilan Symposium on Foundations of Artificial Intelligence. Copyright © 1995, AAAI (www.aaai.org). All rights reserved. 2. 3. 4. 5. Send a messageto him (with a corresponding touch to a name) Send a message to him (without a touched name) Forwardthis messageto him and Bill (two touches: one message, one name) Forwardthis messageto him and Bill (with only a touched message) In each of these cases, the system has to match the touched objects (references to messagesor people) to the ambiguousslots in the sentence. In cases 3 and 5 there are more potential references to touchedobjects than there are screen touches, so additional inference is required to determine the missing information from context, or at least determine specifically which informationis unspecified. This paper discusses the approach to these issues that is embodiedin our two talk and touch prototypes. Thef’trst, an interface to a typical e-mail systemshownin figure 1, allows commands such as those listed above. Weuse this system to introduce and illustrate our approachin section 2. The second, a video communicationsmanagershownbriefly in figure 2, supports commands for establishing video calls and multi-point video-conferences, sending and receiving multLrnedia e-mail, and sharing and manipulating documentsduring a video-conference. Wediscuss this secondsystemin detail in section 3. 2. Multi-modal semantic grammars The first step in our approachto developing an integrated interpretation of speech and screen z In general, semantic touches is to parse the spoken text into a multi-modal semantic grammar, grammarsare language grammarsthat parse text into semantic componentsinstead of syntactic ones [Burton, 1976]. For example, a typical syntactic grammarwouldparse the sentence forward the new messagefrom John to Bill and my team into syntactic constructs such as subject, verb, direct object, and prepositional phrase. Asemantic grammar,on the other hand, wouldparse it in terms of semantic constructs such as command,messagespecification, author, and recipients. This allows for easier and moreaccurate interpretation, as well as improvingperformancethrough early application of semanticconstraints. Wehave extended this idea by adding semantic constructs to the grammarthat refer to other sources of input information, such as screen touches, as well as those that refer to context. For example, the sentence "forward this message to Bill and Chad"would be parsed to recognize that "this message"is not only a messagespecification, but is also a reference either to a touched screen object or to a particular contextually-significant message. A samplegrammarof this sort, for top-level commands in our e-mail prototype, is shownin figure 3. The grammardefines nodes, or "tags," (shownin angled brackets) that correspond to blocks of text in the input string. Eachtag can be seen as a slot that maybe filled by a portion of an input sentence. Instead of the phrase from him being recognized as a prepositional phrase, as it might with a syntactic grammar, it can be recognized as an author of a message (the tag 2 Issues relating to speech recognition are discussed in section 4. In our discussion here we assumethat the spoken cormnandshave been recognized ,and are available in text form. 104 BISFAI-95 From: Proceedings, Fourth Bar Ilan Symposium on Foundations of Artificial Intelligence. Copyright © 1995, AAAI (www.aaai.org). All rights reserved. . . ¯ ¯ . ¯ :...:. :::: | .... ..:: ¯ ... ::::¯ .:... ¯ :.:’:’:::::¯ . i :.’:. ¯ . ¯ " :.’. " " "’i :ii.:..:’." :.::’: Y""" -..’::ii: :" :’.i’i:. ¯... Figure 1: A prototype talk and touch e-mail system workspace <fromPerson>), and in particular as a reference to a touched person on the screen (the tag <pointPerson>). The following are someof the tags used in parsing the sentence forward this message to BillandChad, andtheir corresponding text: <sentence>= "forward this messageto Bill and Chad" <cmdForward> = "forward this message to Bill and Chad" <message>= "this message" <pointltem> = "this message" <rePeople> = "to Bill and Chad" <people> = "Bill and Chad" :;~.~:.~-- m Figure 2: A video communicationsmanager system Krulwich 105 From: Proceedings, Fourth Bar Ilan Symposium on Foundations of Artificial Intelligence. Copyright © 1995, AAAI (www.aaai.org). All rights reserved. <sentence> ::=<cmdFozxvard> I <cmdReply> I <cmdltem> I <cmdCheck> I <cmdReadMessage> I <cmdReadWhich> I <cmdWho>I <cmdWhols> I <cmdDelete>I <cmdScrol]>. <cmdForward>::= <forward> <message> <toPeople> I <forward> <toPeople> I <forward> <message.>I <forward> . <cmdRepfy> ::=<create> <reply> TO<message,> I <create,> <reply> TO <person> I <reply> TO <message,> l <reply> TO <person> I <create> <reply> I <reply>. <cmdItem>::= <create> <item> <rePeople> I <create> <item>. <cmdCheck> ::= <check> <new> MAILFROMMY<groups> I <check> <new> MESSAGESFROMMY <groups> I <check> <new> MAILI <check> <new> MESSAGES. <cmdReadMessage>::= <read> <message>. <cmdReadWhich>::= <read> THE<which> <item> <fromPerson> I <read> THE<which> <item>. <cmdWho>::= <who> RECEIVED<message> I <who> GOT<message> f <who> WASSENT<message>. <c~ndWhoIs>::= <whols> <person>. <cmdDelete,> ::= <delete> <message>. <cmdScroll>::= <scroll> <direction> BY<number>I <scroll> <direction>. <message> ::= <pointltem> I THE<new><item> <fromPerson> I THE<item> <fromPe~son> I <msgNum> I THE<new><reply> <fromPex~on>I THE<reply> <fx~nPerson>. <pointltem> ::= <pointWhich><item> l <pointWhicl~ <reply> I <pointWhich>. <msgnum> ::= <item> NUMBER <number>. <polntwhich> ::= THIS I THAT. <item> ::= MESSAGE I NOTEI LE’ITER. <toPeople> ::= TO<people>. <people> ::= <peopleItem> I <people, Item> AND<people>. <peopleItem> ::= <person> I <groups> I MY<groups>. <fromPerson> ::= FROM <person>. <person> ::= <pointperson> I <name>I <me>. <pointperson> ::= HIMI HERI HEI SHEI <pointwhich> PERSON. <name>::--- PAULA[ BILL I BRUCE I CHAD. <create> ::= CREATEA I CREATEA NEWI SENDA I SENDA NEW. <check> ::= CHECKFOR I SCANFOR I DO I HAVEANYI READMY. <new> ::= NEWI UNREAD. <next> ::= NEXTI NEXT<new>. <previous> ::= PREVIOUSI PREVIOUS<new>. <which>::= <next> f <previous>. <who> ::= TELL ME WHO<else> I TELL ME WHO. <direction> ::= UP I DOWN. <groups> ::= FRIENDSI TEAM. Figure 3: A partial multi-modal semantic _grammarfor e-marl commands A backgroundassumptionthat we are makingis that the spoken text and the screen touches cannot be matchedtemporally. If our systems could simply comparethe times that particular wordswerespokenwith the times of the screen touches, it court of coursedeterminean a priori mostlikely matchbetweenthe touchesandtheir referents. Unfortunately,the currentstate of the market in speech technology makesaccurate time-stamping of multiple input modalities impossible. Moreimportantly, informalstudies have showna tendencyfor people to have their screentoucheslag behindtheir spokenreferences. Forboth reasons,moreinference is requiredto matchspokenreferences with screen touches. 106 BISFAI-95 From: Proceedings, Fourth Bar Ilan Symposium on Foundations of Artificial Intelligence. Copyright © 1995, AAAI (www.aaai.org). All rights reserved. The second interpretation step, then, is to matchthe multi-modalreferences in the parse to the touched screen objects. The simplest case, of course, is whenthe numberof screen touches matchesthe numberof multi-modaltags. In this case, the references in the command are resolved with the appropriate components of the touched objects. For example, suppose the spoken command is ’forward this messageto him and her," and three screen objects were touched. The system determines that a <cmdForward> was given, and sees that the <message>tag refers to a <pointWhich>.This is interpreted by resolving the <pointWhich>with the first screen touch, which is to a message. The system then sees that the <toPeople> contains two <pointPerson> tags, soit resolves the two recipients of the messagewith the second and third screen touches by checking that the two touched objects have assecmted names. The system then executes the FORWARD commandwith the given message and destination names. Thetypechecking of thisresolution is significant, andis crucial to proper handling of more complex situations. Manyobjects andwordson thescreen canbe interpreted in a number of ways. In the e-mail system workspace shownin figure 1, for example, the nameof a message author might be touchedas a reference to the person as an author, or as a recipient, or instead to the messageitself. This is even moreapparent in morecomplexdomains,in which, for example,a documenticon mayindicate a reference to the documentitself, its author, its project, or its purpose. In our previous example, "forward this messageto him and her;" suppose that there are two recorded screen touches, both of whichare to messageindicators and could thus refer to either a message or its author. The system parses the message into the multi-modal semantic grammar from figure 3, and determines that the spoken command is a <cmdForward>,that the <message> is a <pointltem>, and that the recipients are two <pointPerson>s. The system then attempts to resolve the three references to screen touches with the two actual screen touches, and realizes that a simple one-to-one mappingis not possible. The system then checks the previous command to see what messages or people were referenced, to use this context to resolve our current references. If the previous command refered primarily to a message,as is most likely, the system will assumethat the current command is referring to the message from the previous command,and that our two screen touches are resolving the recipients of the message.If, on the other hand, the previous command is purely person-oriented (e.g., whois he), the system will assumethat the previously referred-to person is one of the recipients, and resolve the fn’st screen touch with the messageto forward, and the secondwith the secondrecipient. If, on the other hand, the first screen touch wasto a person’s name,it could not be interpreted as pointing to a message. In this case, both touches are assmnedto refer to recipients, and the messagehas to be determinedfrom context. If there is no wayto determine the message,the user will be asked to specify the message. Becausethe system was able to interpret the bulk of the command,the question posed to the user can be quite focused, such as "Whichmessagedo you want to forward to Chadand Bill?" Krulwich 107 From: Proceedings, Fourth Bar Ilan Symposium on Foundations of Artificial Intelligence. Copyright © 1995, AAAI (www.aaai.org). All rights reserved. Figure 4: A communications workspace Ingeneral, the greater thedisparity between spoken references to touched objects andtheactual screen touches, themoreinference maybe needed to interpret thecommand, andthegreater the likelihood ofhaving toasktheuserquestions before proceeding. Ourgoalin atlthese cases has bcentoutilize whatever information isavailable toconstruct a partial interpret thesentence, and tobc abletoasktheuseras focused a question as possible. Whiletheheuristics we havebeen discussing areonlysomeof manythatarcpossible, we havefound in practice thatoursystem is almost always abletocorrectly interpret ambiguous commands frombeginning users. 3. A second prototype: Video communications management Oursecond talkandtouchprototype, a videocommunications manager, is shownin figure 4. Thesystem supports commands tostartandendvideocalls or multi-point videoconferences, to playand sendmultimcdia mailmessages, to shareand manipulate documents duringvideo conferences, andso on.Ourgoalsindeveloping thisprototype werethreefold. First, wewanted toapply talkandtouch interfaces toa domain thatwasinherently verbal andvisual, asopposed to a domain likee-mall thatwasinherently textoriented. Second, wewanted to increase thenumber of typesof objects thatcouldbc referenced multi-modally, whiledecreasing thenumber of potentially ambiguous references in ourspoken commands. Third, we wanted to integrate our 108 BISFAI--95 From: Proceedings, Fourth Bar Ilan Symposium on Foundations of Artificial Intelligence. Copyright © 1995, AAAI (www.aaai.org). All rights reserved. <sentence> ::= <agent><command> I <command>. <command> ::= <cmdCall>I <cmdSend> i <cmdPickup> I <cmdHangup> I <cmdPlay>. <cmdPlay>-:= <play> <message>. <cmdSend> ::= SEND<toPeople> A <measagzType> I SEND <toPeople,> A <messageType> <msgTytm> I SENDA <messagcType>TO<toPeople> I SENDA <messageTyp¢> TO <toPeopIe> <msgType>. <cmdPickup> ::= ANSWER <call>l PICKUP<call>l SHOW <call>l PICK<itCall> UPI PICKIT UP. <cmdCall>::= CALL <toPeople>l CALL <rePeople><callType>l GET<rePeople,> ONTHELINE I GET<toPeople> ONTHELINE<callType>l SETUP A <confType.> with <rePeople> I:SETUPA <confType> with<rePeople> <callType>. <cmdHangup> ::=HANGUP <call> I DISCONNECT <call.> I END<call>. <call> ::=THECALLTO <toP~ple> I THECALLFROM<f~anPeason>l <pointCaU> I <itCall>. <message>::= <pointMsg>I <itMsg> I THE<messagoType> FROM <IM~nPerson> I THE<msgNum> <messageType,>. <confI’ype.> ::= CONFERENCE CALLI TELECONFERENCE I VIDEOCONI~RENCE. <messageType>::= MESSAGE I REPLY. <toPeople.>::ffi <toPtzson>l<toPvrson>AND <toPerson>l<toPerson>AND <toPea’son>AND <toPcrson> I <toPerson><toPermn>AND<toPerson>. <toPerson> ::= <name,> l <pointPerson>. <pointP~son>::= HIMI HERI THEM. <callType> ::= AUDIO ONLY. <msgType~::= AUDIOONLYI TEXTONLY. ~,nsgNum>::= FIRST I SECOND I ~ I FOURTH. <play> ::= PLAYI SHOW. <pointCall> ::= THISI THISCALL. <fromPetson>::= <name>I <pointPorson>. <name>::= BRUCEKRULWICH l BOBLORDI MINDYCOHNI MARK PAULI PAID ALTO. <agent>::= EINSTEIN. <pointMsg>::= THISI THISMESSAGE. <itCall> ::=IT. <itMsg>::= IT. Figure 5: A grammar for communicationsmanagement (fig. 3) workwith other ongoingresearchat CSTaR, particularly in supportfor multimediacollaboration and communication. Figure 4 showsthe workspaceof our communication managersystem. In the center of the screen arc messages, with the picaa’c of the sender and the narnc and date of the rmssage. These correspondto multimediamessagesthat have beensent or havebeen left by people trying to call. Onthe bottomare two areas for active or pendingcans, with incomming calls on the left and outgoingcalls on the right. Alongthe left side is an area for feedbackfromthe system,with the Iccognizcd spokensentence, the feedbackfrom the talk and touch component,and the feedback fromthe speech recognition subsystem. Figure 5 shows part of the multi-modal semantic grammarfor commandsin the video communications workspacc.This system allows far fewer ambiguousscreen touches than the emarlsystem, dueto the objects on the screen beingmoredistinct and categorized,anddueto lack of support for commands like the forward this message to him and her command that we saw earlier. Grammar tags such as <po~tMessage>,<pointCall>, and <pointPerson>correspondto Krulwich 109 From: Proceedings, Fourth Bar Ilan Symposium on Foundations of Artificial Intelligence. Copyright © 1995, AAAI (www.aaai.org). All rights reserved. Figure 6: A multi-point video conference sharing a document screen touches, and none of the commands at this screen include references to morethan one type 3of screen object. As an example, if the user gives the command"send a message to Bob Lord and him," the systemwill parse the sentence and attempt to resolve the single <pointPerson>reference. If there is a single screen touch recorded, the system will attempt to interpret the touched object as a person, be it the person shownin a picture, or the sender of an indicated message,or the other person on a call. If no touch has been recorded, the system will use contextual references from the previous commands,as discussed earlier. If none exist, a focused question will be posed to the user. A commandsuch as "set up a video conference with Pale Alto, Mindy Cohn, and him" will establish a multi-point video conference. If the <pointPerson>resolves with a touch to BobLord, the conferenceshownin figure 6 will be started as soonas all participants are available. Calls and conferences of this sort support a variety of multi-modal commands,through the grammarshown in figure 7. As with the communications management grammar, the commandsfor video conferencecontrol feature no multi-reference sentences, and are thus muchsimpler to interpret. 3 There are manyissues involved in designingan interface to facilitate straightforward multi-modalinterpretation, whichare beyondthe scope of the present paper. 110 BISFAI-95 From: Proceedings, Fourth Bar Ilan Symposium on Foundations of Artificial Intelligence. Copyright © 1995, AAAI (www.aaai.org). All rights reserved. 4. Discussion and future work The research that we have presented can be viewed from a numberof perspectives, and raises a variety of issues. Oneofthecritical issues in theresearch anddevelopment ofspeech recognition systems hasbccn theneedfora grammar, or othersources of constraints on theinpututterances, to allowfor accurate recognition of continuous speech (e.g., [Reddy, 1976; Leeet.al.,1990;Huang et.al., 1993]. Earlyapproaches useda finite-state grammar, suchas a standard BNFgrammar, forthis purpose. Morerecent research havegeneralized thisapproach toavoid itslimitations byreplacing thefinite stategrammar witha statistical modelofwordordering. Inourresearch, however, we havefound grammar-based speech recognition to be suitable, primarily because of ourneedfor themulti-modal semantic grammars forinterpretation. By usingthesamegrammars forboth purposes, we integrate theprocessing of speech andinterpretation, apply oursemantic knowledge 4 Theremay,however, cariy inprocessing, andfacilitate improved performance. be other benefits of moregeneral approaches, suchas theability to handle newnamesor phrases dynamically. We arecurrently exploring this possibility. Ourapproach hastodateonlybeenapplied to resolving references to touched objects. There arc, however, manyotherusesof gesture in speech[Feldman and Rime,1991]thathavebeen investigated previously inmulti-modal interfaces (e.g., [Bolt andHerranz, 1992; Keens, 1994]). Ourapproach caneasily be extended to include references to screen areasor numerical ranges along axes. It is moredifficult, andthesubject of ongoing research, to incorporate gestures representing thecommands themselves (e.g., an "X"gesture fordeletion) intoourmulti-modal semantic grammars. Lastly, it is crucial thatwe develop morepowerful techniques forincorporating contextual information. Justas we havedeveloped successful approaches fordesigning grammars to support multi-modal references, we haveto enableour grammars to bettersupport references to contextually relevant objects andpeople. If wecanintegrate contextual references andgesturebased commands intoourintegrated approach to multi-medal grammars, we willtrulybe ableto support high-quality interactions between people andcomputers. Acknowledgments: We wishto thankAnatoleGershman, DaveBeck,LucianHughes,Steve Sate,Kishore Swaminathan, andLarryBirnbaum formanyuseful discussions on theresearch presented, andtheinnumerable visitors thathaveseenourdemostrations fortheir feedback. 4 Weare currently using the IBMContinuous Speech SystemTM, which is based on Carnegie Mellon’s Sphinx I system. Wehave also used the PE400TM system from Speech Systems Inc. Krulwich 111 From: Proceedings, Fourth Bar Ilan Symposium on Foundations of Artificial Intelligence. Copyright © 1995, AAAI (www.aaai.org). All rights reserved. References Bolt, R., 1980. Put that there: Voice and gesture at the graphics interface. ComputerGraphics (Proceedings of the ACMSIGGRAPH "80), Vol. 14, No. 3. Bolt, R. ,and Herranz, E., 1992. Two-handedgesture in multi-modal dialog. In Proceedings of the Fifth Annual Symposiumon User Interface Software and Technology, Monterey, CA. Burton, R., 198x. Semantic grammars.In The Encyclopedia of Artificial Intelligence, Feigenbaumand Barr, eds. Burton, R., 1976. Semantic grammar:An engineering technique for constructing natural understanding systems. BBNtechnical report 3453, Cambridge, MA. Feldman, R. and Rime, B., 1991. Fundamentalsof Nonverbal Behavior. CambridgeUniversity Press. Huang, X., Alleva, F., Hon, H., Hwang, M., Lee, K., and Rosenfeld, R., 1993. The SPHINX-I/ Speech Recognition System: An Overview. Computer, speech, and language, volume 2, pp. 137-148. Koons, D., 1994. Capturing and interpreting multi-modal descriptions with multiple representations. In Working Notes of the 1994 AAAISpring Symposiumon Intelligent Multi-Media Multi-ModalSystems, Stanford, CA, pp. 1321. Krulwich, B. and Burkey, C., 1994. Natural command interfaces incorporating speech and gesture. WorkingNotes of the 1994 Conference on Lifelike ComputerCharacters, Snowbird, Utah, 1994. Lee, K., Hon, H., and Reddy, R., 1990. An overview of the SPHINXspeech recognition Transactions on Acoustics, Speech, and Signal Processing, pp. 35-45. Reddy, R., 1976. Speech recognition by machine: A review. IEEEProceedings, 64:4, pp. 502-531. 112 BISFAI-95 system. IEEE