A Conversational Agent to
Navigate in Virtual Worlds
http://www.cs.lth.se/home/Pierre_Nugues/Articles/twlt11/twlt11.pdf
CHI 2000 Workshop on Natural
Language Interfaces
The Hague, The Netherlands
Pierre Nugues, Christophe
Godéreaux, Pierre-Olivier El
Guedj, & Frédéric Revolta
Early Experiment - Shrdlu
• Shrdlu was written by Terry Winograd at
the M.I.T. Artificial Intelligence Laboratory
in 1968-70. Moves objects around and
makes inferences about the world.
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Person: IS THERE A LARGE BLOCK BEHIND A PYRAMID?
Computer: YES, THREE OF THEM: A LARGE RED ONE, A LARGE GREEN CUBE,
AND THE BLUE ONE.
Person: PUT A SMALL ONE ONTO THE GREEN CUBE WHICH SUPPORTS A
PYRAMID.
Computer: OK. (does it)
…
Person: HOW MANY THINGS ARE ON TOP OF GREEN CUBES?
Computer: I'M NOT SURE WHAT YOU MEAN BY "ON TOP OF "IN THE PHRASE
"ON TOP OF GREEN CUBES " .
DO YOU MEAN:
1 - DIRECTLY ON THE SURFACE
2 - ANWHERE ON TOP OF ?
Person: 2
Computer: THREE OF THEM.
...
Person: DOES A STEEPLE
Computer interrupting: SORRY, I DON'T KNOW THE WORD "STEEPLE".
Person: A "STEEPLE" IS A STACK WHICH CONTAINS TWO GREEN CUBES AND A
PYRAMID.
Computer: I UNDERSTAND.
Person: ARE THERE ANY STEEPLES NOW?
Computer: NO.
Person: BUILD ONE.
Computer: OK. (does it)
Graphical Limitations
• Each command needs an interface
• Difficult to enumerate all the possible
combinations
• Takes up room to display interface
• Some commands are hard to find
• In the virtual world, some commands like
“Go through the door” is easier to say than
to navigate with a mouse.
Ulysses
Semantics & Syntax
• Trained on Corpora of 400 utterances
• 350 words plus morphological variations
• Phrase-based structure modified to handle
unknown and missing words
• Recognizes:
– Commands, such as “go to the house”
– deictic clarifications, such as “the yellow one”
– motion modifications, such as “faster”
– motion repetitions, such as “again”
Separates into five commands
• Go: Corresponds to a change of location with a
possible rotation of the embodiment. Includes
advance, enter, leave, climb etc.
• Return: Same as go, but the object visibility
does not matter.
• Rotate: Corresponds to the rotation of the user’s
embodiment head.
• Stop
• Continue
Some verbs, like “retourne”, are still ambiguous.
• Each sentence is broken down into
clauses.
• Each clause has a subject, verb group and
list of complements.
• Each complement has a semantic tag:
time, manner, location.
• Each verb group has the command and
possible object and adverbs.
Maps different styles of commands
together.
•
•
•
•
Go forward.
I want to go forward.
Can you go forward?
I want you to go forward.
Orders clauses based on
connectors and breaks down
clauses into commands.
• Go into the house and then go to the computer.
(1) Go into house
(2) Go to the computer
• Go to the house by passing in front of the flag
pole
(1) Go in front of the flag pole
(2) Go to the house
• Go back into the house
(1) Return to the house
(2) Go into the house
Geometric Reasoning
• For now objects very distinct – houses,
cars, blocks etc.
• Give an alignment to everything that has
alignment
• Distance is relative
Reference Resolution
• Uses a salience algorithm based off work
of Carla Huls.
http://www.cs.mu.oz.au/acl/J/J95/J951003.pdf
• Makes a list of possible objects.
• Places weight based on visibility, proximity
and focus.
• Asks for clarification if still ambiguous.
Dialogue Manager
•
•
•
•
Receives a list of possible parses.
Gets rid of non-executable ones.
If only one command executes it.
If not solicits clarification.
U
s
e
r
Turn around
Look at the
house
Go to the two
cars on the left
of the house
U
l
y
s
s
e
s
Viola
Okay
Viola
U
s
e
r
Turn to the right
Go to that car
Turn to the right
U
l
y
s
s
e
s
Yes
Voila
Viola
U
s
e
r
Go to the cube
Go to the small
ones
Go behind them
U
l
y
s
s
e
s
There are many
Voila
Viola
Questions?