Development of
a Cognitive Agent
Stuart C. Shapiro
Department of Computer Science and Engineering
and Center for Cognitive Science
State University of New York at Buffalo
shapiro@cse.buffalo.edu
S.C. Shapiro
Outline
•
•
•
•
•
•
•
•
Introduction
Intensional Representation
Logic for NLU and Commonsense Reasoning
Discussing Propositions and Sentences
Symbol-Grounding by Perception and Action
Representation and Use of Indexicals
A Personal Sense of Time
Summary
S.C. Shapiro
Goal
• A computational cognitive agent that can:
–
–
–
–
–
–
–
S.C. Shapiro
Understand and communicate in English;
Discuss specific, generic, and “rule-like” information;
Reason;
Discuss acts and plans;
Sense;
Act;
Remember and report what it has sensed and done.
Embodied Cassie
• A computational cognitive agent
– Embodied in hardware
– or Software-Simulated
– Based on SNePS and GLAIR.
S.C. Shapiro
SNePS
• Knowledge Representation and Reasoning
– Propositions as Terms
• SNIP: SNePS Inference Package
– Bi-Directional Inference
• SNeBR: SNePS Belief Revision
• SNeRE: SNePS Rational Engine
• Interface Languages
– SNePSUL: Lisp-Like
– SNePSLOG: Logic-Like
– GATN for Fragments of English.
S.C. Shapiro
GLAIR Architecture
Grounded Layered Architecture with Integrated Reasoning
Knowledge Level
NL
SNePS
Perceptuo-Motor Level
Sensory-Actuator Level
Vision
Sonar
Motion
Proprioception
S.C. Shapiro
Interaction with Cassie
English
(Statement, Question, Command)
(New Belief)
[SNePS]
(Current) Set of Beliefs
[SNePS]
Reasoning
Clarification Dialogue
GATN Parser
Looking in World
Answer
Actions (Updated) Set
of Beliefs
[SNIP]
[SNeRE]
[SNePS]
GATN
Generator
Reasoning
English sentence expressing
new belief answering question reporting actions
S.C. Shapiro
Cassie, the FEVAHR
S.C. Shapiro
Cassie in the Lab
S.C. Shapiro
Garnet Simulation World
S.C. Shapiro
Outline
•
•
•
•
•
•
•
•
Introduction
Intensional Representation
Logic for NLU and Commonsense Reasoning
Discussing Propositions and Sentences
Symbol-Grounding by Perception and Action
Representation and Use of Indexicals
A Personal Sense of Time
Summary
S.C. Shapiro
Entities, Terms, Symbols, Objects
• Cassie’s mental entity: a person named Bill
• SNePS term: B5
• Object in world:
S.C. Shapiro
Intensional Representation
Intensional entities are distinct
even if coreferential.
“The morning star is the evening star.”
“George IV wondered if Scott was the author of
Waverly.”
S.C. Shapiro
McCarthy’s Telephone Number
Problem
Mary's telephone number is Mike's telephone number.
I understand that Mike's telephone number is Mary's
telephone number.
Pat knew Mike's telephone number.
I understand that Pat knew Mike's telephone number.
Pat dialed Mike's telephone number.
I understand that Pat dialed Mike's telephone number.
S.C. Shapiro
Answering the Telephone
Number Problem
Did Pat dial Mary's telephone number?
Yes, Pat dialed Mary's telephone number.
Did Pat know Mary's telephone number?
I don't know.
S.C. Shapiro
Outline
•
•
•
•
•
•
•
•
Introduction
Intensional Representation
Logic for NLU and Commonsense Reasoning
Discussing Propositions and Sentences
Symbol-Grounding by Perception and Action
Representation and Use of Indexicals
A Personal Sense of Time
Summary
S.C. Shapiro
Logic for NLU &
Commonsense Reasoning
Either Pat is a man or Pat is a woman or Pat is a robot.
I understand that Pat is a robot or Pat is a woman
or Pat is a man.
Pat is a woman.
I understand that Pat is a woman.
What is Pat?
Pat is a woman and Pat is not a robot
and Pat is not a man.
S.C. Shapiro
Representation in FOPL?
Man(Pat)  Woman(Pat)  Robot(Pat)
but don’t want inclusive or
Man(Pat) +Woman(Pat) +Robot(Pat)
T
T
T
F
T
So don’t want exclusive or either
S.C. Shapiro
andor
andor(i, j){Pi, ..., Pn}
True iff at least i, and at most j of the Pi are True
S.C. Shapiro
Outline
•
•
•
•
•
•
•
•
Introduction
Intensional Representation
Logic for NLU and Commonsense Reasoning
Discussing Propositions and Sentences
Symbol-Grounding by Perception and Action
Representation and Use of Indexicals
A Personal Sense of Time
Summary
S.C. Shapiro
Discussing Propositions
That Bill is sweet is Mary's favorite proposition.
I understand that Mary's favorite proposition is
that Bill is sweet.
Mike believes Mary's favorite proposition.
I understand that Mike believes that Bill is sweet.
S.C. Shapiro
Discussing Sentences
Hamlet said `To be or not to be'.
I understand that Hamlet said `To be or not To be'.
Polonius said `Neither a borrower nor a lender be'.
I understand that Polonius said `Neither a borrower
nor a lender be'.
Who said `To be or not to be'?
Hamlet said `To be or not To be'.
S.C. Shapiro
Representing Propositions
and Sentences
Propositions and sentences must be
First-class entities of the domain
Represented by terms
S.C. Shapiro
Outline
•
•
•
•
•
•
•
•
Introduction
Intensional Representation
Logic for NLU and Commonsense Reasoning
Discussing Propositions and Sentences
Symbol-Grounding by Perception and Action
Representation and Use of Indexicals
A Personal Sense of Time
Summary
S.C. Shapiro
Symbol Grounding by
Perception and Action
• Agent can demonstrate understanding
– Of properties and descriptions
• By locating correct objects in the world
– Of verbs
• By acting properly in the world
S.C. Shapiro
Symbol-Grounding 1
Go to the green robot.
S.C. Shapiro
Symbol-Grounding 2
Go to the green robot.
I went to the green robot.
I am near the green robot.
S.C. Shapiro
Alignment
•
•
•
•
S.C. Shapiro
Terms at GLAIR KR level
Aligned with
Descriptions and procedures
At GLAIR PM level
Outline
•
•
•
•
•
•
•
•
Introduction
Intensional Representation
Logic for NLU and Commonsense Reasoning
Discussing Propositions and Sentences
Symbol-Grounding by Perception and Action
Representation and Use of Indexicals
A Personal Sense of Time
Summary
S.C. Shapiro
Representation and Use of Indexicals
• Words whose meanings are determined by
occasion of use
• E.g. I, you, now, then, here, there
• Deictic Center <*I, *YOU, *NOW>
• *I: SNePS term representing Cassie
• *YOU: person Cassie is talking with
• *NOW: current time.
S.C. Shapiro
Analysis of Indexicals
(in input)
• First person pronouns: *YOU
• Second person pronouns: *I
• “here”: location of *YOU.
S.C. Shapiro
Generation of Indexicals
• *I: First person pronouns
• *YOU: Second person pronouns
• *NOW: used to determine tense.
S.C. Shapiro
Use of Indexicals 1
Come here.
S.C. Shapiro
Use of Indexicals 2
Come here.
I came to you, Stu.
I am near you.
S.C. Shapiro
Use of Indexicals 3
Who am I?
Your name is ‘Stu’
and you are a person.
Who have you talked to?
I am talking to you.
Talk to Bill.
I am talking to you, Bill.
Come here.
S.C. Shapiro
Use of Indexicals 4
Come here.
I found you.
I am looking at you.
S.C. Shapiro
Use of Indexicals 5
Come here.
I found you.
I am looking at you.
I came to you.
I am near you.
S.C. Shapiro
Use of Indexicals 6
Who am I?
Your name is ‘Bill’
and you are a person.
Who are you?
I am the FEVAHR
and my name is ‘Cassie’.
Who have you talked to?
I talked to Stu
and I am talking to you.
S.C. Shapiro
Outline
•
•
•
•
•
•
•
•
Introduction
Intensional Representation
Logic for NLU and Commonsense Reasoning
Discussing Propositions and Sentences
Symbol-Grounding by Perception and Action
Representation and Use of Indexicals
A Personal Sense of Time
Summary
S.C. Shapiro
A Personal Sense of Time
• *NOW contains SNePS term representing
current time.
• *NOW moves when Cassie acts.
S.C. Shapiro
Representation of Time
before
!
after
before
!
after
!
B1
B6
I
lex
find
S.C. Shapiro
NOW
Movement of Time
t1
NOW
S.C. Shapiro
before
!
after
t2
NOW
before
!
after
t3
NOW
Performing a Punctual Act
t1
before
!
after
t2
before
!
after
t3
time
!
event
NOW
S.C. Shapiro
NOW
Performing a Durative Act
t1
before
!
after
t2
!
time
t3
!
event
NOW
NOW
S.C. Shapiro
Keeping Track of Time 1
Who are you looking at?
I am looking at you, Stu.
Come here.
S.C. Shapiro
Keeping Track of Time 2
Who are you looking at?
I am looking at you.
Come here.
I came to you.
I am near you.
Who are you looking at?
I am looking at you.
S.C. Shapiro
Keeping Track of Time 3
Find Bill.
I found Bill.
I am looking at Bill.
Who are you looking at?
I looked at you
and I am looking at Bill.
Who are you talking to?
I am talking to you.
S.C. Shapiro
Keeping Track of Time 4
Follow a red robot.
I found a red robot.
I am looking at a red robot.
S.C. Shapiro
Keeping Track of Time 5
Follow a red robot.
I found a red robot.
I am looking at a red robot.
I went to a red robot.
I am near a red robot.
I am following a red robot.
S.C. Shapiro
Keeping Track of Time 6
Who are you talking to?
I am talking to you.
Who am I?
Your name is ‘Stu’
and you are a person.
Stop.
I stopped.
S.C. Shapiro
Keeping Track of Time 7
Who are you looking at?
I looked at you
and I looked at Bill
and I looked at a red robot.
Who are you following?
I followed a red robot.
Who are you talking to?
I am talking to you.
S.C. Shapiro
Outline
•
•
•
•
•
•
•
•
Introduction
Intensional Representation
Logic for NLU and Commonsense Reasoning
Discussing Propositions and Sentences
Symbol-Grounding by Perception and Action
Representation and Use of Indexicals
A Personal Sense of Time
Summary
S.C. Shapiro
Goal
• A computational cognitive agent/robot
• That can communicate in natural
language.
S.C. Shapiro
Intensional Representation
• SNePS terms represent mental entities.
• May assert that two entities are coreferential.
• Relations/acts may be declared transparent.
S.C. Shapiro
Logic for NLU and
Commonsense Reasoning
• Designed logical connectives
and rules of inference
More appropriate than in standard FOPC.
S.C. Shapiro
Discussing Propositions
and Sentences
• Propositions and sentences are first-class entities.
S.C. Shapiro
Symbol-Grounding by
Perception and Action
• Use of GLAIR architecture to connect
entities with descriptions/functions used by
sensors and effectors.
S.C. Shapiro
Representation and Use
of Indexicals
• Use of Deictic Center for parser to interpret
indexicals as current referents
• And for generator to generate indexicals
from current referents.
S.C. Shapiro
A Personal Sense of Time
• *NOW is current time.
• Updated when Cassie acts.
S.C. Shapiro
For More Information
•
•
•
•
•
•
Personnel
Manual
Tutorial
Bibliography
ftp’able SNePS source code
etc.
• http://www.cse.buffalo.edu/sneps/
S.C. Shapiro