Dealing with Implicit Negatives

Explicit positive for above
Implicit negatives for below, left, right, etc

in Regier:

E = ½ ∑i,p (( ti,p – oi,p) * βi,p )2,
where i is the node, p is the pattern,
βi,p = 1 if explicit positive,
βi,p < 1 if implicit negative
Learning
System
dynamic relations
(e.g. into)
structured connectionist
network (based on
visual system)
Topological Relations
Separation
 Contact
 Coincidence:

Overlap
- Inclusion
-
-
Encircle/surround
Issue #2: Shift Invariance



Backprop cannot handle shift invariance (it
cannot generalize from 0011, 0110 to 1100)
But the cup is on the table whether you see
it right in the center or from the corner of
your eyes (i.e. in different areas of the
retina map)
What structure can we utilize to make the
input shift-invariant?
Limitations
 Scale
 Uniqueness/Plausibility
 Grammar
 Abstract
Concepts
 Inference
 Representation
Demo of the
Regier System
on the English above
Language and Thought

Language



Thought
cognitive processes
We know thought (our
cognitive processes)
constrains the way we learn
and use language
Does language also influence
thought?
Benjamin Whorf argues yes
Psycholinguistics experiments
have shown that linguistics
categories influence thinking
even in non-linguistics task
5 levels of Neural Theory of Language
Spatial
Relation
Motor
Control
Metaphor Grammar
Cognition and Language
abstraction
Computation
Structured Connectionism
Neural Net
Triangle Nodes
SHRUTI
Computational Neurobiology
Biology
Neural
Development
Quiz
Midterm
Finals
Language, Learning and Neural Modeling
www.icsi.berkeley.edu/AI

Scientific Goal
Understand how people learn and use language

Practical Goal
Deploy systems that analyze and produce language

Approach
Build models that perform cognitive tasks, respecting
all experimental and experiential constraints
Embodied linguistic theories with advanced
biologically-based computational methods
Constrained Best Fit in Nature
inanimate
physics
chemistry
biology
vision
language
animate
lowest energy
state
molecular
minima
fitness, MEU
Neuroeconomics
threats,
friends
errors,
NTL
Simulation-based language understanding
“Harry walked to the cafe.”
Utterance
Constructions
Analysis Process
General
Knowledge
Belief State
Schema
walk
Trajector
Harry
Cafe
Goal
cafe
Simulation
Specification
Simulation
Simulation Semantics



BASIC ASSUMPTION: SAME REPRESENTATION FOR
PLANNING AND SIMULATIVE INFERENCE
 Evidence for common mechanisms for recognition and
action (mirror neurons) in the F5 area (Rizzolatti et al (1996),
Gallese 96, Boccino 2002) and from motor imagery
(Jeannerod 1996)
IMPLEMENTATION:
 x-schemas affect each other by enabling, disabling or
modifying execution trajectories. Whenever the
CONTROLLER schema makes a transition it may set, get,
or modify state leading to triggering or modification of other
x-schemas. State is completely distributed (a graph marking)
over the network.
RESULT: INTERPRETATION IS IMAGINATIVE SIMULATION!
Psycholinguistic evidence

Embodied language impairs action/perception
 Sentences
with visual components to their meaning
can interfere with performance of visual tasks
(Richardson et al. 2003)
 Sentences
describing motion can interfere with
performance of incompatible motor actions
(Glenberg and Kashak 2002)
 Sentences
describing incompatible visual imagery
impedes decision task (Zwaan et al. 2002)

Simulation effects from fictive motion sentences
 Fictive
motion sentences describing paths that require
longer time, span a greater distance, or involve
more obstacles impede decision task (Matlock 2000, Matlock
et al. 2003)
Neural evidence: Mirror neurons

Gallese et al. (1996) found “mirror” neurons
in the monkey motor cortex, activated when
 an
action was carried out
 the same action (or a similar one) was seen.

Mirror neuron circuits found in humans (Porro
et al. 1996)

Mirror neurons activated when someone:
 imagines
an action being carried out (Wheeler et al.
2000)
 watches
an action being carried out (with or
without object) (Buccino et al. 2000)
Area F5c
Convexity region of F5:
Mirror neurons
F5 Mirror Neurons
Gallese and Goldman, TICS 1998
Category Loosening in Mirror Neurons (~60%)
Observed: A is Precision Grip
B is Whole Hand Prehension
Action:
C: precision grip
D: Whole Hand Prehension
(Gallese et al. Brain 1996)
PF Mirror Neurons
1.
Neuron responds to execution (grasping) but to grasping and releasing in observation.
2.
Mirror neurons in parietal cortex.
3.
Difference in left hand and right hand.
(Gallese et al. 2002)
A (Full vision)
B (Hidden)
C (Mimicking)
D (HiddenMimicking)
Umiltà et al. Neuron 2001
F5 Audio-Visual Mirror Neurons
Kohler et al. Science (2002)
Somatotopy of Action Observation
Foot Action
Hand Action
Mouth Action
Buccino et al. Eur J Neurosci 2001
The Mirror System in Humans
BA6
The ICSI/Berkeley Neural Theory of Language Project
ECG
Learning early
constructions
(Chang, Mok)
Computing other relations
The 2/3 node is a useful function that
activates its outputs (3) if any (2) of its 3
inputs are active
 Such a node is also called a triangle node
and will be useful for lots of
representations.

Triangle nodes and
McCullough-Pitts Neurons?
A
B
C
A
B
C
Representing concepts using
triangle
triangle nodes
nodes:
when two
of the
neurons
fire, the
third also
fires
“They all rose”
triangle nodes:
when two of the
neurons fire, the
third also fires
model of
spreading
activation
Basic Ideas behind the model





Parallel activation streams.
Top down and bottom up activation combine to
determine the best matching structure.
Triangle nodes bind features of objects to values
Mutual inhibition and competition between
structures
Mental connections are active neural
connections
Can we formalize/model these intuitions
What is a neurally plausible computational
model of spreading activation that
captures these features.
 What does semantics mean in neurally
embodied terms

 What
are the neural substrates of concepts
that underlie verbs, nouns, spatial predicates?
Spreading activation and feature structures





Parallel activation streams.
Top down and bottom up activation combine to
determine the best matching structure.
Triangle nodes bind features of objects to values
Mutual inhibition and competition between
structures
Mental connections are active neural
connections
Feature Structures in Four Domains
Barrett
Ham
Container
Push
dept~CS
Color ~pink
Inside ~region
Schema ~slide
sid~001
Taste ~salty
Outside ~region
Posture ~palm
Bdy. ~curve
Dir. ~ away
emp~GSI
Chang
Pea
Purchase
Stroll
dept~Ling
Color ~green
Buyer ~person
Schema ~walk
sid~002
Taste ~sweet
Seller ~person
Speed ~slow
Cost ~money
Dir. ~ ANY
emp~Gra
Goods ~ thing