Towards a Knowledgeable Machine that
can Pass an Elementary Science Test
Peter Clark
Vulcan Inc
August 2013
Outline
1. Halo: The Goal and Road Travelled…
 AURA, Inquire, and reflections
2. Exploiting Semi-Formal Representations and Textual
Inference
3. A New Challenge: Fourth-Grade Science Tests
Overall Goals
 Long-Term Goal: The Digital Aristotle
 Have large volumes of knowledge encoded in a
computable form, such that the computer can answer
questions, explain its answers, and ultimately dialog
with users about the subject matter
“Explainable Reasoning”
 History
 Halo Pilot: Assess representation & reasoning technologies
 Formal reasoning works, but acquisition and language are problems
 Halo: Develop high-performance acquisition tool (AURA)
 HaloBook (2010-12): Aim to encode much of a textbook
 Inquire: An iPad app – the knowledgeable book
 Halo 2.0: Reorient towards semi-automated acquisition
 focus on taking K-12 science exams
The Knowledge Encoding Process
…Eukaryotic cells similarly have a plasma membrane, but also
contain a cell nucleus that houses the eukaryotic cell's DNA…
Concept Map (User View)
Logic (Internal View)
∀x isa( x, Eukaryotic-cell) →
∃p,n,d isa(p, Plasma-membrane) ∧
isa(n, Nucleus) ∧ isa(d, DNA) ∧ has-part(x, p) ∧
has-part(x, n) ∧ has-part(x, d) ∧ is-inside(d, n)
The Knowledge Encoding Process
Reasoning: Deductive elaboration of the graph using
other graphs and commonsense rules
EukaryoticCell
PlantCell
Plant
Cell
(more)
Parts:
•Plasma membrane
•Nucleus
•DNA
Parts:
•Plasma membrane
•Cell wall
•Chloroplast
Parts:
•Plasma membrane
•Cell wall
•Chloroplast
•Nucleus
•DNA
Question Answering
Typical examples of questions the system can answer:
During mitosis, when does the cell plate begin to form?
What happens during DNA replication?
What is the relationship between photosynthesis and cellular respiration?
What do ribosomes do?
During synapsis, when are chromatids exchanged?
What are the differences between eukaryotic cells and prokaryotic cells?
How many chromosomes are in a human cell?
In which phase of mitosis does the cell divide?
What is the structure of a plasma membrane?
Outcomes
 The good…
 Experiments suggested Inquire is educationally useful
 Some question classes answered well
 “Suggested question” mechanism helped a lot
 The bad…
 Only covered ~25% of the book after 2 years
 Deductive question-answering somewhat hit-and-miss
 It’s not that manually constructed rulebases are “bad”, but:
 Expensive (of course, costs may be brought down)
 Brittle (unless the task is very tightly constrained)
 Never seem to be finished (permanently incomplete)…
 Textual Inference / Semi-Formal Representations:
 Create language-based representations from (lots of) text
 include words/phrases – deferred ontological commitment
 Imprecise, shallower reasoning
 an evidential process, using multiple sources of evidence
The Dilemma of Knowledge Engineering
Manual methods are expensive, automatic methods are shallow
Outline
1. Halo: The Goal and Road Travelled…
 AURA, Inquire, and reflections
2. Exploiting Semi-Formal Representations and
Textual Inference
3. A New Challenge: Fourth-Grade Science Tests
Levels of Formality
SemiFormal
Text
Textual
entailment
Logic
Logical
entailment
Query
?
?- has-part(ribosome,?x).
1. Representation
Sentence
Parse
Logical
Form
"Channel proteins facilitate the passage of molecules across the membrane."
*S:-17
+----------------------------------+---------+
NP:-3
VP:-13
|
+----------------------------+-----+
N^:-2
V:0
*NP:-12*
|
|
+------------+---------------+
N:-2
FACILITATE
NP:-8
PP:-2
+----+----+
+-------+-------+
+-------+---+
N:-1
N:0
NP:-1
PP:-2
P:0
NP:-1
|
|
+----+--+
+----+--+
|
+----+---+
CHANNEL PROTEINS
DET:0
N^:0
P:0
NP:-1
ACROSS DET:0
N^:0
|
|
|
|
|
|
THE
N:0
OF
N^:0
THE
N:0
|
|
|
PASSAGE
N:0
MEMBRANE
|
MOLECULES
“channel protein”
across
obj
subj
“facilitate”
of
“passage”
“molecule”
“membrane”
subject(facilitate-1, channel-protein-1).
object(facilitate-1, passage-1).
of(passage-1, molecule-1).
across(passage-1, membrane-1).
16
2. Textual Inference
 Reasoning with semi-formal structures
 Find sequence of transformations from text to question
 Requires general lexical and world knowledge
Which proteins help move molecules through the membrane?
IF X facilitates Y THEN X helps Y
“passage”(n) → “move”(v)
“through” ↔ “across”
Knowledge
resources
Channel proteins facilitate the passage of molecules across the membrane.
A. Channel proteins
17
2. Textual Inference
Which proteins help move molecules through the membrane?
1. (simple) question decomposition
What ?x help move molecules through the membrane?
Is ?x a protein?
2a. textual entailment
Channel proteins facilitate the passage of molecules across the membrane.
IF X “facilitates” Y THEN X “helps” Y
Channel proteins help the passage of molecules across the membrane.
“passage”(n) → “move”(v),
“through” ↔ “across”
Channel proteins help move molecules through the membrane.
What ?x
help move molecules through the membrane?
18
2. Textual Inference
Which proteins help move molecules through the membrane?
1. (simple) question decomposition
What ?x help move molecules through the membrane?
Is ?x a protein?
2a. textual entailment Is an evidence-gathering process
Channel proteins facilitate the passage of molecules across the membrane.
IF X “facilitates” Y THEN X “helps” Y
Channel proteins help the passage of molecules across the membrane.
“passage”(n) → “move”(v),
“through” ↔ “across”
Channel proteins help move molecules through the membrane.
What ?x
help move molecules through the membrane?
19
2. Textual Inference
Channel proteins facilitate the passage of molecules across the membrane.
Channel proteins help the passage of molecules across the membrane.
What evidence can I find that
“X facilitates Y” “X helps Y”?
4M rules
12M rules
PPDB
DIRT
(Johns Hopkins) paraphrases
146k rules
WordNet
30k rules
BioKB-101
ontology
20
2. Textual Inference
Channel proteins facilitate the passage of molecules across the membrane.
Channel proteins help the passage of molecules across the membrane.
What evidence can I find that
“X facilitates Y” “X helps Y”?
4M rules
12M rules
PPDB
DIRT
(Johns Hopkins) paraphrases
146k rules
WordNet
30k rules
BioKB-101
ontology
21
2. Textual Inference
Channel proteins facilitate the passage of molecules across the membrane.
Channel proteins help the passage of molecules across the membrane.
What evidence can I find that
“X facilitates Y” “X helps Y”?
4M rules
12M rules
PPDB
DIRT
(Johns Hopkins) paraphrases
146k rules
WordNet
30k rules
BioKB-101
ontology
22
2. Textual Inference
Channel proteins facilitate the passage of molecules across the membrane.
Channel proteins help the passage of molecules across the membrane.
What evidence can I find that
“X facilitates Y” “X helps Y”?

4M rules
12M rules
PPDB
DIRT
(Johns Hopkins) paraphrases
146k rules
WordNet
30k rules
BioKB-101
ontology
23
Domain-Biased Paraphrases (Johns Hopkins)
 Paraphrases learned via bilingual pivoting, and rescored
using distributional similarity.
Some examples from PPDB
amplify
amplify
amplify
amplify
amplify
amplify
amplify
amplify
amplify
amplify
amplify
amplify
amplify
amplify
amplify
amplify
amplify
amplify
amplify
amplify
…
elevate 0.993
explore 0.992
enhance
speed up
strengthen
improve 0.982
magnify 0.98
extend 0.978
accept 0.97
follow 0.965
carry out
broaden0.962
go into 0.962
promote
explain 0.955
implement
leave
0.944
adopt 0.944
acquire 0.942
expand 0.942
…
…
 travel
travel
0.984
0.984
0.982
0.965
0.959
0.951
travel
travel
travel
travel
travel
travel
travel
travel
travel
travel
travel
travel
travel
travel
travel
travel
travel
…
fly
0.893
roll over0.882
relax
0.87
freeze 0.861
breathe 0.861
swim 0.858
move 0.855
die
0.848
swell
0.845
switch 0.842
consumers
bend
0.835
walk
0.835
paint
0.828
work
0.828
move over
feed
0.825
evolve 0.825
survive 0.821
…
…

???
0.838
0.825
???
Performance
 Currently, 3 databases of semi-formal representations
 Current F1 ≈ 30% (e.g., 50% on 10% of qns)
 Answer = weighted sum of evidence
 Learn the weights (via simulated annealing)


Performance

27
Levels of Formality
SemiFormal
Text
Logic
Query
?
?- has-part(ribosome,?x).
Levels of Formality
SemiFormal
Text
Logic
What should
go in here?
Query
?
?- has-part(ribosome,?x).
Outline
1. Halo: The Goal and Road Travelled…
 AURA, Inquire, and reflections
2. Exploiting Semi-Formal Representations and Textual
Inference
3. A New Challenge: Fourth-Grade Science Tests
K-12 Grade Science Tests




Provide a (task-oriented) focus
Simpler (question) language
Involves more common sense
Wide variety of question types
and difficulties
 Caveats
 Multiple choice are common
 Diagrams are common
The 4th Grade NY Regents’ Science Exam
 What types of questions are there?
 What would it take to answer them?
The 4th Grade NY Regents’ Science Exam
 What types of questions are there?
 What would it take to answer them?
The 4th Grade NY Regents’ Science Exam
 What types of questions are there?
 What would it take to answer them?
“Retrieval”
1. Taxonomic
 Question interpretation:
 Decompose question into “isa” queries
 Several good sources of simple “isa” knowledge
 WordNet, Cyc, Wikipedia
 Within text itself
 “isa” knowledge is fundamental to other reasoning types
2. Definitions
Dictionary Resources
erosion: The process of being eroded by wind, water, or other natural agents.
erosion: The wearing away of rocks and other deposits on the earth's surface …
erosion: The gradual wearing away of land surface materials, especially rocks, …
2. Definitions
Dictionary Resources
erosion: The process of being eroded by wind, water, or other natural agents.
erosion: The wearing away of rocks and other deposits on the earth's surface …
erosion: The gradual wearing away of land surface materials, especially rocks, …
Entailment-Style Reasoning
the movement of soil by wind or water
The gradual wearing away of land surface materials, especially rocks,
sediments, and soils, by the action of water, wind, or a glacier.
3. Basic Facts
“Semantic Databases”
 Some basic facts can be pre-extracted and cleaned
 parts, functions, steps in a process, etc.
 + existing resources have some of this knowledge
Building Semantic Databases…
Text
has-part(Leaf,Stomata)
“Stomata in a leaf's surface lead to a
maze of internal air spaces”
Known
parts
LOD
WordNet
AURA
good “parts”
relations
(training data)
Sentences expressing those relations
MultiR (Univ Washington)
Final
parts database
candidate pair, e.g.,
“plant cell” has-part “chloroplast”?
Classifier
Decision
(yes/no +
confidence)
Iterate,
+ Human/
machine
validation
The 4th Grade NY Regents’ Science Exam
 What types of questions are there?
 What would it take to answer them?
“Inference”
4. “Rules” (simple inference)
 Many questions require simple, one-step entailments
 X eats → X gets nutrients
 X breathes oxygen –enables→ X make energy
 X made of metal → X conducts electricity
 Large number of such facts and rules needed
 Manually enter them?
Via:
 Induce them?
Judicious forms of text
Good NLP
 Just read them?
Manual validation
4. Knowledge (Rule) Extraction from Text
Animals take in air by breathing. They need oxygen, which is in the air.
Oxygen allows the animal to make and use energy, which it needs to survive.
Animals also need water to survive. Water is used to break down and move
materials throughout the body. Animals cannot make their own food so they
must eat to get nutrients. Nutrients are necessary for growth and energy.
Assertions
air contains oxygen
animals need oxygen
animals need energy
animals need water
Implications
animal breathes → animal takes in air
animal breathes oxygen -enables→ animal make energy
animal eat -enables→ animal get nutrients
animal get nutrients -enables→ animal grow
animal has water -enables→ animal breakdown materials
4. Knowledge (Rule) Extraction from Text
 Rule acquisition:
 specific patterns in text
X Ys by Z
IF X Zs THEN X Ys
“Animals take in
air by breathing.”
IF an animal breathes
THEN an animal takes in air
 Rule application: using textual entailment-style inference
 If rule condition entailed, then infer conclusion
 Current status: Pretty noisy rules!
The 4th Grade NY Regents’ Science Exam
 What types of questions are there?
 What would it take to answer them?
“Models”
5. Domain Models
 Sometimes you do need some “computational clockwork”
 Qualitative models
 qualitative influences (X goes up → Y goes down)
 what happens to Z if X goes up?
 Process models
 partially ordered network of events
 how does X contribute to Y?
 Acquisition Task ≠ “read the text”
= extract/build model instances from the text
5. Example: Process Models
Process reasoner: Given a process, can answer questions, e.g.
What is the role of Entity in Process?
What Entity performs Role in Event?
During X, what happens after Y?
KA Task = extract a process instance from text:
1. Identify where a process is being described
2. Extract it, e.g., with a set of trained classifiers
“stimulate” [theme: “cell”]
When the cell is stimulated,
gated channels open that
facilitate Na+ diffusion.
Sodium ions then diffuse
down their electrochemical
gradient….
“open” [theme: “gated channels”]
“diffuse” [theme: “sodium ions”,”Na+”
direction: “down ec gradient”]
Extracting Process Models: The annotation tool
Extracting Process Models
flow down
H+ ions
enter
H+ ions flowing down their gradient
enter a half channel in a stator, which
is anchored in the membrane. H+ ions
enter binding sites within a rotor,
changing the shape of each subunit so
that the rotor spins within the
membrane... Spinning of the rotor
causes an internal rod to spin as well.
This rod extends like a stalk into the
knob below it, which is held stationary
by part of the stator. Turning of the rod
activates catalytic sites in the knob
that can produce ATP from ADP and
Pi.
change
gradient
binding site rotor
shape
causes
rotor, rod spin
activate
catalytic site
produce
ATP
ADP, Pi
Another Example: Energy Conversion
 Modeling technique: Energy conversion
 extract event sequence (process model)
 layer energy types on top
 → initial form of energy? final? form that produced X? etc
baby shake rattle
rattle make noise
movement
sound
mechanical energy
sound energy
The 4th Grade NY Regents’ Science Exam
 What types of questions are there?
 What would it take to answer them?
“Diagrams”
6. Diagrams, Images, Tables
 Common in exams; many different styles and challenges
(Non-essential diagram)
6. Diagrams, Images, Tables
 Common in exams; many different styles and challenges
(Hard)
6. Diagrams, Images, Tables
 Common in exams; many different styles and challenges
(Extremely
hard)
Where to?
 Revised picture of intelligence
 Knowledge as a collection of resources, at various
levels of formality
 taxonomic, factual, semi-formal rules, formal models
 Reasoning as a collection of “experts”, with various
specialized skills
 taxonomic, textual entailment, targeted formal systems
 Semi-formal representations avoid some of the
rigidity of deductive logic
 ≠ proof tree, = most plausible chain of inference
 Introspection: Why materialize knowledge at all?
 Allows refinement and inconsistency reduction
What did we learn from Watson?
 The obvious: (judiciously chosen)
 leverage lots of data
 multiple solvers + machine reasoning = better results
 The less obvious:
 evidential reasoning
 not about finding a proof, but searching for evidence
 deduction often comes “tantalizingly close”
“What material is DNA made of?” → “nucleotides”
 no single, pre-defined ontology
“What shape does the six carbon atoms in glucose form?”
 Doesn’t mean we don’t need ontologies!
Summary
 Halo: toward knowledgeable machines
 Now pursuing a quite different model of
intelligence
 Fourth-Grade Science Tests
 Wide variety of question types and challenges






taxonomic
definitional
basic facts
simple (but many possible) inferences from given facts
formal modeling techniques
diagrams
 A good driver and test for this picture!