Reading to Learn
Q3 Review
Peter Clark
John Thompson
Tom Jenkins
Phil Harrison
Bill Murray
Agenda
• This Seedling and Mobius
– Major lessons learned
• Reformulations in CPL
– Whole 5 pages
– Key Sentences
• How do other texts compare?
• Generics
• How to identify “important” text
• Principles for an extensible KB
• Evaluation discussion
• Tuples as another source of knowledge
SRI-Boeing’s Reading to Learn Seedling
• Goal:
– study issues in learning through reading by working with a
reduced version of the problem, namely working with
controlled, rather than unrestricted natural language. The NLP
task is factored into two:
• full NL → CL, CL → logic
this project
• Rationale:
– by sidestepping some of the linguistic issues of full NLP, can
focus on knowledge integration issues
– methods for full NL → CL can be studied separately
SRI-Boeing’s Reading to Learn Seedling
• Approach:
– Rewrite 5 pages of chemistry text into our controlled
language, CPL
– Extend and use our CPL interpreter to generate logic
– Integrate this new knowledge with an existing chemistry
knowledge base (from the Halo Pilot), which has the new
knowledge surgically deleted from it
– Evaluate the performance of the CPL-extended KB with the
original
– Report on the problems encountered and solutions developed
This Seedling in Mobius
Test
Generation
Introspection
Natural Language
Processing
Knowledge
Integration
This seedling
Summary
• Q3:
– Completed coding of key sentences in CPL
– Demonstration of inference with that knowledge
– Study of cues for identifying important text
– Assembly of key lessons learned
– Interaction with ISI
– Exploration of shallow knowledge extraction
• Q4
– Finish interpretation of additional sentences
– Assemble qualitative and quantitive evaluations
– Continue interaction with ISI: Side-by-side study
– Final report
Main Results and Messages
• With some hand-holding, part of the “Mobius loop”
can be done
– But: chemistry is a formidable domain
• Contributions:
– 10 key lessons learned for a larger project
– Qualitative and quantitative evaluation data
10 Key Lessons
• Much of the text is irrelevant (“fluff”)
• Much important knowledge is conveyed by examples & diagrams
• General principles are rarely spelt out clearly
• Text is full of ambiguity, metaphor, and metonymy/“loosespeak”
• Declarative knowledge may be hidden in procedural descriptions
•
•
•
•
•
Text creates disconnected knowledge, which may not chain well
Discourse structure is important
Generic sentences are ubiquitous
Many sentences pose major representational challenges
Traditional KR structures are difficult to extend
Two Reformulations into CPL…
• Reformulation of the whole 5 pages into CPL
– Approximately 250 sentences
– Syntactic conversion + pseudo-logic
– generally not inference capable, esp. generics
• Re-reformulation of first subsection into explicit if-thens
• Inference capable but greater distance from source text
• Reformulation of key pieces into CPL
– approximately 10 if-then rules
– inference capable
– barely recognizable from the original source text
Agenda
• This Seedling and Mobius
– Major lessons learned
• Reformulations in CPL
– Whole 5 pages
– Key Sentences
• How do other texts compare?
• Generics
• How to identify “important” text
• Principles for an extensible KB
• Evaluation discussion
• Tuples as another source of knowledge
Some CPL Rules
IF a substance is an acid THEN the substance tastes sour.
IF an acid contacts an acid-sensitive dye
THEN the acid changes the color of the dye.
IF a substance is a base THEN the substance tastes bitter.
IF a substance is a base THEN the substance feels slippery.
IF a substance is an acid THEN the substance contains hydrogen.
IF a thing is a base THEN the thing is a substance.
IF an Arrhenius base contacts water
THEN the base emits OH-minus ions in the water.
IF an Arrhenius acid is dissolving in water
THEN the dissolving is increasing the concentration of H-plus ions in the water.
IF an Arrhenius base is dissolving in water
THEN the dissolving is increasing the concentration of OH-minus ions in the water
IF a substance is a HCl substance THEN the substance is an Arrhenius acid.
IF hydrogen chloride gas is in water THEN the gas dissolves easily in the water.
IF hydrogen chloride gas is in water THEN the gas reacts with the water.
Reformulation of the 5 pages…
• Note: introductory material, flowery language, fluff,
complex sentences, parentheticals.
IF a substance is an acid THEN the substance tastes sour.
IF an acid contacts an acid-sensitive dye
THEN the acid changes the color of the dye.
IF a substance is a base THEN the substance tastes bitter.
IF a substance is a base THEN the substance feels slippery.
IF a substance is a HCl substance THEN the substance is an Arrhenius acid.
IF hydrogen chloride gas is in water THEN the gas dissolves easily in the water.
IF hydrogen chloride gas is in water THEN the gas reacts with the water.
HCl is the chemical symbol for hydrogen chloride. ← (Implied but not explicit)
IF a substance is an aqueous solution of HCl substance THEN the substance is
hydrochloric acid.
IF a substance is concentrated hydrochloric acid THEN 37 percent of the mass of
the substance is HCl.
IF a substance is concentrated hydrochloric acid THEN the concentration of HCl in
the substance is 12 M.
(surface
logical
form)
CPL
Halo KB
style
the'(e1,x1,e2) & aqueous'(e3,x1) & solution'(e2,x1) & of'(e4,x1,x2)
& hcl'(e5,x2) & know'(e6,z1,x1,x3) & as'(e7,e6,x3) &
hydrochloric'(e8,x3) & acid'(e9,x3)
IF a substance is an aqueous solution of HCl substance
THEN the substance is hydrochloric acid.
(every Hydrochloric-Acid has-definition
(instance-of (Aqueous-Solution))
(has-solute ((a HCl-Substance)))
Summary of Interpretation Challenges
• Interpreting generics.
– "Acids cause some dyes to change color."
• how to handle negation.
– "Some substances containing hydrogen are not acids."
– "The transfer leaves no undissociated acid molecules"
• Vague attributes ("properties", "due to")
– “Properties of aqueous solutions of Arrhenius acids are due to H-plus ions"
– coreference with nominalizations ("react"/"reaction")
– "Hydrogen chloride reacts... The reaction produces..."
• naming: how to represent both the name and the symbol for a chemical.
– "An aqueous solution of HCl is called hydrochloric acid."
• how to get new technical vocabulary + meanings into the system.
– "NaOH dissociates in water."
– "H2O abstracts the proton from HX"
• how to represent definitions.
– "Arrhenius acids and defined..."
• how to state that one category is more general than another.
– "Bronsted-Lowry acids are more general than Arrhenius acids."
Summary of Interpretation Challenges (cont)
• how to represent "sometimes".
– "An HO3-plus ion sometimes reacts with an H2O molecule."
• how to represent modals/tendancies like "can".
– "A molecule of a Bronsted-Lowry acid can donate a proton..."
• how to represent an argument (proof), and generalize from it.
– "Therefore, the H2O molecule acts as a Bronsted-Lowry base.“
– "Substances with negligible acidity contain hydrogen, but the substances do
not behave as acids in water."
• vagueness ("is mostly", "nearby", "some")
– "The NH4Cl is mostly solid particles."
– "Some acids are better proton donors than other acids."
– "A weak acid partly transfers the acid's protons to the water."
– "Proton-transfer reactions are governed by the relative strengths of the bases"
– "The solution has a negligible concentration of HCl molecules."
– "An aqueous solution of acetic acid consists mainly of HC2H3O2 molecules"
– "The aqueous solution has relatively few H3O-plus ions"
• metonymy
– "The H2O molecule in Equation 16.5 donates a proton"
– "In Equation 16.9 HX dissolves in water."
– "Equation 16.9 describes the behavior of a strong acid in water."
Summary of Interpretation Challenges (cont)
• definitions with negation.
– "An H-plus ion is a proton with no valence electron."
• presuppositions
– "Acids cause some dyes to change color."
– "A Bronsted-Lowry acid always reacts with a nearby Bronsted-Lowry base."
• generalized formulae and equations
– "In Equation 16.6 the symbol HX denotes an acid."
• how to compute and represent differences
– "An acid and a base differing only in a proton are called a conjugate pair"
• how to handle definite references ("the" base) that haven't been introduced.
– "Removing a proton from the acid produces the conjugate base."
• change over time
– "The HNO2 molecule becomes the NO2-minus ion."
– "The H2O molecule changes into the hydronium ion"
– "Acids cause some dyes to change color."
• semi-malformed sentences
– "A stronger acid has a weaker conjugate base."
• How to state and represent hypothetical situations.
– "Assume that H2O is a stronger base than X-minus in Equation 16.9."
Summary of Interpretation Challenges (cont)
• Generalization from examples
– “In any reaction we can identify two sets of conjugate acid-base pairs. For
example, consider the reaction…”
• Information in tables and diagrams
Agenda
• This Seedling and Mobius
– Major lessons learned
• Reformulations in CPL
– Whole 5 pages
– Key Sentences
• How do other texts compare?
• How to identify “important” text
• Principles for an extensible KB
• Evaluation discussion
• Tuples as another source of knowledge
Recall from Last Time …
• Most of the textbook sentences are “fluff” and
examples
– and are not needed to solve test questions
• A few key sentences (and a table) are the heart of
this section of the textbook
– and are often given in italics
• These key sentences are not worded as precisely
as needed for automatic translation into axioms
that can chain together to solve a problem
– in fact, some parts are not stated at all
– students look at diagrams and examples and
figure it out
Overview
• 4 key pieces of knowledge in the Section:
– Computing the direction of the reaction
• Rewriting in CPL
• Compare to UT’s KM encoding
• Compare to ISI’s shallow logical form
– Identifying the acids/bases in a reaction
– Computing the conjugate of an acid/base
– Comparing the strengths of two acids/bases
Overview
• 4 key pieces of knowledge in the Section:
– Computing the direction of the reaction
• Rewriting in CPL
• Compare to UT’s KM encoding
• Compare to ISI’s shallow logical form
– Identifying the acids/bases in a reaction
– Computing the conjugate of an acid/base
– Comparing the strengths of two acids/bases
A Key Sentence in Our Textbook
• Let’s look at one example of a key sentence:
• “From these examples we conclude that in every
acid-base reaction the position of the equilibrium
favors transfer of the proton to the stronger base.”
• Restated in Sample Exercise 16.3:
• “Thus, the equilibrium favors the direction in
which the proton moves from the stronger acid
and becomes bonded to the stronger base.”
• “In other words, the reaction favors consumption
of the stronger acid and stronger base and
formation of the weaker acid and weaker base.”
Rewriting a Sentence into CPL
Textbook
“In every acid-base reaction the position of the equilibrium favors transfer of
the proton to the stronger base.”
Naïve Encoding 1
IF there is a reaction
AND one base in the reaction is stronger than the other base in the reaction
THEN the direction of the reaction is away from the stronger base.
[“favors transfer to” → “direction is away from”]
Naïve Encoding 2
IF there is a reaction
AND there is a base on the left side of the reaction
AND there is a base on the right side of the reaction
AND the first base is stronger than the second base
THEN the direction of the reaction is to the right.
Further Refinement of the CPL
Naïve Encoding 2
IF there is a reaction
AND there is a base on the left side of the reaction
AND there is a base on the right side of the reaction
AND the first base is stronger than the second base
THEN the direction of the reaction is to the right.
“The chemical entity whose formula is on the left side of the
equation of the reaction and which plays a base role”
Final CPL Rule That Worked!
IF there is an equation of a reaction
“the base on the LHS”
AND a first chemical entity has a chemical formula
AND the first chemical formula is part of the left side of the equation
AND the first chemical entity is playing a base role
“the base on the RHS”
AND a second chemical entity has a second chemical formula
AND the second chemical formula is part of the right side of the equation
AND the second chemical entity is playing a base role
AND the first chemical entity is stronger than the second chemical entity
(means “stronger base than”)
THEN the direction of the reaction is right [to the right]
AND the equilibrium side of the reaction is right. [lies on the right]
(UT’s rep. uses Reaction, but should use Equation)
Compare Sentence to Final CPL
• In every acid-base reaction the position of the
equilibrium favors transfer of the proton to the stronger
base.
not actually used!
• IF there is an equation of a reaction
AND a first chemical entity has a chemical formula
AND a second chemical entity has a second chemical formula
AND the first chemical formula is part of the left side of the equation
AND the second chemical formula is part of the right side of the equation
AND the first chemical entity is playing a base role
AND the second chemical entity is playing a base role
AND the first chemical entity is stronger than the second chemical entity
THEN the direction of the reaction is right
AND the equilibrium side of the reaction is right.
• (There’s a 2nd rule like this that concludes the direction is left)
KM Generated from CPL
IF
THEN
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
(_Equation7461 equation-of _Reaction7462)
(|_Chemical Entity7468| has-chemical-formula |_Chemical Formula7469|)
(|_Chemical Formula7469| equal _Part7485)
(_Part7485 is-part-of |_Left Side7483|)
(|_Left Side7483| is-region-of _Equation7461)
(|_Chemical Entity7475| has-chemical-formula |_Chemical Formula7476|)
(|_Chemical Formula7476| equal _Part7494)
(_Part7494 is-part-of |_Right Side7492|)
(|_Right Side7492| is-region-of _Equation7461)
(|_Chemical Entity7468| plays |_Base Role7501|)
(|_Chemical Entity7475| plays |_Base Role7508|)
(|_Chemical Entity7468| stronger-base-than |_Chemical Entity7475|)

(_Direction7518 value *right)
(_Direction7518 direction-of _Reaction7462)
(|_Equilibrium Side7524| property *right)
(|_Equilibrium Side7524| equilibrium-side-of _Reaction7462)
chem.
on
LHS
chem.
on
RHS
Structure of the CPL Axioms
1. Find equilibrium side (or direction) of equation
2. Find out if a chemical is
playing a base role in the
equation
4. Check whether one
base is stronger than
another base
4a. Look in Table
3. Find out if a chemical is
the conjugate base of
another chemical
3a. Look in Table, or …
3b. Check whether one
formula differs from
(not in CPL)
another in an H+
Notes on our CPL Rule
• The wording is way different from the original text!
• The literal sentence translation would not have
produced anything that could solve a problem, given an
equation
• “In every acid-base reaction the position of the
equilibrium favors transfer of the proton to the stronger
base.”
– this would create a Favoring event
– the position of the equilibrium is the agent
– the transfer of the proton is the object
– what does this mean?
Overview
• 4 key pieces of knowledge in the Section:
– Computing the direction of the reaction
• Rewriting in CPL
• Compare to UT’s KM encoding
• Compare to ISI’s shallow logical form
– Identifying the acids/bases in a reaction
– Computing the conjugate of an acid/base
– Comparing the strengths of two acids/bases
How UT Encoded This
•
"In acid/base equilibrium reactions, the reaction proceeds in the direction of the side where
equilibrium lies“ [their comment for use in explanations]
•
(every Reaction has …
(direction (
(if (not (the direction of Self))
then
(a Direction-Value with
(value ((if (the output of
To find the direction of a reaction…
(a Compute-Equilibrium-Position with
(input (Self))))
then
Compute the equilibrium position …
(if ((the output of
(a Compute-Equilibrium-Position with
(input (Self))))
=
(the raw-material of Self))
then
If the chemicals match the raw materials
*left
else
*right))))) Then the direction is left, else right
UT’s Compute-Equilibrium-Position
(every Compute-Equilibrium-Position has
(input ((a Reaction)))
(output (
;; See if both the strong acid and base are on the LHS.
(if (;; Check the acids.
((the output of
(a Compare-Relative-Strengths-of-Acids with
If the stronger of…
(input (
(oneof (the raw-material of (the input of Self))
the raw material acid…
where
(the Acid-Role plays of It))
(oneof (the result of (the input of Self))
and the result acid…
where
(the Acid-Role plays of It))))))
= (oneof (the raw-material of (the input of Self))
is the raw material acid…
where (the Acid-Role plays of It)))
and
;; Check the bases.
((the output of
(a Compare-Relative-Strengths-of-Bases with
(input ( (oneof (the raw-material of (the input of Self))
where (the Base-Role plays of It))
(same for bases)
(oneof (the result of (the input of Self))
where (the Base-Role plays of It))))))
=
(oneof (the raw-material of (the input of Self))
then equilibrium is on the
where (the Base-Role plays of It))))
result side
then (the result of (the input of Self))
else (the raw-material of (the input of Self))))))
else the raw material side
Notes on UT’s Encoding
• Very procedural!
• Various procedural methods are encoded
– both qualitative and quantitative
• Nothing like the textbook sentences
• Their representation does not match the natural
conceptual model we expected
– see the next slide
Mismatches between UT and CPL
• UT put a “direction” slot on a Reaction, we expected it
to be on an Equation
• UT has no model of the left and right sides of an
Equation, only the “raw-materials” and “result” slots of
a Reaction
• UT has a Conjugate-Acid-Base-Pair concept, but lacks
the conjugate-base & conjugate-acid relations we
expected
• UT has no slot for the “equilibrium-side” of an
Equation, only the “direction” of a reaction
More Mismatches between UT and CPL
• UT gives us no primitives to use for formula
manipulation (adding an H+), it’s buried within their
Compute-Conjugate-Acid
• UT’s model of Formula does not include a “charge” slot,
they’ve only attached it to the Chemical itself
• UT has no notion of “stronger-base-than,” they only
label a chemical with “intensity” = strong or weak.
• So, it would help if the conceptual model were closer to
natural language!
Overview
• 4 key pieces of knowledge in the Section:
– Computing the direction of the reaction
• Rewriting in CPL
• Compare to UT’s KM encoding
• Compare to ISI’s shallow logical form
– Identifying the acids/bases in a reaction
– Computing the conjugate of an acid/base
– Comparing the strengths of two acids/bases
ISI’s Shallow Logical Form for our Sentence
“From these examples we conclude that in every acidbase reaction the position of the equilibrium favors
transfer of the proton to the stronger base.”
from'(e1,e2,x1) &
these'(e3,s1,e4) &
example'(e4,x1) &
plural'(e7,x1,s1) &
we'(e8,x2) &
plural'(e9,x2,s2) &
conclude'(e2,x2,x3,z1) &
that'(e10,e2,e11) &
in'(e12,e11,x4) &
every'(e13,x4,e14) &
acid-base'(e15,x4) &
reaction'(e14,x4) &
the'(e16,x5,e17) &
position'(e17,x5) &
of'(e18,x5,x6) &
the'(e19,x6,e20) &
equilibrium'(e20,x6) &
favor'(e11,x5,x7,z2) &
transfer'(e21,x7) &
of'(e22,x7,x8) &
the'(e23,x8,e24) &
proton'(e24,x8) &
to'(e25,x7,x9) &
the'(e26,x9,e27) &
strong'(e28,x9) &
base'(e27,x9)
Graph of ISI’s Shallow Logical Form
?
from (x1)
z1 = conclude(x2, x3)
x2 = we
x3 = [missing!]
?
that
in(x4)
x1 = example
?
these
x4 = reaction
z2 = favor (x5, x7)
x5 = position
every(x4) acid-base(x4)
x7 = transfer
of (x5, x6)
of (x7, x8)
to (x7, x9)
x6 = equilibrium
x8 = proton
x9 = base
strong (x9)
Notes on ISI’s Shallow Logical Form
• Not far removed from a syntactic parse
• They plan to do much more development of this
• Will probably produce a literal translation
– there will be a Favoring event, with agent & object
• As with the naïve CPL sentence, a literal translation
would not help solve a Chemistry problem
Overview
• 4 key pieces of knowledge in the Section:
– Computing the direction of the reaction
• Rewriting in CPL
• Compare to UT’s KM encoding
• Compare to ISI’s shallow logical form
– Identifying the acids/bases in a reaction
– Computing the conjugate of an acid/base
– Comparing the strengths of two acids/bases
CPL for 2nd Key Sentence
• “In any acid-base (proton transfer) reaction we can
identify two sets of conjugate acid-base pairs.”
• IF there is an equation of a reaction
AND a first chemical entity has a chemical formula
AND a second chemical entity has a second chemical formula
AND the first chemical formula is part of the left side of the
equation
AND the second chemical formula is part of the right side of the
equation
AND the first chemical entity is the conjugate base of the
second chemical entity
THEN the first chemical entity is playing a base role
AND the second chemical entity is playing an acid role.
• (There’s a 2nd rule like this with first & second reversed)
UT Code for 2nd Key Sentence
(every Chemical has
jump to the other
(plays (
side of the equation!
(if
((the term of (the atomic-chemical-formula of
(the has-basic-structural-unit of Self)))
Reaction
and (not (the Base-Role plays of Self)))
result
then
raw-material
(if ((has-value (oneof
(the result of (the Reaction raw-material-of of Self))
Chemical
Chemical
where (((the elements of
(the term of
“IF one of the chemicals
(the atomic-chemical-formula of
(the has-basic-structural-unit of It))))on the other side of the
reaction…”
= (forall2 (the elements of (the term of
(the atomic-chemical-formula of
(the has-basic-structural-unit of Self))))
(if ((the2 of It2) = H)
then (:pair ((the1 of It2) + 1) H)
“… has an extra H”
else It2)))
or...
then (a Base-Role)
“…THEN this chemical’s a base”
Overview
• 4 key pieces of knowledge in the Section:
– Computing the direction of the reaction
• Rewriting in CPL
• Compare to UT’s KM encoding
• Compare to ISI’s shallow logical form
– Identifying the acids/bases in a reaction
– Computing the conjugate of an acid/base
– Comparing the strengths of two acids/bases
• These last two items are presented in a table
Conjugate Acid-Base Pairs
CPL
IF there is an HCl and a Cl-Minus
THEN the conjugate base of the HCl is the
Cl-minus.
IF there is an H3O-Plus and an H2O
THEN the conjugate base of the H3O-Plus
is the H2O.
Etc.
Textbook
Relative Strengths of Bases
CPL
IF there is a Cl-Minus and an HSO4-Minus
THEN the HSO4-Minus is a stronger base than
the Cl-Minus.
IF there is a HSO4-Minus and an NO3-Minus
THEN the NO3-Minus is a stronger base than the
HSO4-Minus.
IF there is an NO3-Minus and an H2O
THEN the H2O is a stronger base than the NO3Minus.
Etc.
Textbook
Lessons from Key Sentences - 1
• The key sentences did not translate literally into useful logic
– they had to be carefully rewritten in CPL
– and knowledge was added from studying diagrams and
examples
– and they were tested with each other to chain together
• It was difficult to make use of the UT representations
– they were very procedural
– their representations were further removed from the English
– so, we should use more natural representations
• ISI’s shallow logical forms may produce literal translations
– again, not useful for solving problems
Lessons from Key Sentences - 2
• Reading knowledge directly from a Chemistry text would be very
challenging
– the knowledge has to be written precisely enough for a
computer (with little common sense) to encode
– knowledge in tables and diagrams may be critical
– the knowledge has to chain together to solve difficult exam
problems
– we need text that is written much more dryly and precisely
– we need a domain that doesn’t have such difficult exam
problems
Agenda
• This Seedling and Mobius
– Major lessons learned
• Reformulations in CPL
– Whole 5 pages
– Key Sentences
• How do other texts compare?
• Generics
• How to identify “important” text
• Principles for an extensible KB
• Evaluation discussion
• Tuples as another source of knowledge
Are Other Chemistry Texts Better?
• We looked at Web explanations and at ‘Chemistry Made Simple’
types of books
• Discovered that each teacher explains it differently
• Most jump right into quantitative formulas for computing where a
reaction’s equilibrium lies
– but our textbook teaches it qualitatively first, which is rare
• Other sources are not any easier to process
Examples of Other Sources
• “Think of a Bronsted acid-base reaction as a competition between the
2 bases in the system for protons. The stronger base ‘wins” and forces
the equilibrium in the direction of the weaker acid and base.” (Web)
• [some books say that an acid is a proton donor] “… the acid molecule
does not ‘give’ or ‘donate’ the proton, it has it taken away. In the same
sense, you do not donate your wallet to the pickpocket, you have it
removed from you.” (another website)
• “The base is a molecule with a built-in ‘drive’ to collect protons. As
soon as the base approaches the acid, it will (if it is strong enough) rip
the proton off the acid molecule and add it to itself.”
More Examples from Other Sources
• “You see, some bases are stronger than others, meaning some have a large
‘desire’ for protons, while other bases have a weaker drive. It’s the same
way with acids, some have very weak bonds and the proton is easy to pick
off, while other acids have stronger bonds, making it harder to ‘get the
proton’.”
• “Remember that an acid-base reaction is a competition between two bases
(think about it!) for a proton. If the stronger of the two acids and the
stronger of the two bases are reactants (appear on the left side of the
equation), the reaction is said to proceed to a large extent.”
• Note the heavy use of metaphors in these qualitative explanations!
• The more readable by humans, the less readable by computers!
Agenda
• This Seedling and Mobius
– Major lessons learned
• Reformulations in CPL
– Whole 5 pages
– Key Sentences
• How do other texts compare?
• Generics
• How to identify “important” text
• Principles for an extensible KB
• Evaluation discussion
• Tuples as another source of knowledge
Agenda
• This Seedling and Mobius
– Major lessons learned
• Reformulations in CPL
– Whole 5 pages
– Key Sentences
• How do other texts compare?
• Generics
• How to identify “important” text
• Principles for an extensible KB
• Evaluation discussion
• Tuples as another source of knowledge
Review
• Earlier analysis:
– Much of the textbook is “irrelevant” for the
purposes of computer-based reading
• motivational material, illustrative material, humor
– Other sentences/parts are critical
• Questions:
– Can a computer automatically find the critical
items?
– What cues might indicate the important
material?
This brief analysis…
• Here, just consider two categories:
– important vs. unimportant material
• Categories of surface cues:
– linguistic
– context
– layout
– typography (e.g., font changes)
• Looked at several text books:
– B&L, Chemistry Made Simple, Cliffs Notes
Cues for Importance/Unimportance
• Verb tense: past tense suggests irrelevance
– chemical facts are generally presented in the present
tense; past tense usually signals a historical
digression; but biological facts include evolutionary
facts, which require past tense.
• Cue phrases for important generalizations
– “for example” and (less so) “thus” precede examples
but follow important generalizations.
Cues for Importance/Unimportance
• Long sentences (>20) suggest irrelevance
– Average sentence length for chemistry is about 15
words; biology, ca. 24 words.
– 15 words seems to allow a good balance of
simplicity and complexity for stepping through
explanations. CPL should target this number.
– Summaries tend to have long complex sentences
that are harder to process. Also true for sentences
in review texts: Cliffs Notes, Instant Notes.
Cues for Importance/Unimportance
• Everyday words suggest applications.
• Nominalized verbs suggest irrelevance
– exception: basic chemical changes (e.g., reaction,
combustion, evaporation)
• Keywords:
– “if”, “when”, “because”, “for” indicate important
sentences
– “For example” precedes an illustration
• also indicates stuff prior is an important generality
– “although”: typically part of fluffy sentence
Cues for Importance/Unimportance
• Definitional patterns: important!
– “x is substance y”, “x is a y that does z”, “x is
called y”
• First and last sentences in a paragraph tend to be
important (unless transitional)
– set the topic of the paragraph
• Text in bold or italics is often important
• Repetition: could this be exploited?
Summary
• Many surface cues exist
• Could identify important material by
– surface cues
– “deeper” model of the document structure
• e.g. Motivation → General principle → Example →
Reinforce general principle
• Could the document automatically be turned into a
labeled, networked structure like this?
• How document-specific are these patterns?
Agenda
• This Seedling and Mobius
– Major lessons learned
• Reformulations in CPL
– Whole 5 pages
– Key Sentences
• How do other texts compare?
• Generics
• How to identify “important” text
• Principles for an extensible KB
• Evaluation discussion
• Tuples as another source of knowledge
Principles for an Extensible KB
Elaboration Tolerance:
A formalism is elaboration tolerant to the extent that it is convenient to
modify a set of facts expressed in the formalism to take into account new
phenomena or changed circumstances. [John McCarty]
e.g.: add/modify knowledge (semantics) by (only)
adding formulae (syntactics)
Three Key Desirables for this:
• Syntactic simplicity
• Metonymy-tolerant reasoning
• Separate procedural and declarative knowledge
Syntactic Simplicity
•  Many syntactically large and complex structures in
the original Halo KB, e.g.,
(every Acid-Role has
Not elaboration-tolerant
(intensity (
(a Intensity-Value with
(value (
(:pair
;; Case statement for Acids.
(if ((the played-by of Self) isa Ionic-Compound-Substance)
then
(if (((the played-by of Self) isa HCl-Substance) or
((the played-by of Self) isa HBr-Substance) or
((the played-by of Self) isa HI-Substance)
or
((the played-by of Self) isa HClO3-Substance) or
((the played-by of Self) isa HClO4-Substance) or
((the played-by of Self) isa H2SO4-Substance) or
((the played-by of Self) isa HNO3-Substance))
then *strong else
Syntactic Simplicity
•  Better would be to factor them smaller units, e.g.,
Elaboration-tolerant
intensity(HCl-Substance, *strong)
intensity(HBr-Substance, *strong)
intensity(HI-Substance, *strong)
intensity(HClO3-Substance, *strong)
intensity(HClO4-Substance, *strong)
intensity(H2SO4-Substance, *strong)
intensity(HNO3-Substance, *strong)
…
intensity(HF-Substance, *weak)
intensity(HC2H3O2-Substance, *weak)
intensity(H2CO3-Substance, *weak)
…
CPL Produces Syntactically Simple Structures…
“Traditional” KM:
(every Compare-Relative-Strengths-of-Acids has
(output ((if ((the intensity of (the first of (the Chemicals)) = *strong)
and ((the intensity of (the second of (the Chemicals)) = *weak)
then (the strongest of (the Chemicals)) = (the first of (the Chemicals)))))
CPL triples:
IF
(_Intensity9 instance-of Intensity-Value)
(_Chemical8 instance-of Chemical)
(_Intensity5 instance-of Intensity-Value)
(_Chemical4 instance-of Chemical)
(_Intensity5 property *strong)
(_Intensity5 intensity-of _Chemical4)
(_Intensity9 property *weak)
(_Intensity9 intensity-of _Chemical8)
THEN
(_Chemical4 stronger-than _Chemical8)
Metonymy/Loosespeak
• Metonymy: One word substitutes for a closely related word
• Loosespeak: More generally, the “literal” interpretation is wrong
• Examples:
– “The kettle is boiling.”
– “I’m just going to change the washing machine.”
– “It’s your turn to clean out the rabbit.”
– “Remove a proton from the acid”
– “The acid on the left of the equation”
– “The reaction moves to the right”
– “NaCl dissolves in water”
Handling Metonymy/Loosespeak
1. Detect inconsistencies / “unusualities”
Need extensive world knowledge for this
2. If found, create and evaluate alternative interpretations
– Metonymic transformation rules (e.g., Lakoff, Fass)
•
•
•
•
•
PART for WHOLE (“Get your butt over here”)
PLACE for INSTITUTION (“The White House isn’t saying anything”)
PLACE for EVENT (“Remember the Alamo”)
SUBSTANCE for MOLECULE (“NaCl dissolves”)
FORMULA for SUBSTANCE (“NaCl is on the left of the eqn”)
Metonymy Tolerance (“Loosespeak”)
• Could greatly reduce syntactic complexity
– ~50% of HaloKB is doing type conversions
• Example of extensive metonymy:
“HC2H3O2(aq)+…C2H3O2-”
basic-unit
formula
Metonymy Tolerance
(every Compare-Relative-Strengths-of-Acids has
(output ((if (((the1 of (the value of (the intensity of
(the Acid-Role plays of
(the first of (the input of Self))))))
= *strong)
and
((the1 of (the value of (the intensity of
(the Acid-Role plays of
(the second of (the input of Self))))))
/= *strong))
then
(the first of (the input of Self)))))
if we had a metonymy-tolerant
reasoner, we could instead write…
(every Compare-Relative-Strengths-of-Acids has
(output ((if ((the intensity of (the first of (the Chemicals)) = *strong)
and ((the intensity of (the second of (the Chemicals)) /= *strong)
then (the strongest of (the Chemicals)) = (the first of (the Chemicals)))))
Separating Procedural and Declarative Knowledge
• Procedural descriptions are uni-directional, and
difficult to introspect on
• Better: domain-specific, declarative knowledge +
general-purpose procedural algorithms
“Every acid has a conjugate base, formed by removing a
proton from the acid. ... Similarly, every base has associated
with it a conjugate acid, formed by adding a proton to the
base.”
Acid-Chemical = Base-Chemical + H+
Separating Procedural and Declarative Knowledge
Mixed
(every Compare-Relative-Strengths-of-Acids has
(output ((if (((the1 of (the value of (the intensity of
(the Acid-Role plays of
(the first of (the input of Self))))))
= *strong)
and
((the1 of (the value of (the intensity of
(the Acid-Role plays of
(the second of (the input of Self))))))
/= *strong))
then
(the first of (the input of Self))
[Compare-Relative-Strengths-of-Acids-output-1]
)))
“Theory of magnitudes”
Declarative
HCl
*strong
H2CO3 *weak
…
…
+
*strong > *weak > …
+
Procedural (PSM)
Find object(s) with
qualitatively largest
attribute value
Agenda
• This Seedling and Mobius
– Major lessons learned
• Reformulations in CPL
– Whole 5 pages
– Key Sentences
• How do other texts compare?
• Generics
• How to identify “important” text
• Principles for an extensible KB
• Evaluation discussion
• Tuples as another source of knowledge
Possible Quantitative Metrics
• Behavioral:
– Ablation study: Question-answering performance
• Analytic:
– Complexity of CPL vs Halo KB encodings
– Amount of domain K added by Boeing in writing CPL
– % of Halo KB that would be simplified if metonymy handled
– % of original text encodable in CPL
– Time taken to encode the KBs
– % of source text which is important (vs. fluff)
– Bar graph of textual phenomena vs. frequency of occurrence
• e.g., metaphor, examples, metonymy, diagrams
– Measure of redundancy in the text book
Behavioral Evaluation:
Ablation Methodology
Approach:
a. Create set of questions
b. Send qns to Halo KB, measure % correct
c. Ablate the Halo KB, add in ours
d. Send qns to new KB, measure % correct
Issues:
• How to ensure a fair comparison?
– defining the space of questions to look at
• How to ablate the UT KB?
Behavioral Evaluation:
Relevant AP Questions from the Halo Pilot
Questions from Halo Pilot Syllabus & Sample Qns
• Q10. Given an equilibrium reaction, which species in the reaction act as bases?
• Q33. Each of the following can act as both a Bronsted acid and a Bronsted base
EXCEPT ...
Questions from Challenge Exam, Project Halo
• Q18. Given an equilibrium reaction, the species that act as acids include which
of the following?
• Q19. Given an equilibrium reaction, the correct acid/conjugate base pair is ...
• Q37. Which of the following species forms an acid when added to water?
• Q38. Which of the following (lists of chemicals) is in correct order of
increasing acidity?
Behavioral Evaluation:
Variations on a Theme…
• Four main question patterns:
– What is the conjugate base/acid of X?
– Is X stronger/weaker acid/base than Y?
– Find the conjugate acid-base pairs in equation E
– What is the direction of the equilibrium?
Core Knowledge Encodings
Task
Halo KB
CPL
Conjugate pairs
Giant KM procedure for
formula manipulation
Lookup table
Relative strengths
Qualitative absolute strengths
(strong/weak/negligible)
+ qualitative comparison
Relative strength
assertions
Labelling
acid/bases in a
reaction
Giant KM procedure for
reaction manipulation
if-then rule using
conjugate pairs
Computing
direction of the
reaction
KM rule
if-then rule
Core Knowledge Encodings
Task
Halo KB
More general
CPL
Conjugate pairs
Giant KM procedure for
formula manipulation
Lookup table
Relative strengths
Qualitative absolute strengths
(strong/weak/negligible)
+ qualitative comparison
Relative strength
assertions
Labelling
acid/bases in a
reaction
Giant KM procedure for
reaction manipulation
Computing
direction of the
reaction
KM rule
≈
≈
if-then rule using
conjugate pairs
if-then rule
(equivalent)
Behavioral Evaluation: Discussion Points
• We can predict the outcome of any evaluation
– can see the internals of each system
• So what is a fair sample set?
– Generate instantiations of the 4 templates?
– AP exam questions?
– Extend to cover other knowledge in the 5 pages?
• none of it contained in Halo KB
Analytic Evaluation
• Possible Metrics include:
– Complexity of CPL vs Halo KB encodings
– Amount of domain K added by us in writing CPL
– % of KB simplified if metonymy handled
– % of original text encodable in CPL
– Time taken to encode the KBs
– % of source text which is important (vs. fluff)
– Bar graph of textual phenomena vs. frequency
• e.g., metaphor, examples, metonymy, diagrams
– Measure of redundancy in the text book
Agenda
• This Seedling and Mobius
– Major lessons learned
• Reformulations in CPL
– Whole 5 pages
– Key Sentences
• How do other texts compare?
• Generics
• How to identify “important” text
• Principles for an extensible KB
• Evaluation discussion
• Tuples as another source of knowledge
Knowledge Mining
Schubert’s Conjecture:
There is a largely untapped source of general knowledge in
texts, lying at a level beneath the explicit assertional
content, and which can be harnessed.
“The camouflaged helicopter landed near the embassy.”
 helicopters can land
 helicopters can be camouflaged
Our attempt: “lightweight” LFs generated from Reuters
LF forms: (S subject verb object (prep noun) (prep noun) …)
(NN noun … noun)
(AN adj noun)
Knowledge Mining
Newswire Article
HUTCHINSON SEES HIGHER
PAYOUT. HONG KONG. Mar 2.
Li said Hong Kong’s property
market remains strong while its
economy is performing better than
forecast. Hong Kong Electric
reorganized and will spin off its
non-electricity related activities.
Hongkong Electric shareholders
will receive one share in the new
subsidiary for every owned share
in the sold company. Li said the
decision to spin off …
Implicit, tacit knowledge
Shareholders may receive shares.
Shares may be owned.
Companies may be sold.
Knowledge Mining – our attempt
Fragment of the raw data (Brown & Lemay)
;; Atoms can combine
(S "atom" "combine")
;; For example, combustion reactions are redox reactions because elemental
oxygen is converted to compounds of oxygen (Section 3.2).
(S "reaction" "be" "reaction")
(S-ADJ "oxygen" "converted" ("to" "compound"))
(AN "elemental" "oxygen")
;; Plan: Metals react with acids to form salts and gas.
(S "metal" "react" (PP "with" "acid"))
;; Extensive oxidation can lead to the failure of metal machinery parts or the
deterioration of metal structures.
(S "oxidation" "lead" (PP "to" "failure"))
(S "oxidation" "lead" (PP "to" "deterioration"))
(AN "extensive" "oxidation")
Summary
• Q3:
– Completed coding of key sentences in CPL
– Demonstration of inference with that knowledge
– Study of cues for identifying important text
– Assembly of key lessons learned
– Interaction with ISI
– Exploration of shallow knowledge extraction
• Q4
– Finish interpretation of additional sentences
– Assemble qualitative and quantitive evaluations
– Continue interaction with ISI: Side-by-side study
– Final report