University of Texas at Austin
Ken Barker, Shaw Yi Chaw, James Fan, Bruce Porter ,
Dan Tecuci, Peter Yeh
SRI International
Vinay Chaudhri, David Israel, Sunil Mishra
Boeing Phantom Works
Peter Clark
* Funding for various parts of this research was provided by
DARPA, Vulcan, Inc, Boeing, and the US Army.

• Knowledge Systems use formal representations of knowledge to answer unanticipated questions with coherent explanations
• Knowledge System = KB + Q/A +
Explanation Generator +
Knowledge Acq. tools

•
Coverage
U of domain, not domain task
• Various modes of reasoning, well integrated
• Domain level explanation
•
Rapid construction
Just how advanced are they?

*
• Long term: build a Knowledge System encompassing much of the world’s scientific knowledge
• Short term: assess current technologies
• Use a portion of the Advance Placement (AP) chemistry exam as a metric
* Full support for Project Halo was provided by Vulcan Inc, Seattle, WA

Systems must be robust in the face of widely varying, unanticipated questions.
Explanations are as important as correctness.
Hard-ball evaluation, aimed to expose weaknesses.
New domain and short development time require using off-the-shelf KR&R methods and systems.
It was not clear at the outset that these challenges could be met.

• The spectator ions in the reaction of barium nitrate with sodium sulfate are what? (choices)
• Although nitric acid and phosphoric acid have very different properties as pure substances, their aqueous solutions possess many common properties. List some general properties of these solutions and explain their common behavior in terms of the species present.
• Explain why a solution of HClO
4 act as a buffer solution.
and NaClO
4 cannot
• Sodium azide is used in air bags to rapidly produce gas to inflate the bag. The products of the decomposition reaction are what? (choices)
Questions were manually encoded in our formal language

… we can not anticipate the questions, or even the type of questions, so a retrieval method won’t do.
A custom inference method won’t do.
The system must be capable of using its knowledge in unanticipated ways.

•
What are the products of the given decomposition reaction?
•
By definition, oxidation-reduction reactions occur when electrons are transferred from the atom that is oxidized to the atom that is reduced. We need to look for changes in the oxidation states of the elements in the reaction.
• In the reactants, the oxidation state(s) of the element Na is/are (1). In the product, the oxidation state(s) is/are (0)
• Therefore, the reaction causes a change in oxidation state.
• Therefore, this is an oxidation reduction reaction.
• By definition, a Binary Ionic-Compound Decomposition Reaction occurs when a binary ionic compound is heated.
•
Therefore, this reaction is a Binary-Ionic Compound Decomposition reaction.
• In general, a Binary Ionic-Compound Decomposition Reaction converts a binary ionic-compound into basic elements.
• In this reaction, NaN
3 reacts to produce Na and N
2
.
• The products of the decomposition reaction are: (d) Sodium and nitrogen-g

• What are the products of the given decomposition reaction?
• By definition, oxidation-reduction reactions occur when electrons are transferred from the atom that is oxidized to the atom that is reduced. We need to look for changes in the oxidation states of the elements in the reaction.
• In the reactants, the oxidation state(s) of the element Na is/are (1). In the product, the oxidation state(s) is/are (0)
• Therefore, the reaction causes a change in oxidation state.
• Therefore, this is an oxidation reduction reaction.
• By definition, a Binary Ionic-Compound Decomposition Reaction occurs when a binary ionic compound is heated.
• Therefore, this reaction is a Binary-Ionic Compound Decomposition reaction.
• In general, a Binary Ionic-Compound Decomposition Reaction converts a binary ionic-compound into basic elements.
• In this reaction, NaN
3 reacts to produce Na and N
2
.
• The products of the decomposition reaction are: (d) Sodium and nitrogen-g

• What are the products of the given decomposition reaction?
• By definition, oxidation-reduction reactions occur when electrons are transferred from the atom that is oxidized to the atom that is reduced. We need to look for changes in the oxidation states of the elements in the reaction.
• In the reactants, the oxidation state(s) of the element Na is/are (1). In the product, the oxidation state(s) is/are (0)
• Therefore, the reaction causes a change in oxidation state.
• Therefore, this is an oxidation reduction reaction.
• By definition, a Binary Ionic-Compound Decomposition Reaction occurs when a binary ionic compound is heated.
• Therefore, this reaction is a Binary-Ionic Compound Decomposition reaction.
• In general, a Binary Ionic-Compound Decomposition Reaction converts a binary ionic-compound into basic elements.
• In this reaction, NaN
3 reacts to produce Na and N
2
.
• The products of the decomposition reaction are: (d) Sodium and nitrogen-g

• What are the products of the given decomposition reaction?
• By definition, oxidation-reduction reactions occur when electrons are transferred from the atom that is oxidized to the atom that is reduced. We need to look for changes in the oxidation states of the elements in the reaction.
• In the reactants, the oxidation state(s) of the element Na is/are (1). In the product, the oxidation state(s) is/are (0)
• Therefore, the reaction causes a change in oxidation state.
• Therefore, this is an oxidation reduction reaction.
• By definition, a Binary Ionic-Compound Decomposition Reaction occurs when a binary ionic compound is heated.
• Therefore, this reaction is a Binary-Ionic Compound Decomposition reaction.
• In general, a Binary Ionic-Compound Decomposition Reaction converts a binary ionic-compound into basic elements.
• In this reaction, NaN
3 reacts to produce Na and N
2
.
• The products of the decomposition reaction are: (d) Sodium and nitrogen-g

• What are the products of the given decomposition reaction?
• By definition, oxidation-reduction reactions occur when electrons are transferred from the atom that is oxidized to the atom that is reduced. We need to look for changes in the oxidation states of the elements in the reaction.
• In the reactants, the oxidation state(s) of the element Na is/are (1). In the product, the oxidation state(s) is/are (0)
• Therefore, the reaction causes a change in oxidation state.
• Therefore, this is an oxidation reduction reaction.
• By definition, a Binary Ionic-Compound Decomposition Reaction occurs when a binary ionic compound is heated.
• Therefore, this reaction is a Binary-Ionic Compound Decomposition reaction.
• In general, a Binary Ionic-Compound Decomposition Reaction converts a binary ionic-compound into basic elements.
• In this reaction, NaN
3 reacts to produce Na and N
2
.
• The products of the decomposition reaction are: (d) Sodium and nitrogen-g

• What are the products of the given decomposition reaction?
• By definition, oxidation-reduction reactions occur when electrons are transferred from the atom that is oxidized to the atom that is reduced. We need to look for changes in the oxidation states of the elements in the reaction.
• In the reactants, the oxidation state(s) of the element Na is/are (1). In the product, the oxidation state(s) is/are (0)
• Therefore, the reaction causes a change in oxidation state.
• Therefore, this is an oxidation reduction reaction.
• By definition, a Binary Ionic-Compound Decomposition Reaction occurs when a binary ionic compound is heated.
• Therefore, this reaction is a Binary-Ionic Compound Decomposition reaction.
• In general, a Binary Ionic-Compound Decomposition Reaction converts a binary ionic-compound into basic elements.
• In this reaction, NaN
3 reacts to produce Na and N
2
.
• The products of the decomposition reaction are: (d) Sodium and nitrogen-g

*
• KM: KRL-like frame system with FOL semantics.
• …able to represent :
– classes, instances, prototypes
– defaults, fluents, constraints
– (hypothetical) situations
– actions (pre-, post-, and during- conditions)
• …and reason about :
– inheritance with exceptions
– constraints
– automatic classification (given a partial description of an instance, determine the classes to which it belongs)
– temporal projection (“my car is where I left it”)
– effects of actions
• KM answers questions by interleaving two types of inference:
– Automatic classification
– Backward chaining * Details: AAAI’97

• What are the products of the given decomposition reaction?
• By definition, oxidation-reduction reactions occur when electrons are transferred from the atom that is oxidized to the atom that is reduced. We need to look for changes in the oxidation states of the elements in the reaction.
• In the reactants, the oxidation state(s) of the element Na is/are (1). In the product, the oxidation state(s) is/are (0)
• Therefore, the reaction causes a change in oxidation state.
• Therefore, this is an oxidation reduction reaction.
Oxidation Reduction of a Binary Ionic Compound (Sodium Azide)
 the reaction classified as Binary-Ionic Compound Decomposition when a binary ionic compound is heated.
• Therefore, this reaction is a Binary-Ionic Compound Decomposition reaction.

Backward chaining to compute the products of such reactions binary ionic-compound into basic elements.
• In this reaction, NaN
3 reacts to produce Na and N
2
.
• The products of the decomposition reaction are: (d) Sodium and nitrogen-g

*
Two principal types of chemistry knowledge:
– terms
, e.g. “binary ionic compound”
– laws , e.g. problem-solving method for computing products of reactions of binary-ionic compounds
Terms are encoded as definitions to enable automatic classification.
Laws are encoded as rules to enable backward chaining.
* Details: KR’04 (Barker, et.al.
)

Concentration of Solute Law
Context: a mixture M such that: volume(M) = V liters has-part(M) includes
The conditions under which the law applies
Chemical C such that: quantity(C) = Q moles concentration(C) = Conc molar
Input: V, Q
Output: Conc
Method: Conc ← Q/V
The subset of variables that must be bound
The axioms used to compute values for output variables
The subset of variables that will be bound

Knowledge base built in 4 months:
–
Ontological engineering (4 person-months): designed representations, including structure of terms, laws, reactions, solutions, etc.
–
Knowledge capture (6 person-months): consolidated 70 textbook pages into 35 pages of terms and laws
–
Knowledge encoding (15 person-months): coded in KM
500 types and relations, 150 chemistry laws and 65 terms. Compiled a large test suite which was run daily
– Explanation engineering (3 person-months): augmented the representations of terms and laws with templates

*
• After 4 month development effort, the knowledge systems were sequestered and given a test:
– 165 novel questions: 50 multiple choice; 115 free form response
– Questions translated from English to formal language by each team, then assessed for fidelity by Vulcan and team representatives
•
• * Details:
AI Magazine (Winter 2004) www.projecthalo.com
: systems, Q/A, and analyses

• Our system’s correctness score corresponds to an AP score of 3 – high enough for credit at UCSD, UIUC, and many other universities.
• We’ve predicted scoring 85% after a 3 month follow-on project.

*
We analyzed every point lost. Most deductions were due to errors in domain modeling — mistakes that domain experts would not make.
(More later)
Some errors were caused by technology problems.
Details: KR’04 (Friedland, et.al.
)

• Explanations too verbose: e.g. passages repeated multiple times with only small variations – graders expected a general statement that covered them all.
Requires explanation planning
• Questions that require reasoning about our representations:
– Calculate the pH of a particular substance. Explain why the result is unreasonable.
– Explain the difference between the subscript “3” and the coefficient “3” in 3HNO
3
.
– Explain when and why it’s OK for a particular chemistry method to use an equation that only approximates the true answer.

Hydrofluoric acid is a weak acid, Ka = 6.8 x 10 -4 , and yet it is considered to be a very reactive compound. For example, HF dissolves glass. The major reason it is considered highly reactive is:
(a) It is an acid.
(b) It forms H3O+.
(c) It dissociates.
(d) It readily forms very stable fluoride compounds.
(e) It is a weak electrolyte.
All five statements are true. The question requires that the system reason about which of the multiple true statements is most relevant to the claim.

• Halo I was a rigorous evaluation of current
Knowledge System technology.
• In general, the systems were more capable than Vulcan expected.
• The major hurdles to building a Knowledge
System for science are errors (in domain modeling) and cost ($10K/page).

• Knowledge Systems built by domain experts, not knowledge engineers
• 3 test domains: chemistry, physics, biology
• Questions posed by (a different set of) domain experts, in English
• 2 teams, expanded with HCI experts
It’s not clear that these challenges can be met.

Domain experts build knowledge systems
– with no help from knowledge engineers
– working with familiar concepts and without writing axioms
– with little more effort than writing technical papers scientific codes
Domain experts pose questions
– in English Controlled Language (Boeing)
– without knowing the ontology
*
* Details: AAAI’04

•
Observation : Even domain-specific concepts are defined in terms of common abstractions.
•
Approach : We build a library of reusable, common abstractions (“components”) and a set of relations for composing them together. Then, KB’s are built by instantiating and assembling them.

• Common events, entities, and roles
• Currently about 800 components; projected to be about 3000
• Relatively general and atomic. Include enter , but probably not burgle , and definitely not endocytosis.
• Every component richly axiomatized

• draw from related work
– ontology design/knowledge engineering
– linguistics
• semantic primitives
• case theory, discourse analysis, NP semantics
• draw from English lexical resources
– dictionaries, thesauri, word lists
– WordNet, Roget, LDOCE, COBUILD, etc.

• actions — things that happen, change states
– Enter , Copy , Replace , Transfer , etc.
• states
— relatively temporally stable events
– Be-Closed , Be-Attached-To , Be-Confined , etc.
• entities
— things that are
– Substance , Place , Object , etc.
• roles — things that are, but only in the context of things that happen
– Container , Catalyst , Barrier , Vehicle , etc.

relations between events, entities, roles
– agent , donor , object , recipient , result , etc.
– content , part , material , possession , etc.
– causes , defeats , enables , prevents , etc.
– purpose , plays , etc.
properties between events/entities and values
– rate , frequency , intensity , direction , etc.
– size , color , integrity , shape , etc.

Knowledge about Enter
– before the Enter , the object is outside some container
– instances of
Enter inherit axioms from Move , such as: the action changes the location of the object of the Move
– during the Enter , the object passes through a portal of the container
– if the portal has a covering, it must be open; and unless it is known to be closed, assume that it’s open
– after the Enter , the object is inside that container and contained by it
– the resulting
Be-Contained state persists until the object
Exits the container
– etc .

components are linked to WordNet synsets
– given a word, find similar components
– climb the WordNet hypernym tree with search terms; rank the results by distance and specificity
– assemble : Attach , Come-Together mend : Repair infiltrate : Enter , Traverse , Penetrate , Move-Into gum up : Block , Obstruct busted : Be-Broken , Be-Ruined

Using the Component Library for NL Interpretation:
A Sample Physics Question
“A person throws a ball toward the door of a room. The ball passes through the door and into the room…”

• The Ball and the Person must be together at t
0
• The Ball must be held by the Person at t
0
• The Ball must not be otherwise Restrained
• The path to the Door must not be Blocked
• At t
1
• At t
1 the Ball is at the Door and no longer at the Person the Ball is outside the Room but not shut out of it
• The Door must not be Closed
• The Ball passes through the Doorway
• At t
2
• At t
2 the Ball is inside the Room and no longer outside the Room has the Ball among its contents
• Blocked paths can be unblocked
• Closed Doors can be Opened
• …

Details: KCap’01, IAAI’02

• “Understanding” evaluated by ability to answer questions and solve problems.
• There’s a significant gap between the content of NL utterances and the knowledge required for reasoning.
• Can the Component Library fill the gap?