Integrating Probabilistic Reasoning into a
Symbolic Diagrammatic Reasoner
Joseph L. Bokor and Ronald W. Ferguson
College of Computing, Georgia Institute of Technology
Atlanta, GA 30332 USA
{jlbokor, rwf}@cc.gatech.edu
Abstract
A key part of diagram understanding is the problem of
glyph recognition. Glyph recognition is hard, because a
glyph may be drawn in many different ways and with varying levels of precision. A diagrammatic reasoner must be
able to recognize such glyphs. This paper presents a new
glyph recognition mechanism that combines a probabilistic
representation with an existing symbolic diagrammatic reasoner. This reasoner, GeoRep, recognizes glyphs using a
visual domain theory supported by a logic-based truthmaintenance system (LTMS). Here we extend GeoRep’s
LTMS to include nodes that encapsulate naïve Bayes classifiers. The result is a reasoner that can leverage the benefits of both symbolic truth maintenance and probabilistic
networks.
Introduction
Diagrams are important to a wide variety of tasks that
include problem solving, communication, and collaboration (Glasgow, Narayanan, and Chandresekaran, 1995).
These tasks are frequently aided by a diagram’s ability to
capture and convey many spatial relations at a glance.
In most diagrammatic reasoners, the first step is to combine the initial visual primitives into visual symbols or
glyphs. Glyphs, especially when drawn quickly, however,
are often imprecisely drawn. For example, Figure 1 shows
four glyphs that are all easily interpreted as a NAND-gate,
even though three are drawn imprecisely.
Importantly, the imprecision in each glyph cannot be
corrected by assuming tolerance values. GeoRep can recognize constituent relations in glyphs when they are
within tolerance values, but if even one crucial relation
exceeds the tolerance values (as in Figure 1C and 1D), it
fails to recognize the glyph.
To more robustly recognize glyphs, we need a new
method for recognition. This paper discusses the integration of a new probabilistic mechanism into a diagrammatic reasoner. The mechanism encapsulates naïve Bayes
classifiers in nodes in a logic-based truth maintenance
system (LTMS). The Bayes classifiers capture the imprecision and uncertainty inherent in drawn diagrams. The
encapsulation of Bayes classifiers in the existing LTMS
allows the flexibility of probabilistic reasoning with the
efficiency of a symbolic reasoner.
Implementation
GeoRep is a diagrammatic reasoner that takes as input
vector graphics files and outputs a predicate calculus representation of the diagram (Ferguson and Forbus, 2000;
Ferguson et al., 2003). GeoRep creates this representation
using a two-stage architecture. The first stage creates a
dependency network of low-level visual relations. The
second stage uses the dependency network and a rulebased visual domain theory to produce a description of the
diagram.
We have created a new glyph recognition mechanism by
encapsulating naïve Bayes classifiers into LTMS nodes
created by GeoRep’s rules. The classifiers are created during the rule firing process and stored directly in LTMS
nodes. Communication is enabled between the classifier
and the LTMS in order to maintain the proper truth value
for a glyph interpretation.
Integration into the Rule System
Figure 1: A is a precisely drawn NAND-gate glyph. B, C,
and D are examples of imprecisely drawn glyphs.
We now look at how rules in the visual domain theory
create nodes within the LTMS. For each glyph recognition
rule, its set of preconditions captures the visual element
types and the minimal set of spatial relations needed to
attempt further glyph recognition.
When these preconditions are met and the rule fires, it
adds the glyph interpretation to the TMS dependency net-
work (as in the previous version of GeoRep), but also adds
a probabilistic node as a node that conjunctively supports
the current rule’s interpretation (Figure 2). The node is
created with the implicational structure dictated by the
rule, allowing the node to be retracted if any of the trigger
conditions become false (e.g., if a visual element is deleted
from the diagram).
The probabilistic node then maintains the probability of
the current interpretation. To do this, it must estimate the
supporting probabilities of prior nodes, which represent
supporting visual relations. If the visual relation is already
in GeoRep’s knowledge base, the prior node is considered
evidence. If missing, however, a visual test for that relation is called automatically, and if the test is passed, the
relation is added to both GeoRep’s knowledge base and as
classifier evidence. Once the evidence has been collected,
the probability of the current interpretation is computed.
Communication between Representation Levels
Probabilistic nodes must communicate dynamically with
the LTMS, and do so in two ways (Figure 2).
First, they can change the truth value for the glyph interpretation in the LTMS. The classifier in the probabilistic node generates a probability for the interpretation
based on the existing evidence. When this probability
reaches a threshold value, the labeling of the LTMS node
is changed to True. Similarly, if the probability is below
the threshold, the LTMS node will be labeled False.
The second method of communication is via the evidence collection process. To gather evidence, the naïve
Bayes classifier accesses GeoRep’s representations to determine whether a particular relation is present. If a relation is not present, a visual test is run directly on the diagram and the results of this visual test may be stored directly into the existing diagram representation.
This represents a more sophisticated implementation of
top-down influences than in earlier versions of GeoRep.
Previously, rule triggers could make callbacks to visual
Figure 2: The interactions and communications between
the truth maintenance system and the Bayesian networks.
routines, but the results were only used in the instantiated
rule and discarded afterwards. The visual test results were
never added into the diagram representation. This new
implementation allows the anticipated glyph structure
represented by the probabilistic networks to guide further
visual processing.
Results and Future Work
Using this probabilistic mechanism with the LTMS, we
were able to improve the glyph recognition capabilities of
the GeoRep reasoner. The original reasoner is unable to
correctly interpret three of the four NAND-gates in Figure
1 (B, C, and D), because the imprecision in the drawn
glyphs provided inadequate visual relations to fire visual
domain theory rules. However, the new mechanism is capable of recognizing all of the NAND-gate glyphs in the
figure.
In addition to more robust glyph recognition, the new
mechanism allows further processing of a diagram to occur. Within the visual domain theory, rules are often created that are based not on visual elements, but on glyphs
and visual relations between glyphs. When imprecise
glyphs are not recognized, the rules based on glyphs do
not fire and diagram processing is blocked. The new
mechanism overcomes the imprecision, recognizing
glyphs even when poorly-drawn and allows the diagram
processing to continue using rules that previously would
not have been fired.
Although the mechanism increases the diagram understanding capabilities of GeoRep, there is still work to be
done. We are working on a better language for describing
the Bayes classifiers within rules. We are also investigating integrating the probabilistic calculations directly into
the rule triggers, in order to avoid the overhead of creating
separate nodes to contain the Bayes classifiers.
References
Ferguson, R.W., Bokor, J.L., Mappus, R.L., & Feldman,
A. (2003). Maintaining spatial relations in an incremental diagrammatic reasoner. Conference on Spatial
Information Theory, Ittingen, Switzerland, September
2003.
Ferguson, R. W., and Forbus, K.D. (2000). GeoRep: A
flexible tool for spatial representation of line drawings,
Proceedings of the 18th National Conference on Artificial Intelligence (pp. 510-516). Austin, Texas: AAAI
Press.
Forbus, K. D., and de Kleer, J. (1993). Building Problem
Solvers. Cambridge, MA: The MIT Press.
Glasgow, J., Narayanan, N. H., and Chandresekaran, B.
(1995). Diagrammatic Reasoning: Cognitive and Computational Perspectives. Menlo Park, CA: The AAAI
Press/The MIT Press.
McAllester, D. (1990). Truth maintenance. Proceedings of
the Eighth National Conference on Artificial Intelligence (pp. 1109-1116). Boston, MA: AAAI Press.