Observations and a partial prescription for recognition of fractions and
other non-linear written mathematics
Kevin Lin
Richard Fateman
Background
Our goal for SKEME[] is to develop a multimodal application that incorporates speech
and handwriting for the input of mathematical symbols. To simplify the implementation,
we are using packaged programs for the lower level recognition, namely Microsoft
Speech SDK and Microsoft Tablet PC SDK. Here we consider only the handwriting
component, and consider aspects of how to use handwriting as a sole-source for
describing mathematics.
Microsoft Tablet PC SDK is reasonably accurate for input of regular horizontal
characters when they are written linearly. In some modes it also recognizes, by layout
A
analysis, expressions written on several lines. This means that
becomes an expression
B
written on three lines, A-B . (As an aside, the restriction to one line does not solve all
problems.
is most likely to be recognized as “at b” because the default recognizer
prefers words. We hope we can teach the system to prefer + over t.) A possible approach
to solve this problem is to collect the ink strokes such as
and then, based on the
bounding boxes, separate in a functional manner, the ink strokes that constitute parts of a
fraction. After separating the ink strokes into horizontal components, we can pass the
results (perhaps more than once) into the handwriting recognizer, which need only deal
with mathematical expressions of symbols that contain elements oriented horizontally.
We also note that the handwriting recognizer is, in its usual applications, a substitute for
keyboard input. There is no obvious notation on the keyboard to indicate vertical
concatenation of characters, and so we have left the realm of “handwriting to text” unless
we allow for results like \frac{A}{B}, a text version in “TeX” of a built-up fraction.
What we have so far…
So far, K. Lin has a demonstration program to show this approach is feasible. Given a
collection of strokes (a data structure in the context of the Microsoft system), we can look
for horizontal bars. If one is detected, the algorithm can search the remaining elements
for objects whose bounding boxes fall within the left and right boundaries of the
horizontal line (alleged to be a divide bar). If the bounding boxes do not fall (mostly)
within the left and right boundaries, then they are not candidates for numerator or
denominator. Furthermore, the objects need to be in close proximity with the fraction bar
(See diagram below)
Not Part of a Fraction
(too far from horizontal bar)
Not Part of a Fraction
(too far left)
Part of a Fraction
Probably Part of a Fraction
(borderline case)
Horizontal Line
Not Part of a Fraction
(too far from horizontal bar)
Tolerance
It is very unlikely that you will draw a perfectly straight horizontal line. Therefore we
introduce a tolerance level to give leeway for the user’s imperfections in handwriting
recognition. For experimentation we have set the default tolerance level to be 30
(display) pixels, but in a more general approach we should have a percentage calculation,
say that any box with height to width ratio exceeding some tolerance (say 0.1) is likely to
be a horizontal bar. (Note that stylus/pad resolution is much higher than display
resolution.)
Numerator.Right
Numerator.Top
Numerator.Left
Numerator.Bottom
HLine.Top
HLine.Height
Denominator.Top
Denominator.Left
HLine.Bottom
Denominator.Right
Denominator.Bottom
For our prototype we overloaded the same parameter of tolerance level, default 30
pixels, to define how close the objects need to be to the bottom or the top of the
horizontal line. A tolerance of 30 indicates that the numerator can extend no more than
30 pixels lower than the top of the horizontal bar, and extend upward no more than 60
pixels higher than the top of the horizontal bar. Similarly, a tolerance of 30 pixels means
that the denominator of the fraction can rise no further than 30 pixels above the bottom of
the horizontal bar, and a denominator of the fraction needs to be within 60 pixels of the
bottom of the horizontal bar. Example:
t = tolerance (example t = 30)
Horizontal Line
Maxheight = t
Numerator
Denominator
HLine.Top >
(Numerator.Bottom - t)
AND HLine.Top <
Numerator.Bottom + 2*t
HLine.Bottom <
(Denominator.Top + t)
AND HLine.Top >
Denominator.Top - 2*t
The horizontal line can have
a maximum height of 30.
If HLine.Top = 300, find
numerator between 240 and
330
If the HLine.Bottom = 300,
find denominator between
270 and 360.
Where we go from here…
This characterization is far too simplistic. First, the horizontal bar may not be a fraction at
all, but one of the (many) other uses of horizontal lines. There are other reasons for
horizontal strokes. “+” has a horizontal stroke. So also do E, T, =, ~, ≤,≠,,±, etc.
The handwriting recognizer should probably be used first, and if it produces an answer
with high confidence, that should be our result. In the case of low confidence, we can
embark on a segmentation procedure.
The numerator and denominator have horizontal extent as well: we hypothesize that a
reasonably careful writer will try to make the bar wider than the numerator or
denominator, but this may not actually be possible if the bar is draw, as is commonly
done, before the denominator (which may be large). The parts of the fraction may also be
in part further away, as would be the case of exponents in the numerator or denominator.
A more thorough examination of the possibilities in the context of OCR (static pages) has
been given by Fateman [].
In the case of handwriting we can use other information not available to the OCR
program. In particular, if we assume that the normal way of writing a fraction is to write,
in sequence, the numerator, divide bar, denominator, we can use temporal information for
grouping. That is, consider a sequence of symbols a1, a2, a3, divide bar, a4, a5. Assume
also that a1 and a5 are within the expected boxes for the numerator and denominator
respectively. Then we would likely place a2 and a3 also in the numerator, and a4 in the
denominator, because of their temporal order. We might be looking at
a1a2  a3
a5
a
4
Another advantage for handwriting recognition (vs. OCR) is that recognition of densely
typeset formulas requires, in general, separating lines of multi-line formulas. It is
implausible for a handwriting input program to have such multi-line inputs. It is also
possible (and advisable) to make some efforts to assure the writer that the computer has
understood the symbols as they are written: feedback on recognition, followed by
correction, is not part of the usual static OCR process (although this is supported by a
subsequent editing pass in some programs).
An all-encompassing procedure to accurately identify fractions and other markers that are
horizontal lines must have a substantial heuristic component.
Interlocking heuristics for detection of superscript and subscript for OCR have also been
previously described by Fateman []. The additional temporal factors are once again
available for handwriting: we expect handwriting of the base expression followed by
subscript and then superscript. If a bounding (base) box has a smaller bounding box to
the upper right, written after the base box, this can be tentatively identified as a
superscript. Unfortunately, bounding boxes are inadequate unless base-line calculations
are included (this information is available from the handwriting recognizer!) The
example we cite is to look at the bounding boxes for dp versus pd: which has the
subscript? and pd versus dp : which has the superscript? Knowledge of font, baseline
and upper/lower case is significant. On the positive side it may be possible to determine
the termination of a superscript (temporally) by a return to the baseline.
Conclusion
Conventional unconstrained handwriting recognition, even if given hints about
mathematical symbols and typical linear constructions, won’t do 2-D math, but can be
used as a component for recognition within bounding boxes derived from 2-D
arrangements.
PSEUDOCODE / Fraction detection
For all bounding boxes of strokes
Collect P = all plausible horizontal lines
Collect from P, a subset Q = all plausible divide bars
For each item I in Q, examine expressions (possibly multiple strokes)
in bounding boxes mostly above I for possible recognition as
individual characters or symbols in numerators; similarly for denominators.
Order by stroke time; collect plausible numerators and denominators.
References
R. Fateman, Taku Tokuyasu, Benjamin P. Berman, and Nicholas Mitchell:
``Optical Character Recognition and Parsing of Typeset Mathematics.”
Journal of Visual Communication and Image Representation vol 7 no. 1
(March 1996), 2-15.