From: AAAI Technical Report SS-92-02. Compilation copyright © 1992, AAAI (www.aaai.org). All rights reserved.
Automatic Design of Efficient
Visual Problem Representations
Stephen M. Casner
NASAAmes Research Center
Mail Stop 262-4
Moffett Field, CA94035-1000
casner@eos.arc.nasa.gov
Abstract
Automated
reasoning
aboutthe designof effective visual
problem
representations
is possible whenweadoptthe
viewthat visualproblem
representations,
alongwith the
pemeptual
procedures
that humans
useto manipulate
them,canbe described
usinginformation-processing
modelsof the sort introducedby NewellandSimon(1972).
This approach
providesus not only with a means
of
characterizing
visualproblem
representationsin a formal
syntaxbut also witha means
of automaticallymapping
between
"logical"and"perceptual"problem
representations
andprocedures.An automatedsystem
calledBOZ
is described
that beginswitha logical problem
representationandsolutionprocedure,and generatesan
informationally-equivalent
visual problemrepresentation
andprocedure
that allowsthe human
user to obtain the
solution moreefficiently. BOZ’srepresentationmapping
technique
proceeds
by: (1) replacing demanding
logical
inferences
in the solution procedure
with efficient
perceptual
inferences;and(2) structuring information
the visualrepresentation
suchthat searchis minimized.
Theextent to whichthe visual representationsand
proceduresproduced
by BOZagreewith what users
actually see anddo is discussed.
Thelogical proceduredescribesa series of searchand
computation
steps(called logical operators)to be
performed
usingthe set of logical facts. Logicaloperators
occur in two forms. Searchoperatorssuchas Step 1
considerthe logical facts oneat a time, beginningat the
top of thelist eachtime, until a fact is locatedthat
satisfies the relation contained
in the operator.In Step1
the searchis for a fact that satisfies (Origin FLIGHT
’pit). Thisoperatorsucceeds
onthe first fact in the list
andthe variableFLICHT
is instantiated with the value,
flight 117. Computation
operatorsperformarithmetic or
comparison
operationson the valuesretrieved by search
operators.Step11 is a computation
operatorthat
subtracts departurefrom arrival and instantiates
LAYOVER
with the result. Computation
operators
performed
with a logical representationare assumed
to be
mentallyperformed
arithmetic or logical operationsapplied
to a collection of data valuesstoredin the user’s memory.
Visual ProblemRepresentations
A visual problemrepresentationalso consists of a
collection of facts accompanied
by a procedure
for finding
a solution to the problem.Factsin a visual problem
representation,as illustrated in Figure2, havetwounique
features.First, facts in a visual problem
representation
are encoded
graphically by associatingthe elementsof
eachdomainset of informationwith the discriminable
valuesof somegraphical dimension,a techniquefirst
proposedby Mackinlay(1986). For example,eachairline
flight is represented
by a rectangle.Thehorizontal
position of the endsof the rectangleencodethe departure
andarrival timesof that flight. Theshadingof each
rectangleencodes
the availability of that flight. Second,
facts in a visual problemrepresentationgrouptogether
related informationmakingall informationrequiredto draw
a particular inferenceavailableat the same
spatial
location. For example,
all informationabouteachflight,
origin, destination,cost, times, andavailability, is grouped
togetherin a single rectangle.
Logical and Visual Problem Representations
Logicalandvisualrepresentations
of an airline reservation
problemare compared
to illustrate the advantages
offered
by visual problem
representations,
and to showhowboth
logical andvisual problemrepresentationscanbe
characterized
usingthe samesort of informationprocessingmodel.
Logical ProblemRepresentations
A Iogicalproblem
representationincludesa set of logical
facts alongwith a logical procedure
that operateson the
facts to solvea statedproblem.Figure1 showsa logical
representation
for an airline reservationproblemposedas
follows:
Finda pair of connecting
flights that
travel fromPittsburghto MexicoCity.
Youare free to chooseanyintermediate
city as longas thelayoverin that city is
no morethan four hours.Bothflights
that you choosemustbe available. The
combined
cost of the flights cannot
exceed$500.
Perceptualprocedures
differ fromlogical procedures
in
that they are composed
of a set of perceptualoperators
that describesearchand computationsteps performed
specifically within the contextof a visual representation.
Theperceptualoperatorsthat appearin the perceptual
procedure
allow the user to obtain the sameresults as do
the logical operatorsthat appearin the logical procedure
since they compute
the samefunctionsonly in a different
way. Thebottomof Figure 2 showsa perceptual
procedure
for the airline reservationproblem.
Thelogicalfacts(topof Figure1 ) describerelations
between
the elements
of a collection of five domain
sets:
airline flights, departureandarrival times, costs,and
availability. Forexample,
the first fact in thelist indicates
thattheorigin of Flight 117is PIT.
Thevisual problemrepresentationis arguablymore
efficient than the logical problemrepresentation.The
advantages
of the visual problemrepresentationappearto
center aroundthree notions demonstrated
by Larkin and
Simon(1987)and Casner(1990): (1) perceptualoperators
157
°!ii
E
~" li
o~
~ ~!,~^~
.--_
,~.~
==.~
~,li°~ ~ "..~
a
#]
e~
m
Nd
N
P~
,-<
0
9.
~<
158
are sometimes
performed
moreefficiently than logical
operators;
(2) operators
in a perceptualprocedure
can
sometimes
be omitted; and(3) the graphicalstructuring
the informationexpeditessearchfor needed
information.
Severalinstances
of theseadvantages
are apparentin
thevisual airline representation.
First, the visual
representation
allowsthe user to substitutea quick
distance judgement(deterrnineHorzPos)in place of
subtracting
numerically
expressed
departureandarrival
times(computeLayover). Second,Steps 4 and 7 can be
omittedsince the horizontal distancebetween
two
rectangles
canbe determined
without knowingtheir exact
horizontalpositions. This savingscorresponds
to being
able to subtract two numbers
without knowingwhatthe
numbers
are. Thesamereasoning
applies to Steps 13 and
14. Third, the visual representationeliminateseye
movements
whenlookingup time, city, cost, and
availability information
sincethis informationis
represented
in the same
spatial locality (a singleflight
box). Fourth,it allowsusersto limit their searchfor
connecting
flights to only thoseflights that appearto the
right of theoriginatingflight. Fifth, sinceshading
canbe
processed
pre-attentively, users mayimmediately
excludefromtheir searchanyflight squarethat hasno
available seats. Sixth, userscan immediately
rule out
"tall" flights fromtheir searchsincetheseare likely to
violate the cost constraint.
mappingbetweenthe arguments
in the
findFlightWithOrigin and the
searchOb
jectWithLabel operators, we concludethat the
perceptualoperatorcanprovidethe sameinformationas
the logical operator.
It is often the casethat morethanoneperceptualoperator
qualifies as a legal replacement
for a givenlogical
operator. For example,the perceptualoperator
searchObjectWithHeight
(RECTANGLE,
cm) would also
qualify
as a substitute
forthefindFlightWithOrigin
operatorsince wecanderive the onefromthe other by
renaming.Thechoiceof whichparticular perceptual
operatorsto chooseis guidedby a two-tier ranking
scheme
that prioritizes perceptualoperatorsin termsof
their estimatedhuman
performance
costs. Thefirst tier
prioritizes the perceptualoperators
by the difficulty of the
functiontheycompute
(e.g., additionis moredifficult than
searchwhichis moredifficult than a comparison).
The
secondtier ordersthe operatorsby differencesin
performance
costs for operatorsthat computethe same
function(e.g., searching
for anobject of a particularcolor
is easierthansearching
for anobject of a particular size).
When
the perceptual operatorreplacementstep is
finished, the logical procedure
(Figure1) is transformed
into a perceptualprocedure(Figure 2). Perceptualsearch
operatorswhoseonly purposeis to feed valuesto a
computationoperator can generally be skippedand are
markedaccordingly.
Thegoal in designinga goodvisual problem
representation,
then,is to beginwith a logical problem
representation,
andperceptuallyencodeand organize
informationso that anefficient perceptualprocedure
can
be usedto solve the problem.Thefollowing showshow
this can be accomplished.
Designingthe Visual Display
In two steps, BOZderivesthe visual display usingthe
perceptualprocedureit has produced.First, by examining
the perceptual operatorschosento manipulateeach
domainset of information, BOZdetermineshoweach
domainset of informationis to be represented
in the
display. For example, the determineDepartureTime
operatormanipulates
informationaboutflights andtheir
departuretimes. SinceBOZhas decidedthat the best
substitutefor this logical operatoris the
determineHorzPos
perceptual operator, BOZis
constrainedto representdeparturetimesas graphical
objects meaningfullypositionedalongthe horizontal axis
of the display. Similarly, sincethe step of determining
availability has beenreplacedby determiningthe shading
of a graphicalobject, availability mustberepresented
as
shadings.Second,BOZexaminesthe relationships
between
operatorsin termsof the domainsets of
informationthey manipulateto determinehowrelated
informationshouldbe groupedtogether to support
efficient performance
of eachperceptualoperator. For
example,since the domainsets describingtimes, cities,
costs, andavailability are all usedwith the domain
set
flight,gozattemptsto collocate all domain
sets in a
single graphicalobject (i.e., rectangle).Oncethe
graphical objects havebeendesignedBOZtranslates the
original set of logical facts to a set of visual facts and
rendersthemon the computerscreenusing a technique
first proposed
by Mackinlay(1986). Facttranslation
acheivedby replacing the namesand members
of each
domainset of logical informationwith the names
and
members
of the graphical domainthat BOZhas chosento
representit. Thefinal productis the perceptualprocedure
anddisplay shownin Figure2. Theinterestedreaderis
referred to Casner(1991)for a morecompletedescription
of howBOZworksand the generalproblemof reasoning
aboutlogical andvisual representationsautomatically
[Mackinlay,1986].
Design of Efflclent Vlsual Problem
Representatlons
BOZ
is an automated
tool that derivesefficient visual
problemrepresentationsfrom moredemanding
logical
problemrepresentations.BOZrequiresas input a logical
problemrepresentationlike the oneshownin Figure1: a
set of logical facts, anda logical procedure
that usesthe
facts to solvethe problem.BOZworksin two steps,
designinga perceptualprocedure
that allowsthe user to
solvethe sameproblemmoreefficiently, andthen using
this procedure
to decidehowthe informationshouldbe
graphically encoded
andstructured to best supportthe
perceptualprocedure.ThusBOZdelivers two things: a
visual display anda perceptualprocedurethat describes
howthe display canbe usedto solve the problem.
Designingthe Visual Procedure
BOZderives perceptualproceduresfromlogical
procedures
by consideringeachof the logical searchand
computation
operators
in the logical procedureand
attemptingto locate perceptualsearchand computation
operators
that give the user the sameresult. BOZ
accomplishes
this by searchinga catalog of perceptual
operators
organizedaroundthe graphical dimensions
usedto encodeinformationin a visual display. The
perceptualoperatorsusedin the visual airline reservation
procedure
in Figure2 illustrate the idea: searchingfor an
object of a particular shading,estimatingthe horizontal
distancebetween
two graphical objects, comparing
the
size of twoobjects, etc. A perceptualoperatorqualifies
as a legal substitutionfor a logical operatorjust whenthe
two operatorscanbe shownto be renamingsof one
another. For example,since wecan create a one-to-one
159
efficiency advantages
theyyield, it is likely that many
potential designspassedup by BOZfor a particular
problemare in fact moreefficient thanthe oneit produces
usingits crudeone-to-onerepresentationmapping
technique.Whetheror not the possible perceptual
restructurings canbe enumerated
for a given
representationis an openquestion.
Psychological Validity of BOZ’s Visual Problem
Representations and Procedures
Thefollowingillustratestwowaysin whichthe perceptual
procedures
that peoplefollow using a BOZ-designed
representation
candiffer significantly fromthe perceptual
procedure
generatedusing BOZ’sone-to-one
representation
mappingapproach.
Sincethe logical andvisual problemrepresentationsare
expressed
usingthe sameformalnotation, the alternative
representationscanbe useddirectly as simulationsto
makecomparisons
regardingtheir relative efficiency.
Casnerand Larkin (1989) describeshowthese simulations
can be used, in combinationwith subjects’ performance
time data for the airline reservationproblemdeveloped
throughoutthe paper,to empirically determineto what
extent peopleactually follow the hypothesized
perceptual
procedures.
A variety of different perceptualprocedures
canbe
derivedby reordering
the operatorsin the perceptual
procedure.For example,the user, after locating two
connectingflight rectangles,mightcheckthe heightsof
the rectangles(Step15) prior to checkingthe distance
between
the rectangles(Step 11). In general, whenever
the arguments
to a seriesof operatorsdo not dependon
oneanother,the operators
canbe reorderedto arrive at a
variationonthe original procedure.This observationdoes
notposea problem
to BOZsinceit is straightforwardto
enumerate
all possible perceptualproceduresderivable
throughoperatorreorderingby examiningthe data
dependencies
betweenoperators and generatingthat
subsetof the n! permutations
that do not violate the
dependencies.
Moreover,the fact that weare unableto
know
in advancewhichorderingspeoplewill actually use
presents
no problemsince there are in generalno
efficiency gainsto be hadthroughoperatorreordering
alone. Thus,with respectto operatorreordering,the
perceptualprocedureproducedby BOZis guaranteedto
beoptimallyefficient.
Summary
A unifying framework
waspresentedthat allows logical
andvisual representationsand the proceduresthat
manipulatethemto be representedusing the same
notation. An automatedtechniquewasdemonstrated
that
allowsefficient visual problemrepresentations
to be
derivedthroughtransformationof an equivalentlogical
problemrepresentation.Threeformsof efficiency that
visual representationsappearto offer wereexamined.
Twowayswerediscussedin whichthe visual problem
representationsand proceduresthat peopleuse could
differ from thoseproducedusing the automated
technique.
A second,moredramaticwayin whichthe perceptual
procedures
that peopleusecandiffer fromthose
generatedby BOZoccurs whenusers perceptually
restructurethe visual data itself. Visualproblem
representations
offer the user the opportunityto parsethe
dataalongvisual dimensions
other than those usedto
createthe representation.For example,the flight
rectanglesrepresentationwascreatedusing a single
entity, a rectangle,that corresponds
to the logical entity
that appears
in the logical facts: a flight. However,
the
useris not perceptually
limitedto this ontologyat all.
Rather,they are free to carveupthe visual representation
in anymannerthat helpsthemsolve the problem,and are
furthermorefree to invent novelprocedures
of a
fundamentally
different characterthat help them
manipulate
the data in their newlyimaginedform. For
example,whensearchingfor a pair of flights havinga
shortlayoverthe usercanstructure the display around
the set of regionsdefinedby the spacein between
the
flight boxesshownin Figure2. Usingthis perceptual
restructuring, the usercanguidehis or her searchusing
the areabetween
the flights whichformssomewhat
of a
number
6 in the representation
in Figure2. In orderto find
two flights havinga short layoverthe user canconsider
pairs of flights that lie at the endsof thenarrowest
part of
this area. For example,the user mightimmediately
find
the PIT-HOU-MEX
connection since the PIT-HOU
and
HOU-MEX
flight appearat the narrowestpart of the
intermediateregion.
References
Casner,S. M. (1991). A task-analytic approachto the
automateddesign of graphic presentations. ACM
Transactionson Graphics4, April 1991.
Casner,S. M. (1990). Task-AnalyticDesignof Graphic
Presentations.PhDThesis, Intelligent Systems
Program,
Universityof Pittsburgh.
Larkin,J. H. and Simon,H. A. (1987). Whya diagram
(sometimes)worth 10,000words. Cognitive
Science,11, 65-99.
Mackinlay,J. D. (1986). Automatingthe design
graphicalpresentations
of relational information.
ACM
Transactionson Graphics,5 (2), 110-141.
Newell,A., and Simon,H. A. (1972). Human
Problem
Solving. Englewood
Cliffs, NJ: Prentice-Hall.
It is importantto note that perceptualprocedures
derived
throughrestructuringfall well beyondBOZ’scapability for
reasoningaboutthem.Not only is BOZunableto
enumerate
the kinds of proceduresobtainablethrough
restructuring but is also unableto knowanythingabout
howefficient they maybe. WhileI havenot undertaken
a
systematicstudyof perceptualrestructuringsnor of the
160