Intertwining Query Construction and Relevance Evaluation
Gerhard Fischer, Scott Henninger, and David Redmiles
Department of Computer Science and Institute of Cognitive Science
University of Colorado
Boulder, Colorado 80309
[9, 13, 24, 26] (see Figure 1). This mismatch exacerbates
the vocabulary problem in which it has proven difficult to
get an agreed set of terms to describe computer artifacts
ABSTRACT
Traditional information access systems generally assume
that a well-articulated query exists, and that once an object
i found, it can be readily understood. Although this asumption works for retrieving text objects, in more compi x domains, such as retrieving software objects for reuse,
queries must be incrementally constructed and support is
n ded for comprehending what is retrieved. Therefore,
information access methods need support for query con..tnIction and relevance evaluation as an integral part of the
I tion process.
Two prototype systems are described for supporting this
n d: CODEFINDER for query construction and EXPLAINER
r explanations of program examples. These systems interact to support the processes of locating and comprehending software objects for reuse.
[17].
In the domain of text objects, the relevancy of information
found can easily be judged by users. When retrieving more
complex objects, such as software, comprehending these
objects becomes a significant problem (see Figure 2). Text
objects use a familiar form of language that allows the gap
between the situation and system model to be rather small.
This is not the case with software objects, in which users
may have problems understanding the language, abstractions, and interdependencies upon which the software object is built. This causes the gap between situation and
system models to be large enough to require support for
judging whether the item meets the information need.
Cooperative problem-solving systems are needed to change
the role distribution between humans and computers where
currently users construct queries and a system applies
queries to its information space (18]. The traditional distribution gives no support to two critical problems: query
construction and relevance evaluation. Systems should effectively execute well-specified queries, but they must also
support users in articulating what they want and judging the
adequacy of objects found.
KEYWORDS: Information access, software reuse, progr mming methodologies, cooperative problem solving,
r trieval, retrieval by reformulation, explanation, situation
m del versus system model.
Requirements for Information Access
Systems that Support Software Reuse
Information Access in Complex Domains
Traditional information retrieval research assumes that a
well-articulated query can be easily thought out and concentrates primarily on retrieval efficiency [3]. Although
this assumption works in well-known domains, it does not
scale to ill-defined problem domains, in which users need
to elaborate and experiment with a problem before the information need is fully identified. Defining the problem is
a large part of the problem, and support is needed for an
incremental process of exploring the information space
While refining the query.
Software reuse is often touted as the solution to the
software engineering crisis [30, 10,4]. This long-term goal
of software reuse is thwarted by an inherent design conflict:
to be useful, a reuse system must provide many building
blocks, but when many building blocks are available, finding and choosing an appropriate one becomes a difficult
problem. The large number and diversity of objects increases the chances that an object closely related to a
problem task exists. However, finding such an object
among so many is difficult.
Even in well-known information domains, the problem of
query specification can be formidable. Users may know
what they are looking for, but lack the knowledge needed
to articulate the problem in terms and abstractions that the
computer can understand. The underlying problem can be
characterized as a mismatch between the terms the system
needs, the system model, and the situatioll model of the user
Permission to copy without fee all or part of this material is
granted provided that the copies are not made or distributed for
dIrect commercial advantage, the ACM copyright notice and the
of the publication and its date appear, and notice is given
that copying is by permission of the Association for Computing
chinery. To copy otherwise, or to republish, requires e fee
nel/o, specific permission.
1991 ACM 0-89791-383-3/91/0004/0055 ... $1.50
The Information Access Cycle. The process of cooperative problem formation in software reuse intimately intertwines the processes of location and comprehension (see
Figure 3). Once an item is found, users may not be able to
assess its relevancc. The software object must be studied to
determine not only what it does and how it is used. but how
difficult it will be to modify to the current needs. Once an
understanding is achieved, the user is in a better position to
understand what his necds are. and can make another attempt at locating an ohject.
55
Situation Models
System Models
(GRAPHICS:DRAW-CIRCLE x-center y-eenter radius
:INNER-RADIUS i-radius)
- ring
- doughnut
- tire
- CALL BLCIR (xentr,yentr,radius)
CALL SHADE (xerds,ycrds,npts,angle,gaps,ngaps,O,O)
- wheel
- washer
An informal study we conducted revealed that people conceptualize the problem of drawing the object shown with one of the situation
models indicated. Indexing for situation models corresponds to application goals. If drawing a car, drawing a tire would be an
example of an application goal. The two system models ("inner-radius:" option to "draw-circle" for the SYMBOUCS LISP Machine,
and blanking out a circular region before shading a circular curve for DISSPIA) show how system models are indexed by
implementation units.
System Model
Situation Model
.0
a9 • e
The system model can transcend individual functions. In another study, we observed how people adapted a complete program
example from the system to implement a new task in their situation model. Illustrated here is how one subject adapted a diagram of
modularity in a cyclic group to drawing a clock face.
Figure 1: Situation Models and System Models for a Graphics Object
Software Object
Text Object
(defun torus (x y r i-r)
(graphics:with-room-for-graphics
(*standard-output* 100)
(graphics:draw-circle x y r
:inner-radius i-r»)
T.A Standish, An Essay 011 Software
Reuse, IEEE Transactions on Software
Engineering, Vol. SE-lO, No.5,
September 1984, pp. 494-497.
Figure 2: Differences in understanding text and software objects
The given literature reference (Text Object) can be understood directly without any specialized knowledge, but users unfamiliar with
the specifics of the given LISP code (Software Object) will need help to understand whether it meets their needs.
Previous Information Access Studies. Retrieval by reformulation is a method of constructing and refining retrieval
cues. It is based on empirical evidence that ideas are nol
defined precisely to begin with, but are continuously
elaborated upon and evaluated until appropriate cues arc
constructed, and that people think about categories of
things in terms of prototypical examples as opposed to formal or abstract attributes [27,21]. This paradigm has been
applied to create a cooperative relationship between users
and systems giving users the ability to incrementally im-
-
prove a query by critiquing the results of previous queries.
[31] and HELGON [15] demonstrated how the
retrieval by reformulation paradigm could be put into practice.
RABBIT
In another study, we have observed problem-solving behavior of people in a large hardware store (more than
350,000 items) known for its knowledgeable salespeople
[29]. We observed that customers did not come in with
well-articulated queries, but used a number of specification
techniques to eventually arrive at a solution. In many
56
cases, cust o m e ~s. brough,t ,an example close to what ~va s
eedr:d. In additIOn 10 glvm g customers a means to artlC' u-
~a[e
their qu ery, these examples facilitated th e problem·
.
solving process beca use th ey gave the sa lesperson a concre te idea o f what wa s be ing sought and was cru cial 10 th e
of 5uccess; ~n ach i e~' ing. the . gl1at. . SJ lcs peopJc,
1
degrel!
,I
'
eXpfanatioll
Ido'mul~t io
familiar \\'ith potential u.'.e Si tuati on s 10 then de part ment s,
were able to comm unicate to the custo mer ho w ilems could
be used.
Information Access for
Software Objects
Coo eFINOER:
~ xr'~na tion
C'·
CODE FI NDER is an ex ten ~ i on of H ELGO~ tha i r...:t rievcs
software objects (see Figure 4). Empirical studies of
HElGON showed that providing na tural mea ns of qu ery formation, which takes advantage of the way hu ma n memory
works. will lead to he Iter retrieval systems [1 6J.
c l lrii.C liQ r.'modi fic.llion
Figure J : An Informati on Access Model for
Soft\\'are Re use
CoDEFINDER uses an associat ive form o f spreading ~I cti\'a ~
tion [25. 2, 7J, which is hased on a psycho logical model of
human memory [I , 20J to en hance lIELGON. The ge ne ra l
idea hchind spreadi ng ;}ctivation is lu rep rescJl I ite ms and
kevworJs as network nodes, wit h links between [he nodes
representing mu tu al associati on, Cons tructi on of a que ry ,
Soft ware reuse in\'o lv es three cognitive processes: location,
comprehensi nn , and mod ifiCa ti on. The inte rp lay between
localion and co mpre nclI sil )n is discus.<;ed in this pape r. SliPport for end-user modifi :J hilil y in ~'10DIFtER is dea lt w ilh in
Fischer and Girge nsoh n l 1990J ,
whic h consists o f network nodes, causes Jil ac ti vat ion value
to he assigned to th ose nodes. Acti v~li on is the n a llllwe u to
qu erics). Alt hough the experiment w~s performed on a
ve ry sma ll informario n space. subjects experi enced
problems using IiELGON [1 6J.
Subjects
found
CODEFtNDER' s que ry con struct ion methods to be qu ite
nat ural and co nve rged on solutio ns muc h quicker, both in
time and nu mber of itera ti ons, compared with using
HELGON . Some subj ects fail ed to find a solution wi th
HELGON, hu t all succeeded with CODEFtNDER.
Network Stru cture. }\ s shown in Fi gure 5, the re are two
st ruc tures that compose tile network CODEFINDER uses for
ret ricval. The first is J cJtcgory hierarchy, whic h decomposes the inform ation space into a nu mber of ahstract
ca tegories. This st ructure: is constructed by a domain expert fam ili ar with the info rm atio n space an d able to const ruct a useful hierarch y. T he associative net\\'ork is constructed automat ica ll y hy creati ng links betwee n a soCt wa re
object and its key words and param eters a[tributes (see
Figure 4). Key words cnn he assigned by the designe r of th e
so ftware ohject and can be added, deleted, or otherwise
modified directly thro ugh the CODEFi NDER in terface.
iteratively spread along the li nks to <Jssociated nodes. Each
node upda tes its acti vation value by summ ing all of its
inputs by a mathematical formula. Nodes with hi gh activalion values are co nsidered to be re levant to the qu ery.
An impo rt ant characteristic of spreading activat ion is a
tlcxi hle inferenc ing mechan ism that can reCison with incomple te or im precise informat ion [25, 8 J. Ohjects do not
match the que ry in an all-or-none fashion, but in va ryi ng
degrees, Require ments placed by th e que ry l:I r~ iJ1l e rp r et~ d
as sof l l'ons/ruin/s , In cont rast to mntching algorith ms, if a
query does not include key\vords that index a part icul ar
ohjec!, thaI l\bjec l has a chalice of be ing retr ieved if nearby
associat ions exist (see Figure 5 and exam ple he low).
In Figu re 1, we prese nted an examp le of how software
ohjec ts coul d be accessed by application goals ill addi ti on
to implementation units. Figure 5 shows how this can be
applied in the CODEFtt<DER archi tecture. The keywo rd ring
is nClt co nnec ted to draw-ring, wh ich draws the ohject
shown in Figure 1 Oil the Symholics system, If ring is
give n in the query, it wi ll ac ti vate th e draw-circle node,
which in turn acti va tes keywo rd nodes lire and doughot/ l.
These ke ywords will th en wo rk togeth er to aetiv3tt:: the
dr ah'-rillg node. Since rClric\,il l i'i pc r form ~ d by so ft con·
straints, it co mpensates for inconsistent indexing and
omitted keywo rds because key \1,,'ords are dy nami C'n ll y relateu through the items thcy index .
EXPLAINER: Examples and Explanation to
Support Information Acce ss and Modification
Figure 2 il lust rates th at judg ing the rel evan ce of software
objects was ma rc complicated tha n retrieval of simpl e objccls such as literat ure refe rences. Once ret ri eved, softwa re
objects need to he adapted to the task at hand. Both the
j udgment of relevancy and th e adaptation of software o b~
jeets req uire th e user to unde rstand the retrieved object. The
EXPLAtNER systcm su pports thi s need by allowing users to
explore th e design, impl!.!ll1cnlati on, and graphics oUlput of
a software obj ect. f igure 6 shows a pro toty pe of the
EXPLA1:\ER too! be ing used to ex plain the rin g cX3 m p !c.
An informal study was performe d to compa re the query
co nstructio n methods of CODEFIt< DER and HELGON. Suhjects were given a retrieval task and asked to use either
HELGON or CODEFtt<DER to fin d 3 sofllvare ohject (althoug h CODEFI NDER takes advant age IIELGON 's query
specification meth ods, suhjcC'ts were instructed (0 en ter
only keywords and spread act ivalion for CODEFtSDER
57
alea~v
,ttookmar~a
Hierarchv
'..{ o£c
~
.~
tema
dr.",-~II iptical-rinQ
dr,....,-clrcl..-drlvu
-..£1)----<_ I )
"TR'fIIO..U
0
dr.w-rin~
ftI1ltlf)
_'.-lutwurt-_'••-P
:/My-eyl
~
-
Stl'. t ~
LI,.....
~
~
'( ,...." FtftCT '''"
IQuery
THING
GRAPHICS
PARAMETERS , not .lice-fll
Keywords:
t,,..
dQU9hnut
rins
Reillted Item.:
draw-cirele
•
Matc ,ng terns
19 Re ated Keywords
cir·cI,
"a~eo
draw-.lliptical-rin9
draw-rins
draw-half-.lIip ••
draw-partial-ellips.
draw-cireular-are-tc-co
draw- .llip ••
dra...,-quarter-cird.
draw-half-cirele
draw-ci reu I ar-arc- to
draw-cireul ar-are- throus
draw-conic- gection
ccntc,.-y
cenU,.-x
radiu.
y-.-..Iiu.
,,-radiu.
.-.::
"tart-ansi.
to-y
to-x
orbit
.lIip.,
: dock .... i..
: end-angle
path
circular
i
I
tangent
!!!e -"",a t c::ntI1Jl tema;
THING
GRAPHICS
CIRClES
DESCJUPTlON:
Oraw. & nll.t-ln etrcle And baa.nQ out the 1Ilidd.1e,
aMUnl & doUlhnut-shaped object.. The param.etlt
::t..nMr -rw.tw: determines the d2:e at the blJ;1lked. out
inner circle. and must be less than the vaJ\MI ,Iven in
the ,adius pal .uD.etef .
P.t.aAMETrU
P.t.aAGE
RE-4I..50
I
D
Say, Oatabao.
Simpl. Query
Retrieyc
Shew Item
Load Oat-abo..
Require Category Prohibit. Category Require Attribute Prohibit. Attribute
(o;oose n;;. )
draw-rinK
cent.er-x cc:nt.e.r-y radiu.
: inner-r"'ad ius
graphics
dra...,-c ire I. draw·ellipse
HIt
I.ETtJtNS
ACTlVAT10N-VALUE 9.6872849
I.&YWOI.DS
toru. circle tlrtf
... h.!el
doughnut eirel&" outline
CodeF1rw:t.r CON'\and: Add Overy It.". -drw-circT.·
CocMFtnde,. cQIYI.,..d t Retrieve
Codef i nda,. c.",.,...,d: ShOlol Itet\ ·drohl-"'in~·
CodeFinder e_and, •
Figure 4:
CODEFINDER User Interface
The CoOEFINDER user interface is based on IIELGON [15). The Category Hierarchy window displays a graphical hierarchy of the
information space loaded. In this instance, the information space is a sct of graphics functions for the SYMBOLICS LISP Machine. The
Query pane shows the current query. The top part of the query specifies two categories (thing and graphics) and a parameters
attribute. The hottom part specifies keywords and related items. The query parts combine to retrieve the items in the Matching Items
pane. The Rxample oC the Matching Items pane shows the full entry for an item in Ihe information space. The Choose This button
loads the example item into ExPL\JNER for a detailed explanation. The Bookmarks pane holds a history of the objects that have
appeared in the Example oC the Matching Items pane. The Matching Items pane shows all items matching Ihe current query, by
order of relevance to the query. The Related Keywords pane shows keywords retrieved by the query. Any of these keywords can be
added to the query through mouse action. The remaining panes allow users to specify commands by mouse action or keyboarding
(with command completion).
Many studies support the general assumption that examples
are helpful in programming [23, 28] and problem solving
[6, 29]. A preliminary study we carried out observed
specific ways programmers would use examples within our
design framework (see Figure 3). Through a description of
this study, we discuss below the specific issues of judging
relevance, bridging the situation/system model gap, and
supporting problem reformulation.
A Preliminary Study. We observed subjects solving
simple graphic programming tasks in order to determine
what types of questions they would ask about an example
and in general what kinds of information they would seek.
Subjects were given a programming task to complete and
an example from which to slart. They were told that the
example was related 10 a potential solution of the programming task, i.e., as if a query and location had already taken
place. They worked in an EMACS editor buffer and directed
questions and comments to a human; the EXPLAINER
prototype had not yet been implemented.
One task was to write a program to draw a clock face (see
Figure 1). The subjects were given the example now
represented in the EXPLAINER system and shown in Figures
1 and 7. Subjects were observed through questions they
asked, spontaneous talking aloud, and mouse cursor movement as they studied different parts of an example's code.
JudgIng Relevance. When judging the relevance of given
58
i
\
i
\:!
:
i
! "\
I
!i
i
I
i
,~
grilp hlcs
tr i <lng l ps
Figure 5: Indexing by Application Goals
The indexing architecture of CODEFlf"DER makes usc of both a hierarchical arrangeme nt of categories and an associative index of
keywords. In this figure. ovals represe nt categories, the smaller circles represent keywords, and larger circles arc code objects
(k~ywords and code objects together compose the associative network). The function drrm:-circle i~ divided int? two objects, one
represents the function as a whole, and the other represents an option to draW-Circle (draw-mIg), which draws a f1ng. A connCClion
between a keyword and code object means that there is an excitatory association between the keyword and code object. An arrow
from a code object to a categor y means the object is contained within the category.
Discussion
examples, subjects verified that the ex am ple contained certain simple features, such as circle drawing, by inspecting
the sample graphics output (sec Figure 7). Other features,
such as the need to draw numerals, were not so obvious.
Subjects studied and/or asked questions about the part of
the example code that draws the labels_
To build information access systems that arc both useful
and usable [tl J, systems must be more than a passive
repos itory of information. Some of the burden of query
construction and relevance assessment must be shifted to
the syste m by supplying knowledge in the world [26] that
supports the cognitive tasks of location and comprehension.
Know ledge in the world can complement knowledge in the
head. allowing designers 10 concentrate on the creative
aspects of a design [12J. leaving the morc mundane aspects
of the task to the system [18J-
Bridging the Situation/System l\lodel Gap. One consideration in using the clock task was that all the subjects
were familiar with the problem from th eir daily experience_
What they did not know was how to map the task (their
situation model) onto a program in the SYMBOLICS
GRAPHICS system (the system model) . The ex amp le code
obviated some aspects of the system model : e.g., the name
of the specific functions on the SYMBOLICS for drawing
circles ,wd string labels. Expl anations clarified the meaning
of paramders and the purposes of parts of the code, such as
the part that positions the lahel s along the circumference .
CODEFlNDER addresses the problem of location by acknow ledging that hrowsing techniques do not scale up to
hundreds and thousands of software objects and that traditional query se arch methods hreak down. They break down
for two reasons: (I) users arc unahle to articulate complete
queries and (2) in most sys tem s, so ftware objects are indexed under impkment:.lIion units and not application
goals, leading to a mi sma tch between the situation and systelll model. CODEFINDER is a starting point towa rd solving
the se probl ems hy combining retrieval by reformul ation
with associative techniques of retrieval.
Supporting Problem Reformulation. The interdependency of location (COD EFINDER) and comprehension
(E.XPLAINER) in our conceptual framework (see Figure 3)
Occurs when users reformul ate their notion of the ta"k and
its solut io n_ For example, one subject decided that the
numeral s on the clock dial shou ld he calculated as integers
and then converted to strings for plotting. In the provided
ex ample , lahels were computed in a different way. The
sUhject as ked for an additional exampl e that showed the
type of conversion he wanted. This is a rclati\Tly simple
reformulation, and applies to only a part of the problem
SO lution. Reformulations that replac e the initial example
entirel y are expected wh en hoth the CODEFINOER and
EXPLAINER systems ope rate in nll1re rl'alistiL' sl, ttin g".
Retrieval for complex ohjects such as software demands
tools to aid in comprehension as well as location. Users
need to IIl1dersla lld the relrie\'ed object. They need to know
wh en to use a component and II'h([/ resulls they can expect.
The EXPLAINER system emphaSi zes user-directed expl anati on of example programs to support users' comprehension
of sofl\\'are ohjec ts. Exploring a rela ted example can lead
ll sers to ciJ:lnge their de sign idea, causing them to return to
the "K;t l i(ln pr(\((' ~ s.
sa
M~
:
LIe.,. 1111 lO4C lIe.,. All [xpl ..... tion. (In,. All Hil)hlil)ht.
Explainer
>«>/4". tiOM
2!11!
(dlItun ring (x y ,. I .... )
Thh progr ... dr . . . . . ring.
(gra(>hlcs ,,,I th-,.OOft-for-gra(>hlc.
( •• tandM-d-output. 158)
~
(grat)hies,dr..,-circl. x )II ,..
tI.",.,.-rodlu. l-r»)
--
Spoce on tho acreen 1. retlOnitd 'or doing the plot. R filled
circl. t. crewn with ita center It K,y 6nd ",it" radiua r . The
cent.,.. i . blenked out by • circl. o( ,..adius i-t"" o
f- -The function draw-ci,.cl. fa call.d ",ith v9V"enta
I inntf"'-r.cf1ua 1-r.
.1<,
y, ,.. and
xpl"""t on Phra ••
explain man:
hiQhlfght codt
x
how
other example,
why
"._1. OUt""t
vpeln
Explain.,. COMand, •
0
Figure 6: Explaining the Ring Example
Users enter the EXPWNER system after searching for possible examples in COOEFINDER. For each example, EXPWNER begins by
displaying the code and sample graph on the left half of the screen. An initial description of a few lines of text is displayed on the
right. A pane at the bottom allows menu commands to be typed in manually as desired.
The users ask questions. such as "how" or "why", by clicking on a piece of text, code, or graphics. A menu appears. and the users
select the question for the selected item. The pop-up menu is shown here in the center of the screen. The constructed question is
reflected in the Explanations Pane in italic type. EXPWNER'S response follows. The text shown here reflects a history of a
question-answer dialog in progress. The "other examples" menu item allows users to see other possibilities found in CooEFINDER.
Limitations and Future Work. The conceptual
framework in which we developed CoOEFINOER and
ExplAINER was based on small knowledge bases to test the
applicability and functioning of the basic mechanisms.
Problems of scaling information access methods to larger
information spaces are well documented [5, 19], and we
will continue to evaluate our systems on larger information
spaces.
cost-benefit ratio of structure. The value and the cost of a
well-structured knowledge base must be compared to the
benefit that can be obtained from requiring little a priori
structuring. The effort of constructing rich knowledge
representations in CoOEFINOER and EXPLAINER is offset by
what can be gained with better tools in the subdomain of
graphics programming.
In the experiment that led to the development of the
scenario, users were allowed to ask questions about both
domain and system knowledge. To direct the expansion of
the example base, we will test the system against userdefined tasks. Our scenario is based on experiments in
which users were assigned graphic programming tasks.
Looking at users' own problems not only will provide unbiased guidelines for future development [22J, but also will
demonstrate how well the gap between situation and system
models has been bridged.
Conclusions
Another issue that needs
to
3
We have shown that query specification and relevance .
evaluation must be intertwined in information access systems for software objects. Two operational prototype systems, CooEfINDER and EXPLAINER, are being used to explore and clarify issues involved in this integration. Future
evaluations of the systems will continue to improve our
understanding of the relationship between query specification and relevance evaluation.
be empirically explored is the
60
4
5.
Explainer
. '~ ( :. . '"
~. ~~
eye I
jo; - ~rovp
nil
{(2 t:d i t- 2. 0 ~d)
( 01'2 ( , pI 2,B »)
(ra ditJ ~ 3fL (3))
( 9r' ~pr-l c , : .... \ th- r OOr'l - r or - g..-~tJh ' c ~ ( , 't"n,j~,.. d-o ..... t~>.-'t-. ;en )
(9 1" ~ O~ i c ": 1...11 t!"l - 9r~.::hi
tr-on !! l !l~ ~ .:: n (; ~~ ,, ~, ~~)~ - out':' t..' t '
r h '~ p l ot is a \..ri ~ U !l H Iat 1o n o f
g r o uo G o f or der n : G -:; C"g ,,1
""u'~ ';;:,\1c .,t ic n
i n" c yclI C
a{ r.- l)) ", rler e 3~ ::: e, the
"" lJ ltipl i c e t; ve: ide n t.ity .
c, -
(9 rsphic':u1th - 9r4ph ~ :' - S G 5 i e
( l ' t !H.. :j ~ rd - =vt ;:I\... t '
(g r " c. hic5: d r av -c1r c le B 13 ra diu s
( do r {(the t ~ - in cr (/ 2P 1 :0, 0»
2 .~)
:, f h e \/dri~ b l~5 2pi,
:f 'i ll ec r,tl)
Df/2 ,
~n d
r"div~
~re
",!Sl gn ed v",h) e~
C, .• J.
x
, £>(p/~in
y
(th et.-l i st
(l1~t B.g
M a ,.. k er~
theL~ -
Nor. --
ar e
dr~LJn
al on g the cirCu r"lfe r en c e cf a cir cle .
i n e r-
( .. 2.9 th et,,-i nc,..)
( , 3.G
t het..~-' n c r )
Th e v ,,(' ; able theta -i n c ,. i , ~et t o ( .... 201 HLQ). Th e v arta b le
thet ,,-11~t 1s ~et to five p o ~ition~ al o ng the cfr cufere nc e of
t.he circle . The var iab le lebel - li~t 15 ~et t o " li~t o f ~et
v er i able::s ( ... 1.
(- 2p1 t; h et.!I -incr»
( cdr th eta-l f~t»)
(l~ b el - l i~t
'('.~
= e' ',1' ',2' ' . 3 ' ' , (n - I) ')
( c d r l d bel- l i.5t»
{ th eta (c~r t "', e t ~ -l b't ) (C"r" th eto-lfs t; » )
(label
(car
l d bel-lios t )
(c.,r
label-1i5t »)
((null t h~ta-ll!! t ) nil)
( ,eta)oj (f. ,..edhi " (co , ( - t;heo;,o p f/ Z )))
1'1
Id
~ !he variable theta - inc r' dete t" f"llne:!!l t he p O~lti on for t h e
~ n",.. k ers.
~~~~~
'~~
""'~
"~
"~
~
"' ~
"~~
H~
h '~
' ~~~
.l( 9me1e ~'~put
IY
.0:
l/oe1n ,ommancs
iE .... " . a I ner COf"lr'l end : •
30 .:; e
Figure 7:
How EXPLAINER Presents the Example Used in thc Study
Acknowledgments
We would like to thank the other members of the ARI project,
especially Evelyn Ferstl , Petcr Foltz, Walt er Kint sch, and Curt
Stevens with whom we shared m3ny discussions about the ideas
and the systems discussed in this paper. The resea rch was supported by Grant No, MDA903· B6-CO l~3 from the Army Research
Institute.
6,
M.T.H. Chi, M. Bassok, M.W. Lewis, P. Reimann,
R. Glaser. Self-ExJ?lanation s: 1I0w Students Study
and Use Examplcs III Learning to Solve Problems.
Cognitil'e Science 13, :2 (1989), 1-l5-182.
7.
P.R. Cohen, R. Kjcldsen. Information Retrieval by
Constrained Spreading Activation in Semantic Networks. illformation Processing and Management 23,
4 (1987), 255- 26S.
REFERENCES
I
~
')
,,:'
J
R,K. Belew. Adtl/Jlil'C fll/imll ({lio/l RCII'ic\'u!:
M achin c Learning ill A ss()cia l il c NCII ,'orb, Tech nicll
9.
:-.i.J, Belkin, W,B. Crofl. Rctric \:d Techniques.
10. G. Fischer. Cognitiv c View of R~u se anJ Redesign .
IEEE S()frll'are,~SI)('Cia lisslie Oil RClisahility -/, 4 (July
(Jilt!
Techll o!-
]lJ87),60-n
1 l. G. Fisc her. Making Computers more Useful and more
Usable. Proceedings of the 211d imcrnarional COIlf eren ce on 1f1lIllan,CUIIlJlIIICf !n terac tion (HOIlO!Ii!Ii,
/lawai i), Elsev ier Science Pu blishers, New York,
August, 1987, pp. 97-104.
T.J . Biggerstaff, 1\.1. Perli s (Ed,), SO/illlllC
Addison-Weslev Publishing CL1 mpan\, . Rea Jing , rvlA,
1989.
.
-
.
T.A, Vdn Dijk, W. Killtsch, Slrafegies of Discollrse
Acad emic Press, New York , 1983.
COll1prcftcl1.lirJ/l.
RClisa hiliry, Vo!ume f: COIIC/ '/JI,I' olld J[odels.
5,
W.B. Croft, T.J. Lucia, J. Cringean, P. Willett.
Retrieving Doc uments by Pl ausible Inference: An Experimental Study. illforll1l1ti{)/ l Processing and
Man{ll{clIlcllr 25, () (198'),599-614.
Rep()[t -l, Cogniti v\.! ScienCe and i\Ltchinc Intell ige nce
L lro ratory , Univcrsity of 1'.lichigan, 1()~7,
Anllllal Rel 'iell' of flltmnl/lio/l Sciellce
ogy (ARiSf) 22 (I %7), 109-I-lS.
4.
8.
lR . Anderson. Th e Archilcctll r,' of Cog IIi Ii011. Harvard University Press, Cambridge, ;"IA, 1()83.
-
D,C. Blair. 1\I.E. 1'.Ltrnn, An [ v:llll :ttion of Rctri cI';d
Effec tivene ss for a Full-Tc xt DUClIl1lellt-Ectricval S\Stem, CO/ll/ll/ll/iCIJliolls 0/1/1(' ,.\(',\/ .?S (i\b rch I lJ~ 5i,
289-299.
12. G, Fischer. Creativity EnhJllcing Des ign Environments. IJroc('cdings OJ' lirC flll <'maliollal Con/ercn ce
',\fodcIlil1!.; Crcalilil} tln d /{ll o lt 'IciI!;e- lla scd Creatil '('
Desig n ' (tJcr onisla ll ri, ;\1/I II'lI li<l) . October, 1989. pp,
127-132.
61
21. G. Lakoff. Women, Fire, and Dangerous Things:
What Categories Reveal About the Mind. The University of Chicago Press, Chicago, IL, 1987.
13. G. Fischer, P.W. Foltz, W. Kintsch, H. Niepcr-Lemke,
e. Stevens. Personal Illformation Systems and
Models of Human Memory. Department of Computer
Science, University of Colorado, Boulder, CO, 1989.
22. 1. Lave. Cogllition ill Practice. Cambridge University
Press, Cambridge, UK, 1988.
14. G. Fischer, A Girgensohn. End-User Modifiability in
Design Environments. Human Factors in Computing
Systems, CHI'90 Conference Proceedings (Seattle,
WAY, ACM, New York, April, 1990, pp. 183-191.
23. e.H. Lewis, G.M. Olson. Can the Principles of Cognition Lower the Barriers of Programming? In G.M.
Olson, E. Soloway, S. Sheppard (Eds.), Empirical
Studies of Programmers (Vol. 2), Ablex Publishing
Corporation, Lawrence Erlbaum Associates, Norwood, NJ - Hillsdale, NJ, 1987.
15. G. Fischer, H. Nieper-Lemke. HELGON: Extending
the Retrieval by Reformulation Paradigm. Human
Factors in Computing Systems, CHI'S9 Conference
Proceedings (Austin, TX), ACM, New York, May,
1989, pp. 351-362.
24. T.P. Moran. Getting into a System: External-Internal
Task Mapping Analysis. Human Factors in Computing Systems, CHI'S3 COllference Proceedings (Boston,
MA), ACM, New York, December, 1983, pp. 45-49.
16. P.W. Foltz, W. Kintsch. An Empirical Study of
Retrieval by Reformulation on IIELGON. In AA
Turner (Ed.), Mel/tal Models alld User-Centered
Design, Workshop Report (Breckenridge, CO), Institute of Cognitive Science, University of Colorado
(Technical Report 88-9), Boulder, CO, 1988, pp. 9-14.
25. M.e. Mozer. Illductive Illformation Retrieval Using
Parallel Distributed Computation. ICS Report 8400,
Institute for Cognitive Science, University of California, San Diego, La Jolla, CA, June, 1984.
17. G.W. Furnas, T.K. Landauer, L.M. Gomez, S.T.
Dumais. The Vocabulary Problem in Human-System
Communication. Communications of the A CAl 30, 11
(November 1987), 964-971.
26. D.A Norman. The Psychology of Everyday Thillgs.
Basic Books, New York, 1988.
27. D.A Norman, D.G. Bobrow. Descriptions: An Intermediate Stage in Memory Retrieval. Cognitive
Psychology 11 (1979), 107-123.
18. S. Henninger. Defining the Roles of Humans and
Computers in Cooperative Problem Solving Systems
for Information Retrieval. Proceedings of the AMI
Spring Symposium Workshop 011 Knowledge-Based
Human Computer Commu1l/catioll, March, 1990.
28. P.L. Pirolli, J.R. Anderson. The Role of Learning
from Examples in the Acquisition of Recursive Programming Skills. Canadian Journal of Psychology
39,2 (1985),240-272.
19. B.A Huberman, T. Hogg. Phase Transitions in Artificial Intelligence Systems. Artificial Intelligence, 33
(1987),155-171.
29. B. Reeves. Findillg alld Choosillg the Right Object in
a Large Hardware Store -- An Empirical Study of
Cooperative Problem Solvin$ amollg Humans.
Department of Computer SCience, University of
Colorado, Boulder, CO, 1990.
20. W. Kintsch. The Role of Knowledge in Discourse
Comprehension: A Construction-Integration Model.
Psychological Rel'iew 95 (1988), 163-182.
30. T.A. Standish. An Essay 011 Software Reuse. IEEE
Tra1lsactiOllS 011 Software EllgilleeringSE-JO, 5 (September 1984),494-497.
31. M.D. Williams. What Makes RABBIT Run?
International Joumal of Mall-Machine Studies 21
(1984),333-352.
62