Running Head: Analyzing A Novel Expertise
Analyzing a Novel Expertise: An Unmarked Road
Wayne D. Gray
George Mason University
&
Susan S. Kirschenbaum
Naval Undersea Warfare Center Division Newport
Gray, W. D., & Kirschenbaum, S. S. (in press). Analyzing a novel expertise: An unmarked road.
In J. M. C. Schraagen, S. F. Chipman, & V. L. Shalin (Eds.), Cognitive task analysis .
Mahwah, NJ: Erlbaum.
Please send all correspondence to:
Wayne D. Gray
George Mason University
MSN 3f5
Fairfax, VA 22030
1-703-993-1357
gray@gmu.edu
Gray & Kirschenbaum
Analyzing a Novel Expertise
page 2
Analyzing a Novel Expertise: An Unmarked Road
There are many varieties of task analysis-each with its advantages and disadvantages, each
with its adherents and detractors (e.g. see the recent collections published by Annett & Stanton,
1998; Kirwan & Ainsworth, 1992). Most published descriptions focus on how to apply the
technique or why it is a good technique to apply. Few accounts written by advocates of a
technique are specifically directed at problems and pitfalls in applying the technique. This
account is different. Although we are unabashedly enthusiastic advocates of the theory-driven
combination of task analysis and protocol analysis that we employ, we hope that by identifying
problems and obstacles that we encountered that more people will be better prepared and,
therefore, more successful at applying these techniques.
Beware—knowing that the road is narrow, winding, and unmarked does not make the trip
easy. It might, however, discourage someone from setting out in the family sedan. For those who
are better equipped, knowledge of the hazards ahead may help them avoid blindly plunging
forward into a known problem. It is in this spirit that we write this chapter.
The following section provides a brief overview of the techniques we employ. It then
introduces the known obstacles to these techniques. The main part of the chapter discusses these
obstacles in the context of a specific project—Project Nemo.
Theory-Driven Task Analysis and Protocol Analysis
Theory-driven task analysis decomposes the procedural and declarative knowledge required to
perform a task into components supported by the theory. With some additional work on the part
of the analyst, the control structure provided by the theory can use the elements of the analysis
Gray & Kirschenbaum
Analyzing a Novel Expertise
page 3
to form a model of how a user performs the task. Theories with weak or rigid control structures,
such as keystroke-level GOMS or CPM-GOMS (for an overview, see John & Kieras, 1996a;
John & Kieras, 1996b), may produce models that are only capable of performing the exact task
that was analyzed. Theories with more powerful control structures, such as NGOMSL, ACT-R,
Soar, or EPIC (see Gray, Young, & Kirschenbaum, 1997b), may respond adaptively to perform
variations of the analyzed task.
Cognitive theories provide constraints to the final form of the analysis—that is, for how the
components must fit together. However, the components per se vary widely, and the analysis of
expertise into such components is an underconstrained problem. For example, the expertise
exhibited by a chess master in plotting his next move is different from that shown by a medical
expert diagnosing a rare disease (VanLehn, 1989). Once the components (i.e., the knowledge
structures and strategies) of expertise have been delineated, they can be cast into the mechanisms
of a cognitive theory. However, existing cognitive theory provides few a priori constraints for
deriving the components of a hitherto unstudied expertise.
Given a rare form of expertise, one that has been subject to few published reports
(Kirschenbaum, 1990; 1992; 1994), how does the analyst proceed? The method adopted here is a
form of bootstrapping. As shown in Figure 1, an initial task analysis1 guides a protocol analysis.
The task analysis is revised and used as the basis of the next round of protocol analysis. When
the analyst deems that the results of the analysis are as good as the existing data permit, the
analyst moves to the next phase of the effort.
Gray & Kirschenbaum
Analyzing a Novel Expertise
page 4
Initial Task
Analysis
Protocol
Analysis
Results of
Analysis
As good as the
data permit?
No
Revise Task
Analysis
Yes
Next Phase
Figure 1: Bootstrapping
The story told in this chapter is the story of an iterative loop around the stages shown in
Figure 1. The project has now moved to the next phase. The story of the next phase is ongoing;
its outlines are given in Ehret, Kirschenbaum, and Gray (1998) and Ehret, Gray, and
Kirschenbaum (1999).
Dead Ends and Wrong Turns
Studying an expertise that has not been extensively analyzed is like traveling on an unmarked
Gray & Kirschenbaum
Analyzing a Novel Expertise
page 5
road – one with many intersections and forks. What aspects of the expertise are important for the
goals of the analysis? Can knowledge of the components of other expertise guide and inform the
current analysis or does this knowledge serve to lead us astray? If the expertise is a dynamic
expertise – problem solving that takes place over a period of time and that is responsive to events
in the environment – what is the best way to capture key aspects of the expertise without
distorting it due to artifacts of data collection?
These problems are common in analyzing expertise. Indeed, others have warned that an almost
inevitable danger of doing the first, deep-level cognitive analysis of a hitherto unstudied expertise
is that “the final interpretation of the data and its matching against the theory [i.e., task analysis]
may appear to reflect mainly hindsight – pulling out of the data the hypotheses that have been
implicitly hidden in it by the data interpretation process” (Ericsson & Simon, 1993, p. 286).
Although we have read (and heeded) the warnings, what we have not read is a detailed
discussion by analysts of their encounters with problems that threaten the validity of their
conclusions. This chapter fills that void. In it, we provide an autobiographic description of the
problems encountered in our analyses for Project Nemo. We feel no shame at admitting to having
problems. Indeed, when the road is rocky and unmarked, problems must be expected. The shame
lies not in having problems, but in not recognizing problems. The shame falls to the analyst who
mistakes a dead end for the end of the trip or a wrong turn for the right path.
The Task and Our Goals in Analyzing It
Different cognitive task analyses may have different goals. The goal of Project Nemo was to
analyze the knowledge and cognitive processes used by submarine approach officers (AOs) as
Gray & Kirschenbaum
Analyzing a Novel Expertise
page 6
they tried to localize an enemy submarine hiding in deep waters. The project is a collaboration
between a navy and a university researcher. An important role played by the navy researcher
was to feed the results of the project, as they came in, to those parts of the submarine research
community that could make the most use of them. From the beginning of the project, the most
interested parties have been the designers of the command workstation for the next-generation
submarine.
Knowing that the results of the analysis would be used for interface design provided an
important constraint on the knowledge and cognitive processes that Project Nemo analyzed. The
AO possesses specialized knowledge that is acquired over a 20-year period. Rather than focus on
the acquisition, depth, and breadth of this knowledge, we focused on how it is used as the AO
makes progress in his goal of localizing the enemy submarine. Our effort focused on the
knowledge, structures, and strategies unique to the dynamic problem-solving process of localizing
an enemy submarine hiding in deep water.2
Any given task takes place in the context of the artifacts and organization used to perform the
task. The goal of many task analyses, including most GOMS analyses, is to analyze task
interactions at this activity level. Such analyses can either assess the problems with the current
way of performing the task or provide specifications for a new system. The goal of the current
analysis is different. We are neither involved in critiquing the current system nor directly
involved in designing the new system. Rather, our intent is to provide the designers of the new
system with a detailed description of the information processing performed by the AO as he
Gray & Kirschenbaum
Analyzing a Novel Expertise
page 7
localizes an enemy submarine. Hence, in our analysis, we seek to abstract from the activity level
to the functional level (Gray, John, & Atwood, 1993, pp. 244-257).
Issues
We had two sets of problems in understanding AO expertise. The first set included issues in
understanding the control structure of the cognitive processes used by the AO to perform the
localizing task. The second set entailed understanding the nature and limits to the data we had
collected.
The Control Structure of AO Expertise
A common metaphor is to conceptualize problem solving as search in a problem space
(Newell & Simon, 1972). Tasks involving expertise are often thought of as being both wide and
deep. At each step in the problem space, there are many alternative next steps (width). Solving a
problem involves solving many subproblems, and each subproblem can be decomposed into
another subproblem that needs to be solved (depth). In contrast to expertise, the accepted
wisdom is that, for everyday tasks, the search space is limited. The problem spaces for everyday
tasks are either shallow and wide (like choosing a flavor from the menu in an ice cream store) or
narrow and deep (like following a recipe from a cookbook) (Norman, 1989).
Clearly, localizing targets hiding in deep water is not an everyday activity. Therefore, we
reasoned that rather than being shallow and wide or narrow and deep that the AOs’ expertise
must be both wide and deep. This bias led us to make our first and most fundamental mistake
regarding the control structure of the AOs’ cognitive processes for this task.
Gray & Kirschenbaum
Analyzing a Novel Expertise
page 8
Matrix? Subgoals?
One of the most basic control structures used in cognitive task analysis is hierarchical, or goalsubgoal, decomposition. From discussions with experts, however, it was clear that AOs kept
track of many different pieces of several different tasks. Hence, a classic, hierarchical control
structure did not seem accurate.
Our alternative was a matrix goal structure. Although we abandoned the matrix notion before
working out the details, its basic elements were as follows. Imagine a cube with AO goals along
the x axis, information elements about the target, own ship, and ocean conditions along the y axis,
and rules that capture the AOs’ procedural knowledge along the z axis. A given rule would yield
information regarding one or more information element. In turn, a given information element
would be applicable to one or more AO goals. In such a system, one could imagine that the rules
that fired (i.e., the actions that the AOs take) would be those rules that yielded the greatest
amount of new information for the greatest number of AO goals.
Unfortunately, for our preconceptions, we found little in our data that would support a matrix
organization. We gradually abandoned this idea and, for a while, ignored this issue to concentrate
on the issue of how and why one rule rather than another is selected. Our explorations of these
control structure issues led us, albeit a bit unwillingly, to the realization that most AO actions
could be characterized as small steps in a shallow goal hierarchy. However, unlike the everyday
task of choosing one flavor from a wide but shallow ice cream store menu, AOs make many
successive choices. It is the nature of these successive choices that characterize the AOs’
procedural expertise.
Gray & Kirschenbaum
Analyzing a Novel Expertise
page 9
Our current theory of how AOs solve the localizing problem can be summarized by the rather
awkward phrase “schema-directed problem solving with shallow and adaptive subgoaling”
(SDPSSAS). The schema is the task-relevant knowledge accumulated over 20 years of experience
as a submariner (half of it at sea). It is a knowledge structure3 that contains both declarative and
procedural knowledge. An implication of shallow subgoaling is that the knowledge available to
AOs is so rich that the steps required to supplement this knowledge can be fairly shallow.
The second implication is that the problem the AO is constantly solving is “what is the state
of the world – NOW” (where NOW is somewhere on the order of 30 to 300 s). The AO is trying
to find a quiet target hiding in a noisy environment while remaining covert and undetected
himself. What we see him doing is taking short steps that either (a) assess the noise from the
environment or signal from the target – NOW, or (b) attempt to reduce the noise or increase the
signal from the target by maneuvering own ship. As shown in Figure 2, these short steps result in
shallow subgoaling. When a subgoal pops, the schema is reassessed. The result of this
reassessment directs the next step (i.e., selects the next subgoal). This step is accomplished, it
returns information to the schema, the schema is reassessed, and so on
The process of subgoaling is adaptive in two senses. First, the subgoal that is chosen next
reflects the current reassessment of the schema. Second, this choice is sensitive to both the longterm importance of the subgoal as well as its recent history of success or failure. Regardless of a
goal’s long-term importance, AOs will not continue to attempt a goal if successive tries fail.
Instead, they will choose another goal and return to the more important goal later.4
Gray & Kirschenbaum
Analyzing a Novel Expertise
page 10
The dynamic aspect of the AO's task plays an important role in this view of schema-directed
problem solving with shallow and adaptive subgoaling. First, the state of the AO's world is
continually changing – both own ship and target are moving at a given depth, direction, and
speed. For own ship, the value of these attributes can be changed, but neither own ship nor the
target can be stopped. Consequently, time is an important part of the picture. Second, subgoals
are not accomplished once and then discarded. In the AO's world, subgoals bring in certain types
of information or accomplish certain changes to own ship. As the world changes, any given
subgoal may be revisited (e.g., DET-BEARING in Figure 2).
LOCATESUB
(t)
DETBEARING
LOCATESUB (t+1)
EVALSNR
LOCATESUB (t+2)
LOCATESUB (t+3)
SETTRACKER
LOCATESUB (t+4)
DETBEARING
LOCATESUB (t+n)
TMA-SOL
Time (t)
Figure 2: Schema-Directed Problem-Solving with Shallow and Adaptive
Subgoaling (SDPSSAS)
Choosing What to Do Next: Goal-Driven? Event-Driven?
When we put aside the issue of the larger organization of the AO's actions – subgoal or matrix
(at this point, we had not hit on SDPSSAS) – we focused on the more local issue of how or why
an individual rule was chosen. For problems such as arithmetic, physics problems, and Tower of
Hanoi, the pacing of the solution is entirely under the control of the person solving the problem.
The problem can be decomposed into a deep and increasingly wide series of goals and subgoals.
Gray & Kirschenbaum
Analyzing a Novel Expertise
page 11
At any given place in this goal-subgoal hierarchy, the action chosen next completely depends on
the plans and knowledge of the problem solver. Such tasks are said to be goal-driven.
We recognized from the beginning that the AO’s problem is not simply goal-driven in that the
problem state changes without the intervention of the person. Some part of what the AO does is
clearly an event-driven process. The issue is not event-driven versus goal-driven, but what
combination of the two controls the AO's problem-solving behavior and how this combination
can be represented in a cognitively plausible manner.
As we were struggling with the goal-driven versus event-driven interpretations, we were
having problems supporting another early idea – localizing a target as a diagnostic process. The
literature suggests that an important first stage in solving many tasks is diagnosing the problem.
This stage ends when the correct schema is selected (VanLehn, 1989).
We fully expected that the first stage of localizing the target would entail diagnosing the
situation to determine the correct target-finding schema. Indeed, for about the first 18 months,
our encodings had a goal called IDENTIFY-POSSIBLE-INITIAL-SCHEMA. Believing that the
“absence of evidence is not evidence of absence,” we struggled mightily to find support for a
diagnostic process. However, we could not find evidence in our protocols or elsewhere (e.g., from
any of the other AOs and experts whom we consulted) that we could interpret as evidence that
schema selection required extended deliberation. Our tenacity in clinging to the notion of schema
selection is, in part, attributable to evidence that suggested that AOs used different schemas
when the target was a submarine than when the target was a merchant. Unfortunately, this
evidence turned out to be bogus.
Gray & Kirschenbaum
Analyzing a Novel Expertise
page 12
In brief, our analyses indicate that when LOCATE-MERCHANT was the goal, fewer
subgoals were pushed and popped than when LOCATE-SUBMARINE was the goal (see the left
side of Figure 3). More important, most of the subgoals for LOCATE-MERCHANT were
similar to, but qualitatively different from, those for LOCATE-SUBMARINE. As discussed
later, what turned out to be bogus was the apparent qualitative difference between similar
subgoals for different goals. Once this spurious qualitative difference was eliminated, we
concluded that the same LOCALIZE-THE-TARGET schema guided the processing of localizing
either a hostile submarine or a friendly merchant. This conclusion converged with our failure to
find any evidence of schema selection. We currently believe that schema selection is not a
problem. Whether the goal is LOCATE-SUBMARINE or LOCATE-MERCHANT, our expert
AOs have one schema, LOCALIZE-THE-TARGET, that is automatically chosen.
Schema-directed Problem Solving
Conceptualizing the AOs’ task as schema instantiation, not selection, was the key to a
parsimonious account of the general structure of the AOs’ cognitive processes. Localizing the
target was not driven by goals or events, but was directed by a schema. The data gathered were
used to instantiate attributes of this schema. On each cycle of problem solving, the schema was
reevaluated, the currently most critical attribute-value pair was identified, and instantiating this
pair became the current subgoal. New events were incorporated into the schema and affect
problem solving only to the extent that they affect the identification of the currently most critical
attribute-value pair.
Gray & Kirschenbaum
Analyzing a Novel Expertise
page 13
subgoal per goal
operators per subgoal
8
number
6
4
2
0
LOC-MERC LOC-SUB LOC-MERC LOC-SUB
Figure 3: Mean number of subgoals per goal (left two) and mean number of operators per
subgoal (right two) for LOCALIZE-MERC and LOCALIZE-SUB. Also shown are the 95%
confidence intervals for the standard error of the mean.
This view of schema instantiation as schema-directed problem solving fits in nicely with our
emerging awareness of shallow subgoaling. Each time the AO returned to the top-level goal
(LOCATE-SUBMARINE in Figure 2), the schema was reevaluated. The subgoal chosen was one
that would return information regarding one attribute-value pair. Hence, localizing a target is a
wide and shallow task. The width is represented by a well-learned schema. The shallowness is
represented by shallow subgoaling. Localizing the target involves dozens of iterations of schema
evaluation and shallow subgoaling. These iterations continue until the AO has confidence that the
schema as currently instantiated provides accurate information regarding the target’s location.
Gray & Kirschenbaum
Analyzing a Novel Expertise
page 14
Data Analysis: Levels of Analysis and Limits to the Data
Changes in our taxonomy of goals, subgoals, and operators were driven by three sources: our
changing understanding of the general structure of AO cognition (discussed earlier), our failure to
understand the limits to our data (discussed next), and our tendency to embed specifics of
interacting with the simulation in our analysis of the task – that is, a failure to distinguish
between task and artifact (discussed later).
Distinctions Not Supported
Our final encodings resulted in nine categories of operators (Gray, Kirschenbaum, & Ehret,
1997a; Kirschenbaum, Gray, & Ehret, 1997). Each of these categories can be considered as
representing many subcategories. For example, at one time, we tried to distinguish subcategories
of things that could be queried and sources from which information could be received. An AO
could query a display, the own ship operator5, his own memory, an instrument reading, and so
on. Likewise, information could be received from long-term memory, from short-term memory,
from reading a table, from viewing a graphic, from inferring a relationship among other
information, from the own ship operator, and so on. At one time or another, each of these
subcategories was considered – if only briefly – as a candidate for inclusion in the analysis.
In deriving our encoding categories, we focused on two transcripts, one from each of two AOs.
New encoding categories could be proposed by any of our three encoders. Typically, the encoder
would present a rationale for why the subcategory was needed, along with a particular instance
from the transcript that the encoder felt represented that subcategory. If the other two encoders
were convinced by the instance, much discussion would be spent defining criteria by which other
instances could be identified. Two or three of the encoders would together go through one of the
Gray & Kirschenbaum
Analyzing a Novel Expertise
page 15
two sample transcripts. New instances of the encoding category would be identified and debated
until consensus was achieved. After this, two or three of the encoders would independently go
through the other sample transcript in an attempt to identify instances of the new category.
More often than not, each independent encoding would yield a handful of instances (out of
about 300 encoded operators) that could be considered members of the new category. However,
there would be little or no agreement about which encodings were exemplars of the new category
and which were not. At this point, the candidate would be abandoned. We considered it a
plausible distinction, but a distinction for which the data were at too coarse of a level to support.
Problem Solving the Tool as Opposed to Problem Solving the Task
We collected data from AOs who localized targets presented on a dynamic simulation of the
ocean environment – CSEAL (for more information, see Gray et al., 1997a; or Kirschenbaum et
al., 1997). However, our task was not to describe how AOs’ used the simulation (the artifact or
tool) to localize the target (the task), but to separate the specifics of the simulation from the more
general aspects of their problem-solving process (i.e., a functional level of analysis). Time and
again, we found ourselves encoding specifics of using the simulator to localize targets rather than
simply localizing targets.
We attempted to encode every segment of the transcript, but realized early on that a
significant number of segments simply had nothing to do with the problem. The fact that many
of the segments in a verbal protocol have nothing to do with the problem being solved is not
news and, indeed, we expected this. These NAs (not applicable) were easy to identify because
they represented comments about the desirability of eating lunch soon, the drive to the base, the
Gray & Kirschenbaum
Analyzing a Novel Expertise
page 16
weather, the room in which the simulation was run, and so on. Of the 2,882 segments encoded,
946 were in this NA category.
More difficult to distinguish were the segments that did not refer to solving the problem but to
some aspect of the simulation. An analogy to this is talking to a co-author about how to format a
table in Microsoft Word™ as opposed to discussing the data that would go into the table. A
simple example is represented by the following query from the AO to the own ship operator
asking about the units (relative or true) in which a particular display represented the data.
Protocol:
“and so this [display] is relative or true?
(AO points to TB16 block)?”
At some point, we realized that what we were seeing was a ubiquitous phenomenon that was
not limited to our paradigm, but that represented a type of usability issue. We saw this as
problem solving in the tool space, as opposed to the task space, and proposed the tool:task ratio
as a usability metric (Kirschenbaum, Gray, Ehret & Miller, 1996). Following this insight, our
transcripts were recoded with a new operator, instrumentation. As per the prior example, most
of the instrumentation operators were found in groups of one to three amid a series of task
operators. These small groups of instrumentation operators are essentially asides that are
embedded among operators concerned with a particular task goal. Occasionally, both the AO and
the own ship operator abandoned the task of localizing the target and became engaged in an
episode of collaborative problem solving – attempting to figure out how to get the simulation to
take a particular input or to display a particular type of data. Such episodes were recoded as a
new goal – that of supervising the use of the tool (supervisory).
Gray & Kirschenbaum
Analyzing a Novel Expertise
page 17
Over our entire set of encoded data, 421 goals and 2,882 operators were required to encode the
nine scenarios from our six AOs (for more details, see Gray, Kirschenbaum, et al., 1997 or
Kirschenbaum et al., 1997). After removing all supervisory goals, their operators, all
instrumentation operators, and all NA operators, we were left with 397 goals and 1,269
operators. We refer to this remainder as our clean set. The clean set of encodings was used in all
subsequent analyses.
The significance of this reduction cannot be overstated; over half of our encoded utterances
had nothing to do with localizing per se. Having these in the analyses confounded our efforts to
make sense of the data. Once these were removed, regularities that had been obscured became
apparent. For example, the scenarios we studied involved two targets: a hostile submarine and a
friendly merchant. Given the shallow subgoaling we were beginning to believe in, it made sense to
us that there would be more subgoals involved in localizing the quiet submarine than in localizing
the noisy merchant. Indeed, this is what our encodings suggested (see the left two data points in
Figure 3). However, the same subgoal seemed to involve many more steps if its supergoal was
LOCATE-SUBMARINE rather than LOCATE-MERCHANT.
More steps were needed for a subgoal such as determine the signal-to-noise ratio (determineSNR) when determine-SNR was a subgoal of LOCATE-SUBMARINE than when it was a
subgoal of LOCATE-MERCHANT, and this did not seem unreasonable. Indeed, as discussed
earlier, such differences supported our belief that different schemas were used for different
targets. By inference, this finding supported the belief that schema selection was an important
component of the AO's problem-solving process.
Gray & Kirschenbaum
Analyzing a Novel Expertise
page 18
However, once the instrumentation operators and supervisory goals were removed, regularities
appeared. As shown in the right two data points of Figure 3, the subgoals used in LOCATEMERCHANT required the same number of operators as the subgoal of LOCATESUBMARINE. (An extended discussion of this point is provided in both Gray et al., 1997a; and
Kirschenbaum et al., 1997.)
Completed versus Successful Goals
Another part of our struggle to define the number of levels and depth of the goal stack resulted
from an implicit assumption that a completed goal was synonymous with a successful goal. In
our initial attempts to shoehorn reality, we viewed a completed goal as one that returned the
information queried. For example, if the AO queried the bearing rate on a particular target, this
goal would not be completed until the target’s bearing rate was determined. Attempting to trace
the path from initial query to completion led us to postulate a tangled web of semi-infinite
subgoaling. Stepping back and listening to the data led to a different conclusion. We discovered
that, for the AO, knowing that, at a given point in time under current conditions, the bearing rate
(or course, speed, etc.) cannot be determined is an important and complete piece of information.
The insight that a goal can be considered completed without being considered successful
supports the shallow subgoaling component of schema-directed problem-solving. The AO
launches a continuing stream of short queries. Each query returns some information. In the early
stages of localizing (immediately after the target has been detected), the information typically is
something such as the data are too noisy to answer that question (the SNR is literally too low).
This leads the AO to take actions to increase the SNR.
Gray & Kirschenbaum
Analyzing a Novel Expertise
page 19
Summary of Issues
Understanding the control structure of the AOs’ cognition, together with understanding the
nature and limits of the data, were major obstacles in our attempts to do a cognitive analysis of
the AOs’ task. Before beginning this project, we knew that understanding the control structure of
cognition would be the key to the cognitive task analysis (this concern was reflected in the
original proposal to ONR).
At each iteration around the loop (see Figure 1), each component of our analysis seemed
plausible. What troubled us were our efforts to fit the parts together into a coherent whole.
Although during this period we were not building ACT-R models, we were constantly asking
ourselves how the disparate parts could fit into an ACT-R model. It was the failure to answer
this question positively that kept driving us around the loop and deeper into the data.
At this point, we have exited the loop (Figure 1) and moved onto the next phase of the
project. Although the current hypothesis – schema-directed problem solving with shallow and
adaptive subgoaling – is coherent, we are collecting additional data in the hopes of capturing finer
grained data on key aspects of AO problem solving in a dynamic environment.
Conclusions
The current chapter has concentrated on the difficulties of doing a deep-level cognitive task
analysis of a novel expertise. The difficulties are all the more notable in that our team of
researchers brought to the study considerable expertise in cognitive theory and in applying
cognitive theory to real-world tasks. Prior to working on Project Nemo, the co-authors of this
chapter conducted research on COBOL programmers, HAWK Air Defense maintenance workers,
Gray & Kirschenbaum
Analyzing a Novel Expertise
page 20
small-unit tactical team training, phone company operators, as well as more traditional decisionmaking studies of submariners and school children.
Although the road was difficult and unmarked, we have arrived at our destination. Our current
characterization of the AOs’ expertise – schema-directed problem solving with shallow and
adaptive subgoaling – is both simpler and more profound than what we had envisioned when we
began our journey. As far as we can tell, this characterization is unlike any that appears in the
literature on expert performance. As such, it is important that those who are designing the
command workstation understand this characterization of the AOs’ expertise rather than
designing an interface that will support the consideration of multiple hypotheses (as in medical
diagnoses) or the in-depth exploration of several alternative courses of action (as in chess
playing).
Instead of telling stories about how difficult our trip was, we would rather give the reader a
sure-fire guide to plotting a safe path to any destination, on any road, marked or unmarked. We
do not know if such a guide can be written. However, we are sure that we cannot write one.
Unfortunately, the truth remains that, whatever may be done differently, the task of
understanding a hitherto unstudied expertise will never be quick or easy. The problems discussed
in this chapter can be anticipated but not avoided.
Notes
Acknowledgment
More than usual, we thank our agency sponsors and our scientific officer for understanding
that if we knew what we were doing, it would not be called research. We believe that the emerging
Gray & Kirschenbaum
Analyzing a Novel Expertise
page 21
results justify their long-term support of this effort. However, we also understand that there
were times when they might have thought that a successful outcome was unlikely. We thank
Brian D. Ehret who joined our project about two thirds of the way through the events recounted
here. Brian was the third encoder on each of the final encodings of the transcripts. His diagnoses
have guided the current phase of data collection. The work on this project at George Mason
University was supported by a grant from the Office of Naval Research (#N00014-95-1-0175) to
Wayne D. Gray. Susan S. Kirschenbaum’s work has been jointly sponsored by Office of Naval
Research (ONR) (Program element 61153N) and by Naval Undersea Warfare Center's
Independent Research Program as Project A10328.
Authors present address
Wayne D. Gray can be reached at Human Factors and Applied Cognitive Program, George
Mason University, MSN 3f5, Fairfax, VA 22030, USA. E-Mail: gray@gmu.edu. Susan S.
Kirschenbaum can be reached at Naval Undersea Warfare Center Division Newport, 1176 Howell
St., Code 2211, Building 1171/1, Newport, RI 02841-1708, USA. Email:
kirschenbaumss@csd.npt.nuwc.navy.mil.
End Notes
1
The initial task analysis can be as informal as it has to be, but should be as formal as
possible. For Project Nemo, the initial task analysis was based on the published literature
(Kirschenbaum, 1990; 1992; 1994).
2 After a target is detected, it must be localized. Detection tells the AO that a target is out
there. Localizing tells him where it is in terms of bearing from own ship, range, course, and
Gray & Kirschenbaum
Analyzing a Novel Expertise
page 22
speed. Due, in part, to the physics of sound transmission underwater and the need to remain
covert, localizing a target is a mathematically underconstrained problem. Passive sonar is the
only tool available to the AO. From passive sonar, the AO can directly compute the bearing of the
target. Computing the target’s range, course, and speed is a difficult process.
3
As we think of the schema in terms of ACT-R mechanisms, the schema would be a body of
task-specific, declarative memory elements and productions. Any given declarative memory
element is relatively small and limited. However, the set of task-relevant, declarative memory
elements have high interitem association values (see Anderson & Lebiére, 1998).
4
As described by Lovett (1998), this adaptive subgoaling can be modeled in ACT-R 4.0 as
the temporary depression and recovery of the expected value of a goal.
5
The simulation required extensive training to operate. Rather than teaching AOs this
esoteric task, the simulation was run by an experimenter in the role of own ship operator. This
arrangement mimicked procedures onboard submarines and was acceptable to all of the AOs.
References
Anderson, J. R., & Lebiére, C. (Eds.). (1998). Atomic components of thought. Hillsdale, NJ:
Erlbaum.
Annett, J., & Stanton, N. (1998). Introduction to this Special Issue on Task Analysis.
Ergonomics, 41(11), 1529-1536.
Ehret, B. D., Gray, W. D., & Kirschenbaum, S. S. (1999). Scaled worlds and computational
cognitive models as tools to increase the usefulness of cognitive task analysis. manuscript
submitted for publication.
Gray & Kirschenbaum
Analyzing a Novel Expertise
page 23
Ehret, B. D., Kirschenbaum, S. S., & Gray, W. D. (1998). Contending with complexity: the
development and use of scaled worlds as research tools, Human Factors and Ergonomics
Society 42th Annual Meeting . Santa Monica, CA: Human Factors and Ergonomics Society.
Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. (Revised ed.).
Cambridge, MA: The MIT Press.
Gray, W. D., John, B. E., & Atwood, M. E. (1993). Project Ernestine: Validating a GOMS
analysis for predicting and explaining real-world performance. Human-Computer Interaction,
8(3), 237-309.
Gray, W. D., Kirschenbaum, S. S., & Ehret, B. D. (1997a). The précis of Project Nemo, phase 1:
Subgoaling and subschemas for submariners, Nineteenth Annual Conference of the Cognitive
Science Society (pp. 283-288). Hillsdale, NJ: Erlbaum.
Gray, W. D., Young, R. M., & Kirschenbaum, S. S. (1997b). Introduction to this Special Issue on
Cognitive Architectures and Human-Computer Interaction. Human-Computer Interaction,
12(4), 301-309.
John, B. E., & Kieras, D. E. (1996a). The GOMS family of user interface analysis techniques:
Comparison and contrast. ACM Transactions on Computer-Human Interaction, 3(4), 320351.
John, B. E., & Kieras, D. E. (1996b). Using GOMS for user interface design and evaluation:
Which technique? ACM Transactions on Computer-Human Interaction, 3(4), 287-319.
Kirschenbaum, S. S. (1990). Command Decision Making: Lessons Learned (NUSC TM No.
Gray & Kirschenbaum
Analyzing a Novel Expertise
page 24
902149): Naval Underwater Systems Center.
Kirschenbaum, S. S. (1992). The effect of level of experience on information gathering strategies.
Journal of Applied Psychology, 77, 343-352.
Kirschenbaum, S. S. (1994). Command Decision-Making: A Simulation Study (Technical Report
10,350): Naval Undersea Warfare Center Division Newport.
Kirschenbaum, S. S., Gray, W. D., & Ehret, B. D. (1997). Subgoaling and subschemas for
submariners: Cognitive models of situation assessment (Technical Report 10,764-1).
Newport, RI: NUWC-NPT.
Kirwan, B., & Ainsworth, L. K. (Eds.). (1992). A guide to task analysis. Washington, DC: Taylor
& Francis.
Lovett, M. (1998). Choice. In J. R. Anderson & C. Lebiére (Eds.), Atomic components of thought
(pp. 255-296). Hillsdale, NJ: Erlbaum.
Newell, A., & Simon, H. A. (1972). Human problem solving. Englewood Cliffs, NJ: PrenticeHall, Inc.
Norman, D. A. (1989). The design of everyday things. New York: DoubleDay.
VanLehn, K. (1989). Problem solving and cognitive skill acquisition. In M. I. Posner (Ed.),
Foundations of Cognitive Science (pp. 527-579). Cambridge, MA: The MIT Press.