COGNITIVE TASK ANALYSIS
OF TOUR MANAGERIGUIDE
PLYING KUCHING-LUBOK ANTUIBATANG A1
TOUR ROUTE
DOUGLAS JAMES
A thesis submitted
In partial fulfillment of the requirements for the degree of
Master of Science
Faculty of Cognitive Science and Human Development
UNIVERSITI MALAYSIA SARAWAK
DECEMBER 1999
DECLARATION
I declare herewith that no part of the work referred to in this thesis has been submitted in
support of an application for another degree of qualification of this or any other university
or institution of higher learning.
This dissertation is dedicated to my grandfather:
The late Udin Ak Braoh.
ACKNOWLEDGEMENTS
This research was made possible with the endless support, help and guidance of many. My
grateful thanks and appreciation goes to:
Mr. Philip Nuli Anding, my humble and dedicated supervisor, for guiding me with endless
effort through all aspects of the study.
Prof. Madya Dr. Muhamed Awang, my lecturer, for the great effort in imparting his expertise in
the field of cognitive task analysis and the method of verbal protocol analysis. All lecturers and
staff of IAB-Unimas for their professional effort and help in making this course possible and
memorable.
The Managing Director, Mr. Philip Yong and Operation Manager, Mr. Emong Tinsang and all
staff of the BorneoAdventure Travel Agency Kuching, for enabling me to carry out this study,
with special thanks to:
Mr. Danis Luart Lang
Mr. James Jinyong
Mr. Charles Collin Langgie
Mr. Robert Samy
Pn. Annatasia Lim
Mr. Ralp Luku
A wonderful acknowledgement:
To my mother, Bata Ak. Udin for her endless support and inspiration;
To my wife, Mariana Dom, and our sons, Nobel Awang and Leeroy Shah, for their sacrifice,
love and encouragement.
CONTENTS
Page
TITLE PAGE
DECLARATION
DEDICATION
iii
ACKNOWLEDGEMENTS
CONTENTS
LIST O F TABLES
vii
LIST OF FIGURES
viii
LIST OF ABBREVIATIONS
ABSTRACT
ABSTRAK
CHAPTER ONE: THE PROBLEM AND ITS SETTING
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8
Introduction
Background of the Problem
Statement of the Problem
Significant of the Study
Purpose of the Study
Objectives
Scope and Delimitation of the Study
Outline of the Study
CHAPTER TWO: LITERATURE REVIEW
2.1 Introduction
2.2 Knowledge
2.3 Information Processing Theory
2.3.1 Sensory Information Storage
2.3.2 Short-term Memory
2.3.3 Long-term Memory
2.4 Cognitive Task Analysis
2.5 Verbal Protocol Analysis
2.6 Protocol-Based Coding Scheme
2.7 Mental Model
2.8 Cognitive Task Analysis of Tour Guide or Tour Manager in Tourism
2.9 Summary
CHAPTER THREE: RESEARCH METHODOLOGY
3.1
3.2
3.3
3.4
3.5
3.6
Overview of the Chapter
Operational Definitions
Research Methodology and Framework
Subjects
Data Collection
Data Processing and Analysis
3.6.1 Analyzing Conceptual Operator for Tour Guide or Tour Manager
3.6.2 Analyzing Knowledge States for Tour Guide or Tour Manager
3.7 Methodological Assumptions
3.8 Limitations
3.9 Summary
CHAPTER FOUR: FINDlNGS O F THE STUDY
4.1 Introduction
4.2 Analyzing the Knowledge States
4.2.1 Knowledge States
(a) Hospitality Management in Tourism (HMT)
(b) Socio-cultural nnd Art (SCA)
O Economy
(d) Eco-tourism
(e) Education
4.3 Result of Analysis on Knowledge States
4.4 Analyzing the Conceptual Operators
4.4.1 Conceptual Operators
(a) Data Explanation
(b) Verlflcation
O Anticipation
(d) Evaluation
(e) Recognition
(f) Estimation
(g) Postulation
4.5 Results of Analysis on Conceptual Operators
4.6 Verbal Protocol Coding Scheme for Tour Guide or Tour Manager
4.7 Simplified Mental Model for Tour Guide or Tour Manager.
4.8 Summary
CHAPTER FIVE: SUMMARY, CONCLUSION AND RECOMMENDATIONS
5.1 Summary of the Study
5.2 Conclusions of the Study
5.3 Recommendations for Further Research
Reference
Appendix
Appendix
Appendix
A
B
C
Analysis on Knowledge States of Tour Guide or Tour Manager
Analysis on Conceptual Operators of Tour Guide or Tour Manager
Letters of Authority Relating to the Research
LIST O F TABLES
Page
TABLE
2.1
Taxonomy of general and specific task analysis methods (Gordon, 1994)
2.2
Commonly used combinations of task analysis data collection and data
representation methods (Gordon,1994)
3.1
Verbal Protocol Transcript Fragment
3.2
Example of Verbal Data for Analyzing Conceptual Operators
3.3
Example of Verbal Data for Analyzing Knowledge States
4.1
Frequency of Knowledge States encountered
4.2
Frequency of Conceptual Operators encountered
vii
LIST O F FIGURES
Figure
Page
2.1
The stages of memory
2,2
Basic elements of coding scheme
2.3
Detailed description of conceptual operations (Hassebrock and Prietula, 1992)
3.1
Conceptual Framework of the Study
Knowledge States of Tour Guide or Tour Manager
Specific Knowledge States of Hospitality Management in Tourism
Knowledge States of Socio-cultural and Art (SCA)
Knowledge States of Economy
Knowledge States of Eco-tourism
Knowledge States of Education
Knowledge states of tour guide or tour manager
Conceptual Operators of Tour Guide or Tour Manager
Verbal Protocol Coding Scheme for Tour Guide or Tour Manager
Verbal Protocol Coding Scheme for Tour Guide or Tour Manager
Simplified Mental Model for Tour Guide or Tour Manager
LIST O F ABBREVIATIONS
CTA
Cognitive Task Analysis
A1
Artificial Intelligent
ECO
Economy
HMT
Hospitality Management In Tourism
SCA
Socio-cultural and Art
ECT
Eco-tourism
EDU
Education
ANTHRO
Anthropology
CEO
Geography
ACRIC
Agriculture
F&B
Food and Beverage
K
Knowledge
SK
Specific Knowledge
SSK
Sub-specific Knowledge
SSSK
Sub Sub-specific Knowledge
SSSSK
Sub-Sub-Sub-specific Knowledge
BM
Bahasa Malaysia
Ng.
Nanga
LA
Lubok Antu
STB
Sarawak Tourism Board
MOCAT
Ministry of Culture and Tourism
ABSTRACT
Cognitive task analysis delineates how experts use learned skills to perform complex task. It specifies the
knowledge which skilled performers have encoded into schema and mental models, as well as how they
operationalize this knowledge. This study develops a framework adapted from Hassebrock and Prietula
(1992) in conducting cognitive task analysis on the verbal protocols report of the tour guide or tour
manager while carrying the task of tour guiding. The study incorporates five tour guides or tour managers
plying one of the popular tourist routes and destinations in Sarawak. Verbal protocol analysis was done
based on two representations, knowledge states and conceptual operators. The results indicates that there
are seven conceptual operators being used by the subjects to generate and operationalize the five identified
knowledge states while carrying out the tour guiding task. The analysis also indicates that the most
frequently used knowledge states in guiding a tour are knowledge of Socio-cultural and Art and Economy.
While Explanation and Recognition are the most frequently needed operators in executing the task guiding
a tour.
ABSTRAK
Analisa dugas lcognitifmengupas cara mana pakar-pakar menggunakan kemahiran yang diperolehi untuk
melaksanakan tugas yang komplek. Analisa tugas kognitif ini turut menjelaskan pengekodan pengetahuan
kepada skema dun model mental serta bagaimana pengetahuan ini digerakkan oleh seseorang pelaksana
tugas yang berkernahiran. Kajian ini menyingkap kerangka kerja yang disesuaikan daripada kajian
Hassebrock dun Prietula (1992), dalam melaksanakan analisa tugas kognitif ke atas laporan protokol
lisan pemandu pelancongan atau pengurus pelancongan sewaktu menjalankan tugas membawa
pelancong. Lima orang pemandu pelancong terlibat dalam kajian ini melalui laluan dan destinasi
pelancongan yang terkenal di Sarawak. Analisa protokol lisan dibuat berdasarkan dua perwakilan, iaitu
keadaan pengetahuan dun operasi konsep. I-Iasil kajian ini memperlihatkan bahawa adu tujuh operasi
konsep digunakan oleh subjek untuk menjana dan menggerakkan limn keadaan pengetahuan dalam
melaksanakan tugas sebagai pemandu pelancongan. Analysis kujian ini juga menunjukkan hahawa Sociobudaya dan Kesenian, dan Ekonomi merupakan keadann pengetuhuan ynng paling kerap digzlnakan
dalam melaksanakan tugas memandu pelancongan, sementara Penerangan dan Mengenalpasti adalah
dua operator Iconseptual yang sering digunakan dalam menggendalikan tugas memandu pelancongan.
CHAPTER ONE
THE PROBLEM AND ITS SETTING
1.1 Introduction
The travel and tourism industry is of major importance for the world economy today. It is one of
the largest single employers and in many countries it is the largest services exporting sector,
making a significant contribution to the balance of payments. More than 255 million people or
11.l% of total jobs world-wide are occupied in the tourism industry and according to World
Tourism Organization, in 1995 almost 570 million people traveled abroad for tourism reasons
and spent 373 billion US dollars, which accounts for 1.5% of the world GNP (Papatheodorou,
1999).
In Malaysia the importance of tourism development started to gain strength in the 1980s at the
federal level. By mid 1985, the Sarawak State Ministry of Environment and Tourism was
established in line with the Federal policy to accord priority to tourism development. The goal is
to make tourism a major revenue earner in the country. Prior to 1985, the state government
involvement in the tourism industry is channeled through the State Tourist Development
Committee formed in 1976 to assume the planning, coordination and promotion of tourism
development. In recognition of the growing importance of the tourism industry in Sarawak's
economy and its overall development programs, the state government elevated tourism into a
full ministry status and the Ministry of Tourism becomes a reality in 1995 (Rakan Sarawak,
1998). The promotional arm of Ministry of 'Tourism is started with the launching of the Sarawak
Tourism Board in the same year to take over the lead in tourism industry and to work closely
with the Ministry of Tourism in promoting and developing tourism in Sarawak.
As a result of the hard work and endless effort of the Sarawak Tourism Board (STB) there was
an increase of 24% in the number of foreign visitors to Sarawak by 1997, bringing the figure to
more than one million people from Asia, Europe, Africa and the United States (Rakan Sarawak,
1998). Nonetheless the Ministry of Tourism is still emphasizing on improving its services,
through STB, especially on the professional and the quality of the tours manager or guide. Thus
tours manager or guide plays an important role at the grass-root level in the development of
tourism as the job of the tour manager involves primarily rendering service and disseminating
information of interest to his or her clients.
Basically, the job of the tour manager involves planning and conducting tours. Planning and
conducting the tour requires some form of information processing on the part of the tour
manager. Tourists expect value-for-money as they demand for better services and tourism
related products (Poynter, 1993). This requires knowledge and complex cognitive ability in
processing and disseminating information to the tourists. Thus analyzing the concurrent and
retrospective verbalization that the tour manager gives at the time he or she plans or conducts
the tours or trips will give insight on the thinking processes.
1.2 Background of the Problem
Guide or tour managers perform a range of duties. Their roles include conducting and directing
all aspects of the tour, tour company representation, and quality control specialist (Poynter,
1993). This is critical and crucial for the Tour Con~panybecause the duties and jobs are
becoming highly information driven. In Malaysia, all the tourist guides have to pass National
Tourist Guide Examination set by the Ministry of Culture, Art and Tourism (MOCAT, 1999)
before they could get the license to guide a tour. The skilled performance of the tourist guide or
manager could be explored by employing cognitive task analysis (CTA). This method could be
used effectively in the teaching and learning process. Therefore one needs to appreciate and
know the proper way of carrying out CTA.
1.3 Statement of the Problem
Guides and tour managers are entrusted with a heavy responsibility. He is the key person
working for the tour operator (Page and Getz, 1997). His role is to conduct and direct the tour,
and be a representative for the tour operator or travel agency. He has to handle himself with
professional decorum. The tour guide or tour manager also acts in an arbitrator role. When there
is confusion regarding the quality of a service or product to be provided, it is normally expected
that the tour manager will make decisions on behalf of the tour operator and arbitrates any
differences so that tour members receive the quality of service contracted (Poynter, 1993).
Therefore a tour guide requires more than just having the appropriate knowledge and also that
this knowledge must be organized properly to enable its optimal retrieval. Thus, cognitive task
analysis carried out on the verbal protocol of a tour guide, while performing the task, would
enable a better understanding in developing a simple mental models of a tour guide or tour
manager.
1.4 Significant of the Study
A good tour guide or manager is expected to be able to recall and exhibit a broad and in-depth
knowledge about aspects associated with the operation, coordination and conducting of
organized tour group trips. This ability of the guide could enable him or her to provide efficient,
friendly and satisfactory service to his or her clients. The skill performance of this guide could
be reflected in the analyzed concurrent and retrospective verbal protocols.
1.5 Purpose of the Study
The tour guide or tour manager arbitrates any differences, manages and coordinates the tour trip
right from the beginning to the end of the trip. This study would help to understand how to
analyze verbal protocols of a tour guide or tour manager as he or she performs his or her task
efficiently and effectively. Verbal protocol coding scheme for the tour guide would be
developed while analyzing the verbal protocols. Consequently, this verbal protocol-coding
scheme would then be adapted and used to represent a simple mental model of the tour guide or
tour manager. Eventually, the knowledge obtained from analyzing the verbal protocols would
set as an important platform for further research in area of cognitive task analysis of tour
operators.
Foremost, this study strives to analyze verbal protocols, then develop a coding scheme for the
verbal protocol of tour guide or tour manager. Eventually, an appropriate mental model is
developed to represent how the tour guide processes the available information so that he or she
could carry out his or her task properly.
1.6 Objectives
Redding (1995) stated that the goal of cognitive task analysis (CTA) is to delineate the mental
processes and skills needed to perform a task at high proficiency levels. It can be used also to
identify the changes in cognitive structures and process as proficiency develops over time.
Therefore the use of verbal protocol analysis as one of the general methods for carrying out
CTA, can be used to deduce on tasks, goals, knowledge, conceptual operators, lines of
reasoning and strategies used in carrying out unobservable working processes.
Generally this study would explore and develop a simple mental model of a tour guide or tour
manager. The analysis of the concurrent and retrospective verbalizations will specifically be
used to:
(a) Identify the knowledge used by the tour guide or tour manager in carrying out the tour
guiding task.
(b) Identify the conceptual operators used to operationalize this knowledge.
(c) Develop a simple verbal protocol coding scheme based upon the tour guide or tour
manager's knowledge and the conceptual operators needed to process those knowledge.
1.7 Scope and Delimitation of the Study
This study will only focus on how the tour guide or tour manager executes his or her duty
efficiently and effectively. The verbal protocols as concurrent protocols are taken at the moment
the tour guide or tour manager carries out the task of rendering service, processing and
disseminating information of interest to his or her clients. Thus the assumption is the obtained
verbal protocol is the only overall approach the tour guide uses in carrying out the task. If this
were true, then the collected data would give a complete representation and a perfect insight on
the thinking process of a tour guide or tour manager. However, this is difficult to achieve
because not all tasks and actions of the tour guide in executing his task can be recorded verbally.
Therefore the verbal protocols would represent thc verbal communication and thinking aloud
that the tour guide gives while carrying out the task, and also his account of the event he gives
before or after the task was carried out.
The coding schemes of the verbal protocols would be based on two types of protocol
representations: (a) the knowledge state, and (b) conceptual operator. Therefore, the verbal
protocol analysis would only focus on these two representations to understand the way tour
guide or tour manager discharges his or her task.
1.8 Outline of the Study
Chapter two will focus on reviewing literature related to the concepts and frameworks being
used in this study.
The research nlethodology being used in this study will be described in chapter three, followed
by the depiction of appropriate conceptual model with respect to the different stages of the
study. This section will elaborate the procedures of data collection, and the processing and
analyzing of the tour guide verbatim.
Chapter four discusses and gives the result of the analysis of the verbal protocols. Here, the
mental model of tour guide or tour manager will be developed which is based on the conceptual
operators and knowledge states being identified and the coded scheme of the verbal protocol.
Summary of the study, conclusion and suggested recommendations will be done in the final
chapter, chapter five.
Chapter Two
LITERATURE REVIEW
2.1 Introduction
The efficient and effective performance of task depends on the information or understanding
acquired through experience. Likewise, conceptual and skilled knowledge are essential in
discharging a task. Cognitive task analysis represents a transition from emphasis on knowledge
structures to emphasis on the dynamics aspects of mental models (Seamster et al, 1993). Thus
this study would use verbal protocol analysis to identify the knowledge states and operators
being used by the tour guide or tour manager in executing his or her tasks.
2.2 Knowledge
There are at least two senses in which a person can be said to know (McNamara, 1994). One
kind of knowledge can be verbalized, visualized, declared in some manner, and for these
reasons has been called declarative knowledge. A second type of knowledge consists of skills,
cognitive operations, knowledge of how to do things, and has been called procedural
knowledge. Anderson (1983) as cited in Philipchalk and McConnell (1994) assumes that there is
a single, unified memory storage unit that has two major parts call declarative knowledge and
procedural knowledge. The declarative knowledge stores factual information, most of which can
be readily described in words. Procedural knowledge, however is that part of memory which
stores motor skills, strategies for manipulating facts, and knowledge of various rules for
manipulating objects or events. These types of knowledge are essentially important for the
performing of task. Cognitive task analysis is to delineate the mental processes and skills
needed to perform a task at high proficiency levels. The protocol analysis would be used to
identify the knowledge and cognitive operation used by the tour guide or manager in executing
his or her tasks.
Stevensen (1993) mentions that, vast amounts of general knowledge are stored in long-term
memory, knowledge of concepts and knowledge of specific things and events. Systems of
knowledge representation are designed to explain how such knowledge is stored so that it can
quickly and easily be retrieved when needed. Two different kinds of knowledge are represented
in such system, usually called knowing that and knowing how. Knowing that is declarative
knowledge and refers to knowledge of facts, such as knowledge of concepts and knowledge of
individuals and events in the world. Knowing how is procedural knowledge and refers to
knowledge of how to do things, such as multiplying two numbers or driving a car.
2.3 Information Processing Theory
This theory assumes that there is a set of structures for storing information, and a set of
processes for transferring information from one structure to another. We also are not mere
recorders of data, but that we are actively transforming and processing stimulus inputs, much
like a high-speed computer. This approach proposes that information processing occurs at three
different levels (Philipchalk and McConnell, 1994). First, sensory inputs from the environment
are translated, or transduced, into patterns of neural energy by the sensory receptors. The
incoming information is held briefly in sensory registers one for each sensory modality, Initial
processing occurs at this level, and consists primarily of feature detection and pattern
recognition. This is the sensory information stage of input processing. Second, information that
catches your attention is then encoded as it moves to the next stage, which is called short-term
memory (STM) (see figure 2.1). At this point, input would be categorized after comparing it
with information filed away in long-term memory storage. And third, if the input is important, it
would lilcely be rehearsed and then transferred to long-term memory (LTM).
Environment
SENSORY
INFORMATION
STORAGE
Unattended
information
is lost
Attended
information
added to
TERM
LONG-TERM
MEMORY
Figure 2. I : The stages of memory (Philipchalk and ILlcConnell, 1994)
2.3.1
Sensory Information Storage
This is the first stage of memory storage. The sensory receptors hold a more-or-less exact copy
of an input for a fraction of a second before that copy fades or is erased by the next input. Under
normal conditions as he or she loolts from one object to another in his or her visual world, his or
her visual system would hold on to each stimulus pattern for but a fraction of a second before
yet another visual input replaces the previous pattern. As mentioned by Lindsay and Norman
(1977) in Philipchalk and McConnell (1994), this storage time typically lasts from 0.1 to 0.5
seconds under normal viewing conditions. However the storage lasts longer if no new stimulus
erases the input. George Sperling (1963) in Philipchallc and McConne11 (1994) refers to visual
Sensory Information Storage (SIS) as iconic memory. For the amount of information stored in
visual SIS is much like that in an icon, which is a detailed image of a person or event. In similar
fashion, auditory SIS is sometimes called echoic memory, because it seems to be an exact echo
of an auditory input. However, by the time this sensory information reaches our cortex and we
become aware of what we are loolcing at, much of the rich detail is lost.
2.3.2 Short-term Memory
Most people must rehearse a telephone number continuously until he or she dials it. This fact
indicates that the length of time an item typically remains in short-term memory. Short-term
memory (STM) refers to as temporary information storage between the SIS and long-term
memory (LTM). STM has a limited capacity of about seven items. That is the STM ordinarily
cannot store more than about seven items simultaneously as mentioned by Miller (1956) cited in
Philipchalk and McConnell (1994). At the moment that the brain inserts an item into STM, that
item is strong and clear and easy to recall, if he or she does so immediately. But shortly
thereafter, the brain tucks away a second item, then a third item, and a fourth. Although only a
few seconds have passed, the brain will now have much more trouble recalling what the first
item was.
By the time the brain has passed five or six new items down on top of the first, that original item
has lost most of it strength and has faded away. The new items appear to interfere with or erase
the earlier items, just as each new visual pattern he or she looks at wipes clean the stimulus he
or she was looking at just a moment before. While he or she is holding an item in STM, he or
she can recall it more or less at will. And he or she can keep the item around for several minutes
by rehearsing it. Rehearsal refers to the act of silently saying things over and over again. When
a person rehearse an item, that person actually keeps putting it back into temporary storage
again and again. By rehearsing the item in STM several times, he or she can keep it available for
a period of several minutes. However, once the item drops out of this temporary storage, it is
likely to be gone forever, unless some more or less permanent record of the stimulus input has
been made in long-term memory (Philipchalk and McConnell, 1994).
2.3.3
Long-term Memory
If a person rehearses an item for a long enough period of time, we might well not forget it all.
This fact indicates how items get shifted from short-term memory to long-term memory. LTM
is one's store of permanent memories. Inputs important enough to survive his or her short-term
memory are transferred to long-term storage. Salient or critical features of an input are stored
according to various categories. LTM seems to be much more complex. It stores many different
aspects of ones' experiences. First, rather than merely recording total inputs as they are
received, LTM abstracts certain critical features. It then files these meaningful features in an
appropriate memory stores. When a person draws on his or her long-term memory, it then
reconstructs the event from the associations and auditory representations it has in it files.
Ebbinghaus (1 885, 1964) as cited in Philipchalk and McConnell (1994) suggested that people
typically does not remember complex events. Rather, he or she recalls a few high points and
then reconstruct the experience piece by little piece.
Thus, important inputs move from STM into LTM, which is the permanent memory bank of the
brain. Recall from LTM involves remembering significant facts and then reconstructing a
memory by filling in plausible details.
2.4 Cognitive Task Analysis
Redding (1995) mentioned three aspects relating to cognitive task analysis (CTA). First, CTA
uses interviewing, modeling techniques and experimental procedures to determine cognitive
processes underlying skilled job performance. This involves determining how a task is learned,
how to expedite learning, optimal job knowledge, and the required job performance skills.
Second, CTA provides data on the cognitive skills important to the job. Identifies effective
problem-solving and decision making techniques for the job, superior methods for organizing
job knowledge, and efficient ways of allocating attention to the various job tasks.
Third, CTA analyzes decision making and mental models, learning and skill development, the
interrelationships among job concepts and task elements, and group performance as well as
individual differences. Most CTA include the identification of key job components,
identification of the knowledge and skills required for similar job components, and the
identification of the conditions which best facilitate progression to the next level of expertise.
Simon and Kaplan (1989) mentioned that cognitive science has a wide and varied pool of
methods of cognitive research, ranging from computer simulation to naturalistic observation,
from recording the electrical impulses of the brain to performing linguistic analyses, from using
nonmonotonic logic to collecting reaction times or verbal protocols. The following methods are
appropriate for performing a CTA:
a) Structured interviews
b) Verbal protocol analysis
c) Critical decision method
d) Conceptual graph analysis
e) Means and Gott's cognitive task analysis
Gordon (1994) generalized these methods to develop mental models as part of the CTA. The
task analysis methods can be divided into three categories as follows:
a) General methods for data collection
b) General methods for data representation
c) Specific task analysis methods
The first two categories are generic methodologies and the third is specific, commonly used
methods, see Table 2.1 and Table 2.2 below.
General Methods
for Data Collection
Document
and
equipment analysis
Unstructured
interview
Structured interview
Specific Task Analysis
Methods Position Analysis
Questionnaire (PAQ)
Delphi and Focus groups
Controls
and
Displays
Analysis
Hierarchical Task Analysis
Extended Task Analysis
Procedure (ETAP)
The GOMS model
Critical Incident Technique
Group interviews
Sorting and rating
Questionnaires
Verbal
Protocol
Analysis
Observation
General Methods for Data
Representation
List and outline
Matrix (cross-tabulation table)
Structural network
Hierarchical network
Flow Chart
Timeline chart
Functional Analysis System
Technique (FAST)
Cognitive Task Analysis
(Mean & Gott)
Conceptual Graph Analysis
Associative
scaling
algorithms
Activity Sampling
Functional Flow Diagrams
Table 2.1: Taxonomy of general and specific task analysis methods (Gordon, 1994).
Data Collection
Document and
Equipment
Analysis
Unstructured
Interview
List and
outline
*
*
Data Representation Methods
Matrix Structural Hierarchical
Network
Network
*
*
*
*
*
Flow
Chart
*
Timeline
Chart
*
Table 2.2: Commonly used combinations of task analysis data collection and data representation
methods (Gordon, 1994)
From the above Table 2.2, notice that if verbal protocol was the method for data collection, then
the mode for data representation could be in the form of either list and outline, hierarchical
network, and or the flow chart.
2.5 Verbal Protocol Analysis
Protocol analysis, the use of verbal reports as data, has become an increasingly important
technique for studying human intelligence. Its importance stems from the fact that it is one of
the few methods in cognitive science that gathers data with sufficient temporal density to test
models that account for behavior nearly second by second (but not millisecond by millisecond),
(Simon and Kaplan, 1989). Protocol analysis treats verbal report as a source of data to be
accounted for with an experimenter-generated theory, perhaps in the form of computer
simulation. They further mentioned that there are many ways of getting subjects to generate
verbal reports, including questioning them, asking them to report on their mental processes, and
asking them to talk or think aloud. Verbal reports of the talk or think-aloud variety tend to be
the least intrusive and to provide the most accurate records of the nature and sequence of
subjects’ mental processes.
According to Singleton (1989) an operator carrying out a task can be asked to talk about what
he is doing as he does it. Clearly this is useful for tasks which involve extensive scanning of
information sources and deliberations about what action to take. The visible action may
apparently be trivial - adjusting one or two of very many available controls - or may even be
non-existent. Because much of the activity is in searching for information and processing it
leading to a decision, the observed output is obviously a poor indicator of total activity and his
verbal commentary is much better guide to his task performance. Such a commentary can be
recorded or notes can be taken directly by the analyst.
Hassebrock and Prietula (1992) mentioned that protocol analysis methodology is derived from a
cognitive psychological or information processing framework for studying human behavior. In
this theoretical approach, psychologists have stressed the importance of understanding the
cognitive processes (e.g. attention, perception-and memory operations) and knowledge (content,
representation and organization) that underlie complex problem solutions, decisions or
judgements. Thus, the analysis of verbal protocols is one methodology designed to reveal the
knowledge and cognitive processes used by a person while performing a task or behavior.
Redding (1995) stressed that protocol analysis is particularly useful for investigating problemsolving methods and mental models. Most applications of protocol analysis view problem as a
search through a problem space accumulating knowledge about the problem situation, and
applying operators to the knowledge to reach each new point in the space until a solution is
reached.
Medin and Ross (1992) explained that verbal protocols have provided a great deal of insight
into the problem-solving processes and can often be followed up by more traditional dependent
measures. Although there are a number of possible ways to collect data, the most common is to
ask the subjects to think aloud as they go about solving the problems. That is they are asked to
say whatever they are thinking about, but not to embellish or explain it for the experimenter’s
benefit. The idea behind this measure is straightforward that is, people’s thoughts provide
further information about the course of their problem solving. Although thinking-aloud
protocols may seem intrusive, the use of such protocols usually appears to have no effect on the
solution of the problem other than slowing down the performance.
9
Payne et al. (1998) revealed that protocol analysis is one approach to gathering detailed processtracing data on decision making. To use this approach, the subject is asked to think out aloud as
he or she is actually performing the task of interest, such as choosing among several
alternatives. Such a verbal record is called a protocol. Protocols differ from introspection or
retrospective reports about decision processes because the subject is asked to verbalize thoughts
as they occur in the course of making a decision. The protocol data are then analyzed to attempt
to gain insights into the subject’s decision processes. The major advantage of protocol collection
and analysis is that a great deal of data on internal events is made available.
However, Tolbert and Bittner (1991) as cited in Gordon (1994) mentioned that there are three
types of verbal protocols, such as concurrent, retrospective and prospective. In a concurrent
protocol, the worker or expert is asked to perform one or more task and verbalize both the
knowledge and strategies responsible for his or her task activities. Sometimes they are given
specific focal points such as identification of stimuli or triggers that are important or responsible
for their responses, decision strategies, or why they made a certain response.
Sometimes concurrent protocols are difficult to collect, especially if they occur quickly or are
too cognitively demanding (Benjafield, 1992). In such a case, retrospective protocol can be
collected, in which the subject is asked about cognitive processes that occurred at an earlier
point in time (Benjafield, 1992). He further mentioned that concurrent verbalization relies on
short-term memory, whereas retrospective verbalization relies on long-term memory.
In the case of a prospective verbal protocol, a scenario is given for the subject to work on or that
he is to imagine performing the task. The advantage of this method is that the subject would not
be exposed to a situation that is difficult, dangerous, or the task is too costly or time-consuming
to perform. The main drawback, however, is that the verbalized data may quantitatively be
different since there is lack of real tasking cues or triggers to instantiate the mind of the subject
(Peskin, 1998).
Although protocol analysis often allows the researcher to gain important insights into decision
making, there are some disadvantages. Collecting protocol data in quantity is extremely timeconsuming, so small samples of subjects have typically been used (Payne et al., 1998). In
addition, protocol data may not be entirely reflective of subjects’ decision processes. The
protocols may reflect subjects’ biases or may be censored by subjects while they are being
reported. In addition, subjects may be unable to verbalize retrospectively some internal
processes (Nisbett and Wilson, 1977) as cited in Payne et al. (1998). Finally, protocols may not
provide insights into all of the processing performed, and there may not be output corresponding
to all internal states. Subjects may select aspects of processing to verbalize based upon what
they believe is important and may not verbalize those data most valuable to the researcher
Payne et al. (1998).
Although such problems with selectivity in verbal reporting may exist, several researchers have
argued and have provided convincing evidence that decision makers do have self-insight,
(Ericsson and Simon, 1993). They further argued that many studies showing no effects of taking
protocols on decision processes. In studies of decision making, however, the results have been
more mixed. Although Smead et al. (198 1) and Biehal and Chakravarti (1993) as cited in Payne
et al. (1998) reported no significant differences between protocol and no-protocol conditions,
Biehal and Chakravarti (1993) found differences in the extent of alternative-based processing
and problem framing due to verbal protocols, Therefore, although verbal protocols can provide
10
invaluable data on choice processes, one must be very careful to control for any effects of taking
protocols.
However Gordon (1994) expressed that there are certain advantages of verbal protocol as a
method for data collection, as follows:
(a) It is more likely to elicit task-relevant information than having the person respond to
interview questions without any task cues.
(b) It is relatively easy and inexpensive to implement.
(c) It tends to provide information of two types, declarative conceptual knowledge and explicit
rule knowledge for how to perform subtasks, both of which are important for complete task
analysis.
(d) It has face validity.
According to Simon and Kaplan (1989) verbal protocols generally provide explicit information
about the knowledge and information heeded in solving a problem rather than about the
processes used. Thus it is usually necessary to infer the processes from the verbal reports of
information heeded instead of attempting to code processes directly. The talk aloud procedures
being used to study human information processes while performing the task would entail the
recorded protocols, These protocols are then segmented and encoded and the basis of
segmentation would depend on the type of encoding and analysis. Encoding refers to the
process of converting the content of segments into a formal language or as Philipchalk and
McConnell (1994) put it, converting into a form that can be entered into memory. The nature of
task and the hypotheses involved would determine what to encode.
In a research done by Hassebrock and Prietula (1992), “A protocol-based coding scheme for the
analysis of medical reasoning”, the audio recording of a case was transcribed for analysis and
interpretation. The transcript includes patient data statement follows by a new paragraph for the
physician’s statements while reading the data card aloud and another new paragraph containing
the subsequent think-aloud statements. The transcript was segmented by identifying each
segment with a knowledge state, which was in turn associated to one of the possible basic
conceptual operations in the coding scheme of the verbal protocol. The following excerpt is an
example of transcript fragment and coding used Hassebrock and Prietula (1992):
Excerpt 1
Example of and coding
21.
(1)
Ah-ascending aorta is slightly enlarged,
Ah, well that means post-stenotic dilation
or increased ah, blood flow through
it. (2)It would be consistent with ah, ah, transposition of the great vessels with sub-aortic
stances, (3) uh, tetralogy of Fallot would be ah, or a tetrad variant.
1.1 Hypothesis-generation: triggers 3 (post-stenotic dilation; increased blood flow
through aorta)
21.2 Hypothesis generation: association j (TGV,+& sUb,#)
21.3 Hypothesis-evaluation: confirmation 3 (TF; Tetrad variant)
Z’he fzrst numbered paragraph typed in italics represents the physician
reading the card aloud (thus indicating the information available. The
verbalized thinking-aloud responses follow in a new paragraph with the
separate segments numbered and the relevant knowledge states identiJied. The
coding interpretation of the think-aloud response follows with the cue-number
11
(to the left of the decimal point) andsegment-number (to the right o f the decimal
point) noted
For each segment, and to the left of the arrow, there is listed the basic
conceptual operation and the more specific operators used, ranging from the
most general (leftmost) to the most specific form. The knowledge state or states
in each segment are shown parentheses to the right of the arrow. (Hassebrock
and Prietula, 1992, p. 633).
In this study, Hassebrock and Prietula (1992) developed a coding scheme to provide an explicit
framework for documentation of behavioral events and a guide for structuring the analysis. This
coding scheme depends largely on the nature of the task and the theoretical constructs of interest
or a theory of reasoning in the domain.
2.6 Protocol-Based Coding Scheme
This section will focus on the coding scheme based on Hassebrock and Prietula (1992), that they
adopted from Newell and Simon (1 972). This scheme, being used to code topic representations,
suggested that protocols consist of two basic types of semantic elements. Firstly, a given topic
segment consists of knowledge states which are units of information or knowledge retrieved
from memory used by an individual. The second type is, these knowledge states are associated
with conceptual operations which are inferential processes that produce new (or modify
existing) knowledge states. Thus the goal of the analysis is to trace the sequence and patterning
of knowledge states and conceptual operations used in response to a problem-solving task.
However Hessebrock and Prietula included a third element in the coding scheme, that was, a
means of representing physician's use of more global diagnostic strategies and problem solving
representations known as lines of reasoning. A line of reasoning can be viewed as a problemspecific representation of, in this case, physician's medical problem-solving knowledge. Figure
2.2 summarizes the three elements incorporated in the coding scheme, knowledge states,
conceptual operations, and lines of reasoning.
/
Coding
scheme
/
Conceptual
operations
Lines of
reasoning
Data examination
Data exploration
Data explanation
Hypothesis generation
Hypothesis evaluation
Discrepancy processing
Meta-reasoning
Summarization
Figure 2.2: Basic elements of coding scheme (Hassebrock and Prietula, 1992)
Figure 2.2 depicts the three components of the coding scheme. First the knowledge states that
identifies units of medical knowledge used by the physician in diagnostic reasoning and the
occasion at which they were brought into consideration. Data interpretations and diagnostic
hypothesis are the two general categories of knowledge states being represented in this analysis.
Another representation in the scheme identifies the conceptual operations being used to generate
or instantiate states of medical knowledge. Newel1 and Simon (1972) as cited in Hassebrock and
Prietula (1992) referred this to as an inferred cognitive process which modifies (adds,
eliminates) existing or currently active knowledge states and produces new, active knowledge
states (Hassebrock and Prietula, 1992). In response to a specific data cue, a given segment of
verbal protocol will constitute (a) one or more knowledge states (interpretations or hypotheses)
and (b) a conceptual operation that produces the associated knowledge state or states as shown
in figure 2.2.
Hassebrock and Prietula (1992) identified eight basic types of conceptual operations to
characterize distinct segments of a physician's problem-solving behavior. Each will then be
further analyzed in order to provide a more detailed representation of reasoning behavior. The
conceptual operations are analogous to goal-directed medical problem-solving behaviors found
in physicians' diagnostic reasoning.
For instance, data examination, the physician's goal in this problem solving episode is to select
and examine cues and make initial interpretations of the findings in the patient data of a case.
Whereas in data explanation, the goal is to interpret the significance or role of a given data cue
by explaining the underlying pathophysiological basis of a disease. Often, this type of data
explanation is not used to generate a particular hypothesis, but is used to account for the natural
history, which could have led to the development of data finding. Further example is,
hypothesis-generation where the goal is to generate diagnostic hypotheses. Specific operators
provide different means of generating hypotheses in response to a particular data cue, pattern of
cues or from other diagnostic hypotheses. Whereas hypothesis-evaluation is to evaluate a
diagnostic hypothesis by interpreting its ability to account for the presence/absence of data cues,
data interpretations, or its relationship to other diagnostic hypotheses. Hypothesis evaluations
usually assume either a confirmatory or discomfirmatory value. See Figure 2.3 for more detail
description of conceptual operations in the analysis of medical reasoning by Hassebrock and
Prietula (1992).
The third type of protocol representation in the scheme is the lines of reasoning. Hassebrock and
Prietula (1992) described it as a reasoning artifact representing a complex set of related
pathways of thought bound together and defined by linked pathophysiologies and other medical
knowledge evolving in the context of a particular problem instance. In their study, the
identification of a line of reasoning is an attempt to capture the salient aspects of reasoning that
provides coherence and comprehension in the face of information complexity and uncertainty.
Many forms of conceptual and mental models have been proposed for explaining cognitive
activities, including comprehension and problem solving. However, Hassebrock and Prietula
(1992) have developed the line of reasoning construct within the context of a specific domain of
knowledge so that the cognitive properties of a mental model are dependent upon information
constraints associated with the task of medical diagnosis.