Mental models of incidental human-machine interaction
PROJECT REPORT IN THE PERIOD MAY 1- DECEMBER 31
2000
Project-number: MMI99010
Project title: Mental models of incidental human-machine interaction
Research theme: User-centered design
Project dates: May 1, 2000 - April 30, 2004
Project leader: Dr. Gerrit C. van der Veer, Vrije Universiteit, gerrit@acm.org
Researcher Dr. Maria del Carmen Puerta Melguizo mcarmen@cs.vu.nl
Project partner: Dr. Paul van der Vet, Universiteit Twente, vet@cs.utwente.nl
Project partner: Dr. Herre van Oostendorp, Universiteit Utrecht, herre@cs.uu.nl
Introduction
The main goal of this project is to investigate the relationships between characteristics of
design and characteristics of knowledge and understanding of prospective users in order to
develop design guidelines for envisioning (future) systems and systems intended for incidental use.
To design systems for incidental use implies to answer several research questions:
a) What are the relevant aspects of mental models for incidental human-machine
interactions.
b) What characteristics of these mental models may guide adequate design.
c) What is the conceptual framework for investigating mental models.
d) And after developing valid techniques for assessing mental models, how can these
techniques be applied in the design process.
SUMMARY
Starting the first of May 2000, several activities have been performed to respond these
questions. The first thing we did was to try to understand the concept of mental model. We found
1
Mental models of incidental human-machine interaction
that there is not agreement between definitions. Currently, and in collaboration with different
experts in Europe, we are trying to define properly what are and how to study mental models and
other relevant and related concepts. Another relevant aspect of our research is to explore what is
the role that mental models can play during the design process and how to measure them. After
presenting our conceptual framework, we will show some relevant studies and our future research
plans.
1. The Definition(s) of mental models
Following the previous schema, our first goal was to search in the literature what has been
done about mental models, how to “measure” them, and how to apply them in design practice.
The compilation of our search resulted in a document were we show some of the most
relevant contributions in the study of mental models (See Appendix A: “Mental Models: The state
of art”) and its relationship with the design process (See Appendix B: “The need of a conceptual
framework”. Our search showed that there is absence of consensus about the definition of the term
and some definitions of the concept are even contradictory.
Directly related to this activity, we are currently writing a chapter on mental models for
the book Handbook of Human-Computer Interaction (tentative title). Edited by Julie A. Jacko and
Andrew Sears and to be published by Lawrence Erlbaum & Associates. The primary objective of
this Handbook is to compile a comprehensive set of articles that address the principles involved in
conceptualizing, designing, and evaluating computing technologies spanning a variety of traditional
and non traditional platforms including desktop computing, networked environments, mobile
computing and virtual environments. Given both the research-oriented and applied foci, the
handbook will summarize research, technological advancements and specific methodologies in the
field of human-computer interaction.
2
Mental models of incidental human-machine interaction
2. Let’s agree about what is mental model: A JOINT EUROPEAN
EFFORT
The main problem while studying mental models is the absence of agreement between
authors about what is a mental model. Many researchers in different universities have addressed
this problem. We contacted with some of them and currently we are trying to do a joint European
effort to define the concept of mental model. In this task relevant cognitive ergonomists like José
Cañas from the University of Granada (Spain), Cristina Chisalita from the Babes-Bolyai University
(Rumania) and Herre van Oostendorp from the Utrecht University (The Netherlands) are actively
collaborating with us.
As a result of these collaborations several things have been done:
2.1. Collaboration with Herre van Oostendorp
The meetings with him have been focused in the study of different tools to measure mental
models and how to approach the main goal of our project: “to investigate the user’s mental model
in order to develop design guidelines for systems intended for incidental use”.
2.2. Collaboration with José Cañas
During November, Gerrit van der Veer was in Spain visiting different departments of HCI
and Cognitive Psychology. During this time he contacted José Cañas in the University of Granada
and discussed the definition of the concept and the theoretical model.
Currently, we are writting a common article and preparing a paper to present the results
of our view at Icom-3 (The International Congress on memory) in Valencia (Spain) in July 2001 in a
Symposium on “Memory issues in Cognitive Ergonomics”.
2.3. Collaboration with Cristina Chisalita
Recently, M. C. Puerta Melguizo went to Rumania to visit Cristina Chisalita. The
purpose of this visit was to participate in a workshop about UID and the role of mental models (4-8
December, 2000).
3
Mental models of incidental human-machine interaction
In addition, we are collaborating together in the comparison of several methods to study
mental models. The main goal we have is to explore the complementary information that the
different methods can offer us to measure mental models. At the moment we are analyzing a set of
data obtained using both, pathfinder and teach-back procedures.
2.4. Collaboration with Tom van Engers
Tom van Engers works at the Dutch Tax and Customs Administration (The Netherlands).
Currently he is interested in the goal of the shared mental model in a team in order to improve the
communication and collaboration between different experts while designing public administration
systems.
In June Mari Carmen and Tom presented together a talk: “Mental Models and
knowledge productivity: Why there isn’t a royal way to wisdom in the seminar “Improving
knowledge productivity” (Amersfort, 16th June 2000). Another important result of this collaboration
was the experiment we did during this seminar. For a detailed explanation see:
Van Engers, T. Vork, L. & Puerta-Melguizo, M. C. (2000). Kennisproductiviteit in groepen.
Human Resource Development. Thema:Stimuleren van kennisproductiviteit, 2, 76-82.
3. The role of mental model in design: envisioning the future
Following the pragmatic approach of Van der Veer (1990), to understand the role that
mental models play in design it is necessary to specify a design conceptual framework. The
conceptual framework includes and defines clearly the relevant concepts and the design process
itself (see Appendix A).
3.1. Specifying the conceptual design framework
We are using the DUTCH (Designing for Users and Tasks from Concepts to Handles)
design conceptual framework (Van der Veer & Van Welie, 2000). Figure 1 gives an overview of the
whole design process with all the activities and sources of information.
4
Mental models of incidental human-machine interaction
Figure 1. The DUTCH design process.
See Appendix B: “The need of a conceptual framework” for a complete explanation of the
conceptual framework.
3.2. Our hypothesis: Envisioning the future
Our hypothesis is that, in order to improve the interaction between the system and the
user, designers should concern about the User’s Virtual Machine (UVM). That is, the aspects of the
system directly relevant for a specific group of users. And, following the pragmatic approach, we
think that an appropriate mental model has to be the mental representation of the UVM that the
user builds through its interaction with the system image.
UVM  Mental Model  Design
Figure 2. The mental model of the UVM.
5
Mental models of incidental human-machine interaction
And in order to design successful systems, it is important to know what future users would
understand about our envisioning system. In other words, designers need to know if the envisioned
UVM is able to produce an appropriate user’s mental model of the system. And with this goal,
designers should assess the mental models that future users will develop when confronted with a
suitable representation of the envisioning of a new system. So the insight in mental models of
future systems will bring us two types of design knowledge:
1. How will users understand the envisioning and what may be misunderstood. This will
lead to ideas on how to represent the future technology for the user, how to introduce; as
well as how to represent the different “utilities” of the system.
2. What will users expect from the envisioned systems. Future users will come with ideas
about functionality that is intuitive for them, in a way contributing to the design of the
functionality.
We just submitted a paper at the 8th European Conference on Cognitive Science
Approaches to Process Control: CSAPC '01 (see Appendix C: “Assessment of mental models for
envisioning design”). In this paper we present our ideas and the first study assessing mental
models.
4. The starting point and the future
4.1. Working with previous findings in Design
Based in previous work (e.g. Mulder, 2000; Van der Veer, 1990) we suggest to apply
envisioning scenarios and analyze mental representations early in the process. More explicitly,
designers have to consider the different UVM’s or relevant aspects of the system that are relevant
for the different groups of potential users of the system. And as long as envisioning systems, it is
important to evaluate, in an iterative process (see figure 1), all of the possible features that can be
a result of the interaction with the new system. In doing so, it is important to use scenarios and to
6
Mental models of incidental human-machine interaction
explore if the mental representations users build through their interaction with the UVM are the
ones that allow an appropriate interaction in order to reach the desired goals.
As a starting point we are working with some relevant master thesis that in the last years
have been supervised by Dr. Van de Veer. The most representative examples are:
- The master thesis of work of Kok, E. (1998). Mental Models and Individual Differences. A
research method applied to the domain of Physics.
- The master thesis of Klok, J. A. (1998). A needle in a haystack. The design of an
interface that guides the user through the complex conceptual space of an informational retrieval
system.
In addition Dr. Van der Veer and Dr. Puerta Melguizo supervised recently another thesis
were mental models were assessed in order to create adequate guidelines for design:
- Mulder, B. (2000). The role of mental models in designing computer systems.
A short abstract of these master theses can be read in appendix B.
And in collaboration with Chisalita, some of the data extracted from Mulder’s master
thesis are currently being reviewed from a new perspective. Although he did not analyze the
complete set of data, Mulder used both pathfinder and teach-back methods in order to extract
information about the mental model. We are trying to combine the information extracted from both
sets of data to have a clearer insight of the user’s mental model of envisioning systems.
4.2. Measuring mental models
Following the proposal of Cañas and Antolí (1998) we consider that the mental model is a
dynamic representation created in Working Memory (WM) by combining information stored in
Long-Term Memory (LTM) and characteristics extracted from the environment. The knowledge
from LTM is instantiated into a mental model in WM based on triggering events from the situation.
And, as Cañas and Antolí proved, the specific information extracted from LTM depends of the
context and of the task demands. On the other hand the knowledge can be both conceptual and
procedural.
7
Mental models of incidental human-machine interaction
The next question is how to explore those mental models. Although both techniques have
limitations, we think that a combination of pathfinder and teach-back can give complementary
information. Another method we are considering to use is the one proposed by Herre van
Oostendorp and based in the studies of Kintch about discourse comprehension (see van
Oostendorp and Goldman (1999). The construction of mental representations during reading)
4.2.1. Pathfinder
Pathfinder is a graph theoretic technique that derives network structures from proximity
data. In the network concepts are represented as nodes and relations between concepts are
represented as links between the nodes. A weight corresponding to the strength of the relationship
between two nodes is associated with each link. Concepts can be directly linked or not. The
algorithm searches through the nodes to find the closest indirect path between concepts. A link
remains in the network only if it is a minimum length path between the two concepts.
As described by Van Engers (2000), the method consists of several steps:
1. To elicit concepts and create a concept list.
2. To present the concept list to the subjects to be assessed in the form of a pair-wise
comparison test or as a sorting task.
3. To create graphs that represent the concepts and relationships.
4. To calculate the characteristics of these graphs.
4.2.2. Teach-back
The technique was developed by Pask in the framework of the “conversation theory” (Pask
& Scott, 1972). Later, teach-back has been extended by Van der Veer (Van der veer, 1990) as a
hermeneutic method to provoke the user to externalise his/her mental model or representation(s) of
the system. Using this method, participants are asked (individually) to teach to an imaginary
“colleague”, or person with some relevant role, how to solve the problem stated in the teach-back
question. To respond the teach-back question participants can write, make diagrams, drawings
(video can be used as well)… and in this way, subjects are encouraged to externalise their
knowledge about the system. The answered protocols that result form the teach-back questions
8
Mental models of incidental human-machine interaction
are scored along scoring categories in order to get insight in the representation of the problem
space. The function of the categories is to explore the mental models of the participants on
characteristics that are relevant for the research questions of the specific study. Among other
utilities, teach-back method is suitable for detecting individual differences in mental representations
(Kok, 1997; Van der Veer, 1990).
Very interesting research for our purposes is done made by Mulder (2000). Mulder used
the teach-back method in order to explore the mental models participants created after being
confronted to envisioned new information systems using scenarios. He used two kind of teachback questions:
- “what is…”. The main goal of this kind of question was to reveal insight in the
conceptual and semantic knowledge the participant has about the system.
- “how to…”. The goal of this question is to explore the knowledge the procedural
knowledge the user has about the system.
From the teach-back protocols Mulder obtained a set of suitable categories relevant for
designers: “interpretation of functionality in relation to goal”, “affective reactions”, “lack of
recognition of new functionality”, “functionality assumptions”, “dialogue assumptions”, and
“implementation assumptions”. And using these categories, he scored the teach-back protocols of
a group of 30 subjects. In the next phase of his study, the scored data were shown to expert
designers. The most relevant result of his study was the fact that designers found that the
information extracted from the future user, the mental model, could be very useful during the
design process; especially in an early stage of the design.
Teach-back is a technique that focuses on instantiated knowledge (Van der Veer, 1990:
Van der Veer et al., 2000). This means that the type of representation it focuses on may differ
dependent on the instruction. As a matter of facts, the “what is” question seems to focus on the
conceptual knowledge that is relevant to the situation while the “how to” focus on the procedures.
Our hypothesis is that using this questions we will extract the relevant concepts (both semantic and
procedural) that are relevant for the specific situation and that can be later used to analyze the way
they are organized (in this specific situation as well) using the pathfinder.
9
Mental models of incidental human-machine interaction
The paper submitted to CSAPC ’01 pretends to explore such ideas.
4.3. Working with scenarios
Using our conceptual framework we think that in order to improve the design process it is
important to begin to explore the mental model early in the design process; while modeling task
model 2 (TM2). And as long as it is not yet specified in any detail, the only available representation
for modeling TM2 is the use of scenarios. Based on Carroll (1995), a scenario is “a projection of a
concrete description of the activity that the user engages when performing a specific task, a
description sufficiently detailed that the design implications can be inferred and reasoned about”.
Scenarios have been used in the design of future technology in a form that shows an individual
instance of use, in a well-described individual situation. On the other hand they have the advantage
that the details of technology can be left unspecified. The scenario may be formatted as a written
story, as an acted out detailed script (taped on video), as a global script to be acted out by
prospective users, with the help of mock-ups of potential technology, etc. Scenarios can be used
both during the development of TM2 and the evaluation of detailed specifications.
4.4. Working in a design team
The most important goal for the near future, i. e. 2001, is to be a part of one or more real
design team and to explore the role in mental models in a real design project. As a matter of facts,
our research does not make to much sense if we do not know what is going on in a real design
project. This activity will lead us to a better understanding of the design process in practice and,
consequently how to apply the information of the user’s mental models. We want to explore the
best way to communicate and to apply the knowledge extracted from the user’s mental model, the
way future users understand TM2, to improve the design of new systems and systems intended for
incidental use. And to reach this goal we hope that the IOPMMI board can help us contacting the
right type of industrial project as soon as possible.
Apart from this source, we are going to collaborate with Paul van der Vet in exploring
industrial design situations.
10
Mental models of incidental human-machine interaction
APPENDIX A: MENTAL MODELS: THE STATE OF ART
Increasingly people use information technology and complex interactive systems, both in
work situations and for leisure activities. Information technology is embedded in telephones, TV
sets and home computers. Apart from this, public services are growing: electronic counters, ATM
combined with electronic shop, etc. Users of these systems are not "professional" users and,
sometimes, the frequency of use will be low. As a consequence, users normally show low
motivation for practice, for reading directions, for formal training (which anyhow would not be
feasible in most cases). Users, however, need to understand the functionality of the device, the
relation of this to their task, and the dialog for applying the device in order to be able to apply the
system. As a conclusion, design methods for complex but infrequent human-machine interaction
need to focus (among other things) on enabling the development of an "instant" mental model that
allows useful interaction.
As a matter of facts, different findings in Human-Computer Interaction (HCI) and
Psychology show the importance of the use of information from mental models as a basis for
designing systems. Interest of mental models from the HCI perspective is based on the idea that,
by discovering what users know and how they reason about the systems function, it may be
possible to predict learning time, likely errors and the relative ease with which users can perform
their tasks (Preece, 1994).
THE DEFINITION(S) OF MENTAL MODEL
The question of how human beings represent information mentally and how they use it to
interact with the world in adaptive ways is widely investigated for researchers in Philosophy,
Cognitive Psychology and Cognitive Science (Sasse, 1997). As a result, different definitions of the
concept have been made. In this review we will refer to some of the more important ones.
On the other hand although there is no agreement about the exact definition of the
concept, in general, “mental model” refers to the internal representations that people form of the
11
Mental models of incidental human-machine interaction
environment through their interaction with it. This mental model can be different as a function of the
task domain, the previous knowledge people have, and the way they associate it with actual needs
and incoming information from the environment.
1. Craik (1943)
The idea that people rely on mental models can be traced back to Kenneth Craik’s
suggestion in 1943. According to this author, mental models are representations in the mind of real
or imaginary situations and can be constructed from perception, imagination, or from the
comprehension of the discourse. Mental models underlie visual images, but they can also be
abstract, representing situations that cannot be visualised (Johnson-Laird and Byrne, 2000).
The most important characteristic of these representations is that they are "small-scale
models" of the situation and the possible actions. With this representation in mind, we are “able to
try out various alternatives, conclude which is the best of them, react to future situations before
they arise, utilize the knowledge of past events in dealing with the present and future, and in very
way to react in a much fuller, safer, and more competent manner to emergencies which face it”
(Craik, 1943).
Although it gives a clear insight about the concept, this definition does not make explicit
the form of these representations or the processes implicated in the construction of such
representations (Chisăliţă, in press).
2. Johnson-Laird (1983)
Studying the interaction between humans and the world, one of the most influential
theories is Johnson-Laird’s (1983) theory of mental models. The theory seeks to provide a general
explanation of human thought, reasoning processes and language comprehension. Instead of the
traditional symbolic logic theories, he proposed the theory of mental models to explain reasoning
and emphasised the need of consider the semantic content of the representations and not only the
syntax (Chisăliţă, in press).
12
Mental models of incidental human-machine interaction
According to this theory, when we interact with the world, three cognitive processes are
involved:
a. The translation of an external process into an internal symbolic representation.
b. The derivation of new symbols from existing ones by an inference process.
c. The translation of these symbols into actions and predictions of external events.
In other words, while reasoning, people construct working cognitive representations of
phenomena they interact with. This mental representation is build associating the knowledge they
already have with the incoming information. So, while reasoning we construct the mental model
that represents the relevant (semantic) information of the problem.
Another important question Johnson-Laird tries to solve is how mental models are
constructed and applied. In his book “Mental Models”, this author distinguish three different kinds of
representation:
-
mental models: structurally and syntactically analogue to the represented world.
-
propositional representation: mental representations of the content of a verbally
expressible proposition.
-
Images: perceptual correlates of the represented world from a particular point of view.
The main difference between representations is that propositional representations are
abstract symbols, whereas the mental models and the images are analogues. According to this
author, people construct mental models (which have picture-like properties) from propositional
representations (which have language-like properties)1. However, mental models are not perfect
imitations of real-world phenomena, normally they are simpler. But, in order to provide explanation,
the structure of the mental models is similar to the phenomenon they represent. According to
Johnson-Laird they are either analogical or a combination of analogical and propositional
representations.
1
The process by which propositional representations are mapped into mental models is
called procedural semantics (Sasse, 1997). We will not discuss this process here.
13
Mental models of incidental human-machine interaction
3. The early definition(s) from HCI
The same year that Johnson-Laird published his book, another one with the same tittle
appeared. The collection edited by Gentner & Stevens (1983) analyses the concept in an
interdisciplinary theoretical approach: Cognitive Psychology and Artificial Intelligence. In general,
this publication provides some evidence that mental models, and the mechanisms by which they
are constructed, may differ according to the task or problem domain (Sasse, 1997). Among others,
this book contains two of the earliest papers of mental models of computer systems by Norman
(1983) and Young (1983).
3.1. Norman (1983)
Norman defines the mental model as the mental representation constructed through
interaction with the target system and constantly modified throughout this interaction. And from
observations on a variety of tasks, Norman (1983) concluded that:
a. Mental models are incomplete. Mental models are constrained by such things as the
user’s background, expertise, and the structure of the human information processing
system.
b. Mental models have vague boundaries. This implies that operations and systems with
certain relations or similarities can be mixed up (e.g. the mental model of an operational
system and an application program used in the operational system can be mixed up).
c. Mental models are unstable over time. People forget the details of the system they are
using, especially when those details (or the whole system) have not been used for some
period. And, as long as mental models are naturally evolving models, they change not only
because people forget and mix up details but also because new information is
incorporated over the time and through the interaction with the system. So learning and
forgetting are two sides of the same coin or of the fact that mental models are unstable.
d. Mental models are unscientific and contain aspects of superstitions. For example,
people maintain behaviour patterns even when they know they are unneeded. According
to Norman, people maintain this kind of behaviours because normally they cost little in
physical effort and save mental effort, and this is specially apt to be the case when a
person has experience with a number of different systems, all very similar. We have to add
14
Mental models of incidental human-machine interaction
that according to us, although the do not really seem to save mental effort and they know it
is not necessary, people maintain this kind of behaviour because doing them they feel
more comfortable and more confident while using the system. For example, before to shut
down the computer some people first go back to their home directory. When asked, they
explained that they knew it was unnecessary but they felt they were behaving nicely with
the system and, as a consequence, the system will respond according to their
expectancies in order to reach the goal.
e. Mental models are difficult and with restrictions to run.
f. Mental models are parsimonious. Users tend to trade off extra physical actions for
reduced mental complexity.
But even when they are not fully accurate, they are functional. As a matter of facts the
mental model denotes the knowledge structure that the user applies in:
-
planning actions,
-
guiding the interaction with the system and the execution of the planned task,
-
evaluating the results according with the expectations the user has,
-
and interpreting the unexpected results while using the system.
Another important contribution of this author is that, in the study of the aspects that
influence the interaction between human beings and systems, it is necessary to consider five
different concepts or, as Norman called them, things:
a. Target system: it refers to the system with which the user interacts.
b. The conceptual model: an accurate, consistent and complete description of the system
as far as relevant to the user. Norman explains that this model is devised as a tool for the
understanding or teaching of the system to the user and it is normally developed by the
designer or the “teacher”. With this goal the conceptual model must fulfil the criteria of
learnability, functionality and usability. We maintain that the conceptual model can be also
useful during the designing process of the system.
c. The mental model from which the user understands and predicts the behaviour of the
system. The mental model is constructed through interaction with the target system and
constantly modified throughout this interaction.
15
Mental models of incidental human-machine interaction
d. Norman introduces the idea of system image that refers to the perceptible aspects of the
system that are available to the user.
e. The scientist’s conceptualisation of the mental model describes the content and
structure of the user’s mental model as understood by the “scientist”. It reflects the
understanding and the knowledge the psychologist has about the mental model of the
user. Norman stresses the importance of using experimental psychology and observation
techniques to figure out the mental models users actually have. Later, Norman (1986)
distinguished between the designer’s and the researcher’s conceptualisations. As we will
discuss later, we think it is important to stress the idea that, not in all cases of cases of
design, it is possible to distinguish between the designer’s and the researcher’s
conceptualisations. In some cases, in the design teams there is not and expert “scientist”
interested in the study of the mental model. In other words, the image of the expert in
experimental and cognitive psychology dedicated to the study of the mental
representation the user has of the system and collaborating directly in the design
process, very often is not a real one. Instead, during the design process designers pay
very little attention of the way the future user of their system understands and interacts
with the system. According to us, it is really necessary to include experts in mental
models and human computer interaction in the design teams or at least, designers should
be trained to be able to understand the mental model of the user. Fortunately, technology
companies such as Apple Computer, Philips Design, Bell Labs, and Xerox’s Palo Alto
Research Center seeking to design more usable, compelling products, and services are
beginning to tap the human behaviour expertise of psychologists, anthropologists and
other social scientists in growing numbers (Bailey, 1996, Lear, 2000).
3.2. Young (1983)
In the same volume and in parallel with Norman, instead of mental model Young (1983)
uses the term user’s conceptual model (UCM) to refer to "... a more or less definite representation
or metaphor that a user adopts to guide his actions and help him interpret the device’s behaviour."
(p. 35).
Furthermore, Young (1983) introduces a distinction between different types of mental
representations that users can have about the system (strong analogy, surrogate, mapping,
coherence, vocabulary, problem space, psychological grammar, and commonality). Following the
16
Mental models of incidental human-machine interaction
idea that it is possible to have different mental models about a system, representing different kinds
of information, different classifications about the mental models were made.
3.3. Carroll and Olson (1988)
Stressing the idea of its capability to understand, to evaluate and to interpret the way the
system works and the results of the interaction with the system, Carroll and Olson defined a mental
model as: “a representation (in the head) of physical system or software, with plausible cascade of
causal associations connecting the input to the output". In other words, the mental model is a
“mental structure that reflects the user’s understanding of a system”.
Another important characteristic is that the mental model can be conceived as “knowledge
about the system sufficient to permit the user to mentally try out actions before choosing one”.
And it can be created spontaneously by the user or carefully formed and structured
through training.
4. Different classifications of mental models
4.1. The classification of DiSessa (1986)
DiSessa (1986) distinguishes between structural (“what is”) knowledge and functional
(“how to”) knowledge about the system. According to this distinction, structural models provide the
users with a detailed understanding of the systems whereas functional models represent the
properties of the system needed to perform a specific task. In other words, the first kind of mental
model contains information about the internal structure of the system and it is independent of the
specific task. However, the functional models contain information about how to use a selected set
of functionality to perform a real-world task.
According to DiSessa, functional models seem to be ideal for non-experts who want to use
the system as a tool because they take less time and mental effort to acquire and maintain. But,
although Sasse (1997) maintains that walk-up-and-use systems are the exception, we think that
17
Mental models of incidental human-machine interaction
nowadays it is possible to perform more that one task even with this kind of systems. This implies
that users would have to acquire a multiplicity of overlapping functional models (one for every
specific task that can be performed with the system). So the advantage of less mental effort does
not seem so evident any more. In conclusion, especially for complex systems that can perform
different tasks, the neat division between structure and function is not possible. Finally, DiSessa
introduces the idea of distributed models. They are not neat structural or functional but
accumulations of multiple partial explanations users hold of a system, tied in with their previous
existing knowledge and experience.
4.2. The classification of Nielsen (1990)
Another classification is the one made by Nielsen (1990). Nielsen presents a meta-model
to classify the different models used in HCI. Following Sasse (1997) we will simply show the
aspects of Nielsen’s classification that are relevant to HCI:
1. Internalised versus externalised models. An internalised model can reside inside the
human mind or in the system. According to Van der Veer (1999), internal models are for
“execution”. The main idea is that there is an “agent” who uses the model to make
decisions based on the behaviour of the model, and to make predictions on the behaviour
of the modelled reality. If the agent is human we will call it mental model. But if the agent is
the system we will call it program, database, etc. On the other hand, externalised models
are mainly build for communication (Van der Veer, 1999) and are represented in some
explicit form outside the human mind or the system. Examples of external models are the
conceptual model and the scientist’s conceptualisation of Norman (1983).
2. Structural versus distributed models. In parallel with DiSessa (1986), distributed models
contain partial and overlapping knowledge about the system. But in the case of structural
models, Nielsen refers to the models that follow a guiding principle. According to this
classification, mental models can be distributed while the conceptual model and the
scientist’s conceptualisation can be more structural models.
3. Generic versus instantiated models. The instantiated model is the idiosyncratic mental
model of the user (mental model) and the generic model refers to the “prototype” mental
model of the user envisaged by the designer and/or the researcher (the scientist’s
conceptualisation of Norman).
18
Mental models of incidental human-machine interaction
4. General versus specific models. While general models describe users and systems in
general terms, specific models are aimed at describing specific reasoning mechanisms
employed by a particular user group (e.g. novices versus experts) or specific tasks that
can be performed using the system. Related with the idea of specific models later we will
consider the user's virtual machine or UVM. Briefly, the UVM is a detailed description
about the aspects of the system that are directly relevant for a specific task and/or a
specific group of users.
5. Descriptive versus analytic models. Normally, the mental representations that users and
designers have are more informal descriptions of the system. However, other descriptions
of the system use more formal methods and show a more complex analysis of the system
(e. g. the scientist’s conceptualisation of Norman). Maintaining her distinction between
researchers and designers, Sasse (1997) thinks that the analytic models are more used by
researchers than by designers.
6. Static versus dynamic models. This dimension distinguishes between models that do
not change continuously over the time and those which change continuously. Normally,
internalised models (such as mental models) are more dynamic than externalised. The last
ones, can change during the design process but when the system is build, normally
became static.
5. The problem of the multiple definitions of mental models
In the previous paragraphs we tried to briefly expose some of the most relevant and early
definitions of mental models. As we saw a mental model is an internal representation that people
form and use while interacting with the environment (problem, system, etc). In more or less degree,
this representation contains structural information about the properties of the system and functional
knowledge about the task to perform. On the other hand, mental models are incomplete, unstable
and unscientific but they can be used for planning, execution, evaluation, and interpretation of the
system or problem the subject has to solve. Another important characteristic is that they are build
associating previous knowledge stored in memory (Long Term Memory: LTM) with the incoming
information from the context and tasks demands. And as we will immediately see, it seems mental
models are build in working memory. Furthermore, mental models are naturally evolving models.
19
Mental models of incidental human-machine interaction
According to Holland et al., (1989) they can be “run” or manipulated, combining existing rules, to
produce expectations about the environment”2. As a consequence mental models are changed
and, most of the times, refined as additional information is acquired. Finally, it is important to
underline the fact that users can have different mental models of the systems and, as we will see
soon, this can hold differences in performance while interacting with the system.
But, although we could summarize the common ideas researchers have about what is a
mental model, as in many others, one of the big problems in this field is the absence of agreement
between definitions. Differences between approaches are due to the way variables are represented
(instantiation in specific symbols vs. direct representation), the relation of models with the memory
structures (WM vs. LTM), and the phenomena they intend to explain (syllogistic reasoning vs.
causal reasoning). Nowadays, people are still trying to build a common, useful and valid definition
of the concept.
5.1. The reason of the disagreements according to Cañas & Antolí (1998)
According to Cañas and Antolí (1998), the fact that the term “mental model” has been used
by researchers who work in different tasks is the main reason of the disagreements in the definition
of the concept. For example, Johnson-Laird (1983) worked in reasoning, while Norman (1983)
used the concept in the field of Human-Computer Interaction. As a consequence, researchers
focused in different aspects of mental models. Some give more importance to the information
extracted by perceptual processes of the characteristics of the task (e.g. Johnson-Laird, 1983) and
some pay more attention to the knowledge the person has stored in Long-Term Memory (LTM)
about the operations and the structure of the system (e.g. Norman, 1983). As a result, a common
debate in the literature is the question about if mental models are structures of knowledge in LTM
or temporary creations in Working Memory (Van der Veer et al., 2000).
Currently, Cañas and Antolí (1998) are trying to unify both definitions and to develop a
model of mental model formation. The basic hypothesis that they maintain is that a mental model is
2
The authors describe mental construction as a trade off between accuracy (addition of
specialised rules) and efficiency (general rules).
20
Mental models of incidental human-machine interaction
a dynamic representation created in Working Memory by combining information stored in LongTerm Memory and characteristics extracted from the environment.
5.2. The pragmatic approach of Van der Veer (1990): The need of a conceptual framework that
includes the relevant concepts for designers.
Finally, in applying the concept to Human-Computer interaction, Van der Veer (1990)
proposes to behave pragmatically. He considers mental model any type of mental representation
that enables and facilitates the interaction with the system and that develops during the interaction
with the system.
Following this pragmatic perspective, and according to our purposes, we will need to
specify a conceptual framework that includes and defines clearly the relevant concepts for design
production and the role mental models can perform in improving the design process. As a matter of
facts, this will be the main purpose of our next “chapter”.
References
Bailey, R. W. (1996). Human performance engineering: Designing high quality, professional user
interfaces for computer products, applications, and systems. Prentice Hall PTR.
Cañas, J. J. & Antolí, A. (1998). The role of working memory in measuring mental models. In:
T.R.G. Green, L. Bannon, C.P. Warren & J. Buckley (Eds.). Proceedings of the Ninth
European Conference on Cognitive Ergonomics – Cognition and Cooperation. EACE.
INRIA. Rocquencourt.
Carroll, J. & Olson, J. (1988). Mental models in human-computer interaction. In Helander, M., (Ed.).
Handbook of Human-Computer Interaction. North-Holland, Amsterdam, The Netherlands.
Chisăliţă, C. (in press). New directions in theory and methodology of mental models.
21
Mental models of incidental human-machine interaction
Craik, K. J. W. (1943) The Nature of Explanation. Cambridge: Cambridge University Press.
DiSessa, A. (1986). Models of computation. In D. A. Norman & S. W. Draper (Eds.). User-Centered
System Design: New perspectives in Human-Computer Interaction. Hillsdale, NJ: LEA.
Gentner, D. A. & Stevens, A. L. (Eds.) (1983). Mental models. Lawrence Erlbaum. Hillsdale, N. J.
Holland, J.H., Holyoak, K.J., Nisbett, R. E. & Thagard, P. R. (1989). Induction: processes of
inference, learning, and discovery. The MIT Press. Cambridge, Mass.
Johnson-Laird, P. N. (1983). Mental models. Cambridge University Press.
Johnson-Laird, P. N. & Byrne, R. (2000). Mental Models Website: A Gentle Introduction.
http://www.tcd.ie/Psychology/Ruth_Byrne/mental_models/index.html
Lear, A. C. (2000). Uncovering technology’s human side. Computer. Innovative Technology for
Computer professionals, 7, 24.
Nielsen, J. (1990). A meta-model for interacting with computers. Interacting with computers, 2, 147160.
Norman, D. A. (1983). Some observations on mental models. In D. A. Gentner, & A. L. Stevens
(Eds.). Mental models. Hillsdale, NJ: Erlbaum.
Preece, J., Rogers, Y., Sharp, H., Benyon, D., Holland, S., & Carey, T. (1994). Human-Computer
Interaction. Addison-Wesley.
Sasse, M. A. (1997). Eliciting and Describing Users’ Models of Computer Systems. Thesis
submitted to the Faculty of Science of the University of Birmingham.
22
Mental models of incidental human-machine interaction
Van der Veer, G. C. (1990). Human-Computer Interaction: learning, individual differences, and
design recommendations. Thesis submitted to the Faculty of Sciences of the Vrije
University of Amsterdam.
Van der Veer, G. C. (1999). Human-Computer Interaction/ Cognitive Ergonomics- a course outline.
House of the Book of Science. Cluj.
Van der Veer, G. C., Chisalita, C. & Mulder, B. (2000). The conceptualization of mental models as
base for the desing of interactive systems and situations. Paper proposal for ICCM-2000.
Young, R. M. (1983). Surrogates and mappings: two kinds of conceptual models for interactive
devices. In D. A. Gentner, & A. L. Stevens (Eds.). Mental models. Hillsdale, NJ: Erlbaum.
23
Mental models of incidental human-machine interaction
APPENDIX B: THE NEED OF A CONCEPTUAL
FRAMEWORK
1. The agreements and disagreements about mental models in HCI
Sasse (1997) concludes that the different theories of mental models do not agree in:
a. The content and structure of the mental models held by users.
b. How exactly users’ mental models influence their interaction with computer systems and
systems in general.
c. The nature of the process through which mental models are constructed.
But, explicitly or implicitly assume that:
a. Users form a mental model of the internal activities of the computer systems they are
interacting with.
b. The content and structure of that mental model influences how users interact with a
system.
c. Selecting what information about the system is presented to users, and how it is
presented to them can influence the content and structure of a mental model.
d. More detailed knowledge of how users construct, invoke, and adapt mental models
could be used to provide guidance for user interfaces and user training which help users to
form appropriate models, and therefore make an important contribution to HCI.
So although there is no agreement about the exact definition of the concept, in the field of
Human-Computer Interaction (HCI) the concept of Mental Model plays an important role. As Sasse
(1997) concluded, researchers agree in the fact that knowledge extracted from mental models can
be used to guide the design process and, in turns, a good design helps users to form appropriate
mental models. It seems that the closer this mental model resembles the actual system, the better
(Klok, 1998). But still the big question is how exactly mental models can be applied to design
computers and systems in general to facilitate learning and use of the system.
24
Mental models of incidental human-machine interaction
In the following paragraphs we will explain some of the conceptual frameworks used
specifically in HCI to describe the user’s internalised representations of a computer system. As we
will see, different researchers in HCI created its own terminology. But, first of all we will introduce
the basic elements that are necessary to consider in order to study the mental representations
implicated in the interaction with the computer. Following Nielsen (1990) these elements are:
- U: user.
- D: designer.
- C: computer system.
- M: manuals and other documentation.
- T: task performed by the user.
- W: surrounding world in which the user performs the task.
- R: researcher.
2. The choice of mental models as a basis for designing more usable systems.
Norman and Draper (1986) applied the notion of mental model to HCI to bridge the gap
between designers and users of software systems. The main idea they show is that a designer will
have formed a working model of the system that can be different to the user’s mental model. The
fact that the models are different is not necessarily a problem; the problem arises when the user’s
mental model is inappropriate. Norman and Draper proposed that designers could facilitate the
construction of appropriate user’s model of a system trough the design of its user interface. “In
general, if mental models are sufficiently accurate then it is possible to solve unexpected problems,
but inappropriate models can lead to difficulties” (Preece et al, 1994).
2.1. The conceptual design approach of Norman (1986)
One of the most influential proposal as to how mental models can be harnessed to design
more usable systems is the theory of Donald Norman. The main assumption of this theory is that a
well-designed system and user interface will allow the user to develop an appropriate model of the
system and, as a consequence, an appropriate performance and/or interaction with the system.
Well designed interface  adequate mental model of the system  appropriate performance
25
Mental models of incidental human-machine interaction
According to the conceptual design approach, in the design process it is important to
consider:
a. The user’s model or mental model. As we saw it is the user’s representation of the
system that is internalised. Following the classification of Nielsen (1990), mental models
are instantiated and usually specific and descriptive.
b. The design model. This is the model of the design team (and/or the researchers). It is an
externalised and general representation of the system. Normally this kind of model is
structural, descriptive, externalised, and static.
c. The system image. Remember that the system image refers to the perceptible aspects
of the system that are available to the user. In other words, the system image is the way in
which the design model is presented to the user through the user interface,
documentation, instructions, error, and help facilities.
We can add that the user’s mental model of the system is mainly derived from the
interaction between the user and the system image. So, if the designer gets the design model right,
and communicates this model successfully through the system image then, users interacting with
the system will develop an appropriate mental model of the system which allows them to use the
system successfully.
DESIGNER MENTAL MODEL  SYSTEM IMAGE  USER MENTAL MODEL
2.2. The importance of the user’s task and user’s previous knowledge
From the point of view of the designers, the main goal of design is to design a highly
usable and useful system (Hartson, 1998; Van Welie, 2000). And following Norman’s theory, one of
the biggest problems is to communicate properly the design model through the system image.
Due to this difficulty currently some authors (designers and/or researchers) advise that the
design models should be as simple as possible and reflect the user’s task rather than the hardware
and software of the system (Tognazzini, 1991).
26
Mental models of incidental human-machine interaction
Following this idea, Sasse (1997) adds three new sets of representations that are
important to consider during the design process:
a. The user’s task model. Internal representation of the task or tasks to be performed.
According to Sasse it is an instantiated, descriptive, specific and dynamic model of the
task. Another important issue is that it is expected that users with similar experience of the
task will have similar task models.
b. The designer’s model of user’s task. According to Sasse, the designer’s model of the
user’s task is normally a specific and static description of the task. But against the old
fashion idea that the conceptualisation designers have of the user’s task is static, we
think it should be dynamic because the task world is changing continuously. On the other
hand, Sasse stress the idea that the design model have to be based on the user’s model
of the task and not on the designer’s model of the task. In other words, although
designers have their own model about the task, and about the system, in order to design
and adequate and useful system, designers should pay attention of the model the user’s
has rather than in their own model.
c. The researcher’s conceptualisation of user’s task. It is the conceptualisation of the
user’s task made with a formalised task analysis method. The result is normally a generic
and general model rather than an idiosyncratic model of an individual user’s task model.
Additionally, Green (1990) uses the term viscosity to illustrate the relationship between the
task’s demands and the system. According to this author, the viscosity of a system increases with
the number of internal tasks that the user has to know and perform with the system in order to
perform the real-world task. On the other hand, the more internal tasks are required, the bigger is
the cognitive load that the user will suffer while interacting with the system (Payne et al., 1990).
According to Payne et al., the user has to represent the device space that represents the possible
states of the system. He also has to represent the goal space that represents the possible states of
the external world that can be manipulated with the system (or device), and the semantic mapping
that relates the entities between both spaces.
Another aspect that it is important to consider is the previous knowledge that the users
have about the system, about similar systems and about the task. As a matter of facts, the
previous knowledge the users have may influence the way in which they interpret the incoming
27
Mental models of incidental human-machine interaction
information, the system, and their interaction with it. Following this idea, Sasse adds three new
important issues:
a. The user’s existing knowledge. The previous knowledge and experience of the users
may influence the way they interpret the information about the system and the way they
interact with it.
b. The designer’s model of user’s knowledge. Designers normally form an internalised
generic model of the user’s background knowledge and experience that can be relevant to
the task and the system.
c. The researcher’s conceptualisation of user’s knowledge. Can differ from the designer’s
model user’s knowledge and be more externalised and analytic.
If we put together Norman’s and Sasse’s ideas we can distinguish between users,
designers and researchers. On the other hand, we can distinguish between different contents of
the internal representation: Norman speaks about the internal representation of the system
whereas Sasse stresses the representation people have about the task to be performed and the
previous knowledge and the degree of expertise users can have. Finally, both of them distinguish
between the designer and the researcher conceptualisations about the user’s internal
representations.
system
task
previous knowledge
users
internal representation
internal representation
internal representation
designers
internal representation
conceptualisation
conceptualisation
researchers
conceptualisation
conceptualisation
conceptualisation
So far, we consider that the so called internal representation about the system and the
task are the instantiated information from the environment that is represented in working memory
while interacting with the system (or designing it). And the previous knowledge about both, the
system and the task, following Cañas & Antolí (1998), is the one stored in LTM that is combined in
working memory with the incoming information from the environment.
28
Mental models of incidental human-machine interaction
On the other hand, both Norman and Sasse, distinguish between the designer’s and the
researcher’s conceptualisations of the internal representation. According to these authors, and
because they are experts in mental representations, the conceptualisation made by researchers is
more formal and exhaustive than the one made by designers. We can express the idea with other
words, as long as they have different interests, researchers in HCI and designers see the subject
from a different point of view. However, although the differentiation between designer’s and
researcher’s conceptualisation is possible, this is not always a realistic position because the
communication between Cognitive Psychology (and related fields as separate faculties than the
faculty of design) and designers is not always possible. One reason is because, in the design team
there is not always an expert in mental representation but only designers. And the second reason
is that, like in many other fields, there are difficulties in communication between theory and
practice. In most of the cases, experts in mental representation have difficulties showing the
practical consequences of their findings3.
From this problem we can conclude that a multidisciplinary approach for design is needed.
According to Van der Veer (1999), in a design team of complex systems it is necessary to include
different experts that have to work together. It is necessary to include experts in psychology,
anthropology and/or ethnography centred in the study of the human capabilities to process
information and to respond and in the study of the how the group develop in the work situation.
Experts in cognitive ergonomics and experts in designing the workplace and the organization are
also necessary. Experts in representations such a graphical designers and even experts in theatre
and cinematography to represent properly the relevant information at every moment. And of
course, it is necessary to have engineers’ experts in hardware and in software design. Fortunately,
more often social scientists and engineers are working more closely that they have in the past and
the design team deliberately includes members with a variety of different backgrounds and
experiences, so that each system is adequately represented by a specialist (Bailey, 1996). This is
true specially in the case of big technology companies such as Xerox’s Palo Alto, Apple Computer,
Microsoft, Philips Design, etc. In this type of companies they are bringing in anthropologists,
psychologists, and sociologists to help with product development. (Lear, 2000). If not possible to
3
At least not in an easy way.
29
Mental models of incidental human-machine interaction
have such design teams, we at least recommend to designers to be properly educated in order to
understand the way user’s represent their interaction with the system and how to use correctly this
information to design more usable systems.
2.3. The importance of the system image
As long as it is the perceptible aspect of the system for the user, the construction of the
system image is as important as the construction of the design model itself. Tognazzini (1991)
offers some practical guidance on designing the system image:
1. Design user interface objects which encourage and facilitate user behaviour that is
consistent with the design model.
2. Remove elements that are not needed for a specific task from the user’s sight.
3. Minimise the amount of information that users have to remember.
4. Do not use abstract or invisible objects.
And, as long as perceptible, it is also important to consider the properties of the cognitive
(attention, perception, memory, though, language…) and motor human processes that users apply
in their interaction with the system via the system image. In this point, Sasse (1997) again
distinguishes between the researcher’s model of users and the designer’s model of users.
Researchers are psychologists and other cognitive scientists interested in the production of
theories about human attention, memory, cognition, and kinaesthesia. Designers on the other
hand, are less familiar with cognitive theories and Tognazzini (1991) recommends them to consult
a researcher to select the cognitive relevant aspects they have to consider in the creation of the
system image.
Summarising, Sasse concludes that the design model should be based on:
- the user’s task,
- the user’s previous knowledge and experience,
- and the cognitive properties of the user.
30
Mental models of incidental human-machine interaction
 The result of the design model is expressed via the system image. And the users
construct a mental model, as a result of their interaction with the system, via the system image,
and, perhaps training and instruction.
3. The DUTCH conceptual framework
Another design conceptual framework is the one developed by Van der Veer and Van
Welie (2000): DUTCH (Designing for Users and Tasks from Concepts to Handles) for complex
systems. Their design method is driven by an extensive task analysis followed by structured design
and iterative evaluation using usability criteria. To cover the wide range of aspects of design, they
use a combination of multiple complementary representations and techniques. Figure 1 gives an
overview of the whole design process with all the activities and sources of information.
Figure 1. The DUTCH design process.
31
Mental models of incidental human-machine interaction
The main goals are to design usable and useful systems and, with these purposes, their
design process consists of four main activities:
a. Analyzing the "current" task situation.
b. Envisioning a future task situation for which information technology is to be designed.
c. Specifying the information technology to be designed.
d. Evaluating activities to make the process cyclic.
3.1. The task analysis
The first steps are to analyze the current and the future task situations. The first task
model is a descriptive task model and is used for analyzing the current task situation. The second
task model is a prescriptive task model for the system that is to be designed.
3.1.1. Analyzing the current task situation (Task model 1)
First it is necessary a systematic analysis of the current situation. In many cases the
design of a new system is triggered by an existing task situation but, either the current way of
performing tasks is not considered optimal, or the availability of new technology is expected to
allow improvement over current methods. A systematic analysis of the current situation may help to
formulate the design requirements, and at the same time, may later on allow evaluation of the
design.
Task model 1 is developed on the basis of knowledge acquisition activities. Especially in
cases of complex task situations, several different disciplines need to contribute. Techniques that
might apply may be related to knowledge elicitation and psychological techniques, or be derived
from anthropology and ethnography. The final goal of this group of activities is to analyze, describe,
and model the current task world.
3.1.2. Envisioning the future task situation (Task model 2)
Task model 2 is the result of analyzing task model 1 on its values and problems for the
user, confronted with the requirements from the client of the design process as well as with the
technology availability and constraints. The aim of this set of activities is to specify a model of the
32
Mental models of incidental human-machine interaction
envisioned new task world. Task model 2 in general will be formulated and structured in the same
way as the previous model, but in this case it is not considered a descriptive model of users'
knowledge but a prescriptive model about the knowledge an expert user of the new technology
should possess.
In general, for describing the task world, Van der Veer and Van Welie propose to use a
combination of classical HCI techniques such as structured interviews and CSCW (Computer
Supported Cooperative Work or Computer Supported Collaborative Work) techniques such as
ethnographic studies and interaction analysis. They also developed a broad task-analysis
conceptual framework that is based on comparisons of different approaches and on the analysis of
existing and proposed systems for HCI and CSCW (see a more detailed explanation in Van der
Veer and Van Welie, 2000).
3.2. The UVM
After the task modeling activity the system needs to be designed and specified. This
activity is focused on a detailed description of the system as far as it is of direct relevance to the
end-user. Van der Veer et al., (1985) use the term of UVM, the user's virtual machine, to denote all
aspects of a system the user should be aware of during interaction. In other words, the UVM has to
be modeled indicating all the relevant details and knowledge the user has to have about the
system, both semantics (what the system offers the user for task delegation) and syntax (how task
delegation to the system has to be expressed by the user).
The specification of the UVM can be structured into three sub-activities that are strongly
interrelated:
1. Specifying the functionality of the system. This activity includes specifying the system
details as far as relevant to the user (what the system provides to perform the task to a
specific type of user). Aspects like the look and the feel of the technology, the specification
of the graphical interface, and any representation of the system and its products, have to
be considered too because they can strongly influence the ease of use. So, apart from the
functionality of the UVM we need to specify the interaction between user and system, with
33
Mental models of incidental human-machine interaction
its two directions of communication: the dialog (or language interface) and the
representation (or presentation interface).
2. Structuring the dialog between the users and the system. Modeling the language
interface means to define the type(s) of language style in which the user expresses
himself/herself to the system.
3. Specifying the representation or the way the system is presented to the user. Modeling
the system's actions and representation of relevant information for the user is possible to
influence the kind of control the user feels has over the system, as well as the amount of
learning needed to operate the system.
We think that the result of specifying the dialog but mainly the representation corresponds
to the system image of Norman (1983) as long as it represents perceptible and the dynamic
aspects of the system.
The three types of design decisions resulting from this obviously are strongly related in
their consequences for the user, hence, the activity of maintaining consistency is a further
important design activity in the phase of specifying the UVM (Van der Veer and Van Welie, 2000).
Another important issue is that the UVM defines what is relevant for a specific group of
users. In this context, a group is mainly defined by the task domain and the degree of expertise. If
there are different classes of users (different roles) and possibly stakeholders ("indirect" users of
the system, i.e., groups of people whose work is influenced by the implementation of the new
system) there will be a need to specify a different UVM for each role.
As the reader can notice, the idea of UVM and the conceptual model defined by Norman
(1983) are very similar. Both consist in an accurate and complete description of the system as far
as relevant to the user. But using again the differentiation between researchers and designers,
perhaps the distinctions between those concepts are related to the different point of view and
interests that the designer and the researcher have. We hypothesize that the idea of conceptual
model has been developed more according to the researcher point of view while the more detailed
definition of UVM comes from a designer environment.
34
Mental models of incidental human-machine interaction
3.3. The evaluation and the usability testing
During the entire process, some kind of evaluation activity can take place. As soon as an
initial task model is available it can already be evaluated using scenario's and use cases. Later on
when some initial sketches for the new system are known, mockups and prototypes can be used
for early evaluation of design concepts. Each evaluation activity can cause another iteration. For
instance, a task model may turn out to be incomplete after a mockup is evaluated in a critical
scenario. In that case the designers need to go back to the task model and rethink their model. The
activity of evaluation will therefore be mandatory in parallel to each of these activities.
Normally, the development of the UVM is based on detailed specifications of the "new
system" on the basis of task model 2. From there, the details of the technology and the basic tasks
that involve interaction with the system need to be worked out. During this process, the need for
feedback to task model 2 will be inevitable. If design decisions force the specifications to deviate
from the original global intentions, this has to be considered explicitly. Changes in the model of the
information system may result in changes in other parts of the task world, like organization of
people's work, communication structure, and work procedures. If this would cause task model 2 to
deviate from what is considered an adequate answer to the request for system development, this
could lead to reconsidering the whole set of decisions taken so far. Feedback to task model 2 is a
design activity that has to be coordinated and continuously controlled. So, frequent iterations will
be needed between the specifications of the tasks models and the UVM specifications. This
iteration is an explicit part of the method and essential for the development of usable systems.
Evaluation may need a variety of testing techniques, from formal analysis and cognitive
walkthrough to user testing, observations in usability labs, and the application of norms and
standards. Again, different specialists will be needed in an evaluation phase, and a variety of
experts will be needed from ergonomists to hardware and software engineers and programmers. In
some cases of design decisions, guidelines might be of help, in other situations formal evaluation
may be applied. Formal modeling tools like CCT, TAG, or ETAG may provide an indication of
complexity of use or learning effort required (De Haan et al., 1991).
35
Mental models of incidental human-machine interaction
4. Our ideas: the role of the mental model and the UVM
We saw that the main problem with the study of mental models is the lack of a unified and
universal terminology. Tauber (1988) indicates that “the term of mental model has been liberally
applied to virtually any representation of knowledge about devices”. The main goal we had
introducing different models of design processing was precisely, and following the pragmatic
approach of Van der Veer (1990), to try to reach an useful definition of mental model as an
appropriate tool to help and to improve the design process. But the problem remains; in HCI there
is not a single recognised set of terms which is universally employed. And the consequence is that
the different terminology and, even sometimes contradictory statements, will confuse designers
that try to apply the knowledge derived from HCI of users’ models. Furthermore, the theoretical
contributions of HCI offer little concrete knowledge and guidance related to how knowledge of task
domain and task analysis, degree of expertise and previous knowledge, and properties and
limitations of the human information processing system can be applied in the design model. Newell
and Card (1985) finally predicted that all of these problems would lead designers to ignore HCI.
We think that in order to improve the interaction between the system and the user,
designers should be concerned about the UVM. That is, the aspects of the system that are directly
relevant for a specific group of users. According to what we just see, the specification of the UVM
should consist in the specification of the functionality of the system, the type of dialog needed to
interact with it, and the representation of the system, or using the terms of Norman, the system
image. And following the pragmatic approach of Van der Veer, we think that the mental model is
the mental representation of the UVM – of the functionality, the dialog and way the system is
represented - that the user builds through its interaction with the system image, while performing a
specific task, and applying his previous knowledge. And now following Cañas and Antolí (1998),
this representation is created in the working memory. Consequently, one of the most important
issues for the designer is to build the system image in such a way that the user acquires easily an
optimal or appropriate mental model of the UVM.
36
Mental models of incidental human-machine interaction
System
functionality
dialog
UVM
representation
Mental model (WM)
Task requirements
Representation
of UVM
Knowledge
LTM
Figure 2. The mental model is the mental representation of the UVM.
And as we said, an important issue is that the UVM stresses the idea that different classes
of users should have a different UVM, and a different mental model, of the system defined mainly
by the task domain and expertise. In other words, designers have to think about different classes of
users in order to specify correctly their specific tasks requirements.
References
Bailey, R. W. (1996). Human performance engineering: Designing high quality, professional user
interfaces for computer products, applications, and systems. Prentice Hall PTR.
Cañas, J. J. & Antolí, A. (1998). The role of working memory in measuring mental models. In:
T.R.G. Green, L. Bannon, C.P. Warren & J. Buckley (Eds.). Proceedings of the Ninth
European Conference on Cognitive Ergonomics – Cognition and Cooperation. EACE.
INRIA. Rocquencourt.
37
Mental models of incidental human-machine interaction
De Haan, G., Van der Veer, G. C., & Van Vliet, J. C. (1991) Formal modelling techniques in
human-computer interaction. Acta Psychologica 78, 27-67.
Green, T. R. G. (1990). The cognitive dimension of viscosity: A sticky problem for HCI. In: D.
Diaper et al., (Eds.). Human-Computer Interaction- Interact’90. Proceedings of the IFIP TC
13 Third International Conference on Human-Computer Interaction. Cambridge, 27-31
August, pp. 79-86. Amsterdam: North-Holland.
Klok, J. A. (1998). A needle in a haystack: the design of an interface that guides the user trough
the complex conceptual space of an information retrieval system. Master’s thesis.
University of Twente.
Lear, A. C. (2000). Uncovering technology’s human side. Computer. Innovative Technology for
Computer professionals, 7, 24.
Newell, A. & Card, S. K. (1985). The prospects for psychological science in human-computer
interaction. Human-Computer Interaction, 1, 209-242.
Nielsen, J. (1990). A meta-model for interacting with computers. Interacting with computers, 2, 147160.
Norman, D. A. (1983). Some observations on mental models. In D. A. Gentner, & A. L. Stevens
(Eds.). Mental models. Hillsdale, NJ: Erlbaum.
Norman, D. A. (1986). Cognitive engineering. In D. A. Norman & S. W. Draper (Eds.). UserCentered Design: New perspectives in Human Computer Interaction. Hillsdale, NJ: LEA.
Norman, D. A. & Draper, S. W. (1986). User-Centered Design: New perspectives in Human
Computer Interaction. Hillsdale, NJ: LEA.
38
Mental models of incidental human-machine interaction
Payne, S. J., Squibb, H. R. & Howes, A. (1990). The nature of device models: The Yoked Space
Hypothesis and some experiments with text editors. Human-Computer Interaction, 5, 415444.
Preece, J., Rogers, Y., Sharp, H., Benyon, D., Holland, S., & Carey, T. (1994). Human-Computer
Interaction. Addison-Wesley.
Sasse, M. A. (1997). Eliciting and Describing Users’ Models of Computer Systems. Thesis
submitted to the Faculty of Science of the University of Birmingham.
Tauber, M. (1988). On mental models and the user interface. In: G. van der Veer, T. R. G. Green,
J. M. Hoc & D. M. Murray (eds.). Working with computers: Theory versus outcome.
London: Academic Press.
Tognazzini, B. (1991). TOG on interface. Reading. MA: Addison-Wesley.
Van der Veer, G. C. (1990). Human-Computer Interaction: learning, individual differences, and
design recommendations. Thesis submitted to the Faculty of Sciences of the Vrije
University of Amsterdam.
Van der Veer, G. C. (1999). Human-Computer Interaction/ Cognitive Ergonomics- a course outline.
House of the Book of Science. Cluj.
Van der Veer, G. C., Tauber, M. J., Waern Y., & Van Muylwijk, B. (1985). On the interaction
between system and user characteristics. Behaviour and Information Technology 4, 284308.
Van der Veer, G. C. & Van Welie, M. (2000). Task Based Groupware Design: Putting theory into
practice. Proceedings of DIS 2000, New York, United States.
39
Mental models of incidental human-machine interaction
APPENDIX C: ASSESSMENT OF MENTAL MODELS FOR
ENVISIONING DESIGN
Mari Carmen Puerta-Melguizo & Gerrit C. van der Veer
Vrije Universiteit
Amsterdam
The Netherlands
Extended Abstract for CSAPC 2001
The main goal of this paper is to show how relevant is the research and assessment of
mental models in design and specially in the process of designing innovative complex systems and
systems intended for incidental use. The different theories on mental models in HCI assume that
while interacting with the system users build a mental model of the activities of the system and the
content and structure of that mental model influences how users interact with a system. But the
main problem is that there is not agreement about the definition of the concept and even
contradictory statements have been made. Van der Veer (1990) considers mental model any type
of mental representation that enables and facilitates the interaction with the system and that
develops during the interaction with the system. In fact, he showed that several representations
may be used by different subjects for identical interaction situations. Following this pragmatic
approach, to understand the role that mental models play in design it is first necessary to specify a
conceptual framework that includes and defines clearly the relevant concepts for design production
and the role mental models can perform in improving the design process.
Van der Veer and Van Welie (2000) introduced DUTCH (Designing for Users and Tasks
from Concepts to Handles) for complex systems. This design method is driven by an extensive
task analysis followed by structured design and iterative evaluation using usability criteria.
40
Mental models of incidental human-machine interaction
Figure 1. The DUTCH design process.
With the main purpose of designing usable and useful systems the design process
consists of four main activities: 1) analyzing the "current" task situation, 2) envisioning a future task
situation for which information technology is to be designed, 3) specifying the information
technology to be designed and, 4) evaluating activities to make the process cyclic. The UVM,
introduced by Van der Veer et al, (1985) is a nuclear element of the DUTCH design process (see
figure 1). After the task modeling activities the system it is necessary to specify all aspects of the
system the user should be aware of during interaction. To describe the UVM means to specify: 1)
the functionality of the system, 2) the dialog between the users and the system, and 3) the way the
system is presented to the user or the representation. Another important issue is that the UVM
defines what is relevant for a specific group of users (mainly defined by the task domain and the
degree of expertise). If there are different classes of users and possibly stakeholders there will be a
need to specify a different UVM for each role.
Summarizing, in order to improve the interaction between the system and the user,
designers should concern about the UVM. That is, the aspects of the system directly relevant for a
specific group of users. And, following the pragmatic approach of Van der Veer, we think that an
appropriate mental model has to be the mental representation of the UVM that the user builds
through its interaction with the system.
41
Mental models of incidental human-machine interaction
3. The role of mental models in the envisioning future system
As we saw, the first step in the design process is to analyze the current task situation
including the users, the tasks, the objects and the environment. This step is called in DUTCH task
model 1(TM1). But, as important as TM1 it is important to analyze how the world would be if the
new design, specially the UVM, is implemented. Van der Veer and Van Welie (2000) called this
process task model 2 (TM2). TM2 is mainly developed considering the needs extracted from TM1
and the requirements of the client. But as long as we are envisioning the future, during TM2 it is a
lot less important to consider the actual constraints imposed by technology. At this point we are
modeling (and evaluating) new ideas and not actual systems (or even prototypes where
technological constraints has to be taken more seriously).
As long as it is not yet specified in any detail, the only available representation for
modeling TM2 is the use of scenarios. Based on Carroll (1995), a scenario is “a projection of a
concrete description of the activity that the user engages when performing a specific task, a
description sufficiently detailed that the design implications can be inferred and reasoned about”.
Scenarios have been used in the design of future technology in a form that shows an individual
instance of use, in a well-described individual situation. On the other hand they have the advantage
that the details of technology can be left unspecified. The scenario may be formatted as a written
story, as an acted out on process (taped on video), as a script to be acted out by prospective
users, with the help of mock-ups of potential technology, etc. A scenario is triggered by some event
and usually starts with some important goal to achieve. The scenario usually ends when the goal is
achieved (Van der Veer & Van Welie, 2000). Scenarios can be used both during the development
of TM2 and the evaluation of detailed specifications.
For example, Van der Veer proposed the use of scenarios in the design of a device for
future use by hotel guests as a practice for HCI students. The main goal of such a design was to
allow frequent guests to check-in and check-out quickly. A first scenario that some of the students
constructed was just a simple story on paper about an individual hotel guest using a "device" and
not needing to queue for the desk. A later variant consisted of a cardboard box with some
indication of buttons and a screen to be holding in hand. This last variant was actually used by
prospective users to perform an acted out form of the scenario showing how it would work as well
42
Mental models of incidental human-machine interaction
as what these users thought would be the actual interaction process. A more sophisticated form of
scenario can be found in products like HP's video "1995" and SUN's video "starfire", where
professional actors show the process for an individual situation, including some of the possible
positive and negative side effects of the use of this envisioned technology.
To model the future is mainly the development of ideas at the level of task delegation,
organization, and global procedures in a new situation. But TM2 means to begin with the
specifications of the UVM as well. We think that it is at this point where mental models play a very
important role. In order to design successful systems, it is important to know what future users
would understand about our envisioning. In other words, designers need to know if the envisioned
UVM is able to produce an appropriate user’s mental model of the system. And with this goal,
designers should assess the mental models that future users will develop when confronted with a
suitable representation of the envisioning of a new system.
Summarizing, this insight in mental models of future systems will bring us two types of
design knowledge:
1. How will users understand the envisioning, what may be misunderstood and how should
we represent the future technology for the user. This will lead to ideas on how to represent
the future technology for the user, how to introduce; as well as how to represent the
different “utilities” of the system.
2. What will users expect from the envisioned systems. Future users will come with ideas
about functionality that is intuitive for them, in a way contributing to the design of the
functionality.
4. How to approach the mental model
In the study of mental models a very important contribution is the one of Norman (1983).
According to this author different aspects are necessary to consider while studying the interaction
between the user and the system. The concepts he defined where:
a. Target system: it refers to the system with which the user interacts.
b. The conceptual model: an accurate, consistent and complete description of the system
as far as relevant to the user. Norman explains that this model is devised as a tool for the
43
Mental models of incidental human-machine interaction
understanding or teaching of the system to the user. With this goal the conceptual model
must fulfil the criteria of learnability, functionality and usability. We maintain that the
conceptual model can be also useful during the designing process of the system.
c. The mental model from which the user understands and predicts the behaviour of the
system. The mental model is constructed through interaction with the target system and
constantly modified throughout this interaction.
d. Norman introduces the idea of system image that refers to the perceptible aspects of the
system that are available to the user.
e. The scientist’s conceptualisation of the mental model describes the content and
structure of the user’s mental model as understood by the “scientist”. Norman stresses the
importance of using experimental psychology and observation techniques to figure out the
mental models users actually have. Later, Norman (1986) distinguished between the
designer’s and the researcher’s conceptualisations.
At this point, we think it is important to stress the idea that, not in all cases of cases of
design, it is possible to distinguish between the designer’s and the researcher’s conceptualisations.
In some cases, in the design teams there is not and expert “scientist” interested in the study of the
mental model. In other words, the image of the expert in Experimental and Cognitive Psychology
dedicated to the study of mental models and collaborating directly in the design process, very often
is not a real one. Instead, during the design process designers pay very little attention of the way
the future user of their system understands and interacts with the system. According to us, it is
really necessary to include experts in mental models and human computer interaction in the design
teams or at least, designers should be trained to be able to understand the mental model of the
user. Fortunately, technology companies such as Apple Computer, Philips Design, Bell Labs, and
Xerox’s Palo Alto Research Center seeking to design more usable, compelling products, and
services are beginning to tap the human behaviour expertise of psychologists, anthropologists and
other social scientists in growing numbers (Bailey, 1996, Lear, 2000).
With the same pragmatic-practical approach of Van der Veer (1990), and based on
observations on a variety of tasks, Norman (1983) described the properties of the mental models
that may be relevant in HCI:
44
Mental models of incidental human-machine interaction
a. Mental models are incomplete. Mental models are constrained by such things as the
user’s background, expertise, and the structure of the human information processing
system.
b. Mental models have vague boundaries. This implies that operations and systems with
certain relations or similarities can be mixed up (e.g. the mental model of an operational
system and an application program used in the operational system can be mixed up).
c. Mental models are unstable over time. People forget the details of the system they are
using, especially when those details (or the whole system) have not been used for some
period. And, as long as mental models are naturally evolving models, they change not only
because people forget and mix up details but also because new information is
incorporated over the time and through the interaction with the system. So learning and
forgetting are two sides of the same coin.
d. Mental models are unscientific and contain aspects of superstitions. For example,
people maintain behaviour patterns even when they know they are unneeded.
e. Mental models are difficult and with restrictions to run.
f. Mental models are parsimonious. Users tend to trade off extra physical actions for
reduced mental complexity.
But when we try to assess such a representation it is necessary to go further. It is
necessary to have some insight of the processes and representations involved in the construction
of a mental model or using the term of Norman, it is necessary the scientist’s conceptualisation of
the mental model. But as we saw, the problem is the absence of agreement between theories and
even definitions of the concept. According to Cañas and Antolí (1998), the fact that researchers
have used the concept of “mental model” in different research domains is the main reason of the
disagreements. Johnson-Laird (1983) worked in reasoning, while Norman (1983) used the concept
in the field of Human-Computer Interaction. As a consequence, researchers focused in different
aspects of mental models. Some give more importance to the information extracted by perceptual
processes of the characteristics of the task (e.g. Johnson-Laird, 1983) and some pay more
attention to the knowledge the person has stored in Long-Term Memory (LTM) about the
operations and the structure of the system (e.g. Norman, 1983). As a result, a common debate in
45
Mental models of incidental human-machine interaction
the literature is the question about if mental models are structures of knowledge in LTM or
temporary creations in Working Memory (Van der Veer et al., 2000).
Cañas and Antolí (1998) unified both definitions and developed a model of mental model
formation. The basic hypothesis that they maintain is that a mental model is a dynamic
representation created in Working Memory (WM) by combining information stored in Long-Term
Memory (LTM) and characteristics extracted from the environment (see figure 2). The knowledge
from LTM is instantiated into a mental model in WM based on triggering events from the situation.
This instantiated mental model is dynamic and functions as a "runnable" model in WM.
task
artefact
Output
system
Other
cognitive
and
non-cognitive
systems
Perceptual
system
MENTAL
MODEL
Conceptual model
of task and
artefact
Figure 2. The role and place of Mental Models during the interaction with a physical system. Adapted from Cañas &
Antolí (1998).
5. Assessing mental models
Pathfinder and Teach-back methods have been used in order to assess mental models.
For example, Bajo et al., (1988) used the Pathfinder Algorithm to study the differences between
experts and novices in the domain of physics. These authors found that the way people categorize
and group the relevant concepts of the domain changes as a function of expertise. On the other
46
Mental models of incidental human-machine interaction
hand, Van der Veer (1990) used the teach-back protocols to study individual differences between
users of computer systems.
5.1. Teach-back
This technique was developed by Pask in the framework of the “conversation theory”
(Pask & Scott, 1972). Later, teach-back has been extended by Van der Veer (Van der veer, 1990)
as a hermeneutic method to provoke the user to externalise his/her mental model or
representation(s) of the system. Using this method, participants are asked (individually) to teach to
an imaginary “colleague”, or person with some relevant role, how to solve the problem stated in the
teach-back question. To respond the teach-back question participants can write, make diagrams,
drawings (video can be used as well)… and in this way, subjects are encouraged to externalise
their knowledge about the system. The answered protocols that result form the teach-back
questions are scored along scoring categories in order to get insight in the representation of the
problem space. The function of the categories is to explore the mental models of the participants
on characteristics that are relevant for the research questions of the specific study. Among other
utilities, teach-back method is suitable for detecting individual differences in mental representations
(Kok, 1997; Van der Veer, 1990).
5.1.1. One example in physics
Van der Veer et al. (1999) used the teach-back method to study mental models in the
domain of physics. They explored the individual differences in mental models and examined
whether these differences could be systematically related with the level of expertise. Three teachback questions that triggered different kind of answers were designed for the study. The first
question, “real life” did not refer to the physics as dealt in education. The goal of this question was
to investigate if the subjects apply physics laws and principles in real life. The second teach-back
question, “general classroom”, explicitly refered to the domain of physics as presented in
education. The third teach-back question, “actual exercise”, implied to solve a physics problem as
dealt in educational settings.
47
Mental models of incidental human-machine interaction
Sixty-eight novices (37 male and 31 female) and 18 experts (6 female and 12 male)
participated in the study. The answered protocols were scored using the categories “memory”,
“theoretical explanation”, “internal experiment”, “metaphor”, “concept”, “procedure”, and “formula”.
And to investigate expertise effects on the scored categories, an analysis of variance (ANOVA)
was carried out. The authors found significant differences between experts and novices in the
categories “theoretical explanation”, “internal experiment” and “procedure”. The results showed that
novices scored lower in these categories than the experts. Other relevant results were find when
they analyzed separately the data obtained from the three teach-back categories. The results from
the “general classroom” showed experts did not score in the “formula” category while a lot of
students did. These results seem to show that novices often think in terms of formula when an
“educational” question is asked. On the other hand, experts scored in all of the categories while
performing the “real life” question showing that these kind of problems stimulate “physic” reasoning
in experts.
In general, the authors interpreted their results as a probe or the usefulness of teach back
as an approach to investigate expert’s and novice’s mental models.
5.1.2. One example in designing computer systems
A very interesting research for our purposes is the one made by Mulder (2000). Mulder
used the teach-back method in order to explore the mental models participants created after being
confronted to envisioned new information systems using scenarios. He used two kind of teachback questions:
- “what is…”. The main goal of this kind of question was to reveal insight in the
conceptual and semantic knowledge the participant has about the system.
- “how to…”. The goal of this question is to explore the knowledge the procedural
knowledge the user has about the system.
From the teach-back protocols Mulder obtained a set of suitable categories relevant for
designers: “interpretation of functionality in relation to goal”, “affective reactions”, “lack of
48
Mental models of incidental human-machine interaction
recognition of new functionality”, “functionality assumptions”, “dialogue assumptions”, and
“implementation assumptions”. And using these categories, he scored the teach-back protocols of
a group of 30 subjects. In the next phase of his study, the scored data were shown to expert
designers. The most relevant result of his study was the fact that designers found that the
information extracted from the future user, the mental model, could be very useful during the
design process; especially in an early stage of the design.
5.2. Pathfinder
Another technique that has been widely used in the study of mental models is the
pathfinder algorithm created by Schvaneveldt and co-workers (Schvaneveldt, 1990; Schvaneveldt
et al., 1985). Pathfinder is a graph theoretic technique that derives network structures from
proximity data. In the network concepts are represented as nodes and relations between concepts
are represented as links between the nodes. A weight corresponding to the strength of the
relationship between two nodes is associated with each link. Concepts can be directly linked or not.
The algorithm searches through the nodes to find the closest indirect path between concepts. A
link remains in the network only if it is a minimum length path between the two concepts.
As described by Van Engers (2000), the method consists of several steps:
1. To elicit concepts and create a concept list.
2. To present the concept list to the subjects to be assessed in the form of a pair-wise
comparison test or as a sorting task.
3. To create graphs that represent the concepts and relationships.
4. To calculate the characteristics of these graphs.
5.2.1. One example in physics
Empirical data reflects that differences between experts and novices can be caused in part
by the Conceptual Organization that solvers have in memory (Gonzalvo et al., 1994). To describe
changes in mental models in physics as a function of expertise. Bajo et al., (1998) used the
Pathfinder method combined with the SCAN method (Cook,1992) to label the links between
concepts specifying the nature of the relation.
49
Mental models of incidental human-machine interaction
Comparisons between students and experts showed important differences in the way they
scattered the concepts around the network. Students seem to organized the cinematic concepts
close in the network with velocity as central concept whereas experts scattered cinemantic
concepts around the network, close to concepts that define dynamic principles or energy states.
Another important difference is the role of mass. Students seem to relate mass directly to force,
whereas experts relate it to energy states. Both, the importance of dynamic principles and the
connection mass-energy implies a sophisticated understanding of the concept.
Summarizing, experts represent different relations among concepts than novices but the
organization is flexible and dependent on the context. When novices learn about the domain their
Conceptual Structure become more similar to that of the experts. In their third study students
solved problems that emphasized dynamic aspects. After solving the problems, students were
similar to the experts in the importance they assigned to dynamic principles. However, as cluster
analysis showed, students still stress kinematics as an organizing principle while experts did not.
The authors concluded that instructional practice can make use of this flexibility to elicit physic
principles appropriate for solving problem. The results of our study indicate that if the context
encourage students to think to sophisticated physics principles, they are able to do it.
5.2.2. One example in computer systems
Klok (1998) performed another interesting study with the pathfinder technique. Klok
studied the mental model that subjects develop from working with an Information Retrieval (IR)
system. The search-facility of the IR system was based on concepts and relations between them.
Four groups of subjects learned to search among the IR system. The difference between groups
was the representation used for the interaction between the subject and the system: “formal”,
“graphical”, “natural language” and a “combination” of the graphical and natural language
representation. The comparison of the differences in performance and accuracy of the mental
models between the four groups showed that a combination of natural language and a graphical
representation yields better overall results than a single representation.
50
Mental models of incidental human-machine interaction
5.3. The limitations of both techniques
5.3.1. The teach-back limitations
One of the most important limitation of teach-back is that is a descriptive method that does
not provide explanations. And the method is in principle only of use for getting information on
knowledge of which the user is aware, and which can in any way be represented in writing or
drawing. Furthermore, teach-back requires more than a straight forward application of the
technique. Both the instructions and the interpretation require experience with hermeneutic
analysis. However, Van der Veer (1990) has shown that the technique can be learned and that the
results after training have an acceptable level of reliability (e.g. Kok, 1998; Van der Veer et al,
2000)
Another important aspect to consider while using teach-back is that the interpretation
requires some methodological considerations. And the most important one is that the
representation obtained shows only aspects of mental model build after the particular teach-back
question.
5.3.2. The pathfinder limitations
Schvaneveldt (1999) claims that the mental maps represented by the pathfinder algorithm
represent the concepts and relationships between concepts as they reside in LTM however, the
method does not give guidelines for the selection of the concepts.
Another problem is that Pathfinder needs the subjects to make pairwise estimations of the
degree of similarity or relatedness for the concepts in the domain-relevant concept set. Each
concept is presented with all other concepts so the subjects has to make n(n-1)/2 similarity
judgments. The problem is that the higher the number of relevant concepts, the higher the number
of pairs of concepts that the subjects have to judge. For example, for 30 concepts there are
(30*29)/2 = 435 pairs of concepts. Because it could be very tiring for the participants, Goldsmiths et
al (1991) suggested not to use more than 30 concepts.
51
Mental models of incidental human-machine interaction
6. Our current approach
Based in previous works (e.g. Mulder, 2000; Van der Veer, 1990) we suggest to apply
envisioning scenarios and analyze mental representations early in the process. More explicitly,
designers have to consider the different UVM’s or relevant aspects of the system that are relevant
for the different groups of potential users of the system. And as long as envisioning systems, it is
important to evaluate, in an iterative process (see figure 1), all of the possible features that can be
a result of the interaction with the new system. In doing so, it is important to use scenarios and to
explore if the mental representations users build through their interaction with the UVM are the
ones that allow an appropriate interaction in order to reach the desired goals.
The next question is how to explore those mental models. Although both techniques have
limitations, we think that a combination of both can give complementary information. Following the
proposal of Cañas and Antolí (1998) we consider that the mental model is a dynamic
representation created in Working Memory (WM) by combining information stored in Long-Term
Memory (LTM) and characteristics extracted from the environment (see figure 2). As we said, the
knowledge from LTM is instantiated into a mental model in WM based on triggering events from the
situation. And, as Cañas and Antolí proved, the specific information extracted from LTM depends
of the context and of the task demands. On the other hand the knowledge can be both conceptual
and procedural.
On the other hand, teach-back is a technique that focuses on instantiated knowledge (Van
der Veer, 1990: Van der Veer et al., 2000). This means that the type of representation it focuses on
may differ dependent on the instruction. As a matter of facts, the “what is” question seems to focus
on the conceptual knowledge that is relevant to the situation while the “how to” focus on the
procedures. Our hypothesis is that using this questions we will extract the relevant concepts (both
semantic and procedural) that are relevant for the specific situation and that can be later used to
analyze the way they are organized (in this specific situation as well) using the pathfinder.
References
52
Mental models of incidental human-machine interaction
Bailey, R. W. (1996). Human performance engineering: Designing high quality, professional user
interfaces for computer products, applications, and systems. Prentice Hall PTR.
Bajo, M. T., Gonzalvo, P., Gómez-Ariza, C. & Puerta-Melguizo, M. C. (1998). Changes in
categorization as a function of expertise and context in elementary mechanics. X ESCOP
Conference. European Society for Cognitive Psychology. Jerusalem (Israel).
Cañas, J. J. & Antolí, A. (1998). The role of working memory in measuring mental models. In:
T.R.G. Green, L. Bannon, C.P. Warren & J. Buckley (Eds.). Proceedings of the Ninth
European Conference on Cognitive Ergonomic–Cognition and Cooperation. EACE. INRIA.
Rocquencourt.
Carroll, J. M. (Ed.) (1995). Scenario-Based Design. John Wiley & Sons Inc.
Cook, N. J. (1992). Eliciting semantic relations for empirically derived networks. International
Journal of Man-Machine Studies, 37, 721-70.
Goldsmith, T. E., Johnson, P. J., Acton, W. H. (1991). Assessing structural knowledge. Journal of
Educational Psychology, 83, 88-96.
Gonzalvo, P., Cañas, J. J. & Bajo, M. T. (1994). Structural representation in knowledge acquisition.
Journal of Educational Psychology, 4, 601-616.
Johnson-Laird, P. N. (1983). Mental models. Cambridge University Press.
Klok, J. A. (1998). A needle in a haystack. The design of an interface that guides the user through
the complex conceptual space of an informational retrieval system. Master Thesis.
University of Twente.
Kok, E. (1998). Mental Models and Individual Differences. A research method applied to the
domain of Physics. Master Thesis. Vrije Universiteit, Amsterdam.
53
Mental models of incidental human-machine interaction
Lear, A. C. (2000). Uncovering technology’s human side. Computer. Innovative Technology for
Computer professionals, 7, 24.
Newell, A. & Card, S. K. (1985). The prospects for psychological science in human-computer
interaction. Human-Computer Interaction, 1, 209-242.
Norman, D. A. (1983). Some observations on mental models. In D. A. Gentner, & A. A. Stevens
(Eds.). Mental models. Hillsdale, NJ: Erlbaum.
Norman, D. A. (1986). Cognitive engineering. In D. A. Norman & S. W. Draper (Eds.). UserCentered Design: New perspectives in Human Computer Interaction. Hillsdale, NJ: LEA.
Norman, D. A. (1988). The psychology of everyday things. New York: Basic Books.
Mulder, B. (2000). The role of mental models in designing computer systems. Master Thesis. Vrije
Universiteit, Amsterdam.
Pask, G. & Scott, B. C. E. (1972). Learning strategies and individual competence. International
Journal of Man-Machine Studies, 4, 217-253.
Sasse, M. A. (1997). Eliciting and Describing Users’ Models of Computer Systems. Thesis
submitted to the Faculty of Science of the University of Birmingham.
Schvaneveldt, R. W. (1990). Pathfinder associative networks: Studies in knowledge organization.
Norwood, New Jersey: Ablex Publishing Corporation.
Schvaneveldt, R. W., Durso, F. T. Goldsmith, T. E., Breen, T. J., Cook, N. M., Tucker, R. G. &
DeMaio, J. C. (1985). Measuring the structure of expertise. International Journal of ManMachine Studies, 23, 699-728.
54
Mental models of incidental human-machine interaction
Tauber, M. (1988). On mental models and the user interface. In: G. van der Veer, T. R. G. Green,
J. M. Hoc & D. M. Murray (eds.). Working with computers: Theory versus outcome.
London: Academic Press.
Van der Veer, G. C. (1990). Human-Computer Interaction: learning, individual differences, and
design recommendations. Thesis submitted to the Faculty of Sciences of the Vrije
University of Amsterdam.
Van der Veer, G. C., Chisalita, C. & Mulder, B. (2000). The conceptualization of mental models as
base for the desing of interactive systems and situations. Paper proposal for ICCM-2000.
Van der Veer, G. C., Kok, E. Bajo, T. (1999). Conceptualising mental representations of
mechanics: A method to investigate representational change. In: D. Kayser & S.
Vosniadou. Modelling changes in understanding: case studies in physical reasoning.
Pergamon.
Van der Veer, G. C. & Van Welie, M. (2000). Task Based Groupware Design: Putting theory into
practice. Proceedings of DIS 2000, New York, United States.
Van der Veer, G. C., Tauber, M. J., Waern Y., & Van Muylwijk, B. (1985). On the interaction
between system and user characteristics. Behaviour and Information Technology 4, 284308.
Wickens, C. D. & Hollands, J. G. (2000). Engineering psychology and human performance.
Prentice Hall.
55