A Systemic Approach to Usability in Mixed Reality Information Systems Authors:
Susanna Nilsson, PhD student
Björn J. E. Johansson, PhD, Assistant professor
Adress:
Human-Centered Systems
Institution of computer and Information Science
Linköping University
SE – 583 31 Linköping
Fax:
+46 (0) 13
Telephone: +46 (0) 13 28 26 90
Email:
{susni, bjojo}@ida.liu.se
Biographical sketch of the authors
Susanna Nilsson is a PhD Student in Cognitive Systems at the division for Human-Centered
Systems at the department of Computer and Information Science, Linköping University. She
is currently in her second year of study, and she is studying usability and design of
applications in Mixed, and Augmented Reality Systems. The main supervisor for the research
is Professor Erik Holllnagel.
Previous publications:
Nilsson, S. and Johansson, B. A Cognitive Systems Engineering Perspective on the Design of
Mixed Reality Systems. Proceedings of the 13th European Conference on Cognitive
Ergonomics September 20-22, Zürich 2006
Björn Johansson is an Assistant Professor at the Cognitive Systems Engineering Laboratory
at the division for Human-Centered Systems at the department of Computer and Information
Science, Linköping University. Dr Johansson works in projects related to crisis management,
traffic, automation and command and control systems. His research interests concerns
Cognitive Systems Engineering, Communication, Resilience Engineering and behavior in
complex systems. He is also the co-supervisor of the first author of this paper.
Selection of Previous publications:
Johansson, B. & Hollnagel, E. (2007) Pre-Requisites for Lage-Scale Coordination. Cogntion,
Technology & Work. Vol 9 (1), pp 5-13.
Johansson, B., Trnka, J. & Granlund, R. (2007) The Effect of Geographical Information
Systems on a Collaborative Command and Control Task. In (Eds.) B. Van de Walle, P.
Burghardt and K. Nieuwenhuis, Proceedings of ISCRAM 2007, Delft, the Netherlands, 191201.
Johansson, B., Granlund, R. & Waern, Y. (2005). Research on Decision Making and New
Technology - Methodological Issues. In (Eds.) B. Brehmer, R. Lipshitz, & H. Montgomery,
How Professionals Make Expert Decisions, Lawrence Erlbaum Associates, Mahaw, New
Jersey.
Johansson, B., Artman, H. & Waern, Y. (2001) Technology in Crisis Management Systems ideas and effects. Document Design. Journal of research and problem solving in
organizational communication special issue, Pragmatics in Crisis, Vol 2 (3), pp 247-258.
A Systemic Approach to Usability in Mixed Reality Information Systems Abstract
In this paper we describe a systemic approach to usability in information systems, by applying
a cognitive systems engineering perspective (CSE) on usability analysis of Mixed Reality
(MR) Systems. MR systems are general information systems designed for specific
applications. Information systems have a global distribution and usability methods for design
of such systems and interfaces are based on traditional human computer interaction theories
assuming a decomposed view of the user and interface. CSE approaches system design by
viewing the system as a whole, focusing on the goals of the joint cognitive system. CSE is a
holistic approach to usability, allowing more flexibility in methodology than traditional HCI
guidelines. The conclusion is that CSE is an appropriate foundation for usability analysis and
design of information systems such as MR systems, which are global, and general in the
technical solution, but local and specific in their real world application.
Conference theme: Information Systems, Systems theory, Systems thinking
Keywords: Information Systems, Cognitive Systems Engineering, Joint cognitive systems
Mixed/Augmented Reality, Soft systems methodology
A Systemic Approach to Usability in Mixed Reality Information Systems Introduction
Today many technologies, such as computer software, have global distribution with users all
over the world in many different cultures. Technologies developed for specific purposes have
also become more widespread and used in different domains by different social and economic
groups. When new technologies are introduced into a domain it may affect the user and task
on both a practical and a social level. A technical system or interface should have as much
positive effect on the user and her/his surroundings as possible, while at the same time
minimizing the negative effects of the system both for the user and other individuals,
ultimately being beneficial to society in general. Traditional usability guidelines are often
restricted to the technical system, ignoring the context in which the system is used, as well as
the effect the system or interface may have in shaping the same context. When designing an
interface the developer needs a good perception of who the user is and where and how the
system can and should affect the user in her tasks.
Mixed Reality (MR) systems are information systems that merge real and virtual information
with the purpose of aiding users in different tasks (Azuma 1997). A MR system is a general
system much like a computer is general – it has potential as a tool for many different purposes
in many different situations. This means that MR systems should be flexible enough to allow
users to define the application to the specific cultural and social context. Seeing that MR
systems are as diverse in application as the term information system, studying usability in
these systems on a general level is impossible, or at least bound to provide little practical
value. The literature on usability in MR systems is limited which makes it interesting to
explore new ways of addressing design of MR systems. Traditional human computer
interaction (HCI) research makes assumptions about the interaction between user and system
that rarely apply to MR systems. Instead we propose using a systemic approach to usability in
mixed reality information systems – the Cognitive Systems Engineering (CSE) perspective
(Hollnagel & Woods, 1983). The purpose of this paper is to describe how this approach can
be used to formulate a general method for studying usability in information systems.
Background and related research: Systemic approaches and CSE
Traditional HCI is based on a scientific tradition of hypothesis testing to ensure scientific
validity of its claims by allowing repeatability of the results of usability studies. However, the
repeatability of studies involving human activity has been seriously questioned by among
others Simon (1956) as well as in the Soft Systems Methodology (Checkland & Holwell
1998). The foundation of soft systems methodology is “learning from intervention in realworld problem situations” (ibid) rather than classical hypothesis testing. In approaching
complex socio-technical systems, the basis of CSE does not differ much from the soft systems
methodology.
According to Checkland (1999) a systems approach is, much like a scientific approach, a meta
discipline. Studying something scientifically implies using a certain paradigm of methods that
are based on defined laws of nature. Studying something from a systems approach means
assuming that the world contains “structured wholes which can maintain their identity under a
certain range of conditions and which exhibit certain general principles of wholeness” (ibid ).
This mode of thinking is not new, for example, already Carolus Linnæus divided plants and
creatures in nature in a systemic fashion based on their characteristics and properties in his
‘Systema naturae’ (1758, org. 1735). This is just one of many ways of approaching our
surrounding world systematically. Numerous ways of decomposing our surrounding world
into system components have been developed, all with their own specific purpose. When it
concerns information systems, the information processing approach has been, and is,
prevailing. From this perspective, a system (human, animal or machine) is divided into a
number of components that propagates or processes data. The approach was developed by,
among others, Shannon & Weaver (1949). This idea had profound impact, not the least on the
Macy conferences (Dupuy, 2000; Heims, 1993), and its long lasting effects on the theoretical
foundation of cybernetics, cognitive science, computer science and design cannot be ignored
by anyone working in these fields, although often are forgotten in contemporary texts.
The most important consequence of the information processing paradigm is probably that it
has focused largely on structural descriptions focusing on what something is rather than what
it does. Although these two directions in many respects are inseparable, the way of
approaching a problem, or even defining what the problem is, differs greatly. In ‘traditional’
HCI, the interface and the user are almost always seen as separate entities that exchange input
and output. The focus of design based on this approach has consequently been improving that
interaction as much as possible. Little thought is given to the task that is actually going to be
performed by the user when using the program/tool beyond the interface. In contrast, a
functional perspective would instead begin by asking what the purpose(s) or goal(s) of the
system is and what the pre-requisites for achieving that goal(s) are. One approach using this
mode of thinking is Cognitive Systems Engineering (Hollnagel & Woods, 1983; 2005).
Hollnagel and Woods suggested that a ‘cognitive system’ should be described in terms of
functions rather than hypothetical structures of information processing. By approaching the
study of humans and machines from this perspective, CSE avoids the basic problems of
detailed modelling of internal structures and can focus on the performance of a controlling
system in its context.
“Instead of viewing an MMS 1 as decomposable by mechanistic principles, CSE introduces
the concept of a cognitive system: an adaptive system which functions using knowledge about
itself and the environment in the planning and modification of actions.”
(Hollnagel & Woods, 1983, p 583)
More recently, a systemic approach, Joint Cognitive Systems (Hollnagel & Woods, 2005), has
been suggested. The fundament of that approach is functional decomposition, using the
cybernetic concept of requisite variety (Ashby, 1956). By ‘requisite variety’ Ashby refers to
the ability of a system to exercise control over another system. According to him, a system
can only control another system if it can present a higher degree of variety than the system
that it tries to control (only variety can destroy variety). Another way of explaining the
concept is that the ‘repertoire of performance’ that a system can present explains if he/she/it
can control a situation. Constraints are also put on the system by the environment, and
everything that is beyond the control of the system, but puts constraints on it, is considered to
be a part of the environment. The boundaries of a system can thus be defined from the
components ability to exercise control over another component, and vice versa from its
inability to do so (Hollnagel & Woods, 2005). Hollnagel & Woods uses the example of traffic
to illustrate the JCS concept (see figure 1).
1
Man-Machine System, our footnote.
Figure 1. An example of a Joint Cognitive System in traffic
A car and its driver are considered as a JCS. The driver-vehicle system can move in itself, but
its behaviour is constrained by the road (if we assume that it is a normal two-wheel drive with
a ‘sensible’ driver). The road, and the traffic on it, is in turn a part of the traffic infrastructure,
which is constrained by the topography in which it is located. Lastly, the weather will affect
all of these parts and is also completely beyond driver-vehicle control. There are thus a
number of analytical levels that the researcher can chose from depending on what the research
focus is. The important thing to remember is that any driver-vehicle system under study is
part of a larger context that shapes its behaviour. The same conclusion is valid for any usersystem configuration. Goals and variability takes place on several different system levels, and
will thus affect and be affected by other levels. Where to begin and end the analysis must be
judged both by the purpose of the investigation as well as pragmatic aspects.
A holistic approach to analyzing Mixed Reality Systems
Mixed reality Systems are in general information systems that merge the real world with
elements of virtual information to varying extent. Milgram and Kishinos (1994) virtual
continuum is often used to describe the relation between augmented reality, virtual reality and
the stages in between. Mixed Reality is the
collective name for all the stages (see
figure 2). Azuma (1997) mentions three
criteria that have to be fulfilled for a
system to be classified as a Augmented
Reality system: they all combine the real
and the virtual, they are supposedly
interactive in real time (meaning that the
user can interact with the system and get
response from it without delay), and they
are registered and aligned in three
dimensions. Technically, there are two
principally different solutions for merging Figure 2: The Virtual Continuum (figure from
reality and virtuality in real time today – Gustafsson et al 2003 after Milgram and Kishino, 1994)
video see-through and optic see-through (Azuma 1997; Kiyokawa 2007).
According to the CSE approach, one should look at the goal of completing a task as a joint
cognitive system instead of focusing on the merging of the real and the virtual. The task might
be repairing an engine – the human is supported by the virtual elements presented by the
technology. In this section two different aspects of a JCS approach to MR system are
discussed, first the descriptions of MR systems – where the boundaries between system,
human and artifacts are drawn, and secondly the use of the MR systems as a tool or a
prosthesis.
Describing Mixed Reality systems in relation to usability
The definitions of MR systems found in the literature to date are examples of the natural
scientific methodology and this may affect the structural or functional decomposition of MR
systems. In general MR systems are described in very technical terms with details of
components and their functions (see figure 3). Often the hardware and software are described
separately which confirms the clear boundaries within the system.
Figure 3 Decomposing the MRS to hard and software.
The technical description is needed for the repeatability of the studies, but when approaching
usability in these information systems other types of descriptions are necessary. Articles from
the MIXER 2004 workshop (a part of the larger Intelligent User Interfaces conference) claim
to have a usability approach as can be seen from the quotation from this paper:
”…our research aims at providing elements useful for the design of usable MR systems by
focusing on the interaction between the user and the MR system” (Mansoux et al, 2004)
Despite this aim the descriptions and decomposition of the MR systems in the articles all
make the same distinctions between system and components, but adding the user (see figure
4)
Figure 4 The user/s separate from the system
Figure 4 illustrates a traditional usability approach to human-system interaction. The system
and the human working with the system are separate entities and are also analyzed separately.
The systems output becomes the users input and vice versa, leaving two separate input-output
systems that are studied as information processing units (Hollnagel & Woods 2005). One
paper does include a functional, rather than structural decomposition of the MR system but
the physical boundaries are still intact (see figure 5).
Figure 5 Decomposition of a MRS with functional division between tracking (upper left rectangle) and
visualization (lower rectangle) and HMD and robot (upper right square). Still the image shows clear physical
boundaries (Klinker et al 2004).
Typical of traditional HCI the context of the system is not included in the MR system
description. This excludes discussion on how the environment and experimental setup may
have affected the use of the MR system (figure 6). A majority of studies performed on
usability of MR systems so far are laboratory or experimental studies, which may be the
reason for omitting the context dimension is from the discussion. The focus remains on the
individual artifacts and users in the system and their individual performance, not the
performance of the system as a whole – the joint cognitive system.
Figure 6 The MR system as apart of a joint cognitive system
Applying a different aggregation level
Analyzing MR systems from a JCS perspective means approaching usability in the system
from a different level. In this paper we propose to approach the MR system by defining the
system based on the goals of the use and experience of the system as opposed to defining it
from its technical and structural components. The terminology used in papers on MR systems
is often based on an assumption that the MR system is the ‘system’ which the user ‘interacts
with’. However it may be more useful to look at the purpose of the system and the fact that
the user is actually interacting through the system and not solely with the system.
If the goal in fact is to interact with the world via the MR system, then one could base the
decomposition on the interaction technique, instead of individual technological components
(Mackay, 1998).
As seen in figure 4 the boundary between the human user and the artifact is based on the
assumption that the human ends where the artificial system begins – there is an obvious
division between display and the users’ eyes. Even within the artifact the boundary is drawn
based on physics – software and hardware (see figure 3). But this does not have to be the case.
The boundary could be drawn differently, allowing the camera and display to be part of the
human perception, as an extension of the abilities of the user. This boundary also coincides
with a boundary drawn when defining the system based on functionality (i e what the system
component does) instead of structure (e g hardware and software). The function of the
camera, and display in combination with the users visual perception is to convey an image or
other virtual information to the user, who then takes action. The function of the computer,
processor and code is to generate and execute commands, most often in response to user
action, changing the visual appearance in the display and thus the users’ perception of the
world.
Mixed Reality Systems as artifacts - tool or prosthesis?
In video see through MR the user experiences the world only through the artifact, while the
optic see through should have more of an ‘embodiment relation’, in the same way a pair of
glasses act as an extension of our eyes. In a video see through system, the system takes on the
so called amplifier hermeneutic relation (Hollnagel & Woods 2005), where the artifact is an
interpreter and takes care of all communication between the operator and the application. The
MR system is not transparent to the user, meaning that all information is filtered through the
system and then projected on the display, with possible additional information, to the user
who has little control over it.
In MR systems used for technical support the system in itself could be viewed upon as a tool
for retrieving and presenting information, but if it is used for example as a part of distance
expert guidance it can be considered an extension of the expert assistant’s eyes and arms
when giving instructions in the operator’s field of view.
Many of the definitions used to describe MR systems rely on the physical boundaries between
the parts of the system which illustrates a tool view on the combined MR system. The person
in the system is using the system to retrieve information or solve problems. By not focusing
on the physical boundaries, but instead using a JCS approach, focusing on the common goals
of the system, it can be seen as a prosthesis or an extension of the human system capabilities.
The interaction with the environment becomes the tool for achieving the goals, goals such as
repairing an engine or starting up complicated technical devices. The MR system can be
viewed upon both as a tool and as a prosthesis depending on the context and purpose of using
the system. The question is if this affects the development and the design of the system as it
very well should.
Concluding Discussion
MR-systems do not always have clearly defined user groups, however they are most often,
designed to complete certain tasks in specific situations. This becomes obvious if we consider
that two MR systems that are identical in terms of the technical solution (hardware) can be
used in very diverse settings like industry, military or healthcare. This makes the CSE
approach appropriate for studying usability in MR-systems, since, as in CSE, the task, goals
and performance of the user and system are more important than the details of the users’
internal processes and interaction with the artifact. Rather, the composition of user and artifact
should be studied in relation to the goals of the activity in which they are involved. By
applying a systemic perspective based on functional decomposition, these goals become
manifest, guiding the designer in his/her quest. The JCS approach helps the designer identify
the various levels of analysis in a specific case, which is needed if one is to understand how
control relationships are affected by for example the introduction of an MR-system. MR is
also particularly challenging because of the tools/prosthesis crux. Since an MR-system
provides input directly in the user’s perceptual and auditory fields, it both expands and limits
the user’s capacity. It expands it by directing attention and providing instructions, and it limits
it in exactly the same way: when the user is given indications on what to focus on, attention is
also taken away from other things. This is also why it is extremely important of a designer to
consider the actual use context – a limited understanding of the context may have severe
outcomes if the MR-system changes the task at hand in such a way that MR-Human-JCS
performs erroneous. This is an apparent risk if a decomposed approach focusing solely on
human-MR interaction is applied. The CSE/JCS approach is an effort to avoid this by utilizing
a systemic functional analysis of the application domain. Our conclusion is thus that CSE is a
useful approach in addressing design and evaluation of information systems and this paper has
exemplified this by an analysis of an information system that could be used in a wide range of
disciplines.
References
Ashby WR (1956) An introduction to Cybernetics. Chapman & Hall, London.
Azuma, R. (1997) A survey of Augmented Reality. Presence: Teleoperators and Virtual
Environments. 6 (4), August. p355-385
Checkland, P. and Holwell, S. (1998) Information, Systems and Information Systems. Making
sense of the field. ISBN 0 471 95820 4
Checkland, P. (1999) Systems Thinking, Systems Practice. ISBN 0 471 986062
Dupuy, J-P. (2000) The Mechanization of the Mind: On the Origins of Cognitive Science,
ISBN-10: 0691025746
Heims, S.J. (1993) Constructing a Social Science for Postwar America: The Cybernetics
Group, 1946 – 1953, ISBN-10: 026258123X
Hollnagel, E. & Woods, D.D (1983) Cognitive Systems Engineering: New wine in new
bottles. Int. Journal of Human-Computer Studies 51, p 339-356
Hollnagel, E. & Woods, D.D. (2005) Joint Cognitive Systems. Foundations of Cognitive
Systems Engineering. ISBN 0 8493 2821 7
Linnæus, C. (1758) Systema naturæ per regna tria naturæ, secundum classses, ordines,
genera, species, cum characteribus, differentiis, synonymis, locis. 10th edition, Available at:
http://dz-srv1.sub.uni-goettingen.de/cache/toc/D265100.html.
Kiyokawa, K. (2007) An Introduction to Head Mounted Displays for Augmented Reality. In
Haller, M., Billinghurst, M., Thomas, B. (eds): Emerging Technologies of Augmented Reality:
Interfaces and Design. ISBN 159904067 0
Klinker, G. et al (2004) FixIt: An approach towards assisting workers in diagnosing machine
malfunctions Proceedings of the IUI-CADUI Workshop on Exploring the Design and
Engineering of Mixed Reality Systems (MIXER). Funchal, Island of Madeira, January 13th
Mackay, W.E. (1998) Augmented Reality: linking real and virtual worlds. In Proceedings of
ACM AVI '98, Conference on Advanced Visual Interfaces. L'Aquila, Italy: ACM. Keynote
address. pp. 1-9.
Mansoux, B., Nigay, L. & Traccaz, J. (2004) Abstract and concrete interaction with Mixed
Reality systems. The case of the mini-screen, a new interaction device in computer assisted
surgery. Proceedings of the IUI-CADUI Workshop on Exploring the Design and Engineering
of Mixed Reality Systems (MIXER). Funchal, Island of Madeira, January 13th 2004.
Milgram, P. & Kishino, F. (1994) A taxonomy of mixed reality visual displays. IEICE
Transactions on Information Systems, Vol E77-D, No 12 December 1994
Nilsson, S. and Johansson, B. A Cognitive Systems Engineering Perspective on the Design of
Mixed Reality Systems. Proceedings of the 13th European Conference on Cognitive
Ergonomics September 20-22, Zürich 2006
Warren Weaver and Claude Elwood Shannon (1963). The Mathematical Theory of
Communication. Univ. of Illinois Press. ISBN 0252725484.
Simon, H. A. (1956) Rational Choice and the structure of the Environment, Psychological
Review, vol 63, pp 129-138