Acceptance of Augmented Reality
Instructions in a Real Work Setting
Susanna Nilsson
Department of Computer and
Information Science
Linköpings Universitet
SE- 58231 Linköping
susni@ida.liu.se
Björn J.E. Johansson
Combitech AB
Teknikringen 9
SE-583 30 Linköping
bjorn.j.e.johansson@combitech.se
Abstract
The differences between Augmented Reality (AR) systems
and computer display based systems create a need for a
different approach to the design and development of AR
systems. To understand the potential of AR systems in real
world tasks the technology must be tested in real world
scenarios. This case study includes two qualitative user
studies where AR was used for giving instructions to users in
a hospital. The data show that the users in the context of
medical care are positive towards AR systems as a
technology and as a tool for instructions in terms of
usefulness and social acceptance. The results indicate that AR
technology, should it be introduced as technical support in
this context, may become an accepted, and appreciated part
of every day work.
Keywords
Augmented Reality, human factors, user interaction, usability
methods
ACM Classification Keywords
H5.m. Information interfaces and presentation (e.g., HCI):
Miscellaneous. H.5.1 Multimedia Information Systems:
Artificial,
Augmented
and
Virtual
Realities,
H.1.2
User/Machine Systems: Human Factors
Copyright is held by the author/owner(s).
CHI 2008, April 5 – 10, 2008, Florence, Italy
ACM 978-1-60558-012-8/08/04.
Introduction and related research
In many areas today organizations and companies are facing
many challenges with ever-increasing demands on a limited
number of resources. For instance, training and educating of
new staff is both a time demanding and people consuming
task. Introducing new ways of teaching and training could be
one way to reduce the strain on human resources. The field
of Mixed Reality (MR) technology (including Virtual Reality,
VR, and Augmented Reality, AR) has the potential to allow for
training and exercise in a less resource demanding manner.
Mixed and Augmented Reality
Figure 1. The Virtual Continuum
(after [8]).
The field of Mixed Reality (MR) is a relatively new field in
terms of commercially and publicly available applications. As
a research field it has existed for almost two decades with
applications in diverse domains, such as medicine, military
applications, entertainment and infotainment, technical
support and industry applications, distance operation and
geographic applications [1, 2]. The aim of AR interfaces is the
“merging of worlds” by adding virtual information to the real
world. For a system to be classified as an AR system, the
system has to fulfill three criteria according to Azuma (2001):
they all combine the real and the virtual, they are supposedly
interactive in real time, and they are registered and aligned
in 3D [1], [2]. Milgram and Kishinos (1994) virtual
continuum is often used to describe the relation between
augmented reality, virtual reality and the stages in between
[8]. Mixed Reality is the collective name for all the stages
(see figure 1).
To successfully integrate new technologies into an
organization or workplace means that the system, once in
place, is actually used by the people it is intended for. There
are many instances where technology has been introduced in
organizations and then not been used for a number of
different reasons. One major contributor to the lack of usage
is of course the usability of the product or system in itself.
But another issue is how well the system operates together
with the users in a social context – are the users interested
and do they see the same potential in the system as the
people (management) who decided to introduce it in the
organization? Davis describes two important factors that
influence the acceptance of new technology, or rather
information systems, in organizations [4]. The perceived
usefulness of a system and the perceived ease of use both
influence the attitude towards the system, and hence the
user behavior when interacting with the system, as well as
the actual use of the system. If the perceived usefulness of a
system is considered high, the users can accept a system
that is perceived as harder to use than if the system is not
perceived as useful. For an AR system this means that even
though the system may be awkward or bulky (head
mounted), if the applications are good, i.e. useful enough,
the users will accept it. Equally, if the AR system is not
perceived useful, the AR system will not be used, even
though it may be easy to use.
MR and AR are relatively new technologies considering
commercially available products. Few studies addressing the
potential users’ attitude towards this technology have been
done. This paper presents a case study where AR instructions
have been evaluated for use in the domain of public health
care. Public health care, just like many other domains, has an
interest in reducing resources spent on training and
educating of new staff members. In one study the use of AR
for starting up a diathermy apparatus was evaluated and in a
follow up study AR was used to assemble a small surgical
instrument. Both studies had a qualitative approach where
the purpose was to see whether AR technology would be
socially accepted by the staff at the hospital.
User acceptance case studies
Figure 2. A diathermy apparatus
prepared with fiducial markers.
Figure 3. A helmet mounted AR
system
A long term goal of AR research is for AR systems to become
fully usable and user-friendly, and as noted by Livingston [8]
and Nilsson & Johansson [12] there are problems addressing
human factors in AR systems. Research in AR including user
studies
usually
involves
quantifiable
measures
of
performance (eg. task completion time and error rate) and
rarely focuses on more qualitative performance measures. Of
course there are exceptions such as described by Billinghurst
et al where users’ performance in collaborative AR
applications is evaluated by not only quantitative measures
but also with gesture analysis and language use [3]. Another
example is the method of domain analysis and focus on task,
with user profiles presented by Livingston et al [9]. However,
regarding usability studies in the AR research domain, the
methods mainly used are based on traditional usability
methods for graphical user interfaces, sometimes in
combination with usability for Virtual Reality applications
[13]. This approach may not be the best option, since these
approaches are not based on experiences from actual AR
systems users. To investigate user acceptance and attitude
toward AR technology, two user studies were conducted
where the participants were observed using an AR system. In
the first study the participants, who were all professional
medical staff, received instructions on how to interact with a
diathermy apparatus, and in the second study the
participants received instructions on how to assemble a
relatively small surgical instrument, a trocar.
Study 1 – method
The specific aim of study 1 was to investigate user
experience and acceptance of AR systems in an instructional
application for medical equipment. A qualitative user study
was conducted onsite at a hospital. Eight participants (ages
30 – 60), all employed at the hospital, participated in the
study. Four of them had previous experience with the
diathermy apparatus, and four did not. All of the participants
had experience with other advanced technology in their daily
work. First the participants were interviewed about their
experience and attitudes towards new technology and
instructions for use. Then they were observed using the AR
system, receiving instructions on how to start up a diathermy
apparatus (see figure 2). After the task was completed they
filled out a questionnaire about the experience. The
interviews, as well as the observation were recorded with a
digital video camera. During the task, the participants’ view
through the video-see-through AR system was also logged
with a digital video camera.
Equipment.
The AR system hardware part, which can be seen in figure 3,
included a tablet computer (Fujitsu Siemens Lifebook T with
1GHz Intel®Pentium® M, 0.99GB RAM), a helmet mounted
display (Sony Glasstron) with a fire wire camera attached. A
numpad was used for the interaction with the user (see
insert, figure 3). The AR system uses a hybrid tracking
technology based on marker tracking; ARToolKit, (available
for download at [7]), ARToolKit Plus [11] and ARTag [5]).
The software includes an integrated set of software tools
such as software for camera image capture, fiducial marker
detection, computer graphics software and also software
developed specifically for AR-application scenarios [6].
The user task
The users were given instructions on how to activate (and
prepare for operation) a surgical diathermy apparatus (DA),
ERBE ICC-350™ with associated instruments and electrodes
(see figure 2 & 4). In general diathermy is a physical therapy
for deep heating of tissues with high frequency electrical
current. The ERBE ICC 350™ diathermy apparatus is used for
mono- or bipolar cutting and coagulating during invasive
medical procedures. The image in figure 4 also shows the
placement of the markers.
Figure 4. The participants view
of the AR instructions.
The instructions received through the AR system were
developed in cooperation with an experienced operating room
nurse at the hospital in a pre study. After observing and
recording the nurse as she gave instructions on how to use
the DA, the AR instructions were constructed in the same
manner. After a first version of the AR instructions was
implemented in the system, the operating room nurse had
the opportunity to give feedback and corrections. The
instructions were given as statements and questions that had
to be confirmed or denied via the input device, in this case a
num pad with only three active buttons – ‘yes’, ‘no’, and ‘go
to next step’. An example of the instructions from the
participants’ field of view can be seen in figure 4.
Data was collected both through observation and open ended
response questionnaires. The questionnaire consisted
questions related to overall impression of the AR system,
experienced difficulties, experienced positive aspects, what
they would change in the system and whether it is possible to
compare receiving AR instructions to receiving instructions
from a teacher.
Figure 5. Participant receiving
instructions from the AR system
and interacting with the DA.
Study 1 – Results
It was found that all participants but one (out of eight) could
solve the task at hand without any other help than by the
instructions given in the AR system. In general the
interviewed responded that they preferred personal
instructions from an experienced user (6/8), sometimes in
combination with short, written instructions, but also that
they appreciated the objective instructions given by the AR
system. The problems users reported on related both to the
instructions given by the AR system and to the AR
technology, such as problems with a bulky helmet etc.
Despite the reported problems, the users were positive
towards AR systems as a technology and as a tool for
instructions in this setting.
All of the respondents work with computers on a day to day
basis and are accustomed to traditional MS Windows™ based
graphical user interfaces but they saw no similarities with the
MR system. Consistent with the findings in a previous study
[6], none of the respondents referred to interacting with or
through a computer when asked what the interaction felt like.
Instead one respondent even compared the experience to
having a personal instructor guiding through the steps: “It
would be if as if someone was standing next to me and
pointing and then… but it’s easier maybe, at the same time it
was just one small step at a time. Not that much at once.”
(participant 1)
Even though the participants stated that they normally do not
mind working with many different technological devices on a
daily basis occasions that cause problems occurs. For
example when rarely using equipment – it is not easy to
remember how it works if a long time passes in between use:
“I think that especially on this apparatus there are a lot of
details that we get information about and then we forget it
because we don’t use it for a while and then you need to be
updated if you have to deal with something, otherwise you
don’t use the apparatus to its full potential” (participant 3).
Generally, the respondents are satisfied with the instructions
they have received on how to use technology in their work.
One problem with receiving instructions from colleagues and
other staff members is however that the instructions are not
‘objective’, but more of “this is what I usually do”. The only
‘objective’ instructions available are the manual or technical
documentation and reading this is time consuming and often
not a priority. When asked about the general impression of
the AR system one respondent answered: “…totally clear, it
wasn’t any ‘this is how I usually do it and it usually works
fine’. Really clear…” (participant 6)
Figure 6 A HMD with camera
mounted in front of the user’s eyes.
One important feature of the personal instruction situation is
the interactivity – it is important to be able to ask questions
or return to instructions, or adjust the pace. A majority of the
participants stated that they prefer instructions from an
experienced user, which allows interactivity.
Insert: A fiducial marker mounted
on a size adjustable ring.
Figure 7. A trocar fully assembled.
Bottom left: The separate parts of a
The participants were also asked about in what situations or
for what purpose they think an AR system could be useful, if
in any. Two of them mentioned medically invasive procedures
such as laparoscopy. Several of the participants responded
that AR systems could probably be used in any situation
where instructions are needed, such as in education or for
guiding users on how to use technical equipment.
The video based observation illustrated that the physical
appearance of the AR system probably affected the way the
participants performed the task. Since the display was
mounted on a helmet there were some issues regarding the
placement of the display in from of the users’ eyes, so they
spent some time adjusting it in the beginning of the trial.
However since the system was head mounted it left the
hands free for interaction with the DA and the numpad used
for answering the questions during the instructions.
trocar.
As a result of the study, the AR system has been redesigned
to better fit the ergonomic needs of this user group. Changes
have also been implemented in the instructions and the way
they are presented. The new designs of the AR system have
been used in study 2 in this paper.
Study 2 - Method
Study 2 is a follow-up of study 1, but with a slightly different
application and new participants. Twelve professional (ages
35 – 60) operating room (OR) nurses and surgeons at a
hospital took part in the study. The participants all had some
knowledge about the object of assembly (a trocar, see figure
7), although not all of them had actually assembled one prior
to this study. A majority of the participants stated that they
have an interest in new technology, and that they interact
with computers regularly, however a majority of them had
very limited experience of video games and 3D graphics. The
participants were first introduced to the AR system. When the
head mounted display (HMD) and headset was appropriately
adjusted they were told to follow the instructions given by
the system to assemble the device they had in front of them.
After the task was completed the participants filled out a
questionnaire about the experience. The participants were
recorded with a digital video camera when they assembled
the trocar. During the task, the participants’ view through the
video-see-through MR system was also logged on digital
video. As in study 1, data was collected both through direct
observation and through questionnaires.
Equipment
The AR system was upgraded and redesigned after study 1
was completed (see figure 6). It includes a Sony Glasstron
Head Mounted Display (HMD) and an off the shelf headset
with earphones and a microphone. The AR system runs on a
Dell Inspiron 9400 laptop with a 2.00 GHz Intel®Core™ 2
CPU, 2.00 GB RAM and a NVIDIA GeForce 7900 graphics
card. The AR system uses a hybrid tracking technology based
on marker tracking; ARToolKit, (available for download at
Figure 8. A participant wearing the
[7]), ARToolKit Plus [11] and ARTag [5]). The marker used
can be seen in figure 6. The software includes an integrated
set of software tools such as software for camera image
capture, fiducial marker detection, computer graphics
software and also software developed specifically for MRapplication scenarios [6]. One significant difference between
the redesigned AR system and the AR system used in study 1
is the use of voice input instead of key pressing (see figure
8). The voice input is received through the headset
microphone and is interpreted by a simple voice recognition
application based on Microsoft’s Speech API (SAPI). Basic
commands are OK, Yes, No, Backward, Forward, and Reset.
HMD and following the AR
instructions.
Figure 9. The participants view
in the HMD.
The user task
The object the participants were given instructions on how to
assemble was a common medical device, a trocar (see fig 7).
A trocar is used as a “gateway” into a patient during minimal
invasive surgeries. The trocar is relatively small and consists
of seven separate parts which have to be correctly assembled
for it to function properly as a lock preventing blood and gas
from leaking out of the patient’s body. The trocar was too
small to have several different markers attached to each
part. Markers attached to the object (as the ones in study 1)
would also not be realistic considering the type of object and
its usage – it needs to be kept sterile and clean of other
materials. Instead the marker was mounted on a small ring
with adjustable size which the participants wore on their
index finger (see figure 9).
Instructions on how to put together a trocar are normally
given on the spot by more experienced OR nurses. To ensure
realism in the task, the instructions designed for the AR
application were based on the instructions given by an OR
nurse at a hospital. The nurse was video recorded while
giving instructions and assembling a trocar. The video was
the basis for the sequence of instructions and animations
given to the participants in the study. An example of the
instructions and animation can be seen in figure 9. Before
receiving the assembly instructions the participants were
given a short introduction to the voice commands they can
use during the task; “OK” to continue to the next step, and
“back” or “backwards” to repeat previous steps.
The observations and questionnaire was the basis for a
qualitative analysis. The questionnaire consisted of 10
questions where the participants could answer freely on their
experience of the AR system. The questions related to overall
impression of the AR system, experienced difficulties,
experienced positive aspects, what they would change in the
system and whether it is possible to compare receiving AR
instructions to receiving instructions from a teacher. The
questionnaire responses were described and analyzed
quantitatively.
Study 2 - results
All users in this follow-up study were able to complete the
task with the aid of AR instructions. The responses in the
questionnaire were diverse in content but a few topics were
raised by several respondents and several themes could be
identified across the answers of the participants. Issues,
problems or comments that were raised by more than one
participant have been the focus of the analysis.
None of the open ended questions were specifically about the
placement of the marker, but the marker was mentioned by
half of the participants (6/12) as either troublesome or not
functional in this application: “It would have been nice to not
have to think about the marker" (participant 7)
Concerning the dual modality function in the AR instructions
(instructions given both aurally and visually) one respondent
commented on this as a positive factor in the system. But
another participant instead considered the multimedial
presentation as being confusing: “I get a bit confused by the
voice and the images. I think it’s harder than it maybe is”
(participant 9)
Parallax and depth perception issues are commonly known
problems for any video-see-through system and not only the
one used in this study. The problems are occur due to the
cameras angle, which is somewhat distorted from the angle
of the users’ eyes, causing a parallax vision, i.e. the user will
see her/his hands at on position but this position does not
correspond to actual position of the hand. Only one
participant mentioned problems related to this issue: “The
depth was missing” (participant 7). A majority among the
participants (8/12) gave positive remarks on the instructions
and presentation of instructions. One issue raised by two
participants was the possibility to ask questions. The issue of
feedback and the possibility to ask questions are also
connected to the issue of the system being more or less
comparable to human tutoring. It was in relation this
question that most responses concerning the possibility to
ask questions, and the lack of feedback were raised.
The question of whether or not it is possible to compare
receiving instructions from the AR system with receiving
instructions from a human did get an overall positive
response.1 4/12 gave a clear yes answer and 5/12 gave
1
To assure that the interpretation of the open answers was done
correctly the interpretation was triangulated in that two observers did
translations into “yes”, “no” and “unclear” responses. These were
more unclear answers like: “Rather a complement; for
repetition. Better? Teacher/tutor/instructor is not always
available – then when a device is used rarely – very good”
(participant 1). Several of the respondents in the Yes
category actually stated that the AR system was better than
instructions from a teacher, because the instructions were
“objective” in the sense that everyone will get exactly the
same information. When asked about their impressions of the
AR system, a majority of the participants gave very positive
responses: “Very interesting concept. Easy to understand the
instructions. Easy to use” (participant 3) Others had more
concerns: “So and so, a bit tricky” (participant 6). One
question specifically targeted the attitude towards using AR
in their future professional life and all participants responded
positively to this question.
Concluding discussion
When introducing new systems, like AR, in an activity, user
acceptance is crucial. The overall results from both studies
shows a system that the participants like rather than dislike,
and whether they received instructions in two modalities or
only one – both studies indicate that the participants would
like to use AR instructions in their future professional life.
Despite some physical issues with the AR system all users,
but one in both studies, did complete the task without any
other assistance. However, effects of the physical intrusion of
the system upon the users’ normal task should not be
ignored. Even if the system is lightweight and non-intrusive,
it still may change the task and how it is performed. This
then compared and correlated and Cohens Kappa was used to
determine the inter-rater reliability. The result was 1.0 which is well
above the satisfactory level of 0.70 and means that the two observers
made the same interpretations of all the answers.
may not be a problem in the long run – if the system is a
positive influence on the task, user and context, it will with
time and experience grow to be a part of the task (much like
using computers have become part of the task of writing a
paper).
Interactivity is an important part of direct manipulating user
interfaces and also seems to be of importance in an AR
system of the kind investigated in these studies. A couple of
the participants who responded “no” on the question
regarding comparability between AR and human instructions,
motivated their response in that you can ask and get a
response from a human, but this AR system did not have the
ability to answer random questions from the users. Adding
this type of dialogue management and/or intelligence in the
system would very likely increase the usability and
usefulness of the system, and also make it more “humanlike” than “tool-like”. However, this is not a simple task, but
these responses from real end users indicate and motivate
the need for research in this direction. Utilizing knowledge
from other fields, such as natural language processing, has
the potential to realize such a vision.
Acknowledgements
This study is part of a collaboration project between the
Department of Computer and Information Science (IDA) at
Linköping University, and the Swedish Defence Research
Agency (FOI), funded by the Swedish Defence Materiel
Administration (FMV).
References
[1] Azuma R. A survey of Augmented Reality. Presence:
Teleoperators and Virtual Environments. 6:4 (1997) 355-385
[2] Azuma R, Bailot, Y., Behringer, R. Feiner, S., Simon, J. &
MacIntyre, B. Recent Advances in Augmented Reality, IEEE
Computer Graphics and Applications. Nov/ Dec 2001, 34-47
[3] Billinghurst, M., Belcher, D. Gupta, A. and Kiyokawa, K.
Communication Behaviours in Colocated Collaborative AR
Interfaces. International Journal of Human-Computer
Interaction, 16 (3), (2003) 395-423
[4]
Davis, F. D. Perceived Usefulness, Perceived Ease of
Use, and User Acceptance of Information Technology. MIS
Quarterly, vol 13, no 3, (1989) 319-340
[5] Fiala, M. ARTAG Rev2 Fiducial Marker System: Vision
based Tracking for AR. Workshop of Industrial Augmented
Reality, Wienna Austria Oct 4, 2005
[6] Gustafsson, T., Carleberg, P., Svensson, P., Nilsson, S.,
and Sivertun, Å. Mixed Reality Systems for Technical
Maintenance and Gaze-controlled interaction. FOI-R.1695-SE
(Technical report ISSN 1650-1942) 2005
[7] HITLAB, http://www.hitl.washington.edu/artoolkit/ (as of
2007-07-23)
[8] Livingston, M. Evaluating Human Factors in Augmented
Reality Systems. IEEE Computer Graphics and Applications
November/December 2005
[9] Livingston, M. Julier, S.J. and Brown, D. Situation
awareness for teams of dismounted warfighters and
unmanned vehicles. Proceedings of SPIE, Enhanced and
synthetic vision, volume 6226, 2006
[10] Milgram, P. & Kishino, F. A taxonomy of mixed reality visual
displays. IEICE Transactions on Information Systems, Vol E77-D,
No 12 (1994) 1321-1329,
[11] Schmalstieg, D. Rapid Prototyping of Augmented Reality
Applications with the STUDIERSTUBE Framework, Workshop
of Industrial Augmented Reality (IAR) Wienna, Austria Oct 4,
2005
[12] Nilsson, S & Johansson, B. A Cognitive Systems Engineering
Perspective on the Design of Mixed Reality Systems. Proc of
ECCE-13, ETH Zürich (2006) 154-161,.
[13] Träskbäck, M. User Requirements and usability of Mixed
Reality applications. Licentiate Thesis, Helsinki University of
Technology, 2004