Conceptualizing Tangible Augmented Reality Systems
for Design Learning
Rui Chen and Xiangyu Wang
The University of Sydney, Australia
New forms of combining tangible interfaces and Augmented Reality (AR) systems
are the supporting technologies which can be devised to leverage the way learners
develop skills/knowledge in the design process. The paper presents a theoretical
framework which conceptualizes how tangible AR spatial interactive systems can
be utilized to support cognitive development of design learning. The framework
can be applied not only to evaluate existing tangible AR interfaces, but to conceptualize new prototypes for design learning activities.
Aims
Early research on tangible user interfaces focuses on the design of new
systems. More recently, there has been a shift towards research based on
theoretical and conceptual understandings of tangible interactions [1]. For
instance, Ishii and Ullmer [2] developed a classic research framework that
explores possible types of coupling between material and virtual representations through interactions from a data-centric view. The typical tangible
user interface can be illustrated as in Fig. 1 from an example by the actuated workbench [3]. The traditional interactive tangible user interface provides feedback through video projection alone. The actuated workbench
adds an additional feedback loop using physical movements of the tracked
objects. Therefore, it provides an additional feedback loop for computer
output, and helping to resolve inconsistencies. Users can observe each
other’s actions, and users can reach out and physically change the shared
layout without having to grab a mouse or other pointing device.
Augmented Reality (AR) has attracted investigations as an alternative
vehicle to facilitate learning. Doswell and Blake [4] built a Mobile Augmented Reality (MARS) e-learning tool with capabilities of adapting to
J.S. Gero and A.K. Goel (eds.), Design Computing and Cognition ’08,
© Springer Science + Business Media B.V. 2008
697
698
R. Chen, X. Wang
various learning environments with considerations of cultural, geographical and other contexts about learners. AR training/learning systems have
been developed in the area of military combat [5], training in industrial
maintenance [6], and school education [7]. However, an existing gap between the lab and practical use for Augmented Reality was identified. The
concept of Augmented Instructions that mixes AR and traditional printed
materials therefore emerged. For instance, Wagner and Barakonyi [8] developed educational software that uses collaborative AR to teach learners
the meaning of kanji symbols. Certain study [9] has been carried out to explore the potential strengths and limits of AR as a means of engaging and
educating students who traditionally perform poorly in school.
Fig. 1. The Actuated Workbench based on tangible user interface [3]
Combining the tangible user interface and AR systems into tangible ARbased learning environments can present objects with natural affordances
for supporting interactions. Tangible AR can enable learners to acquire
concrete learning experiences through active experimentation. Students
can actually ‘experience’ theory in a more familiar form, since the practical experiment enables the students to “observe and reflect on” the results
of learning tasks and assignments. This theoretical process provides potentially beneficial categorizations of design learning activities with tangible
AR interfaces. High integration of representations from visual and haptic
stimulus is unique feature of tangible AR systems which exposes learners
Tangible Augmented Reality for Design Learning
699
to multi-channels from the concreteness and sensory directness. For example, a common discussion context consisting of same set of physical objects and public screen can then be created to physically bind everyone for
more effective communication and discussion. Everyone has equal access
to the tangible AR interface. They can easily do comparative work with
concurrent interaction which leads to effectiveness of collaborative learning activities compared to the typical desktop setup with mouse and keyboard.
This paper developed an analytical framework of conceptualizing and
integrating tangible Augmented Reality into design learning. This framework, which intends to inform tangible AR system design, is developed by
considering notions from both cognitive development and educational theory. The relevance of the design of computational artifacts to the above
two areas are identified and integrated into the framework, more specifically into the aspects of tangible and spatial interactions. A tangible AR
prototype is also conceptually specified in the paper. The proposed prototype mainly focuses on supporting the design learning activity based on
experiential and collaborative learning theories. From the practical perspective, tangible AR systems developed based on this framework, can be
integrated into design education curriculum as a supplementary teaching
aid.
Significance
Like a constructivist view on learning, meaning is created through interactions. Broader characterizations of tangible interfaces have been instantiated in design frameworks which concentrate on the design of the interaction itself [10]. Design frameworks which focus on spatial aspects were
also considered [11].
A design framework should be developed to consider learners’ physical,
social and spatial interactions. To tackle this issue, integrating tangible interfaces and AR could be one promising technology option. The framework presented in this paper is the first attempt to address this gap, which
can inspire and inform the design of interactive technologies for learning.
The significance of the presented work is a focus on the theoretical framework for guiding the design of tangible AR systems, rather than merely on
system development. This theoretical framework establishes design principles and produces methodological guidelines for creating effective tangible
AR systems by integrating knowledge from various fields that address the
human factors (perceptual, cognitive, and characteristics of real and virtual
learning in design discipline), information visualization, information tech-
700
R. Chen, X. Wang
nology (hardware, software, communications), human-machine interaction, and learning theories. This theoretical framework also provides potentially beneficial categorizations of design learning activities with tangible AR interfaces.
Framework
The theoretical framework depicted in Fig. 2 addresses four knowledge
domains from cognitive science, design process, tangible AR technology
and learning theory.
Fig. 2. Theoretical process for applying tangible AR concept and technology into
design learning
A response to behaviorism [12], people are not “programmed animals”
that merely respond to environmental stimuli, since people are rational beings that require active participation in order to learn, and whose actions
are a consequence of thinking. The term “cognitive science” is used for
any kind of mental operation or structure that can be studied in precise
terms [13]. Changes in behavior are observed, but only as an indication of
what is occurring in the learner’s head. Cognitivism [14] uses the metaphor of the mind as computer: information comes in, is being processed after the stimulus. There is an essential argument from cognitivist paradigm
[14] that the “black box” of the mind should be opened and understood.
Tangible Augmented Reality for Design Learning
701
The learner is viewed as an information processor (like a computer). Opening the “black box” of the human mind is valuable and necessary for understanding how people learn. Mental processes such as thinking, memory,
knowing, and problem-solving need to be explored. The framework in Fig.
2 shows that tangible AR systems are the stimulus to the learners, the brain
is like the processing continuum (see Fig. 2) which includes the reflective
observation through watching and the hapic feedback from feeling. These
give concrete experience based on the perception which gives abstract
computation. After the active experimentation, this can lead to certain response which can benefit some possible learning activities from playful,
reflective, situated and interactive aspects (shown in Fig. 2). Knowing a
person's (and your own) learning style enables learning to be orientated according to the preferred method. That said, everyone responds to and needs
the stimulus of all types of learning styles to one extent or another - it’s a
matter of using emphasis that fits best with the given situation and a person's learning style preferences.
The framework focuses on supporting learning practice based on experiential and collaborative learning theories. Tangible AR learning space offers a richer form of experiential learning not available previously. Tangible interfaces can provide tactile sensational feedback and Augmented
Reality visual interface can enhance human’s perception of digitalized
feedback from such interaction. This combination directly provides concrete experience. Learners receive the stimulus through the processing
continuum which contains the broad diversity of learning styles ranging
from constructive to analytical learning.
Learning Styles
Specific learning styles which are identified from Kolb’s four-stage learning cycle (shown in Fig. 3) [15] is mapped onto the sequence of the processing continuum as shown in Fig. 2. Various factors influence a person’s
preferred style: notably in his experiential learning theory model, Kolb defined three stages of a person's development, and suggested that our propensity to reconcile and successfully integrate the four different learning
styles improves as we mature through our development stages. The theory
establishes a cyclical model of learning which consists of four stages as
outlined in the Fig. 3. One may begin at any stage, but must follow each
other in the sequence.
Kolb’s four-stage learning cycle [15] represents how experience can be
translated through reflection into concepts, which in turn are used as
guides for active experimentation and the choice of new experiences. The
702
R. Chen, X. Wang
major four learning styles are abstract, concrete, active and reflective
learning. The model is an iterative learning cycle within which the learner
tests and modifies new ideas and concepts as a result of reflection and
conceptualisation. One of the critical issues in educational methods is that
the teaching style offered does not always significantly addresses the
learning style preferred by learners. It is therefore useful to combine different teaching methods/techniques to cover different aspects of what
needs to be learned.
Fig. 3. Kolb’s Experiential Learning Cycle [15]
The first stage, concrete experience, is where the learner gains active
experiences through an activity such as a lab session or practical work. A
learning process begins with a concrete experience, which is then followed
by the second stage; reflective observation is when the learner consciously
reflects back on that experience. The reflection can then be assimilated
into a theory by the process of abstract conceptualisation The third stage,
abstract conceptualization is where the learner attempts to conceptualize a
theory or model of what is observed. The last stage, active experimentation
is where the learner is trying to plan how to test a model or theory or plan
for a forthcoming experience. New or reformulated hypotheses are tested
in new situations.
Four learning styles have been identified by Kolb [15] which correspond
to these stages. The styles give a highlight of conditions under which
learners learn better. These can be categorized to the styles as below: Assimilators are the people who learn better when presented with sound logical theories to consider. The assimilating learning preference is for a
Tangible Augmented Reality for Design Learning
703
concise, logical approach. People with an assimilating learning style are less
focused on people and more interested in ideas and abstract concepts. People with this style prefer readings, lectures, exploring analytical models,
and prefer to have time to think things through. Convergers are those who
learn better when provided with practical applications of concepts and
theories. They mostly prefer to technical tasks, and are less concerned with
people and interpersonal aspects. People with a converging learning style
pay more attention to technical tasks and problems than social or interpersonal issues. Another style comes up with accommodators. These people
learn better when provided with “hands-on” experiences [15]. The accommodating learning style relies on intuition rather than logic. These
people desire to conduct a practical, experiential approach based on previous people’s analysis. People with an accommodating learning style prefer
to work in teams to complete tasks. They set targets and actively work in
the field trying different ways to achieve an objective. Last category is divergers. They learn better when allowed to observe and collect a wide
range of information. Particularly, they usually prefer to watch rather than
do, tending to collect information and use imagination to solve problems.
People with a diverging learning style have broad cultural interests and enjoy gathering information. People with the diverging style prefer to work
in team, to listen with an open mind and to receive personal feedback.
Design Development
Design learning inherits certain features of design development. The typical design process was also analyzed and featured as a spiral process as
shown in Fig. 4 to model how the various design elements fit together [16].
It can be extended to model how the various elements involved in design
learning and development fit together. For tangible Augmented Reality
systems, the initial mental image/model is perceived and constructed visually from reflective observation (Augmented Reality) and tactilely from
tangible feedback (tangible interface). For each cycle along the spiral design process, designers proceed through by presenting, testing, and reimaging responses to a set of related problems. Abstract concepts are then
reinforced by this continuous combined feedback from visual and tactile
channels. Designers then converge the current abstract concept to the active experimentation. Following the above immediate action, they return to
problems that have already been studied to revisit earlier decisions
throughout this design activity. The output is the knowledge gained from
the expressive, playful, reflective, situated and interactive learning activities from tangible AR systems.
704
R. Chen, X. Wang
Fig. 4. Design Development Spiral [16]
Physicality
This section brings together a multi-disciplinary group of knowledge towards the central physicality [17]. It is necessary to dig more deeply into
the nature of physicality itself. Physical objects impress people mentally.
There are two issues which have been raised that how physicality plays a
role in traditional computer-aided design and the other way round. Three
properties of 'ordinary' physical things – that is inanimate, non-mechanical,
'natural' things. There are three aspects extended from the properties: directness of effort, locality of effort and visibility of state [17]. These aspects can explain the reason why the visual and tactile feedback can give
the concrete experience provided by the tangible AR system, impact either
explicitly or implicitly. The way designers design requires an understanding of how learners apply principles of cause and effect from the physicality. The design of tangible AR systems which do not conform to these
principles may trigger confusion, disinterest or reflection. These properties
lead to identification of centre-movement-expression (see Fig. 5). For example, when we press the door bell, we expect to hear the ring tone and
Tangible Augmented Reality for Design Learning
705
later hopefully some responses from the inside or somebody opens the
door, but if we won’t get any feedback, people usually will try to press the
door bell again. From this simple case, we see the physicality has direct
and central influences on the movements related to the locality change
from outdoor to indoor and even the aural spatiality. Spatiality is another
property of tangible AR systems, which provides space for actions where
they affect computation. It is unlike traditional desktop systems which utilize an indirect-mapped controller, typically mouse and/or keyboard for
input, tangible AR systems afford opportunities to capitalize on human’s developing repertoire of physical actions and spatial abilities for direct
system input and control. For example, theme parks and interactive exhibitions in museums, art galleries and science centers have created a rich tradition of creating environments which respond to human’s actions and
movement.
Concreteness and
sensory-directness
Centre-movement-expression
Physicality
Effect
Cause
1. directness of effort
2. locality of effort
3. visibility of state
Embodiment effects & Learning
benefits
Fig. 5. The physicality between cause and effect
However, little is known about how to design these augmented reality
environments specifically to support cognitive development of learning.
Design requires an understanding of how and why people’s actions in
space are related to cognitive developmental change.
Embodied Cognition
The embodiment from cognitive science suggests that there are some
stronger linkers between physical activity and cognition [18]. Tangible AR
706
R. Chen, X. Wang
systems are envisaged to offer more opportunities for learners to increase
the physical activities which prove to influence and constrain cognitive
process [19]. While learners perceive visual representations from AR systems and tacitly feel physical materials from tangible interfaces, perceptual
and motor responses are complemented to offer a more accurate assessment of the current learning outcome. Such alignment/matching in turn
tightly inter-leave the perceptual and cognitive processes together. Materializing knowledge into physical forms can also help learners to further advance the understanding of those purely gained from written media. Several informal evaluations of tangible AR systems [2] [7] for learning have
been highlighted the integration of physical and digital representations as
an important distinction between representations. Spatial and temporal relationships between representations are defined as the integration. The integration of representation has also been outlined as one of features for
tangible user interfaces [2].
It is a truism that computation and information are always physical [20].
For example, the magnetic regions polarize even ink on paper. Just as the
embodiedness of the human body is critical to understand cognition,
physical embodiment reminds us of crucial features of computation (see
Fig. 6). For example, that you can only perform finite computation in finite
time and space and that memory 'space' consumes physical space. Numerous pedagogical approaches including the Montessori Method and Frobel’s
Kindergarten curriculum support using bodily engagement with physical
objects to facilitate active learning [20]. This can be proved from a classical example that the way children use their bodies to learn is to use their
fingers to count and perform arithmetic operations.
Pragmatic action involves manipulating physical entities to directly accomplish a task [21]. In a simple case, a child will try to connect Lego
blocks to build a dinosaur that can stand on two feet. Epistemic action involves manipulating physical entities in order to change the nature of the
cognitive operations necessary to complete a task. For example, a child
may consider spending the amount of time to connect and disconnect Lego
blocks to better understand how different configurations relate to the stability of their creation.
Conceptualizing Tangible AR Systems
This section presents the conceptualization of a tangible AR prototype
based on the framework. It is highlighted that tangible manipulation is
bodily interaction with physical objects. These objects are coupled with
Tangible Augmented Reality for Design Learning
707
computational resources [22], allowing users to control computation by
fusing familiar desktop utensils, two-dimensional computer desktop applications, and three-dimensional models, we provide a seamless transition
from the traditional 2D computer desktop to a 3D augmented working environment. Two hexahedral objects are attached together to create the new
tangible AR interface. The shape of the new tangible AR interface removes the restriction of user’s motions in existing tangible AR interfaces.
Users can move and rotate the new interface freely to manipulate virtual
objects in AR environment. This improvement is useful for applications
that require unrestricted rotation motions of virtual objects. A user navigating the virtual environment using the two hexahedral objects interface, a
virtual block is added or removed from the corresponding position of the
destroyed building in the virtual environment can easily get the affection
from the other view. This mode ends when users complete putting all the
blocks in the right place on the broad.
Embodied Cognition
Directly accomplish
a task
Physical interaction
with the
world
Pragmatic action
Mental
action
Manipulating the
physical entities
Epistemic action
Change the nature of the
cognitive operations necessary to
complete a task.
Fig. 6. The relation between the physical worlds to the embodied cognition
708
R. Chen, X. Wang
System Architecture
Fig. 7 shows the overall architecture of the tangible AR system that displays the event and messages flow among the sub-components of the system. It consists of the context server for managing the main logic, the virtual environment and the tangible interfaces for user navigation and
interaction. The context server includes the communication manager for
the event and message handling among the sub-systems. The augmented
reality environment shows the reconstructed urban area (such as buildings,
parking spots, trees and roads) with vivid sounds. It also contains user interaction and event processing modules for state update by the context
server and the tangible interfaces.
AR system
Event Receiver
Graphics&
Behaviour
engine
Tangible interface
Navigation interface
Augmentation
display
Context server
FSP Engine
Communication Manager
UDP wrapper
Navigation
TCP wrapper
Two hexahedral
objects
Interaction
Sound server
Tangible Board
Event
Queue
Transmitter
Fig. 7. System Design
The tangible interfaces include the augmented display and the navigation interface for navigation and the broad interface for user interaction in
the augmented reality environment. Navigation interface allows learners to
walk through the pre-designed area along with the virtual urban area. The
augmentation display shows the overview of the open area and the user’s
current location in the virtual environment. It allows user to draw a path on
Tangible Augmented Reality for Design Learning
709
the map with their hands to navigate in augmented reality environment.
The advantage of using this system design is to emphasize the physical interaction with the world as mentioned in the section of embodied cognition. When the learner performs the physical activities in the virtual urban
area, their hands directly interact with the physical objects which improve
the influence to cognitive process. In another word, the physical and digitalized learning material has been integrated into the representation. The
board interface adopts the Teris puzzle game metaphor. It allows users to
modify the structure of the buildings using various tangible blocks. Each
block position is detected by 10*10 PIC micro-controller driven switch
unit. Each micro-controller cell unit controls LEDs and switches. The
board interface is connected to the communication manager by TCP.
System Setup
Fig. 8 is the sketch for the setup of the tangible AR system and it illustrates
the practical system configuration. Underneath the table is a projector for a
table display and a camera for tracking the hexahedral objects. It considers
one aspect of interface principles, which is seamless interaction with an
AR system. That is created by aligning two hexahedral objects as shown in
Fig. 8 with markers. Eighteen distinguishable markers are attached to the
block, so it can be viewable from the camera of the AR system at any orientation as long as the block is located inside the viewing area of the camera. The camera views the block, and the AR system detects markers on
the block and estimates pose of the block using the detected markers. The
block provides natural interactions with users. Users can hold the block
and move or rotate it freely to manipulate the corresponding virtual objects. At least one side of the block is always visible to the camera as long
as the interface is viewed by the camera. Learners can also hold the block
with one or two hands not occluding all markers on the block. This helps
to increase the physicality described in the previous framework. It directly
effects on the concreteness and sensory-directness. As stated before, the
way designers design requires an understanding of how learners apply
principles of cause and effect from the physicality. Physical objects can
impress people mentally so when the learners play around with the blocks
meanwhile they also receive visibility from different broad. These helps
them learn fast and understand better in space which is related to cognitive
developmental change.
The camera under the table is only used to track object’s position and
orientation. Under the table, a mirror is equipped to extend the line of sight
of the projector and the camera. Beside the table, the augmentation display
710
R. Chen, X. Wang
shows virtual graphical models augmented on the scene of the real table
captured from the camera above the table. The upper camera is also used
to track the 3D position and orientation of tangible interface. This more
naturally engages learners to the learning activities since three aspects
from the directness of effort (manipulating the physical objects), locality of
effort (recognizing the orientation from the Augmentation display) and
visibility of state (overview from the navigation display) are supported to
some extent.
Fig. 8. Setup sketch for tangible AR system
Summary
This paper presented a theoretical framework which investigates how tangible Augmented Reality systems can be utilized to support cognitive development of design learning. The framework was developed by considering
Tangible Augmented Reality for Design Learning
711
different learning styles, design development theory, physicality and
embodied cognition. The framework can be applied to conceptualize new
tangible AR prototypes for design learning activities. It can also be used to
evaluate existing tangible AR interfaces. The paper also presented a proposed tangible AR system, based on the framework. In order to verify its
applicability, future work will apply this framework to a wider variety of
cases in design learning.
References
1. Holmquist L, Schmidt A, Ullmer B (2004) Tangible interfaces in perspective:
guest editors’ introduction. Personal and Ubiquitous Computing 8(5): 291-293
2. Ishii H, Ullmer B (1997) Tangible bits: towards seamless interfaces between
people, bits and atoms. Proceedings of CHI 1997, ACM Press, pp. 234-241
3. Pangaro G, Maynes-Aminzade D, Ishii H (2003) The actuated workbench:
computer-controlled actuation in tabletop tangible interfaces, Proceedings of
the 15th annual ACM symposium on user interface software and technology,
October 27-30, Paris, France
4. Doswell JT, Blake MB (2006) Mobile Augmented Reality system architecture
for ubiquitous e-Learning. Fourth IEEE International Workshop on Wireless,
Mobile and Ubiquitous Technology in Education - (WMTE'06)
5. Department of Defense (2002) Augmented reality for ship survivability, live
fire test and training program. url: https://www2.afams.af.mil/programs
/projects/lftt/projects/aug_shi.htm, last update April 2005
6. Schwald B, Laval B (2003) An augmented reality system for training and assistance to maintenance in the industrial context. Journal of WSCG
7. Kaufmann H, Schmalstieg D, Wagner M (2000) Construct3D: a virtual reality
application for mathematics and geometry education. Education and Information Technologies 5(4): 263-276
8. Wagner D, Barakonyi I (2003) Augmented reality Kanji learning. Proceedings
of the Second IEEE and ACM International Symposium on Mixed and Augmented Reality, Tokyo, Japan
9. Asai K, Kobayashi H, Kondo T (2005) Advanced learning technologies. Proceedings of ICALT: 213–215
10. Hornecker E, Burr J (2006) Getting a grip on tangible interaction. Proceedings
of CHI 2006, ACM Press:437-446
11. Bongers B (2002) Interactivating spaces. Proceedings of 14th Annual Conference on Systems Research, Informatics, and Cybernetics, Baden, Germany
12. Pavlov IP (1927) Conditioned reflexes: an investigation of the physiological
activity of the cerebral cortex
13. Lakoff G, Johnson M (1999) Philosophy in the flesh. Basic Books, NY
14. Luger G (1994) Cognitive science: the science of intelligent systems.
Academic Press, San Diego
712
R. Chen, X. Wang
15. Kolb DA (1984) Experiential learning: experience as the source of learning
and development. Prentice Hall, New Jersey
16. Zeisel J (2006) Inquiry by design: environment/behaviours/neuroscience for
architecture, interiors, landscape and planning, second edition. WW Norton,
New York
17. Kamaruddin A, Dix A (2006) Understanding physicality on desktop: preliminary results. First International Workshop for Physicality, Lancaster
18. Barsalou LW, Wiemer-Hastings K (2005) Situating abstract concepts. In
Pecher, D and Zwaan, R (eds), Grounding Cognition: The Role of Perception
and Action in Memory, Language, and Though,: Cambridge University Press,
Cambridge
19. Barsalou LW, Niedenthal, PM, Barbey, A, Ruppert, JA (2003) Social embodiment. The Psychology of Learning and Motivation 43: 43-92
20. Antle AN (2007) Child-based personas: need, ability and experience. Cognition, Technology and Work, Special Issue on Child Computer Interaction:
Methodological Research, Springer, London
21. Klemmer S, Hartmann B, Takayama L (2006) How bodies matter: five themes
for interaction design. In Proc. DIS 2006, ACM Press: 140-148
22. Ullmer B (2002) Tangible interfaces for manipulating aggregates of digital information. PhD thesis, MIT Media Lab