WiiInteract: Designing Immersive and Interactive
Application with a Wii Remote Controller
Jee Yeon Hwang and Ellen Yi-Luen Do
Georgia Institute of Technology
Atlanta, GA 30308, USA
{jyhwang, ellendo}@gatech.edu
We designed and implemented WiiInteract to provide a
tangible interface that links digital interactivities directly to
children’s real life experiences. WiiInteract consists of four
interactive applications that support playful and tangible
interactions for children. WiiInteract utilize physical
manipulations and movements of digital data through a
sensing device, the Nintendo Wii Remote Controller.
WiiInteract has immersive interactions in the natural
environment and aims to offer a potential creative play
environment through tangible interaction. In this paper, we
report the design and implementation of WiiInteract
applications and conclude with discussion and future work.
cameras, and LEDs. WiiInteract, on the other hand, needs
only a Wii Remote Controller (Wii Remote), which
integrates novel interaction techniques and motion sensing
capabilities into a tangible remote control device. Thus,
WiiInteract enables children to achieve embodied
interactions with its wireless and sensing technology. With
its applications and flexible interaction methods, WiiInteract
can offer expressive and non-competitive interactions. The
educational technology for children has been constrained by
two-dimensional exploration [8]. To improve screen-based
interactions, WiiInteract utilizes its three-dimensional
physical controls and playful applications, which model the
natural environment.
Author Keywords
RELATED WORK
ABSTRACT
Tangible interaction for children, Wii Remote Controller,
immersive interaction processes.
ACM Classification Keywords
D.2.2 [Design Tools and Techniques]: User interfaces; H.5.2
[User Interfaces]: Input devices and strategies, Interaction
styles, Screen design, Style guides.
INTRODUCTION
The tangible user interface (TUI) has emerged as a
powerful method through which to embed digital elements
in the real world. We have a recent trend that applies TUI
applications to playful learning systems for children to join
the physical and digital worlds [13]. In order to facilitate
realistic interaction, WiiInteract connects physical
movements and screen-based digital information with
applying a design approach for TUIs. The design process of
WiiInteract involves considering how to motivate children
so that they will interact and engage with the tangibility as
interactive artists do with their artwork [1]. The existing
tangible interfaces, like I/O Brush [10], StoryMat, a soft
storytelling interface [11], and the Tabletop interface [9] are
creative examples of tangible interfaces, but they require
extra interaction techniques and devices, such as sensors,
There are two categories of related works based on system
components: Tangible interfaces using the Wii Remote, and
other technical devices. The Pinocchio project builds a
system to enable children to experience and learn classical
music by conducting a virtual orchestra with a Wii Remote
[2]. WiiArt project is primarily for artistic and expressive
interactions. It includes prototype applications, such as
Illumination (draWiing), Beneath (Waldo), Time Ripples,
and WiiBand, to compose both images and sounds for art
experiences [7]. One of the non-Wii interfaces is StoryMat,
a tangible interface covered with soft quilt. The software in
the mat works with an ultrasonic transmitter embedded in a
small stuffed animal. The StoryMat mediates a conversation
between children plays the role of a playmate by responding
to the child’s story with a similar story [11]. Another
example is I/O Brush, which contains a video camera,
lights, and touch sensors. The I/O Brush is a design and
development tool that allows users to create visual art with
elements extracted directly from their personal objects and
their immediate environment [10]. In comparison to these
interfaces, WiiInteract uses one simple controller to manage
multimedia options and enables children to experience
diverse connected activities rather than working with one
constrained feature.
WIIINTERACT: APPLICATIONS FOR A TANGIBLE INTERFACE
15th International Conference on Computer Games: AI,
Animation, Mobile, Interactive Multimedia, Educational &
Serious Games, July 28-31, Galt House Hotel, Louisville,
Kentucky, USA
.
Interaction Control with a Wii Remote Controller
The Wii Remote contains an accelerometer, infrared camera,
and built-in 4-point tracking blob, all of which support
motion-sensing technology. After connecting the Wii Remote
to the computer via Bluetooth wireless communication, the
user can start interacting by launching WiinRemote, an
existing program that uses the controller on the computer.
The program provides cursor options and a button-assigned
window. Each application’s inputs or the user’s movements
were assigned to the Wii Remote’s buttons (up/down/left/
right, +/-, A/B, etc.) and the motion parameters through the
Wii Remote. WiiInteract uses a motion sensor instead of an
infrared (IR) sensor for its cursor movement and smooth
navigation in its Flash-based applications. Flash and Action
Scripts were used in the design of WiiInteract’s visual
interface and its immersive applications. The Glove
Programmable Input Emulator (GlovePIE) Script, which
initially was developed for Virtual Reality Gloves, is
another development option to control the remote controller.
However, WiiInteract focuses on Flash and Action Script
components to keep the applications easy-to-execute in any
case or environment. For example, users can make
interactive waves from LakeForest with light tapping
motions, and they can easily make constellation patterns in
StarLight with simple drag-and-drop motions. Further, they
can feel the different aspects of Earth instantly when they
select one of the view options from GlobalSteps.
According to the previous observations on Wii tennis game
plays of 14 researchers for 3 days, Wii Remote is an
effective modality to support immersive and playful
interactions. Wii Remote enables users to control screenbased interface with dynamic motions like waving their
arms in similar to their real-life actions. Thus, when playing
the game, some of the researchers said that their body
movements such as serve and backhand stroke lead them to
have more playful and interactive experiences. WiiInteract
also utilizes the motion sensing methods and interaction
options of the Wii Remote, but the difference is, beyond the
typical Wii games, WiiInteract provides tangible interaction
for children to explore immersive experiences of play in the
real world. In this way, WiiInteract provides tangible
interactions, not for the augmented virtual environment but
for the augmented real environment.
maps on the right that the children can select to see
displayed on the globe. Users can zoom into specific area
when pushing the Wii Remote Controller’s ‘+’ button, and
Figure 1. GlobalSteps enable users to explore various aspects of
earth using its interactive globe.
manages exploration speed by moving the controller
horizontally such as a page flipping motion. In addition to the
unique features on manipulating size or speed, GlobalSteps
has segmented multiple views that connect to the organized
map, Google Earth, to improve its interaction and navigation.
It can be enhanced to a mixed reality tour game by
manipulating some parts of its visual data representation to
zoom in on some aspects of the Earth. This "mixed reality
tour" can give children the opportunity to tour the place around
the world where they have never visited with zero carbon
footprint.
WIIINTERACT INTERACTIVE APPLICATIONS
WiiInteract gives pre-defined menu options and enables
children to set their preferences and change interaction
types. WiiInteract consists of four interactive applications
to support its tangible interaction: GlobalSteps, StarLight,
LakeForest, and TravelWorld.
GlobalSteps
Children can learn various aspects of earth with GlobalSteps.
There are four options to view and rotate the threedimensional globe. Playing with GlobalSteps will help
children grasp the concepts of environmental pollution,
energy consumption, and explore aspects around the globe.
The application also includes additional objects, such as
asteroids and stars that change their colors according to
children’s mood or interacting status [5]. Figure 1 shows the
display of an ever evolving globe and the various status
Figure 2. StarLight users can match stars to the constellation
picture or create new ones.
StarLight
StarLight encourages the study of constellations by children
with direct mapping of stars to the constellation picture and
also enables them to create new ones using the flexible
controller’s movement. Being able to change the connecting
network diagram of stars and to make their own
constellations can help them reflect and understand the
constellations. This constellation-making procedure would be
useful for other applications that a diagram structure may be
useful. For example, computational thinking tools, such as a
tangible state machine game, Escape Machine [12], and a
science game of DNA sequences can also use this
constellation creation structure. In addition, iterative steps in
constellation creation can enable users to engage in
interactive learning and storytelling based on the mythology
of the constellations. In this manner, StarLight has the
potential to visualize the children’s symbolic perception,
expression, and imagination.
and feeling the water flow. In addition, by tapping the
lake’s surface, the user can play with reflective ripples. This
interaction may inspire and increase users’ participation on
natural environment. The authors plan to combine sounds
with LakeForest’s actions to increase children’s immersion
and participation. The research project, Elysian Fields,
shows that the combination of sound and action enables
users to perceive of being physically located in the place
and actually engaging in these activities. This sense of
having a physical effect may also give participants a strong
pleasure from creation [1].
TravelWorld
TravelWorld is a dynamic word and picture matching game.
When children select and drag the message box with while
holding the Wii Remote’s ‘B’ (Backside) button, they can
rotate, shuffle, or move it in any direction. After reading the
essential information from the message, the user looks and
matches each picture with the message. If the graphical or
textual information is insufficient, then the user can search for
additional information from Wikipedia.com by following its
link. TravelWorld is also applicable to a communicative and
social interaction to share not only messages, ideas, and
pictures, but also immersive physical activities. Further,
TravelWorld can be enhanced to work for an interactive
message center or a discussion room for design collaborations
with its flexible message control options and gesture data
representation.
Figure 3. TravelWorld has interactive options for users to travel
around the world and learn value-added geographical information.
LakeForest
LakeForest application is inspired by both the peaceful
enjoyment and the playful acts of ripple making people
often engage in lakeside scenery. On the lake surface,
interactions with the Wii Remote are marked as traced
images of wave and ripple movements. LakeForest displays
realistic and natural images for the context of the play.
Users can move the controller in a horizontal direction,
such as smoothly moving the sunlight, touching the surface,
Figure 4. LakeForest presents water waves and ripple movements
reflecting the Wii Remote’s motions and touches.
FUTURE WORK
WiiInteract offers selective interactions and navigations by
obtaining inputs from the Wii Remote’s buttons and motion
sensors. This interface offers sophisticated motions, such as
pitching, rotating, and rumbling to enable children to
interact more actively with the virtual environments that
simulate our natural environments. Future work of this
project could enhance the sensory aspects of the system
similar to what Johnny Chung Lee did for his Wiimote
Projects. His work connected digital sensors to the real
world objects are demonstrated in Wiimote DesktopVR,
Whiteboard with IR pen, and Finger Tracking using the Wii
Remote Controller [6]. Further extension of WiiInteract
could augment the play interaction with a shape or an avatar
to represent children’s activities on screen in the virtual
world, or display previous play interaction data (e.g., images
of prior journey and explorations, travel logs, etc. from
themselves or other children around the world) to motivate
children to reflect and explore the various activities.
Tangibility and the Child-User Benefits
During the development process of WiiInteract, we were
interested in about how the integration of the representation
of natural environment with tangibility in application design
can help and enhance children’s interaction activities. For
approaches of user studies, we drew from three particular
methodologies that we would briefly describe here.
The first method is surveying each participant about their
experiences and feelings with detailed classifications of
choice options, such as the 13 pleasure categories of play [1].
The framework of the pleasure categories was developed by
the six theorists, all of whom approach play and pleasure
from different perspectives. According to Costello, B., and
Edmonds, E., of A Study in Play, Pleasure, and Interaction
Design [1], the framework can enable the interactive
designers to focus more on the type of playful experiences
that they want their work to elicit. Likewise, we can get
feedback on what children feel and experience from the
tangible applications and compare their responds to the first
assumed results, like creation and fantasy, which were
made during the design process. This process can bolster
the weak points and find their unexpected positive effects.
The second option is to conduct play testing of WiiInteract
several times using different sizes of user groups. When
they form a team, children can participate in interaction more
cooperative and social ways. As shown in the paper,
People, Places, and Play: Player Experience in a SocioSpatial Context [3], the games frequently gives meaningful
social interactions in the real and tangible world of gamer.
Moreover, WiiInteract’s applications provide more
educational and pragmatic experiences. The group activity
is, therefore, effective to obtain social and empirical
evidence to support the assumed benefits of tangibility.
Lastly, asking caregivers (i.e., parents and teachers) may be
an efficient way to assess benefits of tangibles for children.
We can also provide a questionnaire using pictures, short
sentences, and diary options to let children express their
opinions comfortably and conveniently.
CONCLUSION
In this paper, we present WiiInteract, a tangible interface
with four applications that provide immersive, expressive,
and creative experiences with the Wii Remote. Believing
that engaging in well-designed game enhances the user’s
social and other abilities in real life [3], WiiInteract focuses
on offering children educational and playful benefits
through its interconnection between physical controls and
the realistic natural environment.
Studies of playful behavior report that the player switch
back and forth between the explorative goal, “What can this
object do” and the playful goal “What can I do with this
object [1].” The user also becomes easily bored if they
cannot find new features or possibilities to play with. Thus,
in support of tangible interface that provides responsive and
interactively changing environment, WiiInteract seeks to
find a balance between those goals while providing
customizable and renewable content with its applications to
support children’s preferences and frequent activities.
WiiInteract encourages children to the design of interactive
applications to explore the potential of the Wii Remote for
playful learning and interaction. In this way, it focuses on
helping children become more active with their bodily
actions and specific controls of playful applications such as
the modeling and visualizing of the natural environment.
The design ideas and applications of WiiInteract offer a
new approach to the tangible interface using the Wii
Remote. Exploring a variety of application properties, this
design pattern may provide interesting properties and
insights that may benefit design and implementations of
future tangible interfaces.
REFERENCES
1. Brigid Costello, B., and Edmonds, E. A study in play,
pleasure and interaction design. Proc. conf. on Designing
Pleasurable Products and Interfaces, DPPI 2007, ACM
Press (2007), 76-91.
2. Bruegge, B., Teschner, C., Lachenmaier, P., Renzl, El,
Schmidt, D., and Bierbaum, S., Pinocchio: conducting a
virtual symphony orchestra. Proc. ACE 2007, ACM
Press (2007), 294-295.
3. De Kort, Y.A.W., and Ijsselsteijn, W.A. People, places,
and play: player experience in a socio-spatial context.
Proc. Computers in Entertainment (CIE), ACM Press
(2008), Article No. 18.
4. Florida, R. The Rise of the Creative Class: And How It’s
Transforming Work, Leisure, Community and Everyday
Life. Basic Books, New York, 2003.
5. Fullerton, T., Chen, J., Santiago, K., Nelson, E., Diamante,
V., Meyers, A., Song, G., and DeWeese, J. That Cloud
Game: Dreaming (and Doing) Innovative Game Design.
Proc. SIGGRAPH 2006, ACM Press (2006), 51 – 59.
6. Johnny Chung Lee's Wiimote Projects Including
Wiimote DesktopVR, Whiteboard, and Finger Tracking.
http://www.wiimoteproject.com/.
7. Lee, H., Kim, H., Gupta G., and Mazalek, A. WiiArts:
Creating collaborative art experience with WiiRemote
interaction. Proc. Tangible and Embedded Interaction
(TEI) 2008, ACM Press (2007), 33-36.
8. Revelle, G., Zuckerman, O., Druin, A., and Bolas, M.
Tangible user interfaces for children. Proc. of ACM CHI
2005, ACM Press (2005), 2051-2052.
9. Rick, J., Harris, A., Marshall, P., Fleck, R., Yuill, N. and
Rogers, Y. Children designing together on a multi-touch
tabletop: An analysis of spatial orientation and user
interactions. Proc. Interaction Design and Children
(IDC), ACM Press (2009), 106-114.
10. Ryokai, K., Marti, S., and Ishii, H. Designing the World
as Your Palette. Proc. CHI 2005, ACM Press (2005),
1037-1049.
11. Ryokai, K., and Cassell, J., Computer Support for
Children's Collaborative Fantasy Play and Storytelling.
Proc. Computer Supported Collaborative Learning
(CSCL), ACM Press (1999), 510-517.
12. Weller, M.P., Do, E.Y.-L., and Gross, M.D. Escape
Machine: teaching computational thinking with a
tangible state machine game. Proc. Interaction Design
and Children (IDC), ACM Press (2008), 282-289.
13. Xie, L., Antle, A. N., and Motamedi, N. Are tangibles
more fun? Comparing children's enjoyment and
engagement using physical, graphical and tangible user
interfaces. Proc. Tangible and Embedded Interaction
(TEI) 2008, ACM Press, (2008), 191-198.