Human Computer Interaction: The uses and development
of Gesture Recognition for Control of Multimedia Systems.
Anonymous
Southampton University
Anonymous@ecs.soton.ac.uk
ABSTRACT
The way in which a user interacts with a computer is
an important issue considering user experience and
abstraction of the system to increase productivity of
work and also ease of access to the core
functionality. Gesture recognition is an advancement
of user interfaces in which the system uses the
user’s body as the controller to remove a layer of
abstraction from the system.
This report aims to look at the strengths and
weaknesses of using gesture control systems within
different multimedia computer systems such as
video playback, computer games and music and
general content browsing.
Gesture control systems are a promising technology
that has potential for investment in many different
functional uses; media control is an effective use of
gesture technology.
General Terms
Performance, Design, Human Factors.
Keywords
Human Computer Interaction, Interaction, Gesture
control, Touch Screen.
1.
2.
GUESTURE CONTROL SYSTEMS
2.1
Examples of Gesture Control
Systems
There has recently been a push in gesture within
gaming systems lead by the Nintendo Wii in 2006
that aimed to increase the target audience for the
release of the games media. now within 2010 the
other large console companies Sony (Playstation)
and Microsoft (Xbox) have released their own
methods of gesture tracking. The Playstation move,
like the Wii uses a system in which the user’s
gestures are capture using a controller. Microsoft’s
Kinect Product replaces the need for a controller by
using a system that uses a range camera to track
the user’s body movements
Other multimedia systems that use gesture control
do not use the entire body to create gesture control
systems. Examples of products that currently track
gestures by only tracking touch are Apple’s
iPhone/iPod touch which employs a multi-touch
system that detects the multiple fingers on its
surface using a thin transparent conductive material
above the screen. Microsoft’s Surface is a larger
touch screen product that tracks user’s hands to
produce an interactive touch screen table where by
touching and dragging their fingers across the
screen the user can perform tasks such as resizing
their photos and listening to music.
INTRODUCTION
This report will analyze the strengths and
weaknesses of Gesture Control by looking at its
application within multimedia including the positive
effects of its use for each purpose. The
disadvantages will be highlighted near the end of the
report. Within the conclusion of the report there will
need to be a summary of the key aspects of the
control system in several different applications.
Within the conclusion I will be trying to measure the
effectiveness of application by trying to apply “The
Model Human Processor” [1] this will require me to
estimate a time value for a typical gesture controlled
task. To estimate a time value finding a song within a
library will be measured and also measure the time
in which it takes to find an image on the iPhone.
Fig
1
–
Microsoft
(http://www.microsoft.com/surface/)
Surface
Fig 3 – Zombie Outbreak for iPhone (I collected all the
buttons to the top right hand corner of the screen)
Futuristic gesture controlled systems have been
commonly placed within film media such as a hand
tracking gesture system featured Minority Report
(2002) and also within Iron Man (2008) showing how
media has seen these technologies becoming wide
spread.
Fig 2 – Minority Report Glove gesture control system.
(http://zoelaycock.com/blog/wpcontent/uploads/2010/02/minorityreport.jpg)
Having a gesture controlled interface also helps to
increase the feeling of immersion within games as
the user’s actions relate more directly to the actions
of the character they are playing as on screen. With
the introduction of gesture control games have been
released that focus on the users fitness such as Wii
Fit, The Nintendo Wii remote has also been
considered for use helping patients with
hemipariesis.
2.2
Where can Gesture Control be
used, and why?
Gesture control has already seen uptake within the
games market such as motion tracking games that
use a person’s swing to control an on-screen
character’s bat or running on the spot to run within
game, and also with touch screen control featured
within the Nintendo DS touch screen device an
iPhone/iPod touch games. Touch screens have been
used within games as they reduce the amount of
buttons a device needs to contain as well as
reducing the complexity of controlling the games
making them more accessible to potential customers
that wouldn’t otherwise be used to remembering
buttons. Adding gesture control to games instead of
button controls ensures that a game has a greater
amount of flexibility (such as moving the controls
around the screen to suit the individual needs and
preference of the player.
Another use of gesture control is the implementation
of user interfaces in which movies and music can
easily be browsed through in various forms. A
common form at the moment is the use of a swipe
gesture on the system to ‘flick’ through objects
(implemented within Kinect for browsing movies and
also within the iPhone music browsing interface) the
speed of the flick within the iPhone also controls the
speed at which the content is scrolled through
allowing users to browse through their content faster.
Gesture control wouldn’t just be limited to being used
for browsing media collections, basic media
functions could be could be controlled without the
need of trying to find the remote, or even without the
need to turn a wireless controller back on because it
has turned itself off to save battery. Users could
rewind and fast forward media by using the same
flick gesture that has been used for browsing of
libraries, volume by the simple action of rising or
lowering of your hand. With the Kinect system that
recognises when a person is within the camera
frame, pause could be initiated automatically when
there is no one watching the media. (The last person
moves their body outside of the camera’s viewing
angle)
The controls for media could be used within other
place such as browsing websites, reading books
(flick to change page, present within iBooks) and
also the browsing of images.
2.3
How Gestures are used to
Improve User Experience
Fig 5 – Pinch-to-Zoom Patent Apple Inc.
[http://www.patents.com/portable-electronic-devicemulti-touch-input-7812826.html]
Gestures use interface metaphors
heavily to
improve usability of the system as unlike buttons,
users cannot easily guess how to use the system.
The interface could use visual cues to explain how to
interact or maybe even a tutorial. Gesture controlled
systems would need to use symbolism to reduce the
difficulty for the user to perform basic tasks but not
all humans understand all symbolism and as such
there may need to be manuals documenting how to
perform different tasks. Once a gesture has been
related to a function (such as how flicking has been
related to scrolling though a horizontal or vertical list)
other gesture control systems should inherit these
standards to remove interface confusion. If a user is
asked to remember many different functionalities for
the same motion the gesture control system would
not benefit from recall. (Users could recall stop
symbol as the palm of a hand from traffic symbolism)
2.4
The use of Gesture interfaces in
the Pursuit of Simplicity
Advances in HCI are usually in reducing the amount
a person has to think about a task/reduce the time it
takes to act, time to perform an action within a
system can be quantified by Nommell’s theory. An
interface that is simple reduces the amount of time
that an individual would spend scanning for what
they wish to achieve. The User interface hides the
underlying complexities of a system and as such the
effectiveness of a user interface is quantifiable by its
simplicity while allowing for functionality.
2.5
Disadvantages of using a Gesture
Control System
A major flaw in Gesture systems is that they are
reliant on being having recognisable symbolism. This
is especially apparent when the current patenting
system is taken into account. The patent by Apple in
figure 3 is an example of a gesture being reserved
for just one company. Patenting gestures is counterproductive for the progress of technology as it works
at removing recognisability from systems.
As of yet it is difficult to implement Haptic feedback
for gesture controlled devices, have little to no
texture on touch screens make processes hard to
perform without the users fully attention. Haptic
feedback would be an important stage in the
development of gesture related systems as it would
enable blind control of systems. At the moment
some systems are difficult if not impossible for users
with limited accessibility to use.
A major disadvantage of gestures is the initial time it
takes to learn new systems if the gesture aren’t
recognisable, such as now that pinch-to-zoom is
patented for hand held devices other devices will
need to use less recognisable gestures for the same
functionality. If there are too many different gestures
to
remember
systems
could
become
overcomplicated, an example of such would be a
system in which a full television remote was
represented by gestures
if you needed some
functionality that was uncommonly used it would take
dramatically more time to look up the gesture than it
would to search for a button.
Accidental Gesturing and the controlling of when to
register gestures are difficult to distinguish between
in gesture systems. There are issues with this, such
as how long is a touch before it becomes a drag,
which touches are intentional and which are
collisions. One way of distinguishing for non realtime applications (games and other instances where
the user’s reactions are tested) would be assigning a
gesture that is difficult to accidentally perform such
as splitting your fingers down the middle (
start inputting gesture commands.
) to
2.6
How Gestures can be coupled
with other technology to improve user
interface
Gestures could be coupled with other technologies
such as holograms in order to greatly improve the
ease of specific tasks. A task such as spatial control
of objects, this is shown within ‘Iron Man’
Fig 5 Iron Man - (http://www.myninjaplease.com/wpcontent/uploads/iron_man_2_holographics6.jpg)
down to the time it takes to look at each thumbnail. In
both cases it was easy to navigate to the correct media.
4.
ACKNOWLEDGMENTS
This document was built on the ACM SIGCHI
document
template.
www.acm.org/sigs/publications/pubform.doc
5.
REFERENCES
Author[surname, first], Title. Publisher, City of
Publication, Year of Publication.
Card, S. (1986). ‘The model human processor - An
engineering model of human performance’
Fineman, B. (2004). Computers as People: Human
Interaction Metaphors in Human-Computer
Svensson, P. (2006). Nintendo to Sell Wii Console in
Novemberhttp://www.foxnews.com/wires/2006Sep14/0,4670,Ni
ntendoWii,00.html
Petrecca, L. (2007). USA Today. Tech giants target
older buyers - and their cash
Gesture control could also be used in conjunction
with Virtual or Augmented reality to create simulation
games or a HUD with movable items on it so that
users could watch a movie while walking to work,
school etc.
3.
CONCLUSION
Gesture technology is still in the early stages of
development and as such is not entirely viable for most
purposes, but after some further investment the technology
could be used in a wider range of situations. If gesture
standards are developed then there is a lot of potential for
intuitive and simple interfaces for otherwise very
technically difficult tasks such as 3D modelling.
One of the main concerns is the reliability of gestures in
comparison to conventional button orientated systems.
Gesture recognition is a promising technology and should
be incorporated into other systems other than media once
it is more reliable such as heavy machine operation.
In testing the effectiveness of gesture control for finding a
song I found that it takes about 2 seconds to flick to a
song when there are 200 songs, looking for a picture using
gesturing took 5 seconds with 200 images this could be
Freeman W. Weissman C. (1994). Mitsubishi
Electric Research Laboratory. Television Control by
Hand Gestures
Grossman T., Wigdor D., Balakrishnan R. (2004).
Multi-finger gestural interaction with 3d volumetric
displays
Schlomer T., Poppinga B., Henze Niels.,Boll S.
(2008). Gesture recognition with a Wii controller
Molenaar G. (2010). Sonic Gesture.
Buxton B. (2009) Microsoft. Multi-Touch Systems
that I Have Known and Loved.
Leder, R.S. Azcarate, G. Savage, R. Savage, S.
Sucar, L.E. Reinkensmeyer, D. Toxtli, C. Roth, E.
Molina, A (2008) Nintendo Wii remote for computer
simulated arm and wrist therapy in stroke survivors
with upper extremity hemipariesis
Baudel T. (1993) Charade: remote control of objects
using free-hand gestures
Vladimir I. P (1997). Visual Interpretation of Hand
Gestures for Human-Computer Interaction: A Review
Pope H. (1999) Hand at rest grip