1
Lu
Magician or Just Hand-waving:
A Promising Future of Gesture-Control Technology
Dizhen Lu
Abstract
Modern gesture-control systems are an emerging technology. As a revolution of human-machine
interface, gesture-control technology has evolved from inaccurate webcam-like sensor bars to highly
precise USB peripherals and, in the near future, armbands that detect motion and bioelectricity. The
advent of gestural interface satisfies human demand of creating an intimate communication with
machines by utilizing fingers’ capabilities, and it will simplify the way we interact with both the physical
and virtual environment. Since gadgets based on gesture-control technology are becoming more
accessible and affordable, they will become an inseparable part of common life, such as helping elderly
and disabled people to improve living conditions.
Introduction
People have been dreaming of gesture-control technology for years. In some mainstream
movies, such as Johnny Mnemonic (1995) and Minority Report (2002), futuristic gestural
user interfaces have been depicted to emphasize their sci-fi elements and provide an
awesome experience for the audience to imagine what future would be like. For instance,
in order to retrieve essential locative information, Johnny Mnemonic, played by Keanu
Reeves, unlocked a virtual lock pyramid by waving his bare hands. Likewise, in Minority
Report, the police chief, played by Tom Cruise, browsed camera-recording tapes
controlled with intricate hand movements to search for suspicious moments in the archive.
2
Lu
Both dramatic presentations of the gesture-controlled system impressed the audience by
evoking a sense of marvel. On the other hand, these depictions indicate that gestural
interfaces may serve as the medium of human-machine interaction for the next generation,
as the mouse, keyboard and touch screen did for this generation.
The trend of modern gesture-control technology started with the invention of
Nintendo Wii in 2005, which brought gesture recognition to the ordinary homes with
handheld Wii Remotes. Shortly after, the evolution of gestural interface entered into a
controller-free stage, when Microsoft released Kinect in 2010. Stepped away from the
webcam-style add-on that Wii and Kinect use, Leap Motion, a highly sensitive USB
peripheral that prefers to track finger motion, was launched in July 2013. In the coming
future, more sophisticated gesture-control peripherals will be developed with approaches
other than 3D camera and infrared laser, such as armbands that detect bioelectricity.
History of Human-machine Interface
The reason why gesture-control technology will change the world is the simplicity of
natural user experiences. However, when we review all the currently available humanmachine interfaces, they do not serve the principle well; in the other words, they
complicate the way human interact with the machine. Teleprinter-style keyboards were
the primary input device for decades, in which keys not only produce letters, numbers, or
characters but also execute computer commands. Nonetheless, even today, keyboards still
remain central to human-machine interaction, since smartphones and tablets embed a
virtual keyboard to input data. Shortly after graphical user interface was introduced to the
world, the mouse became the main pointing device. The mouse satisfied human needs for
Lu
3
natural interaction with the physical approach: two-dimensional movements under one’s
hand are detected and thereby translated into the motion of a pointer on the screen. As the
mobile industry continues to develop, touchpads and touchscreens are designed to replace
the mouse in facile forms. On touchscreens, people can interact with the device by
physically pressing icons shown on the display, with either their fingers or a stylus. It
may be obvious that, in the evolution of human-machine interface, the incoming devices
always serve the human needs of creating a direct communication with computers by
utilizing fingers’ capabilities. Therefore, it is easy to tell that motion sensing and gesture
control technology will prevail in the next few years.
Different Approaches of Gesture Recognition and Control
Gestures are considered a series of expressive and meaningful body movements, which
involve physical motion of fingers or body with the purpose of conveying significant
information and interacting with others. [1] These movements may serve as a collection
of potential body language, which can be recognized by the machine as valuable
information to be transferred and consequently reconstructed. Various methods could be
used to achieve the ability to track body motion or even finger movements and evaluate
which information is delivered by the gesture. Though a large number of applications of
the gesture-control system have been introduced in the market, there is “some variation
within the tools and environments used between implementations.” [2] In the following, I
will introduce several different controller-free approaches to gesture recognition and
manipulation, such as a webcam-like sensor bar, a stereo-camera USB peripheral, and an
armband, which detects bioelectricity in the muscles.
Lu
4
Sensor Bar with Infrared LED and 3D Camera
Regarding the implementation of a sensor bar with a camera and LED combination,
Microsoft Kinect may serve as a successful example. The Kinect sensor is a horizontal
bar set atop the display with an adjustable pivot. As shown in Figure 1, the device
contains three critical elements that function together to track full-body 3D motion and
generate a physical image: a RGB camera, a depth sensor and a multi-array microphone
running proprietary software [3]. The RGB camera senses three different color
components along with body figures. It uses 8-bit VGA resolution of 640 × 480 pixels
and a frame rate of 30 fps. The depth sensor contains an infrared projector and a
monochrome CMOS sensor placed a few inches apart that captures 3D video data under
any lighting conditions. It also measures the distance of 48 different points on the player's
body by “transmitting invisible near-infrared light and measuring its time of flight after it
reflects off the objects.” [4] The multi-array microphone is capable of distinguishing the
player’s voice from other noises in order to accomplish voice control. The sensor
confines the player to a roughly 6 m2 area, because the combination of the sensors
provides a solid tracking range of approximately 0.7 to 6 meters, while it also offers an
angular field of view of 57° horizontally and 43° vertically. The Kinect software adopts a
machine-learning algorithm to possess and model all the visual data it receives. Therefore,
it is capable of “automatically calibrating the sensor based on gameplay and the player's
physical environment, accommodating for the presence of furniture or other obstacles.”[5]
5
Lu
Figure 1. Taken at Microsoft’s E3 in 2010. The photo shows all the different parts of the sensor.
Stereo-camera USB Peripheral
Even though Kinect is considered to be accurate by less than a centimeter, Leap Motion,
a 3D gesture-control USB gadget launched in July 2013, is touted for its sensational
finger-tracking accuracy, up to 1/100th of a millimeter, in its product description website.
Although explicit technical details have not been unveiled, it is revealed by the designer,
David Holz, that Leap Motion employs a similar optical system as Kinect but with a
different software approach. The Leap Motion controller is a 3-inches long tiny USB
peripheral on the desktop, with two 1.3 million pixels cameras and three infrared LEDs.
It is most likely that the Leap Motion controller maps the users’ fingers by a compact
version of time-of-flight technology, which relies on analyzing the exact time that light
travels and returns in order to determine the distance. As shown in Figure 2, it operates in
a 150° field and at a distance of about 3 feet without consuming CPU. Compared to
Kinect, which can recognize 48 skeletal points on a player’s body and repeat the
procedure 30 times per second, the Leap Motion controller is able to track ten fingers and
6
Lu
their joints at a rate of over 200 frames per second. [6] However, it is not designed for
facial or voice recognition. Therefore, Leap Motion is more like a supplement to a
desktop workstation or PC entertainment.
Figure 2. A user is testing Leap Motion on a Windows All-in-one.
Gesture Control Armband
MYO, a gesture control armband device created by Waterloo-based startup Thalmic Labs,
is expected to be the game-changer of the gesture interface industry. Now all the camerabased gesture control systems face two limitations: defined space and exclusive
implementation. Unlike the other gesture control devices we mentioned above, MYO
frees the users from boundaries of the fixed space, while also enabling the user to
“connect wirelessly to any electronic or digital device over bluetooth”, as said by Stephen
Lake, the CEO of Thalmic Labs, in their Behind the Scenes Youtube video[7]. Though
the lightweight and “ordinary” armband looks like a series of eight mobile phone
7
Lu
batteries, it contains “a combination of motion sensors and muscle activity sensors to
track the possible gestures.”[8] As presented by Stephen[7], the armband will “recognize
the electrical impulses in your muscles so it can actually recognize the gestures you make
almost before you do them.” The device also contains a reliable battery, which can last
for 48 hours on one charge, and an ARM processor to turn raw data into gestures.
Furthermore, MYO is highly accessible and functional in various computing applications,
such as iPhone 5, Parrot AR Drone 2.0, Orbotix Sphero Robot Ball, and Oculus Rift VR,
since Thalmic Labs provide the APIs to build from any implementation.
Figure 3. A demo picture shows how MYO evaluates electrical activity in your muscles.
A Gesture-controlled Future: Assistive Living solutions
In the promising future, human-machine interaction will undoubtedly become more
intimate than ever, while technologies based on gesture control become accessible and
affordable for all people. At that stage, gesture-control devices will become common
accessories, like our watches and phones, and will serve as a key to both the physical and
Lu
8
virtual world. Gesture-control systems will certainly simplify the way you live. They will
allow you to pause or play back your favorite TV series by easily snapping or scrolling
you fingers; they will allow you to raise or lower the room temperature, open or close the
curtains by flicking your wrist. The affordability and accessibility of gestural interface
may result in the opportunity for improving living conditions for elderly and disabled
people. The older population (65+) numbered 40.4 million in 2010, an increase of 5.4
million or 15.3% since 2000, and it is still growing over time.[9] Eventually, there will be
more elderly people, and fewer young people to take care for them. So gesture-control
systems provide a good approach to assist the living of the elderly. Also, gestural
interface can provide a more convenient and humane environment for disabled people.
They may use gestures to control their wheels, open the doors of buildings, press buttons
on elevators, etc. Therefore, gesture control systems may eventually improve human lives
by building an intimate relationship between users and ubiquitous machines.
9
Lu
References
[1] S. Mitra and T. Acharya. "Gesture Recognition: A Survey," Systems, Man, and
Cybernetics, Part C: Applications and Reviews, IEEE Transactions on, vol.37,
no.3, pp.311-324, May 2007
[2] K. Vyas, A. Pareek and S. Vyas. “Gesture Recognition and Control,” International
Journal on Recent and Innovation Trends in Computing and Communication,
vol.1, iss.7, pp.575-581, July 2013
[3] S. Totilo, “Natal Recognizes 31 Body Parts, Uses Tenth of Xbox 360 “Computing
Resources””. Kotaku, Gawker Media, Jan. 2010
[4] R. Cong and R. Winters, “How It Works: Xbox Kinect”. Jameco Electronics
Workship. Available:
http://www.jameco.com/Jameco/workshop/howitworks/xboxkinect.html.
Retrieved in Sept. 19 2013
[5] W. Greenwald. (2010, Nov. 6). “Kinect vs. PlayStation Move vs. Wii: MotionControl Showdown”. Available:
http://www.pcmag.com/article2/0,2817,2372244,00.asp
[6] Leap Motion, Inc. “Our Device” Avaible: https://www.leapmotion.com/product.
Retrieved in Sept. 20 2013
[7] ThalmicLabs. (2013, Apr. 24). “MYO – Behind the Scenes at Thalmic Labs”
Available: http://youtu.be/oV0PbKZyqg8.
[8] A. Kooser. (2013, Feb. 25). “Myo Gesture-control Armband Uses Muscle Power”
Available: http://news.cnet.com/8301-17938_105-57571180-1/myo-gesturecontrol-armband-uses-muscle-power/
10
Lu
[9] Administration on Aging. “A Profile of Older Americans: 2011” Avaiable:
http://www.aoa.gov/Aging_Statistics/Profile/2011/docs/2011profile.pdf.
Retrieved in Sept. 20 2013