Interactive Systems Technical Design
Seminar work: GESTURES
Pekka Välitalo
Daniel Pakkala
Teemu Koponen
ISTD 2003, GESTURES
Introduction/Definitions
• The use of motions of the limbs or body as a means of
expression
• In a computer controlled environment, one wants to use the
human hand as a manipulator, and its use in human-machine
communication
• Gestures originate as gesturer’s mental concept, possibly in
conjuction with speech
• Gestures are expressed through the motion of arms and hands,
the same way speech is produced by air stream modulation
through the human vocal tract
• Observers perceive gestures as streams of visual images, which
are interpret using the existing knowledge about those gestures
ISTD 2003, GESTURES
Motivation
• As the computing, communication and display technologies
process even futher
 The existing HCI (Human-Computer Interaction)
techniques may become a bottleneck in the effective
utilization of the available information flow
• The use of hand gestures provide an alternative to cumbersome
interface devices for HCI
 Visual interpretation of hand gestures can help in
achieving the ease and naturalness desired for it
• More natural interfaces for humans to computing systems
ISTD 2003, GESTURES
Taxonomy of Hand Gestures for HCI
Hand/Arm Movements
Unintentional Movements
Gestures
Communicative
Acts
Mimetic
ISTD 2003, GESTURES
Deictic
Manipulative
Symbols
Referential
Modalizing
Implementation
ISTD 2003, GESTURES
Implementation
• Temporal Modeling
-Gesture Phases
-Preparation
-Nucleus
-Retraction
• Spatial Modeling
-3D Hand/Arm Model
-Appearance-Based Model
ISTD 2003, GESTURES
Applications
• Control of consumer electronics
• Interaction with visualization systems
• Control of mechanical systems
• Computer games
ISTD 2003, GESTURES
Hand Gesture Control of TV Set and a Lamp
• Three types of hand gestures: static hand posture,
quantitative hand motion and qualitative hand motion.(In
the prototype static hand postures used)
• Lars Bretzner & Co, “A Prototype System for Computer
Vision Based Human Computer Interaction”, Technical
Report, Royal Institute of Technology, Stockholm, Sweden
www.nada.kth.se/cvap/adstracts/cvap251.html.
ISTD 2003, GESTURES
In Detail
1. Images grabbed at frame rate
2. RGB -> new color space
3. Color feature detection
->Image features(blobs,ridges)
4. Tracking & Pose recognition
-> pose, position,scale,orientation
5. Match 3.&4. Results with set of
object hypothesis with statistical
approach(particle filtering/condensation)
6. Application control (Recognized
gestures bound with actions)
ISTD 2003, GESTURES
Strengths / Advantages
• No actual physical remote controller needed
• More natural interaction (?)
• Relatively simple recognition technigue
• Nice test case, but...
ISTD 2003, GESTURES
Limitations / Weaknesses
• Usability: Slow response time
• Solution: faster algorithms, more processing power
• Static hand postures & skin colour
• skin and background colour variations?
• solution: recognition based on translations/trajectories
• Two dimensional gesture recognition: angle of the
hand has to be right in relation to the camera.
• Solution: 3D hand model & recognition or multiple
cameras/multiple models for same gesture
ISTD 2003, GESTURES
Industrial Players
•
Cybernet Systems Corporation (www.cybernet.com)
• A company committed to research and commercialize technology
solutions and advancements to human- machine interaction
• UseYourHead: http://www.gesturecentral.com/useyourhead/
•
Immersion (www.immersion.com)
• SuperGlove: www.immersion.com/products/
3d/interaction/cyberglove.shtml
•
Toshiba (www.toshiba.com)
• Motion Processor:
www.toshiba.com/news/980715.htm
UseYourHead
ISTD 2003, GESTURES
•
Siemens (www.siemens.com)
• SIVIT (Siemens Virtual Touchscreen)
• Gesture Computers for the Disabled
www.siemens.de/ct/en/technologies/ic/sivit.html
www.siemens.com/index.jsp?sdc_p=po263380
Motion Processor
SuperGloves
ISTD 2003, GESTURES
International Research Groups
and Projects
• University of California at Berkeley, GUIR (Group for User
Interface Research)
• http://guir.berkeley.edu/
• http://guir.berkeley.edu/projects/quill/
• GMD - German National Research Center for Information
Technology (Fraunhofer-Gesellschaft)
• www.fraunhofer.de/english/
• Interactive Electronic Whiteboard:
www.acoustics.hut.fi/icad2001/proceedings/ papers/mullert.pdf
• Ruhr-University of Bochum, Germany, Institut for
Neuroinformatik
• http://www.ruhr-uni-bochum.de/index_en.htm
• http://www.neuroinformatik.ruhr-uni-bochum.de/top.html
ISTD 2003, GESTURES
Finnish Research Groups
and Projects
• Helsinki University of Technology, Intelligent
Machines and Special Robotics Institute
• WorkPartner, service robot
ISTD 2003, GESTURES
WorkPartner
• Robot control is based on human
gestures and speech instead of
keyboard and joystick
• All operator hardware is included in
the operator´s coat
• The coat is a bright color textile
including PC, speechmike and hand
trackers. The bright color of the
waistcoat is used to help the machine
to recognize and track the operator
and his gestures.
ISTD 2003, GESTURES
WorkPartner
•
•
•
•
Handtrackers are used if direct
teleoperation of the manipulators is
needed in teaching or gripping of
difficult objects
The ring in the end of the
potentiometer wire is simply thread of
the operator’s thumb
Hand positions are calculated from
the length and two direction angles of
the wire and transmitted to the robot
More sophiticated (more complex)
solutions for the robot control are
under research and construction
ISTD 2003, GESTURES
Operator controlling the right arm
of WorkPartner with hand tracker
WorkPartner
Source: http://www.automation.hut.fi/IMSRI/workpartner/
ISTD 2003, GESTURES
• University of Tampere, department of
computer and information sciences, TAUCHI
Group (Tampere Unit for Computer-Human
Interaction)
• Multimodal Interaction Group:
Multimodal information kiosk
interface
• Kiosk can be used with hand
gestures drawn on the
surface of the screen.
• www.cs.uta.fi/hci/
• www.cs.uta.fi/hci/mmig/
kiosk/
ISTD 2003, GESTURES
• Tampere University of Technology,
Department of Information Technology
• www.cs.tut.fi/
• DMI (Digital Media Institute): www.tut.fi/dmi
• Survey: www.cs.tut.fi/kurssit/8107210/ELEET.pdf
ISTD 2003, GESTURES
• VTT Technical Research Centre of Finland,
VTT Electronics
• www.vtt.fi/ele/research/ais
• Using the Soap Box
developed by VTT to identify
gestures and control the
functions of the television.
• www.vtt.fi/vtt/new/
new121.htm
• www.vtt.fi/ele/research/
els/projects/soapbox.html
ISTD 2003, GESTURES
Companies and Research Groups in Oulu
• University of Oulu, Infotech Oulu
• www.infotech.oulu.fi
• MediaTeam: www.mediateam.oulu.fi/
publications/pdf/106.pdf
• VTT Technical Research Centre of
Finland, VTT Electronics
– www.vtt.fi/ele/research/ais
ISTD 2003, GESTURES
Future Developments
Ideally, any and every gesture performed by the user
should be unambiguously interpretable, thus allowing for
naturalness of the interface.
• Real-time interaction based on 3D model gesture analysis.
• Human gestures, especially communicative, naturally employ
actions of both hands  two-handed gestures
• Multi-user interfaces
• Simultaneous use of gestures, speech and gaze for multimodal
control input
ISTD 2003, GESTURES
Further Reading
Pavlovic, V., I., Sharma, R., Huang, T., S., “Visual Interpretation of
Hand Gestures for Human-Computer Interaction: A Review”,
IEEE Transactions on Pattern Analysis and Machine Intelligence,
Vol 19, No. 17, July 1997, pp 677-695.
http://www.cs.rutgers.edu/~vladimir/pub/pami97.pdf
Brezner, L., Laptev, I., Lindberg, T., Lenman, S., Sundblad, Y.,
“A Prototype System for Computer Vision Based Human Computer
Interaction”, Technical Report, Department of numerical Analysis and
Computing Science, KTH, Stockholm, Sweden
ftp://ftp.nada.kth.se/CVAP/reports/cvap251.pdf
Gesture Recognition Home Page: Lots of information about research
groups and commercial products.
http://www.cybernet.com/~ccohen
ISTD 2003, GESTURES