A Fast and Robust Fingertips
Tracking Algorithm for Vision-Based
Multi-touch Interaction
Qunqun Xie, Guoyuan Liang,
Cheng Tang, and Xinyu Wu
2013 10th IEEE
International Conference on Control and Automation (ICCA)
2
Outline
• Introduction
• Related Work
• Proposed Method
• Hand localization
• Fingertips tracking
• The Multi-touch system
• Experimental Results
• Conclusion
3
Introduction
4
Introduction
• Multi-touch technology:
• Sensor Based
• Directly receive finger touch as input
• High cost → limits its application to some extent
• Computer Vision Based
• Good scalability as well as good performance
Image: Oka, K, Sato, Y, Koike, H. "Real-time fingertip tracking and gesture recognition", IEEE Computer Graphics
and Applications, 2012
5
Introduction
• In this paper:
• Propose a robust fingertip tracking algorithm:
• Real-time
• Stereovision-based 3D multi-touch interaction system
• Skin / Depth / Geometry structure
Hand
Detection
Fingertip
Tracking
Multi-touch
System
6
Related Work
7
Related work
L. Jin, D. Yang, L. Zhen, and J. Huang. A novel vision based finger-writing
character recognition system. Journal of Circuits, Systems, and Computers
(JCSC), 16(3):421–436, 2007.
• Geometry properties:
• Curvature
• Edge or shape
• Build a model
• Image Analysis
• Template matching
• Color Segmentation
D. Lee and S. Lee. Vision-based finger action recognition by angle detection and contour
analysis. Electronics and Telecommunications Research Institute Journal, 33(3):415–422, 2011.
8
Related work
• Palm Center:
𝜺
𝜴
[b]
[a]
• Fingertip Detection
Geodesic
distance
GSP points
Neighbor
depth
[a]
[c]
[d]
9
Related work
• [a] Hui Liang, Junsong Yuan, and Daniel Thalmann, "3D Fingertip and Palm Tracking in
Depth Image Sequences", Proceedings of the 20th ACM international conference on
Multimedia, 2012
• [b]Chia-Ping Chen, Yu-Ting Chen, Ping-Han Lee, Yu-Pao Tsai, and Shawmin Lei, "Real-time
Hand Tracking on Depth Images", IEEE Visual Communications and Image Processing
(VCIP), 2011
• [c] Ziyong Feng, Shaojie Xu, Xin Zhang, Lianwen Jin, Zhichao Ye, and Weixin Yang, “Real-
time Fingertip Tracking and Detection using Kinect Depth Sensor for a New Writing-in-the Air
System”, Proceedings of the 4th International Conference on Internet Multimedia Computing
and Service, 2012
• [d] Zhichao Ye, Xin Zhang, Lianwen Jin, Ziyong Feng, Shaojie Xu, "FINGER-WRITING-IN-
THE-AIR SYSTEM USING KINECT SENSOR", IEEE International Conference on Multimedia
and Expo Workshops (ICMEW), 2013
10
Proposed
Method
11
Hand Segmentation
Training data:
• Skin Color filter
• YCbCr color space
• Gaussian Mixture Model
• Describe the skin-color distribute
• Single Gaussian Model:
color vector
• Gaussian Mixture Model:
Weight of each Gaussian model:
12
Hand Segmentation
• Skin Color filter
•
: how skin-like the color is
• Expectation Maximization(EM) algorithm
13
Hand Segmentation
• Depth Cue:
• The points with minimum depth are picked as seeds
• Region grow algorithm
skin
depth
skin + depth
14
Hand Segmentation
• Divide wrist and hand:
• By a boundary curve 𝐵𝑊
[18]
• Minimum depth
,
range threshold
(related to palm size)
r : row index
c : column index
z(r,c) : depth value
• Boundary curve
boundary
[18] Z. Mo and U. Neumann, “Real-time hand pose recognition using low-resolution depth images,” in Computer
Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, vol. 2.
15
Palm Region Extraction
• Observation : Palm is a rectangle-like region
• Method : Project the hand region in all directions
𝐡𝐚𝐧𝐝
𝟎°
𝟗𝟎°
𝐢𝐧𝐭𝐞𝐫𝐬𝐞𝐜𝐭𝐢𝐨𝐧
16
Palm Region Extraction
Intersection
17
Palm Center Localization
•
X
𝟗𝟎
𝟎°°
>𝜆
𝟎° 、𝟒𝟓° 、𝟗𝟎°、𝟏𝟑𝟓°、𝟏𝟖𝟎°
>𝜆
𝟎°
>𝜆
𝟗𝟎°
18
Palm Center Localization
• Palm Center:
• The point with maximum distance from the closest palm boundary[18].
palm region palm boundary
• The size of palm R:
19
Fingertip Localization
• Fingertip :
The point with maximum distance to the palm center
(on the contour of each finger)
P : contour point
C0 : palm center
d2 : distance
R : palm size
• Candidate set F:
1.2
F
20
Fingertip Localization
• Assign an index to each point in candidate set:
𝜃𝑝𝑓 : index
F : candidate set
C0 : palm center
𝜙 ∶ the angle of 𝑃𝑓𝐶0 with negative x-axis
• Sort candidate set by
21
Fingertip Localization
• Distance between successive points :
• If 𝐷𝑝𝑖 > 𝛿 → Start & End point
subset
• Fingertips : maximum distance in each subset
22
Multi-touch system
• TUIO (Tangible User Interface Object)
[24] M. Kaltenbrunner, T. Bovermann, R. Bencina, and E. Costanza, “Tuio:A protocol for table-top tangible
user interfaces,” in Proc. of the The 6th Intl Workshop on Gesture in Human-Computer Interaction and
Simulation, 2005.
23
Experimental
Results
24
Experimental Results
• Xeon 3.07Ghz workstation
• frame rate : 20Hz on average (real-time)
• Modules
• Fingertip tracking
• TUIO server
• TUIO client
[10] C. Shan, Y. Wei, T. Tan, F. Ojardias, ”Real Time Hand Tracking by Combining Particle Filtering and
Mean Shift”, In: International Conference on Automatic Face and Gesture Recognition, 2004, pp. 669-674
27
Conclusion
28
Conclusion
• Fast and robust fingertip tracking
• Without pressuring sensing device & extra marks
• Hand Segmentation
• Depth / Skin
• Fingertip Detection
• Palm region
projection
• Palm center
distance from the boundary
• Fingertip : assign index (angle)