Hand Shadow Gestures
To Rotate And Scale
Sample Image
Tim Taylor, Colorado School Of Mines
Introduction
• Setup
• Projector
• Webcam
• Application
Projector
Web Cam
Previous Work
• 3-D Contour Modeling
• Position Features
• Fourier Descriptors
• Finger Segmentation
• Rule Classifiers
• Hu Invariant Moments
• Finger-Earth Mover’s Distance
• Time-Series Curve
Methods Applied
•
Hu Invariant Moments
𝑥
𝜂𝑖𝑗 =
𝑦
𝑥
𝑥−𝑥
𝑖
𝑦 − 𝑦 𝑗 𝑓(𝑥, 𝑦)
𝑦 𝑓(𝑥, 𝑦)
(1+(𝑖+𝑗)/2)
𝐼1 = 𝜂20 + 𝜂02
𝐼2 = 𝜂20 + 𝜂02
2
+ 4𝜂11 2
𝐼3 = 𝜂30 − 𝜂12
2
+ (3𝜂21 − 𝜂03 )2
𝐼4 = 𝜂30 + 𝜂12
2
+ (𝜂21 + 𝜂03 )2
𝐼5 = 𝜂30 − 3𝜂12 𝜂30 + 𝜂12 [ 𝜂30 + 𝜂12 2 − 3 𝜂21 + 𝜂03 2 ]
+ 3𝜂21 − 3𝜂30 𝜂21 + 𝜂03 [ 𝜂30 + 𝜂12 2 − (𝜂21 + 𝜂03 )2 ]
𝐼6 = 𝜂20 − 𝜂02 [ 𝜂30 + 𝜂12
2
− 𝜂21 + 𝜂03 2 ] + 4𝜂11 𝜂30 + 𝜂12 𝜂21 + 𝜂03
𝐼7 = 3𝜂21 − 3𝜂03 𝜂30 + 𝜂12 [ 𝜂30 + 𝜂12 2 − 3 𝜂21 + 𝜂03 2 ]
− 𝜂30 − 3𝜂12 𝜂21 + 𝜂03 [3 𝜂30 + 𝜂12 2 − (𝜂21 + 𝜂03 )2 ]
Methods Applied
• Hu Invariant Moments Limitations
• Experimental Usage
Methods Applied
• Earth Mover’s Distance
𝑷 = 𝒑𝟏 , 𝑤𝟏 , … 𝒑𝒎 , 𝑤𝑚
𝑸 = { 𝒒𝟏 , 𝑤𝟏 , … (𝒒𝒏 , 𝑤𝑛 )}
𝐸𝑀𝐷 𝑷, 𝑸 =
𝑗 𝑑𝑖𝑗 𝑓𝑖𝑗
𝑖
𝑖
𝑗 𝑓𝑖𝑗
Methods Applied
• Earth Mover’s Distance Limitations
• Partial Matching
Method
• Gesture Recognition
• Find Hand
•
Choose Image from ROI
• Time Series Curve
•
Distance Radially from Center of Mass
• Segmentations to Find Signature
•
Threshold at certain radius
• Calculate Best Gesture Match
• Gesture Usage
• Nearest Neighbor Rule
Method
• Find Hand
5
4
3
2
1
Method
• Time Series Curve
•
•
•
•
Find Center of Mass
Calculate increments
Find Boundary of ROI
Remove Wrist and Normalize
0
0
1
0
Method
• Segmentations to Find Signature
• Thresholding
Method
• Segmentations to Find Signature
• Thresholding
𝑻 = { 𝒕𝟏 , 𝑤1 , … , 𝒕𝒎 , 𝑤𝑚 }
𝒕𝒊 = {𝑡𝑎𝑖 , 𝑡𝑏𝑖 }
𝒕𝒂
𝒕𝒃
𝒘
0.01
1.06
21.90
1.93
2.34
7.76
2.74
3.03
6.12
3.18
3.66
9.07
3.74
5.28
25.25
Method
• Segmentations to Find Signature
• Thresholding
𝑆𝑖𝑚𝑖𝑙𝑎𝑟𝑖𝑡𝑦 =
𝐸𝑀𝐷(𝑇𝑖 , 𝑇𝑗 )
𝑖
𝑗
Method
• Video of Operation
http://youtu.be/RkCfXE1OEOk
Analysis
• Gesture Recognition of Three Gestures
Flat
Flat
Open
Pincers
Five
74
7
1
Open
Pincers
0
26
8
Five
0
14
47
Analysis
• Speed of Operation
• Average 0.79s per frame
• 38500 calls to Time Series Curve function
• 4518 calls to EMD function
Future Work
• FEMD vs. EMD
• Efficiency of Operation
• Application
References
A. Licsar, and T. Sziranyi, “Hand Gesture Recognition in Camera-Projector System,” LNCS,
vol. 3048, pp. 83-93, 2004
H. Ha, and K. Ko, “A Method for Image-Based Shadow Interaction with Virtual Objects”,
Journal of Computational Design and Engineering, vol. 2, pp. 26-37, 2015
J. Martin, M. Santos, and J. de Lope, “Orthogonal Variant Moments Features in Image
Analysis”, Information Sciences, vol. 180, pp. 846-860, 2010
Y. Rubner, C. Tomasi, and M. Lindenbaum, “The Earth Mover’s Distance as a Metric for Image
Retreival,” International Journal of Computer VIsion, vol. 40, no. 2, pp. 99-121, 2000
Y. Yao, and Y. Fu, “Contour Model-Based Hand-Gesture Recognition Using the Kinect Sensor,”
IEEE Transactions on Circuits and Systems for Video Technology, vol. 24, no. 11, 2014
Z. Chen, J. Kim, J. Liang, J. Zhang, and Y. Yuan, “Real-Time Hand Gesture Recognition Using
Finger Segmentation,” The Scientific World Journal, 2014
Z. Ren, J. Yuan, J. Meng, Z. Zhang, “Robust Part-Based Hand Gesture Recognition Using
Kinect Sensor,” IEEE Transactions on Multimedia, vol. 15, no. 5, 2013
Questions