Finger Detection and
Tracking
Gurpreet Nanda
Introduction
● Detect fingers in video from a normal
camera in real time
● Many possible applications for humancomputer interaction
● Applications in augmented reality
Assumptions
●
●
●
●
●
Rigid model of hand
A single raised finger
Hand is the only body part in view
Hand is ~1ft from camera
Stationary Camera
○ This assumption will need to be revised
Tasks
● Hand Segmentation
○ Extracting the hand cleanly
● Finger Detection
○ Determining the location of the finger given the
segmented hand
Previous Work - Hand Segmentation
● Color Segmentation
○ Can be done in the HSV or YCbCr color space
○ Problems:
■ Range of skin colors
■ Background object colors in range
● Foreground Segmentation
○ Frame Differencing
○ Background Subtraction
○ Problems:
■ Requires stationary camera
Previous Work - Hand Segmentation
● Active Contours
○ Deformable Models
○ Gradient Descent
○ Problem:
■ Clutter
○ Mean Shift?
■ Restrictive
https://upload.wikimedia.org/wikipedia/commons/0/00/Snake-contour-example.jpg
Previous Work - Finger Detection
● Von Hardenberg and Berard method
○ Look at square around locations in region of
interest
■ Percentage of filled pixels
■ Number of connected pixels along boundary
● Normalize the hand position
○ Center of gravity of “hand” and direction of slant
○ Search for finger in the normalized position
Implementation
●
Hand Segmentation
● Foreground Segmentation
○ Background Subtraction
● Combined with color segmentation
● Morphological operations to clean the
image
Hand Segmentation
Finger Detection
● Normalize hand position
○ Vector endpoints:
■ Midpoint of hand intersection with bounding box
■ Centroid of hand
○ Project hand image based on the angle of this
vector
Finger Detection
Finger Detection
● Construct a template image of a rectangle
topped with a hemisphere
● Use normalized cross-correlation to find
best match in normalized Image
● Project location back to find finger location
Finger Detection
Results
Results
See videos.
Discussion
● Achievements
○ Functional finger tracking given the assumptions
● Limitations
○ Stationary Camera
○ Distance to the camera/size of finger in frame
○ Stationary Background
Future Work
● Replace foreground segmentation with
active contours segmentation using meanshift
● Avoid large changes in the vector position
to prevent mismatches
● Detect multiple fingers
● Convert code for use on a mobile platform