2007 IEEE International Conference on
Robotics and Automation
Roma, Italy, 10-14 April 2007
FrA4.3
EigenNail for Finger Force Direction Recognition
Yu Sun, John M. Hollerbach and Stephen A. Mascaro
Abstract— This paper presents a technique termed EigenNails to classify fingertip force during contact based on the
coloration patterns in the fingernail and surrounding skin.
Fingertip force is classified into six directions: no force, normal
force only, two directions (left/right) of lateral shear force,
and two directions (forward/backward) of longitudinal shear
forces. Based on the face recognition technique Eigenfaces, a
small number of EigenNails are sufficient to express the color
pattern features for shear force direction classification. Results
show that 98% of 960 fingernail images of 8 different subjects
are correctly classified. The lowest imaging resolution without
sacrificing classification accuracy is found to be 10-by-10.
I. INTRODUCTION
Coloration changes in fingernails have been shown to be
useful for measurement of fingertip forces [6]. Due to the
interaction between the fingernail, bone, and tissue, the blood
pools under the fingernail are affected by the pressure at
the fingerpad. The blood pools create color patterns in the
nail and surrounding skin that provide a surprisingly good
transduction of fingerpad force [7]. Shear forces as well as
normal forces can be measured, although there is coupling
between them [8].
To measure the color change in the fingernail, Mascaro
and Asada [6] developed a photoplethysmograph sensor,
comprised of an array of 6 LEDs to illuminate the fingernail
and an array of 8 photodetectors to measure the coloration.
These components were embedded in epoxy, which was
shaped to an individual’s fingernail and attached for close fit
and controlled lighting environment. A strength of this design
is portability. With a linear model, the sensor predicted
normal force to within 1 N accuracy in the range of 2 N
and shear force to within 0.5 N accuracy in the range of 3
N. Considering that the photodetectors are located close to
the surface of the fingernail and there are a limited number
of photodetectors, to fully measure these color patterns, the
locations of the photodetectors are critical. Human factors
such as differences in fingernail size, in the underlying
biomechanical structures of the fingertip, and in blood circulation dynamics make it difficult to design a sensor with
fixed photodetectors to best suit all fingernails for force
measurement.
In previous work [10], [11], we used instead a high resolution video camera to image the full back of the fingertip.
This work was supported by NIH Grant 1R21EB004600-01A2
Y. Sun is with the School of Computing, University of Utah, Salt Lake
City, UT 84112, USA ysun@cs.utah.edu
J. M. Hollerbach is with the School of Computing, University of Utah,
Salt Lake City, UT 84112, USA jmh@cs.utah.edu
S. A. Mascaro is with the Department of Mechanical Engineering, University of Utah, Salt Lake City, UT 84108, USA
smascaro@eng.utah.edu
1-4244-0602-1/07/$20.00 ©2007 IEEE.
A detailed analysis of the static and dynamic color response
of each pixel (0.04-by-0.04 mm area) in the fingernail was
carried out for all regions of the fingernail and surrounding
skin. The complex color response was characterized by
a Bayesian model trained for each individual. The result
was that the normal and shear force on fingertip could be
estimated in isolation with around 0.3 N accuracy and 6 N
measurement range.
The previous studies also provided some characterization
of the coloration patterns for different directions of shear
force. [7] displayed and thresholded the average fingernail
color patterns for 16 subjects to show the different color
patterns for different force and finger postures. [9] showed
that after cropping and normalization according to the length
and width of the fingernail, the fingernail coloration patterns
under various force conditions (no force; normal force = -3;
longitudinal shear forces = ± 2 N; lateral shear force = +2 N)
are different in a statistically significant sense and common
to all people. [10] quantitatively showed that different areas
in the fingernail respond differently. Some areas respond well
to all components of force, other areas are unique to a force
component, particularly for lateral shear where skin areas are
strongly involved.
Identifying the coloration patterns in the fingernail and
surrounding skin, related to the directions of the shear force
on the fingertip, is a pattern recognition problem. Since the
coloration patterns are complex and high dimensional, it is
quite difficult to apply traditional recognition methods to
decide on the direction of shear force. A similar problem
is encountered in face recognition. [13], [14] presented a
method called Eigenfaces that has been successfully used
in face recognition. Face images are decomposed into a
small number of characteristic feature images, which are
the eigenvectors (principal components) of the training face
images, called Eigenfaces. An image is projected into the
subspace spanned by the Eigenfaces. In the subspace of the
Eigenfaces, the classification can be made by comparing
the distance from the projection of the new image to the
projections of the known faces.
Using the same idea, we decompose fingernail images
into a small number of eigenimages, which we call them
EigenNails. The coloration pattern of a new fingernail image
is recognized in the EigenNail space. The EigenNails, which
are the characteristic features of the training images with
different color patterns, represent the response feature areas
in the fingernail and surrounding skin.
In the remainder of this paper, after presenting the EigenNail method, we present experiments that show a very high
accuracy of classification with very few training images. We
3251
FrA4.3
Fig. 1. Fingernail images with different directions of force on the fingertip. The top row are +Fx , −Fx and +Fy . The bottom row are −Fy , Fz , and
zero force.
also determine the minimum image resolutions required for
accurate classification.
II. EIGENNAIL
From our previous study, we found that there are areas
having significant color changes in the fingernail corresponding to the force direction changes on the fingertip. If we view
each image as a point in a very high dimensional space, the
significant color changes cause data scattering in the space.
Using principal component analysis (PCA) to obtain primary
directions of the scatter, we drastically reduce the dimensions
of the space by using just a few principal components to
represent the color changes.
A. Calculating EigenNail
Suppose a fingernail and surrounding skin are imaged
by an N-by-N photo sensor array. For each image I, the
N 2 area of intensity values can be reshaped into an N 2
dimensional vector. For example, a 300-by-300 image has
90,000 dimensions, and so the the image space is a 90,000
dimensional space. Images with different color patterns are
a set of points in space RN ×N . The principal components
or eigenvectors of the data set in the space RN ×N are
orthogonal vectors representing the distribution of the image
data [5].
Let the training set of fingernail images be I1 , I2 , · · · , IM .
The mean and covariance matrix of the images are
Ī =
Σ =
M
1 Ii
M i=1
(1)
M
1 (Ii − Ī)(Ii − Ī)T
M i=1
(2)
We can write Σ = AAT . Since Σ is a N 2 × N 2 matrix,
it is computationally infeasible to find its eigenvectors.
Fortunately, [13] has proved that if vi is an eigenvector of
AT A, then Avi is an eigenvector of Σ = AAT .
The following is the procedure to calculate the EigenNails
relative to different directions of force for one subject’s
fingernail.
1) Take images of the fingernail and surrounding skin for
6 known force directions on the fingertip: no force,
normal force Fz only, 2 lateral shear force directions
±Fx , and 2 longitudinal shear force directions ±Fy .
The force level on each direction is around 2-3 N. For
the shear force, subjects apply approximately a 2-3 N
normal force to prevent slip. There are 10 images of
each force direction, and so the size of the training
data set is M = 60.
2) Preprocess the images to size 300-by-300 (discussed
later), using the green color channel from the camera.
The green color channel has previously been found to
be better than the red or blue channels. Each image is
reshaped to a 1-by-90,000 vector Ii .
3) Calculate the 60 by 60 matrix AT A and its eigenvectors vi with singular value decomposition (SVD) [5].
A = I1 − Ī I2 − Ī · · · IM − Ī ,
where Ī is the average of all images.
4) Calculate the eigenvectors Avi of the covariance matrix Σ = AAT . These eigenvectors are the EigenNails.
With a high resolution (1024-by-768) color video camera
(Flea camera from Point Grey Research), we have collected
10 images for each of the 6 force directions for 8 subjects.
All the auto adjustment functions of the camera were turned
off to make sure the internal condition of the camera does
not change over images. All images are taken in a controlled
lighting environment.
Figure 1 shows one image for each training group of a
typical subject. Figure 2 shows the top 20 EigenNails from
3252
FrA4.3
Top 20 EigenNails
the 60 training images.
There are several strategies to select the most adequate set
of principal components from training data space. We used
the well-known “knee” method, which examines the plot
of the descending ordered normalized eigenvalues and then
searches for the “knee” where the values fall sharply. Defining λi as the eigenvalue corresponding to eigenvector
Avi ,
the normalized eigenvalues are calculated as λi / M
i=1 λi .
Figure 3 shows that the knee is at 5. The first 5 eigenvectors
captures the major features of the color patterns due to
different directions of force on the fingertip. It is consistent
with the look of the EigenNails. The first five EigenNails
highlight the areas with clear color patterns relative to force
directions. The areas are also consistent with the good linear
response areas to different directions of force found in [10].
Figure 4 shows an original fingernail image and its projections onto the subspace constructed by 5 EigenNails. We
can see that the projection onto the EigenNails space keeps
the major color patterns of the original image.
B. Recognizing Force Directions
The hope is that the training images I will cluster into six
different regions of the image space, corresponding to the 6
force directions. For classification of a new image, the task
then is to see if the image is close to one of the six clusters.
Rather than attempt this classification in the highdimensional space RN ×N , the EigenNails method reduces
the image space dimension to a small number r, based on
the number of principal components selected. Each training
0.35
Normalized eigenvalues
Fig. 2.
0.3
0.25
0.2
0.15
0.1
0.05
0
0
10
20
30
40
50
60
Eigenvalue index
Fig. 3.
Normalized eigenvalues
image is projected to the r-dimensional EigenNail space as
a point PT = [p1 · · · pi · · · pr ], where pi is the projection
value on EigenNail i. The EigenNail images in the same
group form one cluster. The L2 norm distances (Euclidean
metric) between its projection and the centroids of all 6
clusters are used for classification.
To demonstrate the process, Figure 5 shows 6 clusters
using just two EigenNails; the 6 clusters are quite well
separated. The subject produced a new image by exerting
force in the +Fy direction. The projection of this new
fingernail image into the 2D EigenNail space clearly shows
3253
FrA4.3
Accuracy percentage
1
Fig. 4. An original fingernail image and its projection on the space spanned
by 5 EigenNails.
Projection value on eigennail two
5000
0.9
0.8
0.7
0.6
0.5
2
4
6
8
10
Eigennail number
Fig. 6. The accuracy of the recognition increases with the number of
EigenNails. There were 60 training images and 120 verification images.
Subject
2
3
4
5
1
6
7
8
Average
0
−5000
−6000
−4000 −2000
0
2000
Projection value on eigennail one
4000
Fig. 5. Six clusters represent the lateral shear force directions +Fx (o’s)
and −Fx (’s), the longitudinal shear force directions +Fy (.’s) and −Fy
(), normal force Fz only (’s), and no force (+’s). The centroids are
indicated by ’*’. The projection of a new image is marked as ’♦’. The
distances between the new image point to the centroids of each cluster are
shown. The shortest distance is the one to the centroid of +Fy cluster.
Skin color
White
White
White
White
Yellow
Yellow
Yellow
Black
Male/Female
M
M
M
M
M
M
F
F
EigenNail number
3
2
5
2
5
5
6
4
4
Accuracy
100%
100%
93%
100%
100%
100%
92.3%
100%
98.1%
TABLE I
T HERE ARE 8 SUBJECTS VARYING IN RACE AND SEX . T HE “E IGEN N AIL
NUMBER ” COLUMN LISTS THE MINIMUM
E IGEN N AIL NUMBERS FOR
BEST ACCURACY.
T HE ACCURACY LISTS THE PERCENTAGE OF CORRECT
RECOGNITION FOR 120 NEW FINGERNAIL IMAGES FOR EACH SUBJECT.
III. PREPROCESSING
that this image belongs to the +Fy cluster as it should.
Figure 6 shows the recognition accuracy with different
numbers of EigenNails. There are 10 fingernail images for
each group used as training data. The recognition accuracy
is calculated based on 20 new fingernail images for each
group. The result shows that the EigenNail method works
well when the number of EigenNails is equal to or larger
than 2 and works perfectly when the number is equal to or
larger than 5.
Running the EigenNail method for all 8 subjects, Table I
lists the subject index, skin color, sex, EigenNail number, and
accuracy. 5 out 8 subjects have 100% recognition accuracy.
The average recognition accuracy is 98.1%. The minimum
EigenNail numbers for best recognition accuracy are from
2 to 6. The EigenNails can efficiently characterize and
distinguish the color patterns for different directions of force.
According to our experimental results, there is no evidence
that the sex or race plays a role in force direction recognition
by imaging.
The EigenNail approach is very sensitive to the fingernail
size and its orientation. For example, when the orientation of
the fingernail changes 10 degrees, the recognition accuracy
decreases to 60%. Preprocessing is very important to correct
for orientation and to normalize for size.
The orientation of the fingernail can be obtained by
estimating the orientation of the whole finger, located by
background subtraction. There are many ways to estimate
the orientation of an object in computer vision [12]. For our
problem, since as shown in Figure 7, the shape of a finger is
symmetrical, the center line of the finger can be obtained by
calculating the average of the columns of finger pixels along
each row. The orientation of the finger is the orientation of
the center line. A typical bilinear image rotation is used for
orientation correction.
Assuming there is no tilting or rolling, the size of the
fingernail can be normalized according to the width of the
undeformed portion of the orientation-corrected finger. After
the normalization, a bounding-box is determined for the full
back of the fingertip by finding the very front tip and pre-
3254
FrA4.3
Fig. 7.
The original finger and the finger after orientation correction.
define the length and width.
IV. RESOLUTION
The results show that just a low-dimensional EigenNail
space can express the color patterns efficiently. This suggests
that a high-resolution camera image is not necessary. To
determine what the lowest-resolution camera image might
be that still gives good results, the image resolution was
gradually increased from 2-by-2 (Figure 8). The result is
that the recognition accuracy increases with the resolution
until it reaches 100% at 10-by-10.
Accuracy percentage
1
0.8
Fig. 9. The top twenty EigenNails for 10 by 10 fingernail images with 6
different directions of force on the fingertip.
One application of this idea is as a virtual switch panel
as illustrated in Figure 10. Fingertip pressing is interpreted
as pressing on a virtual panel to control a virtual PUMA
robot. This application was tested on several subjects; all
were easily able to use this interface to control the virtual
robot in real-time. A demo video is included.
0.6
0.4
0.2
0
0
10
20
30
Resolution
40
50
Fig. 8. The relation between the recognition accuracy and the resolution
of the fingernail images. The row and column of the images are equal. The
x-axis shows the row/column resolution.
Figure 9 shows the top twenty EigenNails when the image
size is 10-by-10. The top five EigenNails in Figure 9 contain
all the feature areas as the top five EigenNails in Figure 2. 10by-10 is the lowest resolution that is still sufficient for force
direction recognition. This finding holds for all 8 subjects.
V. APPLICATION
The finger imaging force direction recognition technique
is potentially useful for human-computer interfaces, by conveniently adding the detection of fingertip force without
requiring a force sensor. At the same time that a camera
tracks finger position [2], [3], [4], it can also measure contact
force.
One example is to use finger tracking to position a mouse
icon and the EigenNail technique to detect mouse clicking.
Instead of recognizing 6 different directions of force, it
only needs to distinguish zero force from normal force Fz .
Fig. 10. A camera is used to track the position of a finger and to detect
whether the finger is pressing down. The finger input is used to control a
virtual robot by a virtual switch panel.
VI. DISCUSSION
The coloration patterns in the fingernail and surrounding
skin in response to fingertip force have been studied with
principal component analysis. In terms of recognizing six
different force directions (left/right lateral shear, front/back
longitudinal shear, normal force only, or no force) a highly
reduced set of principal components termed EigenNails is
3255
FrA4.3
sufficient for high accuracy. For 8 subjects varying in race
and sex, the average accuracy is 98%.
This method currently is limited to recognizing 6 discrete
color patterns due to orthogonal forces. We found that the
color pattern in the fingernail and surrounding skin changes
continuously with the changes of force direction on the
fingertip. Since the principal component analysis is a linear
projection, the continuity property remains in the EigenNail
space. In the future, we will further investigate the possibility
of using the continuity property and Euclidean distances to
centroids to continuously estimate the force direction.
After comparing the EigenNails of 8 different subjects, we
found that they are not that different. [9] showed that from
a statistical point of view the fingernail coloration patterns
are common to all people after normalization. In the future,
we will study the data scattering in the EigenNail space
for a general population, with a view towards developing
a recognition method that can be used for all people without
individual training.
The lighting in the experiments was fully controllable
to be uniform and consistent. For more general usage,
the lighting environment is not controllable. The eigenface
method does not work as well for varying illumination. A
method called “Fisher linear discriminant analysis” (FLDA)
or “Fisherfaces” can tolerate lighting changes [1], at the cost
of additional complexity.
An image resolution analysis showed that the lowest resolution fingernail image sufficient for fingertip force direction
recognition is 10-by-10. This resolution analysis provides a
guide for camera and lens selection, finger positioning, and
finger tracking resolution (there is a conflict between the
resolution of the fingernail and the resolution of the finger
tracking).
R EFERENCES
[1] P. N. Belhumeur, J. P. Hespanha, and D. J. Kriegman, Eigenfaces vs.
fisherfaces: Recognition Using Class Specific Linear Projection, IEEE
Tran. on Pattern Analysis and Machine Intelligence, vol. 19, no. 7,
1997, pp. 711-720 .
[2] J. Crowley, J. Berard, and J. Coutaz, “Finger Tracking as a Input
Device for Augmented Reality,” Proc. Int’l Workshop on Gesture and
Face Recognition, Zurich, June 1995.
[3] W.T. Freeman, D.B. Anderson, P.A. Beardsley, et al., Computer Vision
for Interactive Computer Graphics, IEEE Computer Graphics and
Applications, Vol. 18, no. 3, 1998, pp. 42-53.
[4] C. Hardenberg, and F. Berard, “Bare-hand Human-computer Interaction,” Proc. ACM Workshop on Perceptive User Interfaces, 2001, pp.
113-120.
[5] I.T. Jolliffe, Principal Component Analysis, Springer, NY, 2002.
[6] S.A. Mascaro and H. H. Asada, Photoplethysmograph Fingernail
Sensors for Measuring Finger Forces without Haptic Obstruction,
IEEE Trans. Robotics and Automation, vol. 17, 2001, pp. 698-708.
[7] S.A. Mascaro and H.H. Asada, “Understanding of Fingernail-bone
Interaction and Fingertip Hemodynamics for Fingernail Sensor Design,” Proc 10th Intl. Symposium on Haptic Interfaces for Virtual
Environment and Teleoperator Systems, 2002, pp. 106-113.
[8] S.A. Mascaro and H.H. Asada, Measurement of Finger Posture and
Three-axis Fingertip Touch Force Using Fingernail Sensors, IEEE
Trans. Robotics and Automation, vol. 20, 2004, pp. 26-35.
[9] S.A. Mascaro and H.H. Asada, The Common Patterns of Blood
Perfusion in the Fingernail Bed Subject to Fingertip Touch Force and
Finger Posture, Haptics-e, vol. 4, No. 3, 2006, pp. 1-6.
[10] Y. Sun, J.M. Hollerbach, and S.A. Mascaro, “Measuring Fingertip
Forces by Imaging the Fingernail,” Proc 14th Symposium on Haptic
Interfaces for Virtual Environment and Teleoperator Systems, 2006,
pp. 125-131.
[11] Y. Sun, J.M. Hollerbach, and S.A. Mascaro, “Dynamic Features and
Prediction Model for Imaging the Fingernail to Measure Fingertip
Forces,” Proc IEEE International Conference on Robotics and Automation, 2006, pp. 2813-2818.
[12] L.G. Shapiro, G.C. Stockman, Computer Vision, Prentic-Hall, Upper
Saddle River, NJ, 2001.
[13] M. Turk and A. Pentland, Eigenfaces for Recognition, J. Cognitive
Neuroscience, vol. 3, no. 1, 1991, pp. 71-86.
[14] M. Turk and A. Pentland, “Face Recognition Using Eigenfaces,” Proc.
IEEE Conf. on Computer Vision and Pattern Recognition, 1991, pp.
586-591.
3256