Task Recognition and Human - Machine Coordination
Through the Use of an Instrument - Glove
by
Jason R. Fortier
A.B., Engineering Science and History (1994)
Dartmouth College
B.E., Engineering (1995)
Thayer School of Engineering - Dartmouth College
Submitted to the Department of Mechanical Engineering
in Partial Fulfillment of the Requirements for the Degree of
Master of Science in Mechanical Engineering
at the
Massachusetts Institute of Technology
February, 2000
ENG
MASSACHUSETTS INSTITUTE
OF TECHNOLOGY
@ 2000 Massachusetts Institute of Technology
All rights reserved
SEP 2 0 2000
LIBRARIES
Signature of Author_
Depa Aent of Mechanical Engineering
January, 2000
Certified by
I
I4 aruhiko Asada
Professor of Mechanical Engineering
ThesjQrvisor
Accepted by
Ain A. Sonin
Chairman, Department Committee on Graduate Students
Task Recognition and Human - Machine Coordination
Through the Use of an Instrument - Glove
by
Jason R. Fortier
Submitted to the Department of Mechanical Engineering on January 25, 2000
in partial fulfillment of the requirements for the
Degree of Master of Science in Mechanical Engineering
ABSTRACT
This work focuses on human-machine interaction using an Instrument Glove as a
communication device. The Instrument Glove provides a new medium for a worker to
send signals to a computer system, based on the person's actions and gestures while
wearing the glove. A task recognition system is used to identify data signals from the
Instrument Glove and correctly label them as a specific task. To implement a task
recognition system, the concepts of feature space and feature vectors are explored as they
apply to pattern recognition techniques. By reducing the dimension of a feature vector
prototype, it can be used more effectively to identify random points in feature space.
This work implements an algorithm using multicollinear analysis to eliminate all
correlated data sets from a prototype feature matrix, thereby producing a more efficient
prototype feature vector to be used for task recognition. The classifier used for the task
recognition process is based on determining the minimum Mahalanobis distance between
a random hand movement, mapped to feature space, and a prototype task in feature space.
It is shown in this thesis that a reduced feature vector has an optimal size that will
maximize the success rate of the classifier. The multicollinear analysis feature reduction
technique produces a far more effective feature subset than the original feature vector, but
it is not an algorithm that will truly optimize the feature reduction process for any generic
task.
Thesis Supervisor: Haruhiko Asada
Title: Professor of Mechanical Engineering
2
ACKNOWLEDGEMENTS
I would like to thank Professor Harry Asada for giving me the opportunity to
study under him during my time here at MIT. The times here have not always been easy,
but he had faith in me and always pushed me to do my best work.
I also want to give very special thanks to Michelle Moore, who has been there for
me any time I needed someone to lean on or talk to over the last year and a half. I don't
know if I would have been able to keep pushing on through if she wasn't always there for
me.
All of my friends from my past lives at Dartmouth, Osmonics, and in Boston have
really been a huge help to me when I was looking for an escape from the lab. It's always
nice to know that there's an outside world where I can get away from it all.
And last, but not least, I am lucky to have such great parents who have offered
support and encouragement throughout my time here at MIT.
3
CONTENTS
1. O utlin e ..............................................................................................
1.1. Introduction ..............................................................................
8
8
1.2. B ackground ............................................................................
10
1.3. O utline of T hesis .......................................................................
11
2. Human-Machine Interaction ...................................................................
13
2.1. Human-Machine Interaction with an Instrument Glove ........................
13
2.2. An Example of Human Machine Interaction ....................................
15
3. Data Collection and Feature Vectors ..........................................................
18
3.1. Feature Vectors and Feature Space ....................................................
18
3.2. Creating Feature Vectors .............................................................
20
3.2.1. Creating a Biomechanical Model of the Hand .........................
21
3.2.2. Using the Biomechanical Model of the Hand ...........................
33
4. T ask R ecognition .................................................................................
35
4.1. Creating a Prototype Feature Vector for Task Recognition .......................
35
4.2. Discriminant Functions ................................................................
37
5. Feature Reduction ................................................................................
42
5.1. Reasons for Feature Reduction .....................................................
42
5.2. Theory of Multicollinearity.........................................................
43
5.3. Application of Multicollinearity for Feature Reduction ..........................
46
5.4. Discriminant Functions in Reduced Feature Space ...............................
47
6. Implementation and Results ...................................................................
49
6.1. Goals, Methods, and Set Up ........................................................
49
6.2. Equipment: The Instrument Glove ..................................................
51
6.2.1. Haptics and Pressure Sensors ............................................
52
6.2.2. B end Sensors ...............................................................
53
4
6.2.3. Position Sensor ............................................................
6.2.4. D ata Form at ................................................................
54
54
6.3. Implem entation .........................................................................
55
7. Industrial Applications ...........................................................................
64
7. 1. Time E stim ation ........................................................................
64
7.2. Human-Robot Coordination .........................................................
66
8. C onclusions .......................................................................................
68
Referen ces ..............................................................................................
70
5
LIST OF FIGURES
Figure 2.1
Figure 2.2
Figure 3.1
Figure 3.2
Figure 3.3
Figure 3.4
Figure 3.5:
Figure 3.6:
Figure
Figure
Figure
Figure
3.7:
3.8
3.9
3.10
Process flow chart for human-machine interaction example ...................
Ring finger kinem atics ...............................................................
15
17
19
22
23
24
24
Little finger kinem atics ..............................................................
25
Thum b kinem atics ....................................................................
26
30
30
31
Using an Instrument Glove to coordinate human-machine interaction ......
Feature extraction from a manual task ........................................
Skeletal system of the hand ....................................................
Index finger kinem atics ..............................................................
Middle finger kinematics ............................................................
Tendon structure in a finger..........................................................
Tendon structure in the hand .......................................................
Joint geom etry .........................................................................
Figure 3.11
Figure 4.1
Tendon displacem ents ................................................................
Figure 4.2
Illustration of a Mahalanobis distance classifier ...............................
Flow chart of a task recognition process ..........................................
Figure 6.1
Figure 6.2
Figure 6.3
Figure 6.4
Figure
Figure
Figure
Figure
Figure
Figure
Figure
Figure
2-D feature space with three decision regions ....................................
32
37
40
50
The Instrum ent G love ...............................................................
51
Force Sensing Resistors on the inner lining of the Instrument Glove ......... 51
T ask 1 (Pen ) ...........................................................................
56
6.5
6.6
T ask 2 (Screw ) ........................................................................
56
56
6.7
6.8
T ask 4 (C alculator) ..................................................................
6.9
6.10
6.11
6.12
Classification results from task 1 ...............................................
Figure 6.13
T ask 3 (Book) ........................................................................
T ask 5 (Plastic Part) ..................................................................
56
56
Classification results from task 4 ...............................................
59
59
60
60
Classification results from task 5 ...............................................
61
Classification results from task 2 ...................................................
Classification results from task 3 ...............................................
6
LIST OF TABLES
Table 6.1
Results of multicollinear feature reduction .....................................
Table 6.2
Results of classification using a feature set reduced by multicollinear
analysis ......................................................................
Table 6.3
Prototype Feature Vectors Used for Comparative Testing ...................
7
57
57
58
CHAPTER 1
INTRODUCTION
1.1 Introduction
Human-machine interaction is becoming increasingly more important in a modern
workplace. As the use of robots and automated machines continues to rise, an ideal robot
is seen by many not as an autonomous machine, but as a tool to assist a human worker.
In fact, the goal of a modern automated manufacturing facility is not to eliminate all
people from the process, but rather to integrate the efficiency of machines with the
knowledge and skill of people. In order for a human and a machine to work together
effectively, a process must be established to coordinate both of their actions.
This
human-machine interaction is an important area of research and the development of a
useful tool to transfer information between a worker and a robot will be useful in many
different fields. The goal of this thesis is to introduce the Instrument Glove as a highly
intuitive tool that is capable of coordinating human-machine interactions by monitoring
motions of the human hand and recognizing specific tasks performed by the wearer.
An Instrument Glove is an ideal tool that can be used by a worker to interact
naturally and intuitively with a machine. The glove is able to monitor all manipulations
of a human hand, and this information will allow a control system to determine the state
of the worker and then activate automated machinery to assist the worker as necessary.
The primary reason for choosing to use an Instrument Glove as a human-machine
coordinating tool in a manufacturing environment is that most of the work performed by
a worker is done with his hands. Therefore, by monitoring the state of a person's hands
at all times, it is possible to get very relevant information regarding the state of the
worker.
If the state of the worker is accurately known at all times, then informed
decisions and actions can be made by a control system so that the worker and any piece
of automated machinery can work together cohesively.
8
Currently, a person interacts with a machine by pressing buttons, pulling levers,
or typing commands on a keyboard.
Or possibly, in a more advanced setting, a
computer-based vision system could be capable of coordinating a robot's actions based
on input from a digital camera. These systems all have limitations and drawbacks that an
Instrument Glove is able to overcome.
For instance, a vision system is an indirect
measurement of a process and it can be very difficult to monitor a human. If the lighting
is poor, if there are any visual obstructions, or if the worker moves his body in a manner
that blocks the view of the camera, then no information can be obtained. In addition,
subtle hand manipulations would be very difficult to observe and record with a vision
system. Buttons, levers, or other direct input devices allow information to be passed to a
robot under all conditions, but they necessitate an extra step by the worker. If a worker is
attempting to perform a task, but has to use one, or both, of his hands to activate a
machine then time is lost and the task becomes more complicated than necessary.
The advantage of an Instrument Glove is that it is very simple to use, so tasks
never become more complicated than necessary, and it enables a direct measurement of
what a worker is manipulating with his hands. By creating a system that is able to
automatically determine the state of a worker without extra buttons or levers, the entire
task process becomes simpler for the worker. This allows for a more efficient working
environment. For instance, if a worker picks up a plastic part with one hand and a screw
with the other, then without every pushing a button the control system can automatically
realize that the next step in this process is to activate an automatic screwdriver. Since no
buttons or levers need to be pushed, this system flows very smoothly, less training is
required for the worker, and more work can be performed in the same period of time.
To accomplish this goal of using an Instrument Glove as a communication device
between a worker and a machine, a classification system must be developed that is
capable of interpreting data from the Instrument Glove as a specific task performed by
9
the wearer of the glove. A control system should be able to recognize discrete tasks and
determine what the next course of action should be, based on current conditions. This
thesis will concentrate on the technical issues involved with recognizing discrete tasks as
a person wearing the Instrument Glove performs them.
1.2 Background
The need for this area of research is found in industrial companies who are
continually trying to improve their working environment, both for the sake of the workers
and for corporate efficiency. Many modern manufacturing plants are focusing on the
need to integrate skilled workers with automated machinery, rather than simply
attempting to eliminate humans from the plant altogether.
Background work has been performed in the area of robotic skill acquisition by
researchers such as Liu and Asada [Liu and Asada, 1992], Asada and Izumi [Asada and
Izumi, 1989], and Yang [Yang, 1990]. But this work focuses on recognizing human
skills and transferring the skills to an automated system. The concept and some of the
technical issues involved in monitoring human manipulations is introduced these works,
but there is no emphasis on human-machine interaction. By using this work as platform
to study human manipulations, new work can be performed in the area of human-machine
interactions.
To create a human-machine interaction environment, a classification system must
be able to identify data from the Instrument Glove as a specific task. Work in the field of
pattern recognition by Duda and Hart [Duda and Hart, 1973] is instrumental in laying the
foundation of the task recognition techniques studied in this thesis. Similarly, work by
Nadler and Smith [Nadler and Smith, 1993] details a strong theoretical background for
pattern recognition, but both of these works focus heavily on optical pattern recognition
by vision systems.
The other heavily documented area of pattern recognition is for
10
speech recognition, but the concept of applying similar techniques to the unique
environment of an Instrument Glove is a new and unexplored area.
This thesis takes the concepts of human task monitoring and of pattern
recognition techniques, as they are used in different fields, and combines the work into a
study of task recognition via an Instrument Glove.
1.3 Outline of Thesis
This thesis focuses on the use of an Instrument Glove to identify specific tasks
performed by an assembly worker, and Chapter 2 begins this discussion by discussing
human-machine interaction in general terms and highlighting an illustrative example that
describes how a task identification system would improve the work process. Chapter 3
looks more closely at the data being analyzed by the Instrument Glove and it introduces
the concept of feature space and feature vectors. Feature vectors are the quantitative
tools used to describe a hand manipulation. This chapter also has an in-depth description
of a biomechanical model of the hand, which creates several additional features that can
be added to the feature matrix.
Chapter 4 discusses the technical issues of task
recognition and the theory of what a classification system must do to successfully
identify a worker's actions based on an incoming stream of digital data.
The
classification system described here is based on the feature vectors introduced in Chapter
3. While it may seems like a greater number of features will always add more useful
information about a task, and therefore assist with the identification process, it turns out
that a computer system is better able to recognize a worker's task if the number of
features it is monitoring is reduced to a reasonable number. This idea of feature reduction
is discussed in Chapter 5, and the concept of multicollinearity is introduced and applied
to reduce the dimension of the feature space.
Once the feature matrix is successfully
reduced to a manageable size, Chapter 6 lists the important experimental results of the
processes described in the earlier chapters.
11
Chapter 7 is set up to explore a few key
applications of this technology in an industrial environment. The final chapter, Chapter 8
reviews the conclusions of this work and states several recommendations for future work
along these lines.
12
CHAPTER 2
HUMAN - MACHINE INTERACTION
2.1 Human - Machine Interaction with an Instrument Glove
In an automated assembly environment, there is a two-fold relationship between
the workers and the automatic machines.
First there is communication between the
human and the machine, and second there is a physical interaction between them as they
work together. The communication from the machine to the worker is usually something
simple like a warning light, a horn, or maybe even a computer display. The worker
communicates with the machines by pressing buttons, typing commands, or triggering a
switch. The physical interaction between the two as they perform their necessary jobs
involves other tasks, such as loading the machine or holding a part in place. Since the
physical interaction and the communication from the worker to the machine both involve
discrete, and often separate, actions, the system could be greatly improved and simplified
if the worker was automatically communicating with the machine simply by performing
the normal physical interaction tasks.
By wearing a specially instrumented glove with sensors imbedded into the liner, a
worker would be able to communicate his exact state completely unconsciously to an
automated control system. The glove would be able to sense what parts the worker has
picked up based on hand configuration, location, and pressure points, and it could
immediately instruct a robot to assist the worker in the most appropriate way, depending
on the circumstances. The worker does nothing more than pick up ordinary parts, or
move a tool, and the robot can immediately jump into action at the proper time. No
buttons or levers ever need to be pushed. The interaction between human and machine
becomes highly intuitive and almost symbiotic as the two work together.
13
To accomplish this task, the computer system must first be instructed, or taught,
what to look for, and when the computer "sees" these "identifying characteristics", it can
put the correct label on the input. The "identifying characteristics" are referred to as
features. By apriori identifying several important features to monitor, and attaching a
specific label to these features, a computer classification system will be able to
successfully identify segments of data.
Using the Instrument Glove, a worker can initially teach several tasks, or actions,
to a computer system. These tasks are given labels that correspond to the action, such as
"picking up the screwdriver" or "attaching widget to the main gadget." By recording the
worker's actions for that particular label, a computer system will be able to identify
which features are most unique for that task and it will save the values for those features.
Later, during normal operation, when a worker performs that same task, the computer
classification system will be able to recognize the pre-recorded features and identify the
corresponding label, such as "picking up the screwdriver."
The process flow chart for this cycle of events is outlined in Figure 2.1. We will
discuss the middle two blocks of this flow chart, creating feature vectors and a
classification system, in more depth in the next few chapters.
By using an Instrument Glove and the task recognition concepts outlined above,
human-machine coordination should be greatly enhanced.
14
I
Worker performs normal task
Instrument Glove passes data
(feature vectors) to a classification system
Classification system identifies worker's
task
Automated system assists the worker
Figure2.1: Processflow chartfor human-machine
interactionexample
2.2 An Example of Human - Machine Interaction
To illustrate the idea of human-machine interaction through the use of an
Instrument Glove, an example is described here. Picture a simple assembly process, such
as screwing down a cap onto the main body of a plastic part. In a traditional automated
environment, with humans and machines working together, a worker may have to hold
both parts with two hands, and then with a third hand press the start button for the
machine. Or in a completely manual workplace, the worker would have to mount the
part, hold it in place with one hand, and then screw the parts together manually, which is
15
a time consuming and unskilled process. With coordinated human-machine interaction
and a task recognition system via an Instrument Glove, this process becomes much
quicker, simpler, and more intuitive. The work simply needs to pick up the cap with the
hand wearing the Instrument Glove and the computer classification system will
immediately recognize this action.
The system will be pre-programmed to begin
screwing the cap down to the main body part as soon as the worker snaps the two parts
into place. By holding the cap in place on the main part body, the identification system
will recognize that the part is in place and an automatic screwdriver assembly can swoop
in and insert all the screws. The worker never needs to press a start button. This process
is illustrated in Figure 2.2.
In this example, the worker will save time and potential strain by never needing to
perform the screwing operation, and he is able to use both hands for his task rather than
worry about pressing a start button on the robot. This increases the efficiency of the
process and makes the task performed by the worker simpler and easier to perform.
16
Automatic Screwdriver
Robot Arm
Instrument Glove
Main Body Part
Cap
Step 1 - Grasp the cap
411~
Step 2 - Place the cap on the main body part
Step 3 - Automatic Screwdriver attaches cap to main body part
Figure 2.2: Using an Instrument Glove to
coordinatehuman-machine interaction
17
CHAPTER 3
DATA COLLECTION AND FEATURE VECTORS
3.1 Feature Vectors and Feature Subspace
The Instrument Glove is capable of outputting a vast amount of data, but how can
this data be used to recognize the manual task being performed by the worker? In general
terms, a task is identified when certain characteristics of the data match some preset
values.
To accomplish this, the classification system must look for specific
characteristics that can describe the task.
These characteristics are called "features".
Each feature is scalar value that represents a piece of data from the glove or a function of
any combination of data from the glove. For example, given a segment of data output
from the glove, such as the bend angle of the index finger MPJ, a feature x1 may be the
mean value of that signal over a fixed time segment. Another feature, x2 , may represent
the mean value of the middle finger MPJ, or possibly the standard deviation of the index
finger MPJ. Combined in a vector form, the "feature vector", x, becomes:
x1
x=
x2
.
Figure 3.1 outlines the process of "feature extraction" from an arbitrary manual
task. First, a person performs some task while wearing the data glove.
The data is
collected on a computer system and then the feature values are calculated from the raw
output data to create the feature vector. In short, feature extraction can be thought of as
rendering the data output more suitable for the identification process. The feature vector
can be thought of as a "distinguishing characteristic" that defines the specific task at
hand.
18
Output Data from the
Instrument Glove
Manual Task
Si
t
S2
t
sm
t
Feature Extractor
x1
mean(s1 )
x2
max(s1 )
X3=
mean(s 2 )
.Xn.
. fn (S)
Figure3.1: Featureextractionfrom a manual task
19
By defining a feature vector, we are defining a multidimensional space called
feature space. A feature vector of length n will create an n-dimensional feature space.
By thinking in terms of feature space, one can see that creating a feature vector from raw
data reduces the entire raw data set to a single point in feature space. All of the relevant
data is mapped from the task space to feature space. Nadler and Smith sum up this
though concisely by saying, "each object encountered in a pattern recognition procedure
is represented by a single point in each hyperspace, whose coordinates are the values of
the features." [Nadler and Smith, 1993].
Since a task can be mapped to a single point in feature space, the classification
process becomes a matter of determining whether or not the current state of the
Instrument Glove data is close to a particular preset point when mapped into feature
space. These details will be explained in Chapter 4.
3.2 Creating Feature Vectors
Above, Section 3.1
introduces the concept of a feature vector and a
multidimensional feature space. The features mentioned in passing were the mean value
and maximum value of two of the analog output signals.
While any value deemed
relevant can be considered a feature in the feature vector, the values considered for this
thesis were the mean values and the standard deviations of each of each analog signal
output from the Instrument Glove. Using these two parameters, the 37 values that are
output from the Instrument Glove map to a 74-dimensional feature space.
The mean and standard deviation of each data signal were chosen as features
because of their ability to describe unique characteristics of the glove data. The mean
value is obviously important since it represents the average range the analog signal
operates in for a fixed time segment. The standard deviation is also considered important
because it is a measure of the fluctuation of the analog signal around the mean value. A
20
low value for the standard deviation could be a very important feature because it is a
quantitative description stating that the analog signal is constant and does not experience
very much fluctuation over the time segment in question. Using these parameters, the
base feature vector has the form:
xi
mean(sensorl)
x2
StdDev(sensorl)
mean(sensor 2 )
x3
X=
x4
=
StdDev(sensor 2 )
x53
mean(sensor 37 )
x 54 .
_StdDev(sensor 37 ).
Once the base feature vector has been established, it is still possible to add
additional features to the matrix, thereby increasing its dimension.
An advantage to
doing this is the fact that there may be other quantitative descriptions of the Instrument
Glove's output data that will be very important in terms of uniquely differentiating a
specific task from all other tasks. Examples of other possible features include minimum
values, maximum values, and higher order statistical moments.
More complicated
features can be used as well, which are evaluated as functions of several of the data
signals.
3.2.1 Creating a Biomechanical Model of the Hand
To create a more complete feature vector that can accurately account for subtle
differences in a person's movements, a biomechanical model of the hand has been
created. The outputs of this model include several values that can be added to the feature
vector. These include the position of each fingertip on the hand, the velocity of the
fingertips, the velocity of each joint, internal tendon forces created by external fingertip
21
forces, and internal tendon displacements caused by hand movements. These features
may prove to be more valuable than those included in the base feature vector due to a
greater dynamic range or more unique values for subtle movements. At this point, any
new features that describe a hand's motion will add extra information that can be used to
distinguish that motion from other hand manipulations.
Kinematics
To model the hand, wrist, and forearm accurately, one must investigate the
anatomy of the human body. The underlying rigid body structure of the hand is the
skeletal system, so we can model the bones of the hand as rigid bar linkages. Figure 3.2
illustrates the basic skeletal system of the hand.
(Distal
Phalanges
Smooth for
finger-nail
.
Middle
Proximal
-
Distal end
Metacarpals
Shaft or
Body
Metacarpals
Base
-.
Capitate
Trapezoid
Carpals Trapezium
Hamate
Scaphoid
Lunate Triquetrum
Figure 3.2: Skeletal system of the hand
Using the skeletal knowledge of the hand, and the knowledge of which angels we
are capable of measuring and calculating via the Instrument Glove, a kinematic model of
the hand can be created. For the five models shown below, Figure 3.3 through Figure
22
3.7, each represents an individual finger of the right hand. The origin coordinate system
of each model, xo-yo-zo, is common to all five models. This is a point located at the base
Xi
ai
di
0,
Link 1
0
11
0
01
Link 2
-900
0
0
02
Link 3
0
13
0
03
Link 4
0
14
0
04
Link 5
0
15
0
05
X_~
N33
Figure 3.3: Index finger kinematics
of the hand near the wrist. The zo axis points straight up, out of the back of the hand, and
the xO axis points forward through the knuckle (MIP joint) of the middle finger.
Looking at Figures 3.3 through 3.7, the models of the index, middle, and ring
fingers, one can see that the outer two joints on the fingers (the DIP and the PIP) are
modeled as a simple, one-degree of freedom pin joint, and the knuckle (MIP joint) is
modeled as a two degree of freedom pin joint. This assumption proves to be highly
accurate based on the physiology of human hand. In these models,
03
and
04
are the bend
angles of the MIP and the PIP joints, as measured by the Instrument Glove, and 05 is the
bend angle of the DIP, which is calculated based on the PIP measurement. An important
point of note is that 01 is a fixed angle, which represents the angle between the origin,
23
4
ai
ai
di
0,
Link 1
0
11
0
0
Link 2
-90"
0
0
02
Link 3
0
13
0
03
Link 4
0
14
0
04
Link 5
0
15
0
05
ZA
z2
Figure 3.4: Middle finger kinematics
xXi
ai
di
Link 1
0
11
0
-01
Link 2
-900
0
0
02
Link 3
0
13
0
03
Link 4
0
14
0
04
Link 5
0
15
0
05
Oi
KI
z,
13
Z3
Figure 3.5: Ring finger kinematics
24
xo-yo-zo, and the knuckle of the appropriate finger.
The Instrument Glove does not
measure this angle, and it remains fixed for a given person for all configurations of the
hand.
Finally,
02
is the absolute abduction angle, which is calculated based on the
relative abduction angles that are measured by the Instrument Glove. In the coordinate
frames of these models,
02
= 0 when the finger is pointing in a straight line from the
origin, and is positive when the finger moves to the left (on the right hand). This is
exactly the same convention used when calculating the abduction angles as described
above. Since this kinematic chain models the MIP joint as two pin joints, spaced apart by
a distance of zero, the parameter 12 = 0, which is the distance between the two joints on
top of each other, does not appear in the figure.
The model of the little finger, Figure 3.6, is slightly more complicated, due to the
fact that the palm can arch significantly near the little finger, allowing it to rotate under
the palm towards the thumb. This is accounted for in the kinematic model by including
an extra
14
q,
Z4
cXi
ai
di
0,
Link 1
-750
ao
0
01
Link 2
750
a1
di
02
Link 3
-900
0
0
03
Link 4
0
14
0
04
Link 5
0
15
0
05
Link 5
0
16
0
06
0,
Figure3.6: Little finger kinematics
25
joint between the origin and the knuckle of the little finger. 02 represents the amount of
rotation of this joint and it is measured directly by the Instrument Glove. Similar to the
other three fingers, 01 is a fixed angle, but in this case it does not intuitively point towards
any physical marker, such as a knuckle; 03 represents the abduction of the little finger, as
measured by the Instrument Glove; 04 and 05 represent the bend angles of the MIP and
the PIP joints; and 06 represents the bend angle of the DIP joint.
The thumb model is obviously a bit different than the four finger models due to
the physical differences of the digits. One of the important considerations that must be
taken into account while developing the kinematic model of the thumb is to keep in mind
which angles the Instrument Glove is capable of measuring. This is of course true for all
the digits, but one must make more of a conscious effort to model around this constraint
when dealing with the thumb because of its great flexibility. Figure 3.7 shows that, as
X4
X3(
cXi
ai
di
01
Link 1
750
a,
di
01
Link 2
900
0
0
-02
Link 3
900
13
0
900-03
Link 4
0
14
0
04
Link 5
0
15
0
05
ZI
d..
Z3,
Figure 3.7: Thumb kinematics
26
with the other four models, the same origin xo-yo-zo locates the model with respect to the
back of the hand. The outer two joints on the thumb (MP and IP joints) are modeled as
one-degree of freedom pin joints, and their bend angles 04 and 05 are measured directly
by the Instrument Glove. The trapeziometacarpal joint (TMJ), which is located at the
base of the thumb metacarpal bone near the wrist, is modeled as a two-degree of freedom
double pin joint. The first joint models the thumb rotation, and its axis extends in a line
from the TMJ through the MIP of the index finger (the index finger knuckle).
This
angular displacement, 02, is measured by the Instrument Glove as the thumb rotation
angle. The second joint rotates on top of the thumb rotation joint, and it accounts for the
abduction of the thumb. This angle, 03, is also measured directly by the glove, and unlike
the four finger abduction angles, requires no calculations.
The data that is shown adjacent to each finger model are the Denavit-Hartenberg
parameters for each kinematic chain.
This data is easily manipulated to provide the
endpoint location for each fingertip. Using the Denavit-Hartenberg parameters, one can
determine the endpoint location of the linkage with respect to the base coordinate frame.
(See [Asada and Slotine, 1986] for reference.) In terms of the hand, this means we can
establish the location of the fingertip with respect to the fixed location of the wrist. The
data from this analysis can be used to create new features for a prototype used for task
recognition (see Chapter 4.1), but the real-time identification process needs to compute
the endpoint fingertip position directly as a function of the Instrument Glove data. To
accomplish this, the index, middle, and ring fingers all have equations of the form:
xe =
licos(0 1) + cos(0 2)[1 3cos(0 3) + 14 cos(0 3+0 4 ) + l5cos(0 3+0 4+0 5)]
Ye = l1 sin(0 1) + sin(0 2 )[1 3 cos(0 3 ) + 14 cos(0 3 +0 4 ) + 15cos(0 3 +0 4 +0 5)]
Ze = -13 sin(0 3) - 14 sin(0 3+04) - 15sin(0 3+04 +05)
27
where Xe =
(Xe Ye Ze)T
is the endpoint (fingertip) position relative to the base coordinate
frame (the wrist). Similarly, for the little finger and thumb it is possible to establish the
general function:
Xe = Xe(e)
where
E is the vector of all the joint angles along the kinematic chain.
Tendons and Muscles as Actuators
Once we have established the kinematics of the hand, we can determine the
internal forces necessary for articulation and manipulation. The Jacobian matrix of Xe(E)
relates differential endpoint displacements to differential joint displacements, or more
importantly it relates the endpoint velocity to the joint velocities. This is written as:
DX
Ji =A
and
X = J16
The principle of duality states that if the system is in equilibrium, we can
determine the relationship between an external endpoint force and the torques at each
joint using this same Jacobian matrix. This relationship is:
t = J1TFextemai
Where t is the vector of all the torques at each joint and Fextemai is any fingertip pressure
that is recorded by the pressure sensors on the Instrument Glove.
Unfortunately, the human body is not so simple, and torques are not generated
directly at each joint. The torque on any joint is generated by muscles located away from
the joint, which are attached by tendons. This physiology requires that we study the
muscle and tendon structure in the hand so that we can determine the linear muscle forces
that are necessary to create the torques at all of the joints in the hand.
28
The physiological actuation system is a complex network of tendons and muscles
located in both the forearm and the hand itself. In fact, there are twenty different muscles
in the forearm that are used to actuate the hand, and an additional twenty in the hand. To
approach this problem, we must focus on isolating specific muscles, and associated
tendons, that are used in a few simple motions.
To model the actuation system, the
tendons and muscles can be treated as a cable and pulley system.
By solving the
kinematics of this actuation system, we will create a second Jacobian matrix that relates
muscle forces with joint torques. Combining this result with the first Jacobian of the
skeletal system, we can create a complete model that is capable of solving for internal
actuation forces as model outputs, based on the hand configuration and external forces as
inputs.
The first hand movement we analyze is the flexion and the extension of the
fingers. By examining human anatomy, we see that there are three main tendons which
lie on the underside of the finger that are responsible for flexing the fingers. The first
tendon attaches the flexor digitorum profundus muscle, located on the underside of the
forearm, to tip of the distal phalanx (the outer bone on the finger). The second tendon
attaches the flexor digitorum superficialis muscle, similarly located on the underside of
the forearm, to the middle phalanx (one bone in from the end of the finger). The third
tendon that contributes to the flexion of a finger connects the proximal phalanx to the
lumbrical muscles located in the palm of the hand. In contrast, to extend a finger, only
one tendon is involved. This is the extensor digitorum tendon, which attaches the distal
phalanx to the extensor digitorum muscle in the top of the forearm. A schematic of this
physical configuration is shown below, in Figure 3.8.
An important point that should be noted when examining the tendon structure in
the hand is the fact that the tendons run a great length from the muscle in the forearm to
the attachment point out near the fingertip, and they are not simply free to assume any
29
path between these two points. Each finger has a sheath, or a tunnel, that extends from
the mid-palm to the distal phalanx. At key mechanical points, primarily on either side of
all the interphalangeal joints, this sheath is rigidly attached to bone so that the tendon
Extensor digitorum
Flexor digitorum superficialis
Flexor digitorum profundus
Lumbrical
Figure 3.8: Tendon structure in afinger
follows a consistent path. This sheath is well lubricated, so we can assume there is no
friction as the tendon pulls on the finger. An illustration of this is shown in Figure 3.9,
which highlights the middle finger, but the structure is similar for all fingers.
414
r2
Zone 11
Zone I
Al
2
)A
2 A
.42
Figure3.9: Tendon structure in the hand
30
3A
Research by Kai-Nan An and William Cooney III, of the Biomechanics
Laboratory at the Mayo Clinic, have quantitatively defined all of these attachment points.
[Chao, et al., 1989] In this manner, we are able to accurately describe the geometry and
all the actuation angles in the fingers as the tendon is pulled from the forearm. The
results from this study are all given in a relative form, as a percentage of the length of the
middle phalanx, so that the results are valid for people with any size hand.
To establish a quantitative relationship between tendon displacements and hand
bend angles, one can see that along the length of the tendon, the total length between
fixed attachment points is always constant, except opposite a joint. For example, when
the finger flexes, all the slack created in the tendon length along the underside of the
finger is the result of the change in tendon length in the sections directly opposite from
the joint. This is illustrated in Figures 3.10 and 3.11.
iia.
Figure 3.10: Joint geometry
By expanding on this concept, we can see that the total tendon displacement,
which causes the finger to move, can be quantified by analyzing the geometry of the
tendon placements at each joint. We describe the geometry at each joint in the following
manner:
wi = [ai2 + bi2 + ci 2 + di2 + 2(bidi-aici)cos6; - 2(aidi+bici)sini]/
31
where ai, bi, ci, and di are known values based on the work of An and Cooney III, and Oi is
the measured bend angle from the Instrument Glove. Looking at the finger as a whole,
with regards to both flexion and extension, we see the following:
W7
W4
.W6
W4
W5
W2
W1
L
L
L4
L3
Figure 3.11: Tendon displacements
From this Figure we see that:
Li= wi
L2= w 2
(lumbrical flexion)
+
w3
(flexor digitorum superficialis)
L3= w 2 + w 3 + w 4
(flexor digitorum profundus)
L4= w 5 + w6 + w 7
(extensor digitorum)
where Lj is the total tendon displacement, and wi is the individual displacements at each
joint, as defined above. We now have a vector of all the tendon displacements as a
function of bend angles, L = L(O).
Using this relationship we can create a second
Jacobian, J2 , that relates differential tendon displacements to differential joint angles.
This is shown below.
J2 =
and
32
As was described earlier, the principle of duality states that the force on each
tendon is related to the torques at each joint by the transpose of the Jacobian matrix. This
relationship is:
t = J 2 TFendons
Where ' is the vector of all the torques at each joint and
Ftendons
is the force exerted by the
muscle on each tendon. By combining this equation with the similar equation derived
from the kinematics of the hand, we can solve for Ftendons based on
Fexternai
and the hand
bend angle data.
J1TFextemal = J 2 TFendons
Ftendons = J 2 - JiTFextemai
Using these results, it is possible to add fingertip position, tendon forces, and
tendon displacements as features in our feature vector.
3.2.2 Using the Biomechanical Model of the Hand
It is important to note some issues that reduce the usefulness of some of this
analysis. The most important point is that since the fingers are actuated by an opposing
cable and pulley system, when tendons on both sides of the finger are under tension, large
muscle forces
(Ftendons)
result from this effort.
are being generated, but no externally measurable forces
(Fextemai)
In this situation the stiffness of the finger and the hand are
increased but the hand is not pushing on any surface. The nature of the problem is such
that we cannot measure the internal forces in the system, so therefore we cannot truly
know the forces being exerted by the muscles on the tendons.
33
Even though we cannot solve for the absolute muscle forces, we can still solve for
the amount of muscle force that is necessary to create an external fingertip force on a
surface. This value will not include the opposing bias forces that create stiffness in the
finger, yet we can see the extra amount of muscle force necessary to push on an object,
and this may be a valuable feature for task recognition. Since we only have force sensors
on the underside of the fingertip, we isolate the flexion tendons that pull to create a
downward fingertip force. We can then solve
Ftendons =- J 2 -TJiTFextemai
where J 2 is a 3x3 matrix that only accounts for the flexion tendons.
This is a
mathematically solvable problem, since this square matrix, J 2 , is invertable.
Another valuable feature is tendon displacements. Using the geometry shown in
Figures 3.10 and 3.11, we can solve for tendon displacements for all hand movements.
Using the forward kinematic analysis of the rigid body structure of the hand, we are also
able to solve for the position of the fingertip at all times. It is also a straightforward step
to solve for the velocity of any point of the hand during a motion, as well as all of the
joint velocities.
As noted earlier, there are several potential advantages of including features from
the biomechanical model to assist with data recognition.
By looking beyond the
individual bend angles and analyzing the fingertip location and tendon actions, more
steady data may be available for analysis. These added features may prove to have a
greater dynamic range, which will make the feature vectors more easily identifiable by a
classification system.
34
CHAPTER 4
TASK RECOGNITION
4.1 Creating a Prototype Feature Vector for Task Identification
The feature vectors created by the Instrument Glove represent random points in
feature space.
Identification of these points is accomplished by comparing them to
another known point in feature space that represents a specific action. This known point
is called a prototype and it is associated with a specific label. It is necessary to determine
the coordinates of a prototype in a separate procedure before the task identification
process can begin. Before we discuss the theory of matching a random feature vector
with a prototype, the method used to create a prototype should be explained.
To create a prototype one must find the mean values of a feature vector, the
variance of each feature, and the covariance matrix of the feature vector. This way, it is
known what the average features are for a specific task, and what the variation of each
feature tends to be. These values all assist when a classification system attempts to
associate a random object with a prototype. In essence, a prototype is simply a point in
feature space with coordinates that are the mean values of all relevant features for a
specific task.
The first step in creating a prototype is to apply a label to a specific manual task,
such as "picking up part A." The goal then is to determine what the mean values of the
features of this action are, so that a new random feature vector in feature space can be
compared to these values. Mean values are used for obvious reasons. For any given
manual task, a person will always hold their hand slightly differently each time, so it
would not be correct to use one instance of that action as a prototype. The action must be
repeated N times by the person wearing the Instrument Glove to determine the mean
35
value and variance of each feature. In addition, the covariance matrix can be calculated
from this data. At the end of this procedure, a label, which we will refer to as LI, is
associated with N trials of data from a person performing the specified task.
The N feature vectors are concatenated into a "feature matrix" and transposed so
that each column of the feature matrix represents a feature, and each row represents a
trial.
1
Feature
Featur e2
(4.1)
Feature n
x 11
X12
... Xln
X X2 1
X2 2
X2n
LXN1
XN2
Trail
... XNn.
1
Trail N
From this feature matrix, X, the mean (mj), variance
(oj2),
and covariances (S) of
the features are easily calculated by
(4.2)
1 N
mj = - Exj
i=1
2
N
(4.3)
G= N
l(xij - mj)
N -I
(4.4)
S =X(xi -mj)(xi -mj) T
n
(n x n)
i=1
By performing these calculations on N trials of a specific task, all the critical
values for a prototype are known.
36
4.2 Discriminant Functions
In simple terms, to accomplish the goal of task recognition one must determine if
a random feature vector generated by the Instrument Glove falls reasonably close to a
prototype in feature space, which represents a specific task. Section 5.1 detailed the
creation of these prototypes, which map to specific tasks, so in this section we look at the
decision process that determines whether or not a random feature vector can be classified
as a specific task.
The mathematical equation used for this decision process is a
discriminant function.
Richard Duda and Peter Hart [Duda and Hart, 1973] describe several different
minimum distance discriminant functions used in the area of pattern classification, but
their basic assumption was that all random feature vectors must map to a specific label.
Their work focuses on the statistical process of finding the label with the highest
probability of being the correct identification. The entire feature space is partitioned with
dividing lines to determine which label is associated with every point in feature space.
f2
Figure4.1: 2-D feature space with
three decision regions
37
The difference between that assumption and this work is that not all of the random
feature vectors created by the Instrument Glove map to a label. For instance, if the
worker wearing the glove is taking a break and not doing any work, then whatever point
the glove represents in feature space at that point in time must not map to any label at all.
This concept is shown in two dimensions in Figure 4.1. In this figure, any object that
falls into regions,
9 1 I,
912, or 913 receive the classification corresponding to that region,
but a random object outside of those regions receives no classification at all.
The simplest way to determine whether or not a random object in feature space
can be classified as a specific action is to measure the distance between the random
object and the closest prototype object and see if the distance is less than an acceptable
value. This expression can be written as:
if (d(xo,xp) < r ) then (Lp)
where
d = distance between objects xO and xP
xO = random object in feature space
xp = prototype object in feature space
Lp = label for associated prototype in feature space
r = threshold distance
Using this procedure, every random feature vector generated by the Instrument
Glove is evaluated against each prototype, one at a time. A classification decision is
made for each decision region surrounding a prototype; either the random feature vector
falls inside this region or it does not. If the random vector falls inside the decision
region, then that vector is classified according to the label attached to that decision
region. If it falls outside of that decision region, then no label is attached and the random
vector is compared to the next prototype.
38
If we consider this problem using a standard Euclidean distance between points, a
few problems arise. First, how is the threshold distance, r, determined? There is no
rigorous method to determine an acceptable value of r. The second problem is related to
the first, and it is that each axis in feature space has its own scale, so that it would be
possible to set r to a value where one axis will always dominate the distance function.
To overcome these issues, the Mahalanobis distance is used for the function
d(xo,xp). The Mahalanobis distance between two points, x0 and xp is defined as
(4.5)
d(xo,xp)
where S is the covariance matrix.
(xO -xP)TS-l(x
-XP)
This measure of distance is superior because it
accounts for standard deviations and covariances along each axis. The value for the
distance, d, is actually a measure of the number of standard deviations between the
points. This eliminates the first problem associated with Euclidean distances, which is
where to set the boundary of acceptability.
By dealing with multiples of standard
deviations, it is more justifiable to make this decision, such as all values within 3
standard deviations of the point xP are acceptable. In two dimensions, this concept is
illustrated in Figure 4.2, where the value m is the prototype point in feature space and the
decision boundary is set at
P standard
deviations from this point.
The illustration
highlights a scenario where the standard deviation of the f2 feature is larger than that of
the f, feature and there is a positive correlation between the two features. In this case, the
random object xi,1 falls within the decision boundary while object
xi, 2
is outside of the
decision boundary, even though it is closer to m in a Euclidean sense. This is due to the
fact that xi,1 is still within
P standard deviations of m, while xi, 2 is not.
39
d
Xi,2
m
Xi,'
Figure 4.2: Illustration of a Mahalanobis
distance classifier
Using a minimum Mahalanobis distance classifier, our discriminant function is
given by
if (d(x,,m) <
) then (Lp)
where
(4.6)
d(xo,m) = j(xo
m)TS-l (xo -i)
and
xo
= random object in feature space
m = mean feature values of prototype object in feature space
Lp = label for associated prototype in feature space
P = threshold distance, measured in standard deviations
S = feature covariance matrix
At any point in time, the Instrument Glove will be outputting data which will be
mapped to a feature vector, so the classification process is simply a matter of having a
number of stored prototypes and computing the distance from the unknown object to each
40
of the prototypes. If that distance is less than the boundary distance, P, then the unknown
object is classified as the associated label.
In the case of overlapping decision regions in feature space, a random vector can
theoretically be mapped to two or more separate labels. This would obviously lead to
erroneous results, since any random vector must only be able to map to one label. In the
case of overlapping decision regions, the highest probability mapping scenario is to map
the random object to the prototype with the shortest Mahalanobis distance. While it is
statistically possible that this will be a false reading, since the random vector is still
within
P standard
deviations of another label point, the shortest Mahalanobis distance
will represent the highest probability mapping.
41
CHAPTER 5
FEATURE REDUCTION
5.1 Reasons for Feature Reduction
We saw in Chapter 3 that there exists an infinite number of features that can be
used in a prototype to describe a given task.
The question that arises is how many
features should be used for an effective and efficient classifier? In the words of Nadler
and Smith, "There are a great number and variety of features that can be designed for a
given problem. Not all of these features need to be or, indeed, can be used." [Nadler and
Smith, 1993]
The basic idea is that a classifier becomes more simple and efficient if a "useful"
subset of features is retained and the "useless" ones are eliminated. It has been shown
that the quality of task recognition actually declines when extra features are added to a
set. "When a small sample is used for training, it turns out that there is a so-called
peaking phenomenon.... Initially performance improves as new features are added, but at
some point inclusion of further features will result in performance degradation. Thus
dimensionality reduction may achieve actual decrease in recognition error rates." [Nadler
and Smith, 1993]
Intuitively, this statement makes sense.
If there are an excessive
number of features that must be matched to a prototype, there is a higher probability that
one feature may lie outside of the Mahalanobis acceptance region, even if the task should
be correctly identified by that region.
After all, humans are not as repeatable as
machines, and there are always subtle differences in the way we perform a given task.
An ideal classifier is not one that attempts to use every possible feature, but rather one
that only uses the most necessary and descriptive features. Features that are the most
unique for a particular label should be retained, and all the other ones discarded.
42
To determine which features are useful and should be retained is not a problem
that is easily solved. In fact, "It has been shown that no algorithm can be constructed for
the selection of an optimal subset of features - other than the trivial algorithm of
exhaustively testing every subset." [Nadler and Smith, 1993] From the viewpoint of
pattern recognition, every problem is unique and the particular features that are most
descriptive for a class of objects may contain no useful information at all for a different
set of objects.
To derive a theoretical approach for picking a useful subset of features, one can
look to statistical methods and the concept of correlation and collinearity. By definition,
two correlated features move with a linear relationship to each other, so it is easy to see
that the two features contain redundant information.
By monitoring one feature, the
value of the other can always be estimated. Therefore, the second feature adds no new
information to the classifier and it can be eliminated from consideration.
By
systematically eliminating all the dependent features, and retaining the independent
features, the dimension of the feature vector can be reduced and no accuracy will be lost
by the classifier.
5.2 Theory of Multicollinearity
Multicollinearity can be defined as an "approximate" linear dependency between
columns of a matrix. [Gunst and Mason, 1980] This is useful from the standpoint of data
reduction because if one column of data is found to move linearly with another column,
or several other columns, it can be removed from the data set with no real loss of
information.
When the reduced data set is analyzed, all the information that
distinguished the eliminated column can be found in the remaining data set.
For the case being dealt with here, each column of data represents a specific
feature. The column length, N, represents the number of trials used to define a task.
43
Therefore, by detecting multicollinearities between feature columns, a specific feature
can be eliminated from consideration during the task recognition process. In other words,
if it can be shown that the mean value of a specific joint in the hand varies linearly with
the mean value of two other joints, then the first joint only provides redundant
information and can be ignored.
Formally, a multicollinearity exists between column vectors aj if
n
(5.1)
Xcjaj = 0
j=1
where cj are constants, not all of which are zero.
To detect multicollinearities, the matrix under observation must be standardized
using a unit length scaling [Gunst and Mason, 1980]. Standardization of vectors using
the unit length scaling transforms all variables to numbers between -1 and +1.
This
ensures that all the columns are of the same order of magnitude and the constants, cj,
accurately describe which columns are most heavily involved with the multicollinearity.
For a feature matrix X, the unit length scaling form is achieved by
X-- -x-
(5.2)
W = Xi
(5.3)
X=-
J
dj
where
- 1N
Xij
i=1
N
(5.4)
dj = I(Xij -Xj)
2
=S
2
(N -1)
J
i=1
In this standardized form, W'W becomes a correlation matrix, shown as:
44
P12
1
(5.5)
...
Pin
P2n
W'W = P21
1 .
Pn2
.Pni
Since the feature matrix X was standardized, the value puv is the correlation
coefficient between the uth and vth variables.
These values show the pairwise
collinearities of the feature matrix. The values of one on the diagonal represents the
standard deviation of each standardized variable.
To detect correlations between three or more variables, one must look beyond the
correlation coefficients shown in W'W and investigate the eigenvalues of the W'W
matrix [Gunst and Mason, 1980]. The definition of eigenvalues and eigenvectors for any
matrix A is
(5.6)
Av = kjvj
so it follows that for any eigenvalue close to zero, j =0,
(5.7)
W'Wvj =Xjvj = 0
since all of the components of the eigenvector vj are less than one in magnitude.
Pre-multiplying by vj produces
(5.8)
V'j W'WVj = 0
Let U = Wvj, so equation (5.8) becomes
(5.9)
U'U=xU2 =0
Since all terms in the summation are non-negative, U'U = 0 implies U = 0. Therefore,
for an eigenvalue close to zero,
(5.10)
WVj
i = 0,
v=vjW = 0
45
Comparing equation (5.10) to equation (5.1) shows that if an eigenvalue is close
to zero, the terms of the associated
multicollinearity.
eigenvector become the coefficients
of
The larger terms in vj determine which columns of W are most
involved in the multicollinearity. Since W is simply the standardized form of the feature
matrix X, we can see exactly which features behave in a multicollinear manner with other
features.
5.3 Application of Multicollinearity for Feature Reduction
The theory of multicollinearity describes a statistical method to determine if
columns of a matrix behave in a collinear, or correlated, manner. In Chapter 4.1 we saw
that a feature matrix that is used to define a prototype is arranged with each column
representing a feature.
Therefore, by applying the concept of multicollinearity to a
prototype feature matrix generated from the Instrument Glove, it is possible to identify
which features behave in a multicollinear fashion.
By standardizing the feature matrix with unit length scaling and determining
which eigenvalues of the X'X matrix are close to zero, the multicollinearities between the
features become apparent. In a systematic manner, each eigenvector whose eigenvalue is
less than 0.01 is examined and the term with the largest absolute value is identified. The
feature that corresponds to this term is then eliminated, since this feature is the one that is
the most heavily involved with the multicollinearity. In other words, for an eigenvalue
less than 0.01, if the third term of the corresponding eigenvector is the largest, in absolute
terms, then feature number three is the most involved feature in terms
of
multicollinearity, so it offers the most redundant information and can be eliminated.
In this manner, after all possible features are eliminated by this method the
remaining features are the most unique.
All of the redundant information from the
multicollinear features is no longer part of the feature matrix.
46
5.4 Discriminant Functions in Reduced Feature Space
Multicollinear analysis must be performed on each prototype, individually, to
determine which features associated with that task are correlated. After this analysis has
been performed and the dimension of the prototype feature vector is reduced, there is no
guarantee that each reduced feature vector will have the same length. Therefore, each
reduced feature vector will determine a subspace of feature space, and each reduced
prototype will have a unique subspace. Even if the dimensions of two reduced prototypes
are the same, chances are that they will not have the same features as components, so
their subspace axes will be different.
Using a Mahalanobis distance classifier allows us to use the classifier in any
dimensional space, so we can perform the same analysis for each reduced prototype.
Since the Mahalanobis distance is a scalar value representing the distance between two
points scaled by the standard deviation along every axis, it is irrespective of the number
of dimensions being considered.
The calculation holds for any dimension and will
simply determine the number of standard deviations between the two points.
Furthermore, since each axis is scaled by its own standard deviation, then comparisons
between different axes are valid. For instance, if the distance between two points in rdimensional space is found to be 3 standard deviations, based on the standard deviations
along each axis, and the distance between two other points in s-dimensional space is
found to be 8 standard deviations, then we can see that the first pair of points are closer in
a relative sense. In this manner, we can compare random vectors produced by a person
wearing the Instrument Glove to several prototypes, even if those prototypes occupy
different subspaces.
Applying this concept to the discriminant function described in Chapter 4,
classification is accomplished as follows. First, each prototype is reduced to its own
47
unique subspace, which is a subset of the complete feature space. Then, to classify a
random vector produced by the Instrument Glove, the classifier reads only the features
contained in the union of all the reduced feature prototypes. This way, the classifier does
not need to read every single available feature, and computation time is saved. One at a
time, the relevant features from the random vector are compared to each prototype and a
Mahalanobis distance is calculated in the proper subspace. If the Mahalanobis distance is
less than the threshold level,
f,
then the random vector is classified as the label associated
with that prototype. If the Mahalanobis distance is outside of the
at least one of the important features lies outside of
P threshold level,
P standard
then
deviations from the
prototype so the random vector is not classified as that label. The random vector is
compared to each prototype in its proper subspace in this manner, and for each
comparison a decision is made; the random point is classified as the associated label if
the Mahalanobis distance is less than P, or the random point receives no label from the
prototype because the Mahalanobis distance is greater than P.
In the case where a
random point can be classified by more than one prototype, the object is classified by the
shortest Mahalanobis distance.
Since a Mahalanobis distance is always scaled by
standard deviations in any dimension, comparisons between different subspaces are valid
and the shortest Mahalanobis distance corresponds to the highest probability of correct
classification.
48
CHAPTER 6
IMPLEMENTATION AND RESULTS
6.1 Goals, Methods, and Set-Up
Based on the discussion of task recognition in the previous chapters, it is our goal
to implement a task recognition system using the Instrument Glove. Experimentation
will show that the minimum Mahalanobis distance classifier is an effective means of
differentiating random human motions and classifying specific motions as unique tasks.
Furthermore, the concept of feature reduction is implemented and the use of
multicollinearity for the purpose of feature reduction is analyzed.
To implement the task recognition system, there are several steps involved that
are briefly outlined in Figure 6.1. First, one must create one or more prototypes that will
be used to classify random hand motions. This is accomplished on a real-time system
using the Instrument Glove. Data is recorded while the task is performed N times in
order to create mean values and standard deviations for every feature. Once this data is
recorded, feature reduction takes place on an off-line system to identify the most
important features for each prototype. The mean values and standard deviations of each
feature that is identified by the feature reduction procedure are then loaded back into a
real-time system. At this point the classification system is able to begin identifying hand
motions based on output data streaming from the Instrument Glove and the prototype
feature vector. The specific details of each of these steps are discussed below.
The hardware and software that is required for this procedure includes the
Instrument Glove, a personal computer, and several custom programs written in C++ and
Matlab. First, a C++ program is used to record data for the prototypes and is part of a
real-time system. Then, two Matlab programs are used during the offline analysis of the
49
Record all features from apriori trials
for N specific tasks
(create a prototype)
On-line
"F
Feature reduction
(multicollinear analysis)
Off-line
Task identification criteria:
Compute fand for selected features to
create a Mahalanobis decision region
Read glove output analog signals
only for selected features
No
Do values of real-time
signals fall within a Mahalanobis
On-line
decision region?
Yes
Task is identified
Figure 6.1: Flow chartof a task recognitionprocess
feature vector. One contains much of the biomechanical model and it is used to create
additional features for the feature vector, while the other performs a multicollinear
analysis on the feature matrix and reduces the dimension of the feature vector according
to the output of the multicollinear analysis. The final piece of software is part of the real
time system and it runs the classification program used to identify specific tasks using
data output from the two Matlab programs.
50
6.2 Equipment: The Instrument Glove
Since the Instrument Glove is the piece of equipment that directly reads all of the
data used for this work, it is important to understand it thoroughly. To capture a person's
hand movements, the Instrument Glove consists of three distinct components. The main
part of the glove, shown in Figure 6.2, is a snug fitting glove that has a number of sensors
imbedded in the lining. These sensors are able to capture all the bend angles of the hand
as it moves through any motions. Attached to the back of the glove is a position and
orientation tracker, which can be seen in Figure 6.2. Under this main part of the glove is
a thin cotton liner which is responsible for monitoring all touch forces, and this can be
seen in Figure 6.3. Three wires extend from the back of the glove to a computer system,
one each for the bend sensors, the position tracker, and the touch sensors. The specific
details of the glove are discussed below.
Figure6.3: Force Sensing Resistors on the
inner lining of the Instrument Glove
Figure 6.2: The Instrument Glove
51
6.2.1 Haptics and Pressure Sensors
One of the important considerations that must be taken into account when using
the Instrument Glove is maintaining the wearer's haptic senses. If the goal of the data
acquisition is to accurately monitor the wearer's movements for analysis, the wearer must
be able to feel the environment and manipulate objects as naturally as possible.
Obviously, unnatural movements that result from impeded haptic senses will lead to
inaccurate results.
In accordance with our goal of creating a haptically unobtrusive glove, it would
be ideal to leave the contact surfaces of the hand exposed. The practical conflict with this
requirement is that in order to measure external forces generated by contact, one must
place a pressure transducer between the hand and the contact surface.
This is an
unfortunate predicament, and the problem is dealt with as best as possible.
The current solution to this problem is the use of small, flexible, force sensing
resistors (FSRs). These FSRs have a thin, 0.018" thick, plastic body and are sufficiently
pliant to allow the hand to go through most normal movements with no interference.
These FSRs have a force range of approximately 0 - 10kg of force.
There are six FSRs wired to the Instrument Glove. Five on these are placed on
the fingertips of each finger plus the thumb, and the sixth in located on the thenar
eminence region of the palm (the fleshy section of the palm at the base of the thumb).
These locations were chosen because they are the most commonly touched parts of the
hand during normal grabbing, pinching, or pushing motions. We have not added more
than six sensors at this time because more sensors will begin to encumber the Instrument
Glove and may inhibit natural movement of the hand.
52
6.2.2 Bend Sensors
The Instrument Glove is able to measure eighteen bend angles on the hand and
wrist, and calculates four more angles from basic physiology. The measurements are
accomplished by using flexible sensors that are imbedded in the glove and are located
over the joints in the hand and wrist. On the thumb, there are two sensors that measure
the metacarpophalangeal and interphalangeal joints (MPJ and IPJ) (the outer two thumb
joints). On the four fingers, there are two sensors on each finger to measure the MPJ and
proximal interphalangeal (PIP) joints, but the distal interphalangeal (DIP) joint (the outer
joint on each finger) is not measured. The DIP joint is instead calculated from the angle
of the PIP joint, since these two joints are physiologically coupled. For a typical person,
(6.1)
6 DIP
= O.5*Opip
There are four abduction sensors on the glove, located between the middle and
index fingers, the ring and middle fingers, the little and ring fingers, and on the thumb.
Two sensors are included to measure the arch of the palm. One is located near the thumb
to measure how much the thumb rotates across the palm towards the little finger and the
other is located near the little finger to measure how much the little finger rotates across
the palm towards the thumb. There are two additional sensors located on the wrist, one to
measure wrist pitch and the other to measure wrist yaw.
The bend sensors have an
accuracy of ±0.50.
Since the Instrument Glove can only measure the relative abduction angles
between fingers, the absolute abducted position of each finger must be calculated.
For
the four fingers, only three relative angles are measured, so we must assume a constraint
on the system to calculate the absolute position of all four fingers. This is accomplished
by estimating a gain value that represents the proportion the middle finger should abduct
in response to a change of the middle-index abduction sensor value and the middle-ring
abduction sensor value. The sensor output, (p, has a sign convention such that when the
fingers are spread, the angle (p is positive, while the absolute abduction angle, 0, has a
53
sign convention such that rotation to the left (towards the thumb on the right hand) is a
positive angle and rotation towards the little finger is a negative angle. Using a gain
value Kp,
(6.2)
9
(6.3)
OAb,Index = OAb,Middle + (PMI
(6.4)
OAb,Ring
(6.5)
Ab,Little
Ab,Middle = ((PMI - (PMR) *p
OAb,Middle
- (PMR
OAb,Ring - (PRP
Typical gain values for Kp are about 0.5.
This approximation holds reasonably well when the hand moves freely and is not
constrained by an external force. But, if a finger is pushing on an object, or is being
obstructed from its normal movement in any way, then these formulas cannot predict the
actual location of the abducted finger.
6.2.3 Position Sensor
The final ability of the Instrument Glove is to measure its relative position and
orientation in space. A remotely mounted transmitter generates a magnetic field, which
is measured by a receiver that is attached to the wrist of the Instrument Glove. In this
manner, the receiver is able to compute its position and orientation, in six degrees of
freedom, relative to the transmitter. The receiver must be located approximately within
4' of the transmitter, but it has an accuracy of ±0.1" and ±0.5'.
6.2.4 Data Format
The Instrument Glove outputs a number of analog signals for use by the computer
classification system. Since the FSRs are simply variable resistors, they require an input
voltage, and after passing through an op-amp circuit, the output voltage is read as an
54
analog signal. The conversion between the FSR resistance and the applied force is a nonlinear relationship that roughly follows an inverse power law curve (approximately 1/R).
The actual relationship between the analog voltage signal and the applied force is a
function of this non-linear relationship and of the resistance used in the op-amp circuit.
For simplicity, force will be dealt with in units of Volts. The analog signals from the
FSRs are acquired using a Keithley DAS 1800 Analog-to-Digital board in a personal
computer (PC). The bend angle data from the glove is input via a serial port on the PC
using software provided by Virtual Technologies Inc, the manufacturers of the
commercially available glove. The data from the position tracker is acquired using a
special ISA board in the PC, available from the manufacturers of the tracker system,
Ascension Technology Corporation.
6.3 Implementation
Several experiments were performed to analyze the capabilities of the minimum
Mahalanobis distance classifier and the effectiveness of multicollinear techniques for
feature reduction.
Prototypes were created for five basic tasks in order to gauge the
accuracy of the task recognition system. These tasks were gripping a pen; picking up a
screw; picking up a book; holding a calculator; and grasping an irregularly shaped plastic
part. Photos of these five tasks are shown below in Figures 6.4 through 6.8.
Multicollinear Analysis Results
The base feature vector had a dimension of 76 before any reduction took place.
This number is too large to be dealt with effectively or efficiently by a classification
system, so one of the goals of the multicollinear analysis is to reduce the size of this
vector. The result of the multicollinear analysis is that the size of the feature vector was
reduced to 11 to 21 features, depending on the task. Position data was excluded from the
reduced feature vector, since all tasks were to be performed in any workspace and should
55
Figure 6.4: Task 1 (Pen)
Figure6.5: Task 2 (Screw)
Figure 6.6: Task 3 (Book)
Figure6. 7: Task 4 (Calculator)
Figure6.8: Task 5 (PlasticPart)
56
only be classified by manipulations of the hand, not the location of the task. The specific
results are shown below in Table 6.1.
Task 1: Pen
reduced to 11 features
Task 2: Screw
reduced to 21 features
Task 3: Book
Task 4: Calculator
reduced to 16 features
reduced to 18 features
Task 5: Plastic Part
reduced to 17 features
Table 6.1: Results of multicollinearfeature reduction
Using these feature vectors as prototype feature vectors, the classification system
was tested to see how effectively random hand motions could be identified as the proper
task. While wearing the Instrument Glove, each task was performed 25 times, and the
numbers of correct and incorrect classifications were recorded to determine an overall
success rate of the system. The overall success of the classification was determined by
subtracting the number of errors from the number of correct identifications, and dividing
by the number of trials.
Success Rate =
(Correct Identifications - Errors)
Number of Trials
As Table 6.2 shows, the results varied depending on the task. Some tasks, such as
picking up the screw, were not classified very well, but others, such as picking up the
book, were done very easily.
Trials
Correct ID's
Errors
(Correct - Errors)
Success Rate
Task 1: Pen
25
17
0
17
.68
Task 2: Screw
25
14
2
12
.48
Task 3: Book
25
22
0
22
.88
Task 4: Calculator
25
15
0
15
.60
Task 5: Plas. Part
25
16
3
13
.52
Table 6.2: Results of classificationusing a feature set reduced by multicollinearanalysis
57
To determine whether or not the feature vector produced by the multicollinear
analysis is an optimal feature subset, it is necessary to compare it to other feature subsets.
Several other feature vectors were created from the same original base feature using
different methods.
One prototype feature vector was chosen by randomly picking
features of approximately the same dimension as the one produced by multicollinear
reduction, and several other feature vectors were chosen manually of varying sizes. Each
of these prototype feature vectors was entered into the identify.cpp program and the
effectiveness of each feature vector was compared to the others. Each of the feature
subsets used for comparison is listed in Table 6.3.
1.
2.
Reduced using statistical multicollinear techniques
Reduced by random feature selection
3.
4.
Complete feature vector
Reduced to 37 manually selected features
Reduced to 32 manually selected features
Reduced to 27 manually selected features
Reduced to 22 manually selected features
5.
6.
7.
8. Reduced to 17 manually selected features
9. Reduced to 12 manually selected features
10. Reduced to 7 manually selected features
Table 6.3: PrototypeFeature Vectors Usedfor Comparative Testing
Figures 6.9 through 6.13 show the graphical interpretation of the comparative
results. One can see that the number of features chosen for the classification system has a
large impact on the effectiveness of the system.
58
Task 1: Pen
1.00
0.900.80 0.70
0.60 0.50
0.40 -
+
Chosen Features
+
Multicollinear Results
A
Random Features
Trendline
0.30 0.20
0.10
0.00
0
10
20
30
40
50
# Features
Figure 6.9: Classificationresults from task 1
Task 2: Screw
1.00
0.900.800.700.60
0.50 (D 0.40 -
,
*
Chosen Features
*
Multicollinear Results
A
Random Features
-- - Trendline
0.300.20 0.10 -
0
0.00
0
10
20
30
40
50
# Features
Figure6.10: Classificationresultsfrom task 2
59
Task 3: Book
1.00
0.90
0.80
0.70
0.60
C,Cc
0.50
C.) 0.40
U 0.30
0.20
0.10
0
10
20
305
0
10
20
30
*
Chosen Features
*
Multicollinear Results
A
Random Features
-- - Trendline
0.00
40
50
# Features
Figure6.11: Classificationresults from task 3
Task 4: Calculator
1.00
0.80 0
z
+
c0.60
S -
*
0
S0.40A
Chosen Features
Multicollinear Results
Random Features
Trendline
0.20
0.00
*
0
10
20
30
# Features
40
50
Figure 6.12: Classificationresultsfrom task 4
60
Task 5: Irregular Plastic Part
1.00
0.80 0.60 -
.
CO)
CO
*
0.40 -
Chosen Features
* Multicollinear Results
* Random Features
- - - -Trendline
'
0.20 -0
Wl 0.00 _
20
10
-0.20
30
40
A
-0.40
# Features
Figure 6.13: Classificationresultsfrom task 5
These plots show several interesting results. The most obvious is the fact that the
number of features included in the prototype heavily influences the success rate of the
classification system.
The tendlines added to each plot are third order best-fit
approximations of the manually chosen feature vectors. These tendlines clearly show
that there is an optimal number of features that should be used for classification, and this
number varies depending on the task. For tasks 1, 2, and 5, the optimal number of
features is in the range of 12 - 15, but task 3 appears to be optimized by about 30 tasks
and task 4 requires only 5. The fact that classification improves as features are initially
added, but then begins to decline after a critical number of features has been reached
agrees
with observations
described by Nadler and Smith in their discuss of
dimensionality reduction with respect to feature vectors. [Nadler and Smith, 1993]
The reason one sees a drop-off in success rate as the dimension of the feature
vector is increased can be rather easily explained. Primarily, the explanation is that the
human hand, and therefore human manipulations, is so versatile that for any given task
61
there are an almost infinite number of slight variations in the resulting motion. When one
is monitoring a great number of features using a minimum Mahalanobis distance
classification system, the probability that a single feature will lie outside the decision
region grows rather quickly. For this reason it is very important not to attempt to classify
a task based on extraneous or unimportant features.
One should only use a reduced
feature set that emphasizes the most important features associated with the task.
When we compare the success rate generated by a manually chosen feature vector
to the success rate of a feature vector chosen my multicollinear analysis, we see that the
later subset is a well-selected one. In each instance, the results from the feature vector
reduced by multicollinear analysis lies above the trendline, indicating that this feature
subset is more effective than our chosen manually selected feature vector of the same
dimension. But we also see that while the results from multicollinear analysis provide a
good feature subset, it does not necessarily produce the best feature subset. In three out
of the five tasks, a manually chosen feature subset of a different dimension produced a
higher success rate.
In order to confirm that the success rate results of the feature subset selected by
multicollinear analysis are not a product of mere chance, a randomly selected feature
vector was also used to compare results. The length of each random vector was chosen to
be 17, which is the average of the multicollinear feature vectors.
By keeping the
dimensions of these two subsets approximately the same, we can reduce the effects of
dimension size on the results and we can better compare the effectiveness of the two
selection methods. Figures 6.9 through 6.13 show that in all cases the randomly selected
feature vector has a lower success rate. In fact, for Task 5 the randomly selected feature
vector produced so many errors that it resulted in a negative success rate. This reaffirms
the earlier results showing that the feature subset chosen by multicollinear analysis may
not be the most optimal of all possible selections, but it is certainly a well chosen one.
62
It should also be noted that only one manually selected feature vector was chosen
for each vector length, but it is quite possible, in fact very likely, that a different feature
subset will produce even better results. An attempt was made for each chosen feature
vector to try to include the most important and descriptive features, but given the possible
number of combinations of features, one must assume that there are many other
combinations that will give varying results.
An important point that should be made based on these results is that each task
has unique characteristics. The response to all five tasks varied significantly enough that
there is probably no single clear-cut method to chose an optimal feature vector for all
possible tasks.
For optimal performance, is more likely that each task needs to be
analyzed separately and a feature vector should be chosen that clearly highlights the
repeatable manipulations of that task and also differentiates it from other tasks that are
part of the classification system. If possible, tasks should be chosen that are more readily
identifiable. For instance, picking up the irregular plastic part never achieved a success
rate higher than approximately 50%, while picking up a book was successful up to 95%
of the time.
The success of the multicollinear method of feature reduction provides us with a
readily accessible algorithm that can be used to generate a reasonably well formed feature
vector. The success rates using this method for the five tasks studied here generally were
in the range of 60%. This rate is relatively high enough, compared to other success rates,
to show that the algorithm does in fact select unique features and it effectively reduces
the size of the feature space to a reasonable level. Unfortunately we also see that optimal
selection of a feature subset is unlikely to take place using multicollinear analysis.
63
CHAPTER 7
INDUSTRIAL APPLICATIONS
While the work done in this thesis has a distinctive academic nature to it, the
applications of this work could have a solid impact in industry. The Instrument Glove
and the task recognition system can be used in a manufacturing plant to satisfy a number
of different needs. Many automated plants are not focusing on replacing all of their
workers with autonomous robots, but rather integrating skilled workers with robots
capable of assisting the worker. This means there needs to be current technology in place
to coordinate the interaction between the workers and the machines, as well as a way to
monitor and continually improve the process.
7.1 Time Estimation
An important application of the Instrument Glove and a task recognition system
that is not immediately obvious, and has not yet been discussed, is in the area of time
estimation. In a manufacturing or assembly environment, particularly an assembly line or
an assembly cell, the estimate of the amount of time it takes to perform a particular task is
of critical importance. As a part flows through the cell, or down an assembly line, the
amount of time it stops at each station affects the entire process, and ultimately it can
affect the throughput volume of the whole plant. Great effort goes into the inexact
science of time estimation for these procedures.
When a new cell or assembly line is designed, or an old one is evaluated and
revamped, an industrial engineer needs accurate numbers for the amount of time it takes
to accomplish each individual task along the way. Using these time estimates, a smooth
part flow can be designed that will avoid bottlenecks and will utilize the full capacity of
each worker to the greatest extent as possible.
64
Ramifications of these time estimates are large for any manufacturing company.
Both high volume and low volume producers feel the economic pressures to be as
efficient as possible and properly use all of the company's resources. If a work cell or
assembly line is poorly planned, the resources may be under-used so production is below
an optimal level, or bottlenecks may develop if the timing is not well coordinated. If
workers are waiting downstream of a bottleneck, money and resources being wasted, and
if workers are rushing because they are part of a bottleneck, quality may go down and
stress may go up.
Currently, to determine the amount of time it takes to accomplish a task an
observer will often simply use a stop watch. This process is not ideal. First of all, it
involves two or more people, so extra labor must be used. Second, a worker who is under
observation may not always perform naturally. While this method is not perfect, it is
better than another method commonly used, which is simply taking an educated guess
based on experience.
To use the Instrument Glove and task recognition system to assist in this field, a
worker simply wears the glove and performs the task under normal operating conditions.
By identifying the start point of any task that the worker performs, the time interval
between recognition points is the amount of time the task takes to perform. This method
is superior for a few reasons. First, only the worker needs to be present. No industrial
engineer or extra labor hours need to be spent to record the data. Second, more data can
be taken since no extra labor is involved in the data collection process. The data is
recorded any time the worker is working, not simply when an industrial engineer is
available for an hour. This way many trials over many days can be performed to get a
more accurate set of data, from which one can easily calculate any statistical analysis
deemed necessary, such as mean values and standard deviations.
65
This method of using the Instrument Glove and a task recognition system may be
more accurate than a traditional method of time estimation for a few reasons. One is that
if a worker knows an observer is standing over him, there is a tendency to not work at a
natural pace. The worker may subconsciously work faster for the testing time period, at a
rate that he will not maintain over the period of an entire day or over the course of a
week. The Instrument Glove is more likely to obtain accurate numbers over the course of
that whole week.
Additionally, when one simply times a task in a traditional manner, no account is
taken for extenuating circumstances or human nature. If it is determined by standard
observation that a worker takes 33 seconds, on average, to move a part from point A to
point B, then that is the number which is used for all future calculations. But under
normal operating conditions it is very possible that the worker takes 33 seconds on
average, but occasionally needs to take a 60 second break to rest his arms. The data
glove will take this 60 second pause into account, because it continues timing until the
next task actually begins. It is less likely that a traditional method of time estimation will
see these pauses since the worker knows he is under observation, or the observer only
times the task and is not aware of how often the worker needs a quick break. Over the
course of a day these breaks will affect the statistical output of that worker and his
station, so this data is important and should be accounted for to get accurate results.
7.2 Human-Robot Coordination
A more obvious application of the Instrument Glove and a task recognition
system is in the area of human-robot coordination. This topic has been discussed in
passing at several points in this thesis. In a manufacturing environment, where workers
and robots are working together, it is extremely beneficial to have a system in place that
makes the interaction between these two as simple as possible.
66
If a task recognition system can recognize the state of a worker and a robot, or
several robots, are programmed to assist the worker as necessary, the worker's job
becomes much simpler and he will be more efficient in completing his tasks. In addition
to increased efficiency, the simplicity of the system had many other benefits as well.
Since the Instrument Glove provides a more intuitive system to communicate with a
robot, there will be less training necessary the worker. There will be no complicated
input system or computer commands that need to be learned by the worker. He simply
performs a natural task and the system is programmed to respond appropriately. And not
only will high level training be reduced, but there will also be fewer errors due to
improperly used machines.
If a worker is operating a highly complicated automated
machine, it is natural that mistakes will be made, especially for an inexperienced
operator. But if the communication with that automated machine is made simpler and
more intuitive, then these errors will be reduced.
67
CHAPTER 8
CONCLUSIONS
In modern manufacturing plants, human - machine interaction is growing in
importance. In an effort to improve quantities of production, the efficiency of the plant,
and the quality of the product, companies are integrating the benefits of automated
machinery and a skilled workforce. In an effort to better use the abilities of human
workers as they interact with increasingly complex machines, this thesis introduces an
Instrument Glove as a new coordinating tool between humans and machines.
The Instrument Glove is a logical choice as a coordinating tool due to the
simplicity of its use and its ability to directly monitor a worker's hand motions. Almost
all of an assembly worker's tasks are performed using his hands, so by monitoring hand
motions, a control system will be able to correctly identify the state of the worker at all
times. If the state of the worker is known, then automated machinery will be able to
correctly assist the worker at the proper time and at the proper place. In addition, the
Instrument Glove has the advantage of being a highly intuitive tool. A worker will only
perform natural tasks, and these tasks can serve as control signals to a robot. This is far
simpler than learning a complicated control system or having to manually activate a
machine every time it is needed.
To coordinate the work of a human worker and a robot through the use of an
Instrument Glove, it is necessary for a classification system to identify discrete tasks as
the worker performs them. To accomplish this, the classification system introduced in
this thesis is a minimum Mahalanobis distance classifier, which functions by matching
random feature vectors output from the Instrument Glove to prototype feature vectors that
are associated with specific task labels. Each reduced prototype feature vector exists in a
subspace of feature space, and random feature vectors are systematically compared to
each prototype in its own subspace.
Determinations are made within a prototype's
68
subspace whether the random feature vector falls within the Mahalanobis decision region,
or not.
To determine
an optimal subspace for each prototype
feature
vector,
multicollinear analysis is performed on a feature matrix to eliminate all correlated data
sets.
In this manner, only the most unique features are retained for later analysis.
Experimental data shows that a feature vector reduced by multicollinear analysis is a
"well-formed" vector that has a far greater ability to classify random objects than an
unreduced feature vector. But these same experimental results also that there is an
optimal number of features that should be used for the highest success rate during a task
recognition procedure. While multicollinear analysis provides a more optimal feature
subset, it does not guarantee that the most optimal number of features are retained, and
therefore it is not necessarily the most optimal feature subset available.
The infinite variability between all types of manual tasks that can be monitored
using an Instrument Glove makes it very difficult to use a single algorithm to always
retain the most optimal feature subset for task recognition purposes.
Multicollinear
analysis provides a very efficient method that produces consistently good results, but it
does not guarantee the absolute best results in terms of success rate during task
recognition.
The benefit of using multicollinear analysis is that this procedure is
contained in an algorithm, so no human involvement is necessary to perform the feature
reduction.
For superior success rates during task recognition, it is necessary to go
through the laborious task of manually selecting an optimal feature subset and
methodically testing a great number of possibilities.
69
REFERENCES
[Asada and Izumi, 1989]
Haruhiko Asada and Haruo Izumi, "Automatic Program
Generation from Teaching Data For the Hybrid Control of Robots", IEEE
TransactionsOn Robotics and Automation, Vol. 5, No. 2, pp. 166 - 173, April 1989.
[Asada and Slotine, 1986] Haruhiko Asada and Jean-Jacques Slotine, "Robot Analysis
and Control", John Wiley and Sons, New York, 1986.
[Chao, et al., 1989]
Edmund Y. S. Chao, Kai-Nan An, William P. Cooney III, and
Ronald L. Linscheid, "Boimechanics of the Hand - A Basic Research Study", World
Scientific Publishing Co., Singapore, 1989.
[Duda and Hart, 1973] Richard 0. Duda and Peter E. Hart, " Pattern Classification and
Scene Analysis", John Wiley and Sons, New York, 1973.
[Gray, 1995] Henry Gray, "Gray's Anatomy - The Anatomical Basis of Medicine and
Surgery", Peter L. Williams, ed., 38t Edition, Churchill Livingstone, New York,
1995.
[Gunst and Mason, 1980] Richard F. Gunst and Robert L. Mason, "Regression Analysis
and Its Application", Marcel Dekker, Inc., New York, 1980.
[Liu and Asada, 1992]
Sheng Liu and Haruhiko Asada, "Transferring Manipulative
Skills to Robots: Representation and Acquisition of Tool Manipulative Skills Using a
Process Dynamics Model", Journal of Dynamic Systems, Measurement, and Control,
Vol. 114, pp. 220 - 228, June 1992.
70
[Mason and Salisbury, 1985] Mattew T. Mason and J. Kenneth Salisbury, Jr., "Robot
Hands and the Mechanics of Manipulation", The MIT Press, Cambridge, 1985.
[Nadler and Smith, 1993]
Morton Nadler and Eric P. Smith, "Pattern Recognition
Engineering", John Wiley and Sons, New York, 1993.
[Palastanga, et al., 1994] Nigel Palastanga, Derek Field, and Roger Soames, "Anatomy
and Human Movement",
2 nd
Ed., Butterworth-Heinemann, Oxford, 1994.
[Schuind, et al., 1994] F. Schuind, K.N. An, W.P. Cooney III, and M. Garcia-Elias, eds.,
"Advances in the Biomechanics of the Hand and Wrist", NATO ASI Series, Series A:
Life Sciences, Vol. 256, Plenum Press, New York, 1994.
[Yang, 1990] Boo-Ho Yang, "Skill Acquisition for Robotic Grinding: Learning of HighLevel Feedback Laws from Teaching Data", Massachusetts Institute of Technology
Master's Thesis, Cambridge, 1990.
71