Translation of Sign Language
to Speech
Sensor Glove Development
Name:
Garry Rank
Student Number: 12046339
Supervisor:
Mr Iain Murray
Australian Sign Language
(ASL)
• Brief History
– Developed by whom.
– Various standards and forms of sign
language.
• One handed alphabet.
• Two handed alphabet.
• Full word gestures.
Why do we want translation?
• To help people to communicate.
• It can not replace full understanding of
sign language. There are many elements
of ASL that are simply too subtle!
• Speed, force, facial expressions, body
language can all change meaning of a
sign.
Glove Specifics
• Why is a glove required?
• Image data extraction not yet powerful
enough to handle everything.
• Using a glove is more effective than a
CCD for:
– Speed
– Sensitivity
– Accuracy
Design Requirements
•
•
•
•
•
Fast
Accurate and sensitive.
Robust
Low cost
Low profile – the glove should not impede
signing.
• Be able to detect everything necessary for
translation.
These are all problems that make most virtual reality
gloves unsuitable for the task!
Required Data
• What sensors are required?
• Less is better
• General hand locations important to ASL:
Top of hand
Side and palm of hand
• Amount of bend of base and first knuckle
of each digit.
• Pitch of the wrist.
• Detection of finger splaying.
• Finger’s crossed.
• General area contacts.
Fingers that Bend
Properties of a bending finger
– Vertical compression over the knuckle.
– Horizontal sliding.
Fingers that Bend contd…
• Capacitors lend a natural solution.
– Sliding motion to change surface area.
– Capacitance is changed.
– Detectable by a micro-controller’s analogue to
digital converter (ADC).
Fingers that Bend contd…
•
•
•
•
Apply voltage.
Wait for capacitor charge.
Turn off voltage.
Wait a short time and read voltage level.
Probably not required.
Capacitor Discharge
Higher capacitance
Lower capacitance
T
From the discharge rate of the circuit, the
voltage at time T can be used to determine
amount of capacitance.
Testing
• An m16c microprocessor development board
provided the ADC.
• 8bit resolution mode used for its increased
speed.
• Results showed 10° accuracy possible, but
interference may produce erroneous values.
• 10bit resolution on the ADC offered a
dramatically increased reliability and accuracy.
Contact Sensors
• Primary and Secondary contact locations.
• Primary
–
–
–
–
Used for fast determination of hand shape.
Finger splaying.
Touching the palm.
Fingers crossed.
• Secondary
– Fingers touching elsewhere.
– Chest, head, arms or elsewhere on the body.
• Different types of contact sensors for primary
and secondary locations.
Primary Contact
• Used to supplement finger bend data to
improve speed of sign recognition.
• Amount of finger bend is redundant data if
it is known a finger is touching the palm.
• Minimum set required – more buttons
greatly increases complexity and chance
of equipment failure.
Primary Contact contd…
• Positive terminal on palm and tops of
fingers.
• Negative terminal on finger tips.
• When the fingers touch the palm a circuit
is closed.
• Same reasoning can be used for finger
splaying.
Primary Contact contd…
Primary Contact contd…
• What to use as materials on the glove?
– Flexible to move with the hand.
– Conductive
• Carbon rubber.
– Used on keypads.
– Flexible and conductive.
– Requires outer surface to be kept clean.
– Not as sensitive to contact as metal.
Probably not required.
Secondary Contact
• A simple button used to detect when a
finger is touching something not
detectable by the glove.
• Not as sensitive to soft contact
– Some signs could be missed.
• Used to supplement overall sign
recognition of the system.
Secondary Contact contd…
Putting it all Together
• M16c microcontroller development board linked
to a PC via serial cable.
• Developed under cygwin programming
environment – but should work for any Unix type
OS.
• Bend sensor:
– 1MΩ Resistor.
– Brass tubing.
– Old mouse cord for over-the-finger exoskeleton.
Software Flowchart
Start Here
Set output pin
high (5V)
Transmit value
to PC
Use ADC for
voltage reading
Wait for
capacitor charge
Set output pin
low (0V)
Wait some
time T
Timing
• A real time system – at least 10 full glove
readings per second.
• Including one for the wrist, three for the thumb
and two for each finger, 24 bend sensors may
be required. Conversion timing is important!
• Approximate 6us conversion time.
• With up to 24 bend sensors, this is means a
144us bend sensor clock time.
• Well within acceptable limits!!!
ADC Input Multiplexing
• The problem now is that your standard
micro-controller doesn’t have enough ADC
inputs for all the required bend-sensors!
• Inputs must be multiplexed.
• Diodes can be used to prevent all
capacitors multiplexed from being
charged.
Probably not required.
Multiplexed Input
If only A, only B or only C is
turned on, then only one
capacitor will affect the
value as read from the
voltage probe (ADC input).
Probably not required.
Contact Sensors
• Much of this work has already been done
by Craig Newman in his thesis “Translating
Sign Language to Speech”, 2003.
• Most notably, pull-down resistors are used
to prevent indeterminate values when the
contacts are not touching.
Contact Sensor Circuit
• Much of this work has already been done by Craig Newman in his
thesis “Translating Sign Language to Speech”, 2003.
• Most notably, pull-down resistors are used to prevent indeterminate
values when the contacts are not touching.
+5V
+5V
Secondary
contact
Input
Pin
Primary
contact
Note that the voltage sources are isolated – this is so that it can be
known if the primary and/or secondary contacts are closed.
Two Gloves
• The sensors so far assume only one glove. Two
handed interaction should also be detected.
• More contact sensors seems like a good idea, or
is it?
• Pins on microcontrollers are I/O.
– They can be input or output.
– By controlling this, much more data can be obtained
from the same number of sensors.
– Incurs a time penalty.
Problems
• The glove needs to be specially made.
– Material moves too much with the hand.
– Capacitors sewn onto the glove move with the material. Bending
of the wrist causes the most problems here.
– Wrist bending is calculated and so these errors can be
algorithmically reduced
• Primary and secondary contacts become more sensitive
with small hardpoints.
– Thin plastic can be used as an exoskeleton on which to mount
sensors.
– Straps to keep the glove fixed on the hand.
• Electrical interference can induce currents that produce
errors in results.
Thanks for Listening!
Feel free to ask some questions.