Bio Com Project
Sarath Chandra
Committee:
Dr. Krishna Kavi
Dr. Robert Akl
Dr. Yuan Xiaohui
Objective
 Objective of my thesis work is to develop a sensor system
that allows soldiers to communicate and convey
information through signals and gestures when there is no
direct Line of Sight.
 The signals and gestures can be transmitted wirelessly to
a HUD(heads up display) which is mounted on the
soldiers head.
Outline
 Motivation
 Gesture set
 Standardized hand signals
 Components
 Previous Works
 Analysis & Classification
 Summary
 Future Work
Motivation
 Tactical Squads and SWAT teams involve in high-risk
situations such as Hostage rescues, VIP protection,
counter terrorist situations etc.
 Verbal communication is not feasible in such high-risk
situations because:
 Absence of Line of Sight
 Interference of sounds like gun firing

Teams divide into different groups
Gestures
 Gestures are a form of non-verbal communication in
which visible bodily actions are used to communicate
important messages.
 Gesture classification has many important uses in our day
to day lives like:

Human computer interaction(HCI) applications

Controlling of prosthetic arms and limbs

Gaming purposes like Wii
Gesture Set
 Used the standardized hand signals for close range
engagement operations used by military and swat
teams.
 Gesture set is broadly divided into finger and arm
movements.
Standardized Hand Signals
Sensors
The different core components of our prototype are:
 Surface EMG sensors
 Bend(flex) Sensors
 Accelerometers
Surface EMG sensors
Surface EMG is electrical activity produced by muscle
activity and measured by sensors directly attached to the
skin
Electromyography
 Electromyography(EMG) measures the muscle electrical
activity during muscle contractions as electric potential
between ground and sensor electrodes.
 Electrical activity is typically in the range of 0-90 micro
volts range and contained within 0-200Hz spectrum.
 2 types of EMG:
(a) Surface EMG
(b) Invasive EMG
Bend Sensors
These devices use a piezo-resistive method of detecting the
bend. The sensor output is determined by both the angle
between the ends of the sensor as well as the flex radius.
Accelerometers
Senses dynamic acceleration(or deceleration) and
inclination(tilt) i.e. acceleration due to gravity ) in 3
dimensions simultaneously
Communication
 Wi-micro dig enables us to interface sensors wirelessly to
our computer.
 Designed to obtain data from sensors for measurement
and analysis.
 It translates analog sensor signals with high resolution into
digitally encoded music industry compliant MIDI
messages.
 The wireless communication protocol followed is
Bluetooth.
Wi-micro Dig
 A thumb sized hardware device that encodes analog
voltage signals generated by up to 8 sensors.
Data Format for MIDI Protocol
Infusion system’s I-Cube X Wi-micro Dig MIDI implementation
uses the following data format for all system exclusive
messages:
Byte
Description
240(F0h)
System exclusive status
125(7Dh)
Manufacturer ID
{DEV}
Device ID
{CMD}
Command or Message ID
[BODY]
Main Data
247(F7h)
End of System exclusive
Other current research works
 Most of the current research going on using EMG for
gesture classification is done for various applications like
HCI, gaming, controlling of prosthetic arms and limbs
based on the electrical information stored in the nervous
system etc.
 One of the notable work was done by Microsoft in which
they use a wearable electromyography based controller
that senses the electrical activity produced in muscles
using surface EMG sensors and these signals provide a
muscle computer interface for interaction with one or
more computing devices.
Microsoft Patent
 A wearable electromyography based controller is used
for interacting with one or computing devices.
 The wearable electromyography based controller
directly senses and decodes electrical signals produced
by human muscular activity using surface EMG sensors.
 The controller consists of surface EMG sensor nodes
integrated into a wearable band like an armband or a
wrist watch.
Wearable EMG based controller
How our prototype works
 EMG sensors are placed on the upper and lower forearm.
These sensors assist in the classification of wrist movements.
 Bend sensors are attached to a glove that is worn on the
hand. Sensors are located over the first knuckle of each
finger and determine weather a finger is partially bent
partially, completely or the finger is straight.
 Accelerometer is attached to the glove at the base of the
wrist. By converting the signal into a 3 –element vector and
using a 3D Mapping technique , it can determine the
position of the speed and direction of the arm movements.
ANALYSIS
and
CLASSIFICATION
 The sensor data is ported into the computer and
analyzed with MATLAB. A custom GUI is built to control
the overall system.
Analysis Methods
Classification is accomplished in Mat lab using a variety of
methods:
 Fourier Analysis
 Threshold analysis
 Support vector machines
 Naïve Bayes
 Wavelet Transforms
Phase 1
 Develop a prototype using off the shelf components from
I-cube X that can measure EMG as a result of hand
movement and convert it into an electrical signal that
can be transmitted wirelessly.
Two channel EMG sensor system plot
Phase 1 cont …
Some of the lessons learnt from phase one are:
 Succeeded in developing an interface to receive EMG
signals and analyze using MATLAB
 Crosstalk between muscles impaired localization
 Vey low classification accuracy
 Signal attributes found are: Raw EMG,FFT
Phase 2
 Investigate the current state of the art EMG sensors and
an in depth analysis of different feasible implementations
and solutions to this project.
Raw EMG,FFT, Cepstrum
Phase 2 cont…
Some of the lessons learnt from phase two are:
 Clinical EMG sensors were found to be much more
accurate
 Understanding gained from these sensors as well as
further research allowed us to modify and reconfigure
the EMG sensor system.
 Increased classification accuracy
 Classification techniques were identified like: Logistic
regression, SVM, Neural networks
Phase 3
 Develop a prototype with either new more accurate
sensors or the optimized Bio flex sensors.
 Develop a classification technique to improve
classification.
3 channel EMG system
Phase 3 cont..
Some of the lessons learnt from phase 3 are:
 Using optimized bio flex sensors a new prototype was created
using 3-5 sensors
 Using SVMs and other techniques several revolutions of
increased accuracy
 Limiting the number of features to abstract and percentage
based components FFT component
Phase 4
 Constructed a 5 channel EMG sensor prototype
 Took more samples involving three subjects for 5 gestures.
 Primary and the secondary sensors were selected.
 Mean, variance and standard deviation were calculated
for each gesture of the respective samples.
 Found very high variance and standard deviation
involving samples of all the 3 subjects taken at once.
Phase 4
 Wavelet transforms are used to find the difference between
these gestures.
 Signal features present in the detail and approximation
coefficients of the respective signals for each gesture are similar
with slightly different features for each gesture.
 Flex sensors are attached to a glove that is worn on the hand.
 Located on the first knuckle of every finger and can determine
weather a finger is bent partially, completely or when there is
no bend.
Gesture 1
Gesture 3
Phase 4 cont …
 Took the difference between samples of different gesture
pairs to find notable difference between the gestures.
 Plotting of the difference between samples showed that
each gesture has noticeable amplitude spikes in a very
narrow band width of 1-1.5 Hz range.
 These narrow bandwidths are so congested that it
becomes difficult to get higher accuracies above 75%.
Difference between Gesture 3 & 1
Phase 5
 Wavelet transforms were used to find any more
noticeable differences between the gestures.
 The approximation and detail coefficients of these
gestures are almost similar with slightly different
waveforms.
 Incorporated additional sensor types such as flex sensors
and accelerometers.
Phase 5
 Thresholds are defined to determine the bent of the
finger.
 95% classification accuracy achieved.
Summary and Conclusion
 Electromyography can be used effectively for the
classification of wrist movements with high degree of
accuracy.
 Flex Sensors can be used to classify the finger movements
effectively because they clearly distinguish the fingers
which are bent versus the fingers which are not bent thus
classifying the finger gestures effectively.
Future Work
 Use a fully integrated sensor system including the
accelerometers to classify the complete gesture set with
high degree of accuracy.
 Transmit the messages wirelessly and securely to display it
on a HUD
 Design an encryption scheme for secure transmission to
the HUD.
Questions??