GDMS Sr Engineer Mike DeMichele
Design of Hands-Free Interface for
Object Relocation
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
User
Proposed design eliminates need for operator to perform Robotic Aid
physical motion
AGENDA
•
•
•
•
•
•
•
•
2
Motivation
Problem Statement and Stakeholders
Technology Introduction
Gap Analysis
CONOPS
Simulation
Trade-off Analysis
Business Case
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
2
PHYSICALLY DISABLED STATISTICS
• 5,596,000 people in the US, reported
some form of paralysis [1]
• 2,900,000 severely paralyzed persons in
America [1]
• Physical movement required by control
interfaces.
• Limiting factor in the system usability
and user workload.
[1] 2012 study performed by the Christopher and Donna Reeve Foundation
3
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
3
COMPLEXITY OF HUMAN MACHINE INTERACTION
•
•
Process driven by input from human user
Relies on interface to continue data exchange
Challenges of using human machine
interaction in a practical setting
•
•
•
•
[3] Robot FRIEND, Institute of Automation (IAT)
of University of Bremen
4
Systems requiring a huge amount
of user input and focus
Time sensitive systems
Systems requiring user to perform
small, precise movements
Systems requiring frequent
operator turnover
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
4
AGENDA
•
•
•
•
•
•
•
•
5
Motivation
Problem Statement and Stakeholders
Technology Introduction
Gap Analysis
CONOPS
Simulation
Trade-off Analysis
Business Case
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
5
PROBLEM STATEMENT
The 2.9 million severely paralyzed individuals in the US require a technology
enable them to relocate objects without requiring motion. This design has the
potential to assist users in other daily activities that ordinarily rely on physical
input, such as navigating a robot to perform fetch and deliver tasks.
6
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
6
STAKEHOLDER ANALYSIS
Stakeholder
Significance
Control Technology
Companies
-
Severely Paralyzed
Tensions
Lead on current development work for
alternatives
Potential to modify control technologies to
best fit need
-
Competition between companies
Tension with government approvers
-
Benefit from Hands-Free Control System
-
Learning time
May initially distrust system
Insurance Companies
-
Create new policies to include HFCS
-
Cost of insurance
Additional costs to insurance
companies
HHS and FDA
-
Under Medical Device Act, aid would require
approval
-
Tension if design is not accepted
In-home Care
Organizations
-
Leading care for severely paralyzed
Potential to integrate new robotic care device
-
Concerns of being replaced/ losing
market dominance
-
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
7
STAKEHOLDER INTERACTIONS
8
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
8
AGENDA
•
•
•
•
•
•
•
•
9
Motivation
Problem Statement and Stakeholders
Technology Introduction
Gap Analysis
CONOPS
Simulation
Trade-off Analysis
Business Case
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
9
TRADE SPACE
ALTERNATIVES
Methods to Perform Hands-Free
Control
• Eye Tracking Software
• Head Movement
(Gyroscope)
• Voice Control
• Muscle Contraction
Detection
• Brain-Computer Interface
[4]RoboNurse, Computers Making Decisions- Standford
[5] Accompany Care-O-Bot, Fraunhofer Institute
10
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
10
SELECTING ALTERNATIVES
Alternatives to Evaluate:
- EEG
- Eye Tracking
- Voice Commands
• EEG will be evaluated as the BCI Alternative as it is the most
researched and cost effective BCI
• Muscle Contraction Detection and Gyroscope will not be evaluated as
these alternatives require physical input or movement
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
11
VOICE COMMAND SUMMARY
• Unlimited Number of Commands
• Ready to Use Technology
• Study by School of Information Technology in China set up
a wheelchair to be driven via voice command
• 92% average success rate
• 2.2s average delay
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
12
ELECTROENCEPHALOGRAPHY (EEG) SUMMARY
• Non-invasive BCI for detecting brain activity
• Uses electrodes to measure ERP pulses along the scalp
• Event related potentials (ERP) - Brain’s electrical response to a sensory, cognitive, or
motor event
• Key to utilizing EEG as a control method is to detect desired ERP from other ERPs (noise)
and map to a command
• Non-invasive BCI Interface for Device Operation
• 96-channel EEG system
• Cursor guidance commands
• User “mastery” within 5 sessions
• Patient initial success rate of 50-70% to 80-100% accuracy
https://drive.google.com/file/d/0BzLzei3_VBoAeFVCSFg0U2ljdDQ/view
*J. del R. Millán is with the IDIAP Research Institute, CH-1920 Martigny, Switzerland, and also with the Laboratory of Computational Neuroscience
13
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
13
EYE TRACKING SUMMARY
• Subject to false positives
• Calibration is key to success
• 2014 study published by Max Planck Institute for Informatics
in Germany
• 40ms slower than using a mouse
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
14
AGENDA
•
•
•
•
•
•
•
•
15
Motivation
Problem Statement and Stakeholders
Technology Introduction
Gap Analysis
CONOPS
Simulation
Trade-off Analysis
Business Case
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
15
GAP ANALYSIS
• The necessity of physical movement of the control interface is a highly limiting factor
is the system usability and user workload.
• For the 2,900,000 severely paralyzed persons in America, the necessity of physical
motion to operate a device interface renders the system unusable.
Win-Win - Create a hands-free control system interface which can match the
quality and cost of a traditional interface.
16
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
16
SCOPE
Before a new control interface can be selected for the robotic aid, we must examine
the maneuvers which the robot will need to perform.
This project will perform a simulation to collect and analyze the data crucial to
creating a hands-free control interface.
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
17
AGENDA
•
•
•
•
•
•
•
•
18
Motivation
Problem Statement and Stakeholders
Technology Introduction
Gap Analysis
CONOPS
Simulation
Trade-off Analysis
Business Case
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
18
CONOPS
A paralyzed person cannot perform basic motor tasks without
assistance. A robot, controlled by the person, can cross the room and
retrieve an item (prepackaged meal, medicine, cell phone).
User with HFCS
19
YouBot by Kuka
[9] Youbot, Vertically-Integrated Projects, GT
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
19
Functional Diagram
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
20
USE CASE
21
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
21
MISSION REQUIREMENTS
R.0 The HFCS shall be operable without tactile user input.
R.1 The HFCS shall not harm the user in any way.
R.2 The HFCS shall provide flexibility for use for a variety of functions.
R.3 The HFCS shall operate in real-time.
R.4 *The HFCS shall be capable of differentiating between [X] commands.
R.5 *The HFCS shall be capable of executing [Y] commands.
* where [X] and [Y] to come from simulation
22
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
22
COMMANDS FOR CONOPS
Loop 1: Correct direct
1. Stop motion
2. Pivot to correct direction
3. Resume forward motion
Assumption 1:
Hand contacting object will result in grasp.
(No slipping, etc)
23
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
23
VIRTUAL ROBOTIC SIMULATOR
• V-REP as a virtual robot simulator
• Physics based collisions
• Large number of ready made models
• Free for educational use
• Matlab
• TRS and RTB - binds VREP and
Matlab
• YouBot will pick up an object and
move it to another location
YouBot, a popular multi-function robot
• Limited preprogrammed movement
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
24
YouBot VIRTUAL ROBOT[9]
• Platform
• Omnidirectional wheels
• Zero turn radius pivoting
• Surface for placing multiple objects
• Arm
• 5 joints for 5 DoF movement (Arm is not capable
of full pitch motion)
• Each joint has a limited range of movement
• Gripper
• Open and close
• To simplify the YouBot movement, all the joint movement can be automated according to
the x-y-z position of the gripper in order to reduce user complexity
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
25
YouBot Dimensions
Platform
• Raise distance from ground .7m
• L x W x D: .58m x .38m x 14m
• Payload 20 kg
• Max Velocity .8m/s
Arm (Scale factor x1.4)
• Total reach .917m
• Payload .7kg
Gripper (Scale factor x1.4)
• Range .007m
Work Envelope .7182 m^2
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
26
AGENDA
•
•
•
•
•
•
•
•
27
Motivation
Problem Statement and Stakeholders
Technology Introduction
Gap Analysis
CONOPS
Simulation
Trade-off Analysis
Business Case
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
27
Simulation Objectives
• For object relocation be domestic robot:
•
•
•
•
How many & what turn commands are needed
How many & what travel commands are needed
How many & what arm commands are needed
How many & what gripper commands are needed
• What tasks are infeasible for robot?
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
28
Simulation Requirements
•
•
•
•
•
•
•
•
The simulation shall model the path of a robot fetching objects in a domestic setting.
The simulation shall model the arm movement of a robot fetching objects in a domestic
setting.
The simulation shall model the gripper of a robot fetching objects in a domestic setting.
The simulation shall collect platform travel statistics.
The simulation shall collect platform turn statistics.
The simulation shall collect arm movement statistics.
The simulation shall collect gripper statistics.
The simulation shall define range of robot.
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
29
SIMULATION OVERVIEW
Fetch item - an item the virtual robot is instructed to pick up
Fetch point - the location of the fetch item, random variable
Task - a randomly generated trip from a start point, to the fetch point, picking up the fetch item and carrying it to an assigned finish point
Start location - initial (x,y) coordinates of the center of the robot’s platform, random variable
Finish location - final location of the center of the robot’s platform, random variable
Start Points (Green), End Point (Red)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
Fetch Points (Blue)
30
SIMULATION STATISTICS
Statistics to Record per Task
1. Start location
2. Fetch item location
3. End location
4. Total time on task
5. Time spent on fetch trip
6. Time spent on return trip
7. Time spent picking up object (frame
between fetch trip and return trip)
8. Total distance traveled
9. # of platform rotations
10. degrees rotated for each rotation
11. # lateral arm movements
12. Arm displacement per movement
(cm)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
31
2D- Simulation Inputs
Start and fetch location
coordinates
Room Map - Red marks obstacles
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
32
2D- Simulation Modules
Name
Description
Output
GeneratePath.m
Select start and fetch coordinates
then uses D* navigation algorithm
from Robotics Toolbox
Array of coordinates stepping through
shortest path to fetch location.
StatisticsScript.m
Locates turns along shortest path.
Calculates angle of turn and
distance traveled on straight-aways
Matrix
Column 1,2 - (x,y)
Column 3 - angle of turn @ pivot point
Column 4 - distance traveled in previous
straight away
GraphicalData
Displays frequency of maneuvers
Bar graphs for frequency of travel distance
and angle
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
33
2D- Simulation Output for Single Run
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
34
2D- Simulation Output for 50 runs
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
35
3D Simulation
Depending on the position of the YouBot base, the arm joint positions are different even for moving
an object on the same height.
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
36
Alternative to Manual Control of Arm/Gripper
• Vision sensor to track objects to be picked up to simplify the
pick up action
• Option for user would be displayed an array of detected distinct
objects and be prompted to choose the distinct object to be
picked up (via voice command)
• Used as an aid to guide the arm movement
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
37
Vision Sensor for HFCS Display
VREP Vision sensor has an adjustable resolution, in the ranges that an object is detected, it will
return a depth value, color and intensity values for every point. From the angle of the vision
sensor, the relative coordinates can be calculated.
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
38
Simulation Results
-6 platform commands - turn left 18 degrees, turn left 2
degrees, turn right 18 degrees, turn right 2 degrees, start, stop
- Vision Sensor and active select algorithm
- Automated collision avoidance
- Return to user function
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
39
Simulation Impact
Completed Mission Requirements
R.4 *The HFCS shall be capable of differentiating between 9 commands.
R.5 *The HFCS shall be capable of executing 9 commands.
Additional Requirement
1. Implement a vision sensor instead of manual control once the object is in “sight”
of the robot.
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
40
AGENDA
•
•
•
•
•
•
•
•
41
Motivation
Problem Statement and Stakeholders
Technology Introduction
Gap Analysis
CONOPS
Simulation
Trade-off Analysis
Business Case
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
41
Decision Criteria
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
42
Weighting the Attributes
Category
Weight
Goal
Capabilities
TRL
0.5
Measures
Measure Weight
Number of
Commands
0.7
0.35
Time Sensitivity
0.3
0.15
1
0.1
User Population
0.75
0.15
New User Time
0.25
0.05
Accuracy
0.4
0.08
Maintainability
0.2
0.04
Reliability
0.4
0.08
0.1
TRL
Usability
Performance
Total Weight
0.2
0.2
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
43
Utility Calculation
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
44
SENSITIVITY ANALYSIS
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
45
Cost vs Utility for 1 Year
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
46
AGENDA
•
•
•
•
•
•
•
•
47
Motivation
Problem Statement and Stakeholders
Technology Introduction
Gap Analysis
CONOPS
Simulation
Trade-off Analysis
Business Case
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
47
VALUE
●
Eliminates need for 24/7 in-home caregiver
●
Fill in-home caregiver shortage [4]
●
Returns level of independence to physically disabled persons
● Unit is defined as a Robotic Aid + Hands-Free Control System
● Sell units to Home Health Care Organizations
○ Open channel for improvement in future releases
○ Allow professionals to monitor ratio of robotic care vs human interaction
● Potential to form partnership
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
○ Leverage knowledge
of in-home care agencies to further design
48
COST
● TOTAL 5 years $699k
● Nonrecurring $195K
● Recurring $504k
REVENUE
● Break even at 300 sales
49
Sell 100 per year
Sell 200 for year 1 and 2, sell 100 after that
QUESTIONS?
Hand-held controller
User
Robotic Aid
Proposed design eliminates need for hand-held controller
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
50
BACKUP SLIDES
51
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
51
References
[1] "Prevalence of Paralysis." Christopher & Dana Reeve Foundation. The Reeve Foundation Paralysis Resource Center, n.d. Web. 19
Nov. 2015.
[2] "Human Factors and Medical Devices." U.S. Food and Drug Administration. U.S. Food and Drug Administration, n.d. Web. 19 Nov.
2015.
[3] "Care-Providing Robot FRIEND." Institute of Automation. University of Bremen, n.d. Web. 19 Nov. 2015.
[4] "Computers and Robots: Decision-Makers in an Automated World." Computers and Robots DecisionMakers in an Automated World.
University of Stanford, n.d. Web. 19 Nov. 2015.
[5] "Care-O-bot 3." Fraunhofer Institute for Manufacturing Engineering and Automation. Fraunhofer Institute, n.d. Web. 19 Nov. 2015.
[6] "Brain Cells Chat, Even Without a Synapse." Science Magazine. AAAS, n.d. Web. 19 Nov. 2015.
[7]"Parts of Central Nervous System." , Control and Coordination, Science Help. Tutorvista, n.d. Web. 19 Nov. 2015.
[8] "A Closer Look at EEG." Epilepsy Society. Epilepsy Society, n.d. Web. 19 Nov. 2015.
[9] "YouBot." YouBot Store. Kuka, n.d. Web. 19 Nov. 2015.
[10] "Emotiv EPOC / EPOC+." Emotiv Epoc. Emotiv, n.d. Web. 19 Nov. 2015.
[11] "Quantitative EEG and Event-Related Potentials." Neuronetrix. COGNISION , n.d. Web. 19 Nov. 2015.
[12] T. Pierce, T. Watson, J. King, S. Kelly and K. Pribram, 'Age Differences in Factor Analysis of EEG', Brain Topography, vol. 16, no.
1, pp. 19-27, 2003.
52
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
52
MARKET VALUE
According to a 2014 New York
Times article [4],
●
●
●
1.3 million new paid caregivers will
be needed to meet demand over
the next decade.
By 2020 the direct care workforce
(5 million people) will become the
largest occupation in the United
States, surpassing the number of
retail salespeople.
About 75 percent of services
provided by home care agencies
are paid by Medicaid and
Medicare.
53
Alternative Scores
EEG CONTEXT
EEG TERMINOLOGY
Signals - The electrical activity that travels through a user’s
brain
Synapses - Spaces between neurons that conduct the electrical
activity in the brain
Noise - General term for unwanted and, in general, unknown
modifications that a signal may suffer during capture,
storage, transmission, processing, or conversion
[6] AAAS, Brain and Behavior
Event-Related Potential - electrophysiological response to a
stimulus
User command - The instruction a user gives to perform an action (user signal to EEG)
Input Command - A collection of steps the end-device will perform in order to execute a
single user command (digital signals to end-device)
56
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
56
BRAIN-COMPUTER INTERFACE (BCI)
Direct communication pathway between the brain and a computing device
BCI Type
Signal
Quality
Sensor Location
Cost
Commitment
Other
1. Invasive
Best
In Brain Matter
High
Risk of scar tissue
2. PartiallyInvasive
Medium
Under skull
High
3. Non-invasive
Poor
On Scalp
Low
EEG is the most studied non-invasive BCI method due to its low cost and ease of use. The data
quality is improving quickly.
57
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
57
Status of EEG as a Control Method
Projects that follow a similar approach using EEG technology:
•
Non-invasive BCI Interface for Device Operation
• 96-channel EEG system
• Cursor guidance commands
• User “mastery” within 5 sessions
• Patient initial success rate of 50-70% to 80-100% accuracy
•
Brain Actuated Control of a Mobile Robot
• Combined with machine-learning techniques
• Six commands combined with “mental states”
• Worse than manual control by a slim factor of 1.5
https://drive.google.com/file/d/0BzLzei3_VBoAeFVCSFg0U2ljdDQ/view
*J. del R. Millán is with the IDIAP Research Institute, CH-1920 Martigny, Switzerland, and also with the Laboratory of Computational Neuroscience
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
58
PHYSIOLOGY OF THE BRAIN
Human Brain
- Comprised of
100,000,000,000 neurons
- Neurons connect via
synapses
- When thought occurs,
neurons generate spikes
of electrical activity
[7] Canadian Institutes of Health Research, Institutes of Neurological, Mental Health and Addiction
59
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
59
ELECTROENCEPHALOGRAHY (EEG)
• Offers a non-invasive method to detect electrical activity by using electrodes to
measure the electrical pulses along the scalp to ultimately graph these impulses
• Predominantly used in medical field for sleeping/brain abnormalities and
detection, stress evaluation, and prosthetics
• Event related potentials (ERP)
• Correspond to a brain response related to a sensory, cognitive or motor event
• The key to using EEG as a control method is to detect the desired ERP from
other ERPs (noise) and map it to a command
• EEG is not widely used as a control method due to poor signal quality from
being outside of the skull. Technological advances are improving quality.
60
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
60
How the EEG Hands-Free System Works
1) When a user thinks, electrical pulses fire within synapses in lobes of the brain specific to
that “thought” - user command
2) The EEG headset reads these signals as voltage and generates a graph indicating the
intensity and pattern of the electrical signals (The same stimuli or thought should produce
roughly the same EEG patterns when repeated)
3) These graphs will then be analyzed and chosen patterns will be stored in the database with
it’s corresponding thought or stimulus
4) These selected patterns will then be mapped to varying device actions through a system
that
this project will create - input command
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
61
EEG receives
signals and captures
them in the form of
graphs
14
Sensors
Raw signals are
stored in database and
inputted into the
system
EEG raw data stream
[8] A Closer Look at EEG, Epilepsy Society
System filters out
noise and converts
signals to digital
format
System sends digital
signals to library to be
matched to a command
System outputs
corresponding
command to
simulator
User concentrates
on a trained user
command
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
62
PATTERN LIBRARY BLOCK DIAGRAM
63
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
63
OPERATIONAL BLOCK DIAGRAM
64
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
64
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
65
EEG SIGNAL PROCESSING
• The EEG signal will be composed of 14 continuous and numerical data streams, one
from each of the EEG.
• General steps to preprocess the signal:
1. High pass filter to remove low frequency noise
2. Use a window function to group a variable length interval of data such as the Hann or
Hamming window function and eliminate edge artifacts
3. Fast Fourier Transform
4. Convert the imaginary part of any complex numbers into amplitude values
• Use a machine learning approach such as neural networks to build a learning model,
then continuously test in real time to measure accuracy
66
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
66
OPTIMIZING EEG Event-Related Potentials
Goal: Determine largest limiting factor in EEG Control System reliability
• Use simulation to examining the effects of different variables on
detection rate and reliability of ERPs such as:
• Optimal training procedures or increasing training time
• Use of different machine learning approaches to improve detection
• Tentative input commands to be detected:
• 6 directional commands for positive and negative displacement along each
of the x, y, z axes
• 1 toggle command for switching platform control vs arm control
• May need to simplify the number of different commands to reduce user
complexity
67
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
67
SIMULATION REQUIREMENTS
S.0.1 Virtual Robot shall accept input from HFCS.
S.0.1.1 Virtual Robot model shall contain a robot with 2D movement capability
S.0.1.2 Virtual Robot model shall contain a robot capable of grasping and
relocating
S.0.1.3 Virtual Robot model shall contain an object to be relocated
S.0.2 Simulation shall accept EEG ERPs from an input device
S.0.2.1 Simulation shall respond to a detected ERP in time < 2 seconds
S.0.2.2 Simulation shall map ERPs to input commands and interface with VREP
S.0.2.3 Simulation should detect and respond to 2+ simultaneous ERP
S.0.3 Virtual robot shall employ automated collision avoidance.
68
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
68
EPOC Cognitive Suite
1)Create user account
2)Train neurological signal to a basic motion
3)Repeat thought to perform signal
4)Map signal to another vehicle motion
69
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad
Sungkar
Source:
Emotiv
EPOC
69
EEG Headset Alternatives
Standard EPOC
• 14 EEG channels and two references for accurate spatial resolution
• High performance wireless device
• iOS and Android compatible
EPOC+
• 9 axis-inertial motion sensors, Bluetooth capabilities, additional applications are
enabled
Raw Data Add-on
• Includes EEG firmware that allows the raw EEG data stream and marker
events in TestBench software
70
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
70
Brainwaves
Neural oscillations are rhythmic or repetitive neural activities in the central nervous system. Neural tissue generate oscillations through
neuron interactions.
A large number of neurons activating for a particular neural computation is called a neural ensemble.
Neural oscillations can be categorized into 5 main frequencies
•
Beta (14-40Hz) - Waking consciousness
•
Alpha (7.5-14Hz) - Conscious relaxation
•
Theta (4-7.5Hz) - Sleeping and light meditation
•
Delta (0.5-4Hz) – Deep sleep
•
Gamma (above 40Hz) – Insight
•
Only recently discovered
71
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
71
http://neuronetrix.com/technology-i-36.html
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
72
72
Age
• A study conducted by Thomas W. Pierce et. al suggested that
age has an effect on EEG
• Results showed that electrode groupings were higher in older
adults than younger adults
• Older adults had more electrode locations that did not load
than the younger adults
73
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
73
[1]T. Pierce, T. Watson, J. King, S. Kelly and K. Pribram, 'Age Differences in Factor
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Analysis of EEG', Brain Topography, vol. 16, no. 1, pp. 19-27, 2003.
74
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
74
Gender
• A study by Corsi-Cabrera et. al suggested differences in the
brain waves between male and female
• Men showed higher beta power, women showed higher alpha
power
• The alpha waves of men decreased during analytic,spatial,
and mixed processing, while women decreased significantly
only in analytic and mixed processing
75
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
75
CURRENT TECHNOLOGIES UTILIZED BY PHYSICALLY
DISABLED PERSONS
• Prosthesis
• Wheelchair
• Voice Recognition
76
[13] Johns Hopkins- Modular Prosthetic Limb
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
76