BMI Principles
Jose C. Principe
University of Florida
Adapted from Hayrettin Gürkök, U. of Twente, NL
Literature
Difficulties in Invasive BMIs
• BCIs offer an easy entry to research
– Non invasiveness straight forward data collection
– Closer to cognition
– Conventional signal processing
• BMIs research infrastructure is much harder
– Work with animals (ethics)
– Difficult instrumentation
– Unclear signal processing
Choice of Scale for Neuroprosthetics
Bandwidth
(approximate)
Localization
Scalp
Electrodes
0 ~ 80 Hz
Cortical Surface
Electrocorticogram
(ECoG)
0 ~ 500Hz
Micro
Electrodes
0 ~ 500Hz
Volume Conduction
3-5 cm
Cortical Surface
0.5-1 cm
Local Fields
1mm
500 ~ 7kHz
Single Neuron
200 mm
Electrode Arrays
two polyimide cables
Utah array
footing
Brain Gate
50
40
30
Michigan probes
Microvolts
20
10
0
-10
-20
-30
-40
0
0.01
0.02
Time (s)
0.03
0.04
J. C. Sanchez, N. Alba, T. Nishida, C. Batich, and P. R. Carney,
"Structural modifications in chronic microwire electrodes for cortical
neuroprosthetics: a case study," IEEE Transactions on Neural
Systems and Rehabilitation Engineering, 2006
Technical Issues with BMIs
• An implantable BMI requires beyond of state
of the art technology:
– Ultra low power
– Ultra miniaturized
– Huge data bandwidth/power form factor
– Packaging
FWIRE: Florida Wireless Implantable
Recording Electrodes
Electrode
attachment
sites
50µm pitch
Electrodes
Modular
Electrodes
IC
Patterned
Substrate
Electrode
Array
Flexible
substrate
IF-IC
12mm
Thru vias to
RX/Power Coil
18 mm
+ Thru vias to
Battery
RFIC
Flip-chip
connection
Battery
TX antenna
28mm
Coil
12.5 mm
Supporting
screws
Supporting
Substrate
3.5 mm
15mm
Coil winding
Coin Battery
(10 x 2.5 mm)
Specifications:
16 flexible microelectrodes (40 dB, 20 KHz)
Wireless (500 Kpulse/sec)
2mW of power (72-96 hours between charges)
RatPack
Low-Power, Wireless, Portable BMIs
•
Requirements
– Total Weight: < 100g
– Battery Powered: Run for 4 hours
•
Implantable
– Biocompatible
– Heat flux: < 50 mW/cm2
– Power dissipation should not
exceed a few hundred milliwatts
•
Backpack
– Small form factor
– Speed vs. Low Power
UF PICO System
PICO system = DSP + Wireless
Generation 3
General Architecture
BCI (BMI) bypasses the brain’s normal pathways of peripheral nerves (and muscles)
J.R. Wolpaw et al. 2002
BMIs: How to put it together?
• NeoCortical Brain Areas Related to
Movement
Posterior Parietal (PP) –
Visual to motor
transformation
Premotor (PM) and Dorsal
Premotor (PMD) Planning and guidance
(visual inputs)
Primary Motor (M1) –
Initiates muscle contraction
Motor Tasks Performed
Task 1
Data
• 2 Owl monkeys – Belle,
Carmen
• 2 Rhesus monkeys –
Aurora, Ivy
• 54-192 sorted cells
• Cortices sampled: PP,
M1, PMd, S1, SMA
40
30
Task 2
20
10
0
-10
-20
-30
-40
-40
-30
-20
-10
0
10
20
30
40
• Neuronal rate (100 Hz)
and behavior is time
synchronized and
downsampled to 10Hz
100 msec Binned Counts Raster of 105 neurons (spike sorted)
Firing Rates
10
20
Neuron Number
30
40
50
60
70
80
90
100
200
400
600
800
1000
Time
1200
1400
1600
1800
2000
Ensemble Correlations – Local in Time – are Averaged with Global Models
Computational Models of Neural Intent
•
Three different levels of neurophysiology realism
– Black Box models – function relation between input desired
response (no realism!)
– Generative Models –state space models
using
neuroscience
elements (minimal realism).
– White models – significant realism (wish list!)
Optimal Linear Model
•
The Wiener (regression) solution
w0
w  ( R  I ) 1 p
•
Normalized LMS with weight decay is
a simple starting point.
w(n  1)  w(n) 
•
•
•

x ( n)  
2
w9
e( n ) x ( n )
Four multiplies, one divide and two
adds per weight update
Ten tap embedding with 105 neurons
For 1-D topology contains 1,050
parameters (3,150)
Z-1 delay of 1 sample
S adder
wi(n) parameter i at time n
3-D, 2-D Trajectory Modeling and Robot Control
•
•
Collaboration with Miguel Nicolelis,
Duke University
Sponsored by DARPA
Time-Delay Neural Network (TDNN)
Principe, UF
• The first layer is a bank of
linear filters followed by a
nonlinearity.
• The number of delays to
span I second
• y(n)= Σ wf(Σwx(n))
• Trained with
backpropagation
• Topology contains a ten tap
embedding and five hidden
PEs– 5,255 weights (1-D)
Multiple Switching Local Models
• Multiple adaptive filters that compete to win the modeling of
a signal segment.
• Structure is trained all together with normalized LMS/weight
decay
• Needs to be adapted for input-output modeling.
• We selected 10 FIR experts of order 10 (105 input channels)
d(n)
Recurrent Multilayer Perceptron (RMLP) – Nonlinear
“Black Box”
• Spatially recurrent dynamical
systems
• Memory is created by feeding
back the states of the hidden
PEs.
• Feedback allows for continuous
representations on multiple
timescales.
• If unfolded into a TDNN it can
be shown to be a universal
mapper in Rn
• Trained with backpropagation
through time
y1 (t )  f ( W1x(t )  W f y1 (t  1)  b1 )
y 2 (t )  W2y1 (t )  b2
Generative Models for BMIs
• Use partial information about the physiological system,
normally in the form of states.
• They can be either applied to binned data or to spike
trains directly.
• Here we will only cover the spike train implementations.
Difficulty of spike train Analysis:
Spike trains are point processes, i.e. all the information is
contained in the timing of events, not in the amplitude of
the signals!
Particle Filters for Point Processes
Instantaneous tuning model
kinemati
cs
Linear
filter
t
nonlinearity
f
spikes
Poisson
model
i
i  f (k  X )
Neural Tuning
function
Kinematic
State
t  f (k  xt lag )
t
t
spiket  Poisson(t )
spike trains
neuron 80 nonlinear estimation
N t( j )
X t i  Ft X t 1i  vt 1i
0.25
optimum delay
NonGaussian
[-300, 500] ms
0.2
[-250, 450] ms
[-200, 400] ms
Prediction
0.15
N
p( x0:t | N1(:tj ) ) 
position
40
desired
KalmanPP
Monte Carlo
k PP
30

i 1
N
w  w p (N
i
t
p (x | N 1:k )   Wki  k (x k  x )
20
i
k
i 1
10
y
0
-10
-20
-30
-40
-50
-50
-40
-30
-20
-10
0
x
10
20
30
40
[-50, 250] ms
0.1
i
t 1
[0,200] ms
0.05
Updating
wti k ( x0:t  x0i :t )
[-100, 300] ms
(KX)
P(state|observation)
[-150, 350] ms
( j)
t
| )
i
t
0
-0.1
-0.05
0
0.05
0.1
KX
0.15
0.2
0.25
Generative Data Modeling
…..
…..
…..
…..
Hidden
Processes
(Brain areas)
…..
…..
…..
…..
…..
…..
…..
…..
Time
Observable
Processes
(probed neurons)
BMI lessons learned
BMIs are beyond the Proof of Concept
stage, but….
Present systems are signal translators
and will not be the blue print for
clinical applications
Current decoding methods use
kinematic training signals - not
available in the paralyzed
I/O models cannot contend with new
environments without retraining
BMIs should not be simply a passive
decoder – incorporate cognitive
abilities of the user
BMI lessons learned
BMIs are beyond the Proof of
Concept stage, but….
Present systems are signal
translators and will not be the
blue print for clinical applications
Current decoding methods use
kinematic training signals - not
available in the paralyzed
I/O models cannot contend with
new environments without
retraining
BMIs should not be simply a
passive decoder – incorporate
cognitive abilities of the user
A Paradigm Shift for BMIs!
DSP algorithm
Neural Signal
Processing
Desired response
•
During training the user actions create a desired response to the DSP
algorithm.
•
During testing the DSP algorithm creates an approximation to the desired
response.
A Paradigm Shift for BMIs!
Neural Signal
Processing
Control Algorithm
Learning Algorithm
X
•
The control algorithm learns through reinforcement to achieve common
goals in the environment.
•
Shared control with user to enhance learning in multiple scenarios and
acquire the net benefits of behavioral, computational, and physiological
strategies
Construction of a New Framework
How to capitalize on the perception-action cycle?
•
•
EXTERNAL
WORLD
Causality line
DOES ACTION
MEET
FUTURE
REALITY?
SENSORY
STIMULUS
LIMBIC
SYSTEM
PAST
Body line
ORGANIZED PAST
EXPERIENCE
FUTURE
•
•
–
–
–
–
–
PREDICTIVE
MODELING
INTERNAL
REPRESENTATION
The brain is embodied and the body is
embedded
Need to quantify Brain State at different
time resolutions
Intelligent behavior arises from the
actions of an individual seeking to
maximize received reward in a complex
and changing world.
The BMI must engage and dialogue
with the user:
•
Exploits better engineering knowledge
Utilizes cognitive states
Dissects behavior top-down
Exploits rewards
Learns with use
Propose Reinforcement Learning to train the
BMI.
Reward Learning Involves a Dialogue
•
Agent: I'll take action 1.
•
repeat
actions
•
•
AGENT
rewards
•
Relation between the agent and its
environment.
Environment: You are in state 14. You
have 2 possible actions.
Agent: I'll take action 2.
Environment: You received a
reinforcement of 17.8 units. You are
now in state 13. You have 2 possible
actions.
states
•
ENVIRONMENT
Goal
Start
CABMI involves TWO intelligent agents in a
cooperative dialogue!!!
User’s
neuromodulation
sets the value
function for the
CA
rewards
actions
states
COMPUTER AGENT
environment
ROBOT
Both the CA and the
user have the same
reward in 3D space
RAT’S BRAIN
Features of co-adaptive BMI
• Enables intelligent system design in BMIs
• Both systems adapt in close loop in a very tight
coupling between brain activity and computer
agent ( CA states are specified by brain activity).
• User must incorporate the CA in its world (can a rat
learn this?)
• CA must decode brain activity for its value
function (can it model the signature of behavior?).
• In fact CABMI is a “symbiotic” biologicalcomputer hybrid system.
31
Experiment workspace [top view]
The user learns first to associate
levers with water reward in a training phase.
In brain control, it progressively associates
the blue guide LED of the robotic arm with
the target lever LEDs.
Only when the robot presses the target lever
it will get reward.
Experiment workspace [top view]
Experimental CABMI Paradigm
Grid-space
Incorrect
Target
Correct
Target
Map workspace
to grid
Robot Arm
3
2
1
3
Rat
2
0
Starting
Position
-3
Match LEDs
-2
-1
0
1
2
1
0
3
27 discrete actions

26 movements
 1 stationary
Match LEDs
Rat’s Perspective
Left Lever
Water Reward
Right Lever
Experimental CABMI Paradigm
Left Target
Right Target
0.3
0.3
0.25
z
0.25
z
0.2
0.2
0.15
0.35
0.3
0.25
0.2
0.15
y
0.15
-0.05
x
-0.1
0
0
0.05
0.1
0.35
0.3
0.25
0.2
0.15
y
x
Right Target
Left Target
0.3
0.3
0.25
0.25
z
z
• CA rewards are defined in 3D
at the target lever positions.
• RL is used to train the CA in
brain control (tabula rasa, i.e.
no a priori information).
• During brain control, shaping
of the reward field increases
the level of difficulty across
multiple sections with an
adjustable threshold target.
0.2
0.2
0.15
0.15
0.35
0.3
0.25
0.2
0.15
y
0
-0.05
-0.1
x
0
0.05
0.1
0.35
0.3
0.25
0.2
0.15
y
x
35
Neuromodulation defines the States
Sampling rate 24.4 kHz
Hall, Brain Research (1974)
Bilateral Premotor/motor
Areas
32 channels
Spike sorted data
Performance metrics
Performance metrics:
1. Percentage of trials earning reward
2. Average control time required to reach a target
4 sessions were simulated using random action
selection to estimate chance performance
for the CABMI in increasing difficulty tasks.
Performance in 4 tasks of increasing difficulty
% trials earning reward
time to achieve reward
Closed-Loop RLBMI
Functional
levers
Robot workspace
in rat visual field
of view.
BLUE – Robot
GREEN - Lever
Non-functional
levers
Top-view of the
rat behavioral
cage.
Event Related Desynchronization (ERD) and
synchronization (ERS)
• It is well established that preparation, execution, and also
imagination of movement produce an event-related
desynchronization (ERD) over the sensorimotor areas, with
maxima in the alpha band (mu rhythm, 10 Hz) and beta band
(20 Hz).
• The mu ERD is most prominent over the contralateral
sensorimotor areas during motor preparation and extends
bilaterally with movement initiation
• ERD during hand motor imagery is very similar to the premovement ERD, i.e., it is locally restricted to the contralateral
sensorimotor areas
Event Related Desynchronization (ERD) and
synchronization (ERS)
• During movement preparation and execution, an increase of
synchronization (ERS) in the 10-Hz band normally appears
over areas not engaged in the task (idling)
• ERS can also be observed after the movement, over the same
areas that had displayed ERD earlier
Beta rebound following movement
and somatosensory stimulation
• The general finding is that beta oscillations are
desynchronized during preparation, execution, and
imagination of a motor act
• After movement offset, the beta band activity recovers very
fast (<1 s) and short-lasting beta bursts appear.
• The occurrence of a beta rebound related to mental motor
imagery implies that this activity does not necessarily
depend on motor cortex output.
• A number of experiments have also shown beta oscillations
to be sensitive to somatosensory stimulation
ERS (Blue) and ERD (Red)
ERD
10.9 Hz +/- 0.9
ERS
12.0 Hz +/- 1.0
Pfurtscheller
ERS (Blue) and ERD (Red)
Pfurtscheller
Beta ERS
Pfurtscheller
Alpha and Beta ERS
Pfurtscheller
Signal Processing for ERD/ERS
• Bandpass filtering between 9-13 Hz will emphasize
this component.
• Estimate the power
• Place a statistical threshold for detection.
• Alternatively use PSD and threshold the appropriate
frequency band.
Paradigm 1
(http://www.dcs.gla.ac.uk/~rod/Videos.html)
Paradigm 2
Event Related Potentials
• ERPs are a signature of cognition. They signal a
massive communication amongst brain areas
(kind of the brain’s impulse response to an
internal stimulus).
• This is very good, but the problem is that it is normally
much smaller than the ongoing EEG activity (i.e. the SNR
is negative).
Event Related Potentials
• The ERP shape is well known and pretty stable across individuals,
and has a known distribution across the channels.
• The P300 is the most used for BMIs because it is task relevant
N100-P200 complex is pre-attentive response
appearing over sensory areas
P300 signals a rare tasks relevant event (Cz)
N400 signals an unexpected event (Cz)
Event Related Potentials
• In order to deal with the negative SNR, we use
averaging of the stimulus.
• If you have a transient that appears in white Gaussian
noise, align the transient and average across trielas you
obtain an increase of SNR by N , where N is the number
of trials.
• This is normally done but has three shortcomings:
– It is not real time
– It assumes that the shape of the ERP is the same
– It assumes that the latency is constant
P300 Event Related Potentials
P300 Event Related Potentials
Negative SNR so need averaging (i.e. repeated presentation of stimuli)
P300 Paradigm
P300 Paradigm
P300 Paradigm 2
The Cortical Mouse
In 1990 the CNEL proposed a new computer interface that would control cursor
movement in the screen using directly brain activity (EEG) and implemented in
a NeXT Computer
Left/Right
YES
• Decision based on single ERPs (N400) in real time
• Neural network classifier implemented in DSP chip
• Overall control (synch, screen, data flow) by the OS
NO
4.5bits/min
Konger, C., Principe, J., ANN classification of ERPs for a new computer interface IEEE IJCNN, 1990
Sina Eatemadi, A new computer interface using event related potentials University of Florida, 1992.
Slow Cortical Potentials
SCP Paradigm
Steady State Evoked Potential (ssEP)
ssVEP Paradigms
ssVEP Paradigms
ssVEP Paradigms
• One of the most reliable effects.
• Need to do FFT of occipital channels and pick
the highest frequency.
• Car race (winner of the first BCI competition)
Taxonomy of BCI paradigms
Taxonomy of BCI paradigms
Taxonomy of BCI paradigms
Taxonomy of BCI paradigms
Mu Rhythm
• When a subject imagines movement or sees movement made by
others a burst of activity in the 8-12Hz range appears over the
sensorimotor areas in the brain
• The subject can synchronize the rhythm and by moving
desynchronize it, hence it ia good signal to be used for motor BMI
tasks.