Closing the Sensorimotor Loop: Haptic
Feedback Facilitates Decoding of Arm
Movement Imagery
M. Gomez-Rodriguez1,2
J. Peters1
J. Hill3
B. Schölkopf1
A. Gharabaghi4
M. Grosse-Wentrup1
1MPI
2Stanford University
for Biological Cybernetics
3Brain-Computer Interface Laboratory, Wadsworth Center
3Werner Reichardt Centre for Integrative Neuroscience
Eberhard Karls University Tuebingen
SMC Workshop in Shared-Control for BMI, October 2010
1
BCI + robot-assisted therapy
Brain Computer Interface (BCI) + robot-assisted physical
therapy for neurorehabilitation of:
Hemiparetic syndromes due to brain damage
may outperform traditional therapy
Traditional rehabilitation
Sensorimotor
loop is broken
They do not help
for severe motor
impairment
Our approach to rehabilitation
We close the
sensorimotor
loop
Synchronize
subject’s attempt
and robot arm
2
Stand-alone BCI and robot-assisted therapy
It has been shown
that
Motor imagery
Robot-assisted
physical therapy
are beneficial for rehabilitation
as stand-alone therapies [2,3]
but loop is still broken!
Next logical step is to combine both in an
integrated rehabilitation to close the loop
3
Hebbian plasticity: Why closing the loop?
Closing artificially the sensorimotor loop is
likely to result in increased cortical plasticity
because we induce Hebbian plasticity.
Hebbian plasticity [1]
“A positive feedback-mediated plasticity in which
synapses between presynaptic and postsynaptic neurons
that are coincidently active are strengthened.”
Requirements
Instantaneous feedback: Delays in the order of ms.
High accuracy: On-line decoding of arm movement intention.
High specificity: Focus on motor and sensorimotor cortex.
4
BCI decoding: Effect of closing the loop
Combining BCI and robot-assisted physical therapy opens
many research questions.
Our work builds on analyzing the effect of artificially closing
the sensorimotor loop on BCI-decoding.
Previous studies:
Passive and active movements induce patterns in the brain similar to those
induced by motor imagery [2, 3].
Random haptic feedback has been shown to be beneficial for BCI-decoding [4].
In our work:
We show how haptic feedback (closing the sensorimotor loop) influences
BCI-decoding.
5
Outline
1. Experimental Design:
Human subjects, recording and task & feedback conditions.
2. Methods:
Signal processing, on-line decoding and conditions comparison.
3. Results:
Analysis of the haptic feedback effect on decoding
performance and spatial/frequency features.
4. Conclusions
6
Subjects and recordings
Human subjects:
6 right-handed healthy subjects
between 22 and 32 years old.
Recording:
35 EEG channels
250 Hz sampling rate
Quickamp with built-in CAR
BCI2000 + BCPy2000
Pre-motor, primary motor and
somatosensory cortex are covered
7
Task and haptic feedback conditions
Subject’s task:
Think about moving the right arm forward (extension) or
backward (flexion) in the same way the robot does.
Condition
Training
Test
Condition I
+
+
Condition II
+
Condition III
Condition IV
+
8
Haptic feedback conditions
+
Trial duration
Training
(25s per condition)
Test
(consecutively
after training)
Rest: 3s
MI: 5s
Rest: 3s
MI: min(5s, robot hits border)
X
Robot moves while
motor imagery
Arrow in a screen
moves-stops
according to classifier
while motor imagery
Arrow in a screen +
robot moves-stops
according to classifier
while motor imagery
9
Outline
1. Experimental Design:
Human subjects, recording and task & feedback conditions.
2. Methods:
Signal processing, on-line decoding and conditions comparison.
3. Results:
Analysis of the haptic feedback effect on decoding
performance and spatial/frequency features.
4. Conclusions
10
Signal Processing
Preprocessing
Surface Laplacian Filter
Band-pass filtering (2-115Hz)
Notch filtering (50 Hz)
Features Computation
Power spectral densities over 2Hz frequency bins for
each electrode are used as features.
Welch’s method over overlapping incrementally bigger time
segments each 5-s movement or 3-s resting periods.
Larger segments → Less noise and more reliable estimates.
Shorter segments → Necessary for on-line feedback.
11
On-line Decoding
During the test periods, on-line classification between
movement and resting using spectral features:
Every 300 ms,
• One classifier output.
• Visual on-line feedback and depending on the condition also
haptic feedback is updated.
+
A linear support vector machine (SVM) is generated each run
on-line after the training period and its outputs are mapped to
probabilistic outputs by fitting a sigmoid.
12
Conditions comparison
To discover how haptic feedback influences the BCI
we compare the BCI performance for each condition of
haptic feedback by computing:
1. Two-way analysis of variance (ANOVA) over probabilistic
outputs in each condition.
2. Average accuracy per condition.
3. Area under the receiving operating characteristic (AUC)
per condition
We expect all three to support the same conclusions to
strengthen the empirical evidence.
13
ANOVA and AUC
ANOVA
AUC
1. For each condition, we group
M probabilistic outputs from
all subjects.
1. For each subject and
condition, we have N
probabilistic outputs.
2. Compute ANOVA at
significant level α = 0.05 with
Bonferroni multiplecomparison correction.
2. We sweep over different
thresholds in (0, 1) to classify
mov/rest and compute the
accuracy for each.
3. ANOVA tell us if we can
reject the hypothesis that
the probabilistic outputs
means are equal between
conditions.
3. The area under the curve
threshold versus accuracy is
our AUC.
14
Outline
1. Experimental Design:
Human subjects, recording and task & feedback conditions.
2. Methods:
Signal processing, on-line decoding and conditions comparison.
3. Results:
Analysis of the haptic feedback effect on decoding
performance and spatial/frequency features.
4. Conclusions
15
ANOVA
Conditions
ANOVA confidence intervals
Training
Test
+
+
+
+
Average probabilistic on-line (every 300ms) output
Condition I outperforms the rest, very
clearly condition IV!
16
+
+
+
+
The results are coherent with
ANOVA!
Test
2. Condition I outperforms
Conditions III and IV for all
subjects, and it
outperforms II for all
subjects except two.
Training
1. In group average,
Condition I outperforms
the rest.
Average accuracy
Average accuracy
17
Conditions
1. In group average,
Condition I outperforms
the rest.
AUC
AUC
+
+
+
Test
Training
2. Condition I outperforms
the rest for all subjects.
+
The results are coherent with
ANOVA and average accuracy!
18
Conditions
Spatial and spectral features
Average classifiers weights for each electrode over the
frequency band (2 – 40 Hz)
Condition I and II
(Robot moves during training)
Condition III and IV
(Robot does not move during training)
When the robot moves, we have higher weights in the
motor/somatosensory area
19
Outline
1. Experimental Design:
Human subjects, recording and task & feedback conditions.
2. Methods:
Signal processing, on-line decoding and conditions comparison.
3. Results:
Analysis of the haptic feedback effect on decoding
performance and spatial/frequency features.
4. Conclusions
20
Conclusions
 Artificially closing the sensorimotor feedback loop facilitates
decoding of movement intention in healthy subjects.
 Our results indicate the feasibility of future integrated
rehabilitation therapy that combines robot-assisted physical
therapy with decoding of movement intention by a BCI.

We assume that the results presented here with healthy subjects can be
transferred to stroke patients.
 We speculate that haptic feedback support subjects in
initiating a voluntary modulation of their SMR.
 In a shared-control scenario in BMIs, we may improve
performance by means of haptic feedback.
21