Toward Brain-Computer Interfacing
Minhye Chang
Contents
5. The Berilin Brain-Computer Interface:
Machine Learning-Based Detection of
User Specific Brain States
7. Brain Interface Design for Asynchronous
Control
Chapter 5
• Berlin Brain-Computer Interface procject
• Premovement Potentials in Executed and
Phantom Movements
• BCI Control-Based on Imagined Movements
• Lines of Further Improvement
Brelin Brain-Computer Interface
Brelin Brain-Computer Interface
• EEG-based BCI system with machine learning techniques
– High quality feedback w/o subject training
– Copes with the huge intersubject variability
• Spatial resolution of the somatotopy
• Discriminability of premovement potentials in voluntary
movements
• Sensorimotor rhythms caused by motor imagery
– 128 channel EEG
– Subjects w/ no or very little experience w/ BCI control
– Information transfer rate above 35bits per minute.
Premovement Potentials
• Bereitschaftspotenzial, RP (readiness potential)
– Activity in the motor cortex leading up to voluntary muscle movement
– To build a classifier
• Letting healthy subjects actually perform the movements
– Movement imagination poses a dual task: motor command
preparation plus vetoing the actual movement
• Predictions about imminent movements
– Exclude a possible confound with feedback from muscle and joint
receptors
– Assistance of action control in time-critical behavioral contexts
Left vs. Right Hand Finger Movements
• Readiness potential
– Pronounced cortical negativation
• Left-hand vs. Right-hand finger tapping experiment
– Electrodes : CCP3 and CCP4
– Predominantly contralateral negativation before movement
Left vs. Right Hand Finger Movements
• Self-paced finger-monements on a computer keyboard
– Tap-rates of 30, 45, 60, and 120 taps per minute
– 128 Ag/AgCl scalp electrodes
– EMG, EOG
• Discriminate premovement potentials as fast as two taps per
second
• Highly subject-specific
Preprocessing
• Extracts the low-frequency content w/ an emphasis on the
late part of the signal
Starting points:
128 samples
Emphasize
the late signal
content
FT coefficients :
Pass-band
Four feature
components
per channel
Classification
• Regularized linear discriminant analysis (RLDA)
– RP features normally distributed with equal covariance matrices.
– The data processing preserve gaussianity
RP-Based Feedback in Asynchronous Mode
• Calibration session
– The data is used to train a classifier.
• A useful continuous feedback in an asynchronous mode
– Classifier must work for a broader interval of time
– The system needs to detect the buildup of movement intentions
• Quite simple strategy : jittering
– Extracts several with some time jitter b/w training samples
– Two samples per key press: at 150 and at 50 ms before key press
– Invariant to time shifts
• EMG activity at about
120ms
RP-Based Feedback in Asynchronous Mode
• Movement intention detector
– Distinguishes b/w motor preparation intervals and “rest” intervals
• In a BCI feedback experiment (-160 to -80ms)
Detection of ‘Phantom Limb Commands’
• ERD(ERS)
• Attenuation (amplification) of pericentral μ and β rhythms in the
corresponding motor areas
• Lack of a time marker signal
– Listened to an electronic metronome
– Deep sound: rest, higher sound: perform either a finger tap or a
phantom movement
• 8 patients with amputations showed significant ″phantomrelated″ ERD/ERS of μ- and/or β -frequencies
– Signed r2 values of the differences in ERD curves
Exploring the Limits
• Gross somatotopic arrangement
– Hand versus foot
• Finely graded representation of individual fingers
• Examine the discriminability of BCI signals from close-by brain
regions
– 128-channel EEGs
– During self-paced movements of various limbs
– Significantly reflect specific activations in sensorimotor cortices
Exploring the Limits
• Averaged premovement potential patterns of one subject in
different self-paced limb moving tasks
• Significantly reflect specific activations in sensorimotor
cortices
BCI Control-Based on Imagined Movements
• In repetitive movements, the discrimination decays already
after about 1s
• Modulations of sensorimotor rhythms evoked by imagined
movements
• 6 subjects who had no or very little experience w/ BCI
feedback
– 118-channel EEGs
– Recorded EOG and EMG
• Calibration measurement (machine training)
– Estimate parameters of a brain-signal to control-signal translation
algorithm
Experimental Setup
• In the training sessions
– Visual stimuli for 3.5s: (L) left hand, (R) right hand, or (F) right foot
– Two types were selected for feedback; binary classifier
• Bias and scaling of the linear classifier
– Different experimental condition of the (exciting) feedback situation
Experimental Setup
• 1st feedback application: position-controlled cursor
– Classifier output translated to the horizontal position of a cursor.
Success
Holding
Activated
for 500ms
Experimental Setup
• 2nd feedback application: rate-controlled cursor
– At each update step a fraction of the classifier output was added to
the actual cursor position.
• 3rd feedback application: basket game
– Operated in a synchronous mode
– Horizontal position was controlled
by the classifier output
Processing and Classification
• Spatial filters
– Optimize the discriminability of brain signals based on ERD/ERS
effects of the motor rhythms
– Common spatial pattern (CSP) analysis
• Feature calculation
– Log of the variance in those surrogate channels
– Calculated every 40 ms from sliding windows of 250 to 1000ms
(subject-specific) for online operation
• Details about the processing methods and the selection of
parameters : Blankertz et al. (2005)
Results
• Information transfer rate (ITR) in bits per minute (bpm)
– Compared to ROC curves, ITR considers different duration of trials and
different number of classes
• Highest ITRs: rate-controlled cursor, asynchronous protocol
Results
• It can be operated at a high decision speed
– Average trial length for 1st, 2nd, and 3rd was 3s, 2.5s, and 2.1~3s resp.
• The fastest subject : average speed of one decision every 1.7s
• Subject who showed the most reliable performance
– only 2% of the total 200trials were misclassified at an average speed
of one decision per 2.1s
– Sentences 135 characters in 30 minutes (4.5 letters per minute) in a
free-spelling mode
Investigating the Dependency
• It is possible to voluntarily modulate motorsensory rhythms
w/o concurrent EMG activity (Vaughan et al., 1998)
• Squared biserial correlation coefficient, γ2
– For the classifier output and for the bandpass filtered and rectified
EMG signals of the feedback sessions
– Occurrence of minimal EMG activity in some trials does not correlate
with the EEG-based classifier
CSSSP
• CSP with Simultaneous Spectral Optimization
–
–
–
–
CSP strongly depends on the choice of the bandpass filter
Broadband filter for general choice
Subject-specific choices
Optimized spatial filters (usual CSP technique) + temporal finite
impulse response(FIR) filter : enhance the discriminability
• Significant superiority of the proposed CSSSP
• The spatial and/or the spectral filter can be used for source
localization of the respective brain rhythms.
Need for Adaptivity
• Nonstationarities in EEG signals
– How much of this nonstatioarity is reflected in the EEG features
– How strongly is the classifier output affected
– How can this be remedied
• The most serious shift occurred b/w the initial calibration
measurement and online operation
• Shifts during online operation were largely compensated for
by the CSP filters or the final classifier
• Simple adaption of classification bias successfully cured the
problem.
Chapter 7
• Introduction
• Asynchronous Control
• EEG-Based Asynchronous Brain-Switches
• Asynchronous Control Design Issues
Introduction
• Neil Squire Society
– The only national not-for-profit organization in Canada
– Neil Squire Brain Interface lab focused on BI system design specifically
for asynchronous control environments.
• Synchronized control environments where the system dictates
the control of the user
• Robust multistate, asynchronous brain-controlled switch in
the most natural manner in the real-world environments
Asynchronous Control
Asynchronous Control
• Output signal levels are changed or commands are issued only
when control is intended
– Intentional control (IC) / no control (NC) state
– Remains neutral or unchanged during the NC state
NC state; stable and unchanged
IC; available for control
• Asynchronous control
– Characteristic of most real-world control applications
– Most people expect from interface technology
System Idling
• The neutral or unchanging system output response during NC
states
• Poor idling is indicated by false switch activations
• How well BI transducers idle
– Rate of false activations or false positive error
• True and false activation rates as performance metrics
With no gas (the NC state), the
engine idles.
Poor at idling
Synchronized BI System
• Synchronized BI system
– System-driven control strategy which cause significant user frustration
and fatigue
• Asynchronous TV controller
– Simply change channels at any time they wish
– The channel selection is stable when users are watching TV (NC state)
• Synchronous TV controller
– Regularly poll the user to ask
– Renders the changing of channels to specific periods
Synchronized BI System
• Two serious drawbacks
– control decision regardless of whether the person is actually intending
control
– User would need to wait for the system polling period to occur
System availability
• Essence of the temporal control paradigms
– Idle support and system availability
• Continuously available control
– Always ready for the user to control
• Periodically available control
–
–
–
–
For Initial trial-based technology development
For restricting the signal processing complexity
Blocks a user’s attempt to control for the periods b/w control periods
“no control is possible at this time”
Control Paradigms
• Unintended action during NC states
– Control paradigms that do not support idling
– “Midas touch problem” by the eye-tracking community
• Four primary control paradigms
– Constantly engaged mode : impractical
– Asynchronous mode : most natural assistive device operation
Asynchronous Brain-Switches
• Specific signal processing algorithms to handle the NC state
– Optimized TP rate and minimized FP rates
– Trade-off b/w TP accuracy and FP rates → specific characteristics of
an application
• Brain-switch based on the outlier processing method(OPM)
–
–
–
–
Extract single-trial voluntary movement-related potentials (VMRPs)
TP rates greater than 90%
FP rates b/w 10 and 30% → limits OPM as an asynchronous switch
Multiposition asynchronous brain-switch
Asynchronous Brain-Switches
• Attempted VMRPs
– Voluntary movement : existing and natural internal control system
– “attempted” : subject with SCI attempt to move their fingers
– By a very different neural mechanism
• The low-frequency asynchronous switch design (LF-ASD)
–
–
–
–
–
Relative power in the 1-4Hz band from ensemble VMRP increased
Focus on low frequency band
Wavelet analysis of the EEG signal over SMA and MI
Lower end of FP activations
TP rates of 30 to 78% during IC states with very low FP rates of 0.5 to
2% during NC
Additional signal processing
• Additional signal processing
–
–
–
–
EEG signal normalization
Switch output debounce
Feature set dimensionality reduction blocks
Increased the TP rate by an average of 33%
• Feature vectors navigate the feature space
– System classification accuracy of more than 97%
• Levine and Huggins, Millán et al. (2004a), Yom-Tov and Inbar
(2003), and Townsend et al. (2004)
Asynchronous Control Design Issue1
• Very low FP rates during the NC state
– High FP error rates cause undue user frustration
– False activation appear uncontrollable to the user
– Subjects would rather experience more trouble performing accurate
hits ( low TP rate)
• Evaluating asynchronous control
– Receiver operating characteristic (ROC) curves
– BI transducers with low FP error rates
Asynchronous BI Performance
• “Tuning” the performance
– ROC curve shows possible operating setups
– by tuning various parameters
• Two-state LF-ASD brain-switch (on/off)
– Scaling the relative magnitudes of NC state feature vectors vs. IC state
ones
– FP rates under 1% during the time b/w FPs (30s or more)
Switch-output jitter reduction
• FPs typically clump together
– Large periods of system idle time with free of FPs
• Switch-output jitter reduction methods
–
–
–
–
Switch debounce block
Improve error rates by reducing the FP jitter in the switch output
Trade-off : transducer availability
Debounce time ↑ → time the transducer is available ↓,
control time ↓
Borisoff et al. (2004)
Asynchronous Control Design Issue2
• Intra-False Positive Rates
– Dependent on the output classification rate of a transducer
– FP rate of 1% with classification output every 1/16th sec
→ FP every 1/16 * 100 = 6.3s
• Average time b/w errors greater than 30s
– Reasonable design goal : FP rate of under 0.25%
• False activation rates : time rates w/ raw percentages
Asynchronous Control Design Issue3
• Temporal characteristics of NC and IC states
– Environmental controllers : periods b/w the IC commands
– Neural control of a robotic device : intercontrol times during periods
of intense usage
• Ubiquitous ON/OFF problem
– Users with amputation have to turn the system on/off by themselves
– Confirms user intent to turn the system to the awake mode
– Eliminate FPs and require a simple commands
Asynchronous Control Design Issue4
• Costs associated with an FP
– Multiple operating levels of the sleep mode
– Full awake mode : sequence through higher modes, higher FP rates
and intentional command sequences
Awake mode
Sleep mode
Low costs of FPs
Easily corrected
High FP rates
Less intricate
• How to structure tasks during different phases
– Customization, training, and testing phases
Asynchronous Control Design Issue4
• Initial customization
– Accurate time-stamping for calibration and classifier
– System-guided and system-paced tasks
– Train and test in very similar control environments
• Apparatus necessary
– Self guided and self-paced tasks → Self-report errors
– Contamination of data
– Accurate assessment of asynchronous system performance
Thank you for your attention