Evaluation of a robot as embodied interface
for Brain Computer Interface systems
Evaluation of a robot as embodied
interface for Brain Computer
Interface systems
L. Tonin, F. Piccione, S. Silvoni, K. Priftis,
Emanuele Menegatti
IRCCS
San Camillo,
Venice, Italy
May 12, 2009
Intelligent Autonomous
Systems Lab
University of Padua
BCI Workshop ICRA 2009, Kobe
Dept. Of General
Psychology
University of Padua
Evaluation of a robot as embodied interface
for Brain Computer Interface systems
What is BCI?
•According to Birbaumer et all.(1999):
“Brain Computer interfaces are
systems that allow to translate in realtime the electrical activity of the brain,
in commands to control devices“
• Our target for the use of BCI system
are patients severely paralyzed
(locked-in) due to injury or disease
•
1.
2.
3.
4.
There are four general steps in a
BCI system:
Signal acquisition
Classification and recognition
Interaction with device
Feedback to Subject
May 12, 2009
BCI Workshop ICRA 2009, Kobe
2
Evaluation of a robot as embodied interface
for Brain Computer Interface systems
Original P300-based BCI of IRCCS San Camillo
• This BCI system is based on P300 wave detection
• P300 is event-related potential
• Visual interface gives to patient stimulus to activate P300 signal
• The P300 detection will control the movement of cursor
• Directional cues are represented by the
single flash of one out four target-direction
arrows
• Participants focus their attention on the
flashing arrow corresponding to the desired
direction of motion
• Validated on 5 tetraplegic patients and 7
healthy patients [Piccione et al. Clinical
Neurophysiology 2006]
May 12, 2009
BCI Workshop ICRA 2009, Kobe
3
Evaluation of a robot as embodied interface
for Brain Computer Interface systems
P300-based BCI of IRCCS San Camillo
Focusing own attention, patient will generate P300 signal:
May 12, 2009
BCI Workshop ICRA 2009, Kobe
4
Evaluation of a robot as embodied interface
for Brain Computer Interface systems
Goals of our project
In this project we aim at investigating two main issues:
• Giving to locked-in patients the control of a physical
actuactor to improve their rehabilitation experience
• Developping telepresence of the patient with the body of the
robot as medium
The questions are:
Can the robot lead patient to full and better control of BCI via a
higher patient engagement in the task?
Might patients identify them-selves into the robot and think of it as an
extension of their perceptions and actions (i.e. their body)?
May 12, 2009
BCI Workshop ICRA 2009, Kobe
5
Evaluation of a robot as embodied interface
for Brain Computer Interface systems
Robot as a physical actuator
•
To be able to move a body again, might give
new strength and will to the locked-in patients
1. Custom built holonomous robot with hexagonal
structure and three omnidirectional wheels
2. The holonomous nature of the robot is very
important to replicate the motion
and the apparence of the interface used
by our BCI
The robot is fitted with:
• Omnidirectional camera
• Audio board
• WiFi connectivity
May 12, 2009
BCI Workshop ICRA 2009, Kobe
6
Evaluation of a robot as embodied interface
for Brain Computer Interface systems
BCI and robot connection
BCI system
SERVER
Command
Robot
Client
Action result
• Interaction between robot and BCI is based on client-server system with TCP-IP
protocol via IEEE 802.11
Security about connection between BCI and robot is guaranteed by:
• System protocol and network protocol check data received;
• Virtual Private Network between BCI and robot with autentication and encryption
May 12, 2009
BCI Workshop ICRA 2009, Kobe
7
Evaluation of a robot as embodied interface
for Brain Computer Interface systems
Test and Demo
We carried out many tests to integrate BCI and robot. These tests
accomplished at IRCCS San Camillo in Venice, and in IAS Laboratory of
University of Padua:
• Robot and BCI computer connected with wired
network
• Testing healthy subject performances to move
robot
•BCI and robot connected in larger network with
VPN connection
(robot was in Padova, and BCI system in
Venice)
May 12, 2009
BCI Workshop ICRA 2009, Kobe
8
Evaluation of a robot as embodied interface
for Brain Computer Interface systems
Test and Demo
During the scientific festival Discovery on Film in Rovereto (May 2008).
A healthy subject at San Camillo in Venice was able to move the robot
using the BCI at museum of Rovereto (about 200 Km from Venice).
The test was successful and subject was able to reach the target.
You can see it at:
www.sperimentArea.tv the web TV of
the Museo Civico of Rovereto
May 12, 2009
BCI Workshop ICRA 2009, Kobe
9
Evaluation of a robot as embodied interface
for Brain Computer Interface systems
Robot as actuator and generator of interface
Robot is able to send back to patient
images and sounds about
environment.
The images token by omnidirectional
camera can be integrated in BCI
interface
to allow patient to see through
robot’s camera.
A new integrated interface for BCI is
under clinical validitation by prof.
Priftis of Psychology Department of
University of Padua
May 12, 2009
BCI Workshop ICRA 2009, Kobe
10
Evaluation of a robot as embodied interface
for Brain Computer Interface systems
Robot as actuator and generator of interface
Figure 6: classification accuracy (%) of the 5 healthy subjects
who performed Experiment 1, and classification accuracy (%) of
one subject who performed Experiment 2.
May 12, 2009
BCI Workshop ICRA 2009, Kobe
11
Evaluation of a robot as embodied interface
for Brain Computer Interface systems
Robot senses
May 12, 2009
BCI Workshop ICRA 2009, Kobe
12
Evaluation of a robot as embodied interface
for Brain Computer Interface systems
Robot as a Telepresence medium
•
The patient is able to see and listen with the senses of the robot and so he is
able both to extend own social life and communicate own needs
•
Telepresence for museum
visits: more autonomous
navigation,obstacle avoidance,
new graphical user interface
•
Telepresence for household
rehabilitation: consumer robot
Rovio holonomous robot with
frontal camera for remote
perception.
May 12, 2009
BCI Workshop ICRA 2009, Kobe
13
Evaluation of a robot as embodied interface
for Brain Computer Interface systems
Robot as a Telepresence for museum visits
Patient in Venice, robot in Agrigento, Sicily, 1467 km
The BCI
Server
TCP/IP
Internet
May 12, 2009
BCI Workshop ICRA 2009, Kobe
14
The Client robots developed at
University of Palermo, Sicily, ITALY
Pioneer3-AT (Outdoor Robot)
PeopleBot (Indoor Robot)
ARES – CISIS 2009 Fukuoka Institute of Technology, Fukuoka 16 March 2009
The Graphical Interface

STEP 1: Choose of the real robot
ARES – CISIS 2009 Fukuoka Institute of Technology, Fukuoka 16 March 2009
The Graphical Interface

STEP 2:
The Session Start and Sensors and Actuators are
Activated
ARES – CISIS 2009 Fukuoka Institute of Technology, Fukuoka 16 March 2009
The Graphical Interface
STEP 2.1:
The Applet for the
visualization of the
navigation
environment
through the robot
camera

ARES – CISIS 2009 Fukuoka Institute of Technology, Fukuoka 16 March 2009
The Graphical Interface

STEP 3:
Robot Movements Activation through the use of
the 4 directional arrows
ARES – CISIS 2009 Fukuoka Institute of Technology, Fukuoka 16 March 2009
The Graphical Interface
ARES – CISIS 2009 Fukuoka Institute of Technology, Fukuoka 16 March 2009
Evaluation of a robot as embodied interface
for Brain Computer Interface systems
Robot as a Telepresence for museum visits
•
Next step: fully autonomous robot, robot has a map of the environment,
robot presents four topological choices (e.g. the four closest masterpieces),
subject selects destinations
May 12, 2009
BCI Workshop ICRA 2009, Kobe
21
Evaluation of a robot as embodied interface
for Brain Computer Interface systems
Robot as a Telepresence for museum visits
•
Next step: fully autonomous robot, robot has a map of the environment,
robot presents four topological choices (e.g. the four closest masterpieces),
subject selects destinations
May 12, 2009
BCI Workshop ICRA 2009, Kobe
22
The Experimental Results: Simulation
BCI- Web_Interface
BCI- Web_Interface
Snapshot of Robot
Position
Snapshot of
Robot Position
Time T1: Command Go Forward
Time T10: Command Stop
The BCI-Robot moving in the DINFO Department.
Many Thanks to You and to:

Dept. of Information Engineering, University of Palermo, Palermo, Italy
Rosario Sorbello: sorbello_rosario@unipa.it
Antonio Chella: chella@unipa.it
Salvatore Anzalone: anzalone@dinfo.unipa.it
Francesco Cinquegrani: cinquegrani@dinfo.unipa.it

Dept. of Information Engineering, University of Padova, Padova, Italy
Emanuele Menegatti: emg@dei.unipd.it
Dept. of Neuro-rehabilitation, S.Camillo Hospital, Venice, Italy
Laboratory: labeeg.nfsancamillo@virgilio.it

Dept. of General Psychology, University of Padova, Padova, Italy
Konstantinos Prifitis: konstantinos.priftis@unipd.it
