ITU Workshop on
“E-health services in low-resource settings:
Requirements and ITU role”
(Tokyo, Japan, 4-5 February 2013)
Cognitive neuroscience approach
to telemedicine
Makio KASHINO, Ph.D.
Senior Distinguished Scientist, Executive Manager
NTT Communication Science Laboratories
Kashino.makio@lab.ntt.co.jp
Tokyo, Japan, 4-5 February 2013
Human Information Lab, NTT CSL
Psychophysics
Neuroscience
Perception,
Emotion,
Motor control
Man-machine
interface
Informatics
!7?#7@A7347BC7
!7?#7@A7347BC7
23453#
678479:
<=>7:
23453#
678479:
25:;73453#
678479:
<=>7:
25:;73453#
678479:
01
Tokyo, Japan, 4-5 February 2013
01
2
Challenges in telemedicine
!
Transmission delay and asynchrony
!
Guiding a remote collaborator
!
Face-to-face-like communication
Tokyo, Japan, 4-5 February 2013
3
Compensation for transmission delay
Smooth
"
Awkward
Smooth
Heavy
No delay
Delay
Tokyo, Japan, 4-5 February 2013
Delay
With delay
+
Delay+flow
Takamuku, Gomi (2011) ECVP2011
4
Compensation for transmission delay
Tokyo, Japan, 4-5 February 2013
Takamuku, Gomi (2011) ECVP2011
5
Compensation for transmission delay
Tokyo, Japan, 4-5 February 2013
Takamuku, Gomi (2011) ECVP2011
6
Involuntary arm movement
induced by vision
17
Manual Following Response (MFR)
110 ms
110 ms
R
L
CR
85 ms
L
0.05
[m]
Y
X
85 ms
・Involuntary
・Fast
91 ms
91 ms
X (Manual Following
MFR
Response)
Movement duration: 1.0 s
Saijo et al. 2005 J.Nneurosci.
assisted
by 4-5
beeping
Tokyo, Japan,
February sounds
2013
Y
]
[m
.5
0
Saijo, Murakami, Nishida, Gomi (2005)
The Journal of Neuroscience
0.05
[m]
Movement duration: 1.0 s
assisted by beeping sounds
Fixation marker
]
[m
.5
0
normalized
alized
Y
85 ms
normalized
0.05
[m]
RC
L
C
110 ms
MFR (Manual Following Respons
]
[m
.5
0
7
Different visual processing systems
for perception and motor control
Fast motion of a coarse pattern is not salient
perceptually, but effective to induce MFR
Tokyo, Japan, 4-5 February 2013
Gomi, Abekawa, Nishida (2006)
The Journal of Neuroscience
8
Audiovisual simultaneity
How to judge A-V timing in the presence of
significant external and internal asynchrony?
Tokyo, Japan, 4-5 February 2013
9
Adaptation to A-V asynchrony
induces…
Tokyo, Japan, 4-5 February 2013
Fujisaki, Shimojo, Kashino, Nishida
(2004) Nature Neuroscience
10
Recalibration of A-V simultaneity
“Simultaneous” response
adaptation (0ms)
N=7
adaptation
(+235ms)
adaptation
(-235ms)
A first
V first
Test AV lag (ms)
Tokyo, Japan, 4-5 February 2013
Fujisaki, Shimojo, Kashino, Nishida
(2004) Nature Neuroscience
11
Challenges in telemedicine
!
Transmission delay and asynchrony
!
Guiding a remote collaborator
!
Face-to-face-like communication
Tokyo, Japan, 4-5 February 2013
12
Perceptual attraction force: Buru-Navi
How to generate “pull” or “push” sensation in
the mobile devices having no external fulcrum?
Amemiya, Ando, Maeda (2007)
Laval Virtual 2007
Tokyo, Japan, 4-5 February 2013
Amemiya, Gomi (2012)
IEEE Transactions on Haptics
13
Navigation by Buru-Navi
Successfully navigated ~90% (21/23) of
visually-impaired people in a maze
Tokyo, Japan, 4-5 February 2013
Amemiya, Sugiyama (2009) ASSETS
14
Challenges in telemedicine
!
Transmission delay and asynchrony
!
Guiding a remote collaborator
!
Face-to-face-like communication
Tokyo, Japan, 4-5 February 2013
15
Implicit InterPersonal Information
IIPI (interpersonal interaction of involuntary and subconscious
body movements and physiological responses) may play critical
roles in communication
→ Decoding and controlling IIPI for better communication systems
Interpersonal
interaction
of
ERPs after Face1
presented (200-400ms)
brain and ANS activities Interpersonal interaction of
body movements
()#
,#
+#
*#
"#
$#
%#
&#
'#
(#
)#
)#
(#
'#
&#
%#
$#
"#
*#
+#
!(#
!'#
!&#
Wavelet-based time-freq. plot, after Face2 presented
Theta, 5-8 Hz (500ms)
Gamma, 40-60 Hz (650ms)
!%#
!$#
!"#
Caltech
ⓒⓒCaltech
")#
presented
Supported by JST CREST (PI: Makio Kashino [NTT], Collaborators:
ally.
$)#
%)#
Sinsuke Shimojo [Caltech], Katsumi Watanabe [U Tokyo])
&)#
')#
Tokyo, Japan, 4-5 February 2013
(Lindsen, et al., NeuroImage, '10)
()#
16
Acoustic IIPI for talker “presence”
Spontaneous utterances recorded and reproduced by
the sound field sharing system “BoSC” (CREST Ise team)
Dynamic
Static
→ Subjective rating, ANS activity, hormone concentration
Tokyo, Japan, 4-5 February 2013
Kobayashi, Ooishi, Enomoto, Kitagawa,
Ueno, Ise, Kashino (in preparation)
17
Acoustic IIPI for talker “presence”
n  Subjective rating
1. Artificial sound?
2. Feel talker presence?
3. Like the talker?
4. Interested in the content?
n  Significant differences also found for sympathetic
nerve activity and salivary hormone concentration
Tokyo, Japan, 4-5 February 2013
Kobayashi, Ooishi, Enomoto, Kitagawa,
Ueno, Ise, Kashino (in preparation)
18
Concluding remark
!
Challenges in telemedicine may be
overcome by taking advantage of
brain mechanisms underlying human
perception, emotion, motor control,
and communication.
Tokyo, Japan, 4-5 February 2013
19