From Neuroscience to Mechatronics
Presentation by Fabian Diewald
JASS April 2006
Outline
The human brain
The vestibulo-ocular reflex (VOR)
The VOR – technical applications
Camera stabilizing system at TU Munich
A possible algorithm
Testing the camera stabilizing system
Another field of neuroscience and technical
application: face recognition
Conclusion and outlook
Learning from nature – a justified
strategy
first vertebrates ~500 million years ago
civilization of man since ~10 000 years
Consequently nature has a great time advantage!
The human brain
cerebellum = "little
brain"
responsible for accurate
movement
instructions by the
forebrain insufficient
instructions have to be
translated into accurate
commands by the
cerebellum
cerebellum essential
part in learning motor
skills
Neurons – elementary components of
the central nerve system
dendrites: "input" of a neuron
axon: "output"
axon terminals/boutons contact other neurons or
muscles
Neurons – elementary components of
the central nerve system
transmission works electrically
information through the axon is encoded in
changes of electrical potential
short impulses with fixed intensity
information is contained in
firing frequency
("pulse rate modulation")
Synapses – link between neurons
gap against electrical
transmission
transmission between
neurons is controlled by
chemicals called
neurotransmitters
controlled transmission
is important in regard to
adaptation and learning
The structure of the cerebellar cortex
parallel fibres
Purkinje cells:
~200 000 synapses per cell,
only output for cerebellar cortex
climbing fibre:
axon of inferior olive;
one single climbing fibre
for one Purkinje cell
wraps around dendritic
Inferior
tree of Purkinje cell; Olive
essential for "learning"
parallel fibres
=axons of granule cells,
synapse on Purkinje cells
granule cells,
~20 per mossy fibre,
half of all neurons are
Granule
cells
granule cells
Purkinje
cells
climbing
fibre
-
+
- Deep +
a
a
cerebellar
nuclei
inferior olive
Diagram from Max Westby
mossy fibres
= information input,
synapse on
granule cells
mossy
fibre
The influential theory of Marr-Albus
twice independently proposed:
Marr, David: “A Theory of Cerebellar Cortex”, 1969,
Journal of Physiology 202: 437-470
Albus, James S.: "A theory of cerebellar function", 1971,
Mathematical Biosciences 10: 25-61
major aspects:
climbing fibre carries a teaching signal
signal influences the synapses of Purkinje cells
consequently the intensity of certain inputs to the
Purkinje cells can be controlled
The influential theory of Marr-Albus –
disappointment by Marr himself
Marr 1982:
"In my own case, the cerebellar study (...) disappointed
me, because even if the theory was correct, it did not
enlighten one about the motor system – it did not, for
example, tell one how to go about programming a
mechanical arm."
Outline
The human brain
The vestibulo-ocular reflex (VOR)
The VOR – technical applications
Camera stabilizing system at TU Munich
A possible algorithm
Testing the camera stabilizing system
Another field of neuroscience and technical
application: face recognition
Conclusion and outlook
The vestibulo-ocular reflex (VOR)
aim: stable picture of our environment/object we are staring at in spite
of moving head
only use of visual information would be too slow for stabilization (visual
processing delay about 50-100ms)
signals of the vestibular organ (motion of the head) are more or less
directly transmitted to the extra-ocular eye muscles, leading to process
times of 5-10ms
vestibular signals are also influenced by visual information
The vestibulo-ocular reflex (VOR) –
experimentation with monkeys
VOR does an excellent job in monkeys as well
as in other vertebrates under normal
conditions
magnifying or miniaturizing glasses cause
abnormal image motion speed perceived
during head motion
monkeys with glasses show problems with
head motion in the beginning
after several days: considerable improvement
is completed
then: taking the glasses away
monkeys show problems again
after some days: normal behaviour again
unconscious use of VOR, motoric issue,
calibration needed
 typical case concerning the cerebellum
The vestibulo-ocular reflex –
paths of information
eye
(visual information)
vestibular organ
(head motion)
direct path:
very quick
not accurate
(not sensible to
changes in the system,
e.g. changes in eye
adaptation path:
muscle strength)
teaching signal
transferred on
climbing
fibres
"strengthens" or
"weakens" the
synapses
inferior olive
eye muscles
adaptive path
over Purkinje
cells:
transferred
on parallel
fibres
synapses of
Purkinje cells
Purkinje cells
cerebellum
Outline
The human brain
The vestibulo-ocular reflex (VOR)
The VOR – technical applications
Camera stabilizing system at TU Munich
A possible algorithm
Testing the camera stabilizing system
Another field of neuroscience and technical
application: face recognition
Conclusion and outlook
The vestibulo-ocular reflex –
possible technical applications
camera systems in general
face-recognition
systems
The vestibulo-ocular reflex –
also an important automotive application
Affordable
driver-assistance systems
wide angle cameras used
to obtain an overview of
the environment
for a closer look (road
signs, number plates etc.)
a telephoto lens is needed,
which is quite sensitive to
motion
Requirements concerning driverassistance systems
camera stabilization only by optical means is too slow
inertial measurement of head angular velocity
needed
affordable hardware
cheap inaccurate sensors must be allowed (multisensor fusion of translation/angular velocity and
visual information)
inaccuracies have to be compensated
automatically
 similar problems/requirements as in biology
the vestibulo-ocular reflex may solve our
technical problem!
Outline
The human brain
The vestibulo-ocular reflex (VOR)
The VOR – technical applications
Camera stabilizing system at TU Munich
A possible algorithm
Testing the camera stabilizing system
Another field of neuroscience and technical
application: face recognition
Conclusion and outlook
The VOR – biological and technical
analogons
How can we implement cerebellar structures as computer hardware?
the structure of the cerebellum is quite homogeneous –
although different information appears in different regions,
these regions are similar
idea of "cerebellar chip"
today: normally no need for cerebellar chip as DSPs are
very multifunctional and cerebellar algorithms can be
implemented software-based (e.g. in Matlab)
DSP dSPACE DS1103 PowerPC,
used in camera stabilizing system at
TU Munich
The VOR – biological and technical
analogons
neuroscience/biology:
vestibular organ in the inner ear
direct pathway vestibular to muscles
cerebellum
extra-ocular eye muscles
mechatronics:
6-DOF inertial measurement unit
CAN bus system
DSP with specific algorithm
servo motors
main difficulties:
commercially available inertial measurement units are too big
for driver-assistance systems
the algorithm biology uses in the cerebellum has to be
detected and implemented, which is obviously difficult
(remember Marr's quotation!)
Camera stabilization system at TU
Munich, Institute of Applied Mechanics
serial gimballed configuration
two perpendicular axes
pan actuator driven by:
4.5 Watt motor
inner frame driven by:
11 Watt Maxon motor with
backlash free HarmonicDrive
gearbox
each axis controlled by differential
encoders with 512 steps
camera: CMOS sensor with effective
resolution of 750x400 pixels
in the future: only a mirror is moved leading
to further miniaturization
Coping with difficulty 1:
size of inertial measurement unit (IMU)
smallest available intelligent IMU 50x38x25 mm³
too big for driver-assistance-systems
new IMU was developed within the FORBIAS
project:
edge length:
3 gyroscopes
(rotation), bandwith
up to 100Hz
accelerometer with 3 axes
on one chip
(translation), bandwith up
to 640Hz
15 mm
Outline
The human brain
The vestibulo-ocular reflex (VOR)
The VOR – technical applications
Camera stabilizing system at TU Munich
A possible algorithm
Testing the camera stabilizing system
Another field of neuroscience and technical
application: face recognition
Conclusion and outlook
Coping with difficulty 2 (main difficulty):
implementing an adequate algorithm (1)
central ideas/principles:
motor movements are calculated according to angular velocity of
measurement unit ("direct path")
theoretically (perfect sensors/perfect production) this is sufficient for
camera stabilization
but: due to inaccuracies in production and sensors themselves there is
always relative movement of the picture of the environment ("optical
flow")
the relation between angular rates and this relative movement is
calculated
the result dynamically influences a matrix w
Coping with difficulty 2 (main difficulty):
implementing an adequate algorithm (2)
central ideas/principles:
this matrix w calculates a certain output out of angular velocity
this output is used to "clean" the signal of the direct path (velocity)
additively
result: after some time the matrix w is "perfect" (learning completed) and
always knows what to add to the measured angular velocity so that there
is no optical flow any more, i.e. optical flow and angular velocity are
decorrelated
consequently further cost-reduction possible by teaching the
system after assembling and storing the matrix w on a chip
but in this case inaccuracies because of e.g. plastic mechanical
deformation during use cannot be compensated
The vestibulo-ocular reflex –
paths of information
eye
(visual information)
vestibular organ
(head motion)
direct path:
very quick
not accurate
(not sensible to
changes in the system,
e.g. changes in eye
adaptation path:
muscle strength)
teaching signal
transferred on
climbing
fibres
"strengthens" or
"weakens" the
synapses
inferior olive
eye muscles
adaptive path
over Purkinje
cells:
transferred
on parallel
fibres
synapses of
Purkinje cells
Purkinje cells
cerebellum
Coping with difficulty 2 (main difficulty):
implementing an adequate algorithm
fz
w
f pf
fy
flow, f x
x
y
z
x
y
z
d
03x3
E 3 x3
p
E 3 x3
03 x3
central idea: decorrelation of angular rates and optical
decorrelator: contains dynamic state matrix w, dynamically influenced by optical flow,
angular velocity and its derivative
f (optical flow = teaching signal)
Decorrelator with weight matrix w  wp
D
decomposition to
"parallel fibre signals"
w 
pf   
w 
T

wd ,
0
d  J w w 
equivalent to synapses of Purkinje cells
p
 fx 
T
w   f  pf      f y  w x w y
 f z 

 p  E 3 x3
03 x 3 

 d  E 3x3 
w z w x w y w z 
3 x3
 0
E 3x3
p  J wp wd  3x3
0


d
3x3
E
3x3
03x3 
 pf
03x3 
03x3 
 pf
03x3 
parallel fibres
w
S
diagonal
sensitivity
matrix
Decomposition:
splitting physical
input B
signal in
b  JSw
brainstem
adequate inputs
for decorrelator
= direct
path
(decorrelator should be able to
learn by derivatives of velocity
as well, not only by velocity!)
biological equivalent:
parallel fibres
Inferior
Olive
I
+
Purkinje
cells
+
P
-
C
Granule
cells
climbing
fibre
+
- Deep +
a
a
cerebellar
nuclei
mossy
fibre
Diagram from Max Westby
from: Günthner, W.; Glasauer, S.; Wagner, P.; Ulbrich, H.: Biologically inspired multi-sensor fusion for
adaptive camera stabilization in driver-assistance systems, Advanced Microsystems for Automotive
Applications AMAA, Berlin, April 25-27, 2006 (in press)
Coping with difficulty 2 (main difficulty):
implementing an adequate algorithm
fz
w
f pf
fy
flow, f x
x
y
z
x
y
d
03x3
z
E 3 x3
p
E 3 x3
03 x3
central idea: decorrelation of angular rates and optical
decorrelator: contains dynamic state matrix w, dynamically influenced by optical flow,
angular velocity and its derivative
f (optical flow = teaching signal)
D
decomposition to
"parallel fibre signals"
w 
pf   
w 
w
S
diagonal
sensitivity
matrix
T
Decorrelator with weight matrix w  wp

wd ,
0
d  J w w 
equivalent to synapses of Purkinje cells
p
 fx 
T
w   f  pf      f y  w x w y
 f z 

 p  E 3 x3
w z w x w y w z 
3 x3
 0
03 x 3 

 d  E 3x3 
b  JSw
brainstem B
+
output
given
I
= direct
pathby weighted sum:
E 3x3
p  J wp wd  3x3
0

+
• proportional output p and
• derivative path d (added after integrator I)
• are calculated out of the "input" angular velocity (vector pf)
by the dynamic matrix w
• J: Jacobean projecting angular rates on degrees
of freedom of motion device

P
d
3x3
E
3x3
03x3 
 pf
03x3 
03x3 
 pf
03x3 
-
C
from: Günthner, W.; Glasauer, S.; Wagner, P.; Ulbrich, H.: Biologically inspired multi-sensor fusion for
adaptive camera stabilization in driver-assistance systems, Advanced Microsystems for Automotive
Applications AMAA, Berlin, April 25-27, 2006 (in press)
Coping with difficulty 2 (main difficulty):
implementing an adequate algorithm
fz
w
f pf
fy
flow, f x
x
y
z
x
y
z
d
03x3
E 3 x3
p
E 3 x3
03 x3
central idea: decorrelation of angular rates and optical
decorrelator: contains dynamic state matrix w, dynamically influenced by optical flow,
angular velocity and its derivative
f (optical flow = teaching signal)
D
decomposition to
"parallel fibre signals"
w 
pf   
w 
S
T
Decorrelator with weight matrix w  wp

wd ,
0
d  J w w 
equivalent to synapses of Purkinje cells
p
 fx 
T
w   f  pf      f y  w x w y
 f z 

 p  E 3 x3
w z w x w y w z 
learning rule:
3 x3
 0
03 x 3 

 d  E 3x3 
E 3x3
p  J wp wd  3x3
0


d
3x3
E
3x3
03x3 
 pf
03x3 
03x3 
 pf
03x3 
b  JSw
brainstem B
+
•
similar
to
LMS
algorithm
of+adaptive control theory
= direct path
• here: teaching signal = optical flow f;
in biology: teaching signal = climbing fibre signals
• covariance learning rule (Sejnowski 1977):
–If parallel-fibre firing pf (angular velocity and derivative) is positively
correlated with climbing-fibre firing f (optical flow), reduce the weight w,
–if parallel-fibre firing pf (angular velocity and derivative) is negatively
from: Günthner, W.; Glasauer, S.; Wagner,
P.;with
Ulbrich,
H.: Biologically
inspired
multi-sensor
fusion
for w,
correlated
climbing-fibre
firing f (optical
flow),
increase the
weight
adaptive camera stabilization in driver-assistance
Advanced
Microsystems
for Automotive
–if parallel-fibre systems,
firing pf (angular
velocity
and derivative)
is uncorrelated
with climbing-fibre
firing f (optical flow), no change
Applications AMAA, Berlin, April 25-27,
2006 (in press)
w
diagonal
sensitivity
matrix
I
P
C
Possible variants of the basic systems
(1)
fz
w
f pf
fy
flow, f x
x
y
z
x
y
z
d
03x3
E 3 x3
p
E 3 x3
03 x3
central idea: decorrelation of angular rates and optical
delay ~100ms
decorrelator: contains dynamic state matrix w, dynamically
influenced by optical flow,
angular velocity and its derivative
f (optical flow = teaching signal)
delay ~10ms
D
decomposition to
"parallel fibre signals"
w 
pf   
w 
w
S
diagonal
sensitivity
matrix
T
Decorrelator with weight matrix w  wp

wd ,
0
d  J w w 
equivalent to synapses of Purkinje cells
p
 fx 
T
w   f  pf      f y  w x w y
 f z 

 p  E 3 x3
w z w x w y w z 
3 x3
 0
03 x 3 

 d  E 3x3 
E 3x3
p  J wp wd  3x3
0
for frequencies above ~2.5Hz
the phase difference can get quite big
=> instability in the learning process possible
b  JS
w
brainstem
B pf-signal
=> delay of
(~100ms)
+ and I
+
=smoothing
direct pathfilter removing high frequencies
("eligibility trace")


P
d
3x3
E
3x3
03x3 
 pf
03x3 
03x3 
 pf
03x3 
-
C
from: Günthner, W.; Glasauer, S.; Wagner, P.; Ulbrich, H.: Biologically inspired multi-sensor fusion for
adaptive camera stabilization in driver-assistance systems, Advanced Microsystems for Automotive
Applications AMAA, Berlin, April 25-27, 2006 (in press)
Possible variants of the basic systems
(2)
fz
w
f pf
fy
flow, f x
x
y
z
x
y
z
d
03x3
E 3 x3
p
E 3 x3
03 x3
central idea: decorrelation of angular rates and optical
decorrelator: contains dynamic state matrix w, dynamically influenced by optical flow,
angular velocity and its derivative
f (optical flow = teaching signal)
D
decomposition to
"parallel fibre signals"
w 
pf   
w 
w
S
diagonal
sensitivity
matrix
T
Decorrelator with weight matrix w  wp

wd ,
0
d  J w w 
equivalent to synapses of Purkinje cells
p
 sign f x 
  E 3 x3
T
w  wf 
 pf
   f y  w x w y w z w x w y w z  p 3 x 3
...  sign
 0
 sign f z 


03 x 3 

 d  E 3x3 
E 3x3
p  J wp wd  3x3
0


d
3x3
E
3x3
03x3 
 pf
03x3 
03x3 
 pf
03x3 
capacity of climbing fibre pathway (teaching signal)
is limited
b  JSwdata flow can be avoided
brainstem=>
B unimportant
+
+
I the teaching
P
= direct path
by only evaluating the sign of
signal,
i.e. the direction of the optical flow
=> learning was not significantly worse compared
to learning with exact values
C
from: Günthner, W.; Glasauer, S.; Wagner, P.; Ulbrich, H.: Biologically inspired multi-sensor fusion for
adaptive camera stabilization in driver-assistance systems, Advanced Microsystems for Automotive
Applications AMAA, Berlin, April 25-27, 2006 (in press)
Outline
The human brain
The vestibulo-ocular reflex (VOR)
The VOR – technical applications
Camera stabilizing system at TU Munich
A possible algorithm
Testing the camera stabilizing system
Another field of neuroscience and technical
application: face recognition
Conclusion and outlook
Other visual technical problems to be
solved for driver assistance
optical flow caused by camera rotation can
successfully be removed by camera stabilization
as just seen, but...
line of sight may have to be kept on the road
rapid changes of viewing direction ("saccades")
have to be implemented
furthermore: closer examination of certain
objects (signs etc.) need visual tracking
Testing of the camera stabilization
system under laboratory conditions
testing in the laboratory by mounting the system
on a hexapod
created sensed angular rates of ~100°/s
optical flow was reduced from 6 pix/frame to
less than 1 pix/frame after 2-3 minutes of
adaptation
the improvement within this time was also
subjectively viewable
Testing of the camera stabilization
system "on the road" (1)
system mounted near the rear-view mirror
additional camera installed as well to detect points of interest for the
camera with telephoto lens
system tested in association with saccade and visual tracking
vehicle velocity and yaw rate added to the system via CAN bus to
improve tracking of space fixed objects
Testing of the camera stabilization
system "on the road" (2)
three modes were tested:
1. lane marker was focused in a distance of 60m, followed up to a
distance of 20m, then the next was focused on
2. lane separation was focused on in a constant distance of 40m
3. nothing was focused on but the camera was stabilized around a
constant line of sight
in all modes: bumps could be compensated
static wide angle
camera
stabilized camera
with telephoto lens
Outline
The human brain
The vestibulo-ocular reflex (VOR)
The VOR – technical applications
Camera stabilizing system at TU Munich
A possible algorithm
Testing the camera stabilizing system
Another field of neuroscience and technical
application: face recognition
Conclusion and outlook
Learning from the human brain –
another example
a current example: face recognition
excellent recognition abilities by humans
under different circumstances, e.g.
illumination
viewing angle
pose
facial expression
...
need for technical face recognition systems
(e.g. war against terrorism)
consequently need to continue exploring the
neuroscience of face recognition
Face recognition – the problem with
different angles of view
How can
we/a
computer
tell that
each face
belongs to
the same
person?
Face recognition
– step:
adaption of
testing
unfamiliar view of the
weights and output as
weighted sum
face, but
nevertheless the
highest output is
produced for "Betty"
"hidden units": one for
each angle of view – the
more similar the input is
to the "prototype" face of
one unit, the more
"active" the unit gets
learning step:
changing the weights
within the network so
that the output is "1"
for the viewed person
from: Vatentin, D.; Abdi, H..; Edelman, B.: What represents a face: A Computational Approach for the
Integration of Physiological and Psychiological Data, 1997
Outline
The human brain
The vestibulo-ocular reflex (VOR)
The VOR – technical applications
Camera stabilizing system at TU Munich
A possible algorithm
Testing the camera stabilizing system
Another field of neuroscience and technical
application: face recognition
Conclusion and outlook
Conclusion and outlook
technical use of neuroscience is the key to
giving machines several typical human abilities
optimizing service intervals and
minimizing complexity of installing systems by
self-learning abilities
cost reduction
...
consequently interesting to a large variety of
technical fields
Thank you!
Thank you for your attention!
Fabian Diewald
Fabian.Diewald@mytum.de