Visual feedback in the control of
reaching movements
David Knill
and
Jeff Saunders
Two types of motor control
• Ballistic
• Feedback control
Ballistic control
Target state
Initial system state
Motor
planning
Motor
commands
Physical
plant
New System
states
Feedback control
Target state
Initial System state
Motor
planning
Motor
commands
Physical
plant
New System
states
Sensory
system
Baseball examples
• Ballistic control
– Hitting
– Throwing
• Feedback control
– Running to catch a ball
Target
Target
Target
Target
Target
Target
Target
Target
Target
Target
Target
Target
Target
Target
Target
Target
Target
Target
Target
Target
Target
Eye-hand coordination
700
600
X position (mm)
500
EOG signal
400
300
Finger position
200
100
0
0
100
200
300
400
500
Time (msecs.)
600
700
800
Movement start
700
600
X position (mm)
500
EOG signal
400
300
Finger position
200
100
0
0
100
200
300
400
500
Time (msecs.)
600
700
800
Speed profile for pointing movement
1000
900
Speed (mm/sec)
800
700
600
500
400
300
200
100
0
0
100
200
300
400
500
Time (msecs.)
600
700
800
Ballistic control
Target state
Initial system state
Motor
planning
Motor
commands
Physical
plant
New System
states
Questions
• Does the visuo-motor system use visual
information about target on-line to update
motor program?
Feedback control
Target state
Initial System state
Motor
planning
Motor
commands
Physical
plant
New System
states
Sensory
system
Questions
• Does the visuo-motor system use visual
information about target on-line to update
motor program?
• Does the visuo-motor system use
continuous feedback from the hand during a
movement to control the movement
Questions
• Does the visuo-motor system use visual
information about target on-line to update
motor program?
• Does the visuo-motor system use
continuous feedback from the hand during a
movement to control the movement
– What visual information is used?
Question
• Does the visuo-motor system use visual
information about target on-line to update
motor program?
Question
• Does the visuo-motor system use visual
information about target on-line to update
motor program?
• Yes - for detectable target motion (e.g.
catching a moving object)
Question
• Does the visuo-motor system use visual
information about target on-line to update
motor program?
• Yes - for detectable target motion (e.g.
catching a moving object)
• ?? - for imperceptible changes in target
position
Experiment
• Perturb position of target during a saccade
(imperceptible change)
• Does motor system correct for change in
target position?
monitor
mirror
infrared markers
on f inger
tabletop ali gned to
virtual targets
15
10
Perturbed trials
Y (mm)
5
0
-5
-10
Unperturbed
trials
-15
-20
0
50
100
150
200
X (mm)
250
300
350
400
100
Y (mm)
50
0
-50
-100
0
50
100
150
200
X (mm)
250
300
350
Results
• Automatically correct for imperceptible
target perturbations.
• Correct for perturbations
– Perpendicular to movement
– In direction of movement
• Reaction time = 150 ms
• Smooth corrections
Question
• Does the visuo-motor system use
continuous feedback from the hand during a
movement to control the movement?
Hypotheses
• Classic model
– Ballistic control during “fast” phase of motion
– Feedback control during end, “slow” phase of
motion
• Continuous model
– Feedback control throughout movement
Arguments against continuous
feedback
• Visuo-motor delay (~100 ms) is too large
for effective control during fast phase.
• Removing vision of hand early in motion
does not affect end-point error.
• Corrections to target perturbations are just
as strong with or w/o vision of hand.
Experiment
• Imperceptibly perturb the position of the
hand during a movement and measure
motor response.
• Add perturbations early and late in pointing
movement.
• Measure reaction time to perturbations.
monitor
mirror
infrared markers
on f inger
tabletop ali gned to
virtual targets
virtual
fingertip
target
(a)
(b)
(c)
unseen
hand
Reaction time predictions
Early
Late
perturbation perturbation
End-phase
feedback
D
>>D
Continuous
feedback
D
D
Sample finger paths
6
4
Y position (cm)
2
0
-2
-4
-6
-8
-5
0
5
10
15
X position (cm)
20
25
30
Autoregressive model
• Baseline (unperturbed) trajectories
y(t)  w8 y(t  8)  w7 y(t  7) 
 w1 y(t 1)
• Perturbed trials
y(t)  w8 y(t  8)  w7 y(t  7) 
 w1 y(t  1)  wP (t)DY
Subject 1: Trajectories for early perturbed trials
30
Positive perturbations
25
X position (cm)
20
Negative perturbations
15
10
5
0
0
100
200
300
400
500
Time (ms)
600
700
800
900
1000
Subject 1: Trajectories for early perturbed trials
30
29
28
X position (cm)
27
Positive perturbations
26
25
24
23
22
Negative perturbations
21
20
0
100
200
300
400
500
Time (ms)
600
700
800
900
1000
Subject 1: Trajectories for late perturbed trials
30
Positive perturbations
25
X position (cm)
20
Negative perturbations
15
10
5
0
0
100
200
300
400
500
Time (ms)
600
700
800
900
1000
Subject 1: Trajectories for late perturbed trials
30
29
28
X position (cm)
27
Positive perturbations
26
25
24
23
Negative perturbations
22
21
20
0
100
200
300
400
500
Time (ms)
600
700
800
900
1000
Subject 2: Trajectories for early perturbed trials
30
Positive perturbations
25
X position (cm)
20
Negative perturbations
15
10
5
0
0
100
200
300
400
500
Time (ms)
600
700
800
900
1000
Subject 2: Trajectories for late perturbed trials
30
Positive perturbations
25
X position (cm)
20
Negative perturbations
15
10
5
0
0
100
200
300
400
500
Time (ms)
600
700
800
900
1000
Perturbation weight function for in-line perturbations
3
x 10-3
Perturbation weight
2
1
0
-1
-2
-3
0
50
100
150
200
250
Time (ms)
300
350
400
450
500
Perturbation weight function for perpendicular perturbations
1
x 10-3
0.5
0
Perturbation weight
-0.5
-1
-1.5
-2
-2.5
-3
-3.5
-4
0
50
100
150
200
250
Time (ms)
300
350
400
450
500
Conclusions
• Visuomotor system uses directional error
signal for feedback control?
• Position / speed error in direction of
movement is not effective feedback signal?
• Why?
– Position along path blurred by motion
– Insensitivity to acceleration along direction of
motion
Question
• What visual information about hand does
visuomotor system use
– Position error?
– Motion error?
– Position and motion?
virtual
fingertip
target
(a)
(b)
(c)
unseen
hand
target
(a)
(b)
(c)
4
x 10-4
2
Perturbation weight
0
-2
-4
-6
-8
-10
-12
0
50
100
150
200
250
Time (msecs.)
300
350
400
450
1
x 10-3
0.5
0
Perturbation weight
-0.5
-1
-1.5
-2
-2.5
-3
-3.5
-4
0
50
100
150
200
250
Time (msecs.)
300
350
400
450
0.5
x 10-3
0
Perturbation weight
-0.5
-1
-1.5
-2
-2.5
-3
-3.5
0
50
100
150
200
250
Time (msecs.)
300
350
400
450
0.5
x 10-3
Perturbation weight
0
-0.5
-1
-1.5
-2
-2.5
0
50
100
150
200
250
Time (msecs.)
300
350
400
450
1
x 10-3
0.5
0
Perturbation weight
-0.5
-1
-1.5
-2
-2.5
-3
-3.5
-4
0
50
100
150
200
250
Time (msecs.)
300
350
400
450
1
x 10-3
0.5
Perturbation weight
0
-0.5
-1
-1.5
-2
-2.5
-3
-3.5
0
50
100
150
200
250
Time (msecs.)
300
350
400
450
Conclusions
• Visuomotor system uses continuous visual
feedback to control reaching movements.
• Feedback signals include positional error.
• Feedback signals include motion error.
• System is approximately linear.
8
x 10-3
6
Perturbation weight
4
2
0
-2
-4
-6
0
50
100
150
200
250
Time (msec.)
300
350
400
450
0.02
Cumulative perturbation weight
0.01
0
-0.01
-0.02
-0.03
-0.04
-0.05
-0.06
0
50
100
150
200
250
Time (msec.)
300
350
400
450
4
x 10-3
2
0
Perturbation weight
-2
-4
-6
-8
-10
-12
-14
-16
0
50
100
150
200
250
Time (msec.)
300
350
400
450
0.05
Cumulative perturbation weight
0
-0.05
-0.1
-0.15
-0.2
-0.25
-0.3
-0.35
0
50
100
150
200
250
Time (msec.)
300
350
400
450
6
x 10-3
4
Perturbation weight
2
0
-2
-4
-6
-8
-10
-12
0
50
100
150
200
250
Time (msec.)
300
350
400
450
Cumulative perturbation weight
0.05
0
-0.05
-0.1
-0.15
-0.2
0
50
100
150
200
250
Time (msec.)
300
350
400
450
6
x 10-3
4
Perturbation weight
2
0
-2
-4
-6
-8
-10
-12
0
50
100
150
200
250
Time (msec.)
300
350
400
450
0.1
Cumulative perturbation weight
0.05
0
-0.05
-0.1
-0.15
-0.2
-0.25
0
50
100
150
200
250
Time (msec.)
300
350
400
450
8
x 10-3
6
Perturbation weight
4
2
0
-2
-4
-6
-8
-10
0
50
100
150
200
250
Time (msec.)
300
350
400
450
0.01
Perturbation weight
0.005
0
-0.005
-0.01
-0.015
-0.02
0
50
100
150
200
250
Time (msec.)
300
350
400
450