Natural Cost functions for contact
selection
Paul R. Schrater
University of Minnesota
Collaborators:
Erik Schlicht, Erik Flister, Charles Sloane
Task-dependent Perception
• Importance of visual information varies with task
• Potential state space of every natural scene is huge.
Example: Information for Distance
How far the ball on the right
from you?
L(a,de )   (a  de ) or exp  (a  de ) 2 
a  argmax
 L(a,d ) pd
e
e
Datadde
Which ball is closer to you?

Which ball is closer to the table?
Questions like these can involve different
state spaces and optimal inference
strategies (Schrater & Kersten, 2000)
de
db
dt
Information for Inter-object distances
Which ball is closer to the table?
Example: Information for Contact
Example: Information for Contact
By Construction
Only reliable features are
Shadows
Example: Information for surface
smoothness
Which Ball is smoother?
Difference Image
Dominant
Information
Specularities
Task-dependent Perception
• Importance of visual information varies with task, even within the same
scene.
• Potential state space of every natural scene is huge.
• Computational resources limited in the brain.
• How does the brain determine how to allocate perceptual resources?
• Need a theory of goal-oriented perception
• Proposal: The brain turns perceptual and motor goals into loss
functions to filter information relevant to the current task
Goal of a task can be encoded in a loss
function
• Perceptual tasks– Actions = judgments
– Typical goal- minimize error in perceptual judgments
– Loss functions
• Error rate (discrete)
• Estimation error (continuous)
• Motor control tasks– Actions=Movement of body parts
– Typical goal- achieve desired body position/motion with little effort.
– Loss functions
• Sequential control decisions
• Cost-to-go to achieve goal position
Loss functions as Information Filters
Risk = Expected Loss
R(u) 
 L(s,u) p(s | D)ds
Assume posterior distribution is approximately Gaussian
s  E p(s |D ) s , and
Let

C  E p(s |D ) s  s s  s 
T

Taylor series in L around the expected state
L(s,u)  L(s,u)   s L(s,u) s  s   s  s   s  s L(s,u) s  s 
T
Plugging into the Risk
R(u) 


L(s,u)   s L(s,u) s  s   s  s   s  s L(s,u) s  s  p(s | D)ds
T
R(u)  L(s,u)  Tr s  s L(s,u) C 
Given s is chosen to diagonalize
C
L2 (s,u) 2
R(u)  L(s,u)  
 i
2
si
After simplifying
Loss functions as Information Filters
Each state variable contributes differentially to the risk:
L2 (s,u) 2
i
2
si
 w i (s,u)   i2
Thus, the impact of loss on the ith component of the state
2
vector is to re-weight its reliability  i

by the curvature
of the loss in that direction.

Loss function for contact selection
• If relevant information is filtered by task goals, it should
work for some non-trivial case:
• Where should you place your fingers when grasping an
object?
Why Non-Trivial
?
?
• Depends on movement goal
• lift, rotate, flip, push, slide
• Seems like a visual task (Can’t
wait till contact to decide) but has
motor goal.
• Grasp quality depends on nonvisual factors. (Mass, COM, Intertia,
Friction, Geometry w.r.t. COM, etc)
Perception for contact selection
• State variables needed:
– Vision does well:
•
•
•
•
Object boundary
Object configuration w.r.t. the body
Surface material properties of the object
Collision time/velocity
– Vision does poorly or not at all
•
•
•
•
•
Mass
Center of mass location
Body inertia
Friction
open question- Is vision simply performing recognition of previously felt
objects, or is it also making inferences based on assumptions– mass uniformity+material inference?
– Shininess => slippery
Goal of contact- Manipulation
The goal of object contact is typically to manipulate the object.
Therefore the information relevant to determining contact should
be derived from that goal.
Thus, we will try to connect manipulation with contact.
Proposal: Brain has internal models for object manipulation.
– Forward kinematic model to predict object motion given finger motions
– Need inverse dynamic model for computing controls required to
produce object motion and maintain contact.
– Need observer model that can supply feedback
• Contact state
• Object state
• Finger state
Internal models for object motion
Key ideas:
– Once an object is grasped, it is effectively part of the hand.
– The motor system has learned internal models for manipulation
(maintaining contact and applying appropriate forces)
• Internal model idea well supported in motor control
– (e.g. Neilson et al, 1985; Ghahramani & Wolpert, 1997, etc.)
• Evidence for internal models for contact– Eye fixations directed to intended grasp locations (Johansson et. al, 2001)
– Plan for object shape (Santello & Soechting, 2000; Jenmalm & Johansson, 1997)
– Plan for grip force for familiar object weight (Johansson & Westling, Gordon et. al,
Witney et. al, 2001, etc )
– Plan for center of mass
– Plan for familiar friction (Burstedt et al, 2000)
Contact as a Kinematic Linkage
Contact without slippage acts
Like a 5-dof joint with transformations
Given link parameters, object velocity
determined by finger & contact velocities
Kinematic Grasp Constraint -> Dynamics
• For details, see Murray, Li, Sastry (1994)
– Equations for the dynamics of the object can be written down using the Pfaffian
kinematic constraints
x  Generalized Object Coordinates
q  Generalized Hand  Object Coordinates
J o  Object Jacobian
J h  Hand Jacobian
  Joint Torques
˜ (q) xÝ
Ý C˜ (q, qÝ) xÝ N˜ (q, qÝ) x  F
M
Standard robotics equation
But defined on object variables
and transformed by adding the
object via the virtual linkage
Mass/ Inertia
EffectiveCoriolis
Gravity/ NonCon
where
˜ (q)  M  J J T M J 1J T
M
o
o h
f h o
F  J o J hT 
N˜ (q)  N o  J o J hT N f
Planning for Object Motion
The goal of the task is to move the object.
Proposal: Cost for desired object motion induces contact selection
Optimal Stochastic Control Model for
Perception and Action
Generative World Model
Generative Perception Model
P(xt |Data)
Actions
L(xt ,uk)
Goals
Beliefs
yk
Sensory Data
Stochastic Optimal Control (Abstract)
System Model
Optimal
Cost-to-go
Typical Loss
Function
Posterior distrib.
on state
Noise Model
Loss
Control law
Goal
loss
Control loss
Loss function for Object Motion
Given:
• Goal states for object:
• Min control cost on motion, both without and with object
Write Cost-to-go to penalize desired object via-states and control cost
where
Rewrite in terms of three control models: pre-,mid-,post-contact
Loss function for Object Motion
• After rewriting, the cost-to-go explicitly incorporates
contact locations (and potentially velocities).
• Contact locations are determined by finger positions,
and finger positions by controls
• Thus, in principle, minimizing cost-to-go with contact
model would allow automatic online selection of
contact conditions.
• Alternatively, contact points could be pre-picked
(estimated)by partially optimizing the cost function.
Preliminary Tests
More Torque
Less Torque
Less
Torque
More
Torque
Predictions:
Optimal finger positions vary with cylinder orientation
Contact velocities will produce impulses in desired direction
of motion.
Qualitative Difference Between Touch and Lift
Touch
0 Degree: Touch
Gravity
X
Y
Z
Qualitative Difference Between Touch and Lift
Lift
0 Degree: Lift
Gravity
X
Y
Z
Preliminary testing of the idea
How optimal are observed contacts?
• Compute the cost function and compare to human
behavior.
– Full model too hard• Cheated by assuming object motion under min control would take
a minimum Jerk path on average.
• Computed forces required to lift object 5cm in 500msec along min
jerk path over a dense set of finger contact locations
• Simulated a contact dynamics model (frictional rolling without
slip). Used kinematic constraints and required object forces to
compute the finger controls needed to accomplish the task.
• Cost could then be computed on the set of finger controls.
Contact Cost for touch task
Object motion risk
Object Jerk Cost
40 mm
0
+90
180 mm
-90
Human vs. Loss function prediction
Purple: Touch contact positions
Black: Lift contact positions
Conclusions- Speculations
• Goals come first- perception should be driven by
satisfying task demands
• Contact selection is a difficult problem (or maybe I
made it into one).
• Perceptual principle of least commitment- don’t commit
to an estimate when a distribution will do.
• New role for vision- Contact model estimation?
• What are the goals of natural conscious perception?