Visual Perception and Robotic Manipulation
Springer Tracts in Advanced Robotics
Chapter 7
System Integration and
Experimental Results
Geoffrey Taylor
Lindsay Kleeman
Intelligent Robotics Research Centre (IRRC)
Department of Electrical and Computer Systems Engineering
Monash University, Australia
Overview
•
•
•
•
•
Stereoscopic light stripe scanning
Object Modelling and Classification
Multicue tracking (edges, texture, colour)
Visual servoing
Real-world experimental manipulation tasks
with an upper-torso humanoid robot
Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics
2
Motivation
• To enable a humanoid robot to perform
manipulation tasks in a domestic environment:
– A domestic helper for the elderly and disabled
• Key challenges:
–
–
–
–
–
Ad hoc tasks with unknown objects
Robustness to measurement noise/interference
Robustness to calibration errors
Interaction to resolve ambiguities
Real-time operation
Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics
3
Architecture
Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics
4
Light Stripe Scanning
Scanned
object
D
B
Camera
Stripe
generator
• Triangulation-based depth measurement.
Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics
5
Stereo Stripe Scanner
Scanned
object
X
Left image
plane
xL
xR
Laser
diode
θ
Left
camera
L
Right image
plane
Right
camera
R
2b
• Three independent measurements provide
redundancy for validation.
Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics
6
Reflections/Cross Talk
Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics
7
Single Camera Result
Single camera scanner
Robust stereoscopic scanner
Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics
8
3D Object Modelling
• Want to find objects with minimal prior knowledge.
– Use geometric primitives to represent objects
• Segment 3D scan based on local surface shape.
Surface type classification
Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics
9
Segmentation
• Fit plane, sphere, cylinder and cone to segments.
• Merge segments to improve fit of primitives.
Raw scan
Surface type
classification
Final
segmentation
Geometric
models
Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics
10
Object Classification
• Scene described by adjacency graph of primitives.
• Objects described by known sub-graphs.
Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics
11
Modeling Results
• Box, ball and cup:
Raw colour/range scan
Textured polygonal models
Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics
12
Multi-Cue Tracking
• Individual cues are only robust
under limited conditions:
– Edges fail in low contrast,
distracted by texture
– Textures not always available,
distracted by reflections
– Colour gives only partial pose
• Fusion of multiple cues
provides robust tracking in
unpredictable conditions.
Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics
13
Tracking Framework
• 3D Model-based tracking: models modelled from
light stripe range data.
• Colour (selector), edges and texture (trackers) are
measured simultaneously in every frame.
• Measurements fused in Extended Kalman filter:
– Cues interact with state through measurement models
– Individual cues need not recover the complete pose
– Extensible to any cues/cameras for which a
measurement model exists.
Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics
14
Colour Cues
• Filter created from colour histogram in ROI:
– Foreground colours promoted in histogram
– Background colours supressed in histogram
Captured image used
to generate filter
Output of resulting filter
Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics
15
Edge Cues
Sobel mask
directional
edges
Combine with colour to get silhouette edges
Predicted
projected
edges
Fitted edges
Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics
16
Texture Cues
Rendered prediction
Feature detector
Outlier rejection
Matched templates
Final matched features
Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics
17
Tracking Result
Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics
18
Visual Servoing
• Position-based 3D visual servoing (IROS 2004).
• Fusion of visual and kinematic measurements.
Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics
19
Visual Servoing
• 6D pose of hand estimated using extended Kalman
filter with visual and kinematic measurements.
• State vector also includes hand-eye transformation
and camera model parameters for calibration.
Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics
20
Grasping Task
• Grasp a yellow box without prior knowledge
of objects in the scene.
Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics
21
Grasping Task
Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics
22
Pouring Task
• Pour the contents of a cup into a bowl.
Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics
23
Pouring Task
Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics
24
Smell Experiment
• Fusion of vision, smell and airflow sensing to
locate and grasp a cup containing ethanol.
Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics
25
Summary
• Integration of stereoscopic light stripe sensing,
geometric object modelling, multi-cue tracking
and visual servoing allows robot to perform ad
hoc tasks with unknown objects.
• Suggested directions for future research:
–
–
–
–
Integrate tactile and force sensing
Cooperative visual servoing of both arms
Interact with objects to learn and refine models
Verbal and gestural human-machine interaction
Taylor and Kleeman, Visual Perception and Robotic Manipulation, Springer Tracts in Advanced Robotics
26