Perceptual
and
Sensory Augmented
Visual Object
Recognition
Tutorial Computing
Visual Object Recognition
Bastian Leibe
Computer Vision Laboratory
ETH Zurich
Chicago, 14.07.2008
&
Kristen Grauman
Department of Computer Sciences
University of Texas in Austin
???
?
Perceptual
and
Sensory Augmented
Visual Object
Recognition
Tutorial Computing
Identification vs. Categorization
2
K. Grauman, B. Leibe
Perceptual
and
Sensory Augmented
Visual Object
Recognition
Tutorial Computing
Object Categorization
• How to recognize ANY car
• How to recognize ANY cow
K. Grauman, B. Leibe
3
Perceptual
and
Sensory Augmented
Visual Object
Recognition
Tutorial Computing
Object Categorization
• Task Description

“Given a small number of training images of a category, recognize
a-priori unknown instances of that category and assign the correct
category label.”
• Which categories are feasible visually?

Extensively studied in Cognitive Psychology,
e.g. [Brown’58]
“Fido”
German
shepherd
dog
K. Grauman, B. Leibe
animal
living
being
4
Perceptual
and
Sensory Augmented
Visual Object
Recognition
Tutorial Computing
Visual Object Categories
• Basic-level categories in humans seem to be defined
predominantly visually.
• There is evidence that humans (usually)
start with basic-level categorization
before doing identification.
 Basic-level categorization is easier
and faster for humans than object
identification!
 Most promising starting point
for visual classification
…
animal
Abstract
levels
…
…
quadruped
…
Basic level
dog
German
shepherd
Individual
level
K. Grauman, B. Leibe
…
cat
cow
Doberman
“Fido”
…
5
Perceptual
and
Sensory Augmented
Visual Object
Recognition
Tutorial Computing
Other Types of Categories
• Functional Categories

e.g. chairs = “something you can sit on”
K. Grauman, B. Leibe
6
Perceptual
and
Sensory Augmented
Visual Object
Recognition
Tutorial Computing
Other Types of Categories
• Ad-hoc categories

e.g. “something you can find in an office environment”
K. Grauman, B. Leibe
7
Perceptual
and
Sensory Augmented
Visual Object
Recognition
Tutorial Computing
Levels of Object Categorization
“cow”
“car”
“motorbike”
• Different levels of recognition



Which object class is in the image?  Obj/Img classification
Where is it in the image?
 Detection/Localization
Where exactly ― which pixels?
 Figure/Ground
segmentation
K. Grauman, B. Leibe
8
Perceptual
and
Sensory Augmented
Visual Object
Recognition
Tutorial Computing
Challenges: robustness
Illumination
Occlusions
Object pose
Intra-class
appearance
K. Grauman, B. Leibe
Clutter
Viewpoint
Perceptual
and
Sensory Augmented
Visual Object
Recognition
Tutorial Computing
Challenges: robustness
• Detection in Crowded Scenes

Learn object variability
– Changes in appearance, scale, and articulation

Compensate for clutter, overlap, and occlusion
K. Grauman, B. Leibe
10
Perceptual
and
Sensory Augmented
Visual Object
Recognition
Tutorial Computing
Challenges: context and human experience
K. Grauman, B. Leibe
Perceptual
and
Sensory Augmented
Visual Object
Recognition
Tutorial Computing
Challenges: context and human experience
Context cues
Image credit: D. Hoeim
Dynamics
Video credit: J. Davis
Perceptual
and
Sensory Augmented
Visual Object
Recognition
Tutorial Computing
Challenges: scale, efficiency
• Thousands to millions of pixels in an image
• Estimated 30 Gigapixels of image/video content generated
•
•
•
•
•
•
per second
About half of the cerebral cortex in primates is devoted to
processing visual information [Felleman and van Essen
1991]
3,000-30,000 human recognizable object categories
30+ degrees of freedom in the pose of articulated objects
(humans)
Billions of images indexed by Google Image Search
18 billion+ prints produced from digital camera images in
2004
295.5 million camera phones sold in 2005
K. Grauman, B. Leibe
Perceptual
and
Sensory Augmented
Visual Object
Recognition
Tutorial Computing
Challenges: learning with minimal supervision
More
Less
K. Grauman, B. Leibe
Perceptual
and
Sensory Augmented
Visual Object
Recognition
Tutorial Computing
Rough evolution of focus in recognition research
1980s
1990s to early 2000s
K. Grauman, B. Leibe
Currently
Perceptual
and
Sensory Augmented
Visual Object
Recognition
Tutorial Computing
Outline
1. Detection with Global Appearance & Sliding Windows
2. Local Invariant Features: Detection & Description
3. Specific Object Recognition with Local Features
― Coffee Break ―
1. Visual Words: Indexing, Bags of Words Categorization
1. Matching Local Features
1. Part-Based Models for Categorization
1. Current Challenges and Research Directions
K. Grauman, B. Leibe
16
Perceptual
and
Sensory Augmented
Visual Object
Recognition
Tutorial Computing
Detection via classification: Main idea
Basic component: a binary classifier
Car/non-car
Classifier
No,Yes,
notcar.
a car.
K. Grauman, B. Leibe
Perceptual
and
Sensory Augmented
Visual Object
Recognition
Tutorial Computing
Detection via classification: Main idea
If object may be in a cluttered scene, slide a window
around looking for it.
Car/non-car
Classifier
K. Grauman, B. Leibe
Perceptual
and
Sensory Augmented
Visual Object
Recognition
Tutorial Computing
Detection via classification: Main idea
Fleshing out this
pipeline a bit more, we
need to:
1. Obtain training data
2. Define features
3. Define classifier
Training examples
Car/non-car
Classifier
Feature
extraction
K. Grauman, B. Leibe