Computer Vision, Part 1
Topics for Vision Lectures
1. Content-Based Image Retrieval (CBIR)
2. Object recognition and scene
“understanding”
Content-Based Image Retrieval
Example: Google “Search by Image”
Basic technique
Extract Features
(Primitives)
Matched
Results
Similarity
Measure
Query Image
Image
Database
Features Database
Relevance
Feedback
Algorithm
From http://www.amrita.edu/cde/downloads/ACBIR.ppt
Each image in database is represented by a feature vector: x1, x2, ...xN, where xi = (xi1, xi2, …, xim)
Query is represented in terms of same features: Q =(Q1, Q2, …, Qm)
Goal: Find stored image with vector xi most similar to query vector Q
• Distance measure:
x i  Q  x1iQ1  x2iQ2  ... xm iQm
• Possible distance measures d(Q, xi):

– Inner (dot) product
– Histogram distance (for histogram features)
– Graph matching (for shape features)
.
.
.
Some issues in designing a CBIR system
• Query format, ease of querying
• Speed
• Crawling, preprocessing
• Interactivity, user relevance feedback
• Visual features — which to use? How to combine?
• Curse of dimensionality
• Indexing
• Evaluation of performance
Types of Features Typically Used
• Intensities
• Color
• Texture
• Shape
• Layout
http://www.clear.rice.edu/elec301/Projects02/
artSpy/intensity.html
Intensity histograms
Color Features
Hue, saturation, value
Color Histograms (8 colors)
http://www.owlnet.rice.edu/~elec301/Projects02/artSpy/patmac/mcolhist.gif
http://www.owlnet.rice.edu/~elec301/Projects02/artSpy/patmac/mcolhist.gif
Color auto-correlogram
• Pick any pixel p1 of color Ci in the image I.
• At distance k away from p1 pick another pixel p2.
• What is the probability that p2 is also of color Ci?
Red ?
k
p2
p1
Image: I
From: http://www.cse.ucsc.edu/classes/ee264/Winter02/xgfeng.ppt
• The auto-correlogram of image I for color Ci , distance k:
 C(k ) (I )  Pr[| p1  p2 | k, p2  IC | p1  IC ]
i
i
i
• Integrates both color information and space information.
From: http://www.cse.ucsc.edu/classes/ee264/Winter02/xgfeng.ppt
Two images with their autocorrelograms. Note that the
change in spatial layout would be ignored by color
histograms, but causes a significant difference in the
autocorrelograms.
From http://www.cs.cornell.edu/rdz/Papers/ecdl2/spatial.htm
From:
http://www.cs.cornell.ed
u/rdz/Papers/ecdl2/spati
al.htm
Color histogram rank: 411; Auto-correlogram rank: 1
Color histogram rank: 310; Auto-correlogram rank: 5
Color histogram rank: 367; Auto-correlogram rank: 1
Texture representations
• Gray-level co-occurrence
• Entropy
• Contrast
• Fourier and wavelet transforms
• Gabor filters
Texture Representations
Each image has the same intensity distribution, but different textures
Can use auto-correlogram based on intensity (“gray-level co-occurrence”)
Texture from entropy
Images filtered by entropy:
Each output pixel contains entropy value of 9x9 neighborhood
around original pixel
From: http://www.siim2011.org/abstracts/advanced_visualization_tools_ss_pao.html
Texture from fractal dimension
From:
http://www.cs.washington.edu/homes/rahul/data/ic
cv07.pdf
Texture from Contrast
• Example:
http://www.clear.rice.edu/elec301/Projects02/artSpy/grainin
ess.html
Texture from Wavelets
http://www.clear.rice.edu/elec301/Projects02/
artSpy/dwt.html
http://www.clear.rice.edu/elec301/Projects02/
artSpy/dwt.html
Shape representations
Some of these need segmentation (another whole story!)
• Area, eccentricity, major axis orientation
• Skeletons, shock graphs
• Fourier transformation of boundary
• Histograms of edge orientations
From: http://www.lems.brown.edu/vision/researchAreas/ShockMatching/shock-ed-match-results1.gif
Histogram of edge orientations
From: http://www.cs.ucl.ac.uk/staff/k.jacobs/teaching/prmv/Edge_histogramming.jpg
Visual abilities largely missing from current CBIR
systems
• Object recognition
• Perceptual organization
• Similarity between semantic concepts
Examples of “semantic” similarity
Image 1
Image 2
Examples of “semantic” similarity
Image 1
Image 2
Examples of “semantic” similarity
Image 1
Image 2
“In general, current systems have not yet had
significant impact on society due to an
inability to bridge the semantic gap between
computers and humans.”
Image Understanding and AnalogyMaking
Bongard problems as an idealized domain
for exploring the “semantic gap”