Statistics of Natural Image
Categories
Antonio Torralba and Aude Oliva. Network: Computation in Neural Systems, 14(2003) 391-412
Jonathan Huang (jch1@cs.cmu.edu)
1/30/2006
Spatial Image Signatures
Averaged pictures of categories of objects, scenes and objects in scenes, computed
with 100 exemplars or more per category. Exemplars were chosen to have the same basic level
and viewpoint in regard to an observer. The group objects in scenes (third row) represent
examples of the averaged peripheral information around an object centered in the image.
Caltech 101

Object Categories:









Airplanes
Brain
Brontosaurus
Chandelier
Garfield
Kangaroo
Octopus
Trilobyte
Etc…
By Antonio Torralba
100 Special Moments (and Conan)
Conan O’Brien
Newlyweds
Little Leaguer
By Jason Salavon
Power Spectra of Natural Images

Fourier Transform:

Magnitude Spectrum:

Spectral Signature (for a set of images S):
“Demo”

Computing the Spectrum (Matlab):


Ifft = abs(fftshift(fft2(I,w,h)));
Visualization:


imshow(log(Ifft)/max(max(log(Ifft))));
colormap(cool);
FFT(Beach)
FFT(Pittsburgh)
1/f Spectra

Natural Image Spectra follow a power law!


As() is called the Amplitude Scaling Factor
2-() is the Frequency Exponent.  clusters
around 0 for natural images.

Any guesses on why this law holds?
Main Idea of Torralba/Oliva Papers

“Statistics of Natural Images vary as a function of
the interaction between the observer and the
world”!
Some examples
Spectral Signatures

Why are Fields, Beaches and Coasts less isotropic than other
natural environments?
Scene Scale


“The point of view that any given observer adopts on a specific
scene is constrained by the volume of the scene.”
How does the amount of clutter vary against scene scale in manmade environments? In natural environments?
PCA on Natural Images
Top Row: PCA on images
Bottom Row: PCA on power spectra
Openness/Naturalness
Projection of images onto the second and third principle components.
SPC2 corresponds to “Openness” and SPC3 corresponds to “Naturalness”
Spatially Localized Statistics

Windowed FFT
Top Row: Man-made environments
Bottom Row: Natural environments

Image statistics become non-stationary as scene scale increases.
What do Images Statistics say about
Depth?
V: Vertical
H: Horizontal
O: Oblique
Comparing Localized Spectral
Signatures and Depth

With increasing depth comes:



An increase in global roughness for man-made structures
A decrease in global roughness for natural structures
Nonuniformity in spatially localized spectral signatures
Examples (man-made)
Examples (Natural)
An Algorithm


First use PCA to reduce dimension
Goal: For a vector of features v, estimate


Since the man-made and natural cases should be treated
separately, we model f(v|art) and f(v|nat) as Gaussians.
The joint distribution is modeled as a weighted sum of
Gaussians:
where
Model Estimation

Expectation-Maximization

E-step:


Compute posterior probabilities of the clusters given
observed data.
M-step:

Update cluster parameters, weighting training data by the
posterior probabilities from the E-step.
Read Torralba and Oliva: Depth Estimation, IEEE PAMI 2002 for the
update equations. They do not all fit on one slide...
Some Results
f(D|category)
Distribution of Scene Categories as a function of mean depth.
Application: Scale Selection
Context in Images

Question: How can these small people possibly affect the image
statistics in any significant way??
Object Detection
Thank You

References





Torralba and Oliva, Statistics of Natural Image Categories. Network:
Computation in Neural Systems 14 (2003) 391-412.
Torralba and Oliva, Depth Estimation from Image Structure. IEEE PAMI Vol 14,
No. 9 (2002).
Oliva and Torralba, Modeling the Shape of the Scene: A Holistic Representation
of the Spatial Envelope. IJCV 42(3), 145-175 (2001).
Srivastava, Lee, Simoncelli, Zhu, On Advances in Statistical Modeling of Natural
Images. JMIV 18:17-33 (2003)
Mumford, Pattern Theory: the Mathematics of Perception. ICM 2002. Vol III. 1-3