What is Vision?
Why is it Hard?
Alan L. Yuille.
UCLA.
The Purpose of Vision.
“To Know What is Where by Looking”.
Aristotle. (384-322 BC).
 Information Processing: receive a signal by
light rays and decode its information.
 Vision appears deceptively simple, but this
is highly misleading.

What can humans see?

Rich description: fox, tree trunk, grass and
background twigs.
 Also: shape of fox’s legs and head, its fur, what
it is doing? old or young? and more.
 Local regions (B,C,D) are highly ambiguous.
Local regions are ambiguous
Airplane
Car
Boat
Sign
Building
Why can Humans see so well?

Humans appear understand images effortlessly.
But this is only because of the enormous amount
of our brains that we devote to this task.
 It is estimated that 40-50% of neurons in the
cortex are involved in doing vision.
 Humans are very visual. We get much more
information from our eyes than other animals.
Vision and the Brain
Half the Cortex does Vision
But human vision is not perfect

Human perception is often incorrect.
 Visual illusions suggest that perception is
often a reconstruction, or even a controlled
hallucination.
 Here are some illusions.
Ames Room
Perspective Cues
Are these lines horizontal?
Which square is brighter?
Visual Illusions

The perception of brightness of a surface,
 or the length of a line depends on context.
 Not on basic measurements like:
 the no. of photons that reach the eye
 or the length of line in the image.
Perception is Inference

Helmholtz. 1821-1894.
 “Perception as Unconscious Inference”.
So why is vision hard?

Vision is an inverse inference problem.
There is a highly complex process that
generates the image.
 This process is studied in Computer
Graphics. It models objects, light sources,
and how they interact.
 Vision must invert this image formation
process to estimate the “causal factors” –
objects, lighting, and so on.
Vision as Inverse Inference:
Inverting image formation.
Inverse Problems are often hard

There are an infinite number of ways that
images can be formed.
 Which ways are most plausible?
.
Vision and Bayes
Bayes’ Theorem gives a procedure to solve
inverse inference problems.
 It states that we can infer the
state S of the world from the
observed image I by,

P(S|I) = P(I|S)P(S)/P(I).
 But doesn’t tell us how to
 Specify these distributions.

Complexity:
The Fundamental Problem?

The set of all images is almost infinite
(Kersten 1987).
 Estimate that only a tiny fraction of all
possible images have been seen by humans
(even small images 10x10).
 The no. of objects is large – 30,000?
 The no. of scene types is large – 1,000?
 Hence I and S are extremely complex.
Variability and Ambiguity

Variability: Images of similar objects
(bicycle) can be very different (left, center).
 Ambiguity: Images of different objects can
be identical (right).
Complexity, Variability, Ambiguity

What is the space of all possible images?
 It is only now becoming possible to explore
this questions – (technological advances).
 Large datasets – 20,000 images (Pascal),
1,000,000 (ImageNet).
 But are these datasets big enough for results
on them to apply to other images?
Datasets: Promise and Peril

Vision evaluates performance of algorithms
on benchmarked datasets.
 But results on these datasets may fail to
generalize to other images.
 Datasets need to be representative of the
complexity of the real world.
 Tyranny of Datasets. Unrepresentative
datasets. Limited visual tasks.
Dataset Examples
Pascal (top). 20,000 images. 20 objects.
ImageNet (bottom) 1,000,000 images. 1,000 objects.
Brief History of Vision

The difficulty of vision became clear in the
1960’s,70’s when Artificial Intelligence
researchers tried to get computer programs to
interpret images.
 Initially AI researchers thought vision was easy.
“Solve vision in a summer”.
 Researchers gradually started realizing that vision
was extremely hard. Much harder than Chess.
.
Big Picture Theories of Vision.
Work in the 1980’s in vision was largely
influenced by big picture theories.
e.g., David Marr’s Vision (1982).
 The theories were interesting, but practical
results were limited.
 Progress started being made by breaking up
vision into subproblems. Inventing or
borrowing mathematical tools for these
problems.

Techniques used in Vision

Multi-Disciplinary set of tools.
 Linear and Non-linear filtering.
 Variational Methods.
 Probabilities on Graphs.
 Perspective Geometry.
 Differential Geometry.
 And many more. We will introduce many of
these at this summer school.
Vision as a Bag of Tricks?

Dividing vision up into sub-problems
enabled progress to be made.
 But lead to a situation where vision was
seen as an enormous bag of tricks.
 “What papers should I read in computer
vision? There are so many, and they are so
different.” Chinese Graduate student.
 Attempts are being made – summer schools,
Wikivision project – to give a core and
foundation for vision,
The mathematical richness of
vision

The complexity and ambiguity of vision is a
challenge.
 Vision problems (broadly defined) have
attracted the attention of learning
mathematicians (e.g., Field’s medalists).
 David Mumford, Shing-Tung Yau, Maxim
Kontsevich, Stephen Smale, Terrence Tao.
Relationships to other
Disciplines

Computer vision relates closely to
 Image Processing
 Machine Learning.
 Partially covered in this school.
 Artificial Intelligence.
 Robotics.
 Study of Biological Vision Systems –
Psychology and Neuroscience.
Imaging Devices

In this school we will be dealing with
“natural images” of everyday scenes.
 But the methods used can be applied to
other imaging modalities.
 Medical images: fMRI, EEC,
 Astronomial Images:…
 Ever-increasing number of imaging devices.
Need a science of visual images.
Structure of the School

Vision can be broken down into low-, mid-,
and high-level vision (very roughly).
 Low-level vision – local image operations
which have limited knowledge of the world.
 Mid-level vision – non-local operations
which know about surfaces and geometry.
 High-level vision – operations which know
about objects and scene structures.
Low-level vision

Image processing.
 Filtering, denoising,
 enhancement.
 Edge detection.
 Image segmentation.
 Right: ideal segmentation
 (followed by labeling).
Mid-Level Vision

Estimation of 3D surfaces:
 Binocular stereo,
 structure from motion,
 other depth cues
 (e.g., perspective, shading,
 texture, focus).
 Fig: Image, Depth (blue to red),
 Segmentation.
Mid-Level Vision: Grouping

Kanisza. Humans have a strong tendency to
group image structures as surfaces.
High-Level Vision

Object detection and Scene Understanding.
 Example: detect objects in Pascal.
High-Level Vision: parts

Detect parts of objects:
High-Level Vision – AI.

Rich Explicit Representations enable:
 Understanding of objects, scenes, and events.
 Reasoning about functions and roles of objects, goals
and intentions of agents, predicting the outcomes of
events. SC Zhu – MURI.
Visual Turing tests

Vision algorithms that have similar
properties to those of humans:
 (i) flexible, adaptive, robust
 (ii) capable of learning from limited data,
ability to transfer,
 (iii) able to perform multiple tasks,
 (iv) closely coupled to reasoning, language,
and other cognitive abilities.
Grammars, Compositionality,
and Pattern Theory.

How to put everything together?
 How to address the complexity problem of
vision.
 Pattern theory offers suggestions:
 Mumford and Desolneux 2010.
 Describe the class of patterns that can
happen in images using grammars.
Grammars and Pattern Theory

Parse images by decomposing them into
elementary image patterns.
 Inverse inference on a grammar for
generating images.
The next few lectures

The next few lectures will concentrate on
basic low-level vision tasks:
 Filtering – linear and non-linear.
 Segmentation. Variational and probabilistic
methods.
Conclusion





Vision is difficult and challenging. Humans
devote half of their cortex to doing it.
It is difficult because it is inverse inference in
an extremely complex domain.
It involves a great variety of mathematical
and computational techniques.
Low-, Mid-, High-Level taxonomy.
Pattern Theory and Grammars.