16-899A Pixels to Percepts
Instructors:
• Alexei (Alyosha) Efros, 225 Smith Hall, CMU
• .Lavanya Sharan, Disney Research Pittsburgh
Web Page:
• http://graphics.cs.cmu.edu/courses/P2P/
Today
Instructors
Why This Course?
Why Perception?
Administrative stuff
Overview of the course
Old Lady or Young Girl?
Lavanya Sharan
Postdoctoral Researcher
Computer Vision
Human Visual Perception
Computer Graphics
Visual perception, both behavioral and computational aspects.
Light or dark?
Material category?
RECENT WORK
Emotion?
Real or fake Oreo?
GRADUATE
WORK
Where does he land?
Alexei “Alyosha” Efros
Associate Professor, Robotics & CSD
Ph.D 2003, from UC Berkeley (signed by Arnie!)
Research Fellow, University of Oxford, ’03-’04
Research Interests
Computer Vision, Computer Graphics, Data-driven
Approaches
PhD Thesis on Texture and Action Synthesis
Smart Erase button in Microsoft Digital Image Pro:
Why This Course?
A perfect storm:
• I wanted to do this for a long time
• but felt I wasn’t qualified
• Lavanya happened to be in Pittsburgh
• and she is more than qualified
• The first-ever textbook on Perception for CS
types (Thompson et al) is coming out
• We will be giving out printouts soon
• There are great perception courses in Psych
dept (e.g. by Klatzky), but our aims are different:
• The aim to be much more applied
• There are real problems in vision/graphics that we can solve
• Exposure for these who might never take perception otherwise
Why study Perception?
• It’s fascinating!
• Ok, but so is astronomy
• It’s still in the “early days”
• Relatively easy to get up to speed
• Still lots of low-hanging fruit
OK, OK, but why should I care?
• I just want to create a seeing robot
• and/or create the next “Avatars”
Two answers: 1) a classic one and 2) a real one
Classic Answer
Understanding Human Visual Perception should
help us design better vision algorithms
• After all, human is the only vision system known to work
Just like understanding how birds fly should
help us design airplanes
• …WAIT A MINUTE
• Ok, so there is a fine line between “helping” and “mindless
copying”
• Beware of papers claiming to be “biologically inspired”
The Real Answer
Human Perception is an integral part of most
vision and graphics endeavors
i.e. you can’t get away from it!
“What does it mean, to see? The plain man's
answer (and Aristotle's, too). would be, to know
what is where by looking.”
-- David Marr, Vision (1982)
Vision: a split personality
“What does it mean, to see? The plain man's answer (and
Aristotle's, too). would be, to know what is where by looking.
In other words, vision is the process of discovering from images
what is present in the world, and where it is.”
depth map
Answer #1: pixel of brightness 243 at position (124,54)
…and depth .7 meters
Answer #2: looks like bottom edge of whiteboard
showing at the top of the image
Which answer is better?
Is the difference just a matter of scale?
Measurement vs. Perception
Measurement vs. Perception
Proof!
Measurement vs. Perception
Pablo Picasso
The Guitar Player
Vision as Measurement
Real-time stereo on Mars
Physics-based Vision
Structure from Motion
Virtualized Reality
Vision as Understanding
• Object / Scene Recognition
• Action / Activity Recognition
• even scene geometry
Perception in Graphics
• One of central goals of graphics is to
simulate visual experience as realistically
as possible
• So, if we could simulate the physics of light
perfectly, why bother with perception?
1) Practical Reasons
2) Artistic Reasons
Complexity
•Amazingly real
•But so sterile, lifeless… Why?
Device Limitations
• Low Dynamic Range
• Limited modalities (visual + audio)
• No smell, taste, temperature
• Limited field of view
Artistic Reasons
• Reality is boring
• Visual content
creators (film
directors, game
designers) want
hyper-reality
How to study Perception?
Ok, you convinced me: Perception is useful.
But why not just “open up the brain” and figure
out how it works?
Two most popular methods in physiology:
• Cell recordings
• Recordings from small number of cells (1-100, out of
millions)
• fMRI
• Overall activity in the brain as function of time
• Very low spatial resolution
Tale of Martians with an old PC
Cell
recordings
fMRI
Instead, in this course…
Course Goals
Read some interesting papers together
• Learn something new: both you and us!
Get up to speed on big chunk of perception
research
Use perception / human studies in your own
research
Try your hand at a perception study
Learn how to speak
Learn how think critically about papers
Course Motto: 50/50
50% Lectures / 50% student presentations
50% textbook / 50% research papers
50% from perception side / 50% from
vision/graphics side
50% Alyosha / 50% Lavanya
Course Organization
Requirements:
1. Class Participation (25%)
• Keep annotated bibliography
• Post on the Class Blog before each class
• Ask questions / debate / flight / be involved!
2. Three Projects (75%)
• Two Analysis Projects (25% + 25%)
• Implement / Evaluate a paper and present it in class
• 1 CS paper, 1 perception paper
• Synthesis (Final) Project (25%)
• Produce a publishable result (study or implementation)
• Can be continuation of analysis project or something
new
• Can be done solo or in groups of 2
Class Participation
Keep annotated bibliography of readings (always a good
idea!). The format is up to you. At least, it needs to
have:
• Summary of key points
• A few Interesting insights, “aha moments”, keen observations,
etc.
• Weaknesses. Unanswered questions. Areas of further
investigation, improvement.
Before each class:
• Submit your summary for current readings in
hard copy (printout/xerox)
• Submit a comment on the Class Blog
• ask a question, answer a question, post your thoughts, praise,
criticism, start a discussion, etc.
Analysis Project
1. Pick a paper / set of papers from the list
2. Understand it as if you were the author
•
•
•
e.g. Re-implement it; try to replicate the study
If there is code, understand the code completely
Run it on data the same data (you can contact authors for data and
even code sometimes)
3. Understand it better than the author
•
•
•
•
Run it on LOTS of new data (e.g. LabelMe dataset, Flickr dataset,
etc, etc), or try to replicate the study with new data
Figure out how it succeeds, how it fails, where it fails, and, most
importantly WHY it fails
Look at which parts of the code do the real work, and which parts
are just window-dressing
Maybe suggest directions for improvement.
4. Prepare an amazing 45 min presentation
•
Discuss with us twice – once when you start the project, and 3
days before the presentation
Synthesis Project
Can grow out of analysis project, or your own
research
1-2 people per project.
Project proposals mid-semester
Project presentations at the end of semester.
Results presented as a CVPR-format paper.
Hopefully, a few papers may be submitted to
CVPR / VSS etc
End of Semester Awards
We will vote for:
• Best Analysis Project
• Best Synthesis Project
• Best Blog Comment 
Prize: dinner in a French restaurant in Paris
(transportation not included!) or some other
worthy prizes
Course Outline (cont.)
•
•
•
•
•
•
Faces (week 11)
Visual Attention (week 12)
Motion Perception (week 13)
Visual Realism (week 14)
Perception and Art (week 15)
Final Project Presentations
Subject to Change
Perception & Art
Livingstone