Visual Recognition and Search

Visual Recognition and Search

Rogerio Feris, Jan 23, 2014

EECS 6890 – Topics in Information Processing

Spring 2014, Columbia University http://rogerioferis.com/VisualRecognitionAndSearch2014

Introduction

Instructors:

Rogerio Feris http://rogerioferis.com

Liangliang Cao http://researcher.watson.ibm.co

m/researcher/view.php?person= us-liangliang.cao

Visual Recognition And Search

Jun Wang http://researcher.watson.ibm.co

m/researcher/view.php?person= us-wangjun

Columbia University, Spring 2014

Introduction

1970s

Early Days of Computer Vision

Visual Recognition And Search Columbia University, Spring 2014

Introduction

First Digital Camera (1975)

 0.01 Megapixels

 23 seconds to record a photo to cassette


Introduction

 Datasets with 5 or 10 images

 Large-Scale Experiment: 800 photos (Takeo Kanade Thesis,

1973)


Introduction

Today

Visual Data is Exploding!


Introduction

Announcement of Pope Benedict in 2005


Introduction

Announcement of Pope Francis in 2013

Rapid proliferation of mobile devices equipped with cameras


Introduction

Era of Big Visual Data

 Billions of cell phones equipped with cameras

 ~500 billion consumer photos are taken each year world-wide

~500 million photos taken per year in NYC alone

 Hundreds of millions of Facebook photo uploads per day


Introduction


Introduction

 How to annotate large amounts of visual data?

 Crowdsourcing: Amazon Mechanical Turk

 ImageNet: Millions of Annotated Images , Thousands of Categories http://www.image-net.org/


Introduction

ImageNet 2012 Challenge

Impressive results obtained by deep convolutional networks


Introduction http://horatio.cs.nyu.edu/


Introduction

Exciting Time for Computer Vision

 + DATA

 + Computational Processing

 + Advances in Computer Vision and Machine Learning


Introduction

Plan for Today

 Visual Recognition and Search: Applications & Challenges

 Student Introductions

 Course Overview and Requirements

 Syllabus Tour

 Cool Data for Projects


Applications

Applications

Visual Recognition and Search


Applications

Self-Driving Cars

 Object Recognition and

Tracking – mostly 3D point clouds (laser scanner)

 Passed Driver’s license test in

Nevada!

Check Sebastian Thrun short TED talk: http://www.ted.com/talks/sebastian_thrun_google_s_driverless_car.html

CVPR 2012 talk with more details: http://techtalks.tv/talks/self-driving-cars/56391/


Applications

Driver Assistance http://www.mobileye.com/

Collision Warning

Automatic Head Light Control

Lane Departure Control etc.


Check interesting talk by Amnon Shashua: http://www.youtube.com/watch?v=cg5SNoTF3Ms


Applications

Seeking the strongest rigid detector (CVPR 2013)

Rodrigo Benenson, Markus Mathias, Tinne Tuytelaars, Luc Van Gool


Applications

Mars Exploration Rover


 Autonomous Navigation

 Highly textured environment – good for stereo!

 Terrain Classification

 Automatic Photo capture of interesting scenes

 Horizontal velocity estimation in landing – feature tracking


Applications

Gaming

 Kinect Camera: RGB + Depth sensor

 Human pose Detection and Gesture Recognition

 See J. Shotton et al, Real-Time Human Pose Recognition in Parts from a Single

Depth Image, CVPR 2011 (Best paper award)


Watch for Abnormal Events!

Applications

What did you notice in the video?

 People walking?

 Vehicles driving?

 Abandoned bag???


Applications


Applications

Surveillance http://www.today.com/video/today/51630165/#51630165


Applications

People Search in Surveillance Videos


Applications

Video Demo: http://rogerioferis.com/demos/PeopleSearch_BlurredFaces.wmv


Applications

Boston Bombing Event

 1071 detected faces from 50 high-res Boston images (all from Flickr)

Ability to spot a person with e.g., a white hat in a crowded scene

Suspect #1 found in 4 images in top 8 results Suspect #2 found in 3 images in top page


Applications


Cashier Fraud Detection:

Faked Scans (aka

Sweethearting fraud)

Giving-away of merchandise without charge to a "sweetheart" customer (e.g., friend, family, fellow employee) by faked product scan

Q. Fan et al, Recognition of

Repetitive Sequential Human

Activity, CVPR 2009


Like a normal field guide…

 that you can search and sort

 and with visual recognition

See N. Kumar et al,

"Leafsnap: A Computer

Vision System for

Automatic Plant Species

Identification, ECCV 2012

 Nearly 1 million downloads

 40k new users per month

 100k active users

 1.7 million images taken

 100k new images/month

 100k users with > 5 images

 Users from all over the world

 Botanists, educators, kids, hobbyists, photographers, …

Slide Credit: Neeraj Kumar

Applications

Google Goggles

Amazon


Applications

Auto-focus by Face Detection http://www.rogerioferis.com/demos/Face

TrackerRealTime.avi

TAAZ Virtual Makeover http://www.taaz.com/


Applications

IntraFace Demo

(http://www.humansensing.cs.cmu.edu/intraface/)


Applications

Finding Visually Similar Objects


Applications

Exploring Community Photo Collections http://phototour.cs.washington.edu/


Applications

Organizing Photo Collections – IBM IMARS http://researcher.watson.ibm.com/researcher/view_project.php?id=877


Applications

Many More Applications…

Medical Imaging

Biometrics

Handwritten Digit Recognition

And so on ….


Challenges

Does it really work?

Many problems still remain to be solved…


Challenges


Slide Credit: Kristen Grauman


Break


Student Introductions

Introductions

Student Info Form (see website: Homework #0)

Send by Monday, Jan 27


Course Overview

Course Goals:

 Understand and analyze the state-of-the-art in visual recognition and search.

 Hands-on experience with cutting-edge methods for visual recognition and search.

Note: 7:00pm – 8:50pm (3 credits)


Course Overview

Pre-requisites:

1) Courses/Background in Computer Vision or

Image Processing is mandatory.

2) Programming skills in C/C++ or Matlab are also required.

Please drop the class if you don’t satisfy these requirements


Course Overview

Course Format:

 Each class will consist of:

 Lecture by instructor

 Student presentations and discussion

 Paper Reviews

 Solo (no groups)

 Project

 Final project report: Paper (4-8 pages)


Course Overview

Grading:

 Class Participation (10%)

 Paper Presentations (20%)

 Paper Reviews (20%)

 Term Project (50%)


Course Overview

Resources:

 Check course webpage for:

 Publicly Available Source Code

 Datasets

 Related Courses


Course Overview

Presentations:

 Students will be divided in groups and will have to prepare one or two presentations in class (list of presentation topics will be provided soon).

 More information about the format and length/time of the presentations will be available in the course webpage.


Course Overview

Presentations:

 Presentation slides will have to be sent to instructors at least one day before the presentation.

 Grading based on clarity, organization, knowledge, and whether the presentation meets the time constraints.

 Check http://www.slideshare.net/cameraculture/how-to-givea-good-talk


Course Overview

Paper Reviews:

 Individual evaluation (no groups)

 Each review (1-2 pages) should address paper strengths, weaknesses, quality of experiments, and describe ways in which the paper could be extended.

 Check the course webpage for more details


Course Overview

Projects:

 Groups of two or three depending on the total number of students attending the class.

 Students are expected to spend significant time on their projects

 Project Proposal, Project Update I, Project Update II, Final

Presentation, and Final Project Report (4-8 pages)


Course Overview

Projects:

 Projects will be graded according to the following criteria:


Syllabus Tour

Syllabus Tour

Broad set of topics; focus on the state-of-the-art

Check the schedule and required reading for each class in the course webpage


Syllabus Tour

Part I: From Low-level to Semantic

Visual Representations


Syllabus Tour

Low-Level Representation: Feature

Detection and Description

Classical Descriptors: SIFT and SURF

Modern Descriptors for Real-Time Applications: FAST, BRISK, …

Check ECCV 2012 Tutorial on Modern Features: https://sites.google.com/site/eccv12features/slides


Syllabus Tour

Mid-level feature coding and pooling

Classical Bag-of-Word Models

Spatial Pyramid Matching


Modern higher-level representations:

Fisher vector and super-vector encoding, LLC, …

[K. Chatfield et al, The devil is in the details: an evaluation of recent feature encoding methods, BMVC 2011]


Syllabus Tour

Encoding Structure: Part-based Models

Poselets (Bourdev et al)

Structureless Rigid

Deformable Part Models (Felzenszwalb et al)


Check ICCV 2013 Tutorial on Partbased models for Recognition: http://www.cs.berkeley.edu/~rbg/ICC

V2013/


Syllabus Tour

High-level Representation: Attributes and

Semantic Features

Output of Semantic Classifiers as Features, Zero-shot Learning, …

Check CVPR 2013 Tutorial on Attributes: https://filebox.ece.vt.edu/~parikh/attributes/


Syllabus Tour

Part II: Tools for Large-scale Image

Classification and Retrieval


Syllabus Tour

Large-Scale Classification

Insights on how to handle huge amounts of data and categories

Deep Convolutional Neural Networks


Check NIPS 2013 Tutorial on Deep Learning: http://cs.nyu.edu/~fergus/presentations/nips2013_final.pdf


Syllabus Tour

Fast Indexing and Image Retrieval

 Metric Learning

 Indexing via Learning-based Hashing


Database


Syllabus Tour

Active Learning


Syllabus Tour

Part III: Case Studies


Syllabus Tour

Case Studies

IBM Smart Vision Suite (SVS) IBM Multimedia Analysis and

Retrieval System (IMARS)


Syllabus Tour

NOT Covered:

 Basic Machine learning algorithms. We assume you are familiar with e.g., SVMs, …

 Computer vision fundamentals. We assume you are familiar with e.g., edge detection, …


Citizen Science Projects

Cool Data for Projects


Cool Data for Projects

Snapshot Serengeti http://www.youtube.com/watch?v=AUL03ivS8bY http://www.snapshotserengeti.org/



5 Terabytes of annotated data



Example Project:

 Build a web demo for fine-grained categorization of animals, using state-of-the-art visual learning techniques

Top classification results and associated confidence:

1. Zebra (0.98)

2. Striped Hyena (0.01)

3. Leopard (0.01)

 We expect a comprehensive experimental analysis for each project


Past 2013 Projects

 Check the previous year project presentations


How to Invent:

See http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010


Next Steps

 Don’t forget to submit Homework #0 (Student Info

Form) if you are taking this class

 Background in computer vision + coding skills in

Matlab or C/C++ are strict requirements for taking the course

 If you don’t satisfy these requirements, please drop the class as soon as possible, as it will help us to know the final number of students in class.

Thank you!


Visual Recognition and Search

1970s

Early Days of Computer Vision

Today

Visual Data is Exploding!

ImageNet 2012 Challenge

Exciting Time for Computer Vision

Plan for Today

Applications