EE565: Advanced Image Processing
Xin Li
LDCSEE, Fall 2009
dehaze
EE565 Advanced Image Processing
Copyright Xin Li 2009
1
EE565 General Information
• Lectures and office hours
Meeting Time: TTh 9:30-10:45 in MRB 107
Office Hours: Mondays 2:00-3:00pm in ESB 939
Fragment Minutes: 15 minutes before and after each lecture
• Contact information
Instructor: xin.li@mail.wvu.edu
For email submission of assignments, please use your MIX account
EE565 Advanced Image Processing
Copyright Xin Li 2008
2
• Prerequisites
EE465: Introduction to Digital Image Processing
or equivalent
• Follow-up (Spring 2010)
EE569: Digital Video Processing (more fun)
• Texts
No textbook is required. The instructor will provide lecture notes
at the course website
http://www.csee.wvu.edu/~xinl/courses/ee565/ee565.html
Additional material (e.g., classical papers, MATLAB demos,
assignments and solutions) will also be posted there
EE565 Advanced Image Processing
Copyright Xin Li 2008
3
• Working load
- 8 assignments
- One midterm and one final project
• Grading
Assignments 40%
Midterm project 30%
(Technical report 5% included)
Final project 30%
(Oral Presentation 5% included)
Auditing policy: you need to turn in all the assignments
EE565 Advanced Image Processing
Copyright Xin Li 2008
4
Importance of Hand-on Experience




Finishing all the assignments are
necessary preparation for working on
larger-size projects
Midterm project will be development
oriented (e.g., implementation of a
published algorithm or some simple idea
of your own)
Final project will be research oriented
(e.g., improve upon a published algorithm
or the idea you have tested in midterm)
Final project could lead to MS thesis or
PhD qualifier exam problem
EE565 Advanced Image Processing
Copyright Xin Li 2008
5
How to Do Scientific Research?

Inquiry (research)-based learning



Difference between taking exams and doing
research
Difference between textbook knowledge and
your own understanding
Competition and collaboration


Your classmates are your competitors
(grading will be ranking-based)
Your classmates are also your collaborators
(cooperation is at the foundation of all
engineering endeavors)
EE565 Advanced Image Processing
Copyright Xin Li 2008
6
Tools for Effective Learning
http://masterxinli.wordpress.com/category/teaching/ee565/
In addition to classroom interaction, Blog offers a
convenient platform for everyone to participate.
I might also try several other techniques:
Think-Pair-Share, Minute Paper and Group Discussion
The most important lesson I have learned through years:
Interest is the best instructor (everything I do is to try to
get you hooked to learning this course – so pls. tell me when
you feel bored)
EE565 Advanced Image Processing
Copyright Xin Li 2008
7
Course Overview

Mathematical modeling of images



Image restoration


Improve image quality and usability
Image communication


Why do we care about images?
Why do we take a mathematical approach?
Move images from here to there and from now
to then
Image analysis

Automatically extract information from images
EE565 Advanced Image Processing
Copyright Xin Li 2008
8
Technological Importance of Images

Improve Human’s vision capabilities




see far (e.g., watch Summer Olympics
in Beijing)
see small (e.g., microscopic structures
such as neurons and cells)
see through (e.g., ultrasound
inspection of pregnant women)
see better (e.g., in the darkness or
adversary environmental conditions).
EE565 Advanced Image Processing
Copyright Xin Li 2008
9
Scientific Reasons
Understanding how we see is the first step towards
understanding human intelligence
EE565 Advanced Image Processing
Copyright Xin Li 2008
10
D. Hubel’s “Eyes, Brain and Vision”
http://hubel.med.harvard.edu/bcontex.htm
EE565 Advanced Image Processing
Copyright Xin Li 2008
11
Neural Network View
“The Next Generation of Neural Networks”
http://www.youtube.com/watch?v=AyzOUbkUf3M
EE565 Advanced Image Processing
Copyright Xin Li 2008
12
Why Mathematical Modeling?

What is mathematical modeling?






A mathematical model uses
mathematical language to describe a
system
Linear vs. nonlinear
Deterministic vs. probabilistic
Static vs. dynamic
Homogeneous vs. heterogeneous
Philosophical considerations

Causality vs. Synchronicity
EE565 Advanced Image Processing
Copyright Xin Li 2008
13
Images of Favorite: Natural Images

What is natural images?



Why natural images?



No rigorous definition to the best of my
knowledge
Loosely speaking, images of natural scenes
acquired by CCD cameras (Others call
photographic images)
An important class of images with a variety of
applications (consumer electronics, biometrics,
entertainment)
A good representative with high modeling
complexity (arguably more challenging than
other class such as medical images)
Cautious note about model complexity
EE565 Advanced Image Processing
Copyright Xin Li 2008
14
To Understand Natural Images
Image Processing is also about Physics
EE565 Advanced Image Processing
Copyright Xin Li 2008
15
Natural Scenes
How many different objects can appear in natural scenes?
Countless – human faces, animals, buildings, mountains …
"Nature is not ecomonical of structures - only of
principles" -Abdus Salam
EE565 Advanced Image Processing
Copyright Xin Li 2008
16
Resolution Invariance
EE565 Advanced Image Processing
Copyright Xin Li 2008
17
Scale Dependency
0.1m
10m
EE565 Advanced Image Processing
Copyright Xin Li 2008
1m
100m
18
Self-Similarity: Fractals
EE565 Advanced Image Processing
Copyright Xin Li 2008
19
Impact of Illumination
Indoor
Example
EE565 Advanced Image Processing
Copyright Xin Li 2008
20
Outdoor example
EE565 Advanced Image Processing
Copyright Xin Li 2008
21
Story of “Lena” Image in USC Dataset
EE565 Advanced Image Processing
Copyright Xin Li 2008
22
From USC to JPEG2K
EE565 Advanced Image Processing
Copyright Xin Li 2008
23
The Space of Natural Images
textures
smooth
regions
contours
(Courtesy of Prof. SC Zhu at UCLA)
By analogy, the space of natural images is very much like our universe
EE565 Advanced Image Processing
Copyright Xin Li 2008
24
Challenge 1: Image Restoration

Practical limitation of image
acquisition system



Limited resolution (image size)
Inevitable blurring and noise
Distortion introduced by image
transmission


Wireless channel: fading errors
Internet: packet loss
If you work on communication, reliable communication of images
through wired or wireless channel is a long-standing open problem
EE565 Advanced Image Processing
Copyright Xin Li 2008
25
Image Denoising
denoising
algorithm
Y=X+W
^
X=f(Y)
W: additive white Gaussian noise
EE565 Advanced Image Processing
Copyright Xin Li 2008
26
Our Tasks

Understand classical Wiener filtering




Gaussian source, Gaussian noise
Theoretically optimal
How does wavelet-based denoising
work?
Why do statistical methods
outperform others (e.g., PDEbased)?
EE565 Advanced Image Processing
Copyright Xin Li 2008
27
Deblurring
deblurring
algorithm
^
X=f(Y|H)
Y=HX+W
H: linear blurring kernel
When H is unknown, it
becomes the notoriously
difficult blind image
Deconvolution problem
EE565 Advanced Image Processing
Copyright Xin Li 2008
28
Idea 1: Motion Deblurring
“Removing camera shake from a single image”
Presented at SIGGRAPH 2006, Boston
http://people.csail.mit.edu/fergus/research/deblur.html
EE565 Advanced Image Processing
Copyright Xin Li 2008
29
Where is Blur?
Easy for human eyes but difficult for computers
EE565 Advanced Image Processing
Copyright Xin Li 2008
30
Image Interpolation
interpolation
algorithm
EE565 Advanced Image Processing
Copyright Xin Li 2008
31
Superresolution
EE565 Advanced Image Processing
Copyright Xin Li 2008
32
Towards Gigapixel
3Mpel
1Gpel
Link 1
http://www.tawbaware.com/maxlyons/gigapixel.htm
Link 2
http://triton.tpd.tno.nl/gigazoom/Delft2.htm
EE565 Advanced Image Processing
Copyright Xin Li 2008
33
Idea 2: Barcode Superresolution
How to extract the 1D barcode information from a 2D image?
EE565 Advanced Image Processing
Copyright Xin Li 2008
34
Image Inpainting
Inpainting
Algorithm
EE565 Advanced Image Processing
Copyright Xin Li 2008
35
Application of Inpainting
EE565 Advanced Image Processing
Copyright Xin Li 2008
36
Inpainting in Image Communication:
Error Concealment
EE565 Advanced Image Processing
Copyright Xin Li 2008
37
Deblocking
JPEG compressed image
at low bit rate
Restored image after
post-processing
EE565 Advanced Image Processing
Copyright Xin Li 2008
38
Deringing
JPEG2000 compressed image
at low bit rate
Restored image after
post-processing
EE565 Advanced Image Processing
Copyright Xin Li 2008
39
Challenge II: Robust Image Coding
5Mpel camera: 3bytes per pixel, 15MB per image
512M memory: $40, $1 per image w/o compression
Memory will become less and less expensive (see next slide)
EE565 Advanced Image Processing
Copyright Xin Li 2008
40
Holographic Recording
Data
1011
1000
0110
0010



Dispersive channel
SLM Image
Detector Image
Recovered
Data
1011
1000
0110
0010



Channel
Courtesy of Kevin Curtis, InPhase Technologies
EE565 Advanced Image Processing
Copyright Xin Li 2008
41
Bandwidth is STILL COSTY
Do you know how much Sprint charges for wireless data?
EE565 Advanced Image Processing
Copyright Xin Li 2008
42
JPEG2000 vs. JPEG
JPEG2000
JPEG
Compression ratio is the same: 217
Online comparison demo:
http://www.aware.com/products/compression/j2kmaindemo.html
EE565 Advanced Image Processing
Copyright Xin Li 2008
43
Our Tasks

Why is wavelet coding better?




Properties of wavelet transforms
Statistical modeling of natural images
Importance of location uncertainty
How do we go beyond wavelet coding?



Image quality assessment
Rethink the role of bits (resolve location vs.
intensity uncertainty)
Biologically-inspired approaches
EE565 Advanced Image Processing
Copyright Xin Li 2008
44
Idea 3: Satellite Image Compression
Imagine you are in real-state business, don’t you want
to give your customers a virtual tour before a physical visit?
EE565 Advanced Image Processing
Copyright Xin Li 2008
45
Challenge III: Image Analysis
From low-level vision (image-in-image-out) to
high-level vision (image-in-information-out)
Automatic target recognition
EE565 Advanced Image Processing
Copyright Xin Li 2008
46
Feature Point Matching at Low-level
EE565 Advanced Image Processing
Copyright Xin Li 2008
47
Object Segmentation at Middle Level
EE565 Advanced Image Processing
Copyright Xin Li 2008
48
Image Retrieval at High Level
EE565 Advanced Image Processing
Copyright Xin Li 2008
49
Popular Demo: Face Detection
http://vasc.ri.cmu.edu/demos/faceindex/03282003/users/665.html
EE565 Advanced Image Processing
Copyright Xin Li 2008
50
Challenges with Face Detection
http://vasc.ri.cmu.edu/demos/faceindex/12182002/users/2622.html
EE565 Advanced Image Processing
Copyright Xin Li 2008
51
Idea 4: Face Image Indexing
How do we tell two
people look alike?
Eye
Face
mouth
nose
Q: Can we automatically sort out
face images based on their
perceptual similarities?
EE565 Advanced Image Processing
Copyright Xin Li 2008
52
Challenge IV: Image-related Security
EE565 Advanced Image Processing
Copyright Xin Li 2008
53
Image Forensics
Courtesy of Dr. H. Farid at Dartmouth:
http://www.cs.dartmouth.edu/farid/research/tampering.html
EE565 Advanced Image Processing
Copyright Xin Li 2008
54