Research Activities at
Computer Vision and Image Understanding Group
Florida State University
Xiuwen Liu
Florida State Vision Group
Department of Computer Science
Florida State University http://fsvision.cs.fsu.edu

Outline
 Motivations
•
Some applications of computer vision techniques
 Computer Vision and Image Understanding Group
 Some of the research projects
 Contact information

Introduction
 An image patch represented by hexadecimals

Introduction
- continued
 Fundamental problem in computer vision
•
Given a matrix of numbers representing an image, or a sequence of images, how to generate a perceptually meaningful description of the matrix?
– An image can be a color image, gray level image, or other format such as remote sensing images
– A two-dimensional matrix represents a signal image
– A three-dimensional matrix represents a sequence of images

A video sequence is a 3-D matrix

A movie is also a 3-D matrix

Introduction
- continued

Introduction
- continued
 Why do we want to work on this problem?
•
It is very interesting theoretically
– It involves many disciplines to develop a computational model for the problem
•
It is the key component to understand and model intelligence
– Note that 50% of the brain is devoted to vision
• It has many practical applications
–
Internet applications
– Movie-making applications
– Military applications

Computer Vision Applications
 No hands across America
• sponsored by Delco Electronics, AssistWare Technology, and Carnegie Mellon University
•
Navlab 5 drove from Pittsburgh, PA to San Diego, CA, using the RALPH computer program.
•
The trip was 2849 miles of which 2797 miles were driven automatically with no hands
–
Which is 98.2%

Computer Vision Applications
– continued

Computer Vision Applications
– continued
DARPA Grant Challenge: http://www.darpa.mil/grandchallenge/index.htm

Computer Vision Applications
– continued
 Military applications
•
Automated target recognition

Computer Vision Applications
– continued

Computer Vision Applications
– continued
 Extracted hydrographic regions

Computer Vision Applications
– continued
 Medical image analysis
•
Characterize different types of tissues in medical images for automated medical image analysis

Computer Vision Applications
– continued

Computer Vision Applications
– continued
 Biometrics
•
From faces, fingerprints, iris patterns .....
• It has many applications such as security, ATM withdrawal, credit card managements .....

Computer Vision Applications – cont.

Computer Vision Applications
– continued
 Content-based image retrieval has become an active research area to meet the needs of searching images on the web in a meaningful way
•
Color histogram has been widely used

Content-Based Image Retrieval – cont.

Vision-Based Image Morphing

Vision-Based Image Morphing
- continued

Computer Vision and Image Understanding Group
 Faculty: Xiuwen Liu, Anuj Srivastava, Washington
Mio, Eric Klassen
 Goals: Develop and implement effective image understanding algorithms and systems for images and videos from multi modalities including visible, infrared, and range sensors
 Approaches: Learning-based vision algorithms, statistical modeling of objects, computational modeling and analysis of textures, statistical modeling of shapes, stochastic optimization, inference algorithms on manifolds, and Bayesian inference

Research Projects
 The group offers a wide range of research possibilities
• Implementation projects
•
Development of new applications
•
Development of new algorithms
•
Theoretical and mathematical analysis of algorithms

Implementation Projects
 These projects involve implementing proven ideas and algorithms on specific datasets with specific interface and programming language constraints
•
For example, Haitao Wu implemented a graphical user interface for a face recognition algorithm we have as his Masters project
•
Yu Wang implemented a web-based interface for a content-based image retrieval algorithm

A Real-time Recognition/Tracking System

Content-based Image Retrieval
Image Query System by Yu Wang

Future Implementation Possibilities
 Implement a Java-based system for face detection
 Implement a Java-based system for learning
 Implement and improve web-based systems for content-based image and video retrieval

Generic Image Modeling
 How can we characterize all these images perceptually?

Spectral Histogram Representation
 Spectral histogram
•
Given a bank of filters F ( a
) , a = 1, …,
K , a spectral histogram is defined as the marginal distribution of filter responses
I
( a
)
( v )

F
( a
)
* I( v )
H
I
( a
) ( z )

1
| I |
 v
δ ( z

I ( a
) ( v ))
H
I

( H
I
( 1 )
, H
I
( 2 )
,  , H
I
( K )
)

Spectral Histogram Representation
- continued
 Choice of filters
•
Laplacian of Gaussian filters
•
Gabor filters
• Gradient filters
• Intensity filter
LoG filter Gabor filter

Spectral Histogram Representation
- continued

A Texture Synthesis Example
 A white noise image was transformed to a perceptually similar texture by matching the spectral histogram
Average spectral histogram error

Texture Synthesis Examples
- continued
Observed image
 A random texture image
Synthesized image

Texture Synthesis Examples
- continued
Observed image Synthesized image
 An image with periodic structures

Texture Synthesis Examples
- continued
Mud image
Synthesized image
 A mud image with some animal foot prints

Texture Synthesis Examples
- continued
Observed image
Synthesized image
 A random texture image with elements

Object Synthesis Examples


As in texture synthesis, we start from a random image
In addition, similar object images are used as boundary conditions in that the corresponding pixel values are not updated during sampling process

Object Synthesis Examples
- continued

Object Synthesis Examples
- continued

Principal Component Analysis

Eigen Values of 400 Eigen Vectors

Principal Component Analysis
- continued
Original Image Reconstructed using 50 PCs
Reconstructed using 200 PCs

Principal Component Analysis
- continued

Principal Component Analysis
- continued

Difference Between Reconstruction and Sampling
Reconstruction is not sufficient to show the adequacy of a representation and sampling from the set of images with same representation is more informational

Face detection based on spectral representations
 Face detection is to detect all instances of faces in a given image
 Each image window is represented by its spectral histogram
•
A support vector machine is trained on training faces
•
Then the trained support vector machine is used to classify each image window in an input image
 More results at http://fsvision.fsu.edu/face-detection

Face detection
- continued

Face detection
- continued

Face detection
- continued

Rotation invariant face detection

Rotation invariant face detection
- continued

Linear representations
 Linear representations are widely used in appearancebased object recognition applications
• Simple to implement and analyze
•
Efficient to compute
•
Effective for many applications a
( I , U )

U
T
I

R d

Standard Linear Representations
 Principal Component Analysis
•
Designed to minimize the reconstruction error on the training set
• Obtained by calculating eigenvectors of the co-variance matrix
 Fisher Discriminant Analysis
• Designed to maximize the separation between means of each class
• Obtained by solving a generalized eigen problem
 Independent Component Analysis
• Designed to maximize the statistical independence among coefficients along different directions
• Obtained by solving an optimization problem with some object function such as mutual information, negentropy, ....

Standard Linear Representations
- continued
 Standard linear representations are sub optimal for recognition applications
• Evidence in the literature [1][2]
•
A toy example
– Standard representations give the worst recognition performance

Proposed Approach
 Optimal Component Analysis (OCA)
•
Derive a performance function that is related to the recognition performance
•
Formulate the problem of finding optimal representations as an optimization one on the Grassmann manifold
•
Use MCMC stochastic gradient algorithm for optimization

Performance Measure
 It must have continuous directional derivatives
 It must be related to the recognition performance
 It can be computed efficiently
 Based on the nearest neighbor classifier
• However, it can be applied to other classifiers as it forms clusters of images from the same class that far from clusters from other classes
• See an example for support vector machines

Performance Measure
- continued
 Suppose there are C classes to be recognized
• Each class has k train training images
•
It has k cross cross validation images

Performance Measure
- continued

 h is a monotonically increasing and bounded function
•
We used h(x) = 1/(1+exp(-2 b x)
• Note that when b  
, F(U) is exactly the recognition performance using the nearest neighbor classifier
Some examples of F(U) along some directions

Performance Measure
- continued
 F(U) depends on the span of U but is invariant to change of basis
• In other words, F(U)=F(UO) for any orthonormal matrix
O
• The search space of F(U) is the set of all the subspaces, which is known as the Grassmann manifold
– It is not a flat vector space and gradient flow must take the underlying geometry of the manifold into account; see [3] [4] [5] for related work

Deterministic Gradient Flow
- continued
 Gradient at [J]
(first d columns of n x n identity matrix)

Deterministic Gradient Flow
- continued
 Gradient at U:
Compute Q such that QU=J
 Deterministic gradient flow on Grassmann manifold

Stochastic Gradient and Updating Rules
 Stochastic gradient is obtained by adding a stochastic component
 Discrete updating rules

MCMC Simulated Annealing Optimization Algorithm
 Let X(0) be any initial condition and t=0
1.
Calculate the gradient matrix A(X t
)
2.
Generate d(n-d) independent realizations of w ij
’s
3.
Compute Y (X t+1
) according to the updating rules
4.
Compute F(Y) and F(X t
) and set dF=F(Y)- F(X t
)
5.
Set X t+1
= Y with probability min{exp(dF/D
6.
Set D t+1
= D t
/ g and set t=t+1 t
),1}
7.
Go to step 1

The Toy Example
 The following result on the toy example shows the effectiveness of the algorithm
•
The following figure shows the recognition performance of Xt and
F(Xt)

ORL Face Dataset

Experimental Results on ORL Dataset
 Here the size of image is 92 x 112, d = 5 (subspace)
• Comparison using gradient, stochastic gradient, and the proposed technique with different initial conditions
PCA ICA FDA

Results on ORL Dataset
- continued
 With respect to d and k train d=3 k train
=5 d=10 k train
=5 d=20 k train
=5 d=5 k train
=1 d=5 k train
=2 d=5 k train
=8

Results on CMU PIE Dataset
 Here we used part of the CMU PIE dataset
•
There are 66 subjects
•
Each subject has 21 pictures under different lighting conditions
-X0=PCA
-d=10
-X0=ICA
-d=10
-X0=FDA
-d=5

Some Comparative Results on ORL
 Comparison where performance on cross validation images is maximized
•
In other words, the comparison is to show the best performance linear representations can achieve
•
PCA – black dotted; ICA – red dash-dotted ;
FDA – green dashed ; OCA – blue solid

Some Comparative Results on ORL
- continued
 Comparison where the performance on the training is optimized
• In other words, it is a fair comparison
• PCA – black dotted; ICA – red dash-dotted ;
FDA – green dashed ; OCA – blue solid

PROBABILITY MODELS FOR IMAGE ANALYSIS
Empirical Studies Indicate Patterns
Histogram of x-derivative
Need models that:
• are low-dimensional (computationally tractable)
• are accurate models of (real) observed clutter
• support the observed patterns

BESSEL K FORM
A Parametric Family:
K is the modified Bessel function of third kind
•Image statistics (under spectral decompositions) exhibit non
Gaussian statistics.
•This density explains the non-Gaussian and heavy-tail nature of observed image statistics.
•The parameters p and c are easily estimated from the data using sample variance and kurtosis.
•This model is derived from first principles.

Original Image
MODELING SUCCESS
Gabor Filter
Filtered Image
Observed Bessel K
Statistics of Filtered Image

SHAPE ANALYSIS
•Represent shapes as elements of infinite-dimensional manifolds
•Analyze shapes using geometry of that manifold
-- connect shapes using geodesic paths on the manifold
-- quantify shape differences using geodesic lengths
-- compute shape statistics ( mean , variance)
•Applications:
-- clustering of objects according to shapes ( learning )
-- shape based recognition of objects ( recognition )
-- predicting shapes of partially-obscured objects ( completion )

Basic Idea: Given two shapes (far left and far right), we connect them using a geodesic path on the shape manifold.
Example
First Shape
Second Shape
Eight shapes along geodesic path
Fish shapes taken from Surrey database

MEAN SHAPES
Four Sample Shapes
Their Mean Shape

CLUSTERING OF SHAPES
Results: 7 resulting clusters, each row is a cluster

3D Model-Based Recognition

Medical Image Analysis
 Advances in medical imaging provide many new opportunities and challenges for computer vision research
 Automated medical image analysis

Medical Image Analysis
- continued

Medical Image Analysis
- continued

Medical Image Analysis
- continued

Medical Image Analysis
- continued

Video Sequence Analysis and Summary
 Motion analysis based on correspondence
 Video stream-based surveillance
 Video summary

Courses
 Most Relevant Courses
• CAP 5638 Pattern Recognition (Spring 2004)
•
CAP 5415 Principles and Algorithms of Computer Vision
– Fall 2004
• CAP 6417 Theoretical Foundations of Computer Vision
– STA 5106 Computational Methods in Statistics I
– STA 5107 Computational Methods in Statistics I I
–
Seminars and advanced studies
 Related Courses
• CAP 5615 Artificial Neural Networks
•
CAP 5600 Artificial Intelligence
• CAP 5xxx Machine Learning

CAP 5638 Pattern Recognition
 It will be offered Spring 2004
• Tuesday and Thursday 6:45-8:00 PM
•
At Love 103
•
The course ref #: 07842
• http://www.cs.fsu.edu/~liux/courses/cap5638/
 It will cover
• The basics for pattern recognition
–
Neural networks
• Machine learning algorithms
• Applications in data mining, pattern discovery, artificial intelligence, and security,
 It should be interesting to anyone interested in more intelligent computer learning algorithms

Funding of the Group
 National Science Foundation
•
DMS
•
CISE IIS
• FRG
• ACT
 National Imagery and Mapping Agency
•
NGA – National Geo-spatial Intelligence Agency
 Army Research Office

Summary
 Florida State Vision group offers many interesting research topics/projects
• Efficient represent for generic images
•
Computational models for object recognition and image classification
•
Medical image analysis
•
Motion/video sequence analysis and modeling
•
They are challenging
• They are interesting

Contact Information
•
•
•
•
• Name Xiuwen Liu
Web site at http://fsvision.fsu.edu
http://www.cs.fsu.edu/~liux
Email at liux@cs.fsu.edu
Office at LOV 166
Phone 644-0050