ENEE631 Spring’09
Lecture 27 (5/11/2009)
Texture Analysis and Synthesis
Spring ’09 Instructor: Min Wu
Electrical and Computer Engineering Department,
University of Maryland, College Park
 bb.eng.umd.edu (select ENEE631 S’09)
 minwu@eng.umd.edu
M. Wu: ENEE631 Digital Image Processing (Spring'09)
Overview and Logistics
UMCP ENEE631 Slides (created by M.Wu © 2004)

Last Time:
– Multi-dimension Lattice sampling
– Sampling rate conversion and applications in video processing

Today:
– Texture analysis and synthesis
– More discussions on image modeling

Project presentation: Thursday May 21, 2009, 11am Kim 2211
– 15min; Arrange to have all members in the team speak; Practice

See course webpage for guides on writing & presentation

Course evaluation (online)
M. Wu: ENEE631 Digital Image Processing (Spring'09)
Lec 27 – Texture & Feature Analysis [2]
From Wang’s book
preprint Fig.4.4
Recap: Sampling Lattice Conversion
Original
Intermediate
Targeted
UMCP ENEE631 Slides (created by M.Wu © 2001)
M. Wu: ENEE631 Digital Image Processing (Spring'09)
Lec 27 – Texture & Feature Analysis [3]
Video Format Conversion for NTSC  PAL
UMCP ENEE631 Slides (created by M.Wu © 2001)

Require both temporal and spatial rate conversion
– NTSC 525 lines per picture, 60 fields per second
– PAL 625 lines per picture, 50 fields per second

Ideal approach (direct conversion)
– 525 lines 60 field/sec  13125 line 300 field/sec
 625 lines 50 field/sec

4-step sequential conversion
– Deinterlace => line rate conversion
=> frame rate conversion => interlace

Simplified conversion
– 50 field/sec  60 field/sec: deinterlace, then simplify to 5 6 frames
– 625 lines  525 lines: simplify to 25 lines  21 lines
– Conversion involves two adjacent lines or frames only
M. Wu: ENEE631 Digital Image Processing (Spring'09)
Lec 27 – Texture & Feature Analysis [4]
UMCP ENEE631 Slides (created by M.Wu © 2001)
From Wang’s book preprint Fig.4.9
M. Wu: ENEE631 Digital Image Processing (Spring'09)
Lec 27 – Texture & Feature Analysis [5]
UMCP ENEE631 Slides (created by M.Wu © 2001)
From Wang’s book preprint
M. Wu: ENEE631 Digital Image Processing (Spring'09)
Lec 27 – Texture & Feature Analysis [7]
UMCP ENEE631 Slides (created by M.Wu © 2004)
Texture Analysis & Synthesis
M. Wu: ENEE631 Digital Image Processing (Spring'09)
Lec 27 – Texture & Feature Analysis [8]
Texture

Observed in structural patterns
of objects’ surfaces
– [Natural] wood, grain, sand,
grass, tree leaves, cloth
– [Man-made] tiles, printing patterns

“Texture” ~ repetition of “texels”
(basic texture element)
– Texels’ placement may be
periodic, quasi-periodic, or random
From http://texlib.povray.org/textures.html
and Gonzalez 3/e book online resource
M. Wu: ENEE631 Digital Image Processing (Spring'09)
Lec 27 – Texture & Feature Analysis [10]
Properties and Major Approaches to Study Texture

Texture properties: smoothness, coarseness, regularity

Structural approach
– Describe arrangement of basic image primitives

Statistical approach
– Examine histogram and other features derived from it
– Characterize textures as smooth, coarse, grainy

Spectral and random field approach
– Exploit Fourier spectrum properties
– Detect global periodicity
M. Wu: ENEE631 Digital Image Processing (Spring'09)
Lec 27 – Texture & Feature Analysis [11]
Statistical Measures on Textures
Figures from Gonzalez’s
book resource
x ~ r.v. of pixel value
R = 1 – 1 / ( 1 + x2 )
~ 0 for constant region;
1 for large variance
E[ (X – x)K ]
3rd moment:
~ histogram’s skewness
4th moment:
~ relative flatness
Uniformity or Energy
~ squared sum of hist. bins
M. Wu: ENEE631 Digital Image Processing (Spring'09)
Lec 27 – Texture & Feature Analysis [12]
Characterizing Textures

Structural measures
– Periodic textures


Features of deterministic texel: gray levels, shape, orientation, etc.
Placement rules: period, repetition grid pattern, etc.
– Textures with random nature


Features of texel: edge density, histogram features, etc.
Stochastic/spectral measures
– Mainly for textures with random nature; Model as a random field

2-D sequence of random variables
– Autocorrelation function: measuring the relations among those r.v.
R(m,n; m’,n’) = E[ U(m,n) U(m’,n’) ]
“wide-sense stationary”: R(m,n;m’n’) = RU(m-m’,n-n’) and constant mean
– Fit into random field models ~ analysis and synthesis


Focus on second order statistics for simplicity
Two textures with same 2nd order statistics often appear similar
M. Wu: ENEE631 Digital Image Processing (Spring'09)
Lec 27 – Texture & Feature Analysis [13]
Examples:
Spectral Approaches
to Study Texture
Figures from Gonzalez’s 2/e book resource
M. Wu: ENEE631 Digital Image Processing (Spring'09)
Lec 27 – Texture & Feature Analysis [14]
Image examples from Bertalmio et al. TIP’03 paper
Texture Synthesis
Recall: error concealment of small blocks
Exploit surrounding edge info. to interpolate
edge
estimation
edge-directed
interpolation

General approach: analysis then synthesis

Image In-painting
– Filling in missing/occluded regions with synthesized version


Maintain structural consistency (in edge & overall color/brightness)
Maintain texture’s statistical continuity (such as oscillation pattern)
for improved visual effect
Ref: M. Bertalmio, L. Vese, G. Sapiro, and S. Osher: “Simultaneous Structure and
Texture Image Inpainting,” IEEE Trans. on Image Proc., vol.12, no.8, August 2003.
M. Wu: ENEE631 Digital Image Processing (Spring'09)
Lec 27 – Texture & Feature Analysis [15]
Figures from Bertalmio et al. TIP’03 paper
Image Inpainting:
Basic Approach
and Example
M. Wu: ENEE631 Digital Image Processing (Spring'09)
Lec 27 – Texture & Feature Analysis [16]
Recall: Characterize the Ensemble of 2-D Signals
UMCP ENEE631 Slides (created by M.Wu © 2004)

Specify by a joint probability distribution function
– Difficult to measure and specify the joint distribution for images of
practical size
=> too many r.v. : e.g. 512 x 512 = 262,144

Specify by the first few moments
– Mean (1st moment) and Covariance (2nd moment)


may still be non-trivial to measure for the entire image size
By various stochastic models
– Use a few parameters to describe the relations among all pixels


E.g. 2-D extensions from 1-D Autoregressive (AR) model
Important for a variety of image processing tasks
– image compression, enhancement, restoration, understanding, …
M. Wu: ENEE631 Digital Image Processing (Spring'09)
Lec 27 – Texture & Feature Analysis [18]
Recall: Discrete Random Field
UMCP ENEE631 Slides (created by M.Wu © 2004)

We call a 2-D sequence discrete random field if each of its
elements is a random variable
– when the random field represents an ensemble of images, we often
call it a random image

Mean and Covariance of a complex random field
E[u(m,n)] = (m,n)
Cov[u(m,n), u(m’,n’)] = E[ (u(m,n) – (m,n)) (u(m’,n’) – (m’,n’))* ]
= ru( m, n; m’, n’)


For zero-mean random field, autocorrelation function = cov. function
Wide-sense stationary (or wide-sense homogeneity)
(m,n) =  = constant
ru( m, n; m’, n’) = ru( m – m’, n – n’; 0, 0) = r( m – m’, n – n’ )

also called shift invariant, spatial invariant in some literature
M. Wu: ENEE631 Digital Image Processing (Spring'09)
Lec 27 – Texture & Feature Analysis [19]
UMCP ENEE631 Slides (created by M.Wu © 2004)
Recall: Special Random Fields of Interests

White noise field
– A stationary random field
– Any two elements at different locations x(m,n) and x(m’,n’)
are mutually uncorrelated
rx( m – m’, n – n’) = x2 ( m, n ) ( m – m’, n – n’ )

Gaussian random field
– Every segment defined on an arbitrary finite grid is Gaussian
i.e. every finite segment of u(m,n) when mapped into a
vector have a joint Gaussian p.d.f. of
M. Wu: ENEE631 Digital Image Processing (Spring'09)
Lec 27 – Texture & Feature Analysis [20]
Recall: Spectral Density Function
UMCP ENEE631 Slides (created by M.Wu © 2004)

Spectral density function (SDF) is defined as the Fourier
transform of the covariance function rx

S (1 , 2 ) 



  r (m, n)  exp[  j ( m   n)]
m   n  
x
1
2
Also known as the power spectral density (p.s.d.)
Example: SDF of stationary white noise field with r(m,n)= 2 (m,n)
 S( 1, 2) = 2

SDF Properties:
– Real and nonnegative: S(1, 2) = S*(1, 2); S(1, 2)  0


By conjugate symmetry of covariance function: r(m, n) = r *(-m, -n)
By non-negative definiteness of covariance function
– SDF of the output from a LSI system w/ freq response H(1, 2)
Sy(1, 2) = | H(1, 2) |2 Sx(1, 2)
M. Wu: ENEE631 Digital Image Processing (Spring'09)
Lec 27 – Texture & Feature Analysis [21]
More on Image Modeling

Good image model can facilitate many image proc. tasks
– Coding/compression; restoration; estimation/interpolation; …

Image models we’ve seen/used so far
– Consider pixel values as realizations of a r.v.

Color/grayscale histogram
– Predictive models

Use linear combination of (causal) neighborhood to estimate
– Random field u(m,n)


Characterized by 2-D correlation function or p.s.d.
Generally can characterize u(m,n) = u’(m,n) + e(m,n)
u’(m,n) is some prediction of u(m,n); e(m,n) is another random field



Minimum Variance representation (MVR): e(m,n) is error of min. var. prediction
White Noise Driven representation: e(m,n) is chosen as a white noise field
ARMA representation: e(m,n) is a 2-D moving average of a white noise field
M. Wu: ENEE631 Digital Image Processing (Spring'09)
Lec 27 – Texture & Feature Analysis [22]
Recall: Linear Predictor
UMCP ENEE631 Slides (created by M.Wu © 2001)

Causality required for coding purpose
– Can’t use the samples that decoder hasn’t got as reference

Use last sample uq(n-1): equiv. to coding the difference (DPCM)
 pth–order
p
auto-regressive (AR) model u (n)   ai  u(n  i)  e(n)
i 1
– Linear predictor from past samples

Prediction neighborhood
– Line-by-line DPCM

predict from the past samples in the same line
– 2-D DPCM

predict from past samples in the same line and from previous lines
– Non-causal neighborhood


Use samples around as prediction/estimation => for filtering, restoration, etc
Predictor coefficients in MMSE sense:
get from orthogonality condition (from Wiener filtering discussions)
M. Wu: ENEE631 Digital Image Processing (Spring'09)
Lec 27 – Texture & Feature Analysis [23]
Commonly Used Image Model

Gaussian model and Gaussian mixture model
Every segment defined on an arbitrary finite grid is Gaussian
-or- follows a distribution of linear combining several Gaussian

Reduce the modeling to estimate mean(s), variance(s), and weightings
(Ref: Prof. R. Gray’s IEEE talk S’07 http://www-ee.stanford.edu/~gray/umcpqcc.pdf)

Markov random field
– Markovianity: conditional independence


Define past, present, future pixel set for each pix location
Given the present, the future is independent of the past
– 2-D spatial causal AR model (under Gaussian noise or MVR)

Gauss-Markov random field model
– Gaussian: conditional independence => conditional uncorrelateness

Bring in multi-scale and wavelet/multi-resolution ideas
Ref: Section 4.1-4.5 of Bovik’s Handbook
M. Wu: ENEE631 Digital Image Processing (Spring'09)
Lec 27 – Texture & Feature Analysis [24]
Recursive Estimation: Basic Ideas

Kalman filter: recursive Linear MMSE estimator

1-D example: provide linear estimation for an AR signal
– System model



AR(M): x(n)=c1x(n-1)+… cM x(n-M) + w(n)
State equation: x(n) = C x(n-1) + w(n) for n = 0, 1, …, x(-1)=0 ~ AR sig
Observation equation: y(n) = hT x(n) + v(n)
M-dimension state vector x(n);
Model noise w(n) and observation noise v(n) are white & orthogonal
– Signal model is Mth-order Markov under Gaussian noise w
– Linear MMSE estimator is globally optimal if model and
observation noise w and v are both Gaussian
– C can be time-variant and physics motivated by applications

General MMSE solution: equiv. to find conditional mean


Filtering estimate: E[ x(n) | y(n), y(n-1), … y(0) ]  xa (n)
One-step predictor estimate: E[ x(n) | y(n-1), … y(0) ]  xb(n)
M. Wu: ENEE631 Digital Image Processing (Spring'09)
Lec 27 – Texture & Feature Analysis [26]
Recursive Estimation (cont’d)

1-D Kalman filter equations


Prediction: xb(n) = C xa(n-1)
Update: xa(n) = xb(n) + g(n) [ y(n) – hT xb(n) ]
– Error variance equations
Pb(n) = C Pa(n-1) CT + Qw
Pa(n) = (I – g(n) hT ) Pb(n)
initialize: x(0) = [w(0), 0, … 0 ]T, xb(0) = 0
Pb(0) = Qw ~ all zero except 1st entry w2
– Kalman gain vector g(n)


g(n) = Pb(n) (hT Pb(n) h + v2) -1
2-D Kalman filtering: define proper state vector


Raster scan observations & map to equiv. 1-D case
Restrict Kalman gain terms to be just surround current
observation to reduce computational complexity
M. Wu: ENEE631 Digital Image Processing (Spring'09)
past
present state
future
Lec 27 – Texture & Feature Analysis [27]
UMCP ENEE631 Slides (created by M.Wu © 2004)
Summary of Today’s Lecture

Texture analysis and synthesis

More on image modeling

Readings
– Texture: Gonzalez’s book 11.3.3; see also Jain’s 9.11
– Image modeling:


Wood’s book Chapter 7 & 9.4; Jain’s book Chapter 6;
Bovik’s Handbook 4.1-4.5
Recursive/Kalman estimation:
Woods’ book Chapter 7; EE621 (Poor’s book)
M. Wu: ENEE631 Digital Image Processing (Spring'09)
Lec 27 – Texture & Feature Analysis [28]
UMCP ENEE631 Slides (created by M.Wu © 2004)
Beyond ENEE631

ENEE633 Statistical Pattern Recognition

ENEE731 Image Understanding

Audio/Speech: ENEE632

Adaptive algorithms: ENEE634

Special-topic research-oriented courses (sometimes offered)
– On medical image processing
– On media security & forensics
M. Wu: ENEE631 Digital Image Processing (Spring'09)
Lec 27 – Texture & Feature Analysis [29]
M. Wu: ENEE631 Digital Image Processing (Spring'09)
Lec 27 – Texture & Feature Analysis [30]