CSE 473/573
Computer Vision and Image
Processing (CVIP)
Ifeoma Nwogu inwogu@buffalo.edu
Lecture 13 + 14 – Texture
1

Schedule
• Last class
– We finished local features
• Today
– Textures
• Readings for today: Forsyth and Ponce
Chapter 6.1 – 6.4
2

Texture
From F & P, used with permission

What is texture?
• Patterns of structure from
– changes in surface albedo (eg printed cloth)
– changes in surface shape (eg bark)
– many small surface patches (eg leaves on a bush)
• Hard to define; but texture tells us
– what a surface is like
– (sometimes) object identity
– (sometimes) surface shape

Texture and Material http://www-cvr.ai.uiuc.edu/ponce_grp/data/texture_database/samples/

Texture and Orientation http://www-cvr.ai.uiuc.edu/ponce_grp/data/texture_database/samples/

Texture and Scale http://www-cvr.ai.uiuc.edu/ponce_grp/data/texture_database/samples/

Texture: Core Problems
• Represent complex surface textures to recognize
– objects
– materials
– textures
• Synthesize texture from examples
– to create big textures for computer graphics
– to fill in holes in images caused by editing

Texture representation
• Core idea: Textures consist of
– a set of elements
– repeated in some way
• Representations
– identify the elements
– summarize the repetition

Notice how the change in pattern elements and repetitions is the main difference between different textured surfaces
(the plants, the ground, etc.)

Different materials tend to have different textures

Filter based texture representations
• Choose a set of filters, each representing a pattern element
– typically a spot and some oriented bars
• Apply all the filter the image at a variety of scales
• Rectify the filtered images
– typically half wave, to avoid averaging contrast reversals
• e.g. should not average dark spot on light background, light spot on dark background to zero
• Compute summaries of rectified filtered images
– e.g. smoothed average
– at a variety of scales to capture
• nearby pattern elements and general picture of pattern elements
• Now describe each pixel by vector of summaries
– which could be very long

How can we represent texture?
• Compute responses of blobs and edges at various orientations and scales

Overcomplete representation: filter banks
LM Filter Bank
Code for filter banks: www.robots.ox.ac.uk/~vgg/research/texclass/filters.html

Filter banks
• Process image with each filter and keep responses (or squared/abs responses)

Representing texture
• Take the vectors of filter responses at each pixel and cluster them, then take histograms

Texture representation
• Textures are made up of repeated local patterns, so:
– Find the patterns
• Use filters that look like patterns (spots, bars, raw patches…)
• Consider magnitude of response
– Describe their statistics within each local window
• Mean, standard deviation
• Histogram
• Histogram of “prototypical” feature occurrences

Texture representation: example
Win. #1 mean d/dx value
4 mean d/dy value
10 original image derivative filter responses, squared statistics to summarize patterns in small windows

Texture representation: example
Win. #1
Win.#2 mean d/dx value
4
18 mean d/dy value
10
7 original image derivative filter responses, squared statistics to summarize patterns in small windows

Texture representation: example
Win. #1
Win.#2 mean d/dx value
4
18 mean d/dy value
10
7 original image derivative filter responses, squared statistics to summarize patterns in small windows

Texture representation: example
Win. #1
Win.#2
Win.#9 mean d/dx value
4
18
20 20 mean d/dy value
10
7 original image derivative filter responses, squared statistics to summarize patterns in small windows

Texture representation: example
Win. #1
Win.#2
Win.#9 mean d/dx value
4
18
20 20 mean d/dy value
10
7
Dimension 1 (mean d/dx value) statistics to summarize patterns in small windows

Texture representation: example
Windows with primarily horizontal edges
Both
Win. #1 mean d/dx value
4 mean d/dy value
10
Win.#2 18 7
Win.#9 20 20
Dimension 1 (mean d/dx value)
Windows with small gradient in both directions
Windows with primarily vertical edges statistics to summarize patterns in small windows

Texture representation: example original image derivative filter responses, squared visualization of the assignment to texture
“types”

Texture representation: example
Far: dissimilar textures
Win. #1
Win.#9 mean d/dx value
4
18
20 20 mean d/dy value
10
7
Dimension 1 (mean d/dx value) statistics to summarize patterns in small windows

Example based texture representations
Q: how does one choose the filters?
• Responses to over-complete filter banks
– build histograms over filter responses
– describe textures using clusters of responses
• Alternative
– build a vocabulary of pattern elements from pictures
– describe textures using this vocabulary

Building a vocabulary
• There are 2 steps to building a pooled texture representation for texture in an image domain.
1. Build a dictionary representing the range of possible pattern elements, using a large number of texture patches
2. Vectorize the patches in images using the clusters learned

Building a dictionary
• Collect many training example textures
• Construct the vectors x for the relevant pixels; these could be a reshaping of a patch around the pixel or a vector of filter outputs computed at the pixel
• Obtain k cluster centers c for these examples
29

Representing an image domain
• For each relevant pixel pixel I in the image
– Compute the vector representation x i pixel for that
– Obtain j , the index of the cluster center c j to that pixel closest
– Insert j into a histogram for that domain
30

Clustering the examples
• For using k -means we can represent patches with
– intensity vector
– vector of filter responses over pixel/patch

Representing a region
• Vector Quantization
– Represent a high-dimensional data item with a single number
• Find the number of the nearest cluster center in dictionary
• Use that to represent the cluster
• Summarize the pattern of patches
– Cut region into patches
– Vector quantize - vector quantized image patches are sometimes called visual words
– Build histogram of resulting numbers

Texture representations
• A vector summarizing the trends in pattern elements
– either overall trend in filter responses
– or histogram of vector quantized patches
• At a pixel
– compute representations for domains centered on the pixel
• For a region
– compute representations the whole region

Texture synthesis
• Problem:
– Take a small example image of pure texture
– Use this to produce a large domain of “similar” texture
• Why:
– Computer graphics demands lots of realistic texture, hard to find
– Fill in holes in images created by removing objects
• Simple case:
– Assume we must synthesize a single pixel in a large image
– Approach:
• Match the window around that pixel to other windows in the image
• Choose a value from the matching windows
– most likely, uniformly and at random

Just like copying…
• but not just repetition
Photo
Pattern repeated
38

Approaches
Generate new examples of a texture.
• Original approach: Use the same representation for analysis and synthesis
– This can produce good results for random textures, but fails to account for some regularities
• Recent approach: Use an image of the texture as the source of a probability model
– This draws samples directly from the actual texture, so can account for more types of structure
– Very simple to implement
– However, depends on choosing a correct distance parameter
39

Other more recent approaches
• Matching methods
• Find another patch that looks a lot like the boundary hole
• Place that patch over the hole and blend (segmentation) the patch and hole together
• Boundary hole filling
• Coherence methods (applying cost function + constraints)
• Variational methods (for non-texture holes)
40

Efros and Leung method
• For each new pixel p (select p on boundary of texture):
– Match a window around p to sample texture, and select several closest matches
• Matching minimizes sum of squared differences of each pixel in the window (Gaussian weighted)
• Give zero weight to empty pixels in the window
– Select one of the closest matches at random and use its center value for p
– Size and shape of the neighborhood matter
41

Initial conditions for E & L
• If no initial conditions are specified, just pick a patch from the texture at random
• To fill in an empty region within an existing texture:
– Grow away from pixels that are on the boundary of the existing texture
42

Usefulness of boundary edges

Window size parameter
44

More results – window size
45

Image extrapolation
46

Image extension results…..
47

Failures
48

Fill in holes by looking for example patches in the image.
If needed, rectify faces (lower images).

State of the art in image fill-in is very good. This uses texture synthesis and other methods.

Texture: crucial points
• Texture is the result of repetition of pattern elements
• Representation of textures
– vocabulary of pattern elements
• either filters or visual words
– summary of how they repeat
• local averages of rectified filter responses or histograms
• Synthesis of texture
– Good, relatively simple procedures are available
• Texture reveals shape

Assignment 2???
Task:
Make the donkey vanish
Fill in the black hole using texture from the white box region
52

Slide Credits
• David A. Forsyth - UIUC
• Derek Hoeim – UIUC
• Jana Kosecka - GMU
53

Next class
• PCA (lectures by Devansh)
• Readings for next lecture:
– Please review your basic linear algebra
• Readings for today:
– Forsyth and Ponce 6.1 - 6.4
54

55