INF 5300
Data fusion for image analysis
Anne Solberg (anne@ifi.uio.no)
Today:
•Plans for INF 5300
•Background
•Multisensor data registration
•Multisensor image classification
•Multitemporal image analysis
•Multiscale image analysis
F1 06.2.08
INF 5300
1
Organization of INF 5300
• Selected topics from image analysis and pattern
recognition.
• Preliminary plan:
• 1-2 new topics will be added
• Exercises will be given for some of the topics.
6.2.08
Data fusion
13.2.08
Thresholding
20.2.08
Moments
27.2.08
Active contour models
5.3.08
Active contour models
2.4.08
Contextual classification
9.4.08
Support Vector Machines
F1 06.2.08
INF 5300
2
Organization of INF 5300
• Teachers: Anne Solberg, Fritz Albregtsen plus 1-2 guest
lecturers.
• Exam: Probably oral exam.
• Recommended background: INF 4300/INF 2310
• Curriculum: mixture of papers and book chapters.
• 1 mandatory exercise - select project in cooperation
with the teachers. Two options:
– Select one of a predefined set of projects related to the
lectures.
– Define a project related to your master/Ph.D. project in
cooperation with the teachers.
– Time schedule:
• Choose project: March 15
• Submit project report: May 1
F1 06.2.08
INF 5300
3
Multisensor image classification
Goal:
• Classify a scene into different classes or identify a
certain object in a scene based on images from
different sensors.
– Commonly done is several steps:
• Segmentation or other preprocessing
• Feature extraction
• Classification
F1 06.2.08
INF 5300
4
Sensors and media
• Remote sensing:
– Electromagnetic signals, from visible to
thermal and infrared and microwave
energy
– Sensors: Optical, microwave, thermal,
laser altimeter
• Medical image analysis:
– Ultrasound: acoustical signals
– Magnetic Resonnance Imaging
(MR)/CT
– PET/SPECT
– X-ray
• Sonar imaging:
– Multifrequency echosounders
F1 06.2.08
INF 5300
5
What does the term ”multi” mean?
• ”Multi” is used for several cases:
– Multisensor images (e.g. Optical and radar)
• Multifrequency
• Multimodal images (from different modalities like MR and CT)
– Multitemporal images
– Multiscale images
– Multipolarization images (for radar images only)
F1 06.2.08
INF 5300
6
Multifrequency imaging
• Multispectral and hyperspectral sensors provide
information on different frequencies. Different targets
are sensitive to certain frequencies.
– Imaging at several frequencies>separating more detalied
classes
• This applies to other types of signals, too.
– Multifrequency echosounders used for fishery
F1 06.2.08
INF 5300
7
Multitemporal data
• By imaging on object at different times, more
information can be derived.
– Discriminate between objects based on their temporal
variation (e.g. vegetation classification)
– Get better classification or detection (e.g. Medical
ultrasound)
• More observations can reduce the uncertainty.
– Reduce the effect of noise or cloud coverage in parts of the
image.
• Challenges:
– Sensor calibration/weather changes
– Changes in the classes occurr
• A new road is built, trees are cut
F1 06.2.08
INF 5300
8
Multisensor data
• Each sensor has it own characteristics
–
–
–
–
–
–
–
Noise level
Reliability
Ability to discriminate certain object types
Resolution
Availability/Coverage
Limitations due to e.g. weather/clouds
Imaging geometry and geometrical errors
F1 06.2.08
INF 5300
9
Example – multisensor image
Two radar images of an
oil spill in the Baltic sea
taken a few hours apart
(Envisat and Radarsat)
F1 06.2.08
INF 5300
10
Multisensor data registration
• Data registration is a pre-requisite for data fusion
• Data registration can be simple if the data are
georeferenced (map co-ordinates are known)
– Registration is then just resampling the images to a common
map projection
• An image matching step is often necessary to obtain
subpixel accuracy in the co-registration.
F1 06.2.08
INF 5300
11
Image registration – an overview
The main steps in a full image registration process are:
1. Feature extraction
2. Feature matching
3. Transformation selection
4. Image resampling
F1 06.2.08
INF 5300
12
Registration step 1: feature extraction
• Purpose: extract regions, edges, contours etc. that
can be used to represent tie-points in the set of
images.
• Tie-points: characteristic points that can be identified
in all images.
• Area-based methods:
– Match the grey levels in the images directly using statistical
measures.
• Feature-based methods:
– Define a set of useful features (e.g. edges, lines,
intersections etc.) tailored to the application.
– Extract these features using either spatial-domain methods
or frequency-domain methods (Fourier- or wavelets)
F1 06.2.08
INF 5300
13
Registration step 2: feature matching
• Purpose: establish the correspondence between the
tie-points found during step 1.
• Area-based methods:
– Match using correlation, Fourier-transform methods or
optical flow.
• Feature-based methods:
– Perform the matching in the Fourier-domain using the
equivalence between correlation and multiplication in the
Fourier-domain.
• Optical flow methods:
– Estimate the relative motion between the two images.
• This is commonly used in video analysis.
F1 06.2.08
INF 5300
14
Registration step 3:
Transformation selection
• Purpose: find the mapping function that describes
the relation between the positions of the tie-points in
the different images.
• Affine tranforms are often used.
Affine transformasjoner (lineære):
x’=a0x+a1y+a2
y’=b0x+b1y+b2
Kvadratiske transformasjoner (kalles warping):
x’=a0x2+a1y2+a2xy+a3x+a4y+a5
y’=b0x2+b1y2+b2xy+b3x+b4y+b5
F1 06.2.08
INF 5300
15
Registration step 4:
Image resampling
• Purpose: compute the new pixel co-ordinates in the
co-registered images and interpolate the pixel values.
• Transforming the co-ordinates will give non-integer
co-ordinates. Interpolation is used to combine pixel
values from neighboring pixels.
• Normally, either nearest-neighbor interpolation or
bilinear interpolation is used.
• See INF2310 for details of affine transforms and
interpolation.
http://www.ifi.uio.no/~inf2310/v2006/forelesninger/20060228_geometriskeOperasjoner.pdf
F1 06.2.08
INF 5300
16
Fusion at different levels
The data can be fused at different level:
• Signal-based fusion
– Fuse the signals prior to image formation
• Pixel-based fusion
– Merge the multisensor images on a pixel-by-pixel basis
• Feature-based fusion
– Merge features derived from the different signals or sensors.
• Decision-level fusion
– Fuse the information at a higher level by first classifying
each image independently, then combining the preliminary
classifications.
F1 06.2.08
INF 5300
17
Classification using a Gaussian
classifier – single sensor data
• Divide the available ground truth data into a training set and a
test set.
• Train the classifier by estimating μi and Σj for each class
μˆ s =
1
Ms
ˆ = 1
∑
s
Ms
∑
Ms
m =1
xm ,
∑ (x
Ms
m =1
− μˆ s )( xm − μˆ s )
t
m
where the sum is over all training samples belonging to class s
• Classifying a new sample:
– Compute for each class the conditional probability density:
1
⎡ 1
⎤
t
p(x |ωs ) =
exp ⎢ − ( x − μ s ) Σ −s 1 ( x − μ s )⎥
1/ 2
P /2
⎣ 2
⎦
(2 π ) Σ s
– Compute the posterior probability
P(ω s | x ) ∝ p ( x | ω s ) P (ω s )
– Assign the label corresponding to the class with the highest
posterior probability
F1 06.2.08
INF 5300
18
Pixel-level fusion
F1 06.2.08
INF 5300
19
Pixel-level fusion
• Advantages:
– Simple. No special classifier software needed.
– Correlation between sources utilized.
– Well suited for change detection.
• Limitations:
– Assumes that the data can be modelled using a common
probability density function. Source reliability cannot be
modelled.
– Not suited for heteorogenous data.
F1 06.2.08
INF 5300
20
Feature-level fusion
F1 06.2.08
INF 5300
21
Feature-level fusion
• Advantages:
– Simple. No special classifier software needed.
– Sensor-specific features gives advantage over pixel-based
fusion.
– Well suited for change detection.
• Limitations:
– Assumes that the data can be modelled using a common
probability density function.
– Source reliability cannot be modelled.
F1 06.2.08
INF 5300
22
Decision-level fusion
Compute the class-conditional probabilities for each class,
then combine these probabilities.
F1 06.2.08
INF 5300
23
Decision-level fusion
• Image X1,... XP from P sensors. C are the class labels
of the scene.
• Assign each pixel to the class that maximizes the
posterior probabilities
• The measurements from the different sensors are
often assumed to be conditionally independent:
F1 06.2.08
INF 5300
24
Combination of single-sensor classifiers
• Consensus theory deals with
methods for combing single-sensor
classification results.
• The output from each sensor
(probability estimates) are often
weighted according to the source’s
reliability.
• Linear opinion pool:
• Logarithmic opinion pool:
F1 06.2.08
INF 5300
25
Selecting the weights λi
• The weights λi are often selected based on heuristics
• The can be found based on grid search and crossvalidation
– Divide the training set in two, training and validation
– Select the weights that give best performance on the
validation set.
• Neural-networks can be trained to estimate weights.
F1 06.2.08
INF 5300
26
Decision-level fusion
• Advantages:
– Suited for data with different probability densities.
– Source-specific reliabilities can be modelled.
• Limitations:
– Special software often needed.
F1 06.2.08
INF 5300
27
Neural nets for
multisensor classifcation
F1 06.2.08
INF 5300
28
Multitemporal image
classification - applications
• Monitor/identify specific changes
– Different fusion levels can be used.
– Examples of pixel-level methods: image math (difference or ratio),
image regression, principal component analysis.
• Data normalization must be done prior to using these methods.
• Improved quality in discriminating between a set of classes.
– Cover limitations due to e.g. partly cloud coverage, snow or soil
moisture conctents.
– A fusion model that takes source reliability into account should be
used.
• Discriminate between classes based on their temporal signature
development
– Example: Discriminate vegetation based on how the chlorophyll
content varies during the growth season.
F1 06.2.08
INF 5300
29
Example – multitemporal image
13 different SAR images during August-December 1991 from agricultural
areas. The ability to identify ploghing depends on soilmoisture content and
temperature at the given date.
F1 06.2.08
INF 5300
30
Multitemporal classifiers 1
Direct multidate classification:
• Merge the data on a pixel-basis into one
measurement vector.
• Classify this using traditional classifiers.
• Advantage:
– Simple. Temporal feature correlation between image
measurements utilized.
• Limitations:
– Not suited for data sets containing noisy images.
– All image data must be stored.
F1 06.2.08
INF 5300
31
Multitemporal classifiers 2
Cascade classifiers:
• At time t1, classify image x1.
• At time t2, compute the conditional probability for
observing class ω given the image x1 and x2:
P(ω| x1, x2)
• Advantages:
– Temporal correlation of class labels considered.
– Information about special class transitions can be modelled.
• Limitations:
– Special software needed.
– All image data must be stored.
F1 06.2.08
INF 5300
32
Multitemporal classifiers 3
Markov chain/Markov random field classifiers:
• Model the probability of class changes from time t1
to time t2.
• Advantages:
– Spatial and temporal correlation of class labels utilized.
– Information about special class transitions can be modelled.
• Limitations:
– Special software needed.
• Markov random field classifiers will be explained
under contextual classifiers.
F1 06.2.08
INF 5300
33
Multitemporal classifiers 4
Temporal trajectory classification:
• Use integrated models that incorporate the expected
development of the classes during a time period.
• Example: vegetation in the growing season:
– Tilstander: vinterdvale, blomstring, fullt utviklet, høst
• Classification is done based on the likelihood of observing a
chain of class transitions (and when they occurr).
• Markov chains are used to model class transitions.
• Advantages:
– Can discriminate between classes not separable at a single point in
time.
– Can be used at different levels.
• Limitations:
– Feature level approaches can be sensitive to noise.
– A time series (>2 images) needed (sometimes difficult to get).
F1 06.2.08
INF 5300
34
Multiscale image classification
• Most common approach: resample one of the images
to the resolution of the other images. Classify using
traditional classifier.
• If resampling all images to the image with the
coarsest resolution, fine details might be lost.
• Resamping all images to the finest resolution can be
done by copying coarse-resolution pixels (or
performing interpolation of neighboring pixels)
– Interpolation should be used with care as it alters the grey
level distributions.
• More advanced classifiers for true multiscale data
exists. The need for such models depends on the
desired detail level in the results.
F1 06.2.08
INF 5300
35
Multiscale image visualization
• Many satellite sensors give a combination of a highresolution panchromatic band (grey-level) with high
resolution (e.g. 1m) and a set of multispectral bands
with lower resolution (e.g. 4).
• Many methods for overlaying the multispectral image
on the panchromatic image with fine spatial
structures exists.
• Common methods are based on wavelets or Fouriertransform to model data at various spatial scales.
F1 06.2.08
INF 5300
36
Summary – multisensor image
classification
• In general, there is no consensus on which multisource or
multitemporal methods that is overall best. Choices should be
application specific.
• The fusion level should be considered carefully. Pixel-based
fusion can be suited for simple analysis, but decision-level gives
best control and allows weighting the influence of each sensor.
• To find the best classifier for a multitemporal data set, the
complexity of the class separation problem and the available
data set must be selected.
• Multiscale data can either be resampled to a common
resolution, a classifier with implicit modelling of the relationship
between the different scales can be used. The latter is
recommended for problems involving small or detailed
structures.
F1 06.2.08
INF 5300
37