Sensors and media
INF 5300
Data fusion for image analysis
• Remote sensing:
– Electromagnetic signals, from visible to
thermal and infrared and microwave
energy
– Sensors: Optical, microwave, thermal,
laser altimeter
Anne Solberg (anne@ifi.uio.no)
Today:
• Medical image analysis:
•Background
– Ultrasound: acoustical signals
– Magnetic Resonnance Imaging
(MR)/CT
– PET/SPECT
– X-ray
•Multisensor data registration
•Multisensor image classification
•Multitemporal image analysis
• Sonar imaging:
•Multiscale image analysis
– Multifrequency echosounders
F1 20.3.06
INF 5300
1
F1 20.3.06
Multifrequency imaging
INF 5300
2
Multitemporal data
• Multispectral and hyperspectral sensors provide
information on different frequencies. Different targets
are sensitive to certain frequencies.
– Imaging at several frequencies>separating more detalied
classes
• By imaging on object at different times, more
information can be derived.
– Discriminate between objects based on their temporal
variation (e.g. vegetation classification)
• Challenges:
• This applies to other types of signals, too.
– Sensor calibration/weather changes
– Changes in the classes occurr
– Multifrequency echosounders used for fishery
• A new road is built, trees are cut
F1 20.3.06
INF 5300
3
F1 20.3.06
INF 5300
4
Multisensor data
Multisensor data registration
• Each sensor has it own characteristics
–
–
–
–
–
–
–
• Data registration is a pre-requisite for data fusion
• Data registration can be simple if the data are
georeferenced (map co-ordinates are known)
Noise level
Reliability
Ability to discriminate certain object types
Resolution
Availability/Coverage
Limitations due to e.g. weather/clouds
Imaging geometry and geometrical errors
F1 20.3.06
INF 5300
– Registration is then just resampling the images to a common
map projection
• An image matching step is often necessary to obtain
subpixel accuracy in the co-registration.
5
Image registration – an overview
F1 20.3.06
INF 5300
6
Registration step 1: feature extraction
• Purpose: extract regions, edges, contours etc. that
can be used to represent tie-points in the set of
images.
• Tie-points: characteristic points that can be identified
in all images.
• Area-based methods:
The main steps in a full image registration process are:
1. Feature extraction
2. Feature matching
3. Transformation selection
4. Image resampling
– Match the grey levels in the images directly using statistical
measures.
• Feature-based methods:
– Define a set of useful features (e.g. edges, lines,
intersections etc.) tailored to the application.
– Extract these features using either spatial-domain methods
or frequency-domain methods (Fourier- or wavelets)
F1 20.3.06
INF 5300
7
F1 20.3.06
INF 5300
8
Registration step 3:
Transformation selection
Registration step 2: feature matching
• Purpose: establish the correspondence between the
tie-points found during step 1.
• Area-based methods:
• Purpose: find the mapping function that describes
the relation between the positions of the tie-points in
the different images.
• Affine tranforms are often used.
– Match using correlation, Fourier-transform methods or
optical flow.
Affine transformasjoner (lineære):
x’=a0x+a1y+a2
y’=b0x+b1y+b2
Kvadratiske transformasjoner (kalles warping):
x’=a0x2+a1y2+a2xy+a3x+a4y+a5
y’=b0x2+b1y2+b2xy+b3x+b4y+b5
• Feature-based methods:
– Perform the matching in the Fourier-domain using the
equivalence between correlation and multiplication in the
Fourier-domain.
• Optical flow methods:
– Estimate the relative motion between the two images.
• This is commonly used in video analysis.
F1 20.3.06
INF 5300
9
F1 20.3.06
Registration step 4:
Image resampling
The data can be fused at different level:
• Signal-based fusion
– Fuse the signals prior to image formation
• Pixel-based fusion
– Merge the multisensor images on a pixel-by-pixel basis
• Feature-based fusion
– Merge features derived from the different signals or sensors.
• Decision-level fusion
• See INF2310 for details of affine transforms and
interpolation.
http://www.ifi.uio.no/~inf2310/v2006/forelesninger/20060228_geometriskeOperasjoner.pdf
INF 5300
10
Fusion at different levels
• Purpose: compute the new pixel co-ordinates in the
co-registered images and interpolate the pixel values.
• Transforming the co-ordinates will give non-integer
co-ordinates. Interpolation is used to combine pixel
values from neighboring pixels.
• Normally, either nearest-neighbor interpolation or
bilinear interpolation is used.
F1 20.3.06
INF 5300
11
– Fuse the information at a higher level by first classifying
each image independently, then combining the preliminary
classifications.
F1 20.3.06
INF 5300
12
Pixel-level fusion
Pixel-level fusion
• Advantages:
– Simple. No special classifier software needed.
– Correlation between sources utilized.
– Well suited for change detection.
• Limitations:
– Assumes that the data can be modelled using a common
probability density function. Source reliability cannot be
modelled.
– Not suited for heteorogenous data.
F1 20.3.06
INF 5300
13
F1 20.3.06
Feature-level fusion
INF 5300
14
Feature-level fusion
• Advantages:
– Simple. No special classifier software needed.
– Sensor-specific features gives advantage over pixel-based
fusion.
– Well suited for change detection.
• Limitations:
– Assumes that the data can be modelled using a common
probability density function.
– Source reliability cannot be modelled.
F1 20.3.06
INF 5300
15
F1 20.3.06
INF 5300
16
Decision-level fusion
Decision-level fusion
• Image X1,... XP from P sensors. C are the class labels
of the scene.
• Assign each pixel to the class that maximizes the
posterior probabilities
• The measurements from the different sensors are
often assumed to be conditionally independent:
Compute the class-conditional probabilities for each class,
then combine these probabilities.
F1 20.3.06
INF 5300
17
F1 20.3.06
INF 5300
18
Selecting the weights λi
Combination of single-sensor classifiers
• The weights λi are often selected based on heuristics
• The can be found based on grid search and crossvalidation
• Consensus theory deals with
methods for combing single-sensor
classification results.
• The output from each sensor
(probability estimates) are often
weighted according to the source’s
reliability.
• Linear opinion pool:
– Divide the training set in two, training and validation
– Select the weights that give best performance on the
validation set.
• Neural-networks can be trained to estimate weights.
• Logarithmic opinion pool:
F1 20.3.06
INF 5300
19
F1 20.3.06
INF 5300
20
Neural nets for
multisensor classifcation
Decision-level fusion
• Advantages:
– Suited for data with different probability densities.
– Source-specific reliabilities can be modelled.
• Limitations:
– Special software often needed.
F1 20.3.06
INF 5300
21
F1 20.3.06
Multitemporal image
classification - applications
– Different fusion levels can be used.
– Examples of pixel-level methods: image math (difference or ratio),
image regression, principal component analysis.
• Data normalization must be done prior to using these methods.
• Improved quality in discriminating between a set of classes.
– Cover limitations due to e.g. partly cloud coverage, snow or soil
moisture conctents.
– A fusion model that takes source reliability into account should be
used.
Direct multidate classification:
• Merge the data on a pixel-basis into one
measurement vector.
• Classify this using traditional classifiers.
• Advantage:
– Simple. Temporal feature correlation between image
measurements utilized.
• Discriminate between classes based on their temporal signature
development
– Example: Discriminate vegetation based on how the chlorophyll
content varies during the growth season.
INF 5300
22
Multitemporal classifiers 1
• Monitor/identify specific changes
F1 20.3.06
INF 5300
23
• Limitations:
– Not suited for data sets containing noisy images.
F1 20.3.06
INF 5300
24
Multitemporal classifiers 2
Multitemporal classifiers 3
Cascade classifiers:
• At time t1, classify image x1.
• At time t2, compute the conditional probability for
observing class ω given the image x1 and x2:
Markov chain/Markov random field classifiers:
• Model the probability of class changes from time t1
to time t2.
• Advantages:
P(ω| x1, x2)
– Spatial and temporal correlation of class labels utilized.
– Information about special class transitions can be modelled.
• Advantages:
– Temporal correlation of class labels considered.
– Information about special class transitions can be modelled.
• Limitations:
– Special software needed.
• Limitations:
– Special software needed.
F1 20.3.06
INF 5300
25
Multitemporal classifiers 4
26
• Most common approach: resample one of the images
to the resolution of the other images. Classify using
traditional classifier.
• If resampling all images to the image with the
coarsest resolution, fine details might be lost.
• Resamping all images to the finest resolution can be
done by copying coarse-resolution pixels (or
performing interpolation of neighboring pixels)
– Tilstander: vinterdvale, blomstring, fullt utviklet, høst
• Classification is done based on the likelihood of observing a
chain of class transitions (and when they occurr).
• Markov chains are used to model class transitions.
• Advantages:
– Can discriminate between classes not separable at a single point in
time.
– Can be used at different levels.
• Limitations:
– Feature level approaches can be sensitive to noise.
– A time series (>2 images) needed (sometimes difficult to get).
INF 5300
INF 5300
Multiscale image classification
Temporal trajectory classification:
• Use integrated models that incorporate the expected
development of the classes during a time period.
• Example: vegetation in the growing season:
F1 20.3.06
F1 20.3.06
27
– Interpolation should be used with care as it alters the grey
level distributions.
• More advanced classifiers for true multiscale data
exists. The need for such models depends on the
desired detail level in the results.
F1 20.3.06
INF 5300
28
Summary – multisensor image
classification
Multiscale image visualization
• Many satellite sensors give a combination of a highresolution panchromatic band (grey-level) with high
resolution (e.g. 1m) and a set of multispectral bands
with lower resolution (e.g. 4).
• Many methods for overlaying the multispectral image
on the panchromatic image with fine spatial
structures exists.
• Common methods are based on wavelets or Fouriertransform to model data at various spatial scales.
• In general, there is no consensus on which multisource or
multitemporal methods that is overall best. Choices should be
application specific.
• The fusion level should be considered carefully. Pixel-based
fusion can be suited for simple analysis, but decision-level gives
best control and allows weighting the influence of each sensor.
• To find the best classifier for a multitemporal data set, the
complexity of the class separation problem and the available
data set must be selected.
• Multiscale data can either be resampled to a common
resolution, a classifier with implicit modelling of the relationship
between the different scales can be used. The latter is
recommended for problems involving small or detailed
structures.
F1 20.3.06
F1 20.3.06
INF 5300
29
INF 5300
30