Image Classifications and Accuracy
Assessments
University of Namibia
DEPARTMENT OF GEOGRAPHY, HISTORY & ENVIRONMENTAL
STUDIES
Michael Mutale
Digital Image Classification
Why classify?
• Make sense of a landscape
– Purpose: Place landscape into categories (classes)
– Forest (vegetation type), Agriculture, Water, Geologic terrains,
Mineral exploration, etc.
• Classification scheme = structure of classes
– Depends on needs of users
Digital
Thematic
2
Why Classified Image?
• Image has been processed to put each pixel into a
category
• Result is a vegetation map, land use map, or other
map grouping related features
• Categories are defined by the intended use of the
map
• Can be few or many categories, depending on the
purpose of the map and available resources
3
Digital Image Classifications
• Forms an important component for examination of digital
images
• Refers to a variety of different techniques that share some
features in common
• Done through a process of assigning pixels to classes
• Classifiers use statistical “clustering” techniques to decide
which pixels should be grouped together
• The comparisons result into assembled groups of similar
pixels into classes that are associated with the
informational categories of interest to users of remotely
sensed data
4
Digital Image Classifications
• Classes form regions on an image
• After classification, digital image is presented as a mosaic of
uniform parcels, identifiable mainly by color
• In principle, pixels within classes are spectrally more similar
to one another than they are to pixels in other classes
• Classification image is defined by examining numeric image
(Spectral classes – i.e. those that are inherent in remote
sensor data and must be identified and then labeled by
analyst), then grouping together (Information classes – i.e.
those that human beings define) those pixels that have
similar spectral values
5
Unsupervised Classification
• The identification of natural groups or
structures, within multispectral data
• Involves grouping of spectral classes which
are reasonably uniform internally with
respect to brightness in several spectral
channels
• Implies definition, identification, labeling
and mapping natural classes
• Isocluster - iterative self organizing way of
performing clustering
• Clusters are merged if either number of
members (pixel) in a cluster is less than a
certain threshold or if centers of two
clusters are closer than a certain threshold
• Clusters are split into two different clusters if
cluster standard deviation exceeds a
predefined value and number of members
(pixels) is twice threshold for minimum
number of members
Computer or algorithm automatically groups
pixels with similar spectral characteristics
(means, standard deviations, covariance
matrices, correlation matrices, etc.) into unique
clusters according to some statistically
determined criteria. The analyst then re-labels
and combines the spectral clusters into
information classes.
6
Band 1
1. Data is clustered but
blue cluster is very
stretched in band 1.
Band 2
Band 2
Band 2
Example: ISODATA
Band 1
2.Cyan and green
clusters only have 2 or
less pixels. So they
will be removed.
Band 1
3. Either assign
outliers to nearest
cluster, or mark as
unclassified.
7
Band 1
1. First iteration. The
cluster centers are set
at random. Pixels will
be assigned to the
nearest center.
Band 2
Band 2
Band 2
Example: K-means
Band 1
2. Second iteration.
The centers move to
the mean-center of all
pixels in this cluster.
Band 1
3. N-th iteration. The
centers have
stabilized.
8
Advantages of Unsupervised Classification
• No extensive prior knowledge of the region is required (i.e. no need
to select representative examples of each class to be mapped)
• Only knowledge of the region is required to interpret the meaning of
the results produced by the classification process
• Reduction in human errors
• Classes resulting tend to be much more uniform with respect to
spectral composition
• Pre-conceived aspect about a region is removed
• Unique classes are recognized as distinct units
• Inclusions of distinct smaller area, that may remain
unrecognized in the supervised classification
9
Disadvantages & Limitations
• Relies on “natural” groupings
• Difficulties in matching “natural” groups to informational
categories
• Spectrally homogeneous classes within the data may not
necessarily correspond to the informational category
• Limited control of the list of classes and their specific identities
(i.e. matching classification between dates or adjacent areas)
• Spectral properties of specific informational classes changes
over time, thus relationships between informational classes and
spectral classes are not constant and relationships defined for
one image cannot be extended to others
10
Supervised Classifications
• The process of using samples of known
identity (i.e. pixels already assigned to
informational classes (categorical/nominal))
to classify pixels of unknown identity (i.e. to
assign unclassified pixels to one of several
informational classes)
• Impose your perceptions on spectral data
vs. Spectral data imposes constraints on
your interpretation
• Samples of known identity are called
training pixels
• Usually selected in digital image
• Training pixels typify spectral properties of
categories they represent (they ought to be
homogenous in respect to informational
category to be classified)
Once training sites are selected,
every pixel both within and
outside the training sites is then
evaluated and assigned to the
class of which it has the highest
likelihood of being a member.
11
Pixel-based vs Object-Oriented
• Traditionally, most digital image classification was based on
processing the entire scene pixel by pixel. This is commonly
referred to as per-pixel (pixel-based) classification
• Object-oriented classification techniques allow the analyst
to decompose the scene into many relatively homogenous
image objects (referred to as patches or segments) using a
multi-resolution image segmentation process
• Various statistical characteristics of these homogeneous
image objects in the scene are then subjected to traditional
statistical analysis
• Object-oriented classification based on image
segmentation is often used for analysis of high-spatialresolution imagery (e.g., 1  1m Space Imaging IKONOS and
0.61  0.61m Digital Globe Quick Bird)
12
Supervised Classification Methods
• Choice of a particular classifier or decision rule depends on nature of input
data and desired output
• Parametric classification algorithms assumes that observed measurement
vectors Xc obtained for each class in each spectral band during training
phase of supervised classification are Gaussian; that is, they are normally
distributed
• Nonparametric classification algorithms make no such assumption
• Several widely adopted nonparametric classification algorithms include:
–
–
–
–
one-dimensional density slicing
parallepiped
minimum distance
nearest-neighbor
• The most widely adopted parametric classification algorithms:
– maximum likelihood
• Hyperspectral classification methods
–
–
–
–
–
Binary Encoding
Spectral Angle Mapper
Matched Filtering
Spectral Feature Fitting
Linear Spectral Unmixing
13
Supervised Classifications
Involves three basic stages:
• Training stage – identification of representative areas
• Classification – categorizing all pixels in the image into
classes
• Output – Thematic map / Tables / Statistics / GIS input
14
Training Stage
• Selection of training sites is a key step in supervised classification
• Requires substantial reference data and a thorough knowledge of the geographic
area to which the data apply
• Aims at assembling a set of statistics that describe the spectral response pattern for
each class
• Determines the location, size, shape and orientation of “clouds” of points for each
class
15
Classification Stage
16
Guidelines for Selecting Training Data
• At least 100 pixels is needed for each class
• Size of the training area should be big enough to avoid mix-pixels
• Avoid data clustering which may under-represent variation within the
image
• Overall, homogeneity is much more desirable than heterogeneity
within a class
17
Classification Stage
Minimum-distance-to-Means Classified
• Determines mean or average spectral value in each band for each
category
• Then calculates distance between the value of unknown pixel and
each of the category means
• Based on results, the unknown pixel is assigned to “closest” class
• If pixel is farther than what analyst defined as minimum distance from
any category, then the pixel would be classified as “unknown”
• Although this method is simple and computationally efficient, it has
the following limitations:
• It is insensitive to different degrees of variance in spectral
response data
• Thus, this classifier is not widely used in applications where
spectral classes are close to one another in measurement space
and have high variance
18
Minimum Distance
19
Classification Stage
Parallelepiped (Box) Classifier
• Multi-dimensional analogs of rectangular areas are called
parallelepipeds
• Classifier only considers range of values in each category training set,
thus sensitive to category variance
• Range defined by highest and lowest DNs in each band
• Appears as a rectangular
• Although this method is also fast and computationally efficient,
difficulties are experienced when category ranges overlap (overlap is
caused by high covariance, which are poorly described by rectangular
decision regions, due to slanting)
• Box classifier can be modified into a series of rectangles with stepped
20
borders
Box Classifier
21
Box - Modified
22
Classification Stage
Maximum Likelihood Classifier
• Quantitatively evaluates both variance and covariance of category
spectral response patterns
• Assumes a Gaussian (normally distributed) cloud of points
• Probability typically set at threshold from [0, 1]
• 0 means zero probability of similarity, 1 means 100% probability of
similarity
• Computed statistical probability of a given pixel value being a member
of a particular class (highest probability value)
• Large number of computations required, however
23
Maximum Likelihood
24
Spectral Angle Mapper (SAM)
• Compares test image spectra to a
known reference spectra using
spectral angle between them
• Small angles between two
spectrums indicate high similarity
and high angles indicate low
similarity
• Not sensitive to illumination since
SAM algorithm uses only vector
direction and not vector length
• Works well in areas of homogeneous
regions
25
Advantages - Supervised Classifications
• Control over selected list of informational categories tailored to
a specific purpose and geographic region (important when
making multi-date comparisons with neighboring regions)
• Tied to specific areas of known identity, determined through
process of selecting training sites
• Detection of mis-classified pixels possible
26
Disadvantages - Supervised Classifications
• Imposing a class structure to data
• Operator-defined classes may not match natural classes that
exist within data, thus they may not be distinct or well defined in
multidimensional data space
• Spectral properties are secondary to informational category (as
classes are defined using the latter)
• Selection of training sites can be time-consuming, expensive and
tedious undertaking
27
Accuracy of Classification
• Classified maps are not a perfect representation of reality
• Accuracy statistics provide objective information about the
quality of the land cover classification
• Accuracy statistics address overall quality and per-class
quality
• Accuracy assessment can be costly
• Accuracy assessment is often dropped from a project
because of cost or time limitations
28
Types of Errors
• Position error
– Often due to misregistration between the base information
and the area being mapped
– Can be difficult to notice unless compared to accurate base
data
• Thematic error
– Due to misidentification of individual features
– Usually the focus of accuracy statistics
• Accuracy assessment sampling often skewed toward detecting
thematic error
– Accuracy assessment sampling design usually allows for
“reasonable” position errors by assuring sample points are
not close to a class edge
29
Steps for Assessing Accuracy
• Sample design
• Collection of validation data
• Compiling validation data
• Analysis
30
Sample Design
• Sampling design attempts to minimize bias
• The following decisions must be made:
–
–
–
–
Number of samples
How will the samples be distributed
How will a sample area be defined (point, area)
How will sampling take place (field, aerial photos)
• In many cases adjustments to the “pure” sampling design
must be made to accommodate practical realities such as
access to sampling points.
• Sample data must be independent of training data
• Many sampling designs are influenced by the amount of
money available and not “pure” statistical theory
31
Collection of Validation Data
• Accurate locality (i.e., latitude/longitude, UTM coordinates)
data should be associated with each sampling point
• Often useful if geo-coded photographs are acquired in the
field
• Information should be collected that is relevant to the map
being assessed
• Ancillary data such as aerial photography can be used in
place of data collected in the field if the mapped classes
can be accurately identified
32
Compiling Validation Data
• If photos are used to record land cover type they must be
acquired in the same time frame as the image used to
create a map
• For each sampling point the classification value (i.e., land
cover type) as determined in the field must be recorded
• The classification values from the validation data (reference
data) and the classified map should be tallied in a
contingency table to facilitate analysis
33
Analysis of Results
• Analysis is used to determine the accuracy of the map
• Using simple formulas the data in a contingency table can
be analyzed to determine a range of accuracy figures
• Accuracy figures are often presented from the users
perspective
If I select any water pixel on the classified map, what is the
probability that I'll be standing in water when I visit that
pixel location in the field?
• And from the producers perspective
If I know that a particular area is water, what is the
probability that the digital map will correctly identify that
pixel as water?
34
Example of an error matrix showing pixel counts
User's Accuracy: A map-based accuracy
User’s Accuracy (water) = (# of pixels correctly classified as water) / (total # of pixels classified as
water) = 367/400 = 91.75%
Producers accuracy: A reference-based accuracy
Producer Accuracy (water) = (# of pixels correctly classified as water) / (# ground reference
pixels in water) = 367/382 = 96.07%
From Canada Centre for Remote Sensing, Natural Resources Canada, 11/21/2005
35
36
Other Accuracy Terms
• Overall Accuracy = # pixels correctly classified/total # of
pixels) = 90 / 100 = 90%
• Omission Error: Excluding a pixel that should have been
included in the class (i.e., omission error = 1 - producers
accuracy)
• Commission Error: Including a pixel in a class when it
should have been excluded (i.e., commission error = 1 user's accuracy.
• Kappa Coefficient: Measures the improvement of the
classified map over a random class assignment
37
Another Example
38
Typical Land Cover Accuracy Figures
• Forest/Non-forest, Water/No Water, Soil/Vegetated:
accuracies in the high 90%
• Conifer/Hardwood: 80-90%
• Genus: 60-70%
• Species: 40-60%
• Bottom Line: The greater the detail (precision) the lower
the per class accuracy
•
Note: If including a Digital Elevation Model (DEM) in the classification, add 10%
39
Classification Error Matrix
• Confusion matrix or a contigency table
• Compares, on a category-by-category basis, relationship
between known reference data (ground truth) and
corresponding results of classified image
• Major diagonal of the error matrix
• All non-diagonal elements of the matrix represent errors of
ommision (exclusion; column; reference data in vertical) or
commission (inclusion; row).
40
Classification Error Matrix
Classified Image
Reference Image
Total
Reliability
1
239
94%
1
0
309
70%
228
3
5
599
60%
2
397
8
4
521
76%
4
48
132
190
78
453
42%
1
0
19
84
36
219
359
61%
Total
233
328
429
945
238
307
2480
Accuracy
97%
66%
84%
42%
80%
71%
Water
Sand
Forest
Urban Corn Hay
Water
226
0
0
12
0
Sand
0
216
0
92
Forest
3
0
360
Urban
2
108
Corn
1
Hay
65%
Omission (wrongly omitted: in columns – excluding diagonal fields)
Commission (wrongly added: in rows – excluding diagonal fields)
41
Change Detection Matrix
B
Classified
B G G G
Reference
W W W G G
G
W W B
G
B
G
W W B
G
B
G
G
X G
W G
B
B
G
B
G
B – Bare Soil
G
G
B
G – Grassland
Crossed Table
Classified
Reference
Number
of Pixels
Area
(km)
Area%
W W W
B
B
2
0.125
8
G
G
W W
B
G
2
0.125
8
G
G
G
B
W
4
0.25
16
G
B
0
0
0
G
G
9
0.5625
36
G
W
3
0.1875
12
W
B
0
0
0
W
G
0
0
0
W
W
5
0.3125
20
1.5625
100
W W W G
W
W – Water Body
Total
Note: Pixel size / spatial resolution is 250m
42
Change Detection Matrix
Crossed Table
Refere
nce
Number
of Pixels
Area
(km)
Area%
B
B
2
0.125
8
B
G
2
0.125
8
B
W
4
0.25
16
G
B
0
0
0
G
G
9
0.5625
36
G
W
3
0.1875
12
W
B
0
0
0
W
G
0
0
0
W
W
5
0.3125
1.5625
Total
Reference
Classified
Classified
B (%)
G (%)
W
Total
(%)
(%)
B
8
8
16
32
25%
G
0
36
12
48
75%
W
0
0
20
20
100%
20
Total
8
44
48
100
100
Accuracy
100
81.8
41.6
Reliability
Note: Pixel size / spatial resolution is 250m
43
Accuracy (Producer Accuracy)
• Present accuracy of classification
• Measure of commission error
• Fraction of correctly classified pixels with regard to all pixels of
that ground truth class
• For each class of ground truth pixels (column), number of
correctly classified pixels is divided by total number of ground
truth or test pixels of that class
• E.g. for Bare or ‘Water' class, accuracy is ((8/8)*100 = 100% or
(226/233 = 0.97 in first example) meaning that approximately
100% (bare) or 97% (Water) of Bare or ‘Water' ground truth
pixels also appear as Bare or ‘Water' pixels in classified image
44
Reliability (User Accuracy)
• Present reliability of classes in classified image
• Fraction of correctly classified pixels with regard to all pixels
classified as this class in classified image
• For each class in classified image (row), number of correctly
classified pixels is divided by total number of pixels which were
classified as this class
• E.g. for Bare or ‘Water' class, reliability is 226/239 = 0.94
meaning that approximately 94% of ‘Water' pixels in classified
image actually represent ‘Water' on ground
45
Overall Accuracy / Reliability
• Overall accuracy is calculated as total number of correctly classified
pixels (diagonal elements) divided by total number of referenced pixels
Water
Sand
Forest
Urban
Corn
Hay
Total
Reliability
Water
226
0
0
12
0
1
239
94%
Sand
0
216
0
92
1
0
309
70%
Forest
3
0
360
228
3
5
599
60%
Urban
2
108
2
397
8
4
521
76%
Corn
1
4
48
132
190
78
453
42%
Hay
1
0
19
84
36
219
359
61%
Total
233
328
429
945
238
307
2480
Accuracy
97%
66%
84%
42%
80%
71%
• = (226 + 216 + 360
+ 397 + 190 + 219) /
2480
• = 65%
65%
46