Digital Image Processing
FW5560
Lecture 17
Accuracy Assessment
Information derived from remotely sensed data are important for
environmental models at local, regional, and global scales. The remote
sensing–derived thematic information may be in the form of thematic
maps or statistics derived from area-frame sampling techniques.
The thematic information must be accurate because important decisions
are made throughout the world using the information.
Unfortunately, the thematic
information contains error.
Remote sensing–derived
thematic maps should normally
be subjected to a thorough
accuracy assessment before
being used in scientific
investigations and policy
decisions.
It is first necessary to clearly
state the nature of the
thematic accuracy assessment
problem at hand, including:
• what the accuracy
assessment is expected to
accomplish,
• the classes of interest
(discrete or continuous), and
• the sampling design and
sampling frame.
A simple random sample is selected so that all samples of the same size
have an equal chance of being selected from the population.
Systematic sampling is a statistical method involving the selection of
elements from an ordered sampling frame.
Cluster sampling involves selecting the sample units in groups.
Stratified systematic unaligned sample
Stratified sampling involves selecting independent samples from a
number of subpopulations, group or strata within the population. Great
gains in efficiency are sometimes possible from judicious
stratification.
Proportionate allocation uses a sampling fraction in each of the
strata that is proportional to that of the total population.
Optimum allocation (or Disproportionate allocation) - Each
stratum is proportionate to the standard deviation of the
distribution of the variable. Larger samples are taken in the
strata with the greatest variability to generate the least possible
sampling variance.
Characteristics of a typical error matrix consisting of
k classes and N ground reference test samples.
The ideal situation is to locate ground reference test pixels
(or polygons if the classification is based on human visual
interpretation) in the study area.
These sites are not used to train the classification algorithm and
therefore represent unbiased reference information. It is possible
to collect some ground reference test information prior to the
classification, perhaps at the same time as the training data. But
the majority of test reference information is often collected after
the classification has been performed using a random sample to
collect the appropriate number of unbiased observations per
Category.
Fitzpatrick-Lins (1981) suggests that the sample size N to be
used to assess the accuracy of a land-use classification map
be determined from the formula for the binomial probability
theory:
Z 2 ( p)(q)
N
E2
where p is the expected percent accuracy of the entire map,
q = 100 – p, E is the allowable error, and Z = 2 from the
standard normal deviate of 1.96 for the 95% two-sided
confidence level.
For a sample for which the expected accuracy is 85% at an allowable
error of 5% (i.e., it is 95% accurate), the number of points necessary for
reliable results is:
Z 2 ( p)(q)
N
E2
22 (85)(15)
N
 a minimumof 203points.
2
5
With expected map accuracies of 85% and an acceptable
error of 10%, the sample size for a map would be 51:
22 (85)(15)
N
 51 points
2
10
Evaluation of Error Matrices
After the ground reference test information has been collected
from the randomly located sites, the test information is
compared pixel by pixel (or polygon by polygon when the
remote sensor data are visually interpreted) with the information
in the remote sensing–derived classification map.
Agreement and disagreement are summarized in the cells of the
error matrix. Information in the error matrix may be evaluated
using simple descriptive statistics or multivariate analytical
statistical techniques.
Descriptive Statistics
The overall accuracy of the classification map is determined by dividing
the total correct pixels (sum of the major diagonal) by the total number
of pixels in the error matrix (N ).
Computing the accuracy of individual categories, is more complex because the analyst
has the choice of dividing the number of correct pixels in the category by the total
number of pixels in the corresponding row or column.
Total number of correct pixels in a category is divided by total number of pixels of that
category as derived from the reference data (i.e., the column total). This statistic
indicates the probability of a reference pixel being correctly classified and is a measure
of omission error. This statistic is the producer’s accuracy because the producer (the
analyst) of the classification is interested in how well a certain area can be classified.
Descriptive Statistics
If the total number of correct pixels in a category is divided by the total
number of pixels that were actually classified in that category, the result
is a measure of commission error. This measure, called the user’s
accuracy or reliability, is the probability that a pixel classified on the
map actually represents that category on the ground.
Discrete Multivariate Analytical Techniques
Have been used to statistically evaluate the accuracy of remote
sensing–derived classification maps and error matrices since
1983.
The techniques are appropriate because remotely sensed data
are discrete rather than continuous and are also binomially or
multinomially distributed rather than normally distributed.
Statistical techniques based on normal distributions simply do
not apply. It is important to remember, however, that there is not
a single universally accepted measure of accuracy but instead a
variety of indices, each sensitive to different features.
Kappa Analysis
Khat Coefficient of
Agreement: Kappa
analysis yields a statistic,
which is an estimate of
Kappa. It is a measure of
agreement or accuracy
between the remote sensing
–derived classification map
and the reference data as
indicated by a) the major
diagonal, and b) the chance
agreement, which is
indicated by the row and
column totals (referred to
as marginals).
Conditional Kˆ Coefficient of Agreement can be used to
calculate agreement between the reference and remote sensing–
derived data with chance agreement eliminated for an individual
class for user accuracies using the equation:
N ( xii )  xi   xi 
ˆ
K
N ( xi  )  xi   xi 
where xii is the number of observations correctly classified
for a particular category (summarized in the diagonal of the
matrix), xi+ and x+i are the marginal totals for row i and column
i associated with the category, and N is the total number
of observations in the entire error matrix.
For example, the conditional coefficient of agreement for the
residential land-use class of the Charleston, SC, dataset is:
407(70)  88 73
ˆ
K Residential 
 0.75
407(88)  88 73
This procedure can be applied to each land-cover class of
interest.