This file was created by scanning the printed publication.
Errors identified by the software have been corrected;
however, some errors may remain.
On Roles and Goals
for Map Accuracy Assessment:
A Remote Sensing Perspective
Curtis E. Woodcock
Abstract. - Expanded roles and goals for accuracy assessment are possible based on changes in the assumptions regarding the properties of
classes in thematic maps. Existing views of thematic maps use classical set theory, which assumes each location in the map can be assigned
unambiguously to a single class. Similarly, classes are assumed
exhaustive and mutually exclusive. The substitution of fuzzy set
theory for classical set theory relaxes these assumption and affords new
opportunities and challenges. The main concept required for use of
fuzzy set theory is acknowledgement of varying levels of class
membership for individual map units. Three areas where improvements are possible using accuracy assessment based on fuzzy sets are:
1) providing additional information on map accuracy, including information on magnitudes of errors, and alternative methods of estimating
the frequency of errors; 2) expanding the kinds of queries that are possible with regard to area estimation, including estimates of areas of
map classes as a function of class membership levels; and 3) improved
landscape characterization by explicit acknowledgement of the heterogeneity within remote sensing pixels or map polygons. Associated
with these expanded roles and goals for accuracy assessment is a need
for more research and development.
INTRODUCTION
For a long time thematic maps have been an important method for display
of spatial data. Their importance continues, and possibly has been enhanced by
the explosive growth in the use of geographic information systems (GIs).
There is a clear consensus that accuracy assessment of thematic maps is an
important aspect of both map production and use. The importance of accuracy
assessment has become particularly apparent due to the use of remote sensing
and computer-based methods for production of thematic maps. All maps
Associate Professor of Geography, Boston University Bmron, MA
contain errors, but the use of more automated methods has brought the issue of
map quality and accuracy assessment to the fore.
Accuracy assessment has served two primary communities: the users of
maps and the producers of maps. From the production perspective, accuracy
assessment is a critical dimension of the process of evaluating alternative mapping methodologies and sensors. Map accuracy assessment has provided the
metric for comparisons in a field where comparisons are frequent. From a
user's perspective, accuracy assessment provides important information on the
quality of the maps, which in turn can significantly influence the manner in
which maps can be used. While the motivation for users is typically an indication of the reliability of the map, the dominant concern remains the same -- the
overall accuracy of the map.
The purpose of this paper is to argue for expanded roles and goals for
accuracy assessment, particularly from the perspective of map production.
These more ambitious roles and goals of accuracy assessment are made possible by a change in the fundamental assumptions about the nature of the classes
in thematic maps. Conventionally, the classes in thematic maps are assumed to
follow classical set theory, in which each location in the landscape is assumed
to belong to a single map class. Additionally, the map categories are assumed
to be mutually exclusive and exhaustive. Sets, or map classes in this case, that
meet these conditions are often termed crisp sets. The continuum of variation
found in landscapes cannot be captured by classes using the assumptions of
crisp sets. The substitution of fuzzy set theory for classical set theory relaxes
assumptions regarding the mutually exclusive nature of map classes and affords
new opportunities and challenges. I suggest three initial categories of issues
arising from the substitution of fuzzy set theory. This list is not intended to be
exhaustive, but rather a start. Each of these has received some initial research
attention, but more is clearly needed and warranted.
Background on Fuzzy Set Theory
The concept of a fuzzy set was introduced by Zadeh (1965) to describe
imprecision that is characteristic of much of human reasoning. Fuzzy sets provide a quantitative approach for dealing with vagueness in complex systems. A
primary difference between fuzzy set theory and classical set theory concerns
membership functions. In classical set theory, each object or element is either
a member of a set or it isn't. Fuzzy set theory allows for grades of membership, providing considerable flexibility beyond that available using classical set
theory. For thematic maps, fuzzy sets afford the opportunity to acknowledge
ambiguity about the class membership of locations in the map. Sites with properties near the definitional boundaries between classes can be characterized
accordingly.
ROLES AND GOALS FOR ACCURACY ASSESSMENT
Issue 1: Enhanced Accuracy Information
Gopal and Woodcock (1994) have provided an initial attempt at accuracy
assessment methods for thematic maps based on fuzzy set theory. These
methods expand the kind of information resulting from map accuracy assessment. The basic approach to accuracy assessment is similar to conventional
methods with respect to sample selection and the need for ground truth (Congalton, 1991). The primary difference in data collection is the requirement for
a fuzzy membership value for each possible map class for each sample site. A
series of fuzzy operators are used to evaluate the data collected at accuracy
assessment sites. Two operators, MAX and RIGHT remain concerned with the
primary question of overall map accuracy. However, two independent measures of map accuracy result, with MAX providing a more conservative estimate
of map accuracy than RIGHT. The difference in the results for these two
operators, combined with the results from a DIFFERENCE operator provide
indications of the magnitude of the errors in the map. Conventionally, frequency of errors has been the criterion determining overall map accuracy.
Given the ability to measure the magnitude of errors, it would be beneficial to
incorporate this information into measures of overall map accuracy.
The nature of the errors in the map, or an analysis of the confusion
between classes proceeds in a similar fashion to conventional confusion
matrices using a CONFUSION operator. One added dimension associated with
the use of fuzzy sets is the AMBIGUITY operator, which tracks the frequency
of ties between classes. The results from the AMBIGUITY operator are
arranged in a matrix similar to the CONFUSION operator, and are particularly
interesting if they are asymmetric. In this situation it is possible to infer the
distribution in the map of areas receiving equal fuzzy membership values for
two different classes.
Overall, the use of fuzzy operators provides more information about the
accuracy of a map than is possible using conventional methods. The use of
both the MAX and RIGHT operators increases the information about the frequency of errors, and the overall accuracy of the map. Explicit consideration
of the magnitude of errors is also an important development. However, the
work of Gopal and Woodcock should be considered only a start. The relaxation of the limitations required by classical set theory provides many opportunities for improved accuracy assessment which remain to be explored.
Issue 2: An Expanded View of Area Estimation
The use of fuzzy sets and the concept of levels of class membership
expand significantly the kinds of queries possible regarding the areal extent of
classes. Data from conventional map accuracy assessments and marginal map
totals have frequently been used to provide improved estimates of the true area
of map classes (Card, 1982; Hay, 1988; Jupp, 1989). However, these
approaches are limited to the constraint imposed by classical set theory that the
sums of the areas of all map classes must be unity. Once the idea that various
grades of class membership exist, it is possible to pose questions regarding the
areal extent of classes as a function of level of class membership. When
viewed from this perspective, the sums of the areas for all classes need not sum
to unity. In fact, values below unity would be expected for strict levels of
class membership, and values above unity for more lenient levels of class
membership.
Card's (1982) methods for area estimation have been adapted for use in
estimating the area in maps having various levels of class membership. The
basis for Card's approach is to assign the area in the map represented by each
sample (which is a function of the number of samples in the map class, and the
map area of the map class) to the true class of the sample site. By retaining
the column totals (map classes) for sample sites from the confusion matrix and
the marginal map proportions and substituting a confusion matrix that includes
only sample sites exhibiting specified levels of class membership, it is possible
to estimate the true area of each class for specific levels of class membership
(Woodcock and Gopal, 199?). The idea that the sums of the areas of classes
might not equal unity is initially unsettling. However, when viewed from outside the normal questions of area estimation, it is easier to understand. For
example, one way to view the problem is more like the general question of
which locations in a map meet a certain set of conditions. This kind of problem is now common in GIs, and expectations for the results of several such
queries to sum to unity rarely exist.
The ability to estimate the areas of classes as a function of levels of class
membership should improve the utility of maps. An example from vegetation
mapping helps illustrate this point. Currently, users are limited to questions
regarding the areas dominated by a particular vegetation type. However, mixes
of vegetation types are common. Using the methods mentioned above, it
would now be possible to ask questions regarding the areal extent of locations
where a particular vegetation type is at least present, if not dominant. These
kinds of queries are particularly important in wildlife habitat studies, where the
presence of a particular species of tree may be important for nesting purposes.
This direction of research is obviously new, and lacks the benefits of peer
review. However, the possibility for improving the information that can be
provided to map users is considerable.
Issue 3: Improved Landscape Characterization
One trend in remote sensing has been toward providing maps at regional,
continental, and global scales. To accommodate mapping areas of these sizes,
increasing use is being made of satellite imagery with coarse spatial resolutions, such as provided by the NOAA AVHRR (see for example, DeFries and
Townshend, 1994). One issue arising in the use of coarse spatial resolution
imagery in making thematic maps concerns the internal homogeneity of individual pixels. Portraying large areas as a single class provides an imprecise view
of landscapes. Prior research has indicated significant errors in the estimation
of map proportions can result from the use of large pixel sizes, including the
overestimation of the area of larger classes and the disappearance of smaller
classes (Turner et al., 1989; Moody and Woodcock, 1994).
One way to view this problem concerns the ability of classes based on
crisp sets to characterize landscapes at these scales. The use of fuzzy sets
allows for explicit acknowledgement of internal heterogeneity for large pixels.
One community within remote sensing assumes all pixels are mixtures, and
employs methods designed to estimate the proportions of constituent components (Adams et al., 1986; Roberts, et al., 1993; Defries et al., 1995). The
use of fuzzy set representations for classes in thematic maps may provide a
bridge between this mixture modeling community and the production of
thematic maps. The key element that fuzzy sets can provide in this context is a
method of translating various component mixtures into levels of class membership.
DISCUSSION AND CONCLUSIONS
This paper calls for a broader role for map accuracy assessment based on
the more realistic underlying assumptions of fuzzy set theory. The expanded
opportunities carry the requirement of an extensive supporting research agenda.
For example, adaptation of some of the recent developments for conventional
accuracy assessment for fuzzy sets would be beneficial (Foody, 1992; Ma and
Redmond, 1995). While a substantial research agenda is needed, it should be
noted that the general steps required for accuracy assessment based on fuzzy
sets are quite similar to conventional methods.
The goals of accuracy assessment can be expanded beyond consideration
of the nature and frequency of errors in maps to include assessment of the
magnitude of those errors and alternative measures of their frequency. Estimation of the area of map classes as a function of class membership levels
significantly expands the kinds of queries that are possible, and hence improves
the utility of maps. Additionally, the use of fuzzy sets in thematic maps can
improve landscape characterization by allowing explicit acknowledgement of
heterogeneity within map units.
REFERENCES
Adams, J. B, Smith, M.O., and P.E. Johnson, 1986. Spectral Mixture
Modeling: A New Analysis of Rock and Soil Types at the Viking Lander
1 Site, Journal of Geophysical Research, 9 1(B8):8098-8112.
Card, D.H., 1982. Using known map category marginal frequencies to improve
estimates of thematic map accuracy, Photogrammetric Engineering and
Remote Sensing, 48(3):43 1-439.
Congalton, R.G., 1991. A review of assessing the accuracy of classifications of
remotely sensed data, Remote Sensing of Environment, 37 :35-46.
Defries, R.S., and J.R.G. Townshend, 1994. NDVI-derived land cover
classifications at a global scale, International Journal of Remote Sensing,
l5(17):3567-3586.
DeFries, R.S., Field, C.B., Fung, I, Justice, C.O., Los, S., Matson, P.A.,
Matthews, P.A., Mooney, H.A., Potter, C.S., Prentice, K., Sellers, P.J.,
Townshend, J.R.G., Tucker, C.J., Ustin, S.L., and P.M. Vitousek, 1995.
Mapping the land surface for global atmosphere-biosphere models:
Toward continuous distributions of vegetation's functional properties,
Journal of Geophysical Research, 100(D10):20,867-20,882.
Foody, G.M., 1992. On the compensation for chance agreement in image
classification accuracy assessment, Photogrammetric Engineering and
Remote Sensing, 58(10): 1459-1460.
Gopal, S., and C.E. Woodcock, 1994. Theory and Methods for Accuracy
Assessment of Thematic Maps Using Fuzzy Sets, Photogrammetric
Engineering and Remote Sensing, 60(2): 181- 188.
Gopal, S., C.E. Woodcock, and G. Unis, 199?. Labeling Map Polygons Using
Remote Sensing: Calibrating Decision Rules, submitted to International
Journal of Geographical Information: Analysis, Theory and Decision,
November, 1994, revised August, 1995, revised again November 1995, in
press.
Hay, A.M., 1988. The derivation of global estimates from a confusion matrix,
International Journal of Remote Sensing, 9: 1395-1398.
Jupp, D.L.B., 1989. The stability of global estimates from confusion matrices,
International Journal of Remote Sensing, 1O(9):1563-1569.
Ma, 2. and R.L. Redmond, 1995. Tau coefficients for accuracy assessment of
classification of remote sensing data, 1995, Photogrammetric Engineering
and Remote Sensing, 6 1(4):435-439.
Moody, A., and C.E. Woodcock, 1994. Scale-Dependent Errors in the
Estimation of Land-Cover Proportions -- Implications for Global LandCover Datasets, Photogrammetric Engineering and Remote Sensing,
60(5):585-594.
Moody, A, Gopal, S. and Strahler, A., 1995. Sensitivity of neural network
outputs to subpixel land-cover mixtures in coarse resolution satellite data,
submitted to Remote Sensing of Environment, July 1994, revised,
September, 1995.
Roberts, D. A., M. 0 . Smith, and J. B. Adams, 1993. Green Vegetation,
Nonphotosynthetic Vegetation, and Soils in AVIRIS Data, Remote Sensing
of Environment, 44:222-269.
Turner, M., O'Neill, R., Gardner, R. and B. Milne, 1989. Effects of changing
spatial scale on the analysis of landscape pattern, Landscape Ecology,
3(3/4): 153-162.
Woodcock, C.E., and S. Gopal, 199?. Fuzzy Set Theory and Thematic Maps:
Accuracy Assessment and Area Estimation, in preparation for
International Journal of GIs.
Zadeh, L., 1965. Fuzzy Sets, Information and Control, 8:338-353.