Digital Image Processing FW5560 Lecture 17 Accuracy Assessment Information derived from remotely sensed data are important for environmental models at local, regional, and global scales. The remote sensing–derived thematic information may be in the form of thematic maps or statistics derived from area-frame sampling techniques. The thematic information must be accurate because important decisions are made throughout the world using the information. Unfortunately, the thematic information contains error. Remote sensing–derived thematic maps should normally be subjected to a thorough accuracy assessment before being used in scientific investigations and policy decisions. It is first necessary to clearly state the nature of the thematic accuracy assessment problem at hand, including: • what the accuracy assessment is expected to accomplish, • the classes of interest (discrete or continuous), and • the sampling design and sampling frame. A simple random sample is selected so that all samples of the same size have an equal chance of being selected from the population. Systematic sampling is a statistical method involving the selection of elements from an ordered sampling frame. Cluster sampling involves selecting the sample units in groups. Stratified systematic unaligned sample Stratified sampling involves selecting independent samples from a number of subpopulations, group or strata within the population. Great gains in efficiency are sometimes possible from judicious stratification. Proportionate allocation uses a sampling fraction in each of the strata that is proportional to that of the total population. Optimum allocation (or Disproportionate allocation) - Each stratum is proportionate to the standard deviation of the distribution of the variable. Larger samples are taken in the strata with the greatest variability to generate the least possible sampling variance. Characteristics of a typical error matrix consisting of k classes and N ground reference test samples. The ideal situation is to locate ground reference test pixels (or polygons if the classification is based on human visual interpretation) in the study area. These sites are not used to train the classification algorithm and therefore represent unbiased reference information. It is possible to collect some ground reference test information prior to the classification, perhaps at the same time as the training data. But the majority of test reference information is often collected after the classification has been performed using a random sample to collect the appropriate number of unbiased observations per Category. Fitzpatrick-Lins (1981) suggests that the sample size N to be used to assess the accuracy of a land-use classification map be determined from the formula for the binomial probability theory: Z 2 ( p)(q) N E2 where p is the expected percent accuracy of the entire map, q = 100 – p, E is the allowable error, and Z = 2 from the standard normal deviate of 1.96 for the 95% two-sided confidence level. For a sample for which the expected accuracy is 85% at an allowable error of 5% (i.e., it is 95% accurate), the number of points necessary for reliable results is: Z 2 ( p)(q) N E2 22 (85)(15) N a minimumof 203points. 2 5 With expected map accuracies of 85% and an acceptable error of 10%, the sample size for a map would be 51: 22 (85)(15) N 51 points 2 10 Evaluation of Error Matrices After the ground reference test information has been collected from the randomly located sites, the test information is compared pixel by pixel (or polygon by polygon when the remote sensor data are visually interpreted) with the information in the remote sensing–derived classification map. Agreement and disagreement are summarized in the cells of the error matrix. Information in the error matrix may be evaluated using simple descriptive statistics or multivariate analytical statistical techniques. Descriptive Statistics The overall accuracy of the classification map is determined by dividing the total correct pixels (sum of the major diagonal) by the total number of pixels in the error matrix (N ). Computing the accuracy of individual categories, is more complex because the analyst has the choice of dividing the number of correct pixels in the category by the total number of pixels in the corresponding row or column. Total number of correct pixels in a category is divided by total number of pixels of that category as derived from the reference data (i.e., the column total). This statistic indicates the probability of a reference pixel being correctly classified and is a measure of omission error. This statistic is the producer’s accuracy because the producer (the analyst) of the classification is interested in how well a certain area can be classified. Descriptive Statistics If the total number of correct pixels in a category is divided by the total number of pixels that were actually classified in that category, the result is a measure of commission error. This measure, called the user’s accuracy or reliability, is the probability that a pixel classified on the map actually represents that category on the ground. Discrete Multivariate Analytical Techniques Have been used to statistically evaluate the accuracy of remote sensing–derived classification maps and error matrices since 1983. The techniques are appropriate because remotely sensed data are discrete rather than continuous and are also binomially or multinomially distributed rather than normally distributed. Statistical techniques based on normal distributions simply do not apply. It is important to remember, however, that there is not a single universally accepted measure of accuracy but instead a variety of indices, each sensitive to different features. Kappa Analysis Khat Coefficient of Agreement: Kappa analysis yields a statistic, which is an estimate of Kappa. It is a measure of agreement or accuracy between the remote sensing –derived classification map and the reference data as indicated by a) the major diagonal, and b) the chance agreement, which is indicated by the row and column totals (referred to as marginals). Conditional Kˆ Coefficient of Agreement can be used to calculate agreement between the reference and remote sensing– derived data with chance agreement eliminated for an individual class for user accuracies using the equation: N ( xii ) xi xi ˆ K N ( xi ) xi xi where xii is the number of observations correctly classified for a particular category (summarized in the diagonal of the matrix), xi+ and x+i are the marginal totals for row i and column i associated with the category, and N is the total number of observations in the entire error matrix. For example, the conditional coefficient of agreement for the residential land-use class of the Charleston, SC, dataset is: 407(70) 88 73 ˆ K Residential 0.75 407(88) 88 73 This procedure can be applied to each land-cover class of interest.