Classification Accuracy Assessment

Document Sample
Classification Accuracy Assessment Powered By Docstoc
					ENVS720: Lecture 9

 Digital Image Processing
 Classification Accuracy Assessment


             ENVS720
           Classification
        Accuracy Assessment

• Classification error matrix (confusion matrix,
  contingency table)

• Provides a comparison on a category-by-
  category basis of classification results vs.
  known reference (ground-truth) data


                      ENVS720
                                Classification
                             Accuracy Assessment
                              Ground Truth (Known Types)

                                A   B       C         D     E
Classification Results




                         A      100 0       0         0     0     100
                         B      0     76    0         3     0     79
                         C      0     20    90        2     0     112
                         D      0     4     0         95    0     99
                         E      0     0     10        0     100   110
                                100   100   100       100   100   500

                                            ENVS720
           Classification
        Accuracy Assessment

• Major diagonal: proper assessment
  – A, E  100%


• Non-diagonal elements: errors




                    ENVS720
         Two Types of Errors
1. Omission (exclusion)
  –   Errors down columns (B: 24%)
  –   Opposite is producer’s accuracy (B: 76%)


2. Commission (erroneous inclusion)
  –   Errors along rows (B: 3/79, 3.8%)
  –   Opposite is user’s accuracy (B: 76/79, 96.2%)



                        ENVS720
           Overall Accuracy

Dividing the total number of correctly classified
pixels by the total number of reference pixels.
(461/500) * 100% = 92.2%

What is acceptable accuracy?
No rule, depends on purpose of study.
Generally >=80% acceptable (land cover studies)

                      ENVS720
                 Problem
• Depends on training data set!

• Might not reflect properly what happens
  outside the training data set

• Note: Ideal assessment (one-to-one) is utopic



                     ENVS720
                Test Areas
• Areas of representative, uniform land cover,
  different from the training areas.

• Used for postclassification accuracy
  assessment




                     ENVS720
                 Sampling
• Methods:

  – Random
  – Systematic
  – Combination of the two




                      ENVS720
                  Sampling
• Parameters:

  – Sample unit (pixels, clusters, polygons)

  – Sample size (e.g. 50 samples of each class, and
    increase this number for larger areas)




                        ENVS720
            Evaluating
   Classification Error Matrices

• Overall measure of accuracy
• Measures of accuracy for individual classes

• Percentage accuracy
• Chi-squared
• Kappa analysis yields K-hat index   (estimate of Kappa)



                     ENVS720
                 Chi-squared
                        Observed  Expected
             2   [                       ]2
                             Expected

                              Chi-squared is never negative!


• Tests the extent of agreement/ contingency
  between observed (classed pixels) and expected
  (ground-truthed pixels).

• Ideal case: chi-squared  0 (check significance of
  test, look at probability)

                           ENVS720
                K-Hat Index
         ˆ ObservedAccuracy ChanceAgreement
         k
                  1 ChanceAgreement      ˆ
                                      0  K 1
                                  K-hat can also be negative!


• Shows the extent to which the correct values of an
  error matrix are due to “true” vs. “chance”
  agreement.

• Ideal case: c.a.  0, o.a.  1, K-hat  1


                        ENVS720
                      K-Hat Index
                          r               r
                      N  x ii   (x i  x i )
                ˆ
                k       i1              i1
                                     r
                         N   (x i  x i )
                              2

                                    i1
•   r: # of rows, columns in error matrix
•   N: total # of observations in error matrix
•   xii: major diagonal element for class I

•   xi+: total # of observations in row i (right margin)
•   x+i: total # of observations in column i (bottom margin)
                                  ENVS720
              K-Hat Index
• K-hat provides a basis for determining the
  statistical significance of any given
  classification matrix

• Quality of accuracy estimate depends on the
  quality of the info used as ground truth
  (which has its own accuracy estimate)


                     ENVS720
        Accuracy Assessment
• Accuracy assessment as a function of
  intended use

• Error accumulation




                       ENVS720

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:15
posted:8/8/2012
language:English
pages:16