Document Sample
schaefer-etal Powered By Docstoc
					 A New Approach For Testing the Accuracy
   of Vertebrate Occurrence Predictions

 Sandra M. Schaefer
   William B. Krohn
Raymond J. O’Connor

 Maine Cooperative Fish and
  Wildlife Research Unit, and
Department of Wildlife Ecology
  University of Maine, Orono

Traditionally focused on single species being
  predicted on relatively small study areas
Habitat Model Testing

    Multiple Species

 Large geographic areas
Traditionally, GAP models
are tested by comparing model
predictions to site-specific
occurrence information

Omission Error (OE) – failure
to predict species known to occur
on the site.

Commission Error (CE) – predicting
the presence of species that do
not occur on the site.
     ME-GAP Test Sites

1. North Maine Forestlands
2. Nesowadnehunk Field,
   Baxter State Park
3. White Mountains National Forest
4. Sunkhaze Meadows
   National Wildlife Refuge
5. Holt Research Forest
6. Petit Manan
   National Wildlife Refuge
7. Rachel Carson
   National Wildlife Refuge
8. Moosehorn
   National Wildlife Refuge
9. Mount Desert Island/
  Acadia National Park
               Site-Specific calculations

CE = # of species predicted but not present on the test site
           Total number of species present on the site

OE = # of species present but not predicted on the test site
           Total number of species present on the site
Overall results of accuracy assessment on ME-GAP predicted
species distributions. Medians and ranges were calculated within
taxonomic group across all sites.

Taxonomic Class            O mission (% )                Commission (% )
(number of sites)   Median             Range      Median             Range
A mphibians (2)      0.0                    –      0.0                 –
Reptiles (3)         10.0            0.0 - 20.0    5.0             0.0 - 10.0
Mammals (4)          5.4             3.0 - 11.4    18.9            11.4 - 36.4
Birds (9)            0.0              0.0 - 5.0    91.9           17.4 - 138.2
All Classes (19)     0.0             0.0 - 20.0    34.2            0.0 - 138.2
                Challenges in Testing

• Errors can be caused by multiple factors (i.e test site
 size, field inventory effort, species biology).

• Purpose of the model needs to be considered in testing,
 because it may influence how the errors are interpreted.

• Interpretation can be complex. Especially for commission
 error where the cause may be either apparent or actual.
Species-Specific Testing Approach
 Assessing model accuracy by calculating
OE and CE for each species across multiple
        sites within the study area.
               Species-Specific calculations

CE = # of sites where the species was predicted but not present
              Total number of potential occurrence sites

OE = # of sites where the species is present but not predicted
            Total number of potential occurrence sites
         How complete are the field inventories?

                     Error Range (ER)

   Difference in the highest and lowest possible OE and CE.
Calculated based on assumptions of field inventory completeness
                      (Nichols et al. 1998).

  complete – assumes that all species occurring on the site
            were found during the field inventories.

 incomplete – assumes that not all species on the site were
              found during the field inventories.
1) Calculate species-specific ER for avian species known to
   regularly breed in Maine.

2) Determine if there is a relationship between the ER
   and the extent of a species distribution.

3) Determine if there is a relationship between the ER and
   how likely a species is to occur during a field survey
  (for statewide species).

4) Compare the test results from the species-specific and the
   site-specific approaches.
Calculating the species-specific ER
    Potential Occurrence

Species could potentially occur
on a site if the site is within the
range limit.

Field survey data was also
included if it indicates the
species occurs on the site.
                 Species Occurrence Table

  Green heron                                   Sites
                     1      2      3      4      5      6   7   8   9
RANGE                --     --     1      1      1      1   1   1   1
SUR VEY              --     --     A      P      A      A   P   P   P
PREDICTION           --     --     P      P      P      P   P   P   P

1 = Within ME-GAP Range   P = Presence   A = Absence
Data Analysis
1) Spearman’s Rho used to test for a relationship between a
    species distribution and the commission ER.

2) Spearman’s Rho used to test for a relationship between an a
   a priori ranking system called Likelihood of Occurrence Ranks
   and the commission ER.

3) Site-specific commission error results compared to the
   species-specific error ranges.

    Grouped into primary breeding category (i.e., barren, early successional
   forest coniferous, forest deciduous, forest generalists, and wetland).

   Mean error for the site-specific method was plotted against mean error for
   species-specific method.

  One-way ANOVA was used to determine if a significant difference existed
  between the means of the 2 methods.
       Likelihood of Occurrence Ranks (LOORs)
                             (Boone and Krohn 1999)

LOORs are an a priori system of ranking species based on
how likely they are to be seen during a standard wildlife inventory.

Developed to help interpret the causes of commission error.
(Schaefer and Krohn 2002).

Atlas occurrence information was used to generate a spatial
incidence for each species. The incidence came from dividing the
number of survey blocks in the atlas having confirmed or potential
breeding by the number of survey blocks within the species range.
                          Assumes Complete Field Survey         Assumes Incomplete Field Survey
                    A.                                      B.

                    180                                    180
                    160                                    160
                    140                                    140
                    120                                    120
                    100                                    100
                    80                                      80
                    60                                      60
Number of Species

                    40                                      40
                    20                                      20
                     0                                       0
                          0   0.2    0.4   0.6   0.8   1          0    0.2    0.4    0.6      0.8   1
                              Commission Error                           Commission Error

                    C.                                     D.

                    120                                    130
                     80                                     80
                          0   0.2    0.4   0.6   0.8   1    -20   0    0.2     0.4   0.6      0.8   1

                                    Omission Error                           Omission Error

Frequency distribution of OE and CE from predicted avian occurrences
with the assumption of complete (A and B) and incomplete (C and D) field survey data.
                                    120                                                      C

                Number of Species
                                     50                                          B
                                     40           A
                                          1   2        3      4     5        6       7   8   9
                                                      Number of Test Sites

Frequency distribution of the number of test sites on which each bird species could
potentially occur based on range limits in Maine. A= those species with limited distribution,
B= those species that are moderately distributed, and C= those species that are statewide.
    Commission Error Range

                             Number of potential occurrence sites

Commission ER for all avian species across all test sites.
            rho = -0.583      P < 0.001
Commission Error Range

                                  Likelihood of Occurrence Ranks

                         Relationship between CER and the LOORs.
                                  rho = -0.657 P < 0.001
       Mean Commission Error



                                        Early         Forest      Forest      Forest      Wetlands   Barren   All Habitats
                                     Successional   Deciduous   Coniferous   Generalist

                                                                      Habitat Categories

Means and 95% confidence interval of CE, by major breeding habitats, for the
site-specific (light bars) and the species-specific (dark bars) approaches to testing
predicted avian occurrences.

• Calculating species-specific ER provided an opportunity
  to assess the overall predictive quality of the habitat models, as well
  as determine the variability of error for each species.

• CER was significantly correlated with species distribution as
  well as with how likely a species was to be observed on
   a field inventory.
                       Conclusions (cont)

• If a high ER is reported for a species that has a high likelihood
  of occurrence then the most likely cause for the over prediction is
  in the model.

• However, if a species has a low likelihood of occurrence
  and a high ER, then the over prediction error is likely due to
  having incomplete field surveys for the species.

•   Both methods are influenced by the data available for use in the
    testing process.

•    The site-specific method provides a generalized idea of how well
     the models are capturing species presence and absence
    and across the entire state.

•   The species-specific approach gives a more detailed description
    of which species are reporting the highest levels of error.
    This helps to answer the question of why the predictive error
    is being reported.
                  Take Home

 I recommend using both methods in assessing model
accuracy because the two approaches provide different
information about the quality of predicted occurrences.
                  William B. Krohn
                Raymond J. O’Connor

                  Daniel J. Harrison
                   Steve R. Sader
                  Randall B. Boone
                  William Haulteman

Supporting Organizations and Agencies
          Gap Analysis Program, USGS BRD
   Maine Cooperative Fish and Wildlife Research Unit
           Department of Wildlife Ecology
                 University of Maine

Shared By: