Scoring by cuiliqing


									                                             Regulatory Scoring
Rule title:
Colorado Roadless Areas
0596-AC74                                                                     RIA Separate?         Yes
Stage                                                                         Publication Date
Proposed Rule                                                                 4/15/2011
Rule summary:
The Forest Service, U.S. Department of Agriculture (USDA), is proposing to establish a state-specific rule to
provide management direction for conserving and managing inventoried roadless areas on National Forest
System (NFS) lands in Colorado. A proposed rule was published in the July 25, 2008, Federal Register. In
response to public comment on the 2008 proposed rule and a revised petition submitted by the state of
Colorado on April 6, 2010, the Forest Service is publishing a new proposed rule.

Openness                                                                       Score     Comments
1. How easily were the RIA, the proposed rule, and any supplementary
materials found online?                                                        1         1A
2. How verifiable are the data used in the analysis?                           2         1B
3. How verifiable are the models and assumptions used in the analysis?         2         1C
4. Was the Regulatory Impact Analysis comprehensible to an informed
layperson?                                                                     2         1D

Total Openness (Sum of 1-4)                                                    7

Analysis                                                                    Score        Comments
5. How well does the analysis identify the desired outcomes and demonstrate
that the regulation will achieve them?                                      3            2A

6. How well does the analysis identify and demonstrate the existence of a
market failure or other systemic problem the regulation is supposed to solve? 0          2B
7. How well does the analysis assess the effectiveness of alternative
approaches?                                                                   4          2C
8. How well does the analysis assess costs and benefits?                      2          2D

Total Analysis (Sum of 5-8)                                                    9

Use                                                                            Score     Comments
9. Does the proposed rule or the RIA present evidence that the agency used
the Regulatory Impact Analysis?                                                1         3A
10. Did the agency maximize net benefits or explain why it chose another
alternative?                                                               1   3B
11. Does the proposed rule establish measures and goals that can be used
to track the regulation’s results in the future?                               0        3C

12. Did the agency indicate what data it will use to assess the regulation’s
performance in the future and establish provisions for doing so?               1        3D

Total Use (Sum of 9-12)                                                        3

Total Score                                                                        19
Criterion                        Score Com. No. Comment
                                                The NPRM and the environmental
                                                impact assessment, but not the RIA,
                                                turn up with a keyword search in
                                       There is no link to the
                                                RIA in the NPRM. Much of the
                                                economic analysis is actually in the
                                                environmental impact analysis, which
                                                is linked. The USDA home page has a
1. How easily were the RIA,                     landing page for the roadless rule that
the proposed rule, and any                      has links to all the relevant
supplementary materials                         documents, but this is difficult to find
found online?                    1     1        without a phone call to the agency.
                                                Most data are not directly sourced in
                                                the RIA; the RIA refers readers to the
                                                environmental impact analysis. The
                                                environmental impact analysis says
                                                that full supporting documents are
                                                available at the Forest Service office
                                                in Lakewood, CO; some are available
                                                online. Data in the environmental
                                                impact statement are usually sourced
                                                to public documents or internal
2. How verifiable are the data                  databases that are not very
used in the analysis?            2     2        transparent.

                                                   The RIA mostly presents conclusions
                                                   from the environmental impact
                                                   statement without page references. It
                                                   usually makes assertions about cause
                                                   and effect (e.g., some tree cutting
                                                   reduces fire damage or increases
                                                   safety) with no citation to studies or
                                                   sources. The RIA includes a short list
                                                   of references, most of which do not
                                                   have links. Some parts of the
                                                   environmental impact statement
                                                   simply list assumptions or conclusions
                                                   with no documentation; others are well-
                                                   documented. The environmental
3. How verifiable are the                          impact statement has an extensive list
models and assumptions                             of references, about one-quarter of
used in the analysis?            2      3          which are linked.
                               The text of the 16-page executive
                               summary does not clearly explain
                               what the alternatives are or what kinds
                               of activities count as costs versus
                               benefits. Many unfamiliar acronyms
                               and jargon make understanding
                               difficult. The environmental impact
                               analysis is much better written and
                               much more informative than either the
                               NPRM or the RIA. Results are hard to
                               understand because the results of the
4. Was the analysis            three alternatives to the baseline are
comprehensible to an           not presented as differences from the
informed layperson?    2   4   baseline.
                                Score   Com. No. Comment

5. How well does the
analysis identify the desired
outcomes and demonstrate
that the regulation will
achieve them?                   3
                                                  Potential benefits named in the RIA include a list of
                                                  private benefits from permitted activities, non-
                                                  market environmental benefits, and changes in
                                                  Forest Service costs. A table lists nine "values"
Does the analysis clearly
                                                  offered by public commenters that reflect both
identify ultimate outcomes                        private and environmental benefits. It is not always
that affect citizens’ quality of                  clear whether these count as benefits or costs of
life?                            4      5A        the regulation.
                                                  Except for the value of output from mineral leases,
                                                  the outcomes are not usually measured. The RIA
                                                  mostly describes how the alternatives would affect
                                                  the amount of activity (e.g., tree cutting, road
                                                  construction) that might affect these outcomes.
                                                  Most of the RIA's assessments (e.g., more or less
                                                  risk of fire, threats to species, water quality)
                                                  presume an automatic link between activity and
                                                  outcome that is often documented in the
                                                  environmental impact statement but not the RIA.
Does the analysis identify                        The RIA claims effects are described qualitatively
how these outcomes are to                         due to the absence of site-specific proposed
be measured?                 3          5B        projects.
Does the analysis provide a
                                                  There are a lot of assertions in the RIA about what
coherent and testable theory
                                                  will lead to what without much of a theory
showing how the regulation                        explaining why. The environmental impact
will produce the desired                          assessment has better explanations of how road
outcomes?                    4          5C        building and tree cutting affect various outcomes.
                                                  Only the environmental impact assessment cites
                                                  some sources (but not much data) in support of
                                                  claims that various activities will have various
                                                  effects. Rarely is there enough information to
Does the analysis present                         determine if the particular differences between
credible empirical support for                    alternatives can be expected to generate large or
the theory?                    2        5D        small effects.
                                                  The RIA notes that economic projections are
Does the analysis
                                                  approximations but does not offer an uncertainty
adequately assess                                 analysis. It also notes that production of
uncertainty about the                             commodities or amenity services will vary with
outcomes?                      2        5E        changing market conditions.
6. How well does the
analysis identify and
demonstrate the existence of
a market failure or other
systemic problem the
regulation is supposed to
solve?                       0
                                        This should have been easy to do but was not
                                        done. Rulemaking is a response to a petition from
                                        the state of Colorado seeking state-specific
                                        regulations to manage roadless areas. The
                                        "Purpose and Need" section lists many reasons
                                        someone should manage the roadless areas but
                                        does not describe a systemic problem with the
                                        state of current management. It just says the state
Does the analysis identify a            and the Forest Service agree there is a need. The
market failure or other                 2008 RIA came closer to naming a systemic
systemic problem?              1   6A   problem.

Does the analysis outline a             The RIA might imply that management of roadless
coherent and testable theory            areas under existing "forest plans" might not
                                        adequately reflect state-specific and local needs
that explains why the
                                        (as the 2008 RIA implied in regard to its baseline
problem (associated with the            2001 rule). This might have been developed into
outcome above) is systemic              some theory explaining inadequacy of the baseline
rather than anecdotal?       0     6B   approach, but this was not done.
Does the analysis present
credible empirical support for           The only evidence is an assertion that the Forest
the theory?                    0    6C   Service and the state agree a regulation is needed.
Does the analysis
adequately assess
uncertainty about the
existence or size of the
problem?                       0    6D   No relevant content.
7. How well does the
analysis assess the
effectiveness of alternative
approaches?                    4

                                         The environmental impact statement identifies
Does the analysis enumerate              seven alternatives, four of which are chosen for
other alternatives to address            further analysis. RIA examines four alternatives,
the problem?                  5     7A   one of which is the baseline.
                                         The alternatives are very narrow, with little scope
Is the range of alternatives             for flexibility. Analysis considers two different
considered narrow (e.g.,                 inventories of roadless areas, different amounts of
some exemptions to a                     acreage restricted to "upper tier," and several
                                         different sets of authorized or prohibited activities,
regulation) or broad (e.g.,
                                         such as tree cutting and roads for oil, gas, and
performance-based                        mineral leases. The three regulatory alternatives
regulation vs. command and               are all variations on the same basic approach,
control, market mechanisms,              rather than different forms of regulation.
nonbinding guidance,                     Differences in the amount of road construction and
                                         tree cutting appear to be very small over several
information disclosure,
                                         million acres. Thus, the alternatives vary primarily
addressing any government                in scope of the rule from status quo to most
failures that caused the                 expansive application, as well as restrictiveness of
original problem)?           3      7B   permitted activities.

                                         The analysis identifies outcomes produced by each
Does the analysis evaluate
                                         alternative for each issue considered. This was
how alternative approaches               usually ordinal, since not much was measured. But
would affect the amount of               the assessment is quite extensive, given the large
the outcome achieved?          4    7C   number of values considered.
                                         The baseline is no state-specific regulation, with all
                                         roadless areas managed according to "forest
Does the analysis                        plans." The RIA does not explain what this means
adequately address the                   (unlike the 2008 RIA); the environmental impact
baseline? That is, what the              study has a brief explanation. The analysis offers
                                         many comparisons of the alternatives but does a
state of the world is likely to
                                         poor job of showing the net effects of the
be in the absence of federal             alternatives relative to the baseline. One page of
intervention not just now but            text does this comparison for oil, gas, and coal
in the future?                  3   7D   production.
8. How well does the
analysis assess costs and
benefits?                    2

                                      The RIA mostly reports changes in activity, not
                                      costs. Effects on mineral production and on other
                                      values under different alternatives could have been
                                      presented as opportunity costs if the RIA had
                                      properly compared the three alternatives to the
                                      baseline. Unlike the 2008 RIA, it does not quantify
Does the analysis identify
                                      value of property at risk from fire, a potential cost of
and quantify incremental              restricting access. It is surprising the RIA did not
costs of all alternatives             employ any techniques for quantifying or
considered?                  2   8A   monetizing the value of non-market goods.
                                        The only expenditures discussed are Forest
                                        Service expenditures. Changes in expenditures in
                                        roadless areas are assumed to be matched by
                                        reductions in other areas due to budget
Does the analysis identify all          constraints. No assessment of expenditures
expenditures likely to arise            related to opportunity costs, such as cost of forest
as a result of the regulation? 3   8B   fires.
                                        Price effects for mining/energy are considered, and
                                        the Forest Service concludes these are negligible.
                                        The preamble states that there will be no
                                        noticeable change in prices since changes in
Does the analysis identify
                                        production will not occur until well into the future.
how the regulation would                Nothing else considered, even though wild fire risk
likely affect the prices of             could have a big effect on housing and insurance
goods and services?           1    8C   prices.

Does the analysis examine               Changes in human behavior are discussed as far
                                        as how the range of alternatives will affect industry
costs that stem from
                                        and human enjoyment of wildlife. Due to the
changes in human behavior               unclear baseline, it is not always clear which
as consumers and producers              outcomes count as benefits and which ones count
respond to the regulation? 2       8D   as costs.

If costs are uncertain, does
                                        The analysis states that projections are uncertain,
the analysis present a range            and to the extent that this statement applies to
of estimates and/or perform             outcomes that might be opportunity costs, it is an
a sensitivity analysis?      1     8E   acknowledgement of cost uncertainty.

                                        There is no single net benefit figure. An extensive
Does the analysis identify              table lists how the different alternatives accomplish
the alternative that                    more or fewer of various activities or outcomes. So
maximizes net benefits?       2    8F   it is a step toward a net benefit comparison.

Does the analysis identify
the cost-effectiveness of               Since outcomes were not usually quantified, no
each alternative considered? 0     8G   analysis of cost-effectiveness was done.

                                        The RIA calculates total value of oil, gas, and coal
                                        development; jobs; labor income; and state and
                                        local mineral lease revenues. These address only
                                        one effect of the rule, on mineral/energy
                                        development. We had to infer that these are the
                                        main cost categories because the analysis does
Does the analysis identify all
                                        not explain which values count as costs and which
parties who would bear costs            ones count as benefits. Maps and tables suggest
and assess the incidence of             how some costs and benefits may be distributed
costs?                         3   8H   geographically.
                                        Values associated with roadless characteristics are
                                        listed and commenters who advocated these
                                        values are discussed. We had to infer that these
                                        are the main benefit categories because the
                                        analysis does not explain which values count as
                                        costs and which ones count as benefits. The
                                        analysis discusses benefits to wildlife and how
                                        scenery improves the quality of people's lives. Also
Does the analysis identify all
                                        some discussion of how industry benefits under
parties who would receive               each alternative. Maps and tables suggest how
benefits and assess the                 some costs and benefits may be distributed
incidence of benefits?         2   8I   geographically.
Criterion                     Score Com. No. Comment

                                              The RIA is mentioned in the NPRM but never
                                              cited as a reason for any decision. The proposed
                                              rule is a revision of a 2008 proposed rule. The
                                              2008 rule never became final because the state
                                              of Colorado decided to revise its petition after
                                              soliciting public comments on the proposed
                                              federal rule. The NPRM states that the proposed
9. Does the proposed rule or
                                              rule reflects public comments and the state's
the RIA present evidence                      revised petition. The RIA does not appear to have
that the agency used the                      influenced the decisions about the regulation, and
analysis?                    1     9          the Forest Service does not claim to have used it.

                                              The RIA provides some basis for assessing net
                                              benefits qualitatively. However, neither the RIA
                                              nor the NPRM explains how the Forest Service
                                              made tradeoffs among the various values. No
                                              basis is explained for making the changes from
10. Did the agency maximize                   2008, other than that Colorado revised its petition
net benefits or explain why it                in response to public comments. Clearly, analysis
chose another alternative? 1       10         of net benefits played no role here.
11. Does the proposed rule
                                              No measures or goals were established. With
establish measures and
                                              some additional work and decisions about
goals that can be used to                     priorities, the RIA could be used to establish
track the regulation's results                measures and goals, but measures and goals
in the future?                 0   11         are not immediately obvious.
12. Did the agency indicate                   The RIA contains no discussion of data to be
what data it will use to                      used to measure future performance. The RIA
                                              suggests that the Forest Service may have
assess the regulation's
                                              access to data on some but by no means all
performance in the future                     outcomes/values affected by the regulation.
and establish provisions for                  These could be used to determine whether the
doing so?                      1   12         rule is having the desired effects.
Rule Title RIN        Agency Pub Date RIA separate? Total (G+H+J) Openness
Colorado Roadless Areas USDA 4/15/2011 Yes                     19            7
Analysis       Quality (G+H) Use       1   2   3   45
           9                16     3   1   2   2   2    3
5A       5B       5C       5D       5E       6       6A       6B       6C
     4        3        4        2        2       0        1        0        0
6D       7       7A       7B       7C       7D       8       8A       8B
     0       4        5        3        4        3       2        2        3
8C       8D       8E       8F       8G       8H       8I       9       10
     1        2        1        2        0        3        2       1        1
11       12
     0        1

To top