Get documents about "ria-assessing-options
Document Sample
Guidelines on
Assessing Policy Options
March 2007
Regulatory Impact Analysis Unit
Effective Markets Branch
ISBN 978-0-478-30496-1 (HTML)
ISBN 978-0-478-30497-8 (PDF)
617695
Guidelines on Assessing Policy Options
Outline of document .....................................................................................................2
Flow chart ....................................................................................................................3
Introduction ..................................................................................................................4
Purpose of this document .........................................................................................4
Audience ..................................................................................................................4
Cost Benefit Analysis framework..................................................................................5
What is cost benefit analysis? ..................................................................................5
Cost benefit analysis is only one input into the decision making process.................5
Levels of cost benefit analysis ..................................................................................6
Mainly textual/qualitative cost benefit analysis......................................................6
Formal quantitative cost benefit analysis ..............................................................6
Alternatives to cost benefit analysis..........................................................................7
Where there are multiple objectives......................................................................7
Cost effectiveness criterion ...................................................................................7
Steps to take ................................................................................................................8
1 Establish the counterfactual (status quo)...............................................................8
2 Identify the range of feasible options .....................................................................8
3 Identify and understand the impacts of each option ..............................................8
4 Assess the magnitude of each impact of each option..........................................11
5 Review the options ..............................................................................................14
6 Compare the options and identify the best option................................................14
7 Present the results...............................................................................................16
What level of analysis is required?.............................................................................16
Appendix 1: Guidance on information gathering and brainstorming...........................18
Appendix 2: Principles to follow..................................................................................19
Appendix 3: Converting values to their ‘present value’...............................................20
Outline of document
This document provides guidance on assessing options in policy development. It is relevant to
all kinds of policy development – social, economic, environmental, and so on.
It introduces the generic cost benefit analysis framework (identify the options, identify the
impacts, assess the magnitude of the impacts, choose the best option), and explains how this
should be used in policy development, regardless of the subject matter. The important message
is that the level of analysis (ie: whether the analysis is mainly textual/qualitative or mainly
quantitative) needs to be commensurate with the significance of the proposal (which is
determined by how large the impacts of the proposal are and how controversial the proposal is).
The regulatory impact analysis (RIA) requirements for assessing options are set out, as these
provide standards that departments need to meet when developing regulatory proposals (ie:
proposals that alter legislation or regulations).
The appendices contain guidance on information gathering and brainstorming for assessing
options, principles to follow when assessing options, and guidance on converting values to their
‘present value’.
ria-assessing-options 2
Flow chart
Step in cost benefit analysis Relevant pages
1 Analyse the status quo 8
Identify the full range of
2 8
feasible options
Identify and understand the
3 8-11
impacts of each option
Assess the magnitude of all 11-14
options’ impacts
4
• How much analysis is 6-7, 16-17
required?
5 Review the options 14
Compare the options and
6 14-15
identify the best option
7 Present the results 16
ria-assessing-options 3
Introduction
1. In a robust policy development process, you need to define the problem, set your objectives,
consider the full range of feasible options, and then assess these options to determine which
is preferred. Thorough and accurate assessment and analysis of the impacts of options is a
crucial part of policy development. Without it, the option that is recommended may not be
the best option, which could have wide-ranging adverse impacts on affected parties (be it a
worsening of conditions or a sub-optimal improvement).
2. Assessing options thoroughly is usually a difficult and complex exercise. Most importantly,
you need to ensure enough time and resource are devoted to this step of policy
development and assessment is undertaken in a structured way.
Purpose of this document
3. To improve the quality of policy development by providing guidance on assessing options.
This document sets out the principles you should follow and steps you should take to ensure
that your assessment of options is thorough and accurate. It discusses the basic steps of
cost benefit analysis as they apply to policy analysis. It acknowledges that cost benefit
analysis is only part of the decision making process in a policy environment, though cost
benefit analysis in some form or other assists to ensure that options are assessed robustly.
4. This document is distinct from the RIA Guidelines1, which set out the requirements for RIA,
including the RIS requirement.
Audience
5. The guidance in this document applies to all types of policy – social, economic,
environmental, and so on. All policy analysts should be familiar with this guidance to ensure
that they are able to assess impacts of policy options thoroughly and accurately.
1
Available on the Policy Development Toolkit: https://psi.govt.nz/pdtoolkit/default.aspx
ria-assessing-options 4
Cost Benefit Analysis framework
6. These guidelines suggest using a cost benefit analysis framework because it is simple,
systematic, and ensures that all of the relevant information is put before decision-makers.
What is cost benefit analysis?
7. Cost benefit analysis (CBA) means:
• analysis of the status quo
• identifying the full range of feasible options
• identifying and understanding the impacts of each option (including the status quo)
• assessing the magnitude of all options’ impacts
• comparing the options and identifying the best option (by weighing up the costs and
benefits, often using a net benefit and/or benefit cost ratio criterion).
8. The cost benefit analysis framework is generic enough to be used in all policy areas. Cost
benefit analysis does not necessarily mean calculating the dollar value of all impacts.
Another common misconception is that cost benefit analysis applies primarily (or only) to
economic proposals – this is incorrect: cost benefit analysis can and should be applied to all
types of policy development. It is simple, systematic, and ensures that all of the relevant
information is put before decision-makers; it is not area specific and you do not need a
thorough understanding of economics to undertake a mainly textual/qualitative cost benefit
analysis or quantify some impacts.
Cost benefit analysis is only one input into the decision making process
9. The decision as to which option is best does not stop once you have completed your cost
benefit analysis. A comparison of options by a simple comparison of each option’s costs and
benefits implies that all impacts are valued equally. Sometimes this is not appropriate. In
these cases you should focus on the policy objectives and whether certain impacts should
have a greater or lesser weighting than others.2
10. The source of these objectives or weightings may come from an assessment of society’s
preferences,3 which may be estimated through careful consideration of submissions during
consultation, polling and surveys, or looking at where political representatives stand on
issues. Alternatively, your Minister may wish to include the Government’s preferences, and
leave Parliament to debate the appropriate weights to use. You need to be aware of and
explicit about these weightings, whichever set of preferences is used.
2
Risk weightings. When addressing a risk problem, you should consider, and be explicit about, what risk
profile you are applying to the impacts, ie the level of risk aversion being used. A higher level of risk
aversion means that costs from an adverse event occurring will be more important relative to the costs of
mitigating the risk than with a lower level of risk aversion, so that more should be done to avoid the
adverse event. Scenario analysis might be useful in some circumstances when considering the impact of
different risk profiles (depending on the nature and significance of the option). Equity weightings. Impacts
are likely to be weighted for equity or distributional concerns. To do this, you need to make judgements
about how much weight to give to impacts on different affected parties. This weighting is also important for
how to value distributional impacts, for example an option may be favoured that has the same net benefit
as another but distributes those benefits more equally (who is affected can be as important as the
aggregate impact).
3
Whichever is most appropriate in the situation.
ria-assessing-options 5
Levels of cost benefit analysis
11. There are different levels of CBA. The level you should use depends on how significant the
proposal is. The main principle is that the level of analysis should be commensurate with the
significance/magnitude of the proposal. How significant a proposal is depends on how large
the impacts are and how controversial the proposal will be. For more guidance on the level
of analysis you should undertake, see What Level of Analysis is Required?, on page 16.
12. Carrying out an appropriate level of analysis is important for ensuring that your policy
development is robust, and is also relevant for ensuring compliance with the regulatory
impact analysis (RIA) requirements (see RIA guidelines4).
Mainly textual/qualitative cost benefit analysis
13. A mainly textual/quantitative CBA may be appropriate for less significant proposals. It is
different from quantitative CBA in two ways:
• Whereas quantitative CBA requires that all or most impacts are quantified in dollar terms,
a textual CBA will describe all of the impacts, giving assessments of magnitude where
these are available. It is a ‘weighing up’ exercise – you state the relative size of impacts
compared with other quantifiable or unquantifiable impacts.
• Whereas in quantitative CBA you calculate the net benefit and/or benefit-cost ratio, in a
textual CBA the preferred option is identified by comparing the relative size of impacts
without necessarily making overall calculations.
Formal quantitative cost benefit analysis
14. When to undertake a quantitative cost benefit analysis. You should strive to undertake a
quantitative CBA when the proposal is likely to be significant. For the most significant
proposals, this will be required in order for your analysis to be adequate (as assessed
against the RIA adequacy criteria). Formal quantitative cost benefit analysis can be
expensive so for less significant proposals, a mainly textual analysis may be preferred.
15. What is formal quantitative cost benefit analysis? Formal quantitative cost benefit analysis
means, in addition to identifying the full range of feasible options and all of the impacts of
these options:
• quantifying all impacts in dollar terms
• converting all costs and benefits to their present value (for more detail on calculating the
net present value, see the appendix 3)
• identifying the best option by:
• identifying the option with the highest net benefit (benefits minus costs); and/or
• identifying the option with the highest benefit-cost ratio (benefits divided by costs).5
16. What if I cannot quantify some impacts? For more guidance on how to quantify, see Assess
the magnitude of each impact of each option, page 11. You can still carry out a formal
quantitative CBA even if you are unable to quantify some impacts. One way to factor in
these impacts is to exclude them when you calculate the net benefit or benefit-cost ratio,
then test how high the benefits or costs would have to be to influence your decision. For
example, suppose you are using a net benefit criterion and for one of the options you are
able to quantify all impacts apart from one of the benefits. If you find that this benefit would
4
Available on the Policy Development Toolkit: https://psi.govt.nz/pdtoolkit/default.aspx
5
Note that for both of these tests, the benefits and costs need to be discounted.
ria-assessing-options 6
need to be worth at least $200,000 for the option to be preferred, you can still make a
decision based on a quantitative CBA by estimating whether the benefit would be valued at
$200,000 or more.
17. More guidance on formal quantitative CBA is available in Treasury’s Cost benefit Analysis
Primer, available at www.treasury.govt.nz.
18. For more guidance on the level of analysis you should undertake, see What Level of
Analysis is Required? on page 16.
Alternatives to cost benefit analysis
19. This document provides a brief description of two alternatives to cost benefit analysis, but
the guidance in the rest of the document is only about cost benefit analysis, as this is the
recommended approach.
Where there are multiple objectives.
20. Where multiple objectives are involved (for example, there are a number of criteria in
addition to or instead of a cost benefit criterion), you should assess the options against all of
the objectives. This is often referred to as ‘multi-criteria analysis’. Note that multi-criteria
analysis is different from setting policy objectives and then doing a cost-benefit analysis:
multi-criteria analysis means focussing only on the objectives and largely ignoring other
costs and benefits. If you are assessing options against multiple objectives, it may be useful
to produce a table that indicates how well each option achieves each objective. While
focussing on the objectives of the proposal (rather than the costs and benefits) is a useful
exercise, you should always bear in mind all impacts of a proposal and the potential for
unintended consequences. You should also note that multi-criteria analysis is a useful
complement to cost benefit analysis.
Cost effectiveness criterion
21. Cost effective analysis can be used on those occasions when the government specifies an
objective below which it will not be willing to trade off other objectives. For example, in
introducing competition to the Employers' Account of the ACC Scheme, it was agreed that
claimant access to entitlements was not to be compromised by insurer insolvency. The
appropriate mechanism for assessing options for achieving this objective then becomes cost
effectiveness analysis. That is, the option expected to achieve the specified objective at
least cost. Analysis of benefits, in this case, is not required. Cost effectiveness analysis
should only be used where the different benefits associated with different options is not
important – this is not likely to be often (cost benefit analysis is usually more appropriate).
22. Cost effectiveness analysis measures provide an index of the relative cost to society of
various options for promoting a particular objective (usually expressed as cost per unit of
benefit). Within the context of risk regulation, for example, the task of this approach is to
ascertain which policies minimise the cost of reducing a given risk.
23. A critical difference between cost-effectiveness analysis and quantitative CBA is that, for the
former, benefits need not be valued explicitly. The cost-effectiveness measure calculates the
cost per unit of benefit but otherwise does not assign dollar values to outcomes. The data
needs for cost-effectiveness tests will consequently be less. It should also be noted that cost
effectiveness analysis does not easily deal with multiple objectives.
ria-assessing-options 7
Steps to take
1 Establish the counterfactual (status quo)
24. Before you identify options, you need to establish the status quo (no change from current
policy, including future policies that have already been agreed to), which is the
counterfactual against which all impacts are assessed. It is important to assess all impacts
against the status quo, rather than, for example, comparing costs against a more
prescriptive regulatory option.
25. Ensure you have a thorough understanding of the status quo. Often this is difficult. When
thinking about what the status quo is, you should think not only about what the current
situation is, but also more broadly about what will happen in the future if the government
takes no further action, taking into account any relevant predicted trends. The nature of
society, income distribution, cultures, goods, consumers, technology and other factors may
change over time. For example, if there is a housing shortage, even if government takes no
new action there may be incentives for commercial companies to build houses, which may
partially mitigate the shortage.
26. You should also note that there are two main approaches to comparing the options with the
counterfactual.
• The first is to analyse the options in terms of how they deviate from the status quo. This
means describing them as how they are different from the status quo.
• The second is to analyse the other options as complete packages. This means describing
each option as a whole.
27. Whichever option is chosen, when you present the impacts of the options you should look
only at the costs and benefits that are additional to those that would have been
incurred/received if no further action were taken (ie the marginal costs and benefits). Current
costs and benefits are part of the background.
2 Identify the range of feasible options
28. After you have understood the counterfactual (status quo), you need to identify the full range
of feasible options. Feasible options are possible options that will solve the problem and
meet the public policy objectives. Identifying a number of options is important as it is the only
way of knowing whether what you are proposing is the best of the feasible options – if you
focus on only one option, it is possible, and in a lot of cases, very likely, that the option you
recommend is not the best option (you may have missed a better option). Options need not
be alternatives – often a combination of options will be most effective (eg: regulation
accompanied by an information campaign). Options should be developed in consultation
with affected parties, experts, literature, seeing what is done overseas etc. It is important
that you consider sub-options. Often tweaking an option in one or more ways will improve
the net benefit that can be obtained from the option. More guidance on identifying options is
available in Identifying Policy Options (Ministry of Economic Development).6
3 Identify and understand the impacts of each option
29. Arising from each option is a collection of impacts, which are made up of immediate impacts,
indirect impacts, and dynamic impacts (which arise over time as a result of the environment
changing within which people operate, especially as a result of incentives). The more
impacts you identify, the more comprehensive and accurate your analysis will be. The key
6
Available on the Policy Development Toolkit: https://psi.govt.nz/pdtoolkit/default.aspx
ria-assessing-options 8
principle is that the level of analysis should be commensurate with the significance of the
proposal.
• You need to understand what each impact is and what its characteristics are (timing,
magnitude, and distribution). You need to be thorough in order to avoid recommending an
option that has unintended consequences. For example, if one of the impacts of an option
is that businesses will be required to do certain things, you should note:
• which businesses will be required to meet the requirements
• what impacted businesses will need to do to ensure they meet the requirements
• the impact meeting the requirements will have on their business.
This may require understanding of the obligations that the affected businesses already
face.
• The impacts of an option may be complex, and you will need to understand this
complexity:
• Impacts may vary across affected parties (for example, by size of business or
business sector).
• There may be unintended consequences.
• The causal chain may involve several steps (some of which may be difficult to
understand).
• The causal chain may be influenced by external factors or feedback channels where
later impacts influence former impacts and people’s behaviour, which may have a
complicated interaction with the impacts of the option.
• You will also need to be clear about whether each impact is likely to be ongoing (and how
long it will last for).
30. Identify all affected parties.
• Ensure you identify all population groups and sub-groups likely to be affected by the
option. A common breakdown is government, industry, consumers and environment,
though it may be more useful to identify the specific groups by affected party. You may
need to further subdivide groups of affected parties - for example by size, income,
geographical location (regional or rural), age, family unit, cultural background, or the level
of information parties have.
• For each affected party you need to identify the full set of impacts on that party: how the
option, if adopted, would affect the actions, obligations, circumstances, costs, and
benefits to all affected parties.
• You should also note distributional impacts. Distributional impacts occur whenever there
are winners and losers from an option (for example, where the costs are borne by one
group in society and the benefits accrue to another group). In most cases the impacts will
fall differently across people with different income levels.
31. Identify all types of impacts, including.
• health/well-being7
• environmental
• cultural
• New Zealand’s compliance with international obligations8
7
See Health Impact Assessment at http://www.nhc.govt.nz/phac/health-impact-assessment.htm.
8
See Legislation Advisory Committee Guidelines (Chapter 6) http://www.justice.govt.nz/lac/.
ria-assessing-options 9
• social
• economic
• legal and legislative implications
• impacts on future generations.
32. Move from immediate effects to short term indirect effects and dynamic effects. A good
place to start is brainstorming the impacts of the option. Normally you should think of the
immediate change as a starting point and from this map out the indirect, dynamic, and
feedback effects. For example, if the proposal is to reduce a tax, it may be useful to first note
that those who pay the tax will pay less and then note that this will lead to certain impacts on
their welfare and behaviour, which may have further indirect effects, dynamic effects and
feedback effects.
33. Stem diagrams can be a useful tool to help you think about the causal chain of effects.
A B
C E G
D F
H I
34. The arrows indicate causal links (for example, impact A causes E, which causes H and I).
Note that while impact B causes G, G also has a feedback effect on B.
35. Identify impacts on those within and outside New Zealand. You need to identify impacts both
on those within and outside New Zealand. Even if impacts on overseas parties are not
considered important, they should at least be noted. While many proposals will affect New
Zealand parties and other parties similarly, there may be occasions where it is appropriate to
identify impacts specific to New Zealand parties or non-New Zealand parties. This can
become complex when, for example, impacts on businesses are being considered where
those businesses are owned by people who reside overseas but the businesses are based
in New Zealand.
36. Identify changes to risk levels. You should identify the impacts each option would have on
the likelihood, frequency, or consequences of risks resulting in an adverse event. There is a
difference between risk and uncertainty – risk is where the probability of an adverse event
occurring is less than certain; uncertainty is where the probability is unknown. Identifying
changes to risk levels should be one of your central concerns when you are assessing
options that are designed to change the risk associated with an event/situation and/or
changing who bears the risk.
37. Consider behavioural responses and acknowledge your assumptions about behaviour. In
virtually all situations, impacts will depend on people’s behaviour. In order to be confident
you have correctly identified the impacts of an option, you need to ensure you have
considered the likely behavioural responses. If you have made assumptions about
behavioural responses, which could include assumptions about how affected parties
ria-assessing-options 10
understand the world, you need to acknowledge these assumptions. For more guidance on
this see ‘Behavioural Analysis for Policy’9.
38. Consider the effects of controls on the impacts. The fact that the world is constantly
changing means the impacts of your options may change over time. You need to be aware
of how external factors are likely to influence the impacts. For example, if an industry is
growing, the effect of requirements in terms of compliance costs may increase over time.
39. Monitoring and enforcement. You should consider how compliance with options and the
success of the policy will be monitored and enforced. This is important because an option
may be inappropriate or even counterproductive if it is difficult to enforce.
40. Do not assume 100% compliance. There is unlikely to be 100% compliance with existing or
proposed changes to the law. Consider the impacts of lower levels of compliance.
41. Consider the risk of government failure. It is important to consider the possibility of
government interventions not working – for example, where a licensing requirement is
imposed and the requirements are so onerous that no-one applies for the licence.
Government failure is where government intervention does not solve the problem or makes
the problem worse or worsens the overall situation (even if the problem is solved).
Government failure can bring the law into disrepute.
4 Assess the magnitude of each impact of each option
42. After you have identified each option’s collection of impacts, you should assess the
magnitude of each impact. This makes comparison between each of the costs and benefits
(and hence comparison of the options) easier. To do this you are likely to need to take
similar steps to those required to identify and understand the impacts (see above).
43. In order to be able to conduct a quantitative cost benefit analysis, you need to measure the
costs and benefits in dollar terms. Even if you are undertaking a mainly textual cost benefit
analysis, you should quantify at least some impacts: the level of analysis needs to be
commensurate with the significance of the proposal. As the impacts become more significant
(and the relationships between the impacts more complex), more advanced/accurate
methods of quantification should be used (see Methods of quantifying impacts below).
44. Assessing an impact’s magnitude means estimating how large or small the impact is. This
may include assessing:
• the number of people affected
• the monetary cost
• how long something will take (eg an activity that will be required if the options is adopted,
such as paperwork)
• how much benefit will be obtained
• how much value will be lost or gained
• the number of lives saved.
45. To quantify impacts, there are two basic steps:
• collecting data
• making calculations to draw inferences from the data.
9
Available on the Policy Development Toolkit: https://psi.govt.nz/pdtoolkit/default.aspx
ria-assessing-options 11
46. Data collection will vary in difficulty, cost, and feasibility, as will the calculations required (see
Methods of quantifying impacts below). You should ensure that the data is unbiased and
representative of the population upon which the impacts are likely to fall.
47. If you are having trouble quantifying impacts, note:
• It is better to provide estimates than no quantification. Estimates may include the relative
size of impacts.
• Base your calculations on ‘expected’ impacts, rather than ‘best case’ or ‘worst case’
potential impacts. The expected impact is the impact that will occur on average and is
calculated by multiplying the magnitude of the impact if it did occur by the probability of
the impact occurring.
• Some impacts are simply difficult to quantify, such as comfort, environmental, and cultural
impacts. The quantification techniques below may be applicable, but if you are unable to
quantify certain impacts, you should ensure that you have a good idea of the significance
and relative importance of these impacts and be able to convey these qualitatively (ie in
words). Note that the fact you cannot quantify some impacts does not mean that you
cannot conduct a quantitative CBA (see paragraph 16).
48. Understand the complexities of each impact’s magnitude. As with understanding the
qualitative characteristics of the impact, you should understand the complexities of the
magnitude, including variations in the size of impacts across different parties.
49. Avoid double counting. Double counting is a mistake that can be made. For example, if costs
to businesses will be passed on to consumers, the cost should only be counted once. In this
example, it may be difficult to determine how much of the cost will be passed on to
consumers; the effect will depend on the elasticities of demand and supply in the relevant
market.
Methods of quantifying impacts
50. There are a number of approaches to quantifying impacts. Which approaches you use
depends on what you are quantifying and the resources available to you. The methods
below can be relevant to both a textual or quantitative CBA.
51. Normally a basic level of quantification is not difficult. You need to turn your mind to what
you can calculate, especially simple methods of quantification:
• Time - You could estimate wages multiplied by time taken.
• Health improvements - You could estimate the cost of treating any illnesses that will be
averted.
• Compliance costs to businesses - You could ask businesses how long it would take them
to meet requirements and multiply this by an estimated hourly rate.
52. Talk to affected parties. Affected parties are likely to have a detailed knowledge of how
options will impact upon them, including information that will help you quantify impacts
(especially costs). For example, businesses are likely to be in the best position to know how
much it will cost them to meet new requirements and how changes to the law will influence
their activities, including the way they do business and their profits.
53. Estimating by analogy. One way of quantifying impacts is to look at the impacts of similar
policies and make adjustments to these to arrive at your best estimate of the impacts of the
option you are considering. For example, when estimating the registration costs to
psychotherapists of new registration requirements under the Health Practitioners
Competence Assurance Act, the Ministry of Health established that the determinants of the
ria-assessing-options 12
cost are the size and complexity of the profession. After comparing psychotherapy with other
similarly sized and similarly regulated health professions it came to a figure of approximately
$500 per registered psychotherapist.
54. Quantifying social impacts. Some examples of methods of quantifying social impacts are:
• estimating gains in years of education that may result from a policy
• the value of reassurance
• estimating quality adjusted life years
You may need to apply other methods in to calculate these. Information on these
approaches can be found in a number of sources.
55. Market prices. Where market prices are relevant, quantification is relatively simple – the
value lost or gained can be measured by the expected change in the price of the good or
service, which indicates how much people are willing to pay for the good or service or how
much they would be willing to accept to give up the good or service.
56. Econometric analysis. Econometric analysis involves collecting data on certain variables and
testing the causal relationships between these variables (eg: changes over time to income
levels, education levels, profits made in certain industries etc). If similar policies have been
implemented in the past, it may be possible to use econometric analysis to directly estimate
the impact of an option on a problem. Otherwise you can estimate the effect on a problem
by assuming/estimating how the option affects other explanatory variables, and then analyse
the effect of changes to those variables on the problem.
57. Data from previous studies, including previous policy development work, can be used to
quantify impacts. There are several extensive ‘benefits transfer’ databases of previously
quantified impacts resulting from policies worldwide. Note that benefits transfer calculations
should only be done by experienced economists, given the uncertainty of transferring
values.
58. Revealed preference techniques that use current behaviour to estimate future behaviour.
• ‘Hedonic pricing’ is a technique used to estimate impacts by considering their influence
on market prices. If influences can be understood, the insights can be transferred.
Example 1: The value of education improvements can be estimated by examining the
increase in incomes (of those who have the education) that results from these
improvements. Example 2: The benefits to residents from restricting industrial
development to non-residential areas could be estimated by comparing the value of
houses adjacent to a factory with those in the same suburb that are not. Example 3: The
value of a park could be estimated by comparing the value of houses adjacent to similar
parks with houses not adjacent to parks.
• ‘Travel costs’ estimate impacts by considering the direct cost a party encounters to
achieve or avoid something. Example 1: The value of a labelling requirement to
consumers could be estimated by finding out how much effort a typical consumer
expends to find out the information themselves. Example 2: The value of a park could be
estimated by calculating how much trouble people would go to in order to visit the park.
59. Stated preference techniques that ask parties the value to them of impacts. This can include
asking a party how much they would be willing to pay to accrue benefits, or would need to
receive in compensation to incur costs. For example, a company operating a factory might
be asked how much it would be willing to pay to continue polluting, while individuals might
be asked how much compensation they would require to tolerate the pollution. Such surveys
need to be carefully designed to minimise the potential for respondents to misrepresent their
true valuations. In the pollution example, the company, if it believes it will not actually be
ria-assessing-options 13
required to pay to pollute, may overstate the amount it is willing to pay to continue polluting,
or the individual may overstate the amount of compensation required for them to be willing to
tolerate the pollution. Further, the amount individuals are willing to pay to prevent a polluting
factory from locating nearby is likely to be less than they would be willing to accept as
compensation for the nearby pollution.
60. Seeking expert advice. In some cases it may be useful to engage experts to help you assess
the magnitude of each option’s impacts. However, as with all analysis, you should ensure
value judgements of experts do not influence the analysis.
5 Review the options
61. Take an iterative approach. Inquiry into the impacts of options will tend to lead to more
refined options as you realise how options can be altered to reduce negative impacts and
increase positive impacts. When you change an option, you need to ensure your list of
impacts remains thorough and accurate.
6 Compare the options and identify the best option
62. Once you have finalised all of the options and have a thorough and accurate list of each
option’s impacts, including a proper understanding of the qualitative characteristics and
magnitude of each impact, you need to make a decision as to which option is best. In order
to do this, you need to:
• determine what your decision criteria are (benefit cost ratio and/or net benefit) and apply
the decision criteria to each of the options
• make sure you have acknowledged your criteria and assumptions
• test the sensitivity of your results to changes in your assumptions.
Decision criteria
63. For a mainly textual/quantitative cost benefit analysis, the decision criterion is likely to be:
Which option is the best when the benefits are compared with the costs? This will
necessarily be a judgement call and will depend largely on estimates of the relative value of
impacts. The more quantification you have done, the easier it will be to make a judgement.
Where there is virtually no quantification, in many cases the decision as to which option is
best will be very much a ‘best guess’.
64. As set out above (see paragraph 7), in a quantitative CBA a net benefit and/or benefit-cost
ratio calculation are required in order to identify which option is best.
65. Sometimes the option with the highest net benefit will not be the option with the greatest
benefit-cost ratio. For example:
ria-assessing-options 14
Option One
NPV of benefits equals $35 million
NPV of costs equals $20 million
BC ratio equals 1.75 (benefits divided by costs)
NB equals $15 million (benefits minus costs)
Option Two
NPV of benefits equals $20 million
NPV of costs equals $10 million
BC ratio equals 2
NB equals $10 million
66. In this example, the net benefit criterion favours option one, while the benefit-cost ratio
favours option two. If the options involve government spending, generally you should choose
the option with the highest benefit-cost ratio because this results in the highest net benefit
over all of the government’s spending projects. The exception to this is where you are faced
with two options that have higher benefit-cost ratios than proposals in other areas (for
example, if transport proposals have higher benefit-cost ratios than proposals in other
areas). In these situations it may be better to go with the option that has the largest scale to
ensure that government spending generates the highest net benefit overall.
67. If the proposal is regulatory, it is less clear which criterion you should prefer, but in general
you should prefer the benefit-cost ratio because, like a government budget constraint,
society’s resources (for example, the amount of time people can spend on complying with
requirements) is finite so getting the best return for society’s resources from each policy
development process is likely to generate the highest net benefit overall.
68. More guidance on formal quantitative cost benefit analysis is available in Treasury’s Cost
Benefit Analysis Primer, available at www.treasury.govt.nz.
Acknowledge your criteria, values, and assumptions
69. You need to acknowledge the criteria, values, and assumptions that have determined your
conclusion as to which option is best. If, for example, the sole criterion is the net benefit
calculation, you should state this. However, if you have applied weightings to different costs
and benefits, have used additional criteria (such as the policy objectives), or have made any
assumptions (such as the discount rate – see appendix 3), you should be explicit about this.
Where there is uncertainty, you should acknowledge the assumptions you have made about
what the likely scenarios are. Be careful not to set up your assumptions in such as way that
they lead to a pre-destined outcome.
Testing the sensitivity of your results to changes in your assumptions
70. Sensitivity analysis involves varying the main assumption(s) (eg the discount rate) of the
analysis to determine the extent to which the results are dependent on those assumptions. It
also provides valuable information on reliability. It is particularly useful where there is
considerable risk or uncertainty surrounding the estimates of the main cost(s) and or
benefit(s). Where this is the case, the net benefit and benefit-cost ratio calculations should
be repeated using other reasonable assumptions on the value of the major impacts. An
option should be able to demonstrate a positive outcome under most of the relevant
scenarios tested if confidence is to be had in its appropriateness.
ria-assessing-options 15
7 Present the results
71. In presenting the results of your assessment of the options (and showing how the preferred
option would change things – ie by comparing the status quo with the preferred option), it is
important to document the methods used to calculate the costs and benefits, including:
• the criteria you have used to come to your conclusion
• all major assumptions
• any weightings that have been applied to certain costs and benefits
• deficiencies in the information used
• the scope of the analysis
• possible estimation biases
• any major intangible costs and benefits
• any other information considered necessary for a third party to be able to assess the
analysis objectively.
What level of analysis is required?
The following text is taken from the RIA Guidelines, but has been reiterated here because it is
best practice.
72. The level of analysis needs to be enough to enable the decision-maker to be confident about
which option is best and the impacts this will have. It should be commensurate with the
significance/magnitude of the proposal. There are four parts to this:
• identify the range of feasible options (ie the possible options that would meet the public
policy objective and solve the problem)
• identify the impacts of each option
• assess the magnitude of each impact
• choose the preferred option.
73. Doing this analysis requires information gathering, which is likely to involve:
• consulting affected and interested parties (stakeholders, experts, and within government)
• collecting data
• researching experiences within New Zealand and overseas.
How much analysis is enough?
74. This is necessarily a judgement call, but as a guide the level of analysis should be enough to
enable the decision-maker to be confident about which option is best and the impacts this
will have. You need to think carefully about what they need to establish to make the case for
the preferred option being better than alternative options (including the status quo). Some
guidelines are set out below.
75. Unless the proposal is very minor, regardless of the significance/magnitude of the proposal:
• all feasible options should be considered
• all impacts should be identified, including all costs (including business compliance costs),
benefits, risks, and other impacts – this includes health, environmental, economic, social,
and cultural costs and benefits
ria-assessing-options 16
• a qualitative or quantitative assessment of each impact's magnitude should be made (ie:
how large or small each impact is)
• there should be clear reasons why the preferred option has been chosen over the other
options.
76. For proposals that are relatively small, a mainly textual/qualitative analysis is acceptable -
the level of quantification can be less than if the proposal is significant. If a fully quantitative
analysis was done for a small proposal, the level of analysis would be more than is
commensurate with the magnitude of the proposal (and would not be a good use of
government resources). Some costs and benefits, such as health and environmental costs
and benefits, can be difficult to quantify. However, where viable, quantification of the
magnitude of impacts is expected, even in a mainly qualitative/textual analysis. Normally
providing some quantification is not difficult – you need to turn your mind to what you can
calculate/estimate. A prime reason for the analysis being inadequate is that there is little or
no quantification.
77. Identifying all impacts is important, including identifying all types of impacts (cultural,
economic etc). To ensure that impacts are properly identified, you need to think beyond what
the direct impacts are. It is common for indirect impacts to be missed – for example, impacts
of general regulation often have unforeseen impacts on specific sectors, such as the
education sector.
78. For proposals that are more significant/of a larger magnitude, the following is expected:
• more information gathering - wider consultation, more data collection, more research
• more rigorous analysis
• all impacts of all options need to be quantified where possible
• there needs to be evidence of clear and detailed reasons why the preferred option has
been chosen, including any relevant assumptions, formal quantitative cost-benefit
analysis where possible (especially where the proposal is very significant), and any
judgements made outside of the impact analysis.
79. In most cases the analysis will be inadequate if:
• no options other than the preferred option were properly considered
• impacts were not identified, especially major impacts
• no or little attempt was made at quantification, especially where the impacts in question
are relatively easy to quantify.
ria-assessing-options 17
Appendix 1: Guidance on information gathering and brainstorming
Use a checklist to help you fully scope your list of impacts. We recommend using the ‘Impact
Checklist’10 as a starting point, though you may wish to amend the checklist to ensure other
areas are considered.
Review the literature. A lot of literature discusses the effects of different types of policies. To
determine what works, it is important to look at what has been written on the policy questions
you are considering. One way to accomplish this is examining what is known about the
impact of particular interventions from a range of studies. Studies should be weighted
according to their rigour and applicability. Not all policy processes can afford this level of
analysis, or the time required for it. However, almost always, good policy requires some
attempt to summarise the overall findings of a number of research efforts. If something works
in a number of places, for a number of different people, it is more likely to work in New
Zealand.
Assess what impacts similar policies have had. In New Zealand and overseas, policies that
are similar or identical to some of your options may have been implemented. If this is the
case, there may be commentary on the impacts these policies have had, as well as
knowledge within the governments of those countries and among affected parties. Care
should be taken to ensure that such studies are comparable with current New Zealand
circumstances, either ‘as is’ or with adjustments. There are several extensive ‘benefits
transfer’ databases of previously quantified impacts resulting from policies worldwide.
Talk to other government agencies and people within your own agency. Talking to people in
government can be an efficient way of improving the analysis of your options. Policy analysts
can draw on their previous experiences with policy processes and their knowledge of the
effects of past government interventions. The departmental consultation process is a useful
way of obtaining comment from government agencies, but you should talk to those who are
likely to have a particular interest or expertise in the area early in your policy development
process to give them time to provide full comment.
Talk to affected parties. Affected parties are likely to have a detailed knowledge of how
options will impact upon them. For example, businesses are likely to be in the best position
to know how much it will cost them to meet new requirements and how changes to the law
will influence their activities, including the way they do business and their profits. You may
wish to talk to affected parties directly or through representative organisations. If you take the
representative organisations route, make sure the representative organisation is relaying the
views of the people who are likely to be affected; otherwise you may not be receiving the
right information. It can be important to consult widely in order to avoid regulatory capture.
For example, while directly affected parties may favour one option, this option may not be the
overall best option. Note that talking to affected parties involves a trade-off between refining
the proposal to allow informed comment and not pre-judging the best outcome and making
comments redundant. You can find names of businesses who are interested in being
consulted on the Business Consultation Database website:
http://www.businessconsultation.govt.nz.
A useful way of consulting on the impacts of options with affected parties is to develop a full
description of each option and a preliminary assessment of the impacts of each option, then
ask affected parties to comment on your assessment of the impacts, including whether there
are further impacts. Where the RIA requirements apply, discussion documents must include
questions and/or discussion of the problem, options and impacts of options (see RIA
Guidelines11).
Talk to experts. For example, scientists, social scientists, economists, engineers.
10
Available on the Policy Development Toolkit: https://psi.govt.nz/pdtoolkit/default.aspx
11
Available on the Policy Development Toolkit: https://psi.govt.nz/pdtoolkit/default.aspx
ria-assessing-options 18
Appendix 2: Principles to follow
Be impartial. People can give more or less weight to, or include or ignore, certain impacts
because they do not consider them to be important, or because the impacts do/do not
support their preferred option. These kinds of judgements are relevant when you are making
an overall assessment of options, but in order to make the overall assessment you first need
to have a full list and full understanding of all of the impacts of each option. Impacts that at a
first glance appear minor may upon closer inspection be major (for example, after you have
obtained more information on the impact from consulting with affected parties).
Use an evidence base. All assessments of impacts, whether quantitative or
textual/qualitative, should be based on evidence, with data sources and assumptions clearly
identified.
Make your assumptions clear. It is normal and expected that you will have made certain
assumptions when identifying impacts. You should be conscious of these assumptions, keep
a record of them, and note them when you are presenting your analysis to stakeholders,
government agencies, Ministers, or other parties.
ria-assessing-options 19
Appendix 3: Converting values to their ‘present value’
Suppose that an option has costs that are incurred over the next two years and benefits that
are realised in ten years, or from now into the future indefinitely (for example, an option to
build a new road). In order to compare costs and benefits across time, all costs and benefits
need to be converted to their present value, which means the value of the cost or benefit is
converted to its value as if it was incurred/received today. The need for such a conversion
arises from the simple observation that the value of a dollar received today is normally
valued more highly than a dollar received at some time in the future. Converting values to
their present value (ie discounting values that will be realised in the future) means you are
comparing apples with apples. There are three major reasons for this difference:
• The "rate of time preference". Most people are impatient and prefer consumption
undertaken now rather than later. Thus, a dollar available now is more highly valued
than one received later.
• Uncertainty. There is necessarily some degree of uncertainty as to whether a future
dollar will actually be received. Its value is lessened in proportion to the expected size
of this uncertainty factor. Hence the discount rate may depend on the degree of
uncertainty.
• Wealth. People will be wealthier in the future than they are now, so impacts incurred
now are valued more than the same impacts incurred in the future.
Note: all costs and benefits must be real (inflation adjusted) values.
The discount rate seeks to capture the sum of the above factors. A standard formula for the
discount rate is d = ρ + µ.w , where ρ is the rate of time preference (including uncertainty), µ
is the marginal utility of consumption (how much value an additional dollar is worth – a value
of 1 implies that a generation with twice as much wealth will value a dollar at half that which
the current generation does) and w is the per capita growth rate. For example, if ρ = 0.03
(3%), µ = 1, and w = 0.02 the discount rate, d, equals 0.05 or 5%.
To calculate the present value of a cost or benefit, multiply the magnitude of the cost or
1
benefit by where
(d + 1) t
• d = discount rate;
• t = the time period.
The time period is defined with reference to the discount rate. For example, if d = 5% per
year, and t = 2, 2 represents the second year. If d = 5% per month, and t=2, 2 represents the
second month.
While conceptually simple, there is considerable debate about how discount rates should be
calculated. The discount rate used may depend on the type of benefit or cost being
discounted. The discount rate applied to government expenditure can be the cost of capital
to the Crown, or the government bond rate appropriate to the term of the project. This
incorporates the opportunity cost of resources forgone to pursue the proposal or project.
Further, it may be possible to justify applying much lower discount rates to environmental
values in the expectation that as people become wealthier and the extent and nature of the
environment decline, people in the future can be expected to place greater value on the
natural environment. Social projects can be particularly sensitive to discount rates, and
accordingly many health projects often use a low discount rate of 3% because of the long
length of time before benefits accrue. Using a range of discount rates to demonstrate the
sensitivity of your project to the discount rate, rather than just choosing one discount rate, is
often recommended.
ria-assessing-options 20
Care needs to be taken in the choice of discount rates. Across government, these choices
can impact on:
• the allocation of resources between the private and public sectors
• the extent to which it is possible to support the imposition of regulatory costs on
business and community interests
• the extent to which the interests of current generations may be favoured over those of
future generations.
• More guidance on the net present value concept and discount rates is available in
Treasury’s Cost Benefit Analysis Primer, available at www.treasury.govt.nz.
ria-assessing-options 21
