1 BENEFIT ASSESSMENT TOOL – GUIDANCE ON USE 1 This note sets out guidance on how to complete the Bank of England’s benefit assessment tool. The main aim of the tool was to make a rigorous, consistent and transparent assessment of the relative benefits of the information collected by the Bank. This process is explained more fully in the CBA Handbook: ‘Cost-benefit analysis of monetary and financial statistics, a practical guide’. The process for using the benefits assessment tool 2 The tool has been developed to give non-monetary benefits scores for statistical data. These scores provide a useful gauge of the relative importance of data, which can be set alongside the information on the costs of providing the data. Those data which look relatively costly and are providing only low benefits to users are the most obvious collections to question first. 3 The benefits assessment tool has been used in a number of ways within the statistics area of the Bank. Some analysts have looked through the different uses of the data for an entire collection (form / table); other analysts have looked at the benefits of each individual box on a data collection or by sections of a data collection. Because of the nature of banks’ costs, it is only through cutting whole sections of forms or entire collections that significant cost reductions for reporters can be achieved. Therefore it may be best to concentrate on measuring the benefits on the same basis – ie at a whole form/section level. 4 It is only the most important use of the data that is being assessed. Multiple uses are taken into account within the tool, so there is no need to re-run the benefits for different users/uses unless there is some uncertainty over the main user/use of the data. 5 The benefit assessment would normally be completed by the most experienced analyst conducting the review of the form / table, taking account of the views of known major users of the data – in some cases there may be widespread users and a public consultation may be required. 6 Where the analyst is very experienced in the uses and users of the data, a basic benefits assessment could be made at an early stage of the review. The results could help to focus on different options for whether to continue the data collection, or possibilities of lowering reporting costs. Of course, initial views may change as new information comes to light during the course of the review, so it is worth keeping an open mind on any initial assessments and revisit benefits scores where appropriate. 7 There needs to be some mechanism for ensuring consistency across benefits assessments for different data collections, which are also typically undertaken by different analysts. The Bank found the most effective way to do this was through a meeting of all analysts involved in reviews. This group 2 would examine the relative scores for each section of the benefits assessment and would revisit scores that looked out of line with others. 8 The tool is a means to an end, and it should accurately reflect where data have high, medium or low benefit. But it may also need to be adapted to particular circumstances. Relative scores of benefits and costs may indicate where savings could be made, which can be particularly helpful when there is pressure to lower reporting or compilation burdens. It will also inform priorities and help general resource allocation. How to fill in the form 9 There are two types of scores on the form: gold boxes, where one (and only one) option needs to be chosen in each of the boxes by placing a 1 in the relevant box; and green boxes, where a 1 is entered for each that applies. A score out of 100 will automatically appear at the end; the higher the score the greater the benefits of the data collection. There are also blue boxes to capture qualitative information about the data collection being assessed. The blue boxes Fig 1 Data being reviewed Form / section / table (where applicable) Section of form / table (where applicable) Description of any other uses (include internal statistical area uses e.g. cross-checks, sample selection) 10 The blue area at the top is to record which part of the form / table is being assessed and which are the main data output(s) involved. Because the benefits tool can be used for a wide range of data sources - from individual boxes, sections of forms, full tables / forms etc - it is important here to be clear what is being assessed. The final blue panel is to record if and where the information is used internally (within the compilation institution); for instance, if it is used for cross-checking information on other sections / tables / forms. These uses can be important and should be noted. Policy use and relevance (50% of total score) Fig 2 3 Policy use Very high used in key policy decisions, or flagship publications and statistics (choose 1 only) High used in important policies, publications and statistics Medium used in less important policies, publications and statistics Low hardly used or in secondary publications Very low Very rarely used None Policy relevance Very high The key component of the activity / policy use contribution of these data (choose 1 only) High A major input to the activity / policy use Medium A generally important component of the activity / policy use Low A sometimes important component of the activity / policy use Very low A rarely important component of the activity / policy use None 11 These two categories assess the importance of the activities that the data are used for (policy use), and then the significance of the data for those activities (policy relevance). There two separate scores to allow one to differentiate, for example, between very important activities where the data plays either a major or minor role, and less important activities where data are either very important, or less so. 12 Policy use is rated between very high and none, with accompanying descriptions to help guide the assessor. Although this is subjective, there should be some evidence to base the decision on. 13 Policy relevance is similarly rated and captures how important the data are for the various policy uses listed. Policy relevance can be important even if the policy use is not and vice versa. Relevance should reflect how much importance the main users place on the data for decision making. 4 Standards and regulation (15% of total score) Fig 3 Standards and regulation Legal obligation No choice but to implement (excluding ESA95) (choose 1 only) Meets agreed standard For full consistency with international standard (ESA, SNA, BPM) Helps international Other countries also produce the data but not comparably across standards comparisons None 14 In this section record if data are required by law (excluding ESA95), to meet an agreed international standard (eg ESA95, SNA95 or BPM5), or if they data help international comparisons. Additional uses (10% of total score) Fig 4 Additional uses Helps outside research (choose all that apply) Helps inform general public or media Helps other economic policies Published, eg Stats Release Consistency check with Other data / counterperties etc. For each that applies, enter 1 in the associated green box. 15 This section awards a score for data that we know have other uses outside of the main user and use (as measured in the policy use and relevance section). This can apply to outside research (eg commentators, academics); to the general public or the media; or to other economic policies (eg where government economists take interest in the data). Value added (15% of total score) Fig 5 Value added High Main source of high level data (i.e. aggregate boxes) for internal analysis (choose 1 only) Medium Further breakdowns of main aggregate data Low Similar data available elsewhere, or data could be estimated None No value added 16 This section captures how much we can gain from these data, over and above what is available from other sources. A high rating should mean that both the aggregate data and any sectoral or industrial breakdowns are only available from this source, or it is the main source of a high level aggregate. A medium score means that it is additional information to that available elsewhere such as 5 a further breakdown of a main aggregate captured elsewhere. A low score means that similar data are available elsewhere, or that the data could be estimated. Quality (10% of total score) Fig 6 Quality High Census (choose 1 only) Medium Large sample Low Poor quality - small sample / low coverage 17 This section looks at the underlying statistical quality of the data – how representative is the sample, are there frequent significant revisions, do the data coherently fit in with what users want, or are they a close second? A simple proxy here is to look at the size of the panel as a proportion of the census in terms of data coverage. High would be full census; medium less than census but capturing a representative panel; low is poor quality, where the panel is small and probably not representative. More advanced measures are also worth investigating, depending on how the data are compiled and what the sources are. New Data Requests 18 The tool can be also used for new data requests. The principle is the same but the score will be determined by the expected benefits from the expected policy use and relevance. Again costs here can be weighed up against benefits for different options, for instance for different levels of data quality. A new data request is likely to involve fixed set up costs and therefore the hurdle in terms of potential benefits is likely to be high. It is important to be transparent about this with users and data suppliers. Finally 19 The benefits tool gives a benefits score at the end. This is a percentage (max 100, min = 0), and the higher the score, the more beneficial the data. After having filled out the form, it is a good idea to check the score fits with what you would have broadly expected. It is also be helpful to benchmark the score against other forms, and discuss how other people have used the tool to rate their own tables / forms. The distribution of scores can be helpful to assess the important and less important data, but only where there is consistency in the way the tool has been used. It is worth archiving benefits scores, and ideally the thinking that they are based on, for future reference. 20 For data with no or very low benefit scores, an immediate question might be: why it is still being collected? Similarly, where new data requests score low on benefits, it may not be necessary to follow a full cost-benefit analysis.
Pages to are hidden for
"Cost Benefit Analysis Tool"Please download to view full document