Learning Center
Plans & pricing Sign in
Sign Out

Benchmarking response - detail


Benchmarking response - detail

More Info

1.   This paper is BAA plc’s response to the CAA’s consultation paper “The Use of
     Benchmarking in the Airport Reviews” issued in December 2000, which forms
     part of the CAA’s work programme relating to the regulatory review of BAA’s
     Heathrow, Gatwick and Stansted airports.

2.   The paper summarises BAA’s overall approach to benchmarking and suggests
     how benchmarking might best be incorporated into the regulatory framework.

3.   This response should be read in conjunction with the CAA’s document. The
     page and paragraph numbers and headings refer to those in the CAA’s paper
     unless otherwise stated. Appendix 4 references comments in this response to
     questions posed in paragraph 4 of the CAA paper.

4.   BAA recognises the CAA’s rationale for including benchmarking as a tool to use
     to assess its regulated business. BAA will support this process in helping to
     achieve a fair regulatory result. This can only be achieved if the CAA
     comprehensively takes into account the many factors that differentiate airports.

BAA response                          1                                         CAA/44
The use of benchmarking                                                   February 2001
Why Benchmark?
5.   The CAA’s general contention (paras 1.1 to 1.6 of the issues paper) is that
     benchmarking is required because of the ‘inevitable’ asymmetry of information
     between regulator and regulated company and the assumed ability of the latter
     to ‘game’ the system.

6.   The first point to make is that BAA does not believe that there is evidence of
     ‘gaming’ in its case. This applies to BAA forecasts of both operational profits and
     capital expenditure and is discussed more fully in MMC4, Chapter 4. Nor does
     BAA believe there is any evidence of ‘gaming’ in the current quinquennium and
     will be submitting detailed accounting information to this effect in due course.
     BAA believes that the current price formula for Q3 represents a reasonable
     balance between the various stakeholders.This was achieved by detailed
     regulatory examination of financial forecasts and little by way of cost

7.   The second general point concerns asymmetry of information. BAA is surprised
     that the CAA perceives this to be a problem given the significant amount of
     information about the company which is in the public domain. However, if the
     CAA do believe this to be a particular problem with respect to cost information
     then the first step should be to identify those specific areas which the CAA
     believes are characterised by asymmetry. The problem can then be addressed
     by an exchange of information. To the best of BAA’s knowledge the CAA has
     not to date sought additional detailed operating costs information from BAA, and
     BAA would have been, and is, happy to give all assistance to the CAA in
     minimising any concerns about information shortfalls.

8.   Thirdly, it is also generally recognised that cost benchmarking is effectively a
     second best outcome to outsourcing - i.e. if BAA were to outsource all its
     operational activities on a transparent competitive tender basis then the CAA
     would possess all the cost information it required. Whilst BAA does not believe
     outsourcing of all activities to be in the best interest of customers , it does
     currently outsource a significant proportion of its operational activities. A number
     of BAA’s costs are also non-controllable being determined largely by external

BAA response                            2                                          CAA/44
The use of benchmarking                                                      February 2001
     agents, e.g. rates, police costs, airport licences. It is therefore fair to say that any
     ‘asymmetry of information’ is confined to BAA’s controllable costs which are not
     outsourced. A significant proportion of these relate to compliance with DETR
     security and CAA SRG safety standards which cannot be meaningfully
     benchmarked with overseas airports as the latter either do not carry out these
     functions or do so to different national standards. An alternative approach to a
     top down mathematical modelling of all costs followed by a series of difficult
     adjustments might therefore be to select the cost functions where asymmetry of
     information is deemed to be a problem and then to request comparative or
     process information from BAA. CAA have chosen not to adopt this approach to
     date but BAA nevertheless feel it has some merit and it is discussed further

The Attractions of Benchmarking
9.   The above comments refer to the perceived regulatory need for cost
     benchmarking. This is not to say that benchmarking in general is without its

10. In previous MMC reviews BAA submitted several benchmarking studies covering
     key areas of the business. For instance in MMC4, published in 1996, the
     following areas were examined in detail:
     (i)     standards of service (see paragraph 6.76-6.82);
     (ii)    rents (8.59-9.65);
     (iii)   airport charges (5.45- 5.47);
     (iv) retail prices (8.21); and
     (v)     construction costs (9.27-9.43).

11. In many cases the correct comparators for the business are not other airports.
     Different conditions will apply to different areas of the business. Rents, for
     example, need to be considered in the context of the London area. Whilst
     comparisons with other airports were undertaken in 1996, adjustments were
     needed to take account of their national property markets and rental levels.

BAA response                             3                                          CAA/44
The use of benchmarking                                                       February 2001
12. Construction costs, on the other hand, given the scale and worldwide nature of
     procurement, can more easily be benchmarked against US or European norms,
     though even in this case there are differences in local labour costs and exchange
     rates which would have to be accounted for.

13. The areas discussed in MMC4 illustrate the strengths of benchmarking, as used
     consistently by BAA,in determining product price and quality.

Top Down Operational Cost Benchmarking
14. Benchmarking of costs has a different purpose from benchmarking of quality and
     prices as discussed above. When conducted by the company its purpose would
     be to identify areas where costs are higher than might be expected and therefore
     suggest potential areas for process improvement. The regulator’s use of cost
     benchmarking would be for the different purpose of setting quinquennial cost
     targets for the regulated company as part of the price formula review.

15. BAA concluded such a cost benchmarking exercise in 1990 and submitted this
     as part of the 1991 price review (see MMC2). It is worth noting that even for the
     airports considered appropriate as comparators for the BAA airports, the
     following not inconsiderable problems arose with the data collected;
     (i)     The exercise took over a year to complete due to the need to identify all
             differences, calculate the cost effects, and verify the calculations;
     (ii)    The process was entirely dependent on the co-operation by overseas
             airports and the time taken for them to produce the calculations;
     (iii)   Residual doubt over several of the adjustments remained;
     (iv) As a result there was an implicit large error in results; and
     (v)     The results were unsuitable for identifying areas where scope for
             improvement existed as it was never certain whether differences were due
             to measurement errors or real underlying efficiency differences.

16. The MMC stated (in para 5.37) that “international airport comparisons are
     notoriously difficult to make because of the differences in the activities
     performed, in the way the financial accounts are drawn up, in the Government
     subsidies involved and in the design and service standards”. It also noted that

BAA response                              4                                          CAA/44
The use of benchmarking                                                        February 2001
     “in staff costs per pax……varying exchange rates and accounting practices
     present a particular complication and the economies of scale argument is again

17. In the event the information submitted to 1991 on cost benchmarking was used
     by the MMC primarily to establish in broad terms if “scope for radical
     improvements in BAA staff productivity” existed, i.e. only a very broad null
     hypothesis was being tested. In the 1996 review benchmarking of cost
     information was not used at all by the MMC.

18. Further studies of benchmarked costs across European Airports have been
     conducted since 1991, notably by Cranfield University for clients in Australia
     (FAC) and Spain (AENA). Again these were primarily used to look at very broad
     efficiency comparisons to justify particular state ownership structures rather than
     the much narrower point of setting detailed price caps. The same technical
     difficulties in comparing costs were found in the Cranfield reports as had been
     experienced by BAA.

Issues to be considered in Top Down Cost Benchmarking
19. BAA’s view is that benchmarking could provide useful information , but that it will
     only be useful if the following issues are dealt with fully;

     Data Adjustments
20. It is worth listing the differences between cost bases of airports within Europe
     which would have to be adjusted for in any benchmarking study (see Table 1).
     Both the CAA (in paras 3.22 to 3.27) and the MMC in MMC4 have already
     referred to these but there are three aspects to the adjustments process which
     are worth highlighting;

     (i)   Some of the issues listed can be circumvented by restricting analysis to a
           ‘costs before depreciation, funding and taxation’ basis. Indeed, data
           problems more or less compel this. However, this invariably means that
           there is a distortion caused by different companies choosing different points
           on the capital/labour (or other resources) trade-off line;

BAA response                            5                                        CAA/44
The use of benchmarking                                                    February 2001
     (ii)    Although adjustments can be made for differing activities and direct costs,
             there are often significant issues concerning overheads. Definition of what
             is included in direct/overhead costs is often unclear, and varies
             considerably from airport to airport. Also how should adjustments be made
             for overheads which would be allocated to cover non-standard activities and
             cost bases? Different views of overhead allocations can lead to a 30-35%
             range in possible cost outcomes. Any adjustment of overheads would have
             to rely on knowledge of the non-standard activities, e.g. ground handling,
             ATC, international contracting, as well as knowledge of the core airport
             business. In practice overhead costs are usually ignored in such
             benchmarking exercises, which leads to a distortion in comparability
             between direct and indirect costs.

     (iii)   The adjustments required, which are summarised in Table 1 below, will
             require detailed information directly from the benchmarked overseas
             airports. Even with the co-operation of industry task forces and using
             personal contacts such exercises take a lot of time and there is doubt, in
             BAA’s opinion, about the accuracy of some of the adjustments data
             provided. This is primarily because there is little incentive for airports,
             whether state-owned or private companies to release data which may lead
             them being exposed as bad performers. Assurances of anonymity do little
             to change this. As a result any overall findings on comparative costs
             cannot be properly audited - the best that can be achieved is an assertion
             that to a probability of x% the results are within a range of £a-b per
             passenger (which will probably be very large). BAA strongly questions the
             fairness of setting price caps based on information of this kind which cannot
             be audited.

    (iv) Two key airports, namely Amsterdam and Frankfurt are preparing IPO’s. It is
             unlikely that they will make available key information in the detail required
             while this process is ongoing. Exclusion of these from the benchmarking
             process would, in our opinion, seriously devalue the results.

BAA response                              6                                           CAA/44
The use of benchmarking                                                         February 2001
TABLE 1 List of Adjustments required for International Benchmarking Exercises
       Different Activities     :   ATC
                                    Handling activities
                                    International work
                                    Crash and rescue
       Costs Add ons/           :   Rates
       ‘free rides’                 Police costs
                                    Airport licences
                                    Corporation Tax
                                    Pensions/Social Security costs

       Different Standards      :   Security - passenger/cabin baggage search
                                             - access control
                                             - hold baggage screening
                                :   Fire
                                :   Other airfield
       Accounting differences   :   Asset Valuation (replacement
                                    methodologies, asset ownership)
                                    Depreciation (lives, write off policies)
                                    Inter company charges

       Financing costs          :   Ownership structure
                                :   Debt/equity
                                :   Local interest rates
                                    Tax breaks on debt

       Other Differences        :   Local utility costs
                                :   Local property costs
                                :   Local staff costs
                                :   Exchange rates

       In house/outsourcing     :   Cleaning

       Service standards        :   Information provision
                                    Terminal duty teams

N.B. This list is not exhaustive; it is intended only as a guide to the magnitude of
      differences which may affect the validity of results derived from
      benchmarking of operating costs.

BAA response                         7                                           CAA/44
The use of benchmarking                                                    February 2001
Choosing Appropriate Airport Benchmarks
21. As regards the practical application of benchmarking to airports (paras 3.31 to
     3.34), BAA is in general agreement with the proposed methodology. It should be
     stressed that any comparisons can only be useful if inputs and outputs are
     aligned and appropriate adjustments made to the data sets. Appendix 1 sets out
     some key data for the airports listed.

22. BAA agrees with the CAA’s definition of primary outputs (discussed in
     paragraphs 3.13 to 3.21), being related to volume throughputs measured in
     passenger numbers and traffic movements, considered in conjunction with
     quality of service. These high level outputs alone, however, are an insufficient
     basis on which to compare costs. The following factors will all impact
     considerably on the levels of operating costs.
     (i)     the mix of passengers between domestic, international, EU or long haul,
             charter or scheduled;
     (ii)    the proportion of transfer passengers;
     (iii)   the mix of movements between large and small aircraft; and
     (iv) the peakiness of traffic.

23. Paragraph 20 and Table 1 above detail the many areas of adjustment which
     would be necessary in order to achieve good quality comparisons. In terms of
     the airports listed in para 3.29, many of the airports would, in BAA’s opinion, not
     be suitable comparators for the three BAA airports.

24. BAA does not consider US airports to be suitable comparators for BAA airports
     for the following key reasons (which are more fully explained in Appendix 2).
     (i)     US airports are institutionally very different from UK airports in terms of
             ownership and financing;
     (ii)    the services provided by airports are typically significantly different from UK
             airports, particularly ownership and operation of terminals, and
     (iii)   airport/airline relationships are very different in terms of contractual
             relationships which give airlines in certain cases a veto over airport capital

BAA response                              8                                           CAA/44
The use of benchmarking                                                         February 2001
     Any cost or efficiency comparisons must adequately compensate for these
     varying responsibilities and the consequent effects on costs and financing.

25. Traffic mix at US airports suggests they should be rejected as comparators. JFK
     has an international/domestic passenger mix of 57/43, and Newark 23/77,
     compared to Heathrow’s 89/11. Heathrow’s total passenger numbers are
     roughly twice those of JFK or Newark. Other comparators in the list (Chicago,
     Atlanta, Tokyo, Los Angeles, Dallas and Denver) are dominated by domestic
     passengers and should not therefore be seriously considered, given the different
     cost profiles associated with the provision of international and domestic facilities.
     The international passenger costs far more to handle than the domestic
     passenger. He or she will have more baggage, spends longer at the airport, and
     therefore needs more space.

26. The proportion of transfer passengers is also an important cost driver. Transfer
     passengers spend a limited time airside and do not usually use landside
     facilities. Airport costs associated with processing transfer passengers are
     correspondingly significantly lower than for other passengers. U.S. airports
     typically handle much greater levels of transfer passengers. (Chicago O’Hare
     70%, Dallas 65%).

27. BAA therefore believes that any attempted comparison of BAA London and U.S.
     airports would be extremely difficult and ultimately of little practical use. For
     Heathrow, therefore, Amsterdam, Frankfurt and Paris (CDG) are suggested as
     potential comparators. These have more similar passenger profiles, and the
     advantage that European airports have more commonality in the range of
     services offered. Even here caution is urged given the different ownership and
     cost related issues which exist within Europe.

28. For Gatwick, the same issues of international/domestic splits, transfer passenger
     mix and total passenger numbers exist in the suggested list of comparators.
     Gatwick’s international/domestic split is 91/9. BAA considers that the least
     inappropriate comparators are Madrid, Munich, Zurich and Rome.

BAA response                            9                                          CAA/44
The use of benchmarking                                                      February 2001
29. For Stansted, the international/domestic split is 85/15. Taking into account size
     and growth profiles, the least inappropriate airports to use are Dublin, Luton,
     Hamburg and Palma. Dublin is included here rather than for Gatwick, due to its
     size, profile and complementarity of traffic.

     Use of mathematical models
30. As regards the currently available econometric and mathematical programming
     techniques, (paras 2.5 to 2.11), it is apparent that there exist drawbacks in each
     technique. BAA commissioned a review of these issues, an extract of which is
     reproduced at Appendix 3 ;
     (i)     Cubbin and Tzanidakis’ study assesses the comparability of the efficiency
             estimates of regression analysis (RA) and DEA models as applied to the
             water industry in the UK. The sensitivity of the results to variations were
             significantly different for both techniques;
     (ii)    Parkin and Hollingsworth examined validity issues in DEA, and concluded
             that a mis-specified model would give severely misleading results; and
     (iii)   Thanassoulis’s study compared DEA and RA and concluded that DEA
             outperformed RA on accuracy of estimates, but RA gave greater stability of
             accuracy. Although challenged by Riddington and Cowie, Thanassoulis
             further concluded that DEA can be highly inaccurate for units that are
             ‘unusual’ and so lack suitable comparators. This could be particularly
             pertinent for airport comparisons.

Residual Differences with Top Down Cost Benchmarking
31. Even if it were possible to choose comparator airports with minimal exogenous
     differences, understand all the cost and activity differences between them, make
     good quality auditable adjustments, and overcome all the modelling difficulties
     involved, there would still be residual differences between benchmarked airports.
     Some of these have already been referred to but it is important to list and
     understand these before any future analytical work is commissioned:
     (i)     Capital Intensity; the substitution of capital for labour will introduce
             differences in operating costs between airports. Without any analysis of
             relative capital efficiency of airports (which the CAA do not seem to be
             proposing) this will remain as an important residual difference;

BAA response                              10                                          CAA/44
The use of benchmarking                                                         February 2001
     (ii)    Service Standards; there is inevitably a trade-off between operating and
             capital costs and service standards which any benchmarking exercise is
             unlikely to capture;
     (iii)   Operating Costs link to revenue; BAA has significantly higher resources
             committed to the generation of non-aeronautical revenue at its London
             airports than would be expected at other airports. These have been
             developed over a number of years and have associated additional costs
             (e.g. marketing, product promotion, retail operations staff). It would be
             wrong to compare the cost base without taking this into account;
     (iv) Airport investment is ‘lumpy’. This means that airports are at different
             stages of the cost cycle. Any comparison has to take due account of this.
             For example, newer facilities such as Malpensa and Denver are likely to
             show different cost patterns from Gatwick or Heathrow;
     (v)     Product innovation; “Innovation, action and achievement at BAA’s airports”
             sets out BAA’s track record on product and process innovation over the last
             5 years. These achievements do not happen by accident but require
             ongoing investment in human capital which inevitably means higher costs
             when looking at any snapshot of an airport’s profit and loss account; and
     (vi) Indirect costs; this problem has already been referred to in para 20(ii)
             above but will be further compounded by the five residual differences cited

32. BAA’s main conclusion from undertaking airport top down cost benchmarking
     itself as well as reviewing subsequent external studies attempting the same
     exercise is that the error range is probably at least as large as the real underlying
     range in efficiencies between airports. In other words one can never know if an
     observed difference in costs per passenger is due to a real difference in
     efficiencies or simply measurement error caused by all the difficulties already
     discussed. The data is also at such a high level that it cannot even point to
     which areas of the business to investigate in more detail should inefficiencies be

BAA response                             11                                         CAA/44
The use of benchmarking                                                       February 2001
The Use of Benchmarking by other Regulators
33. As regards the use of benchmarking by other regulators (paras 1.17 to 1.32),
     different approaches have been used for the price setting process. These appear
     to be characterised by fitting the models to the available data from comparator
     companies. BAA does not regard this as a robust and consistent approach, and
     is particularly concerned that decisions may be made on imperfect and
     potentially flawed data. This underpins BAA’s view that the results of such
     analyses are only as good as the information fed into them.

34. To illustrate this, OFWAT’s experience was that useful and detailed ‘bottom up’
    analysis could be performed on the companies it regulates in the UK. Clearly,
    OFWAT could rely on consistency of information, where the 20 companies being
    reviewed were all required to deliver the same product to the same standard,
    were all subject to the same Regulatory Accounting Guidelines, and the regulator
    could legitimately request detailed data on a consistent basis. However,
    international comparisons, as will be the case with BAA, proved far more difficult,
    with the result that only generalised conclusions could be drawn. In the case of
    BT, only summary data could be collected for comparators, with assumptions
    having to be made about what was included. OFGEM also had problems in
    ensuring consistency and accuracy, and decided not to give the ‘top down’
    approach significant weighting.

35. ORR’s review of Railtrack included a study by NERA of LEK’s efficiency
    comparisons of Railtrack against several U.S. railroad companies. The NERA
    study was unable to draw any conclusions on efficiency gains from this
    international comparison due to problems of differences in economies of scale
    and intensity of use. As a result ORR made little use of benchmarking studies in
    determining Railtrack’s efficiency targets but instead examined other regulators’
    efficiency assumptions, Railtrack’s observed cost reductions and academic
    literature on cost efficiencies post privatisation.

Process Examination - An Alternative Approach

BAA response                           12                                      CAA/44
The use of benchmarking                                                  February 2001
36. BAA believes that an examination of key processes is likely to be a more fruitful
     approach in assessing an airport’s efficiency rather than a top down
     benchmarking approach or an approach which attempts to consider the airport
     as a whole. Key processes should be examined using the following steps;
     (i)     Identify the key processes and costs at BAA London airports. Non-
             controllable areas and non material processes/costs should be stripped out;
     (ii)    Identify those which are outsourced or subject to competitive tender;
     (iii)   Provided CAA are satisfied that tendering had been conducted efficiently
             and effectively then no further analysis is required as market efficiency will
             by definition have been achieved;
     (iv) For the remaining key processes/costs which are not appropriate for
             benchmarking and have not been outsourced, the following questions can
             be posed:
     (a)     why has the activity not been outsourced?
     (b)     what evidence is there of proactive management using best practice tools
             and techniques?

37. Inefficiencies identified through process examination would provide management
     with clear objectives, and lead to quantifiable improvements through a target
     setting process. The following activities are proposed for this assessment, on the
     basis that they lie at the core of the business, and together account for a
     significant proportion of controllable business costs;

     (i)     Procurement;. many key areas are either outsourced (e.g. cleaning,
             maintenance, utilities) or have performance targets which could be
             compared externally.
     (ii)    Construction costs; there is benchmarking data available in several
             formats. For example, success in achieving per unit cost reductions over a
             period, product purchase costs (e.g. cost of steelwork, concrete supplies
             etc) or project by project comparisons (e.g. comparative costs of multi-
             storey car parks) have all been used to measure performance. Other
             parameters will also be important, such as adherence to industry standards,
             safety measures etc.

BAA response                             13                                         CAA/44
The use of benchmarking                                                       February 2001
     (iii)   Security staffing levels and costs; examination of the processes involved,
             the extent to which tactical forecasting is used to determine staffing levels
             for given traffic patterns, and average staff costs for the activity, would be
             an effective way of judging BAA’s performance.

38. Benchmarking is a useful tool and BAA has used this for product price and
     quality issues for several years.

39. In terms of regulatory theory cost benchmarking is only required where
     asymmetry of cost information exists and/or there is evidence of ‘gaming’. BAA
     believes there is no evidence of the latter and that if information asymmetry is an
     issue, it can be addressed by other means apart from benchmarking.

40. Cost benchmarking presents many challenges when being conducted on a top
     down airport-wide basis as including:-
     (i)     finding comparable airports;
     (ii)    identifying all the adjustments required;
     (iii)   obtaining the necessary data for these adjustments;
     (iv) statistical problems inherent in using the mathematical models proposed by
             the CAA (OLS, DEA, SFA etc).

41. Serious residual problems with the data are likely to remain even if these
     challenges can be resolved.

42. BAA’s experience suggests that results from top down cost benchmarking
     exercises are likely to have very wide error ranges which will at best only be very
     broadly indicative.

43. Nevertheless BAA accepts that cost benchmarking is a valid regulatory exercise
     and will of course fully co-operate in carrying out any international cost
     comparisons. The suggestions in terms of adjustments required and correct
     comparator airports should be seen in this light.

BAA response                              14                                         CAA/44
The use of benchmarking                                                        February 2001
44. As an alternative approach BAA will be producing evidence on key
     processes/costs which will identify:

     (i)     Those which are material and controllable;
     (ii)    Outsourcing/tendering where carried out;
     (iii)   Benchmarking information where available;
     (iv) Management initiatives/innovation.

BAA response                           15                                    CAA/44
The use of benchmarking                                                February 2001

To top