CS AAA ARD by jizhen1947

VIEWS: 6 PAGES: 51

									CROSS-COUNTRY ARD SECTOR AAA ASSESSM ENT




                October 14, 2004
                                              TABLE OF CONTENTS




EXECUTIVE SUMMARY……………………………..…………………………………............i-iii

1. OBJECTIVES AND APPROACH………………...……..…………..………. ........................... 1

2. ASSESSMENT OF THE AAA PROGRAM OF WORK............................................................. 3
     A.   Overall Findings ................................................................................................................ 3
     B.   Findings By Quality Dimension........................................................................................... 4
     C.   Systemic and Cross-Cutting Issues...................................................................................... 9
     D.   Priority Areas for Improvement and Next Steps ..................................................................13

TABLES
1.1 Expenditure on AAA by ARD and Bank-wide.................................................................... 2
2.1 QAG ESW and AAA Ratings, ARD Sector............................................................................... 3
2.2 Overall Assessment of Seven Countries .............................................................................. 4

BOXES
2.1 What is Good AAA............................................................................................................... 4
2.2 The Ukraine Policy Group……………… ............................................................................ 9
2.3 Broad or Narrow Sector Work? ..........................................................................................10
2.4 A Supply-Driven Success .....................................................................................................13

ANNEXES
1.   Approach Paper.......................................................................................................................15
2.   List of AAA Reviewed..............................................................................................................18
3.   Country Findings .....................................................................................................................19
4.   Comments on the Technical Quality of Specific Studies in Bangladesh, Mexico and Tanzania......40
                                EXECUTIVE SUMMARY

1.      An assessment of Analytical and Advisory Activities (AAA) was carried out by QAG in
June/July 2004 for the Agriculture and Rural Development (ARD) sector, to review the quality
of sector work in the context of the Bank’s strategy paper, Reaching the Rural Poor, approved by
the Board in October 2002. The QAG review examined the “coherence” between the ARD
sector strategy and a sample of recent sector tasks undertaken in seven countries, focusing on
aspects such as technical quality of ARD analytic work, contributions to the CAS, PRSP and
country rural strategies, as well as impact on the Borrower, Bank lending strategy and on other
donors and stakeholders.

2.      Considerable improvement in the quality of AAA activities managed by ARD was
registered during the last two ESW assessments (FY00-02) managed by QAG. Results from the
current review show continued improvement and the overall rating for 20 activities managed by
ARD regional units was 95% Satisfactory or better (all ratings in this report are based on QAG’s
new six point scale). While the quality of individual tasks is high, the amalgamation of tasks at
the country level reveals shortcomings in coherence and impact. Of the seven countries
examined in this review, three (Mexico, Morocco and Ukraine) had satisfactory ARD sector
work overall, though no country was rated Highly Satisfactory. Two countries (Kazakhstan and
Tanzania) had Moderately Satisfactory ratings, and two countries (Bangladesh and Ghana) were
rated, respectively, Moderately Unsatisfactory and Unsatisfactory for quality of ARD sector
work.

3.      Implementing the goals of the new rural strategy remains a work in progress. Reaching
the Rural Poor was a major exercise to reposition and refocus the ARD sector as a priority locus
of business activity for the World Bank based on the preponderance of poverty in rural areas.
The strategy challenged those inside (and outside) the sector to reassert the strategic importance
of rural development through enhanced performance in lending and AAA. It emphasized that
“improvements in the well-being of the poor will only be possible through enhancement of their
productive, social and environmental assets. This means increasing the productivity and growth
of both the farm and non-farm economies.”

4.      Attaining the goals set out in the Reaching the Rural Poor strategy is bound to be a slow
process. With many of the tasks included in the sample having been launched prior to Board
approval of the new direction, the strategy is too new to have had a significant impact on the
strategic direction of these activities, or on the volume of AAA in the sector. From an overall
perspective, however, the Panel found that in several instances the effectiveness of ARD sector
work at the country level lacks consistency and coherence over time, is poorly integrated with
lending programs, and reveals weak linkages with other sectors and other donors’ programs.

5.     The Panel recognizes that the means for achieving many of the strategy’s objectives
depend on budgetary decisions taken by Country Directors and thus lie outside the control of
ARD managers and staff. But some are within their control, and that provides the Panel with an
opportunity to offer recommendations that may assist in achieving the goals. At the same time,
the Panel wishes to alert country departments of their role in making the ARD strategy work by
according it the priority it warrants in country work programs. The ARD strategy is a Board-
approved document that clearly identifies the fact that poverty is more prevalent in rural areas.
Cross-Country ARD Sector AAA Assessment                                                               ii


Yet the Panel found that rural poverty is still dealt with in a cursory manner in CASs, CEMs, and
even PRSPs.

6.        Key findings emerging from this review include the following points:

     •    Technical quality of analytic work at the task level was uniformly high for tasks managed
          by ARD units, but most of the tasks managed by non-ARD units that were identified as
          having sections pertinent to the rural sector were not particularly relevant.

     •    ARD AAA is too often fragmented, ad hoc, financed off-budget, and poorly documented in
          the Bank's internal records, weakening institutional memory and undermining the coherency
          of AAA work programs.

     •    Sector "silos" are very apparent in the AAA work program, with little or no evidence of
          interaction between sector departments.

     •    The integration of ARD AAA into lending was highly variable. Three countries showed
          good integration of sector work (Morocco, Mexico, Ukraine) but two had only tenuous links
          to the portfolio (Bangladesh, Ghana). Sector-wide, coherence and integration was the
          weakest dimension of AAA work.

     •    While AAA often shows good impact on other donors, the Bank's tendency to want to always
          take the lead, and often to overlook analytic work done outside the Bank, has probably
          undermined true synergy in many countries.

     •    Reports too often were not translated into local languages or formally published, which
          reduced impact. Also, the Bank needs to make more use of Web-based dissemination for
          AAA reports.

     •    The quality of management input into AAA tasks was variable. Larger analytic tasks showed
          strong involvement from the conceptual stage through to implementation and dissemination,
          but in smaller tasks management attention was often hard to detect after the initial concept
          stage.

     •    The Bank appears to have a very porous institutional memory and an entirely inadequate
          filing system (electronic or otherwise). Many reports were missing or could only be located
          with difficulty.

7.       The following recommendations are addressed to specific Bank audiences that have a role to
         play in helping strengthen the impact of AAA work on reducing rural poverty:

For the ARD Sector Board
     •    Continue to publicize and disseminate best practice in AAA.

For ARD Regional Sector Units
     •    Review measures that would help ensure the availability of experienced sector economists
          that can devote most, if not, all their time to AAA activities.
Cross-Country ARD Sector AAA Assessment                                                             iii



   •   Sector staff should continue to champion the importance of AAA by emphasizing the
       preponderance of poverty in rural locations and the value of ARD AAA in meeting the
       Bank’s poverty alleviation objectives.

   •   Greater use should be made of QERs, especially for less experienced task managers. This
       should be combined with more extensive use of external reviewers.

   •   Make more extensive use of studies conducted outside the Bank, conduct literature searches
       before beginning a task, and consider updating old studies before launching new ones
       covering the same topic.

   •   Greater efforts are needed to accurately record AAA tasks in the SAP and to ensure that
       reports are properly archived and readily available to the staff and the client.

For Country Departments

   •   Achieving the Bank’s poverty reduction objectives requires greater prominence for the rural
       sector in country programs. This calls for a more consistent allocation of funds to ARD
       AAA to maintain the increase seen in FY04.

   •   Ensure that rural poverty aspects are properly integrated into the analytic work of other
       sectors and guard against the tendency for staff to remain fixed within narrow sector “silos.”

   •   Earmark funds for dissemination of AAA findings and recommendations and for posting
       them on the web.


For QAG

   •   This pilot review was not able to fully address some issues due to the limited interaction with
       staff and managers. This aspect should be reviewed in any future assessment of this type.

   •   The impact of sector AAA on country lending programs and other stakeholders is best
       examined in conjunction with field visits that provide a much deeper basis for informed
       judgments.
                           I.    OBJECTIVES AND APPROACH

1.1      This exercise is a pilot review of the Agriculture and Rural Development (ARD)
Analytical and Advisory Activitie s (AAA) undertaken during FY01-04 in seven countries and
five regions, broadly representative of the different conditions being faced by ARD staff. The
sample also included rural- related AAA studies conducted by other units. The objectives of the
review are to assess the overall quality and adequacy of AAA in the rural sector, the impact of
AAA and the Sector Strategy Paper, Reaching the Rural Poor, on country rural strategy and
activities of other donors, and its contribution to the formation of the Bank’s lending
strategy/project design in the sector. Although many of the tasks in the sample were launched
before the sector strategy paper was approved by the Board, the Panel found that most of them
were closely aligned with the broader vision of the strategy. One objective on which the Panel
could not make a judgment through the pilot assessment process was the extent to which the rural
sector strategy paper and AAA activities have brought about changes in the content and quality
of the Bank’s lending activities in the rural sector. Documentation available on the entire AAA
process was gathered and read by the panelists. Limited information was collected on the topic of
“Bank inputs and processes,” but no country management staff were interviewed. With the
exception of Bangladesh and Mexico, no field visits were undertaken. Five countries were
chosen on the basis of their inclusion in a current or previous QAG country AAA assessment,
and two countries were suggested by ARD as representing examples of very good and rather
inadequate AAA. (Detailed terms of reference are provided in the Approach Paper, which is
Annex 1 of this report.)

1.2      The review was conducted by a team of seven sector specialists and employed a phased
approach. First, 31 individual tasks (27 ESW and 4 NLTA) were assessed. Out of this total, 20
activities were managed by ARD units, while the balance were tasks managed by other units but
identified as having relevance to work being carried out in the rural/agriculture sector. It should
be emphasized that the non-ARD AAA activities reviewed here were examined strictly from the
point of view of their contribution to advancing the ARD analytic agenda; this was not an
assessment of their individual merit.

1.3     In the second phase, individual task results were aggregated at the country level with
panelists paying close attention to the comprehensiveness of the AAA program, its likely impact
on, and strategic relevance to, the country’s rural strategy, taking into account the sector strategy
paper approved by the Board and referred to above. The panel also looked at the quality of the
Bank’s management of the ARD AAA program in each country. (Results from this part of the
assessment must be handled with caution since in at least three countries only a small number of
ARD AAA activities were carried out, while in others some of the tasks were only indirectly
related to the rural sector.) The assessment’s third phase is the overall sector performance. With
the studies included in the sample accounting for about 13% of total AAA activities conducted in
the sector, the sample size is not large enough to yield robust results at the Bank-wide level, but it
does provide an opportunity to learn several lessons and establish trends. Finally, three AAA
tasks were examined by an independent specialist in agriculture and rural development to identify
strengths and weaknesses related to technical quality and policy relevance (see Annex 4).

1.4     Table 1.1 provides a breakdown of the expenditure on AAA tasks managed by the ARD
units and Bank-wide. At this time, there are no benchmarks to determine the appropriate level of
sector activities and funding, because these must be determined on a country-by-country basis
according to need. As can be seen, until FY04 the amount allocated to ARD AAA has been
essentially flat. In FY04, there has been a significant increase in expenditure on ARD AAA, with
about half the increase coming from trust funds and the other half from the Bank’s budget, but at
Cross-Country ARD Sector AAA Assessment                                                                 2


this time it is unclear if this recent increase is a temporary ‘blip’ or a longer-term shift to a higher
level of funding, though the increase is in line with the recommendations of this report.

TABLE 1.1 EXPENDITURE ON AAA BY ARD AND BANK-WIDE
                                  FY00-04 ($ million)

                                                                    ARD as
                                             ARD         Bank      % of Bank
                              FY00            9.2        196.6        5%
                              FY01            6.8        181.1         4%
                              FY02            9.4        195.7         5%
                              FY03            10.3       253.3         4%
                              FYO4            16.4       245.4         7%


1.5      In line with the agreement reached with the ARD Sector Board, this pilot review was a
desk study and interviews with task and sector managers were limited to a single meeting. This
introduced some limitations. Conducting the assessment without interviewing all sector and
country staff involved in a task often made it impossible to pursue central issues in detail. There
were no individual task discussions (though some had previously been assessed by QAG), and
the lack of a field visit meant that the opinions of the client and other stakeholders could not be
assessed directly. Moreover, in three of the sample countries, only a small number of ARD AAA
activities were carried out, while in others, some of the tasks were only indirectly related to the
rural sector. Finally, robustness of the findings from aggregating the seven country assessments
is affected by the small sample size.

1.6    Ratings were carried out using QAG’s newly-introduced six-point rating scale, with the
following guidelines for assessment:


     1 – Highly Satisfactory              Best practice in most respects and no significant deficiencies
     2 – Satisfactory                     Satisfactory or better on all aspects
     3 – Moderately Satisfactory          Satisfactory or better on all key aspects but significant missed
                                          opportunities
     4 – Moderately Unsatisfactory        Significant deficiencies in a few key aspects
     5 – Unsatisfactory                   Significant deficiencies in several key aspects
     6 – Highly Unsatisfactory            A broad pattern of deficiencies
Cross-Country ARD Sector AAA Assessment                                                         3


           II. ASSESSMENT OF THE AAA PROGRAM OF WORK

A. OVERALL F INDINGS

2.1     Recent QAG assessments of ESW tasks (FY99-02) have found the quality of ARD
analytic work to be close to the Bank-wide average (Table 2.1). The present assessment, which
is based on a review of 31 tasks in seven countries, found that tasks managed by ARD units were
of high quality (95% satisfactory rating for 16 ESW tasks and four non-lending TA) and were
closely aligned with the agenda outlined in the sector strategy paper. Sections of studies
managed by other units that were identified as being pertinent to the rural sector, however, were
found to be of limited relevance and of inferior quality. When aggregating the individual tasks at
the country level, the review found a pattern of deficiencies in analytic work and limited impact
on the country/Bank strategic approach to the sector. Although the sample is too small to derive
conclusive Bank-wide results, the Panel judged the quality of ARD AAA work at the country
level for two out of the seven countries in the sample to be Moderately Unsatisfactory overall
and the others were rated Satisfactory. The strongest aspect of the ARD AAA work program at
the country level was the internal quality of analysis, while strategic relevance, coherence and
integration, and likely impact demonstrated significant shortcomings.

                          Table 2.1 ESW and AAA Ratings, ARD Sector

                                                             % MODERATELY
                                                             SATISFACTORY
                                                               OR BETTER
                QAG ESW Assessments, FY99-02
                  • Bank-wide rating                                 85%
                  • ARD rating                                       81%
                QAG ARD Sector AAA Assessment, 2004
                  • ratings for 20 AAA tasks                         95%



2.2    Within the sample of countries assessed by QAG, there were of course specific examples
where these weaknesses were not apparent, but overall the shortcomings seem to be in these
areas:

   •   Inappropriateness of the coverage, scope and relevance of ARD AAA program in the
       context of the CAS framework, and in relation to the importance of the rural sector in the
       economy, and to the percentage of people classified as poor;

   •   Failure to effectively engage government, local agencies and other donors in carrying out
       the tasks and disseminating results;

   •   The absence of a coherent long-term strategy for undertaking AAA activities in a
       country, and to clearly define objectives and target audiences; and

   •   Limited impact of the analytical work on the client and the Bank due to shortcomings in
       dissemination, the short “shelf-life” of reports, and inadequate systems for archiving,
       updating, and accessing AAA reports.
Cross-Country ARD Sector AAA Assessment                                                                4



                                    Box 2.1 What Is Good AAA?

   Drawing on its accumulated experience in assessing AAA, QAG has identified key characteristics of
   good sector work. Although they were formulated for ESW, they are equally true for AAA and
   were used in this assessment:
   •   Clarity of purpose, intended audiences, and expected impact
   •   Operational relevance and client receptivity
   •   Clients and stakeholders as partners with well defined roles
   •   Sound, customized analysis blending local knowledge with global insights
   •   Substantive conclusions, clearly presented and argued
   •   An effective program of dissemination and follow-up
   •   Realistic budget and timetables
   •   Agreed quality assurance mechanisms and monitoring milestones




B. Findings by Quality Dimension
2.3     As pointed out in paragraph 1.5 results from aggregating the findings from the seven
countries included in the sample are not statistically robust and should be interpreted with care.
An assessment of this limited sample shows the strongest aspects at the country level to be the
internal quality of activities, dialogue and dissemination and Bank inputs. Areas in need of
improvement are strategic relevance, coherence and integration, and likely impact (Table 2.2).
The following paragraphs summarize the findings for each of the six dimensions used in the
assessment. On just about every dimension of the review, the findings include examples of good
practice but also many areas in need of improvement.

                     TABLE 2.2 OVERALL ASSESSMENT OF S EVEN COUNTRIES


                 Overall Rating                4   MODERATELY UNSATISFACTORY

                 Strategic Relevance           4   Moderately Unsatisfactory
                 Internal Quality              2   Satisfactory
                 Dialogue and Dissemination    3   Moderately Satisfactory
                 Coherence and Integration     4   Moderately Unsatisfactory
                 Likely impact                 4   Moderately Unsatisfactory
                 Bank Inputs                   3   Moderately Satisfactory


2.4    Strategic Relevance. This dimension measures the appropriateness of coverage, scope
and relevance of sector AAA programs in the context of the CAS framework and countries’
PRSP/CDF or equivalent strategy, and relevance of sector AAA to the Bank’s sector strategy
paper.

    • Examples of Good Practice: (i) Strategic planning. The Morocco sector studies
      followed the CAS outline of a three-year program to expand and complement the
      available analysis and contributed to the development of strategic planning; (ii) Focus
      on poverty aspects. In Tanzania, where there is considerable rural poverty and a history
      of zero sector growth, the AAA program was focused on changing incentives to promote
Cross-Country ARD Sector AAA Assessment                                                         5


        growth in a manner that takes account of poverty implications; and (iii) Sector Strategy
        Paper. Most tasks are closely aligned with the broad vision of “Reaching the Rural
        Poor,” are consistent with that paper’s strategic options, and have broadened the analysis
        from agriculture to rural development more generally. The Kazakhstan Livestock study
        is a good example of a sound strategic choice of analytic orientation with excellent
        coverage of the poor population.

    •   Aspects Requiring Improvement: (i) Coverage of sector studies. The totality of the
        AAA program does not address integrated agriculture and rural development and growth
        issues in a coherent manner, and frequently overlooks key issues in the sector; (ii)
        Descriptive studies. A number of studies included in the sample are descriptive of the
        current situation rather than being forward looking and aimed at helping government
        overcome emerging problems; (iii) Conclusions and Recommendations . In many
        studies the conclusions and recommendations tend to be quite broad, giving the borrower
        few indications of priorities and sequencing of recommendations; and (iv) Studies
        alignment with the CAS. In some instances, the AAA activities were not aligned with
        the poverty reduction objectives of the CAS, and in the case of Ghana, where poverty is
        predominantly rural, the absence of an ARD AAA program is a serious shortcoming.

2.5     Internal Quality. This dimension examines the technical quality of sector AAA analysis,
and the quality of the empirical evidence used (both quantitative and qualitative).

   •    Example of Good Practice: (i) Use of earlier studies. In Ukraine, AAA tasks as a
        group built on strategic work carried out in the 1990s, addressed key issues, and were
        high quality. The work greatly benefits from its use of empirical evidence, notably the
        database established within the Agriculture Policy Group; and (ii) Studies rated Highly
        Satisfactory. Other AAA tasks rated highly satisfactory include the Bangladesh Climate
        Change Study, the Morocco Agriculture Sector Note, and the Morocco Cereals report.

   •    Aspects Requiring Improvement: (i) Use of Data. Inadequate attention to the
        collection of raw data within the country (this could frequently be best achieved by
        engaging local institutions as was done in Kazakhstan and Ukraine); (ii) International
        Experience . Many studies fail to make effective use of examples from, and convincing
        explanation of the experience in, other countries that have successfully addressed similar
        sector problems; (iii) Actionable Recommendations . Studies frequently contain
        numerous recommendations that are not very explicit and “actionable,” and (iv)
        Literature Search. Studies are making limited use of empirical evidence and frequently
        fail to make extensive search of sector-relevant literature (including non-Bank analytic
        work, which is overlooked in many studies).

2.6     Dialogue and Dissemination. This dimension examines the contribution of sector AAA
programs to promoting policy dialogue and institutional development at the country level, the
quality of client participation, and partnership arrangements with other donors.

   •    Examples of Good Practice: (i) Client led studies. Where studies are managed by the
        country (or independent local units) and with broad participation from a wide range of
        stakeholders, ownership is enhanced, especially where this is combined with the Bank
        assuming a supportive role and furnishing the local experts with worldwide example of
        how other countries have dealt with similar issues; (ii) Dissemination of studies:
        Dissemination efforts through workshops and the translation of reports into the local
        language enhance the impact of the studies on policy makers in the country; and (iii)
        Collaboration with donors. There are many examples of joint financing of studies with
Cross-Country ARD Sector AAA Assessment                                                         6


       other donors but by far the most effective dialogue occurs when the donors are actively
       participating in the preparation of a study. In general, donors’ programs appear to be
       influenced by the findings of Bank AAA. Individual tasks noted for good attention to
       dissemination include the Mexico Land Policy Review, the Kazakhstan Fisheries sector
       study, and the Bangladesh Poverty Assessment.

   •   Aspects Requiring Improvement: (i) Borrower participation: Active client
       participation in the preparation of studies, which generally enhances ownership, is limited
       in most instances; (ii) Shelf life of studies. Many reports included in the sample have a
       far too-short shelf life in an environment where policy issues really do not change that
       much across several years. There are few examples of attempts to keep reports alive and
       to update (rather than re-create) them at appropriate intervals; (iii) Dissemination of
       studies. The arrangements for disseminating studies are often an afterthought. Efforts are
       unplanned, unbudgeted, and simply not undertaken in a workmanlike manner. As a
       result, client uptake is weak. Reports are rarely formally published, and few are available
       in short policy note format (translated to the local language) and posted on the web.
       There are even instances where policy studies intended to influence policy makers were
       not delivered to government; and (iv) Capacity building. Few of the AAA programs
       include capacity building as an explicit objective. Where this has taken place (Ukraine,
       Morocco, and to a lesser extent Kazakhstan), it has substantial impact on the country’s
       analytical capacity and the ownership of the studies and their recommendations. (See Box
       2.2 for an example of an innovative capacity building effort that has proven highly
       effective.)

2.7     Coherence and Integration. This dimension assesses the coherence of sector AAA in
terms of inter-linkages across other AAA tasks and sector or thematic areas, integration of sector
AAA with lending programs, synergies with other donor programs, coherence of sector AAA
over time, and consistency of effort.

   •   Examples of Good Practice: (i) Impact on Bank lending. In Ukraine, AAA work has
       had direct impact on the Bank’s lending program; and (ii) Programmatic approach. In
       Morocco, a three-year programmatic approach to AAA has resulted in excellent
       coherence of analytic work over time, while consistency of effort benefited from
       designating a lead sector economist with adequate resources for this role.

   •   Aspects Requiring Improvement: (i) Linkages with other sectors. Frequently,
       linkages with other sectors are not readily apparent and in fact AAA tasks are often
       “enclave”-type undertakings. In the cases of PERs, for example, which are supposed to
       pull together themes for various sectors, the technical input from ARD sector specialists
       is often limited to reviewing drafts at a late stage when such inputs are least effective.
       Even in countries in which the Bank is heavily involved in areas such as rural
       infrastructure, rural education and health, integration of ARD sector work was often hard
       to detect; and (ii) Linkages with lending operations . Sector studies are frequently
       undertaken to justify/support operations in advanced stage of preparation instead of
       preceding such preparation efforts.

2.8     Likely Impact. This dimension measures the likely impact of AAA programs on clients,
on the Bank, on other donor programs, and on other stakeholders and beneficiaries (NGOs, civil
society, the private sector).

   •   Examples of Good Practice: (i) Client led studies: Attempts to building ownership and
       coalitions for change have the biggest impact where studies are channeled though local
Cross-Country ARD Sector AAA Assessment                                                          7


       working groups and participatory methods are used for analytic tasks; (ii) Impact on the
       Bank and Client. Morocco’s p       rogrammatic approach to AAA has improved impact
       within the Bank, and could have broader benefits if the approach is adopted in other
       sectors and countries. In Ukraine, partnerships with key donors are well developed and
       policy priorities are clearly agreed within the group. The establishment of the Policy
       Analysis Group is a best practice, an innovative example of enhancing country capacity
       and knowledge transfer. Individual tasks which were highly rated for likely impact
       include the Mexico Crop Insurance report, the Mexico Land Policy review, the Ukraine
       Agriculture Policy Note, the Morocco Agriculture sector note, and the Bangladesh
       poverty assessment.

   •   Aspects Requiring Improvement: (i) Client impact. Studies that fail to have an impact
       on the borrower suffer from inadequate client focus, absence of an effective and long-
       term strategy for dissemination, the absence of clear conclusions and “actionable” and
       sequenced recommendations; and (ii) Protecting the Bank’s Knowledge base. Failure
       to complete ana lytic tasks and to ensure ready access to final reports diminishes the
       impact of sector studies on the Bank.

2.9     Bank Inputs and Processes: This dimension examines the adequacy of funding, the
efficiency of resource use, the quality of support from other Bank units, managerial attention to
AAA quality, the effectiveness of the peer review process, monitoring and evaluation of sector
AAA, and the quality of AAA information in the SAP and Business Warehouse.

   •   Examples of Good Practice: (i) Management Input. In general, it appears that
       managerial attention is quite high at the inception phase of large tasks, and in these
       instances there is evidence of lively discussion on the merits and composition of the
       activity in point. Where country management is also involved in AAA tasks, this
       frequently pays off in terms of high quality product and high impact; (ii) Efficient use of
       resources. Major tasks are making efficient use of available resources, and (iii) Client
       and stakeholder feedback. In Ukraine, the Agricultural Policy Group has proven to be a
       very effective conduit for sharing views and soliciting feedback on sector analytic work.
       At the individual task level, no AAA tasks were rated highly satisfactory with respect to
       Bank inputs and processes.

   •   Aspects Requiring Improvement: (i) Multi-year budgeting. Under current Bank
       systems, task teams are unable to make efficient use of resources because of uncertainty
       of budget allocation from one fiscal year to the next; (ii) Country team’s participation.
       Greater active involvement from country economists and other senior members of
       country team in the selection and preparation of AAA activities would probably help to
       strengthen the integration of the tasks with others being undertaken in the country; (iii)
       Monitoring and Evaluation. Consistent monitoring of ARD’s sector activities at the
       country level and the consistency of these activities over time was particularly lacking;
       (iv) Peer reviews. The Bank’s peer review process appears to have deteriorated;
       evidence points to cursory reviews, infrequent attendance by designated reviewers in
       review meetings, and patchy use of external reviewers; and (v) Implementation and
       Dissemination. Management attention to AAA tasks seems to fall off after the initial
       concept review, which weakens the impact of the work.

2.10 There are several implications arising from the findings on Bank inputs. As shown in
Table 1.1, with the exception of FY04, budgetary allocations (both TF and BB) for AAA
activities in the sector have remained essentially flat; this was also true for the Bank as a whole
though here the sharp increase in expenditures preceded that in ARD by one year. The Panel was
Cross-Country ARD Sector AAA Assessment                                                           8


unable to judge whether the sudden increase in budget in FY04 is the beginning of a trend or a
‘blip,’ but noted that about half the increase was from trust funds, which would seem to indicate
an aggressive strategy on the part of staff to secure more resources rather than an explicit policy
decision by country departments to provide enhanced budgets to ARD AAA. Unless the level of
expenditures is sustained at about the FY04 level, the Panel is concerned about the prospects for
improving and sustaining the Bank’s analytic and policy commitments in agriculture and rural
development, for several reasons.

   •   First, it is increasingly difficult for sector managers to retain well-qualified economists
       who are fully dedicated to sector work. At best, sector economists who are fungible with
       lending work are selected, and when this occurs it is inevitable that first priority is given
       to lending activities. This is a logical sector management reaction to limited and
       uncertain AAA budgets, but it has serious implications for the flow of quality AAA work
       and the generation of knowledge. There are several examples of experienced sector
       economists leaving the sector for PREM or other positions and so being “lost” to the
       Network.

   •   Second, there are serious deficiencies at the AAA work program level which would not
       be redressed by occasional increases in funding. The review came across only one
       example of, say, a planned, rolling three-year work program of AAA and a notional work
       program, loosely defined, ready to respond according to resource availability. The
       general absence of a meaningful AAA work program means that it is very difficult to
       assemble a comprehensive and cumulative knowledge base for the country, and to
       maintain an effective staff skill mix focusing on analytic work. Thus AAA work tends to
       be reactive rather than proactive, and inconsistent in focus from year to year.

   •   Third, under the Bank’s present structure and budget, task managers of AAA more often
       simply manage the process, outsourcing the fieldwork and writing to consultants. The
       Panel raises the question of the implications of this. Where will the sector get its next
       generation of task managers skilled in sector work, policy analysis, and policy dialogue?
       What are the implications for quality and the “knowledge bank” if this should persist?

   •   Fourth, insufficient funding for AAA also means that task managers must spend
       considerable time chasing and managing trust funds and other off-budget sources of
       funding. This would not appear to be the best use of their time, nor is it appropriate that a
       core Bank activity and area of comparative advantage should be so exposed to the whims
       of trust funding. (As seen in Table 1, however, compared with the rest of the Bank BB
       funding accounted for a much higher share of the total.)
Cross-Country ARD Sector AAA Assessment                                                                     9



                                  BOX 2.2: THE UKRAINE POLICY GROUP

     Ukraine was one of the slowest agricultural sector reformers in the Former Soviet Union during the
     1990s. The Bank’s traditional approaches to inducing policy change were not working. We were
     seen as outsiders who did not understand their problems. The decision was made to establish a
     policy analysis unit, with a staff of about 10 Western-trained Ukrainian nationals. The goal of this
     unit was to carry out policy analysis for key policy makers who were having problems getting access
     to analytical support for their ideas. Several lessons were learned:

         •   National Policy Advisors are Critical. In the Ukrainian context, policy advice provided by
             foreign advisors had minimal impact. Western trained national policy analysts could more
             effectively assimilate new policy ideas, and communicate them to senior policymakers.

         •   Re-training the Old Guard is not effective. This project demonstrated that the focus on
             educating a new generation yields far better results than trying to reform the “Old Guard,”
             and that it may often be more effective to create new institutions than to try to reform old
             ones.

         •   Support of a “Champion” is Essential. The policy analysis unit was fortunate in that it had
             a series of enthusiastic “champions” at the highest levels of Government who supported
             them and made use of their policy advice.

         •   Financial Autonomy is Important. Financial autonomy has been important to the group in
             terms of its ability to provide objective, independent advice and also to move their support
             to wherever it was required by reformers in Government or the Parliament.

         •   Donor Coordination Improved. Having a commonly-funded policy advisory unit was
             important in harmonizing donors’ policy advice.

         •   Cost Effectiveness. For the Bank, this was a low cost - high impact intervention. Using
             about $700,000 in Bank Budget over 6 years, the region mobilized an additional $5 million
             in funding from USAID, EU and DFID. The policy analysis unit has produced hundreds of
             quick turnaround reports, comments on laws, and meetings with policymakers which have
             significantly improved the agricultural policy environment.




C.       Systemic and Cross-Cutting Issues

2.11 One over-arching issue encountered in the assessment was the rather low level of
importance assigned to ARD AAA in country programs in relation to the importance of the rural
sector in the economy and as a source of the poor. In many respects agriculture and rural
development appear to have taken or been assigned a secondary role, even though these are
sectors that might appropriately be seen as a core business line for the Bank, with high
importance for poverty reduction and the prospects for achieving the Millennium Development
Goals. Panel members expressed concern over the disconnect between the importance of
agriculture and rural development to economic performance, growth, and poverty reduction, and
the amount of budget allocation to ARD AAA. In many countries, only one piece of ARD AAA
is conducted per year, and in some years, none are undertaken. It is true that new activities, such
as PRSPs, do have significant ARD components yet do not get “counted” as ARD activities. Yet
the Panel questions whether, as the “Knowledge Bank” on which many other donors also depend,
Cross-Country ARD Sector AAA Assessment                                                               10


the World Bank is investing sufficiently in ARD knowledge and whether it is fulfilling its Board
mandated strategy. This is a matter for the attention of country departments.

2.12 Directly related to the question of defining the audience is clarity of objectives. ESW and
NLTA have multiple objectives depending on the particular piece of work and context. A well-
designed piece of AAA identifies up-front the prime audience and casts the report with the prime
audience in mind. What the Panel failed to find was a systematic discussion at inception of: (i)
the use to which a report would be put, (ii) a clear identification of the target audience, and (iii)
how the target audience might affect the design of the work. (See Box 2.3 and Annex 4 for
further details). Seldom is the prime purpose and audience clearly spelled out in the concept
paper, and thus many Bank reports miss an opportunity to send focused and sharp messages to
the prime client. This also has consequences for the dissemination strategy, which often seems to
lack focus and follow -up.

2.13 Related to this is the issue of demand-driven versus supply-driven AAA work. In the
recent past, it was recognized that the Bank undertook too much supply-driven AAA and country
uptake and ownership tended to suffer. “Reaching the Rural Poor” concluded that to be effective,
rural strategies must be “owned” by the country, and preferably the team undertaking the work
should include government officials and local experts. The Morocco rural strategy is a good
example of such an approach. But there are also dangers of taking a demand-driven approach to
the extreme where the Bank has very weak or little influence on the analytic agenda. In countries
where decisions on sectoral investment allocation are made by political elites who are not very
interested in agriculture or rural issues, adherence by the Bank to the principle of demand-driven
sector work tends to exacerbate such biases. In these countries, the ARD sector work often tends
to be under-funded relative to its importance to the economy and to poverty.

2.14 Sometimes this becomes extreme, such as in Kazakhstan, where only work approved and
co-financed by the Kazakh government is undertaken. The Panel found that although AAA is
being carried out on several special topics that inform the Bank’s lending in the country, there

                            BOX 2.3: BROAD OR NARROW SECTOR WORK?

   In practice, a feasible agenda for sector reviews has to focus either on broad-ranging description
   with relatively superficial analysis (“wide and shallow”), or a relatively narrow focus with more
   detailed analysis of causes of problems (“narrow and deep”). The prevalent approach to sector work
   appears to be the wide-and-shallow type. When the sector is as complex as the rural sector, this
   descriptive task alone is a large undertaking. Combining this with in-country expert opinion and the
   Bank’s general experience-based positions on policy matters, one model of sector work is a report
   that presents a large amount of data along with fairly generic policy recommendations.
   The narrow-and-deep model of sector work is a report that focuses on a few issues, and analyzes
   them in detail. An example of this genre is the paper on Mexico’s program of privatizing land
   holdings. The work involved two substantial surveys of residents affected by the policy changes, an
   econometric analysis of the data, and conclusions based narrowly on the findings.

   The principal problem with narrow-and-deep sector work is that it is costly per conclusion. It would
   be impossible to fund such work for every lending program and policy reform supported by the
   Bank. The principal problem with wide-and-shallow sector work is that many of the
   recommendations given are weakly evidence-based and may fail to convince a skeptic that they are
   is preferred to alternatives. Thus, what genre of sector work is preferable? Clearly, both genres
   have a function that provides value to the Bank and its clients. The important point is that well-
   managed sector work will identify, early on, the clear purpose of the work and this will inform the
   decision on wide versus deep.
Cross-Country ARD Sector AAA Assessment                                                                        11


has been no recent comprehensive sector strategy, survey, or assessment. The risk in such a
situation is that the Bank conducts studies of limited relevance and is unable to undertake broad
policy/strategy pieces of work that serve to provide a vision for the sector. 1 In so doing, the Bank
is losing knowledge and compromising its unique strength of having a broad and encompassing
view of the sector. It can no longer provide informed strategic advice to certain governments and
is in danger of losing its seat at the agriculture and rural development policy table.

2.15 AAA work reviewed in Mexico provides an interesting counter-example of supply-driven
AAA, where the Bank unilaterally undertook a major piece of informal sector work that became
hugely important to the incoming government and provided a roadmap for Bank intervention and
dialogue (see Box 2.4). The panel concludes that a balance is required between supply-and
demand-driven approaches, and that careful judgment is needed on that balance. Some work that
is initiated unilaterally by the Bank to fill knowledge gaps or to respond to topics ignored by the
country is needed. At the same time, extreme demand driven approaches as is seen in
Kazakhstan come at a cost to the Bank’s knowledge and comprehensive view. As in much of
life, polar solutions are seldom ideal.

2.16      Cross-sectoral collaboration in ARD AAA is seldom being practiced. The panel could find
only one example where cross-support from specialists in other sectors was employed. There is
considerable evidence of ARD staff collaborating with others in activities such as the PRSP, CAS,
CEM and PERs, but there is very little the other way round. The sectoral silos remain robust in the
Bank. The present structure and task budgeting makes cross support problematic. Staff aspire to task
management and there appear to be no incentives for collaboration. Yet sectors do not operate i      n
isolation and strong inter-relationships and overlaps occur across sectors and expertise. Ways to
facilitate cross support would benefit the Bank. The use of specialist anchor staff is not systematic,
either. Some regions use anchor and DEC staff well, but they are the exception rather than the rule.
The evidence seems to be that the Bank is not bringing state-of-the-art expertise to the task at hand
and therefore the tasks often do not represent cutting-edge knowledge on the topic.

2.17     Related to cross-sectoral collaboration is the use of external expertise. In some cases,
international experts were used as reviewers or were part of the team. But this seemed too infrequent.
Further, subcontracting AAA or parts of AAA (data collection and analys is, say) to local think-
tanks/consulting firms was the exception rather than the rule. In instances where this was practiced, it
appeared to be an effective way to leverage resources while at the same time contributing to local
knowledge-enhancement and country ownership. Some TTLs use external expertise well on teams
but very infrequently as peer reviewers. ARD once had a roster of internal and external reviewers
that staff could draw on but this list appears to have disappeared. Clearly external revie wers who are
knowledgeable on international development and Bank practice would enhance the quality of AAA.
The panel recommends that staff be more pro active in using external reviewers.

2.18 Another aspect of overly-inward looking Bank AAA work is neglect of non-Bank
analytic studies. The Panel was struck by the fact that in not one of the seven interviews was
there a single mention of ESW undertaken by others that was used by the Bank. This includes
donors, multilaterals, academics, and think tanks. It was noted that while ESW reports review
the Bank’s literature well, there seldom is mention of literature external to the Bank. It is as if “if
it is not done by us, it doesn’t exist.” Clearly, authors of ARD ESW are missing valuable sources
of information, lessons of experience, empirical findings and theoretical developments when they
restrict their review to in-house work.

1
     In Kazakhstan, for example, a very good piece of work, requested and partly financed by the government, was
done on fisheries. This was an expensive piece of ESW yet will only benefit a few thousand fishermen. Over the
period reviewed, no similar study has been conducted on agriculture, a sector with many multiples of the number of
people.
Cross-Country ARD Sector AAA Assessment                                                          12



2.19 Country Departments and ARD. One purpose for producing “Reaching the Rural Poor”
was to inform a wider audience in the Bank of the importance of the rural sector to the objectives
of the Bank. In particular, it was hoped that the evidence-base showing poverty to be
concentrated in rural areas might influence country departments to accord a higher priority to
work in rural space. Despite a vigorous dissemination campaign by ARD when the report was
published, there is no evidence from the sample that country departments have yet been
influenced by the new rural strategy. Indeed, in one instance the Panel learned that even the rural
sector manager had not yet read the report!

2.20 Quality Enhancement Reviews. None of the reports reviewed had taken advantage of a
QER. ARD could play a role in promoting QERs. Since the feedback from a QER is
substantially different from that received from a peer review, QERs are very instructive and are
likely to be especially useful in the case of an inexperienced task manager. They could be
regarded as a “mentoring” tool. Undertaking a QER at concept or at least at the “white cover”
stage would enhance the quality of AAA and provide managers greater confidence on the quality
of the work.

2.21 Accessibility of Documents. The Bank appears to have a very porous institutional
memory and an entirely inadequate filing system (electronic or otherwise). Many reports were
missing or could only be located with difficulty. In one instance, a report was located because a
panelist knew the author who had since retired, and obtained a copy from him; in another, a
consultant had to be tracked down as the TTL did not have a copy; in a third case, only because a
panelist knew the report existed was the report found – it was not recorded in the system. If staff
are unaware of the existence of studies, how much less so are country counterparts? There is a
massive institutional failure and a massive loss to the institution in reports lost, shelved, unused
or under-used, or otherwise inaccessible to both staff and clients. Better ways to systematically
save and access AAA products should be found.

2.22 Further, the lead-up documentation to AAA is frequently missing. Project concept notes
are either non-existent or missing from files, as are the minutes of peer review meetings. This
makes assessment by QAG and sector managers particularly difficult. Better managerial
oversight of AAA work is needed. The Bank should consider the parallel with lending where a
sector manager signs off on the PAD before it goes out. Sector managers could also sign off on
AAA.
Cross-Country ARD Sector AAA Assessment                                                                 13




                                  BOX 2. 4: A SUPPLY-DRIVEN SUCCESS

     As demonstrated by recent events in Mexico, the transition to a new government offers a timely
     opportunity for the Bank to synthesis the p   ortfolio of AAA work, whether or not a government
     requests it. Before the 2000 elections, the Mexico Country Department arranged for a stocktaking of
     issues drawing on existing knowledge of staff working on different sectors in Mexico. This
     consisted of a wide ranging series of policy notes, prepared outside the formal AAA program,
     largely as an unfunded mandate. As part of this process, comprehensive policy notes were prepared
     on major agricultural policy and institutional issues. These notes were based on the extensive AAA
     program that had been conducted in Mexico on agricultural and rural issues.

     When the election led to a change in Presidency for the first time in 70 years in Mexico, the Bank’s
     policy notes served as a briefing for the new Government. Later, the new Government agreed to
     have the policy notes published as a book, "Mexico-A Comprehensive Development Agenda for the
     New Era" (2001). The book is widely acknowledged with favorable citations, and represents a
     successful AAA contribution. It enabled the Bank to gain a "seat at the table" with the new
     Presidency. It also led to a major change in openness in Mexico's Presidency towards the Bank's
     analytical work. The book became an important element of the Bank's analytical contribution in
     Mexico although it is not recorded in the list of tasks undertaken by the Region.




D.       Priority Areas for Improvement and Next Steps

2.23 The following recommendations are addressed to specific Bank audiences that have a role
to play in helping strengthen the impact of AAA work on reducing rural poverty:

For the ARD Sector Board
     •   Continue to publicize and disseminate best practice in AAA; and
     •   Review measures that would help ensure the availability of experienced sector
         economists that can devote most if not all their time to AAA activities.

For ARD Regional Sector Units

     •   Sector managers need to be more proactive in ensur ing that sector staff follow Bank
         processes designed to enhance AAA quality, including proactive use of peer review
         systems;

     •   Sector managers should earmark funds for dissemination of AAA findings and
         recommendations;

     •   Sector staff should continue to champion the importance of AAA by emphasizing the
         preponderance of poverty in rural locations and the value of ARD AAA in meeting the
         Bank’s poverty alleviation objectives;

     •   Greater use should be made of QERs, especially for less experienced task managers. This
         should be combined with more extensive use of external reviewers;
Cross-Country ARD Sector AAA Assessment                                                         14


   •   Make more extensive use of studies conducted outside the Bank and consider updating
       old studies before launching new ones covering the same topic; and

   •   More attention is needed toward accurately recording AAA tasks and ensuring that
       reports are properly archived and readily available to the staff and the client.

For Other Regional Sector Units

   •   More attention is needed to ensuring that rural poverty aspects are properly integrated
       into the analytic work of other sectors, to overcome the tendency for staff to remain fixed
       within narrow sectoral “silos.”

For Country Departments

   •   Greater attention needs to be paid to the importance of the rural sector in country
       programs if country units are to fulfill the Bank mandate of poverty reduction. This will
       require a more consistent allocation of funds to ARD AAA to maintain the increase seen
       in FY04.

For QAG

   •   This pilot review was not able to fully address some issues due to the limited interaction
       with staff and managers. This aspect should be reviewed in any future assessment of this
       type; and

   •   The impact of sector AAA on country lending programs and other stakeholders is
       probably best examined in conjunction with field visits which provide a much deeper
       basis for informed judgments.
Cross-Country ARD Sector AAA Assessment                                                                    15


                                                                                                   Annex 1
                       AGRICULTURE AND RURAL DEVELOPMENT

                PILOT R EVIEW O F THE Q UALITY AND EFFECTIVENESS OF ARD AAA

                                              Approach Paper

Background

1.       The Agriculture and Rural Development Sector Board has requested the Quality
Assurance Group to conduct a pilot review of the impact of its SSP and ARD AAA
activities 2 on:

           a)    Country strategy and the formulation of sector strategies by the country and the
                 Bank; and

           b)    The integration of ARD AAA activities into Bank country operations, including
                 policy dialogue and lending.

2.       Table 1 below shows the volume of ARD AAA activities managed by the regions.
What is less clear, however, is the impact ARD AAA has had on the Bank’s approach to the
agricultural sector overall and to the formulation of strategies at country level. Nor is there a
clear indication of how the Sector Strategy Paper influenced AAA and lending activities.
There are two further uncertainties associated with the ARD AAA activities. The first relates
to the correlation between the quality and quantity of these activities and the coverage of the
rural issues in a country’s CAS and PRSP (according to assessments conducted by QAG
during FY98-02, the 34 ESW activities in the rural sector received a 71% overall satisfactory
rating compared with a Bank-wide overall rating of 88%). The second is the correlation
between the quality of ARD AAA and the quality of project design and overall performance
at exit.

Table 1: Region specific ARD1 AAA Activities Managed by Regions: FY00-03
                                      No of Activities                                Total Cost
                    NLTA            ESW 2                Other ESW 3         Total            ($'000)
    FY00                5            22                      6               33               5,282
    FY01                2            16                     15               33               7,634
    FY02               12             9                     12               33               5,578
    FY03               16            13                     18               47               7,268
Notes: Source Business Warehouse; 1. Activities designated as the responsibility of the Agriculture and
Rural Sector Board; 2. refers to the reports output type; 3. ESW output types: Consultations;
Consultations/Country dialogue; Conference/Workshop; Other; Policy Note.




2
  ARD AAA stands for all AAA recorded as RDV, whether they are managed by ARD regional units or by other
sector/country units.
Cross-Country ARD Sector AAA Assessment                                                     16


                                          APPROACH
3.      Included in the review will be all ARD AAA activities conducted during the five- year
period FY01-04 and their likely impact on the rural strategy and lending. Activities will be
reviewed in the context of a selected number of countries. Main objectives of the review will
be to evaluate the following aspects:

   •   Impact of the Strategy Papers (Vision to Action and Reaching the Rural Poor) on
       ARD AAA and lending activities;

   •   Overall quality and adequacy of ARD AAA activities;

   •   ARD AAA contribution to and impact on the CAS and PRSP, and understanding of
       the factors leading to the impact;

   •   Impact of ARD AAA activities on the country’s rural strategy and on the activities of
       other donors; and

   •   ARD AAA contribution to the formation of the Bank’s lending strategy in the sector
       and impact on the quality of project design and lending volume and understanding of
       the factors leading to the impact (AAA activities completed during FY 00-01 will be
       examined for actual impact on projects approved by the Board in FY03-04 and the
       rest for their potential to influence future lending).

4.      The review will eva luate the impact of AAA activities conducted by both ARD
regional units and those managed by other sector/country units (e.g., CEM, PER, Poverty
Assessments), including regional and global ESW that may have relevance to the country’s
rural sector strategy and lending. Particular attention will be paid to perceived shortfalls in
the level of ARD AAA activities in the country and to the identification of areas where
additional studies could enhance the quality of Bank activities and support the activities of
the member country. In order to reduce the burden of this review on regional staff,
preference was given in the selection of countries to countries where QAG has recently
completed a Country AAA assessment. Selected countries, which are subject to review and
approval by the concerned regions, include:

           Mexico and Ukraine      Country AAA completed
           Kazakhstan              Country AAA completed
           Bangladesh and Tanzania Country AAA ongoing
           Ghana                 No AAA assessment and relevant ARD AAA available
           Morocco                 No AAA assessment available

5.     The review will be conducted by a core team comprising a senior retired Bank staff
member and an outside reviewer. Specialized reviewers, with sector/project expertise, will
be engaged as needed to examine in more detail individual activities. Support to the core
panel will be provided by the ARD anchor. An initial desk review of all activities previously
evaluated by QAG will be carried out with a view to identify those areas where additional
work is called for. Where the Country AAA assessment is ongoing, the core panel will work
closely with the members of those panels. Except where a Country AAA assessment is
ongoing, the present review will only entail one meeting between the panel and the staff and
managers responsible for the ARD work in the country. At this time no separate field visits
Cross-Country ARD Sector AAA Assessment                                                  17


are expected. A list of primary studies to be examined during the review in the seven
selected countries is attached.

6.      Questionnaires to be used in conducting the review will be modeled on the Country
AAA assessment. They will evaluate quality and impact at the task, country and Bankwide
levels. However, ratings will be strictly for QAG’s piloting purposes and will not be shared
with the Regions or the Sector Board. QAG is currently preparing these questionnaires, and
drafts will be sent to the Sector Board for review and comment.

7.      At the end of the assignment, the core panel will produce individual country reports
summarizing the findings and recommendations for the country. It will also produce an
overall report outlining the major findings of the assessment. In a separate report to QAG,
the panel will evaluate the review process, provide a judgment on whether or not this
approach could be replicated to other sectors, and recommend additional measures to
strengthen the process.

COST ESTIMATES

8.      Inputs to the review will be provided by the core panel, specialized reviewers, ARD
support, and QAG administrative/management inp uts. Estimated costs associated with each
of these aspects are as follows:

         a)   Core panel: 2 persons x 25 days = 50 days

         b)   Sector panelists: 15 in-depth reviews (1.0 day/activity), and 30 activities
              conducted by non-ARD units (0.5 day/activity) = 30 days. (In the case of the
              two countries where the review is ongoing, the costs will be covered by the
              AAA review).

         c)   ARD input = 30 staff days

         d)   QAG management and administration = 40 staff days

The above works out to a total of 150 days or roughly $105,000.

TIMING

9.     It is suggested that the review should commence in mid-May so that the core panel
could interact with the two on-going AAA reviews. The individual country assessments will
be aggregated into an overall report by mid-July.

                                          PANELISTS

                             Amnon Golan           (Moderator)
                             Malcolm Bale         (Coordinator)
                   John Haywood                       Rory O’Sullivan
                   David Steeds                       Derek Byerlee
                   Fred Swartzendruber                Bruce Gardner
                                                                                   Annex 2
       Cross-Country ARD Sector AAA Assessment                                                                           18


                                           LIST OF AAA REVIEWED
      FY            Proj. ID                        Title                             TTL              Output   Cost
Bangladesh
FY01               P071573     Banglades h Rural Decentralization         CONSTANTINO           Consultations          58

FY01               P072118     Bangladesh - Agriculture Policy Dialogue   CHOWDHURY             Consultations          44

FY01               P067728     Natural Resources Management Note          CHOWDHURY             Policy Note            36

FY02               P075239     Rural Development Sector Dialogue          EPWORTH               NLTA                   54

FY03               P074941     Ag. growth. and rural pov. dyn:A reg. p    DIOP                  Policy Note            74

FY04               P078291     Promoting Rural Non-Farm Growth                                  ESW

FY04               P088653     Scaling up Microfinance                                          ESW

GHANA
FY01               P068979     Poverty Update and Profile                 JONES               CoConference

FY03               P077543     Country Gender Assessment                  MUTEMBA               Policy Note

                   P078688     Poverty Note                               DEMERY                Report

FY04               P085043     Ghana Energy PSIA                          KEENER                Policy Note

                   P077582     CEM: Poverty, Growth, Budgetary            CARNEIRO              Report

                   P083876     Public Expenditure Review                  ANDRADE
Mexico
FY01               P060321     RUR.FIN.IN MARG.AREA                       BRIZZI                Report                 200

FY01               P060324     FOOD MARKETING                             BRIZZI                Report                 346

FY01               P071457     MX-Land Policy Review                      LAVADENZ PACCIERI     Policy Note            276

FY01               P072427     MX-Crop Insurance                          VARANGIS              Report                 165

FY02               P074770     MX Sugar                                   LARSON                Policy Note            119

FY02               P074768     Land Administration                        LAVADENZ PACCIERI     NLTA                   87

FY03               P074698     MX Coffee                                  VARANGIS              Report                 122

FY04               P085943     Rural Poverty Analysis                                           ESW
Morocco
FY03               P073174     MA-Agriculture Sector Note                 WARD                  Policy Note            175

FY03               P077456     FOREST AND LAND REVIEW                     PSWARAYI-RIDDIHOUGH Policy Note              87

FY03               P081087     MA: Agricultural Reform and Irrigation     WARD                  Report                 163

FY04                           MA: Agricultural Reform 2                                        Report

FY04                           Agriculture Reform                                               NLTA
Tanzania
FY02               P072131     RURAL DEVELOPMENT                          TOWNSEND              Report                 359

FY04               P085776     SEA for Sustainable Coastal Livelihoods    BOYLE                 NLTA                   20

FY04               P080072     Rural Development                                                Report

FY04               P087208     Agribusiness Supply Chain Analysis
Ukraine
FY01               P072194     LOCAL CAPACITY BLDG TA (PHASE 2)           SHUKER                NLTA                   102

FY02               P074847     RUR FIN POL DIAL TA                        CHAVES                NLTA                   72

FY04               P077461     AGRIC POLICY NOTE                          SHUKER                Policy Note            208

Kazakhstan
FY03               P077376     Forestry and Rangeland Management

FY03               P078300     Livestock Sector Study                     Jungbluth

FY04               P083363     Fisheries Sector Study                     Sutton

                                                                                                                Annex 3
                                                    COUNTRY FINDINGS
Cross-Country ARD Sector AAA Assessment                                                    19


                     BANGLADESH S ECTOR AAA COUNTRY ASSESSMENT

1.      Summary Comments. Serious deficiencies were found in the overall quality of
Agriculture and Rural Development AAA in Bangladesh during the period 2001-2004.
However, this finding has to be considered in the context of the special circumstances of
country-Bank relations prevailing at the time, the state of the country’s agriculture, and the
state of sector knowledge. From ARD’s perspective, the relationship between the Bank and
Bangladesh was not considered conducive during FY01-03 to engaging more substantially in
either new lending or AAA activities. Three out of the seven studies carried out during this
period and identified by the Region as being relevant to the rural sector were found not to be
AAA activities (they were either consultation with government or codes that covered staff
time in maintaining a dialogue with their counterpart during this difficult period). The
Region agrees that they have to address this coding problem.

2.       During the 1980s and ‘90s, the Bank played a key role in Bangladesh agriculture in
bringing about liberalization of markets and prices and privatization of input supply and
irrigation. Employing high quality sector work and a large team of specialists in Dhaka, the
Bank helped the Government of Bangladesh (GOB) attack constraints to agricultural growth
and backed the effort with a vibrant pipeline of projects. Throughout the 1990s, economic
growth accelerated and by 2000 Bangladesh had quite remarkably, achieved food-grain self-
sufficiency. However, deteriorating governance soured government-Bank relations so that
from 2000 to 2003 lending was low case, and funding for AAA was restricted mostly to
PREM-managed tasks such as the Public Expenditure Review (PER) and its associated
Poverty Assessment (PA).

3.      A new CAS, produced in 2001, conformed well to the Rural Sector Strategy, and set
“an integrated approach to rural development, including supporting growth in agriculture and
non- farm activities” as “one of the three most powerful lines of attack against poverty.”
Notwithstanding, this theoretical thrust, however, only minimal resources were provided to
support agriculture. There was good support for rural infrastructure, and rural health and
education, which stimulated the private sector but did nothing to sustain earlier gains in
agriculture. Importantly, there was no integrated approach to rural development and poverty
reduction that included agriculture. Consequently, the portfolio dwindled; sector work
fragmented, and demoralized agricultural staff left the Region. Given the hiatus in 2000-03,
action is even more urgent now. Another noteworthy factor is the Bank’s previous
instrumental role in helping develop a good donor coordina tion program in Bangladesh. In
the future, this facility could be used to interact with government, perhaps through a
dedicated workshop, to define and prioritize issues constraining rural advancement and
proposing a coherent program of AAA tasks.

4.      A new team of Bank country and sector management took over responsibility for the
Bangladesh program in 2003 and started to inject life back into the agricultural program. It
should be recognized, therefore that this assessment reviews AAA, at a particularly difficult
time in Bank-GOB relations, when the sector’s impressive performance had induced
complacency in the country and in the Bank, and when Bank management was in transition.
These factors combined to produce a period of AAA weakness that must be considered an
aberration in the history of Bank- Bangladesh ARD relations.

5.     Strategic Relevance. Studies with relevance to the ARD sector (PER, Poverty Assessment
and Climate Change) were focused on CAS objectives but the rest of the program was
Cross-Country ARD Sector AAA Assessment                                                   20


supply-driven and descriptive of the current situation rather than being forward looking and
aimed at helping government overcome emerging problems. Importantly, the totality of the
AAA program failed to address integrated rural development and rural growth in a coherent
manner. The CAS clearly articulated a prime objective of reducing rural poverty but this was
not matched by coherent, prioritized AAA work geared to producing practical answers to
rural problems. This lack of relevance to immediate issues was cited as a key reason for poor
relations between Bank’s sector staff and the line ministry.

6.      The Poverty Assessment fed recommendations directly and purposefully into the I-
PRSP and the Climate Change task produced valuable indicators for a long-term vision of
development. However, AAA failed to address priority needs and the rural sector still lacks
answers to vital questions concerning crop diversification, agro-processing and marketing,
stimulating the non- farm sector, how best to introduce new technology, improving land and
labor productivity, efficient water management, and how best to provide credit for middle
level farmers and entrepreneurs. It should be noted, however, that new studies launched in
FY04 will address many of these issues but as they were still in progress when the current
review was completed, they could not be assessed by the Panel.

7.      Internal Quality. Agricultural data in Bangladesh are accessible, sound and up to
date. They provide a wealth of information for high quality analysis, as evidenced by the
Agriculture Growth and Rural Poverty Dynamics study. The quality of analysis in AAA was
mostly satisfactory. A number of studies used the Bangladesh Bureau of Statistics (BBS)
and international experts, for example IFPRI economists for the Agriculture Growth Study,
to undertake analyses. Small studies, although technically sound, tended to be descriptive of
the current situation and to bring no new analysis to the agricultural debate.

8.      Recommendations followed logically from analyses and conclusions, but were mostly
broad and general. A complaint from the MOA was that recommendations lacked specificity
and practicality regarding what should be done and when. Sequencing of recommendations
was rare. Most AAA reports avoid being prescriptive, deliberately leaving decisions to
government. Although this is correct, AAA should nevertheless aim to produce options for
debate, otherwise recommendations become too vague; which leaves the potential role for
project intervention equally vague.

9.     A key factor influencing quality was management commitment to, and support of, the
task. Where the task from its inception was considered important by the Bank and by
stakeholders, e.g., the PER, adequate funding for employing a skilled team and for getting
good feedback resulted in high quality product.

10.      Likely Impact. AAA tasks varied greatly in their likely impact on clients.
Importantly, it was often unclear just who was the client for a particular AAA task. Some
tasks were directed at government decision makers and civil society, whereas others
appeared undecided between the government and Bank management. This ambiguity of
client focus resulted in ineffective presentations and weak impact. By contrast, AAA tasks
that involved PREM interaction with the Planning Commission and Ministry of Finance had
good impact at senior government levels. Where, however, AAA was supply-driven and had
little support from Bank management or champions within government, it tended to fizzle out
and have zero impact. Capacity building efforts were effective in some AAA activities, for
example where BBS was involved, but sometimes appear to have taken place in virtual
isolation from government staff.
Cross-Country ARD Sector AAA Assessment                                                      21


11.    Government ownership, dialogue and dissemination were particularly weak elements
in much of the AAA program. It is vital to the health of the AAA program that in future,
government should be true partners in prioritizing and designing AAA. Specific resources
should be made available for effective interaction with stakeholder. The impact of AAA on
the agricultural lending program was minimal. There was a complete disconnect between
completed studies and the ongoing ARD portfolio during the period under review. AAA
tasks did not tackle emerging problems so both the Ministry of Agriculture and Bank
management lost confidence and interest in the program.

12.     Donors look to the Bank for leadership in agricultural strategy and use quality AAA
findings as indicators in their own programs. Impact of the PER and Poverty Assessment
studies on donors was considerable. However, small pieces of supply-driven AAA had little
or no visible influence on donors’ programs.

13.     Comprehensiveness. In the Panel’s view, the AAA program was fragmented,
supply-driven and ad hoc during this difficult period when the Bank virtually withdrew from
the sector. The sum total of studies, which omitted some key issues (see paragraph 6), in no
way covered the needs of a forward- looking ARD program. The government and the Bank’s
agricultural staff, particularly those few remaining in Dhaka, recognized the need to sustain
sector gains but lack of adequate funding and lack of a critical mass of skilled staff prevented
meaningful AAA being carried out.

14.     Planning and Bank Processes. During the period 2000-2003 planning for the
overall AAA program was weak. There was no coherence in the program and it was not
geared to the strategic needs of the country or to the development of a meaningful lending
program. Planning of the individual larger pieces of AAA – the PER, Poverty Assessment
and Climate Study – was good. At the outset, plans for these tasks defined the client and
provided resources for stakeholder interaction and results dissemination. Other pieces,
however, were supply-driven and suffered accordingly.

15.     The Poverty Assessment demonstrated what could be achieved even during a period
of poor Bank-government relations. It conformed precisely to the CAS, was geared to the
                          -
immediate needs of the I PRSP, was well planned and resourced, and was given good
support by Country and Sector Management. Consequently, it was taken seriously by
government and stakeholders, which made feedback and ownership meaningful. The impact
of this AAA was considerable.

16.     The Climate Change Study, which could have easily turned out to be somewhat
academic, was also well planned and resourced, given support and taken seriously by
management, government and stakeholders. Again, its impact on changing attitudes in long
term planning of government and donors could be considerable. The contrast between how
these AAA tasks were managed, compared with the rest of the program, provides clear
lessons about ‘seriousness’ and impact of AAA.

17.     The vicious downward spiral that set in during 2000-2003, whereby lack of support
for agriculture led to lack of staff capacity, which led to lack of sector strategy and further
lack of support for the sector, must be reversed. The impressive gains in agriculture made
during the ‘80s and ‘90s can only be sustained with donor support for further modernizing
the sector. But building an effective pipeline will require coherent and coordinated AAA,
undertaken by skilled professionals. This will, of course, require adequate funding, but more
Cross-Country ARD Sector AAA Assessment                                                 22


importantly it will require experienced staff backed by consistent management attention and
support.
Cross-Country ARD Sector AAA Assessment                                                      23


                        GHANA S ECTOR AAA COUNTRY ASSESSMENT

18.     Summary Comments. In the absence of ARD-specific AAA work, this evaluation
covers relevant aspects of other AAA tasks suggested by the Region. Some of the poverty
work done in Ghana was found to be of high quality, and there is now a significant body of
research dating to the early 1990s, which demonstrates that poverty reduction has been rather
successful in urban areas, notably in Accra, while deprivation continues or is increasing in
rural areas with minimal infrastructure, poor access to markets and services, and few
economic options. The recent CEM does an excellent job of cross-sectoral synthesis of this
evidence and convincingly presents the case for better sector work in Ghana, especially with
respect to agriculture. Another factor may also be at work in Ghana, relating to the very high
level of effort in recent years dedicated to developing the CDF framework, preparing the first
PRSP and PRSC, and complex multi-donor negotiations around these initiatives. This work
may have crowded-out some of the sector work and management attention which otherwise
might have been channeled toward strengthening the existing rural/agricultural portfolio, and
strengthening the integration, coherence and dissemination of AAA activities, especially in
cross-cutting areas such as poverty and social impact analysis, which are highly relevant to
the ARD agenda in Ghana but are managed under other sectors.

19.     Strategic vision and leadership are especially important to provide renewed focus and
dynamism when a sector program runs into serious difficulties, as has been true of
agriculture in Ghana during the 1990s. If the institutional judgment is that this is not an area
of comparative advantage for IDA (a legitimate option), then in the interest of transparency it
would be better to acknowledge this openly, and work with the Government and other donors
to ensure that appropriate resources and commitment are forthcoming.

20.     Strategic Relevance. The Panel noted the absence of ARD managed sector work
during the review period, and the fact that none is proposed in the latest CAS (dated February
2004). In fact, the rural development staff appears to have taken a back seat on critical
issues, which are at times being covered (thinly) by other units, who have considerably less
technical expertise (e.g., aspects concerning non-traditional agricultural exports). At the
same time, according to the Bank's poverty studies, poverty in Ghana is essentially rural
(more than 80% of the poor live in rural areas), is increasingly concentrated among farming
households in northern savanna areas and coastal non-forest areas, and is especially severe
among female- headed subsistence farming households. The Bank and the government both
agree tha t the agricultural sector is critical to achieving sustainable, broad-based growth,
especially in diversifying exports and developing new products and markets for which Ghana
has comparative advantage. The absence of an ARD AAA program thus represents a serious
strategic omission in Ghana, as it is hard to see how poverty reduction objectives can be
achieved without a major effort to address these problems. As a group, the AAA tasks
reviewed here do not yet add up to a comprehensive picture of what the Bank would need to
do on the analytic side to increase the chances of achieving the MDGs and to improve the
present lending program. The recent Country Economic Memorandum (February 2004)
makes a useful first step in this direction, by calling for more in-depth sector work "to
identify the bottlenecks to agricultural growth,” and notes that "policy dialogue must advance
beyond cocoa... in essence moving from subsistence agriculture to market-based high value
production.”

21.     Internal Quality. Tasks examined by the Panel demonstrate a strong continuity of
analytic effort on poverty issues since the mid-1990s, and, taken together, would appear to
Cross-Country ARD Sector AAA Assessment                                                        24


support a strong case for rethinking the Bank's approach to rural development in Ghana.
Some of the individual reports w    ould be strengthened by sharpening their conclusions and
recommendations, which are sometimes sidelined within a text box or table without a clear
summing- up of key points and next steps. The electricity tariff social impact assessment has
excellent internal quality, but the present phase I report is only partially relevant to the rural
sector, and the proposed phase II has not been committed, though it would be directly
focused on poor rural areas of northern Ghana. But on the whole, these non-ARD pieces
maintain important Bank engagement with analytic issues that will need much more attention
in the future, and represent a good starting point for reinvestment in the sector.

22.     Likely Impact. Based on the present absence of an ARD AAA program, which could
make use of the non-ARD tasks reviewed here, the program is unlikely to have a significant
impact. The Bank's rural projects in Ghana have performed poorly in recent years, and many
of these deficiencies are exacerbated by the lack of relevant sector work to inform project
design and improve the choice of targets, instruments, and partners. For example, peer
reviewers criticized a CDD operation now in preparation as taking a very outdated approach,
and missing opportunities to build upon the Bank's own best practices. The task team argues
that land tenure and legal issues unique to Ghana seriously constrained design choices, yet
this is precisely the kind of situation which would call for focused and timely AAA work
prior to project preparation. Similar design issues are found in the new land administration
project, which is designed to tackle a complex problem that would greatly benefit from up-
front analytic work. Indeed, the 2004 CEM, which was reviewed for this exercise, comments
that land administratio n efforts are likely to fail in the absence of a range of upstream
improvements in agricultural input supply as well as downstream improvements in marketing
and infrastructure.

23.     Comprehensiveness. Shortcomings on this aspect are similar to those identified
above. The most comprehensive sector work in Ghana by far is poverty work done in the
overall context of the GPRS, but the non-ARD tasks reviewed here in no way substitute for a
properly- focused effort to understand why previous rural and agricultural operations have
performed poorly, and to identify alternative approaches which might offer better chances of
achieving the MDGs.

24.     Planning and Bank Processes. The AAA tasks reviewed show little evidence of
attention from the ARD sector managers and limited coordination of cross-sectoral issues. At
least one study appears to have been closed at a draft stage without soliciting comments from
the client or planning any dissemination activities. During the review period, the country
team appears to have been extremely busy in support of the PRSP/PRSC and CDF initiatives,
but these have not been guided toward developing a coherent ARD AAA program, which in
turn could serve as a basis for improving the troubled rural portfolio in Ghana. Records are
often difficult to locate, and in some cases, there remains little institutional memory about
how a particular task evolved and why.
Cross-Country ARD Sector AAA Assessment                                                          25


                     KAZAKHSTAN S ECTOR AAA COUNTRY ASSESSMENT

25.     Summary Comments. Since 2001, and the abrupt change in Bank-Kazakh relations
following Kazakhstan's oil boom, sector work in Kazakhstan has been carried out jointly
with the government under the Joint Economic Research Program (JERP). This involves the
selection of an agreed sector work program and the joint financing of tasks. Several aspects
of this arrangement are noteworthy. First, there are few member countries where
governments agree to co-finance (50% in this case) sector work, and this arrangement, which
has been brokered with Kazakhstan, is a most interesting development with possible lessons
for other MIC’s. Second, because studies are 100% demand driven, the likelihood that senior
officials will actually read and use them is far greater than in most other countries. But third,
on the downside, because of the demand driven nature of the arrangement, the Bank seems to
have limited influence over the selection of studies entering the work program. A pertinent
question in this case is the extent to which sector work in this kind of arrangement should
also be driven by Bank policy preference (such as the poverty driven imperative of the ARD
strategy). The three tasks reviewed here, covering Fisheries, Livestock, and
Forestry/Rangelands were part of this jointly implemented and agreed program. These tasks
involved the participation of local cons ultants and experts working with Bank staff and
consultants. The reports were clearly of good technical quality, with the Bank evidently
providing effective leadership in collecting and harmonizing disparate contributions and
weaving them into a coherent framework while addressing the key policy issues using cross
country experience. But the linkage of these studies to the CAS and overall Bank policy
needs to be examined.

26.     Although the program has the clear advantage of being demand driven, the Panel was
concerned that its overall thrust may not cover the highest sectoral priorities for stimulating
economic development in the rural areas and combating poverty and queried the flexibility
that the Bank had for the allocation of WB funds in its dealings with the JERP.

27.      Strategic Relevance. While the three tasks all contribute to specific areas of concern in
the rural space they differ in their potential impact, and the Panel considers that other tasks may
have been more relevant in addressing sector -wide and poverty-related issues. The livestock
report covers a highly relevant and important subsector, which contributes as much as 40% of
agriculture GDP and contains a considerable proportion of the rural poor. On the other hand, the
fisheries report, while addressing important technical challenges in the sector (preserving the
Caspian sturgeon, an effort that calls for close collaboration with the other countries bordering
the sea, and the co-management of small lakes), deals with a topic that may not be of the highest
priority in Kazakhstan (officially, only 13,200 people are engaged in commercial fishing). While
the report recognizes that artisanal fishermen are also important, and tries to investigate them, it
does not cover this group in any depth or as a major focus, as the ARD strategy would
recommend, and fails to address the concerns of persons in the poverty target group. But at the
same time the report provides and excellent technical assessment of the sector, which was surely
appreciated by the JERP administrators and those working in the sector. The Forestry/
Rangelands study was in the end split into two. Rangelands, of very high relevance and
importance on economic, poverty and environmental grounds, and forestry that in Kazakhstan is
important more on environmental grounds (the Panel noted, however, that partly as a result of the
latter study, a Forestry Project is now at an advanced state of preparation). Activities dealing
directly with agriculture and agricultural products were missing from this AAA program:
agricultural production, agricultural research, agricultural extension, marketing, and seed
production. While some of these will have been handled in earlier studies, and others were
Cross-Country ARD Sector AAA Assessment                                                          26


handled in the context of project preparation, there are still key unresolved issues in Kazakhstan
in these subsectors of wide importance and the Panel wondered if this program could not have
been better targeted. Perhaps this will raise challenges for the future work program to be agreed
under the JERP Program. Elsewhere, it was noted that regional studies covering microfinance
and irrigation had contributed to a better understanding of key ARD related issues. Both these
studies were useful additions to Kazakhstan's AAA program but are not a substitute to tasks
concentrating on country specific issues.

28.      Internal Quality. Overall, the technical quality of the reports is good and they seem to
address the key issues in the subsectors in which they are working. Technical reports written by
local consultants have been used judiciously to provide a solid technical underpinning to the
policy recommendations made. External comparisons have been well selected to put the sectors
in the relevant perspective.

29.      Likely Impact. The Panel considered that information contained in these reports could
have a significant impact on the chosen subsectors provided the Kazakh authorities follow up the
recommendations made. The problem is that the extent to which this will actually happen is still
unknown. The outcome will depend to some extent on the ability of the Bank and local experts
to "sell" the findings of the reports and it is still too early to know the outcome in this regard.
One potential important factor that will help a successful outcome is the emphasis in all the
reports on improvements that can be made in the Public Investment Program for the subsector
being considered. This linkage with the key national instrument for making things happen could
facilitate a positive outcome. Overall, it would it would be advantageous for the national
counterparts in the JERP Program to be fully involved in dissemination as well as design of this
AAA program.

30.      Comprehensiveness. The Panel had concerns about the comprehensiveness of the AAA
program in the ARD sector, even allowing for separate regional studies that have included
aspects of the rural sector. For Kazakhstan, this then raised the question of the influence that the
Bank had in setting the AAA agenda in the context of the JERP. As mentioned above, the
current program does not cover key subsectors for economic growth and poverty reduction in the
rural areas, which puts into question the adequacy of the Bank's AAA program in the country.

31.      Planning and Bank Processes. A number of the issues raised above, including the lack
of attention to certain key subsectors, highlight the difficulty of planning the AAA program in the
JERP environment. The government may be less interested in working on key issues (e.g., with
much of the land being tied up in large holdings, what can be done to improve the livelihood of
small farmers), which face difficult policy choices. Since AAA activities in Kazakhstan receive
about one third of all BB resources, it is important for the final program to mirror Bank priorities
as much as it responds to the needs of the Government. The Panel is aware that this would be
difficult in the Kazakhstan context but every effort should be made to make this happen.

32.     While pointing out potential risks (see above), the panel commended the close working
relationship with the client in designing and implementing the AAA program through the JERP.
It is expected that this joint working arrangement would bring benefits in subsequent
dissemination efforts, the transfer of knowledge, and the technical quality of the final product.
All the reports appeared strong on technical aspects with excellent policy roadmaps for the
Kazakhstan’s decision makers. There was widespread evidence in supporting documentation of
good coordination with other development partners, which should strengthen the CDF approach
of other partners.
Cross-Country ARD Sector AAA Assessment                                                     27


33.     The cost effectiveness of AAA needs to be assessed when designing the program. For
the ARD sector, this assessment will often be in terms of resources allocated to tasks compared
with the number of potential beneficiaries, particularly the poor. This will help keep the AAA
program in line with the Bank's ARD strategy. Dissemination remains a vital task if AAA is to
have the full impact hoped for. This may well require a multi-year dissemination pr ogram,
coming back yearly with an assessment of progress and policy change adoption. This could
greatly enhance the effectiveness and payoff from AAA.
Cross-Country ARD Sector AAA Assessment                                                       28


                        M EXICO S ECTOR AAA COUNTRY ASSESSMENT

34.     Summary Comments. Mexico is an advanced middle- income country with
considerable capacity for analytical work. However, the Bank is valued for its AAA,
especially in work that is on the frontier in terms of analytical skills and that provides access
to international good practice. In addition, lending to Mexico is non-additional in ministry
budgets, so that AAA work is demanded and valued largely as a stand-alone product, rather
than as a means for enhancing the quality of lending.

35.     ARD AAA work in Mexico emerged as part of the post- liberalization agenda, after
implementation of NAFTA. It has been one of the largest ARD AAA programs, and with its
emphasis on reforms of product and factor markets, was clearly strategic in terms of CAS
and Mexican goals. Overall, the program of 7 AAA tasks delivered during FY01-04 has had
considerable impacts on policy and capacity building, and to some extent on the lending
program. However, emphasis on AAA has fallen off sharply beginning in FY03, in part due
to a decline in overall AAA, and resulted in a significant reduction in the ARD share of AAA
from 24% in FY00-FY02 to 7% in FY03-FY04. The current program is not well positioned
to engage in policy dialogue on major emerging issues such as competitiveness, more
effective use of the large public expenditures on the sector, and adjustments required for full
implementation of NAFTA by 2008.

36.     Strategic Relevance. The AAA program emphasizing product markets (grains,
perishables), commodity studies (coffee, sugar), and factor markets (rural finance, insurance,
and land) clearly responded to the post- liberalization agenda to develop efficient and
competitive markets, in the wake of state withdrawal from the sector. Agriculture and rural
development was highlighted as a strategic priority in the 1999 CAS, which guided the
selection of tasks to FY02. Likewise, the task on rural poverty initiated in FY03 (but not yet
delivered), was consistent both with the CAS priorities, as well as the newly-approved Bank
corporate strategy, Reaching the Rural Poor. Some of the tasks, notably sugar, crop
insurance, and land policy, also responded to strong demands from the GOM.

37.     Internal Quality. QAG has previously evaluated five of the seven completed tasks,
and generally rated them favorably, with the tasks on land policy rated as highly satisfactory.
Some of the tasks were weak in recommendations, providing too many general
recommendations that did not provide sufficient guidance on priorities, the role of the public
sector, and institutional options for effective implementation.

38.     Likely Impact. The AAA program has had considerable impacts. Three of the
individual tasks have been rated highly satisfactory in terms of policy changes and capacity
building (land, sugar, and crop insurance). In spite of the non-additionality of lending, the
task on rural finance has been used in the design of two operations, and the task on land is
also likely to be followed by a lending operation. Several of the tasks (insurance, land, and
coffee), have also resulted in changes in overall government strategies toward the sector, and
the work on coffee, food markets, and crop insurance has influenced private sector decisions.
Overall, however, impacts may have been even higher with a more effective and longer run
strategy on dissemination. Only one of the reports was formally published, and none are
available in short policy notes (in Spanish) or on the web.

39.     Comprehensiveness. The AAA program was quite comprehensive through FY02.
This is reflected in the compilation of the book for the incoming government at the end of
Cross-Country ARD Sector AAA Assessment                                                       29


2000, which provides an excellent overview of the major challenges for agricultural and rural
development, and concrete recommendations for moving forward. Relative to the overall
AAA program for Mexico, a much higher rate of rural AAA listed in the CASs was
comp leted in this period. Rural staff have also made significant contributions to AAA led by
other sectors, especially the Southern States Strategy.

40.    While the program has shifted directions since FY02 toward other key priorities,
notably rural poverty and water, the panel feels that the Bank has lost momentum in some of
the core issues for the sector, relating to effective use of public expenditures, institutional
capacity to address competitiveness within a value chain framework, and adjustment to full
implementation of NAFTA.

41.     Planning and the Bank Processes. Although the overall AAA program in Mexico
was rated by QAG as lacking coherence and strategic direction, especially on macro- level
issues, this was less of a problem at the sectoral level. A judicious use of demand and supply
driven approaches were applied, that resulted in a reasonably coherent program that at the
same time responded to client requests. More recently, some of that coherence has been lost,
as the Bank appears to be reluctant to push for AAA on issues of clear strategic relevance for
the sector, such as competitiveness, which has appeared in the last three CASs but never
implemented.

42.    Major findings from the current review are as follows:

Strong Aspects

       • Effective deployment of high quality Bank staff, often from the anchor, DEC or
         universities, to carry out complex tasks (such as crop insurance) that are of good
         technical quality;
       • Generally good dialogue with Mexican officials in several ministries, with good
         continuity over time to FY02;
       • Tasks are having a useful impact, with tangible evidence of results on the ground.

Aspects Needing Attention

       • Dialogue needs to be sustained over time on core issues, and with a variety of
         ministries and organizations, even with changes in personnel in both the Bank and
         in the GOM;
       • Increased attention to the share of agriculture and rural development in AAA (since
         FY02); and
       • Allocation of resources up front to dissemination, to allow final publication and
         translation of policy notes that could be made ava ilable to a wider group of
         stakeholders, and to maintain institutional memory.
Cross-Country ARD Sector AAA Assessment                                                 30


Systemic Lessons

       • AAA should be regarded as a stand-alone product that can have significant impacts,
         regardless of the amount and quality of subsequent lending;
       • Preparation of policy notes for incoming governments is an effective tool for
         synthesizing AAA work and providing immediate input into policy dialogue;
       • The formulation of AAA programs requires both demand and supply led approaches
         to setting priorities and developing a coherent program; and
       • The Bank should systematically tap experiences and good practice in institutional
         and policy reform in OECD countries, to inform policy dialogue in advanced
         middle- income countries such as Mexico.
Cross-Country ARD Sector AAA Assessment                                                      31


                      M OROCCO: SECTOR AAA COUNTRY ASSESSMENT

43.     Summary Comments. The Panel assessed three tasks completed during the review
period. These tasks have slightly confusing names and will be referred to here as policy note,
cereals, and irrigation. A fourth task produced a document entitled Integrated Forestry
Development in the Middle East and North Africa. This is a regional task, with a particular
emphasis on Morocco (and Tunisia) only in an annex with good material on land ownership,
land fragmentation, and inactive land markets but very little on the relevance of land issues to
forestry development. Although coded and funded as part of the Morocco AAA program, the
country to which the report’s forest recommendations are least relevant is Morocco, since
Morocco already has a good forestry strategy and a Bank-financed project is under
preparation. Thus, it was an anomaly that this work was included in the Morocco AAA
program.

44.     A major recommendation of the review is that so long as government takes no action
on large-scale irrigation or on the more difficult aspects of cereals policy reform, the Bank
should consider deferring any new analytical work and concentrate instead on trying to get
action taken on the work already done, even if this requires commissioning further work in
these areas. Among the key findings of the review are the following:

       •     The rural dimension has to be brought more explicitly into design of the AAA
             program for ARD. Perhaps this is already being done informally, but it needs to
             be made explicit;

       •     A multi- year programmatic approach to sector work, while still requiring
             confirmation in annual work programs, provides important comfort to sector
             managers and staff that they can embark on a program of work with the client
             knowing that the bank will stay the course;

       •     A two-part presentation of the written product in a client- led process, as done
             for both cereals and irrigation, enables the Bank to meet the challenge of
             preserving its objectivity. Part one of the presentation summarizes the results of
             the work undertaken jointly, while part two offers a (constructive) critique of
             the work;

       •     A well- timed, custom- made, policy note can be very effective in engaging a
             new government in consideration of policy reforms in contentious areas;

       •     In cases where collaborative work with other donors might detract from
             perceived objectivity, such as Moroccan officials’ skepticism about the motives
             of USAID and EU regarding reduction of protection of cereals, the Bank should
             do the work alone (with the client); and

       •     On processes, the peer review process for the three activities reviewed by the
             Panel was very mixed, and the quality of information in the SAP and BW was
             unsatisfactory.
Cross-Country ARD Sector AAA Assessment                                                     32




45.      Strategic Relevance. A major rural development strategy had been undertaken in
1998, which was judged (and is still judged) to be highly relevant. Building on this strategy,
the July 2001 CAS envisaged completion of work on agricultural incentives in 2001,
undertaking work on agriculture/water strategy in 2002, and “expanded” ICRs on large-scale
irrigation in 2001 and rural water supply in 2003. The scope of this analytical work was fully
appropriate.

46.     Given the CASs emphasis on focusing on the rural poverty challenge, and a history of
zero rural growth in the 90s, the AAA program was rightly focused on changing incent ives to
promote growth, in a manner that would take account of poverty implications. This was the
thrust of the policy note that was put to the new government at the end of 2002. At
Government’s request, the Bank agreed to conduct joint analytical work on cereals trade and
price policies and large-scale irrigation management first, to be followed by research and
extension, with institutional restructuring to come later. None of this work was undertaken
for the purpose of underpinning lending, although lending could well arise in respect of, for
instance, direct income support during the cereals policy transition. Undertaken as a fully
collaborative exercise with two government-working groups, the program was designed to
maximize knowledge sharing. Overall, relevance was highly satisfactory.

47.     Internal Quality. Without breaking any new ground, the three written outputs
contain good quality analysis. The policy note produced for the new government was a
model of the genre, synthesizing a vast amount of material in 15 pages, with a clear statement
of recommendations, and indications of how the Bank would be willing to help. Both the
cereals and irrigation papers are very good short statements of complex issues, drawing on
worldwide experience in developing policy options, and not shrinking from indicating
differences between government and Bank positions.

48.     No surveys were launched to look for new evidence, but appropriate consultancies
were arranged to fill gaps in, for instance, legal constraints on irrigation management reform,
and modeling the effects of cereal pricing reforms on production systems. Related work
under the Poverty Study (completed in April 2004) analyzed the distributional effects of
cereals reforms, and made it possible to pinpoint the income effects as a clear guide to
refining needed accompanying measures. Each of the three main pieces of work, and the
work on forestry too, contain good discussions of options followed by clear statements of
conclusions and recommendation. These stateme nts are mutually consistent in striving for
pro-poor growth. Overall, quality was fully satisfactory.

49.     Likely Impact. Impact on the client is somewhat mixed. Impact in terms of building
ownership and coalitions for change has already been achieved, via the composition of the
working groups and the fully participatory methods that they adopted. Bank participation
was low-key, and rightly so. Some changes in government policy have recently been
announced in respect of aspects of cereals marketing reform, including some pilot activities,
but nothing has yet been decided for large-scale irrigation. In respect of cereals, moreover,
the more difficult areas of trade and consumer subsidies are still pending. Liberalization of
the agricultural price system, which could have been chosen as a success indicator, has still
not yet been fully attained. By contrast, impacts on capacity building and knowledge transfer
have already been attained.
Cross-Country ARD Sector AAA Assessment                                                     33


50.     The impact on the Bank is likely to be high. The programmatic approach to AAA has
clearly been effective, and is likely to be adopted by other sectors in Morocco (if not already
done) and in other countries within the ARD sector. The country knowledge base of the
Bank and its staff has greatly benefited from operating with client- led working groups.

51.     Comprehensiveness. Reference has already been made to the sequencing of
analytical work, starting with cereals and large-scale irrigation, followed by research and
extension and institutional restructuring. Building on the 1998 strategy, the CAS laid out a
fairly modest lending program ($100 million and 2-3 projects per year) but with about one-
third for ARD. These included three successive operations for community-based
development in projects in small-scale irrigation, rainfed areas, and forestry and natural
resources management. Each of these projects has, in fact, come to fruition. This suggests
that no further analytical work, beyond project preparation, was needed in these subject-
matter areas and that the selectivity applied to design of the AAA program was highly
appropriate.

52.     These remarks apply to the agricultural aspects of ARD but nothing has been said
about the non-agricultural aspects. The CAS had envisaged an expanded ICR on rural water
supply, and a rural infrastructure project in FY03. The panel has seen no documents on these
or related matters and is therefore concerned about the comprehensiveness of the whole AAA
program for ARD.

53.     Planning and Bank Processes. The multi- year programmatic approach to design of
sectoral AAA seems to be working very effectively. Noteworthy aspects of the program in
Morocco include:

   •   Appropriateness of coverage and scope of the sector AAA program;

   •   Relevance of the program to the CAS and the SSP, including water and forest sector
       policies;

   •   The contribution of the program to promoting policy dialogue;

   •   Quality of client participation in defining the scope of the work, carrying it out, and
       formulating conclusions and recommendations; and

   •   The coherence of the program over time and the consistency of effort.
Cross-Country ARD Sector AAA Assessment                                                       34


                        TANZANIA S ECTOR AAA COUNTRY ASSESSMENT

54.      Summary Comments. This assessment is based on three AAA tasks. One, entitled
Rural Development Sector Work (June 2002), was really two pieces of work: a review of
institutional arrangements fo r promoting cross-sectoral rural development; and an
examination of (relatively poor) agricultural export performance. This is a high quality
product. The other two tasks are: a poverty and social impact analysis of crop boards, as a
follow-up to the work on agricultural exports; and an analysis of agribusiness supply chains.
Both these tasks are incomplete. For the former, for which the first draft report is expected
shortly, scope and strategic relevance are highly satisfactory but it is too early to m     ake
assessments of any other aspects, although they are also likely to of high quality. For the
latter, for which a first draft report recently became available, the Panel identified a number
of deficiencies, though these may well be addressed during the Regional review of the draft
report.

55.     While the AAA program was not undertaken to underpin lending, it is surprising that
the actual (in contrast to CAS-planned) lending program included a Participatory
Agricultural Development and Empowerment project that seems to have no tie to the CAS or
to any AAA. New lending in research and extension is being contemplated and it is
surprising that prior sector work has not been initiated instead of, for instance, work on
supply chains. As for the non-agricultural aspects of ARD, the CAS envisaged nothing
beyond (indirectly) fiscal decentralization. The panel has seen no documents on any other
related matters and is concerned about the comprehensiveness of the whole AAA program
for ARD.

56.       Key findings from the Panel’s review of the Tanzania program include:

      •   Overall, the program was strongly linked to initiating/moving the dialogue forward.
          Dialogue with Government on exports seems to have been stalled when reforms were
          rolled back in 1999-00. The Bank’s AAA program was intended to restart it, and
          subsequently to deepen it though poverty and social impact analysis. The work on
          institutional arrangements for rural development was also intended to move the
          dialogue forward;

      •   The work on exports (both tasks) and supply chains was cutting edge, and undertaken
          to share knowledge and build capacity;

      •   Broadening of the agenda from agriculture to rural development was the right move
          (and in line with the Bank's sector strategy);

      •   The work on crop export boards is innovative. It addresses the contentious and often
          neglected area of second-generation reform of marketing institutions. If successful, it
          will become a best-practice case to be scaled up in those countries that are wary of
          full liberalization but unsure of what institutional and regulatory arrangements to
          keep in place;

      •   An aspect common to work on exports and supply chains was shortcomings in the
          quality/use of empirical evidence. Egregious errors need to be cleaned up in the
          supply chains report;
Cross-Country ARD Sector AAA Assessment                                                      35


   •   Another aspect common to exports and supply chains was an inadequate conclusions
       and recommendations section. For exports, the panel would have welcomed greater
       differentiation in conclusions and recommendations between the crop boards. For
       supply chains, it is important that a clear statement of conclusions and
       recommendations be made in the text and succinctly put into the summary;

   •   Likely impact would be enhanced if conclusions and recommendations were clearly
       stated; and

   •   The rural dimension has to be brought more explicitly into design of the AAA
       program for ARD. Perhaps this is already being done informally, but it needs to be
       made explicit.

57.     Strategic Relevance. The CAS had envisaged only one piece of AAA in ARD: a
rural development strategy in 2002 to follow up on two major pieces of agricultural sector
work that had been completed before mid 2000. This broadening of emphasis from
agriculture to rural development was sound. The budget of $175,000 implied spending of
only $60,000/year, surely too little for a country the size of Tanzania and for a sector of such
importance for poverty reduction. In the event, actual spending of $190,000/year on four
tasks was a far more appropriate (about $150,000 of this total was funded by BB). Much of
the actual program (two tasks and about 70% of spending) was on agricultural exports,
hardly a top priority for poverty reduction, but perhaps the earlier work had dealt adequately
with commodities for domestic markets.

58.     Among items not included in the program, some analysis of the scope for growth
would have been warranted. With the Ministry of Finance unwilling to invest in rural
development so long as agriculture under performed by not achieving growth of 7%/year,
and with the CMU having reservations about rural development, a short analytical piece
could have been very useful showing what growth could reasonably by expected and what
policies and public expenditures would be needed to achieve it. This could have drawn
heavily on the two major pieces of work that had been completed mid-2000.

59.    The rationale for undertaking work on supply chains is not clear, in large measure
because no concept paper could be found, but also because the draft report does not make the
case convincingly. Although small cost ($40,000), an alternative could have been to take a
fresh look, via an expanded analytical ICR or as a self-standing ESW piece, at the
disappointing performance of research and extension.

60.      Internal Quality. The work on institutional arrangements for rural development was
not analytically demanding but was done competently. The work on agricultural exports was
in many ways of high quality, but could have been improved by (e.g., examining strengths
and weaknesses of cooperatives and their relations with crop boards, and providing
illustratio ns of good marketing practices from other countries). In addition, the report points
out significant differences in export crop performance in Ethiopia, Kenya, Tanzania, and
Uganda, but does not convincingly explain them. The work on supply chains contains little
real analysis of performance of the chains with respect to producer and consumer welfare,
and how the chains might be made more effective. Limited, and poor use of, empirical
evidence is a problem in the work on supply chains. Absence of clear conclusions and
recommendations is also a problem, which, if not corrected, would prevent any likely impact.
Cross-Country ARD Sector AAA Assessment                                                       36


In export crops, however, conclusions and recommendations were not candidly stated for
tactical reasons, but should nonetheless have been better differentiated between the boards.

61.     Likely Impact. There are many vested interests in preserving the status quo
regarding crop boards, and many vested bureaucratic interests in preserving the status quo
regarding "turf" in rural development. These two pieces of work have, as yet, had only
moderate impact on building ownership and coalitions for change in these respects. There
has been some impact on reducing the burden of local taxes on exports, although possible
backsliding by way of substituting fees for reduced taxes will have to be watched. There
have been some minor improvements in the behavior of crop boards, but significant change
will not happen until the PSIA work on options for the boards has been completed and
digested. As for impact on the Bank, some recommendations were incorporated in the PRSC
agenda, and the continuing work will feed into the next CAS. The work was not designed to
underpin lending, except for supply chains, although lending in other areas could well ensue
in, for instance, transitional arrangements for crop boards' operations. In addition, farmers'
groups and private businesses have been provided with thorough analytic work on
agricultural exports that they may be able to use to lobby government more effectively.

62.      Comprehens iveness. The heavy focus on agricultural exports seems out of
proportion to the scope for poverty reduction, but perhaps earlier work dealt adequately with
commodities for domestic markets.              The work on supply chains has some
overlap/complementarity with exports and crop boards, but this is not explained. The
impression is that these tasks were undertaken quite separately. Other than the work on
institutional arrangements for rural development, the panel does not know of other AAA
tasks dealing with other aspects of rural development.

63.     Planning and Bank Processes. The inability to find concept papers and reviewers'
contributions for the initiation of work on agricultural exports, and later supply chains, may
just be an administrative lapse. If not, it reflects something seriously amiss in the planning of
AAA. At least, initiation of the work on institutional arrangements for rural development,
and on crop boards, was given serious consideration as demonstrated in the supporting
documents.
Cross-Country ARD Sector AAA Assessment                                                     37


                       UKRAINE S ECTOR AAA COUNTRY ASSESSMENT

64.     Summary Comments. While the operating environment in Ukraine is complicated
by the frequent changes of government and wild swings in the policy environment, the Bank
has adjusted well to this situation through the creation of the Agriculture Policy Analysis
Group, a group of western-trained Ukrainian economists who act as proxies for the Bank in
agricultural analytical work and policy dialogue with government officials. Challenges
remain in terms of how to place this institution, now three to four years old, on a sustainable
financial footing, but the Region is to be congratulated on the innovative and entrepreneurial
way in which this capability has been developed and the way it is used to enhance policy
change.

65.     This aside, the Panel is concerned that strategic and analytic knowledge of the rural
sector in Ukraine within the Bank is being eroded. Perhaps because of the success of the
Policy Group, AAA funded through bank budget over the review period has been minimal --
to the extent that it is zero in some years. The sector manager cannot operate a coherent and
ongoing sectoral AAA work program when funding is insufficient and intermittent. An
added difficulty occurs when tasks are multi- year but funding comes annually with no
guarantee on the level of funding in the second or subsequent years.

66.     Strategic Relevance. The AAA is set in a context of an intermittently dysfunctional
policy dialogue. Despite this, the tasks are relevant and consistent with the CAS they fit well
into the objective of being current on policy advice and agricultural strategy in the event of
an opening in the policy dialogue. But coverage appears thin with only three pieces of AAA
shown as tasks in the last four years. Staff are aware of gaps that exist because of budget
constraints. There is little evidence of ex-ante success indicators. Work is undertaken in a
timely manner. Given the very unstable operating environment, the AAA has focused on key
issues and the work of the “Policy Analysis Group” in Kiev, has been very instrumental in
arresting poor policy proposals, preventing back-sliding, and conveying World Bank policy
messages. The work is well aligned with the country SSP (now becoming somewhat dated
and being replaced by a new document produced in collaboration with OECD), but a
remaining concern is that the sectoral knowledge is being degraded by severely-constrained
AAA budget.

67.     Internal Quality. Although little ESW has been completed recently, that which has
been done is of very high quality and addresses key issues. Well-qualified and experienced
specialist economists have undertaken complex analytical tasks and produced very good
reports. While the Panel did not make an independent assessment, it appears that the quality
of work undertaken by the Agricultural Policy Group is equally good. This group has also
built up a database, which has benefits for World Bank sectoral economists when they
undertake AAA.

68.     Likely Impact. The AAA work is designed and carried out in a way that offers the
promise of significant impact on the client. The difficulty arises from the (sometimes)
unresponsiveness or even hostility of the client to this work and the weak ability to
implement policy change even when a willing government is in office. The establishment of
the Policy Analysis Group is an innovative approach and an excellent example of enhancing
capacity and knowledge transfer. It is a best practice. The AAA work is highly relevant to
the Bank and appears to have informed lending operations. The limitation arises from the
highly constrained budget for AAA in ARD. The Bank has close consultations with key
Cross-Country ARD Sector AAA Assessment                                                       38


donors and joint funding of AAA is often undertaken. Thus the opportunity for AAA to
impact donors exists. Partnerships with key donors are well developed and policy priorities
appear to be quite clearly agreed upon. Civil society organizations are quite nascent in
Ukraine. The opportunity to have an impact on them is quite limited. On the other hand, the
emerging private sector stands to bene fit considerably from AAA, which has led into policy
changes and operations.

69.      Comprehensiveness. The unit has done a good job identifying priority areas that
require analysis. Work has become tightly focused on three cross-cutting themes common to
the entire CD. But it has not been possible to conduct a coherent work program as resources
for AAA have limited work to less than one activity per FY. In this fiscal year, we
understand that there is no budget allocation for ARD AAA. Staff acknowledge gaps in
knowledge and an aging portfolio of sector strategy work. Further, and of concern to the
institution, many pieces of AAA are informal and often unrecorded (as they may be financed
off-budget). The concern is that this work does not become part of the institutional memory
and is quickly lost to the institution.

70.     Planning and Bank Processes. In the Panel's view, this is a very weak link in the
country program. For a country where agriculture is so critical to national employment and
growth, and where agriculture is so in need of fundamental reforms if it is to deliver those
benefits, it is indeed surprising to observe that the sector receives such limited budget support
and that dedicated sectoral economists on staff are fewer and fewer. The few pieces of AAA
undertaken have been well planned and executed, but a sectoral AAA work program and plan
really does not exist. Budget allocations are intermittent and, when made, inadequate. Staff
are forced to contract-out much of the work and to spend considerable time soliciting and
managing alternative non-BB financial support.

71.     The September 2003 CAS for Ukraine notes several areas of interest (a forward
agenda) identified by local policymakers as priorities. Some directly mention agriculture
(land titling, secure transactions) while others affect agriculture less directly but nonetheless
importantly (environmental sustainability, energy reforms, targeted poverty interventions,
and improvements in the business environment). The AAA undertaken on the rural sector is
consistent with these priorities. The ARD AAA that has been undertaken appears to have
had an important influence on the thinking of policymakers in Ukraine and has contributed
directly to operations. Most operations related to agriculture have taken place within the
context of the three PALs that have been the centerpiece of operations, and these components
have been informed by prior ARD sector work. This is a particularly strong aspect of the
ARD work, a finding that is common with that of the general AAA assessment for Ukraine
undertaken in 2003. The major issue requiring attention is the disconnect between the
importance of agriculture to economic performance and growth – the imperative of making a
successful transition to a market-driven economy in the rural sector – and the amount of BB
resources allocated in the country program to agricultural AAA. In recent years, the budget
allocation for agricultural AAA has financed less than one piece of work per year, and even
then task managers must seek outside resources to undertake the work at the scope and
quality expected of bank reports. In other words, necessary, front- line work that can be
legitimately funded from BB is being under funded, forcing diligent staff to seek off-budget
funding and to work off budget to complete AAA tasks.

72.     The lack of regular and sufficient funding for ARD AAA has staffing implications. It
is increasingly difficult for sectoral units to employ dedicated (or even 'a' dedicated) sectoral
Cross-Country ARD Sector AAA Assessment                                               39


or policy economists. When individuals have joint responsibilities with lending, projects
tend to crowd out sector work. And lack of sector work means that the Banks analytical
expertise in ARD is being seriously undermined. It may get to a situation where we cannot
credibly fill a seat at the rural policy table.
Cross-Country ARD Sector AAA Assessment                                                      40


                                                                                      ANNEX 4

         COMMENTS ON THE TECHNICAL QUALITY OF SPECIFIC STUDIES
                  IN BANGLADESH , MEXICO , AND TANZANIA


      COMMENTS ON “PROMOTING THE R URAL NON-FARM S ECTOR IN BANGLADESH
                                    Main Report, Volume II

73.     This paper contains a huge compilation of data and description about rural
Bangladesh, and the economic connections and relationships among agriculture, the rural
non- farm economy, and the rest of the country. It looks not only at the current situation but
also trends linking past changes and current conditions, notably how trends in productivity,
trade, investment, urbanization, and policies have affected the rural poor. A problem
however is that the bulk of the analysis consists of associations rather than identification of
underlying causal factors, which ends up providing disappointingly little grounding for
policy or lending recommendations.

To review the Main Contents

74.      The introductory section (pages 1-10) contains basic history and facts, and most basic
for the rest of the paper, a discussion of the conceptual framework for the work. This
framework however consists mainly of definitions of concepts such as “rural area” and “non-
farm” which of course help in keeping straight the categorization of data, but do not
accomplish an ana lytical task. Figure 1.1 provides a visualization of “the rural- urban and
farm- non-farm continuum,” which shows linkages in five connected locations from village to
metropolitan city (one of two dimensions), and from primary production to final goods and
services (the second dimension). This organizational setup is a necessary first step in
analyzing the economic situation in Bangladesh when the focus is to be on a largely non-
standard object of study like the rural non-farm sector, but here it is, conceptually, also the
last step taken. More in the way of conceptual framework is needed. For example, page 5
states that “average labor productivity” is higher in RNF activities as compared to
agriculture. Presumably, this is measured by output per worker although this isn’t stated.
What we need from an analytical framework is a capability to interpret an indicator such as
labor productivity. It could be a matter of factor ratios (more capital per worker in RNF), or
relatively higher skills in RNF work, or disequilibrium in labor markets (with excess labor in
agriculture). One has to conceptualize these or other reasons in order to formulate
hypotheses about what policies or investments make sense for the RNF sector or for a
decision to give RNF investment priority over alternatives (like agricultural technology
investment). Average labor productivity being higher doesn’t necessarily imply that the
value of marginal product of labor is higher in RNF (though it may well be). The paper
should have pointed to the distinction and tried to assess the facts of this matter. This is the
kind of conceptualization necessary to ask the most useful questions of the data, and the lack
of that in the paper ends up being costly in terms of relevance of the findings.


75.    On page 5, data are cited showing that the RNF sector generates a high share of rural
income, and that share is growing relative to the agricultural sector, and that as quoted above
labor productivity is higher. Evidence is cited that higher income is associated with RNF
Cross-Country ARD Sector AAA Assessment                                                        41


growth, inferring that RNF growth has been pro-poor. What is the implication? It seems to
be that it is a good idea to encourage RNF growth. Yet Figure 1.2 shows that agricultural
income predominates for the lowest deciles of the income distribution. So if we have two
possible alternative policies or investment strategies for reducing rural poverty, one focused
on increasing income from agriculture and the other on increased income in the RNF sector.
Which is preferred? Pages 6-7 argue that growth of the RNF sector is essential. So one
might conclude that RNF promotion is preferred. Yet the first “emerging opportunity” put
forth is about yield growth in agriculture, as “an essential pre-condition for accelerated RNF
growth” (page 8). The remainder of the introductory section goes back to RNF promotion,
but I was mystified as to what the rest of the paper was going to do to help make choices
about policy or investment alternatives. The concluding section of the introduction says the
paper “focuses particularly on identifying constraints to promoting” the RNF sector in
Bangladesh, and will conclude with a strategy for promoting that sector (pages 8-9). So the
paper by-passes key issues of priority mentioned above. This is fair enough, but it does limit
the relevance of the work.

76.     Section 2 describes and mobilizes data on the linkages between agriculture and the
RNF sector. The “bottom line” for analytical purposes is a set of constraints to growth of
“agriculture related RNF activities” (page 16). These are: (1) low yields and lack of year-
round supply of high value crops; (2) low yields in livestock; (3) lack of food quality
assurance; (4) improvements needed in handling, storage, processing, and transporting of
perishable commodities; (5) improvements needed in quality of inputs; (6) lack of a “fully
integrated supply chains for fruits and vegetables;” (7) tariffs, taxes, and regulations that
discriminate against (small-scale) agro-processors; and (8) needed infrastructure and finance
for RNF linkages forward and backward. These are reasonably likely constraints but the
factual basis for choosing them, or prioritization among them, is not provided.

77.     Section 3 addresses poverty, inequality, and access to the RNF economy. Data
outlined in the introductory section are elaborated upon, with more locational detail about
employment and income sources. Constraints to better access are identified as: lack of
education, distance to source of RNF employment and existence of markets in nearby
villages, electricity availability, credit availability, and gender-related constraints (page 25).
The analytical basis for identifying these constraints is a regression analysis of household
data on non-farm employment, with statistically significant variables taken to imply the
constraints listed. That is, if a variable is significantly and positively related to non-farm
employment, that is taken as evidence that increasing the correlates of that variable will
increase employment, which is reasonable.

78.     Section 4 provides further regional detail, and brings in the urbanized parts of
Bangladesh. The section concludes not with constraints but with contributors to more
balanced (regionally more uniform) growth and income levels. These factors are an efficient
transportation system, political and fiscal decentralization, increased openness to
international trade, and strategic investment is smaller towns. Here the basis for choosing
these factors is less apparent than in Section 3, and for some factors, notably the role of
international trade, evidence is entirely lacking

79.    Section 5 considers the climate for investment in rural areas, particularly for small or
medium-scale enterprises. Growth of such enterprises is taken to be a promising avenue of
RNF development. Survey responses indicated that constraints to such growth are lack of
access to finance, exposure to flood and other disaster, electricity, road conditions, and
Cross-Country ARD Sector AAA Assessment                                                       42


(deficiencies of) transport to market. Governance issues (regulatory burden and corruption)
were seen as bigger problems by large, urban-based firms. Proprietor education is important
to the success of smaller enterprises.

80.     Section 6 covers service provision to rural enterprises. Many of the same constraints
discussed in previous sections are important here also, notably transportation and finance, but
in this section with more emphasis on communications, particularly telecommunications.
This section also sees the need for “strategic public intervention in providing the regulatory
framework and in ensuring access of the poor to services” (page 71) to an extent not
approached in the preceding sections.

81.     Section 7 (pages 75-84) is where all the preceding strands are pulled together to
outline “strategy and options for promoting the rural non- farm sector” (page 75). Here the
doubts engendered by the wide range but too casual substantiation of the findings of the
preceding sections come to fruition (on non-fruition). The recommended strategy has two
main thrusts: (i) “improving the rural investment climate,” and (ii) “devising an institutional
framework for efficient delivery of services” (page 77).

82.    The components of (i) are: (Section 7.6) removal of anti-rural bias in trade and
macro-economic policies; removal of constraints in physical and financial infrastructure,
notably in (Section 7.7), roads, (Section 7.8) electricity, (Section 7.9) telecommunications,
and rural finance (Section 7.10); (Section 7.12) improving disaster mitigation; and (Section
7.11) providing public goods (information, research in agriculture) and a regulatory
framework for the agro-food supply chain. The discussion then goes on to (ii)
recommendation for implementing the strategy through institutional changes, including
involvement of RNF stakeholder groups in government decision- making, decentralization of
public decision- making to local levels, a framework for NGO participation, and new ways to
provide business development services (pages 83-84).

83.     The strategy is reasonable but the ties back to the analytical work of the paper are
weak overall and for some components hardly exist. A skeptical reader would ask, why
these investments and policies and not others? From the point of view of task managers, we
really do not have evidence on rates of return (or benefits and costs) to be expected from the
recommended actions, even in rough terms. Perhaps though this is too much to ask of even a
fairly large piece of economic sector work. The paper does lay out a lot of policy and
investment options, placed in the context of the specific situation of Bangladesh today. It
nonetheless remains the case that in order to prioritize among the ideas put forward, it would
take substantial further analysis beyond what this paper has provided.

84.      One may also ask about a basic presumption of the paper – that investment and
policies that encourage the growth of RNF are preferable to a growth and poverty reduction
strategy for Bangladesh that does not aim specifically at RNF development. On this question
the paper cites, without much detail, a literature on RNF sectors in other countries that is said
to support the wisdom of RNF focus. I rather doubt the decisiveness of the findings of that
literature in allowing us to conclude that the menu of Section 7 will be a more effective
poverty reduction strategy for rural people than, for example, investment in research on rice
or other big internationally traded crops, improved elementary education of rural youth,
investment in public health-related activities, to name a few. Not that the paper could have
examined all alternatives; I just think it should have examined in more economic-return
detail the benefits and costs of the topics the paper did cover.
Cross-Country ARD Sector AAA Assessment                                                        43


     COMMENTS ON “M EXICO LAND POLICY – A DECADE AFTER THE EJIDO R EFORM”
85.     This is an excellent piece of work. I will go over its main points as I see them and
then give my assessment. I am looking at the paper with a view, as requested, to its
analytical quality, usefulness as an instrument for policymakers in the country, and extent to
which it provides useful information to Bank managers in country sector strategy and to task
managers in the design of lending activities (note from Amnon Golan).

86.     The paper contains a lot of interesting and relevant factual and historical background
in its Sections 2 and 3 (pages 2-34), but the heart of its analytical contribution is in Section 4
(pages 34-51), “Impact of the legal changes of 1992 and the PROCEDE.” This is a really
nice exercise in social science research. It attempts to provide on-the-ground empirical
evidence on the effects of Mexico’s ambitious program to establish (something approaching)
individual property rights in land ownership by residents of ejidos, which had been, roughly,
under a regime of collective ownership. This change was made in hopes of economic gains
for the farmers but was controversial, as the paper describes, because of fears of the kind that
typically arise when private-property/market institutions are given greater scope. It is
therefore quite important to assess with as much care and objectivity as possible what has
happened in the decade since 2002 and what is plausibly attributable to the policy changes.

87.     The paper’s analysis is based on a large (1297 observation) panel of households
surveyed twice, in 1994 and 1997 (so we really only get to five years after 1992). The
surveys, done with FAO and the University of California (not the “University of Berkeley”
as page 35 has it), appear to have been carried out with high standards and must have been
costly to conduct. The survey asked not only about technical and economic aspects of
agricultural production, but also about a range of socio-economic characteristics of the
households and about respondents’ subjective perceptions regarding the PROCEDE (Table
6).

88.     A reasonably sophisticated econometric analysis estimated effects of the policy
changes on participation in land rental and sales markets, access to credit, household income
growth, and access to natural resources (common lands). The findings were of greater
participation in land markets, no loss of access to common lands, and no improvement in
access to credit. In addition, the subjective responses indicated improvements in “social
peace,” presumably because of settling uncertainties about who had rights to what. Most
importantly, the study found reasonable strong evidence that the policy changes increased
household incomes, and argued, to me convincingly, that this was likely because the greater
tenure security meant people could work off the farm without the risk that by leaving their
land relatively less fully tended to take such jobs they would open themselves up to loss of
land rights to others in the community.

89.     The recommendations section of the paper is reasonable, but many of them do not
follow directly from the findings of the analytical work. The real contribution of the paper in
my view is the support it gives to the wisdom of the 1992 policy changes and the PROCEDE
process (notwithstanding the political problems of incentives within that process that the
paper straightforwardly discusses).

Turning to the criteria for evaluation of the paper
Cross-Country ARD Sector AAA Assessment                                                      44


90.     Analytical quality: excellent, on the frontiers of applied econometric work in policy
analysis, first rate attention to detail in the surveys, substantively appropriate asking of
questions and creative ways of marshaling data to answer them.

91.     Usefulness to bank managers: sector strategy. Very high marks. A prime difficulty
with policy recommendations by the Bank is that they can be taken as too much assertion
based on general principles without fact-based consideration of a particular country’s
situation. The findings of this paper go a long way to answering that sort of criticism.
Recommendations for further implementation of PROCEDE-like policies now have a much
more solid grounding than they would have in the absence of the research.

92.     Usefulness to task managers. Here the limited scope of the paper makes the
usefulness much more constricted. The paper provides no support for (or against) most of
the kinds of loans the Bank makes. The area most directly relevant might be investment in
financial institutions to overcome credit constraints facing rural households. But even here
the negative finding of the research – that PROCEDE and related strengthening of property
rights in land have not enabled a significant increase in land-based lending (because, as I
understand it, the ownership rights cannot be conveyed to a bank or other lender through
foreclosure, so hamstringing the use of land as collateral) – has direct implications for the
expected returns to rural- finance institutions.

93.     The contrast with the Bangladesh paper is striking. I was critical of that paper
because it did not provide the kind of analytical concentration and quantified findings that the
Mexico paper does. On the other hand, the Bangladesh paper took on a much wider range of
issues and options. Would the Bank rather have what I would characterize as a relatively
narrow and deep analysis, as the Mexico paper provides, or a wide and shallow one as the
Bangladesh paper provides? The exercise so far is inclining me to prefer the narrow and
deep approach for economic and sector work. The broader assessment is necessary to see the
range of what might be possible. But, without being fully familiar with the range of
preparatory work the Bank does, I believe the broader assessment is more effectively done by
the country experts within the Bank as part of the broader country assessment papers, and the
economic sector work is more usefully focused on a relatively narrow range of options which
it should analyze in depth sufficiently to be able to come up with some quantification of
potential benefits and costs of policy or investment alternatives.

                     COMMENTS ON “TANZANIA RURAL D EVELOPMENT”

94.    This paper is not a description of a single project of sector work, but rather contains
two parts: a report on a fairly broad program of activity highlighting rural development, and a
focused analysis of agricultural export growth, as “a key pillar of the strategy to reduce
income poverty” in Tanzania (page vi).

95.     The first major section of the work (pages 5-26) addresses Tanzania’s rural
development strategy and its implementation. For purposes of carrying out analytical work,
it is most helpful that the Government has prepared this strategy. Having the Government’s
own strategy in place limits the huge range of alternative policies and programs that might
otherwise have to be considered. The strategy has four “pillars”: (1) promoting widely
shared growth across components of the rural economy, from small farms to tourism to
fisheries; (2) increasing access to services, from education and health to technology and
energy; (3) reducing risks and vulnerability; and (4) good governance. Each of these covers
Cross-Country ARD Sector AAA Assessment                                                       45


a lot of territory, so implementation requires narrowing down the sphere of action as well as
spelling out implementation processes.

96.     In addition to narrowing the scope in a broad sense, the Government of Tanzania has
already accomplished some necessary further focus with respect to the first pillar through its
agricultural sector development strategy, aimed at improved farm income and reduced rural
poverty through improved productivity and profitability of the agricultural sector. The
Report, after several page s devoted to institutional issues of implementation which are
sensible but not analytical challenging (or even interesting in my view), takes on the
substantively central question of prioritization of implementation alternatives. This is done
principally by prioritizing expenditure options.

97.      The discussion of priorities begins with those the Government identified in its broad
Poverty Reduction Strategy, namely, primary education, primary health care, agricultural
research and extension, rural roads, water, judiciary, and HIV/AIDS. The Report takes the
appropriate analytical approach of stating the chief criterion for prioritization as being choice
of options where the marginal returns to public expenditure are highest, and proceeding to
the point that the marginal return to additional expenditures is equal across expenditure areas
(page 16). The practical analytical question though is how to carry out this approach. Given
the huge data and analytical requirements of applying these principles, the paper starts out
with the reasonable approach of reviewing what has been learned from this kind of
prioritizing activity in other countries, notably, China and India, which have been intensively
studied, and some particular quantitative studies in Africa. This information is then nicely
merged with considerations about Tanzania to argue that two areas of public investment offer
notably high returns: agricultural research and rural roads (pages 17-18). The remainder of
Section 2 discusses prioritization further but goes not carry out any further quantitative
analysis. It is nonetheless quite valuable to sketch out the criteria and methods of doing such
analysis, and underlining the importance of an analytical economic (as opposed to political or
rhetorical) approach to prioritization.

98.     The second major section of the Report (pages 27-59), and the five Annexes (pages
70-202) that make up the bulk of the Report, focus on agricultural exports. This section
provides an excellent compendium of facts and trends, and does this not only for aggregate
agricultural exports but also for individual key commodities. Moreover, the discussion
moves beyond the domestic factual basics to international comparisons and analysis of the
economic behavior of Tanzanian producers. These steps are essential in the knowledge base
for policy, because one has to know how Tanzanian agriculture will respond to alternative
price and other policies in order to gauge the benefits and costs of the policies, and this is
knowledge that is all too often lacking in discussion of policy alternatives even for the
advanced industrial economies. The sub-section on supply response and the DRC indicator
of competitiveness is a nice illustration of how to combine an intuitive analysis with key
facts, and findings from a surprisingly extensive outside literature, to provide parameters one
needs to carry out a quantitative analysis of policy alternatives.

99.     The conclusion and recommendations (Sub-section 3.11) reviews the situation and
ends with six bulleted recommendations (page 59). Given the wealth of well- focused detail
that precedes those recommendations, it is disappointing that the recommendations do not
pull the analysis together to quantify what is at stake in the policy choices recommended.
For example, the first recommendation is that “taxes on export crops must be reduced.”
Instead of just presenting this in the imperative mode, it would have been far preferable to
Cross-Country ARD Sector AAA Assessment                                                        46


present at least a rough quantification of the gains to be had by reducing those taxes. For
example, if Tanzania were to tax its coffee exports 10 percent less, how much more would
producers get, how much more would they produce in response, what is that additional value
generated for the Tanzanian economy, and how does that compare to the tax revenues lost.
For most commodities in most developing countries, there would not be sufficient analytical
information even to begin an answer. This paper provides a great deal of the necessary
information for Tanzania, so it is frustrating to see the final steps not taken (or if remaining
defects of knowledge prevent that, the paper spelling out what additional data are needed and
how they would be used).

100. The remaining five recommendations are broader, and not capable of benefit-cost
quantification given the information available. But the example of the first could be made
more helpful by giving policymakers a better grip on what kind of information and analysis
would be required to assess the real gains available from those policies.                   The
recommendations on investment and growth could have been expanded upon briefly to
discuss the increase in complexity necessary to assess the dynamic effects of policies, where
the big gains are likely to be reaped only after many years of consistently followed policies.

101. Following up the thread of the preceding remarks, it would have been useful for the
paper to have had a brief concluding section that integrated the two main sections. The first
section sets up general “best practice” procedures for policy prio ritization. The second
section provides a good case study of how to implement those procedures using the highly
relevant case of exported commodities. The implementation exercise is useful not only for
its specific content but because it gives concreteness to the more general ideas of the first
section. Without the implementation exercise, the general prioritization could easily appear
to government officials or others influential in decision- making to be too “academic” or
lacking in practicality. A brief concluding section could have stressed the division of
intellectual labor between the two main sections (and would have given further impetus to
pushing the export policy section toward a more finished benefit-cost analysis).

102. Although I have ended up being critical of the Report for not going as far as it could
have gone, this does not detract from the high quality of the material that the Report contains.
In usefulness to Bank managers in rural development strategy formulation, and to task
managers in lending, the data and approach to analysis of data provide an excellent baseline
of information and a benchmark for analytical contributions needed in the evaluation of
particular policies or lending programs. The usefulness to Tanzanian policy professio nals is
similar but in addition is helpful, as far as it goes, as an example of best practice in practical
policy analysis.

								
To top