HOW TO STRENGTHEN by qyz18834

VIEWS: 11 PAGES: 16

									             PART IV
       HOW TO STRENGTHEN
         A GOVERNMENT
          M&E SYSTEM


O
           ne of the lessons to heed in building an M&E system is the im-
           portance of conducting a country diagnosis (chapter 12). This pro-
           vides a sound understanding of current M&E efforts, the civil
service environment, and opportunities for strengthening M&E and using M&E
information for core government functions. Perhaps equally important, a di-
agnosis helps focus key stakeholders within government and in the donor com-
munity on the strengths and weaknesses of current M&E arrangements in the
government. Such a diagnosis can also ensure that these stakeholders share
a common awareness of the issues. A diagnosis naturally leads to an action plan
that identifies the main options for strengthening a government M&E system
(chapter 13).
                                                                                                12
The Importance
of Country Diagnosis


I
    t should be apparent that there is a great diversity in country approaches
    to M&E. Countries such as Brazil stress a whole-of-government approach
    to setting program objectives and creating a system of performance indi-
cators. Others, such as Colombia, combine this with an agenda of rigorous im-
pact evaluations.
Yet others, such as Australia, the United States, and   planning, to aiding ongoing management and de-
the United Kingdom, stress a broader suite of           livery of government services, to underpinning ac-
M&E tools and methods: performance indica-              countability relationships.
tors, rapid reviews, impact evaluations, and per-
formance audits. Some countries have succeeded          It is clearly important to tailor efforts to build or
in building a whole-of-government M&E system.           strengthen government M&E systems to the needs
Others, such as Uganda, use a largely uncoordi-         and priorities of each country. Conducting a di-
nated and disparate collection of about 16 sepa-        agnosis of M&E activities is desirable because it
rate sector monitoring systems. Most of the             can guide the identification of opportunities for
poorest countries—those required by multilateral        institutionalizing M&E. A formal diagnosis helps
donors to prepare poverty reduction strategies—         identify a country’s current strengths and weak-
stress the regular collection of performance in-        nesses in terms of the conduct, quality, and uti-
dicators to measure progress toward the MDGs.           lization of M&E. And a diagnosis is invaluable in
                                                        providing the basis for preparing an action plan.
Why Conduct a Diagnosis?                                The action plan should be designed according to
This variety tells us that not only are the starting    the desired future uses of monitoring information
points faced by each country different, but so are      and evaluation findings. An early diagnosis can
the destinations to which they aspire. There is no      help inform judgments about the likelihood of
single best approach to a national or sector M&E        these intentions ever being achieved.
system.1 Instead, which approach a country should
use depends on the actual or intended uses of the       A diagnosis can be conducted—or commis-
information such a system will produce. As dis-         sioned—by government or donors, or it may be
cussed in chapter 3, those uses range from as-          desirable jointly. The process of conducting a di-
sisting resource-allocation decisions in the budget     agnosis provides an opportunity to get impor-
process, to helping prepare national and sector         tant stakeholders within government—particularly

                                                                                                                67
I V: H O W T O S T R E N G T H E N A G O V E R N M E N T M & E S Y S T E M




                 senior officials in the key ministries—to focus on          istries, which were largely opposed to the intro-
                 the issue of institutionalizing an M&E system. For          duction of a mandatory set of requirements for
                 most if not all developing countries, there will al-        conducting evaluations, it not only provided a
                 ready be a number of M&E activities and systems.            sound factual basis for the evaluation strategy
                 But a common challenge is a lack of coordination            but also sensitized these stakeholders to the im-
                 or harmonization between them, which can result             portance of the issue and fostered their somewhat
                 in significant duplication of effort. A diagnosis           grudging acceptance of the strategy itself.
                 that reveals such problems can provide a stimu-
                 lus to the government to address these prob-                A diagnosis also provides a baseline for measur-
                 lems, and by providing a shared understanding of            ing a country’s progress over time; it is a long-haul
                 the nature of the problems, it can also help fos-           effort to build and sustain both demand and sup-
                 ter a consensus on what is needed to overcome               ply for M&E. As noted in chapter 10, most coun-
                 them.                                                       tries have not developed their M&E systems in a
                                                                             linear, predictable manner; instead, they have de-
                 Such consensus is particularly important for M&E,           veloped them opportunistically, depending on
                 which is essentially a cross-cutting activity affect-       emerging opportunities and roadblocks, and as
                 ing all ministries and agencies. Any whole-of-              they develop their understanding concerning
                 government reform, such as the creation of a na-            which initiatives are or are not working well.
                 tional M&E system, requires substantial effort
                 and a high level of central coordination, as well           In this environment, it is important to regularly
                 as the active cooperation of sector ministries and          monitor and evaluate the M&E system itself—
                 agencies. Achieving real coordination among all             just as any area of public sector reform should be
                 these actors is typically not easy, so any process          regularly assessed. Indeed, conducting regular
                 such as preparation of an M&E diagnosis pro-                M&E efforts to strengthen an M&E system is one
                 vides one opportunity to get the key stakehold-             way those in charge of such efforts can lead by
                 ers to talk to each other about M&E and to attempt          example. Some aspects of an M&E system are
                 to reach some agreement on what to do about im-             amenable to regular monitoring, such as the num-
                 proving the government’s approach.                          ber of evaluations completed or the extent to
                                                                             which their recommendations are implemented.
                 This is illustrated by the experiences of Uganda,           Other aspects may require more in-depth evalu-
                 Mexico, and Australia. The finding that there were          ation from time to time, such as the extent of uti-
                 16 M&E subsystems existing in Uganda produced               lization of M&E information in budget decision
                 a response of concern—even outrage—among                    making or the quality of monitoring data.
                 senior officials. That response was instrumental
                 in prompting a decision to create NIMES to ad-              Thus a diagnosis is a type of evaluation and can iden-
                 dress the problems of harmonization and exces-              tify the degree of progress achieved and any nec-
                 sive demands on the suppliers of monitoring                 essary mid-course corrections. It is also noteworthy
                 information in sector ministries and agencies and           that national audit offices have played an important
                 at the facility level (see chapter 9). And the find-        role in reviewing the performance of M&E sys-
                 ing from a rapid diagnosis of M&E activities in Mex-        tems, through performance audits (see, for ex-
                 ico’s social development agency (SEDESOL), that             ample, Australian National Audit Office 1997; Mackay
                 there existed eight uncoordinated monitoring                2004; Office of the Auditor-General of Canada 2003;
                 systems within that one agency, also prompted the           GAO 2004) and in prompting their governments to
                 senior management of the agency to take steps               make needed improvements—for example, in Aus-
                 to harmonize these systems (World Bank 2004c).              tralia, Canada, and the United States.

                 The process of conducting a diagnosis in Aus-               What Issues Should a Diagnosis
                 tralia in 1987 provided the basis for the govern-           Address?
                 ment’s evaluation strategy. Although this process           In the simplest sense, a diagnosis would map out
                 did not lead to a consensus among sector min-               what is working and what is not—the strengths
68
                                                                              1 2 : T H E I M P O R TA N C E O F C O U N T R Y D I A G N O S I S




  Box 12.1: Key Issues for a Diagnosis of a Government’s M&E System


  1. Genesis of the existing M&E system—Role of M&E advocates                      istence of any disconnect between the budget process and
     or champions; key events that created the priority for M&E in-                national planning; opportunities to strengthen the role of
     formation (for example, election of reform-oriented govern-                   M&E in the budget
     ment, fiscal crisis)
                                                                                 • Extent to which the M&E information commissioned by key
  2. The ministry or agency responsible for managing the M&E sys-                  stakeholders (for example, the finance ministry) is used by oth-
     tem and planning evaluations—Roles and responsibilities of                    ers, such as sector ministries; if not used, barriers to utiliza-
     the main parties to the M&E system, for example, finance min-                 tion; any solid evidence concerning the extent of utilization by
     istry, planning ministry, president’s office, sector ministries, the          different stakeholders (for example, a diagnostic review or a
     Parliament or Congress; possible existence of several, unco-                  survey); examples of major evaluations that have been highly
     ordinated M&E systems at the national and sector levels; im-                  influential with the government
     portance of federal/state/local issues to the M&E system                 5. Types of M&E tools emphasized in the M&E system: regular per-
  3. The public sector environment and whether it makes it easy or               formance indicators, rapid reviews or evaluations, performance
     difficult for managers to perform to high standards and to be held          audits, rigorous, in-depth impact evaluations; scale and cost of
     accountable for their performance—Incentives for the stake-                 each of these types of M&E; manner in which evaluation prior-
     holders to take M&E seriously, strength of demand for M&E                   ities are set—focused on problem programs, pilot programs, high-
     information. Are public sector reforms under way that might ben-            expenditure or -visibility programs, or on a systematic research
     efit from a stronger emphasis on the measurement of govern-                 agenda to answer questions about program effectiveness
     ment performance, such as a poverty-reduction strategy,                  6. Who is responsible for collecting performance information and
     performance budgeting, strengthening of policy analysis skills,             conducting evaluations (for example, ministries themselves or
     creation of a performance culture in the civil service, improve-            academia or consulting firms); any problems with data quality
     ments in service delivery such as customer service standards,               or reliability or with the quality of evaluations conducted;
     government decentralization, greater participation by civil so-             strengths and weaknesses of local supply of M&E; key capac-
     ciety, or an anticorruption strategy?                                       ity constraints and the government’s capacity-building priorities
  4. The main aspects of public sector management that the M&E                7. Extent of donor support for M&E in recent years; donor projects
     system supports strongly—(i) Budget decision making, (ii) na-               that support M&E at whole-of-government, sector, or agency
     tional or sector planning, (iii) program management, and (iv)               levels—Provision of technical assistance, other capacity build-
     accountability relationships (to the finance ministry, to the pres-         ing and funding for the conduct of major evaluations, such as
     ident’s office, to Parliament, to sector ministries, to civil society)      rigorous impact evaluations

     • Actual role of M&E information at the various stages of the            8. Conclusions: Overall strengths and weaknesses of the M&E
       budget process: such as policy advising and planning, budget              system; its sustainability, in terms of vulnerability to a change
       decision making, performance review and reporting; possi-                 in government, for example, how dependent it is on donor fund-
       ble disconnect between the M&E work of sector ministries                  ing or other support; current plans for future strengthening of
       and the use of such information in the budget process; ex-                the M&E system




and weaknesses of the M&E system—and the                            somewhat differing emphases. The issues include
reasons why.                                                        the following:

A diagnosis of M&E would be expected to map out                     • The genesis of the existing M&E system (as-
a number of key issues (box 12.1). These key is-                      suming some sort of system or systems al-
sues are relevant whether the focus of the diag-                      ready exist)
nosis is at the national level or at the level of an                • The system’s management and the roles and
individual sector ministry or agency, albeit with                     responsibilities of the key stakeholders
                                                                                                                                                   69
I V: H O W T O S T R E N G T H E N A G O V E R N M E N T M & E S Y S T E M




                 • The public sector environment and whether                 in coping with multiple, unharmonized donor
                   there are incentives to take M&E seriously                requirements for M&E. Donor pressure is often
                 • The current main uses of M&E information, es-             the primary driver of government efforts to
                   pecially the role of M&E information in the               strengthen M&E systems, and the strength of
                   budget process and the use of M&E informa-                country ownership of these efforts may not be
                   tion by sector ministries and agencies                    strong. A deeper discussion of diagnostic issues
                 • The types of M&E most frequently used                     is presented by Mackay (1998b).2
                 • Responsibilities for collecting performance in-
                   formation and conducting evaluations                      Depth of Diagnosis
                 • The extent of donor support for M&E                       A question that is often asked is how long it should
                 • Overall strengths and weaknesses of the M&E               take to conduct an M&E diagnosis. There is no sim-
                   system or systems.                                        ple answer to this question; it all depends on the
                                                                             purposes for which a diagnosis is intended, the
                 The purpose of a diagnosis is more than a factual           range of issues under investigation, and the avail-
                 stocktaking. It requires careful judgment con-              able time and budget. In some cases a week-long
                 cerning the presence or absence of the success              mission to a country has provided a sufficient
                 factors for building an M&E system, as discussed            starting point for a broad understanding of the key
                 in chapter 10. Thus it is important to understand           issues facing a government interested in strength-
                 the strength of the government’s demand for                 ening its M&E functions. At the other end of the
                 M&E information and whether there is an influ-              spectrum is a more formal, detailed, and in-depth
                 ential government champion for M&E.                         evaluation of a government evaluation system,
                                                                             such as the one the Chilean government com-
                 Conversely, it is important to know if there are bar-       missioned the World Bank to undertake (Rojas and
                 riers to building an M&E system, such as lack of            others 2005). The Chile evaluation involved a
                 genuine demand and ownership; lack of a mod-                team of seven people working for many months.
                 ern culture of evidence-based decision making and           It entailed several missions to Chile and involved
                 accountability (due, in some countries, to issues           interviewing large numbers of government officials
                 of ethics or corruption); lack of evaluation, ac-           in central and sector ministries, as well as detailed
                 counting, or auditing skills; or poor quality and           reviews of evaluation quality and evaluation uti-
                 credibility of financial and other performance in-          lization. Such in-depth diagnoses can cost as much
                 formation. This understanding naturally leads to            as several hundred thousand dollars.
                 the preparation of an action plan to strengthen ex-
                 isting M&E systems or to develop a new system               Other issues may need to be investigated in-depth,
                 entirely (discussed in chapter 13).                         such as the quality and credibility of monitoring
                                                                             information and of the sector information sys-
                 Although the preceding issues are largely generic           tems that provide this information.3 Another pos-
                 to all countries, it is necessary to adjust the focus       sible issue is the capacity of universities and other
                 according to the nature of the country. Thus                organizations that provide training in M&E; such
                 middle-income or upper middle-income coun-                  training is a common element of action plans to
                 tries might well possess a strong evaluation com-           help institutionalize M&E. A diagnostic guide for
                 munity, centered in universities and research               assessing training organizations is provided by
                 institutes. But the supply of evaluation expertise          Adrien (2003); an actual diagnosis for Ghana is also
                 would be much weaker in many of the poorest                 available using this guide (Adrien 2001).
                 countries—those that prepare poverty-reduction
                 strategies, for example (see chapter 9). Also,              In between the two extremes of a week-long mis-
                 poorer countries are likely to have a strong focus          sion and an in-depth evaluation are the Bank di-
                 on poverty-monitoring systems in particular and             agnoses conducted for Colombia and Uganda.
                 are likely to experience much greater difficulties          Each involved several missions for discussions



70
                                                                 1 2 : T H E I M P O R TA N C E O F C O U N T R Y D I A G N O S I S




with senior officials and representatives of civil so-   If a donor project to support the development of
ciety, document review (such as government pol-          an M&E system is also envisaged, then it would
icy statements, documents relating to relevant           be important to include in the diagnostic team
donor projects, and any previous country diag-           some donor staff with relevant experience. Close
noses—of M&E, public sector reform, or public            involvement of senior government officials in the
expenditure management issues, for example),             diagnosis is, of course, important—to tap into
and formal conference or seminar presentations           their knowledge and judgments about the current
on the government’s M&E systems.                         M&E system and the ways it could be strength-
                                                         ened and to ensure their acceptance of the diag-
The Colombia diagnosis involved a number of              nostic findings and recommendations.
Bank staff working closely with their government
counterparts over the course of several missions         Conclusions
to the country (Mackay and others 2007); this di-        A diagnosis of a country’s systems for M&E can
agnosis is included in annex B. Illustrative ToRs for    provide a solid understanding of their strengths
a possible future in-depth diagnosis of Colombia’s       and weaknesses. This is clearly important for de-
M&E system are presented in annex C.                     veloping an action plan that is appropriately tai-
                                                         lored to some vision of the future M&E system,
                                                         particularly in terms of the desired uses of mon-
The Uganda diagnoses—in 2001, with a follow-up
                                                         itoring information and evaluation findings, which
in 2003—were conducted by a Bank consultant
                                                         are specific to each country. Such a diagnosis
over the course of several missions, with the close
                                                         would also provide a baseline measure against
collaboration and support of government coun-
                                                         which future progress can be evaluated, and fur-
terparts (Hauge 2001, 2003). The 2001 Uganda di-
                                                         ther modifications can be made to the system as
agnosis is summarized in table 9.1.
                                                         opportunities emerge and setbacks or barriers
                                                         are encountered.
The scope of a sector diagnosis would be narrower
than one focused on the government as a whole,           The process of conducting a diagnosis provides
although many if not most of the key issues would        a vehicle for involving senior officials in the key
be the same. A sector diagnosis would need to            central and sector ministries, together with donor
focus, among other things, on the role of sector         staff, in considering the purposes, uses, and ar-
M&E activities in contributing to any whole-of-          chitecture of the government’s M&E system(s).
government M&E systems. It would, of course, be          Diagnoses can provide surprising findings about
possible to conduct a sector diagnosis as part of        a multiplicity of uncoordinated and duplicative sys-
a national diagnosis; this could be useful in un-        tems. Such findings can help foster consensus
derstanding the issues of institutionalizing M&E         on an action plan to strengthen the system.
at the sector level, or if the government is con-
sidering piloting new M&E initiatives in selected        Creating a consensus on roles and responsibilities
sector ministries.                                       under a whole-of-government M&E system may
                                                         not be easy, however. Sector ministries and agen-
Depending on the issues to be addressed in a di-         cies might prefer not to be subject to centrally de-
agnosis, it might well be necessary to assemble a        termined, formal requirements for M&E. And
team of experts with a range of backgrounds. A           central ministries themselves might jostle for con-
team might therefore include individuals with            trol of a new or rejuvenated M&E system.4 A more
expertise in some or all of the following: the man-      collaborative, less adversarial approach can help
agement of a government M&E system; per-                 reduce these differences.
formance indicators and systems; statistical
systems; evaluation; public sector management re-        A diagnosis should include a factual stocktaking
form; and performance budgeting.                         and careful judgments concerning the presence



                                                                                                                                71
I V: H O W T O S T R E N G T H E N A G O V E R N M E N T M & E S Y S T E M




                 or absence of the various success factors for               vestigation of the current extent of utilization of
                 building an M&E system, such as a committed,                M&E information.
                 influential champion for M&E. A rapid diagnosis
                 can provide an overview of a number of key                  Most diagnoses are neither very rapid nor very
                 issues but could not be expected to be either               time consuming or in-depth; they fall between
                 complete or balanced. A detailed, in-depth di-              these two extremes. Nevertheless, a sound diag-
                 agnosis would include drill-downs into specific is-         nosis does require considerable care; the ex-
                 sues considered important for the country, such             pertise and quality of judgment of those who
                 as the quality of data systems or a detailed in-            prepare the diagnosis is crucial.




72
                                                                                               13
Preparing Action Plans



A
      central theme of this volume is that there is no “best” model of what
      a government or sectoral M&E system should look like. Countries dif-
      fer substantially in the emphasis they choose to give to the different
purposes of an M&E system—such as to support budget decision making or
ongoing management of activities, or for accountability purposes.

And each country is unique in terms of its M&E            ment; to support ministries and agencies in
activities, functions, and systems and in terms of        managing their activities; or to strengthen ac-
its public sector culture and environment. Thus,          countability relationships (chapter 3).
an action plan to create or strengthen an existing
government system for M&E has to be tailored           2. The formal roles and responsibilities of the
closely to country circumstances. This is why it is       key stakeholders of the M&E system—those
so important to conduct a diagnosis of a country’s        who would be expected to produce monitor-
M&E strengths and weaknesses as a basis for de-           ing information and evaluations and to make
veloping an action plan (chapter 12).                     use of them. It is important to locate respon-
                                                          sibility for management or oversight in a pow-
Vision for the Future System                              erful ministry, committee, or other entity.
A diagnosis might ideally lead to a clear state-
ment of what the M&E system would desirably            3. Whether a whole-of-government system, in-
look like at some time in the future. The dimen-          cluding all central ministries, sector ministries,
sions of the system—its architecture—would in-            and agencies, is to be developed or if the sys-
clude, in particular, the following elements:             tem is to be more narrowly focused on indi-
                                                          vidual sectors or agencies. Some governments
1. Which of the four possible uses of M&E infor-          develop an M&E system only for donor-funded
   mation will be the objectives of the system, rec-      operations in their country.1
   ognizing the trade-offs between them: to
   support budget decision making or national          4. The levels of government at which the M&E sys-
   planning; to help ministries in policy formu-          tem will be developed: central, state/provincial,
   lation, policy analysis, and program develop-          or local.




                                                                                                               73
I V: H O W T O S T R E N G T H E N A G O V E R N M E N T M & E S Y S T E M




                 5. The range of M&E tools on which the system                 M&E in pilot ministries or agencies, for exam-
                    will focus: performance indicators, rapid eval-            ple, where there already exists a strong com-
                    uations, rigorous impact evaluations, and so               mitment to conducting and using M&E and
                    forth. It is possible to create a reliable per-            where there is already some successful track
                    formance monitoring system without con-                    record. Pilots could be developed at other lev-
                    ducting any evaluations; however, conducting               els, such as in individual states or municipali-
                    evaluations requires good performance infor-               ties. Pilots such as rapid evaluations or rigorous
                    mation, either from a monitoring system or                 impact evaluations could also be envisaged
                    from special surveys, or both.                             for particular evaluation tools if there is little
                                                                               or no experience with them in the country.
                 Implementation Issues                                       • When and how quickly mainstreaming should
                 Once the vision of the fully functioning M&E                  occur. In Chile, for example, particular M&E
                 system has been formulated, it is possible to                 tools were mainstreamed in a sequential man-
                 develop an action plan to achieve it. This action             ner: performance indicators in 1994, rapid re-
                 plan should draw on the international lessons                 views in 1996, rigorous impact evaluations in
                 from building country M&E systems (chapter 10).               2001, and comprehensive spending reviews in
                 Of course, it would not make sense to develop an              2002 (chapter 6). Mexico is planning to main-
                 action plan unless there is already some sub-                 stream its new, whole-of-government M&E
                 stantive demand for M&E within the government.                system over a three-year period (2007–09)
                                                                               (Hernandez 2007).
                 An action plan can also be designed to strengthen
                 demand, for example, by ensuring that all key               The speed of implementation of an action plan is
                 stakeholders play some substantive part in plan-            clearly very important. There is an issue here of
                 ning or managing the M&E system, such as                    “digestibility”—how much M&E change or re-
                 through their involvement in an oversight or plan-          form ministries and agencies can absorb, and
                 ning committee. Particular incentives might need            how quickly. We have learned from countries that
                 to be created to further strengthen demand; these           have successfully created an M&E system that it
                 incentives could involve a mixture of carrots, sticks,      is a long-haul effort requiring patience and per-
                 and sermons (chapter 11). In other words, it would          sistence (chapter 10); M&E champions in gov-
                 be a mistake to create an action plan that focused          ernment, in contrast, tend to be impatient when
                 purely on technocratic, supply-side issues.                 reforming their M&E systems. Another lesson is
                                                                             to institutionalize an M&E system as rapidly as pos-
                 Strengthening an existing M&E system or building            sible before the champion(s) eventually depart.
                 an entirely new one can be approached in many
                 ways. Implementation issues that need to be ad-             As emphasized in chapter 10, it is helpful to reg-
                 dressed in an action plan include the following:            ularly monitor and evaluate the M&E system itself,
                                                                             to identify how successfully its various compo-
                 • The extent to which there will be a focus on              nents are being implemented. This provides a
                   strengthening what already exists: improving              sound basis for any needed changes to the nature,
                   financial management information systems                  scale, and timing of implementation of the action
                   (to track budget appropriations and actual                plan. And this is obviously important if a pilot
                   spending) or improving ministries’ monitoring             approach has been adopted.
                   systems, especially administrative data on gov-
                   ernment activities, beneficiaries, and outputs            Action plans may focus on some or all of the fol-
                   (note that these data are typically collected at          lowing stakeholders: government ministries and
                   the facility level).                                      officials, such as central and sectoral ministries
                 • Whether new approaches should be piloted,                 and agencies; subnational governments at the
                   with a view to their subsequent mainstream-               state, province, or district levels; the Parliament or
                   ing if successful. These could include the im-            Congress, including elected representatives, com-
                   plementation of new, formal requirements for              mittees, and their staff; and civil society groups, in-
74
                                                                                      1 3 : P R E PA R I N G A C T I O N P L A N S




cluding universities, research institutes, and NGOs.   ownership had a cost: up to one-third of the eval-
Commonly adopted actions could include, for ex-        uations suffered from some kind of method-
ample, providing M&E training; promulgating any        ological weakness.2 This is indicative of another
necessary laws, decrees, and regulations; prepar-      trade-off: internally conducted evaluations may
ing M&E guidelines and standards; strengthen-          have high ownership, but external evaluations
ing and harmonizing monitoring systems; and            may be more likely to be rigorous and objective.
conducting a range of types of evaluation, such as
rapid reviews and rigorous impact evaluations.         A simple results chain for building an M&E system
                                                       is shown in figure 13.1. This provides a simplified
Trade-Offs                                             representation of how an action plan would be ex-
There are many issues and trade-offs to consider       pected to lead to various kinds of output, such as
when developing an action plan. One is that a          the number of officials trained in M&E, the num-
whole-of-government M&E system that is managed         ber of evaluations conducted, and so on. These
by the finance ministry for purposes of perform-       outputs in turn would be expected to lead to in-
ance budgeting would need to ensure broad cov-         termediate outcomes such as strengthened gov-
erage of all programs. One way to achieve breadth      ernment demand for M&E, and to final outcomes,
of coverage is to rely on performance indicators;      including the utilization of monitoring informa-
their drawback, however, is that they usually pro-     tion and evaluation findings by government and
vide little information on the reasons results have    others. It would be hoped that these outcomes
been achieved or not—in other words, the causes        would help lead to final impacts, including im-
of good or bad performance.                            proved government performance, improved de-
                                                       velopment effectiveness, improved service
Rapid reviews can provide some insights into           provision, and poverty reduction.
causality, and as they are cheaper than some other
evaluation methods, they can also provide rel-         Menu of Actions to Improve M&E
atively broader coverage of government pro-            Table 13.1 presents a menu of possible actions to
grams. The most reliable evaluation findings can       achieve improvements in the demand for and
come from a rigorous impact evaluation, although       supply of M&E. This menu is far from compre-
these can be very expensive to conduct, so it is       hensive. Rather, it illustrates the range and nature
harder for a finance ministry to use them to eval-     of actions that can be taken to build or to
uate a broad range of programs (chapter 2). There      strengthen systems for monitoring and evaluation.
is often a trade-off between choice of M&E tools       One example of an action plan, which follows di-
and the desired uses of M&E information.               rectly from a diagnosis of M&E in a country, is
                                                       shown in box 9.2 for Uganda.
Another trade-off is between who commissions
evaluations and who is meant to use them. Chile’s      Another example is the World Bank project de-
finance ministry commissions a range of types of       veloped to help the government of Colombia
evaluation and uses them effectively for its own       strengthen its national M&E system, SINERGIA
purposes; but sector ministries and agencies are       (chapter 7; annex B). This $14 million project has
highly resistant to using these evaluations for        four components: (1) support better monitoring
their own, internal purposes—the entities do not       of government spending at the regional and local
“own” the evaluation findings (chapter 6). It can      levels; (2) consolidate the institutionalization of SIN-
be difficult for a centrally imposed M&E system        ERGIA within the government; (3) support the
to be accepted by sector ministries.                   development of regional and local M&E systems,
                                                       partly through a pilot approach; and (4) establish
Australia’s finance department endeavored to ad-       mechanisms to improve the quality, relevance,
dress this by leaving to sector ministries the prime   and cost-effectiveness of public information. This
responsibility of planning and conducting evalu-       project is based on the Bank’s diagnosis of SIN-
ations—this ensured they had a high ownership          ERGIA (annex B). This project also includes a large
of evaluation findings (chapter 8). However, this                                (Text continues on page 80)

                                                                                                                               75
I V: H O W T O S T R E N G T H E N A G O V E R N M E N T M & E S Y S T E M




  Figure 13.1: A Results Chain for Building a Government M&E System



                 Improved service provision, economic growth, and poverty reduction




                                                       ➩
                                         Improved development effectiveness
                                                                                                                          Impact




                                                       ➩
                                       Improvements in government performance


                                                       ➩
  M&E used for: government decision making on policies, plans, and budget resource allocation; implementation and
  management of government activities; monitoring of activities, accounting of expenditures, evaluation of programs
  and projects; government analysis and policy review; government accountability.
                                                       ➔


                                                                                                                           Final
                                                                                                                         outcomes
                       Government                                     Parliament                 Civil society

 M&E information directly         M&E information directly         Parliament assesses      Civil society assesses
 supports budget balancing,       supports ongoing                 and debates              government performance
 national planning, and           management of government         government               and inputs freely to
 policy formulation.              activities.                      performance.             policy debates.
                                                       ➔




  Formal M&E framework or system is established by government, leading to the systematic planning, conduct,
  reporting, and use of monitoring information and evaluation findings.
                                                       ➔




                                                                                                                         Intermediate
                                                                                                                           outcomes
 Strengthened         Strengthened         Strengthened        Strengthened demand           Strengthened demand
 demand for           supply of M&E        government          for M&E in Parliament;        for M&E in civil society;
 M&E in               and M&E skills       M&E systems.        strengthened M&E              strengthened supply
 government.          in government.                           skills of parliamentary       of M&E and skills
                                                               staff.                        in civil society.
                                                       ➔




  The action plan leads to the production of a range of outputs, such as number of officials trained in M&E;
  harmonized data systems; improved quality and focus of available monitoring indicators; improved quality and            Outputs
  range of evaluations completed; strengthened incentives for ministries to conduct and use M&E.
                                                       ➔




  A package of activities to strengthen country M&E functions is undertaken by the government and donors, such as
  national seminars on country M&E systems; diagnoses of national/sectoral M&E functions; audits of data systems;
  provision of M&E training—including trainer training—or scholarships to officials, NGOs, universities/research
                                                                                                                         Activities
  institutes, parliamentary staff; various types of evaluation are conducted on pilot/demonstration basis.



76
     Table 13.1: Possible Actions to Strengthen M&E Systems at the Country/Sectoral Levels


                                                                                                                                                           Strengthen supply and
           Strengthen government          Strengthen supply and develop                                          Strengthen demand for M&E in          development M&E skills in civil
              demand for M&E              M&E skills within government         Strengthen M&E systems               civil society, Parliament            society, Parliamentary staff

     Provide carrots, sticks, and
     sermons to strengthen demand
     (see table 11.1).

     Donors advocate M&E and focus                                                                               Donors advocate M&E and focus on
     on results. Disseminate examples                                                                            results. Disseminate examples of
     of influential evaluations.                                                                                 influential evaluations.

     Donors preserve and disseminate                                        Donors preserve and disseminate      Donors preserve and disseminate
     experience of other countries with                                     experience of other countries with   experience of other countries with
     M&E systems.                                                           M&E systems.                         M&E systems.

     Hold national brainstorming                                            Coordinate/harmonize donor           Hold national brainstorming
     seminars on options for                                                country M&E.                         seminars on options for
     government M&E systems.                                                                                     government M&E systems.

     Hold seminars on donor evaluation                                                                           Hold seminars on donor evaluation
     findings (for example, World                                                                                findings (for example, World
     Bank’s) relevant to the country—                                                                            Bank’s) relevant to the country—
     demonstrate the usefulness of                                                                               demonstrate the usefulness of
     evaluation.                                                                                                 evaluation.

     Hold regional seminars on            Foster regional networks of M&E   Encourage close coordination with
     government M&E systems.              managers and practitioners—a      work of other donors to strengthen
                                          community of practice.            the government’s central and
                                                                            sectoral M&E systems.

     Conduct national/sectoral            Conduct national/sectoral data    Conduct national/sectoral            Conduct national/sectoral             Conduct diagnosis of organizations
     diagnosis of M&E functions—          audits. Conduct diagnosis of      diagnosis of M&E functions—          diagnosis of M&E functions—           that provide M&E training.
     highlight problems and               organizations that provide M&E    demand, supply, systems, options     highlight problems and
     opportunities.                       training.                         for action plans.                    opportunities.

     Embed M&E as component of            Embed M&E as component of         Donors give loans to support M&E     Conduct service delivery surveys of
     donor/government public sector       related public sector             systems.                             client satisfaction and perceptions
     management and public                management/public expenditure                                          of the quality of government
                                                                            Link efforts to strengthen M&E
     expenditure management work.         management work, for example,                                          surveys and publicize the results
                                                                            systems with other performance-
                                          on performance budgeting;                                              widely (for example, CRCs).
                                                                                                                                                                                                 1 3 : P R E PA R I N G A C T I O N P L A N S




77
                                                                            related government reforms.

                                                                                                                                                                (Table continues on next page)
78
     Table 13.1: Possible Actions to Strengthen M&E Systems at the Country/Sectoral Levels                                              (continued)


                                                                                                                                                               Strengthen supply and
          Strengthen government         Strengthen supply and develop                                                Strengthen demand for M&E in          development M&E skills in civil
             demand for M&E             M&E skills within government               Strengthen M&E systems               civil society, Parliament            society, Parliamentary staff

                                        results-based management and
                                        strategic planning; sectorwide
                                        approaches; sectoral reform
                                        programs; service delivery; financial
                                        management and accountability,
                                        including financial management
                                        information systems; statistical
                                        system capacity building (statistics
                                        offices, ministry management
                                        information systems, statistics
                                        production, including service
                                        delivery surveys); national audit
                                        office capacity building;
                                        government decentralization.

     Identify and support M&E                                                   Encourage stronger government
     champions. Promote collaboration                                           internal coordination of M&E,
                                                                                                                                                                                                  I V: H O W T O S T R E N G T H E N A G O V E R N M E N T M & E S Y S T E M




     among all key government                                                   including links between M&E,
     stakeholders in the M&E system—                                            budget, planning and ministry, and
     especially in planning and                                                 management info systems.
     oversight of the system.

     Seek to mandate M&E via            Disseminate evaluation standards        Seek to mandate M&E through          Seek to mandate M&E through           Disseminate evaluation standards
     government decisions, decrees,     and methods.                            government decisions, decrees,       government decisions, decrees,        and methods. Support research
     regulations, and laws.                                                     regulations, laws. Develop M&E       regulations, laws. Introduce          institutes, universities, NGOs to
                                                                                guidelines and manuals.              freedom-of-information legislation.   evaluate and review government
                                                                                                                                                           performance—for example, CRCs,
                                                                                                                                                           budget analysis.

                                        Provide training in a range of M&E                                           Encourage media to report on          Train parliamentarians and staff to
                                        tools, methods, and techniques to                                            government performance.               analyze government performance.
                                        officials—at national, state,
                                                                                                                                                           Provide a range of M&E training in a
                                        municipal levels.
                                                                                                                                                           range of M&E tools, methods, and
                                                                                                                                                           techniques to staff of universities,
                                                                                                                                                           NGOs, and so forth.
                                                         Provide M&E trainer training and                                                                                                      Provide M&E trainer training and
                                                         support for civil service colleges, for                                                                                               support for universities/research
                                                         example, via twinning arrange-                                                                                                        institutes, for example, via twinning
                                                         ments with developed country                                                                                                          arrangements.
                                                         counterparts.

                                                         Review ministries’ performance
                                                         indicators. Strengthen ministries’
                                                         management information systems.
                                                         Support service delivery surveys.

      Create a fund for evaluation.                      Create/staff M&E/statistics/policy                                                               Create evaluation fund—allow civil   Create evaluation fund—allow civil
                                                         analysis units.                                                                                  society access.                      society access.

      Promote donor funding of major                     Donors fund major evaluations/                                                                   Support community involvement in     Donors subcontract
      impact evaluations/reviews—to                      reviews—on “public good” grounds.                                                                participatory poverty assessments,   evaluations/reviews to
      demonstrate feasibility and utility.                                                                                                                and so on.                           universities/research institutes.

                                                         Allow greater donor reliance on                  Allow greater donor reliance on                 Encourage civil society and
                                                         national M&E expertise when                      government to undertake M&E of                  parliamentary scrutiny of
                                                         conducting evaluations.                          donor-funded projects and                       government evaluation reports.
                                                                                                          programs.

                                                         Support development of                                                                                                                Support development of national
                                                         government evaluation networks                                                                                                        evaluation associations.
                                                         and national evaluation
                                                         associations.

      Seek donor support for joint                       Donors support joint evaluations
      evaluations with governments and                   with governments and country-led
      for country-led evaluations.                       evaluations.

     Note: If actions in these five columns are similar or related, this can be seen by looking along each row. CRC = citizen report card; NGO = nongovernmental organization.
                                                                                                                                                                                                                                       1 3 : P R E PA R I N G A C T I O N P L A N S




79
I V: H O W T O S T R E N G T H E N A G O V E R N M E N T M & E S Y S T E M




                 number of planned diagnoses of more detailed,               By now it should be abundantly clear that it would
                 specific issues:                                            not make sense to attempt to apply a standardized,
                                                                             “cookie-cutter” approach to developing a gov-
                 • A review of existing institutional arrangements,          ernment M&E system. The action plan for build-
                   particularly the extent to which M&E infor-               ing a government M&E system needs to be tailored
                   mation from SINERGIA has been used to sup-                closely to each country’s individual circumstances
                   port national planning and budget decision                and to the government’s particular vision of the
                   making. This review will recommend increas-               future system. This vision would encompass the
                   ing the government’s level of utilization of              specific uses to which it intends to put monitor-
                   M&E information.                                          ing information and evaluation findings, whether
                 • A review of M&E structures, institutional                 that is to be a whole-of-government system or an
                   arrangements, and capabilities in two line min-           individual ministry or agency, the levels of gov-
                   istries. This will help establish M&E quality             ernment to which it will apply, the particular range
                   standards, which will be required of all min-             of M&E tools to be adopted, and so on.
                   istries and agencies.
                 • An assessment of the M&E capabilities of the              There are many possible dimensions to an M&E
                   unit in the DNP that manages SINERGIA and                 system, and there are many trade-offs to be con-
                   provides technical assistance to other min-               sidered carefully. An action plan for building an
                   istries and agencies.                                     M&E system does not have to be enormously
                 • A review of existing laws, decrees, and regula-           complex, although there is always the danger
                   tions relating to M&E.                                    that the desired system will be overdesigned and
                 • A review of the cost, quality, and cost-                  thus much harder to achieve; it is very much the
                   effectiveness of the various evaluation tools             case that less is more. A similar danger is that the
                   and techniques currently used in the SINERGIA             speed with which a system can be built will be
                   system.                                                   overestimated.
                 • A diagnosis of the capacities needed to estab-
                   lish PBB.                                                 A concrete action plan provides a focus for key
                 • A diagnosis of the quality and utilization of             stakeholders in the government and for donors.
                   performance information and of information                It also provides a yardstick against which actual
                   systems in two pilot municipalities.                      progress toward the vision of the future M&E
                 • An assessment of mechanisms for local account-            system can be gauged—by regular M&E of both
                   ability in some good practice municipalities.             the system and the action plan. This will not only
                 • An assessment of the methodology currently                facilitate identification of emerging opportuni-
                   used by the DNP to assess the performance of              ties; it will also enable any implementation diffi-
                   all 1,100 municipalities in Colombia.                     culties to be identified early and addressed. The
                 • A review of data quality and the extent of data           emergence of roadblocks and opportunities ex-
                   harmonization among the main monitoring                   plains the experience of many countries: because
                   systems in Colombia’s central government.                 of these challenges, government M&E systems are
                 • An assessment of the quality, relevance, and use          usually not developed in a linear, predictable
                   of the DNP’s main system for monitoring gov-              manner (chapter 10).
                   ernment performance.
                                                                             The international community is still accumulating
                 Conclusions                                                 experience on how best to strengthen government
                 A useful strategy for building a government sys-            M&E systems—which approaches and systems
                 tem for M&E is to start with a diagnosis of current         are most effective and in which types of country,
                 M&E activities within the government. At the                such as in middle-income countries or in poor
                 same time, a vision of what a well-functioning              countries with very weak capacities. This under-
                 system would look like should be developed.                 lines the importance of further building the body
                 These basic building blocks naturally lead to the           of evidence concerning how best to institution-
                 development of an action plan, including a phased           alize an M&E system. These and other frontier is-
                 approach to its implementation.                             sues are considered in Part V .
80

								
To top