Project Report of Mis System in Bank by hgz16414


More Info
                        GOOD PRACTICE NOTE

                        and Evaluation
                        for AKIS Projects
                        Framework and Options

                        Prepared by Gary Alex and Derek Byerlee
                        with inputs from the AKIS Thematic Team

                        December 2000

                        The World Bank
Work in progress        Rural Development Family
for public discussion   Agricultural Knowledge & Information Systems (AKIS)
Agricultural Knowledge & Information Systems is a thematic team focusing on agricultural
   extension, education, and research within the Rural Development Department of the
    Environmentally & Socially Sustainable Development Network of the World Bank.

Foreword                                                                    iv

Abbreviations                                                               v

Executive Summary                                                            1

  I. Monitoring and Evaluation for AKIS Projects                             2

  II. Challenges for M&E Systems in AKIS Programs                            4
            Problems Inherent in Technology Program M&E                      4
            M&E in AKIS Projects                                             5

 III. The Framework for M&E Systems                                          6
            The Logical Framework (Logframe)                                 6
            Impact Models for AKIS Sub-Systems                               7
            Impact Model for Agricultural Technology Funds                   9

 IV. AKIS Performance Indicators                                             9

  V. Data Sources and Collection                                            15
           Project/Program Data—Management Information Systems              15
           National Level Databases                                         17
           Special Studies                                                  17

 VI. M&E in the Bank Project Cycle                                          18

VII. Recommendations                                                        21
          A. Describe Details of M&E System in Project Appraisal Document   21
          B. Request M&E Data from the Early Stages of a Project            21
          C. Develop Permanent M&E Capacity for Implementing Institutions   22
          D. Establish A Comprehensive Set of Key Performance Indicators    22
          E. Establish Project Baseline Data                                22
          F. Avoid Large Surveys                                            22
          G. Use a Combination of Data Collection Methods                   22
          H. Ensure Utilization of M&E Data                                 23

VIII. References and Additional Readings                                    23

Annex A. Illustrative Indicators for AKIS Projects                          25

Annex B. Example of Routine Project Report Format:
   Key Indicators for PRODETAB (Brazil)                                     34


This AKIS Good Practice Note summarizes cur-              The Agricultural Knowledge and Informa-
rent thinking and good practice with monitor-         tion Systems (AKIS) Thematic Team includes
ing and evaluation of agricultural technology         World Bank staff working in or interested in
projects and programs. The importance of good         research, extension, and education programs.
monitoring and evaluation (M&E) is widely rec-        The team objective is to enhance the effective-
ognized, but establishing M&E systems for re-         ness of Bank support to agricultural knowledge
search and extension is difficult, and World          and information system development. This, in
Bank projects have lagged in introducing sound        turn, contributes to the Bank’s objectives of al-
M&E arrangements for projects and for national        leviating poverty, ensuring food security, and
institutions.                                         improving sustainable management of natural
     “AKIS Good Practice Notes” are designed          resources. The AKIS team emphasizes policy,
to disseminate views, experiences, and ideas          institutional, and management issues associated
that may assist World Bank Task Team Lead-            with agricultural research, extension, and edu-
ers, national counterparts from Borrower coun-        cation. Other thematic teams focus on technical
tries, and other partners to prepare and              issues. The team mission is to “promote the
implement projects to strengthen agricultural         development of sustainable and productive
research, extension, and education programs.          agricultural research, extension, and education
The Good Practice Notes contain valuable              systems in Bank client countries.”
information about lessons learned from inno-              This AKIS Good Practice Note was prepared
vative experiences in World Bank projects and         by Gary Alex and Derek Byerlee with input
elsewhere, and make this information readily          from Jacob Kampen, Dely Gapasin, Madhur
available for comment and use by project              Gautam, Matt McMahon, and other AKIS The-
teams.                                                matic Group members.

                                                                                  Marie-Hélène Collion
                                                                           Chair, AKIS Thematic Team


AKIS     Agricultural Knowledge and Information Systems
APL      Adaptable program lending (or adaptable program loan)
ATF      Agricultural technology fund
CAS      World Bank Country Assistance Strategy
CDB      Country database
ER       Expert review
ICR      Implementation Completion Report
INFORM   ISNAR’s Information Management System for Agricultural Research
IPR      Intellectual Property Rights
ISNAR    International Service for National Agricultural Research
KPI      Key performance indicator
LIL      Learning and Innovation Loan
M&E      Monitoring and evaluation
MIS      Management information system
OED      World Bank Operations Evaluation Department
PAD      World Bank Project Appraisal Document
PDB      Project database
SAR      World Bank Staff Appraisal Report
SS       Special study
TTL      Task team leader (for World Bank project)

                                  Executive Summary

            onitoring and evaluation (M&E) sys-            or issues to be addressed in project evaluations
           tems have been weak in World Bank               (the Mid-Term Review).
           Agricultural Knowledge and Infor-                    Other good practices for M&E of AKIS
mation System (AKIS) projects, both for the                projects and programs include:
projects themselves and for the AKIS programs
that they support. Increasingly scarce resources           •   Requiring M&E data from the early stages
for research and extension dictate a need to maxi-             of a project: The M&E framework should
mize efficiency in use of public investment in these           be established and data collection begun at
programs. Improving monitoring and evaluation                  the outset of a project or program. It is dif-
systems for agricultural research, extension, and              ficult to retrofit an M&E system without
education programs is a long-term process criti-               compromising quality, objectivity, and use-
cal to improving program management and                        fulness later, when data are needed for man-
sustainability, and to building effective institu-             agement or evaluation.
tions to promote economic growth, reduce pov-              •   Institutionalizing M&E capacity within
erty, and conserve environmental resources.                    implementing institutions: Sustainable
     Principles for effective M&E are the same                 AKIS programs and institutions must have
for Bank-financed projects and for ongoing re-                 M&E systems to improve program manage-
search, extension, and education programs                      ment and to demonstrate impacts that jus-
within AKIS institutions. M&E should be based                  tify program funding. Establishing M&E
on an impact hypothesis linking activities to                  capacity within the institution provides for
desired outcomes and impacts. This hypothesis                  most M&E needs for Bank projects and for
is reflected in the logical framework (logframe),              future needs of the institution.
as used in design of Bank projects.                        •   Establishing a comprehensive set of key per-
     To establish a sound basis for project M&E,               formance indicators: Project M&E should be
Project Appraisal Documents (PADs) must de-                    based on a set of hierarchically linked key
scribe M&E systems in adequate detail, address-                performance indicators for inputs-outputs-
ing questions of what information is to be                     outcomes, as reflected in the logframe. Tar-
collected, how (using what procedures), by                     gets for key indicators should be defined by
whom, when, where, and why (how it will be                     quantity-quality-time (TQQ). Generally a
used). The M&E plan should describe arrange-                   good M&E system requires a more exten-
ments for obtaining baseline or control data;                  sive set of indicators than are included in
assess capacity for carrying out M&E; define                   the logframe.
indicators and targets; identify investments               •   Establishing project baseline or control data:
needed to strengthen M&E capabilities of imple-                Baseline data are important but frequently
menting agencies; and identify key assumptions                 not available at project start-up. Some pro-

2     Monitoring and Evaluation for AKIS Projects: Framework and Options

    vision is needed early in the project to en-             key characteristics of client groups: AKIS
    sure the availability of baseline or control             M&E systems must be able to tell who is
    data for comparison of “with-project” and                benefiting from a program as well as what
    “without-project” scenarios.                             the benefits are, and projects must be able
•   Avoiding large surveys: Large-scale surveys              to document impacts on women, the poor,
    are time-consuming and costly and method-                and minority groups.
    ology variations between surveys often limit         •   Ensuring utilization of M&E data by
    comparability. Large sample surveys are of-              decisionmakers: M&E systems must in-
    ten of limited value for Bank projects, but may          clude strategies for use of data, including
    be more useful for national AKIS programs                reporting formats and schedules designed
    that have a longer time perspective.                     to be useful to program managers and fin-
•   Using a combination of data collection                   anciers.
    methods: M&E for AKIS projects generally
    must integrate data from various sources,                Keep it simple! This is the cardinal rule for
    including the project’s internal management          M&E systems. M&E arrangements must be
    information system, special studies (often           implementable and feasible within the financial
    carried out by independent institutions),            and human resources available. Often the more
    and national data sets.                              elaborate the system and data requirements, the
•   Disaggregating data whenever possible by             less likely it is that the system will be able to
    gender, income level, ethnicity, and other           deliver on its promises.

       I. Monitoring and Evaluation for AKIS Projects

This AKIS Note summarizes experience and                 vestments, and particularly research and exten-
current thinking on monitoring and evaluation            sion programs, are being questioned as to im-
systems for AKIS projects. The objective of the          pact on social indicators.
Note is to assist World Bank Task Team Lead-                 Lack of consistency in definitions complicates
ers (TTLs) and Borrowers to develop practical            understanding of monitoring and evaluation.1
and effective systems to monitor and evaluate            Monitoring may refer to: routine collection of
project performance and impacts. Monitoring              information, tracking implementation progress,
and evaluation (M&E) systems are required for            measuring efficiency, and questioning “whether
all Bank projects (Casley and Kumar 1987;                the project is doing things right.” Evaluation is
World Bank 1989), and are receiving increased            generally defined in terms of: analyzing infor-
attention now that all projects are being asked          mation, ex-post assessment of effectiveness and
to demonstrate results. Agricultural sector in-          impact, confirming project expectations, mea-

     1 The World Bank has defined monitoring as “the continuous assessment of project implementation in

relation to agreed schedules and of use of inputs, infrastructure, and services by project beneficiaries,” and
evaluation as “periodic assessment of the relevance, performance, efficiency, and impact (both expected and
unexpected) of the project in relation to stated objectives” (World Bank 1989).
                                                         Monitoring and Evaluation for AKIS Projects                3

suring impacts, and questioning “whether the             sary elements of research, extension, and agri-
project is doing the right things.” Monitoring           cultural education programs.
(in the sense of continuous collection of data on            M&E systems should aid in assessing two
a project) is required both for implementation           aspects of a project—performance and results.
and ex-post assessment, whereas evaluation (in           M&E of project performance essentially assesses
the sense of making judgments and decisions              project implementation efficiency, whereas
about a project) is also a continuous process            M&E of project impact assesses project effec-
during project implementation and after                  tiveness. Project performance (process) leads to
completion. In practice monitoring and evalu-            results (impacts). This distinction between per-
ation (M&E)are difficult to separate and M&E             formance and impact is not entirely clear as
is best defined as “a continuous process of col-         there are often intermediate impacts (or results)
lection, analysis, and use of data.”                     between project performance and impacts. De-
     Effective M&E systems are important to              spite its importance, few World Bank-financed
enable project management to know whether                AKIS projects have adequate M&E systems (see
implementation is going as planned, and to pro-          Box 1). M&E plans are sometimes prepared as
vide information needed to adjust implemen-              last-minute additions to satisfy project prepa-
tation plans to achieve desired results. M&E             ration requirements and many current M&E
systems are also of critical importance to pro-          systems contribute little to project management
vide evidence of project accomplishments and             and performance assessment.
to defend project funding requests. By support-              Project M&E systems must be comprehen-
ing implementation and budget requests, sound            sive, well planned, and adequately funded. This
M&E systems contribute substantially to en-              can require substantial funding, which reduces
hancing program sustainability, and are neces-           funds available for program implementation.

  Box 1.   Weaknesses in AKIS Project M&E Plans

  A desk review of 45 appraisal documents (SARs/         veloped on an ad hoc basis late in the project (for
  PADs) for AKIS projects financed from 1991 to 1999     example, at the Mid-Term Review), when decision-
  revealed that only 24 percent had adequate M&E         makers expect evidence of progress and program re-
  plans. Most appraisal documents included a sepa-       sults. In some cases, projects collect large amounts
  rate M&E section (78 percent), specified information   of data, but little of this is analyzed or used. The ba-
  to be collected (78 percent), and provided financing   sic problem is lack of demand for M&E data from
  to strengthen national program M&E systems (62         policymakers and managers (including World Bank
  percent). The number of M&E plans judged “ad-          management).
  equate” increased from 14 percent of 21 projects fi-        TTLs are almost unanimously concerned with
  nanced during 1991-93 to 33 percent of 24 projects     M&E, and most projects are actively working to es-
  during 1994-99. The quality of key performance in-     tablish M&E systems and local capacities. TTLs rec-
  dicators showed little change between these periods.   ognize that demands for evidence of program
  Even for projects financed from 1994 to 1999, only     impacts will not go away, and that research and ex-
  42 percent had well-defined and quantified indica-     tension programs must demonstrate a plausible link
  tors for outputs, and 17 percent for outcomes.         to impacts on poverty and other social objectives. A
       A key informant survey of TTLs for 18 AKIS        clear lesson from the survey is that projects must es-
  projects confirmed the weaknesses in M&E systems       tablish their M&E systems early, and develop strat-
  for research and extension projects. Most projects     egies to increase use of M&E data for program
  have input monitoring systems to track financial       management and constituency building.
  flows and physical progress; some track outputs, but
  few have well-defined outcome or impact indicators.    Source: Based on Report for AKIS Retreat 2000 by
  Typically, indicators and monitoring systems are de-   Madhur Gautam and Gary Alex.
4    Monitoring and Evaluation for AKIS Projects: Framework and Options

One rule of thumb used by some agencies is that      of a Bank-financed project and a national tech-
M&E costs should generally not exceed five           nology system are similar, they are not neces-
percent of program costs. There is always a ten-     sarily identical, and a distinction often must be
sion between M&E data needs and acceptable           recognized in M&E system design.
levels of funding. Compromise is necessary and           In practice, principles of M&E are the same
this AKIS Note aims to provide practical guide-      whether for a Bank-financed project, a national
lines to improve project M&E.                        institution, or a “program” financed within a
    M&E systems are essential to Bank projects,      national technology system (for example, an
because of the Bank’s fiduciary responsibilities.    extension program or a livestock research pro-
They are also needed as permanent elements           gram). In discussing M&E systems, this Note
within country technology programs, and AKIS         will use the terms “project” and “program” in-
projects with institution-building objectives        terchangeably, and, when referring to a specific
must address the issue of establishing perma-        situation, will specify applicability to “Bank
nent M&E systems. Although the M&E needs             project M&E” or “national program M&E.”

    II. Challenges for M&E Systems in AKIS Programs

Monitoring and evaluating project performance        serving environmental resources, promoting
and impact are difficult exercises for most de-      food security, increasing exports, increasing
velopment programs, but pose special problems        economic growth, and reducing social conflict.
in World Bank-financed AKIS projects.                It is difficult to measure some of these, and
                                                     trade-offs between objectives are often neces-
Problems Inherent in Technology Program              sary.
M&E                                                       Cause-and-effect attribution of impact is
                                                     problematic because poverty, economic growth,
Assessing technology program performance             and environment are affected by weather, poli-
and impact is inherently difficult. Research out-    tics, policies, institutional changes, and many
comes and technology adoption are by nature          other factors. As well, different agencies are
uncertain, and establishing a timetable for re-      involved in basic, strategic, applied, and adap-
sults is difficult. Technology development and       tive research, extension, and technology adop-
dissemination do not follow a linear path—the        tion. Program performance and impact depend
environment for innovation changes continu-          on coordination between technology institu-
ously with new research results or changes in        tions and input supply systems, rural credit,
institutional, market, or social conditions. Even    product marketing systems, infrastructure de-
with success, lag times for impact of agricul-       velopment, trade and macroeconomic policies,
tural research are long, with applied research       and other factors.
often requiring 10-20 years to produce full im-           Technology program funding is in decline,
pacts on productivity.                               and institutions must limit investment in M&E
    Technology programs increasingly target          even though collection of reliable data can be
the diverse objectives of reducing poverty, con-     costly. Furthermore, lack of reliable national
                                                      Challenges for M&E Systems in AKIS Programs           5

data is a serious obstacle in many countries. In        should therefore be flexible and able to assess
addition, lack of awareness and managerial              performance and impact of the institution as a
demand for M&E data is an overriding prob-              whole, of individual programs within the insti-
lem that inhibits development of M&E systems.           tution, or individual projects under a program.
As most countries lack capacity for M&E,                     In practice, Bank projects are usually able
decisionmakers have low expectations for M&E            to address most project M&E needs through an
information. This leads to neglect of systems for       M&E system established by the implementing
generating information, and a tolerance for poor        institution with overall responsibility for the
performance of the systems. Donors have often           Bank loan. Such a system is often an appropri-
been the only supporters of M&E, but as tech-           ate or necessary component of a Bank project,
nology systems mature, funding groups (min-             as the implementing institution is responsible
istries, donors, or the private sector) will            for public funding and must have an M&E sys-
demand more effective program M&E.                      tem adequate to manage these funds. This may
                                                        require a centralized M&E unit within the main
M&E in AKIS Projects                                    implementing institution (the Ministry of Ag-
                                                        riculture) collaborating with M&E units in other
AKIS projects face two apparent dilemmas in             co-implementing institutions (extension agen-
planning M&E systems. The first is whether to           cies), and in decentralized regions where project
develop an M&E system specific to the needs             activities take place or have influence. Institu-
of the Bank project, or to the broader needs of         tionalizing an M&E system within implement-
the national research, extension, or agricultural       ing institutions is key, as this will then meet Bank
education program. Both are needed and,                 needs, as well as provide a permanent M&E sys-
though similar, each has somewhat different             tem beyond the life of the Bank project. Effec-
requirements. Bank project M&E systems focus            tive M&E systems should encourage an M&E
specifically on project implementation and out-         mentality throughout the technology system.
comes, even though this is usually only one part             The second apparent dilemma relates to
of the national technology system. This is, how-        project objectives. AKIS projects typically seek
ever, “must-have” information for the TTL and           productivity impacts3 (often occurring over a
as a minimum Bank projects must establish ap-           long period of time), but these usually require
propriate M&E systems to cover the activities           institutional development to bring about the
under the project.                                      productivity impacts, and to provide a basis for
     National technology programs need M&E              continued innovation and sustainable impact on
systems that are comprehensive and not focused          productivity. Should project M&E focus on the
solely on Bank-funded activities. These systems         institutional development or on the productiv-
enable an institution to focus on results, learn        ity impacts? Institutional development without
from experience, and enhance institutional              productivity impact does not justify investment,
sustainability.2 National program M&E systems           but neither is an investment sound if it gener-
provide information needed to set priorities,           ates a quick productivity impact without the
plan program implementation, and defend bud-            institutional development necessary to sustain
gets, work plans, and strategies. The systems           it over time.

    2 Current strategies emphasizing institutional pluralism in technology systems result in additional com-

plications, because all activities are not under one institution and each institution may require its own M&E
    3 “Productivity impact” is defined broadly to include any impact on the agricultural production system

and rural development, including environmental, economic, and social welfare.
6     Monitoring and Evaluation for AKIS Projects: Framework and Options

    Most Bank-financed projects combine both            systems that capture data relevant to the prin-
institutional and productivity impacts, though          cipal project objectives.4 Adaptable Program
with the balance varying from project to project.       Lending (APL) offers an option for handling this
Bank-financed projects must recognize these             problem in some cases, by emphasizing insti-
two types of impact, define Project Develop-            tutional development objectives (and M&E) in
ment Objectives in light of individual country          earlier phased projects, and productivity im-
and program circumstances, and design M&E               pacts (and M&E) in later projects.

                 III. The Framework for M&E Systems

All projects are essentially an investment plan         Figure 1 Hypothesis of AKIS Project
based on an hypothesis of how project-financed          Impact Process
activities will lead to desired outcomes and
changes in social and economic conditions. This                    Agricultural Education and
hypothesis provides the basis for project moni-                    Institutional Development
toring and evaluation.                                                           ⇓
                                                                     Knowledge Generation
The Logical Framework (Logframe)                                                 ⇓
                                                                   Knowledge Dissemination
The sequential steps of expected cause-and-                                      ⇓
effect due to project activities constitute the                          Opinion Change
project logic (or the “impact model”). For AKIS                                  ⇓
projects, this can be represented in a multi-step                     Knowledge Change
causal hypothesis as illustrated in Figure 1.                                    ⇓
     For bureaucratic consistency in World Bank                          Practice Change
and most other project systems, this multi-step                                  ⇓
process by which investments lead to impacts                          Productivity Change
is compressed into a four-step format in the logi-                               ⇓
cal framework (logframe) (see Table 1). The                          Change in Social Goals
logframe represents the project hypothesis of
                                                         Note: This simplified diagram omits important feed-
activity-result relationships, and is the basis for
                                                         back loops that regulate and strengthen the technol-
project design and evaluation (World Bank                ogy generation and dissemination process.
1996). However, it may be useful when devel-

     4 It is in Bank-financed institutional development activities that M&E for Bank-financed projects may

differ most markedly from M&E for the national technology program. Some indicators (for example, level of
stakeholder participation in priority setting, share of private sector financing for services, and number of
collaborative research programs) may be a high priority for understanding impacts on institutional develop-
ment, but less important for the national system’s routine M&E programs.
                                                                    The Framework for M&E Systems         7

Table 1   The World Bank’s Logical Framework

                                  Key Performance              Monitoring &
    Narrative Summary                Indicators               Evaluation Plan      Critical Assumptions
 Sector-related goal                                                             (Goal to Bank Mission)
 Project development                                                             (Objective to Goal)
                              Inputs: (budget for each
 Project components/

oping a project M&E system to consider the full          complicated because impacts may be diverse
chain of causality, and not be constrained by            and difficult to measure.
the four-step format of the logframe.                         Research program performance and impact
    Logframe outputs and objectives should               assessment is, perhaps, even more challenging,
appropriately address equity issues relating to          due to the nature of research, the longer lag
gender, ethnicity, and poverty group. Key per-           times involved, and the more complex chains
formance indicators should disaggregate data             of cause and effect. Technology development
on the same basis to encourage equity-sensitive          draws on input from various stages of research
project implementation, monitoring, and evalu-           (basic, strategic, applied, adaptive), and from
ation. Guidelines for such indicators can be             producers’ own inputs from traditional knowl-
found in “Gender-sensitive Monitoring and                edge and technology adaptation. Impacts from
Evaluation for Rural Development Projects”               research accrue and technologies are proven
(World Bank 2000).                                       useful only later, following technology dissemi-
                                                         nation and adoption by farmers. However,
Impact Models for AKIS Sub-Systems                       methodologies for estimating productivity im-
                                                         pacts are well developed. ISNAR has done ex-
Extension programs are based on a project im-            tensive work on methods for M&E in research
pact hypothesis of: extension activities, increas-       organizations (Ballentyne and others 1993;
ing extension service contact with farmers,              ISNAR 2000).
changing farmers’ opinions, changing farmers’                 Agricultural education is even further re-
knowledge, increasing adoption of new tech-              moved from direct impacts on productivity.
nologies, changing productivity, and changing            Agricultural education investments lead to more
social indicators. Extension activities that tar-        (or better) trained individuals, who are em-
get particular crops or production systems may           ployed in positions in which their skills enable
have close, well-defined links to impacts on pro-        them to perform better. Their improved indi-
ductivity and social goals, but even in these            vidual performance leads to better performance
programs M&E is complicated by the substan-              of research, extension, and other programs, with
tial lag between extension activity and produc-          a consequent improvement in agricultural pro-
tivity impacts and by influence of external              ductivity. For agricultural education, project
factors. In extension programs that serve as gen-        performance M&E is fairly straightforward for
eral rural information systems, or respond to            training and institutional development, but edu-
diverse production and social problems with no           cation impacts on social goals are loosely linked,
defined commodity, program M&E is further                diffuse, and difficult to estimate.
8     Monitoring and Evaluation for AKIS Projects: Framework and Options

    For any AKIS program, an M&E system to                 •     be based on a logical framework of cause-and-
assess project progress at different levels of the               effect relations that conceptually relate pro-
logframe (project cause-and-effect hypothesis)                   grams to impacts on targeted social goals;
requires a range of monitoring tools, many al-             •     address both project performance and
ready in use and available to be applied on a                    impact;
systematic basis (Table 2).                                •     utilize different tools for monitoring differ-
    A comprehensive approach to AKIS pro-                        ent stages of the technology generation,
gram M&E should:                                                 adoption, and impact process;

Table 2      Methods for M&E for Assessment of AKIS Projects
      Program          Stage of        Monitoring                  Method of            Probable Frequency of
       Level           Activity         Indicators                 Assessment                 Assessment
    Budget          Funding         Funding                    Budget and ex          At initial project approval
                                                               ante impact            and then annually
                                                               analysis for initial
    Inputs          Procurement     Input quantities           Program                Annually
                                                               accounting and
                                                               financial audit
    Outputs         Extension       Contacts with              MIS data; case         Routinely, at least annually
                    execution       farmers; opinion           studies, surveys,
                                    change; and                or beneficiary
                                    knowledge change           assessments
                    Research        Knowledge                  MIS data; external     Variable: perhaps every 3-5
                    execution       generation;                reviews                years
                                    technologies               complemented by
                                    developed;                 ex ante impact
                                    recommendations            analysis and
                                    released; research         output metrics
                    Agricultural    Graduates and              MIS data;              Variable: annual to every
                    education       competence of              institutional          3-5 years
                    execution       graduates                  reports and case
                                                               studies or surveys
                                                               of graduates and
    Outcomes        Technology      Technology                 Special studies;       Variable: perhaps through
    (Project        adoption        adoption and initial       MIS data               infrequent studies, often
    Development                     impacts                                           requiring baseline data
    Objective)      Institutional   Institutional              Special studies;       Variable: perhaps through
                    development     productivity (i.e.,        MIS data               infrequent studies, often
                                    numbers of                                        requiring baseline data
                                    extension contacts
                                    or technologies
    Impacts         Impact          Social goals               National and           Regular collection of data
    (Goal)                                                     global data;           with impact expected only
                                                               modeling               over a number of years.

Source: Adapted from Alex (1998).
                                                                The Framework for M&E Systems                 9

•   relate project progress and results to           gram. A model of ATF impacts is illustrated in
    planned targets based on ex-ante projections     Figure 2.
    of impact;                                           Comprehensive assessment of performance
•   establish a management information system        and impacts requires measurement of impacts
    for regular reporting on inputs and outputs      “A”, “B”, “C”, and “D” in Figure 2, as well as
    and, when possible, on outcomes;                 measurement of the performance efficiency (in
•   utilize special studies for monitoring and       terms of inputs and outputs) of the ATF program
    evaluation of output quality and outcomes;       and its individual subprojects. In practice, mea-
•   utilize currently available information and      surement of impacts at “A” is relatively straight-
    review processes to keep costs down and          forward; “B” and “C” are much more difficult;
    remain sustainable; and                          and “D” is nearly impossible to measure.
•   include mechanisms to feed information
    back to decisionmakers.                          Figure 2 Impact Model of an Agricultural
                                                     Technology Fund
Impact Model for Agricultural
Technology Funds
Agricultural Technology Funds (ATFs) and                                 Grants Program
other projects that have multiple subprojects
illustrate another level of complexity in assess-                A                        B
ing performance and impact of technology sys-
tem programs. These projects have impacts both
on direct program participants (beneficiaries)           Direct Sub-                          Institutional
and spillover effects on indirect beneficiaries.       Project Impacts                          Change
(“Beneficiaries” presuming that the indirect ef-
fects are positive.) Spillovers on indirect ben-                          C            D
eficiaries are difficult to measure. In addition,
ATFs generally also promote institutional
change within the technology system, change                               Indirect Impacts
which itself leads to indirect impacts of the pro-

                    IV. AKIS Performance Indicators

Performance indicators (or “KPI”—key perfor-         impact indicators that measure outcomes
mance indicators) are the basis for project or       (changes related to the “Project Development
program M&E. Performance indicators derive           Objective”) and impacts on social goals. Track-
from logframes and provide for performance           ing input use and direct project accomplish-
measurement at two levels: process indicators        ments often requires multiple indicators, but
that measure program inputs and outputs, and         these are relatively easily measured. Outcome
10       Monitoring and Evaluation for AKIS Projects: Framework and Options

    Box 2    Sample Key Performance Indicators                Research institution development project:

    Examples of possible indicators for an extension          •   20 scientists trained to the M.Sc. level in year 3,
    project and an institutional development research             40 in year 4, and 15 in year 5. (Input level).
    project are listed below.                                 •   By the end of year 2, six provincial research
                                                                  centers have completed strategic plans based
    Production-oriented extension project:                        on sound social and economic analysis; by end
                                                                  of year 4, at least 12 collaborative research pro-
    •     35 farmer training programs are held each year          grams are being executed with collaborating
          for five years with each course including practi-       institutions and in line with strategic plans;
          cal demonstrations and having at least 50 par-          and by year 5, 80 percent of research programs
          ticipants (Input level).                                have at least 50 percent of trials carried out in
    •     Number of farmers trained and know how to               collaboration with small farmers (Output
          use organic fertilizers in conjunction with             level).
          chemical fertilizer: 10,000 by end of year three    •   By the end of year five, at least 12 research pro-
          and 20,000 by the end of year five (Output              grams are receiving at least 25 percent of their
          level).                                                 funding from the private sector, are rated satis-
    •     Number of farmers using organic fertilizers in          factory or better by external peer reviews, and
          conjunction with chemical fertilizer: 20,000 by         are producing technical recommendations with
          beginning of year three and 40,000 by the be-           demonstrated potential to increase small-farm
          ginning of year five (Outcome level).                   profitability (Outcome level).
    •     Within 10 years, net farm income in the region      •   Within 10 years, net farm incomes increase by
          has increased by an average of 20 percent (Goal         an average of 20 percent due to adoption of new
          level).                                                 technologies (Goal level).

and impact indicators are more difficult, as they             •   Significant and selective in that the moni-
are a result of project activities, but are influ-                toring system should limit the number of
enced by many other factors.                                      indicators to a manageable number;
     Project indicators will vary depending on                •   Practical in terms of the ability to collect
specific project strategies and objectives. Sound                 data in a timely and cost-effective basis and
indicators must be characterized by time, quan-                   with a reasonable degree of reliability;
tity, and quality (TQQ). They should specify                  •   Provide information on impact disaggre-
when change in the indicators is expected to be                   gated by gender, poverty status, ethnicity,
achieved, the extent of change expected, and the                  or other characteristic of client group;
quality standard for the expected change. Quali-              •   Acceptable to the Borrower, the Bank, and
tative indicators are acceptable, but must be                     other stakeholders;
specific enough to be unambiguous as to                       •   Quantifiable to the extent possible, or, if
whether or not they are achieved. General prin-                   qualitative, based on a common under-
ciples for selecting key performance indicators                   standing of what constitutes success; and
require that they be:                                         •   Responsive in reflecting changes due to
                                                                  project activities and sensitive in demon-
•       Relevant to the development objective of the              strating change in as short a time frame as
        project;                                                  possible.
•       Unambiguous as to definition, valid and spe-
        cific in reflecting changing conditions, and              Budget and Input Level. Technology insti-
        consistent and verifiable in measurement;             tutions need well established systems for bud-
•       Meaningful and of interest to project man-            geting, procurement, and audits. These systems
        agement and stakeholders;                             track funding allocation, input (works, goods,
                                                                             AKIS Performance Indicators         11

services) use, and expenditures by program and             economic analysis uses similar methodology to
project. Reporting systems should provide regu-            evaluate investment impacts (see Box 3).
lar information on cost and input allocation to
specific programs, projects, and cost centers. For              Output Level. AKIS projects typically focus
research programs, management software pack-               on two objectives: institutional development to
ages, such as ISNAR’s INFORM program                       strengthen the institutions and technology sys-
(ISNAR 1991), provide research cost data in de-            tem necessary to develop and disseminate im-
tail by project and program. These can be useful,          proved technology and management in the
but may be overly complex for many research in-            agricultural sector; and productivity change due
stitutions, so less formal methods of estimating           to technological innovation introduced through
expenditures by project may be fully acceptable.           the technology system. The two objectives are
Simple, “bare bones” systems are needed.                   related and mutually supportive, but require
      The planning process establishes budgets             different indicators for outputs and outcomes.
and input requirements as part of priority set-            An illustrative list of key performance indica-
ting. These cost projections (budgets) must be             tors is included in Annex A for the two broad
linked to expected outcomes and are an essen-              types of AKIS project objectives for research,
tial element for evaluation of any program, as             extension, and education.
benefits are assessed in relation to the program                Project outputs are direct products of project
cost. Accurate and timely budget and expendi-              activity and are within the control of the project.
ture data are important to any evaluation of               Project output level M&E combines relatively
program efficiency and effectiveness. Expendi-             simple quantitative measures of productivity
ture data are commonly used and helpful in                 with more complex assessments of the quality
tracking implementation. If used alone, however,           of the outputs produced. Data on outputs
it is an inappropriate and insufficient indicator of       should come from management information
project implementation progress and perfor-                systems (MISs), which provide routine report-
mance. Ex ante economic analysis relating project          ing against expected output targets established
costs and benefits is a standard means of evaluat-         during project planning. Reporting systems
ing proposed investments, and is therefore rel-            should be structured to provide this informa-
evant to evaluating investment budgets; ex post            tion on a regular basis to guide program man-

  Box 3   Economic Analysis of AKIS Projects

  Comprehensive economic analysis of AKIS projects         •    Using economic surplus, or modified economic
  that includes major institutional development com-            surplus, analyses for economic analysis of tech-
  ponents is generally not very meaningful or helpful           nology programs;
  in evaluating investments (Horstkotte-Wesseler and       • Using ex ante economic analyses to aid in prior-
  others 2000). In place of a comprehensive economic            ity setting and planning within the national tech-
  analysis, good practice in AKIS project preparation           nology system; and
  should include:                                          • Building economic analysis capability within
  • Evaluating research investments at the level of             national programs to provide a basis for on-
       the research program, without limitation as to           going economic analysis of technology pro-
       whether the source of financing is from Bank             grams.
       loans;                                                   Maredia, Byerlee, and Anderson (2000) review
  • Using cost effectiveness analyses for evaluating       procedures for ex post economic analysis of research
       some extension investments and most agricul-        programs. Their paper provides an overview of cur-
       tural education investments;                        rent practice and experience, summarizes and clari-
  • Using break-even analysis to evaluate extension        fies methodological issues, and identifies guidelines
       programs or projects proposed for Bank financing;   for good practice in impact evaluation.
12    Monitoring and Evaluation for AKIS Projects: Framework and Options

agement. Qualitative assessment of outputs re-          developing meaningful indicators, as research
quires special studies and independent data             activities may be completed as planned, but
collection, which should be built in as a regular       completion says little about the quality or sig-
part of program M&E.                                    nificance of the work.1 Resulting increases in
     For extension projects with institutional de-      knowledge may have no immediate effect out-
velopment objectives, output M&E is based               side of the laboratory, but be of great value at
on physical completion (for example, staff              some future time.
trained, extension agents posted) and qualita-              As with extension program M&E, monitor-
tive assessment of established capacities. For          ing research outputs requires a combination of
productivity-oriented projects, output M&E re-          output metrics (for example, experiments com-
lies on data from program reports, especially           pleted, papers published) and special studies
in terms of deliverables in extension system            (external peer reviews) to assess research pro-
contacts with farmers (for example, numbers of          gram quality. External peer reviews are useful
farmers trained, farmer questions answered,             to obtain technical experts’ assessment of re-
field day attendance, newspaper or magazine             search activities from three perspectives: a pro-
readership, radio campaign listeners). Misra            spective review (to assess implications and
(1997) describes approaches and indicators for          relevance of research); an in-process review (to
M&E of extension programs.                              assess efficiency of research and performance
     Quantitative measures of interaction with          against targets for outputs), and a retrospective
farmers then need to be supplemented by qual-           review (to assess scientific or technical quality
ity measures, such as subjective program assess-        of work completed).
ments or measures of the program’s ability to               Agricultural education projects with insti-
promote adoption of new technology (behav-              tutional development objectives rely on
ior change). Indicators of potential to influence       physical measurements of outputs (faculty
behavior (for example, attitudes towards exten-         trained, curricula developed) and related spe-
sion services, change in knowledge about tech-          cial studies to assess quality of the outputs.
nologies) are usually assessed through field            Developing capacity for assessing increase in
surveys, case studies, or beneficiary assess-           numbers of individuals trained is much easier
ments. Extension program M&E systems must               than for assessing capacity for improved qual-
assess the quality of both the message (the rel-        ity of training.
evance of the technology or advisory services)              Agricultural education programs with di-
and the medium (the effectiveness of the com-           rect training objectives have outputs that are
munications or delivery system). Questions con-         easily quantified in terms of numbers of gradu-
cerning the message are in a sense an assessment        ates or person-months of training. Quality as-
of research or technology generation, as well as        sessment is more difficult, and involves special
of the extension service’s ability to access rel-       studies in the form of surveys of change in com-
evant technologies.                                     petencies or knowledge of graduates. Data and
     For research programs, output indicators for       special studies evaluating market demand for
institutional development projects are also             training (rates of employment of graduates, sal-
based on physical completion (for example,              ary premiums commanded by graduates, or
laboratories constructed, staff trained) and qual-      willingness to pay for training) are possible tools
ity of established capacity. Productivity-ori-          for assessing the quality of agricultural educa-
ented projects present serious difficulties in          tion programs.

    1 Research impact depends mainly on the results of the “best” research and relatively little on the total

quantity of research.
                                                                        AKIS Performance Indicators      13

     Project Development Objective (Outcome)             behavior (adoption) and initial effects of adop-
Indicators. Outcome indicators reflect the ini-          tion. Examples of the former include rates of
tial results or intermediate impacts of a project.       adoption of new technology, changes in gov-
Outcomes are closely related to project activi-          ernment policy based on research findings, or
ties, but are not under the full control of the          new investment made possible because of new
project. The outcome indicators must measure             technology or management options. Initial ef-
achievement of the project objectives, whether           fects of behavior change may include yield in-
relating to institutional development or to pro-         creases, decreases in deforestation or erosion,
ductivity change.                                        production cost decreases, or (for effects of
     Institutional development outcomes vary con-        policy reforms) further change in technology
siderably depending on technology system                 adoption rates or investment.
needs. These may reflect changes in the institu-              Project outcomes cannot always be reported
tions themselves, institutional productivity, or         on a routine basis by a project management in-
institutional relations with clients. Changes in         formation system. For this reason, special stud-
institutions may be measured by numbers of               ies are usually necessary to measure outcomes.
functional facilities (for example, laboratories,        Baseline studies followed by regular longitudi-
rural communications centers), number of                 nal studies over the course of a project are use-
qualified staff, ratio of scientists to support staff,   ful to measure changes in a target region.
numbers of research or extension service pro-            National surveys may be an ideal, but are
viders, new institutional relationships, ability         generally too costly, require too much time to
to address new problems, peer reviews of tech-           produce useful data, and are not often recom-
nical programs, stakeholder attitudes, shift of          mended. AKIS projects should frequently build
funding (for example, to environmental pro-              routine field case studies into M&E systems for
grams or small farmer crops, to competitive              extension and research. In fact, it may be diffi-
funding, to on-farm research), and numbers of            cult to envision an efficient technology system
collaborative programs.                                  without a functional system for such studies to
     Institutional development indicators reflect-       assess how well a program is doing and how
ing extent and quality of service provision might        well technologies are being accepted. A key chal-
include publications per scientist, client contacts      lenge is that of relating “with-project” changes
per extension agent, technology releases per             to the counterfactual “without-project” scenario.
research program, quality rating of competitive               In addition to surveys or case studies to as-
grant research programs, and client satisfaction         sess project outcomes, other special studies
ratings for extension services. Financial perfor-        might be needed to track specific planned and
mance and sustainability indicators might in-            unplanned impacts or program operational is-
clude percentage of funding from private sector          sues (for example, technology access by women
or farmers, cost per farmer extension contact,           or minority groups, and changes in land tenure
cost per student graduated, and ratio of salary          and other social conditions in the area). Benefi-
to operating cost budgets.                               ciary and stakeholder participation in M&E is
     Productivity outcomes would require similar         critical to ensure that AKIS programs are re-
indicators for both extension and research.              sponding to genuine needs of intended clients.
Metrics may be fairly straightforward, as ini-           This can be accomplished by using survey tech-
tial impacts can be measured in the expected             niques or participatory evaluation.
“uptake pathways” in which effects of techni-                 One participatory M&E approach is benefi-
cal changes are relatively closely linked to and         ciary assessment, which assesses change in
attributable to research and extension outputs.          farmer opinions and knowledge, changes in be-
A wide range of possible indicators can reflect          havior and technology use, and changes in pro-
two results of program activities—change in              duction, incomes, and environmental conditions
14      Monitoring and Evaluation for AKIS Projects: Framework and Options

(see Box 4). Beneficiary assessment uses infor-            a plausible link between project activities and
mal surveys, structured direct observation, and            the social goal. Goal level indicators that are as-
focus group interviews to elicit information. In-          sociated with agricultural technology and ap-
formant selection is a critical issue as selection         propriate to AKIS projects might include:
is rarely random and poor sampling can intro-              increased factor productivity; social rate of re-
duce significant bias. Beneficiary assessment              turn to investment; positive trends in environ-
can be useful, but it can be difficult so consulta-        mental indicators; and reduction in poverty
tion with experienced practitioners is recom-              rates and improved nutrition.
mended.                                                         Sector goal-level impact indicators are simi-
                                                           lar for research, extension, and agricultural edu-
     Sector Goal Indicators. Sector goal indica-           cation. These are affected by many factors other
tors measure change in the broad development               than AKIS programs, and indicators usually
goal to which the project contributes. Changes             show response only with a considerable lag
in these indicators are indirectly attributable to         time. Ex post impact analyses are important to
the project, and usually cannot be measured                demonstrate past success and guide future in-
during the life of a Bank-financed project. Al-            vestments (see Box 4). The trick with impact as-
though demonstrating direct project impact on              sessment is that of comparing the “with project”
a goal level indicator may not be within the               situation with the counterfactual “without
“manageable interests” of a project, it is impor-          project” scenario.1 Despite the difficulties with
tant that the project provide a plausible hypoth-          attribution and detecting change, AKIS projects
esis of how project activities will lead to change         should link activities to impacts on indicators
at the level of the sector goal. This often requires       of social goals as a basis for justifying invest-
other sources of research and analysis to build            ments.

     Box 4   Beneficiary Assessment—Getting the Client’s Views

  Beneficiary assessment involves a process of infor-      ciary assessments varied widely, but that the ap-
  mation gathering to assess the value of an activity      proach was considered quite effective as a tool for
  as perceived by its intended beneficiaries. For AKIS     M&E. Farmers in direct contact with extension
  programs, it involves structured conversational in-      agents benefited substantially, with resulting im-
  terviews with farmers, extension agents, extension       pacts on productivity. Farmer-to-farmer technol-
  managers, agribusiness personnel, and researchers.       ogy transfer and service provision to women
  Extension program beneficiary assessment is based        farmers was generally ineffective. Farmers’ major
  on interviews with a fairly large sample of farmers,     criticism of extension was the narrow focus of ser-
  stratified as needed by region, gender, ethnicity,       vices. Extension managers found beneficiary as-
  exposure to extension services, area coverage by         sessments to be a positive exercise, effective in
  extension programs, or other factors. The interview      introducing policy changes, and resulting in more
  stresses listening. The purpose of the beneficiary       participatory approaches, including better links to
  assessment is to influence program policy and pro-       existing and local organizations and greater re-
  gram management.                                         sponsiveness to farmer needs.
       A review of experience in ten African exten-
  sion projects found that the structure for benefi-       Source: Salmen (2000).

    1 Various methods may be used to establish controls for impact assessment: randomized controls with

beneficiaries randomly placed into two groups; constructed controls with participants paired with non-
participants; statistical controls comparing groups after correcting for other characteristics; reflexive controls
comparing before and after; generic controls comparing against norms for change; and shadow controls com-
paring against estimates of normally expected change.
                                                                      Data Sources and Collection   15

    Models may be useful in relating AKIS             and Anderson 2000). Development and veri-
programs to changes in social goals and in es-        fication of such models are important tools for
timating impacts that are not otherwise               assessing various influences on agricultural
readily measurable (Wood and Pardey 1997).            production processes, and may be appropri-
Methodologies for use of such models are still        ate for financing under AKIS projects. Such
under development, and more work may be               modeling is likely to be more appropriate and
needed to develop practical models that               useful as a research activity within a country
bridge the gap between measurable outcomes            research program, and less practical for esti-
(adoption and effects of adoption) and im-            mating impacts for a Bank-financed project
pacts on social indicators (Maredia, Byerlee,         M&E system.

                      V. Data Sources and Collection

Research, extension, and agricultural education       Project/Program Data—Management
programs (and bank-financed AKIS projects) gen-       Information Systems
erally require M&E units to carry out required
data collection, analysis, and reporting functions.   Project input-output data are generally avail-
Murphy and Marchant (1988) discuss issues in          able from a project management information
planning M&E systems for extension agencies,          system (MIS), the project’s internal system for
and Casley and Kumar (1987) discuss issues for        collection, analysis, and dissemination of
agricultural sector programs in general. AKIS         project information. Lecuit and others (1999)
project M&E systems must balance data needs           provide guidelines for MISs for social funds,
between the ideal (and costly) and the practical,     with principles and terms of reference that
and will generally need to draw on data from          may be useful to AKIS projects, especially
various sources (see Table 3), including especially   those with competitive grants programs.
project (or program) management information           Babu, Singh, and Sachdeva (1997) describe
systems, special studies, and national data sets.     management information systems for exten-
A Uganda nutrition project is cited as a good         sion programs. For Bank-financed projects,
example of an M&E system that effectively com-        the MIS should be described in the Project
bined several methods of data collection: a Man-      Appraisal Document (PAD) as the project’s
agement Information System to monitor inputs          internal data management system. This re-
and outputs at the community level and feed           quires special attention to: ensuring adequate
information back to the center; surveys using         staffing; designating responsibilities for com-
randomized experimental design to determine           piling, checking, and follow-up on reporting;
baselines and impacts of specific programs; and       quantifying expected inputs and outputs; es-
an ongoing participatory evaluation process to        tablishing indicators or milestones for each
verify information and impacts and to identify        subactivity; and establishing procedures for
needed management adjustments (Garcia, Al-            regular flow of information between decen-
derman, and Rudqvist 1999).                           tralized implementation units and a central
16    Monitoring and Evaluation for AKIS Projects: Framework and Options

Table 3    Illustrative Indicators Available from Various Sources

      Indicator Data Available from Project Management Information Systems:
      ♦ Inputs used: goods procured, buildings                   ♦ Percent of funding for environmental
           constructed, people trained.                              programs
      ♦ Percent of financing from different sources              ♦ Number of women extension clients
      ♦ Percent of funding through competitive grants            ♦ Percent of radio broadcast time in minority
      ♦ Technology recommendation releases                       ♦ Research papers published
      ♦ Research plans/strategies adopted                        ♦ Farmers trained
      ♦ Extension service responses to farmer requests           ♦ Audience for mass media
           for information                                           programsnewspaper, magazine, radio,
      ♦ Farmer field day/demonstration attendance                ♦ Number of graduates
      ♦ Research trials completed                                ♦ Number of extension agents
      ♦ Number of demonstrations                                 ♦ Number of publications
      ♦ Number of students                                       ♦ Number of instructors
      ♦ Number of participating institutions                     ♦ Percent of trainees from minority groups
      Indicator Data Available from Surveys and Special Studies:
      ♦ Yield/productivity impact from farmer                    ♦ Cost of production or change in farm
           adoption of specific technical, managerial, or            income due to adoption of specific
           organizational innovations                                technical, managerial, or organizational
      ♦ Benefit incidence for women, minority groups,            ♦ Employment, environmental and other
           and poor from adoption of technical,                      impacts due to adoption of specific
           managerial, or organizational innovations                 technical, managerial, or organizational
      ♦ Farmer knowledge of specific technical,                  ♦ Farmer adoption of specific technical,
           managerial, or organizational innovations                 managerial, or organizational innovations
      ♦ Scientific/technical quality of program                  ♦ Employment or skill level of graduates
           execution (especially for research programs)              from training programs
      ♦ Farmer satisfaction with extension services              ♦ Private sector investment in research or
           and research-generated technologies                       extension
      Indicator Data Available from National Data Sources:
      ♦ Rural income levels                                      ♦ Public expenditures on agricultural
      ♦ Agricultural GDP                                         ♦ Average yield of target crops
      ♦ Value of agricultural exports                            ♦ Average production per animal unit
      ♦ Public expenditures on agricultural extension            ♦ Index based on crop yields and production
                                                                     per animal unit for major agricultural
      ♦ Public expenditures on agricultural education            ♦ Institutional budget allocations for salaries
                                                                     or operating costs as a % of total budget
Note: All indicators disagregated by sex, where possible.

M&E unit. Project management information                    tice for routine reporting on indicators from
systems should provide gender-disaggregated                 the PRODETAB project in Brazil. Such data
data wherever possible.                                     should be available to TTLs for each supervi-
    Annex B provides an example of good prac-               sion mission.
                                                                       Data Sources and Collection    17

National Level Databases                               attitudes, knowledge, and technology adoption;
                                                       research programs need to know results of new
As it is generally not cost effective for a project    technologies; and training institutions need to
to measure national change in goal-level indi-         know how graduates perform. In some cases, it
cators, information needs at this level rely on        may be best for monitoring to be done by an
national databases. Quality of national data may       independent institution to avoid overburden-
be suspect due to poor sampling procedures or          ing the implementing institution, or to avoid
intentional distortions, but it is readily avail-      bias in monitoring change within the institution.
able and comprehensive, represents consider-           The alternative is to finance special studies
able investment in data collection, and is official,   through a project office independent of the
government-sanctioned data. Changes in na-             implementing agency. No matter how special
tional-level indicators are usually gradual and        studies are to be carried out, there is a danger
not observable over a typical five-year Bank-          of them being neglected during project start-up,
financed project. Nevertheless, goal-level indi-       as studies are relatively low cost, management
cators should be tracked to focus research and         intensive, and easily put off until data are
extension staff on the ultimate objectives of the      needed for a Mid-Term Review or other critical
funding their programs receive and to provide          project evaluation. The major types of special
data for future impact assessments. Even               studies utilized in AKIS program M&E are:
though current programs may not yet be gen-                 Sample surveys. Formal sample surveys are
erating impacts, past research and extension           useful to provide comprehensive and statisti-
programs should provide a basis for ex post            cally valid data on program outcomes and im-
impact assessments. Since agricultural statistics      pacts. However, formal sample surveys also
are subject to wide year-to-year fluctuations due      have disadvantages in that many institutions
to weather and commodity price changes, track-         lack capacity to conduct sample surveys, wide
ing three-year averages and/or statistical trends      fluctuations in agricultural production make it
rather than annual data may be advisable to            difficult to establish statistically significant
dampen the fluctuations.                               trends in production data over short time peri-
                                                       ods, and survey data collection costs are high.
Special Studies                                        Large-scale national surveys are probably best
                                                       avoided, unless these are linked to a regular
Bridging the gap between indicators from rou-          program of national agricultural statistics. More
tine reporting on project activities and from          focused regional or localized surveys may be
national data sets is a challenge for AKIS pro-        more manageable, and are useful in character-
grams, and reflects the difficulty inherent in         izing agricultural conditions and changes in a
linking project performance and eventual               particular area.
impacts of programs. Special studies are a prac-            Rapid rural appraisals. Rapid rural ap-
tical tool for collecting information on agricul-      praisals are a compromise between formal
tural systems and program outcomes and                 sample surveys (which generate statistically-
impacts. They are most important for assessing         valid, quantitative data but are costly and time
quality of outputs and for measuring outcomes.         consuming) and informal field visits (which
Special studies should generally provide for           provide anecdotal information) (Kumar 1993).
disaggregating data by gender, ethnicity, or           They are low cost, quick, open-ended and flex-
other key characteristics of client populations.       ible, but are not statistically-valid and may
    Special studies can usually be built into the      be open to charges of bias. Five core methods
regular work plan of AKIS institutions, as ex-         of rapid rural appraisal approaches are: key
tension agencies must track changes in farmer          informant interviews, focus group interviews,
18   Monitoring and Evaluation for AKIS Projects: Framework and Options

community interviews, structured direct obser-       sources and can be useful to: analyze potential
vation, and informal surveys.                        social, technical, and financial factors affecting
    Special topic studies. Special studies are       uptake of new agricultural technologies; assess
often useful to analyze technology program           potential financial and economic benefits from
impacts and develop an analytical base for rou-      technological innovation; assess differential
tine M&E data collection. Ex ante economic           impacts of programs on different groups; and
analysis and economic models of the impact of        identify key issues for effective extension pro-
technological innovation are common types of         gram operations.
special topic studies. Others might focus on spe-        External peer reviews. Peer reviews are
cific impacts on women, land tenure, food            used in highly technical program areas, most
prices, living standards, or nutrition.              commonly in research. They provide subjective
    Diagnostic case studies. Diagnostic case         assessments of program activities, potential,
studies provide a holistic, in-depth descriptive     and accomplishments, and are useful in devel-
assessment of a program or local situation.          oping linkages between scientists and institu-
These may combine information from various           tions working in related fields.

                  VI. M&E in the Bank Project Cycle

The different stages of the Bank project cycle       be updated every two to three years to monitor
each require special attention to project M&E        changes in the selected areas. Studies during
systems and use of indicators. Ezemenari,            project preparation, particularly participatory
Rudqvist, and Subbarao (1999) relate M&E to          needs assessments, can analyze current farm,
the Bank project cycle as outlined in Table 4.       household, and local conditions. These studies
They note the particular importance of evalu-        can assess prospects for technological change
ation in the context of Learning and Innova-         to impact on current production systems, iden-
tion Loans (LILs) and Adaptive Program               tify social considerations in implementing pro-
Loans (APLs).                                        grams, and identify current and potential future
    Gender, ethnicity, and poverty impacts           sources from which producers receive techni-
should be considered at all stages of the project    cal support.
cycle. Data for planning, management, and                 Appraisal/Negotiations. At appraisal pro-
evaluation should be disaggregated according         jects should identify output, outcome, and im-
to these and other relevant variables.               pact indicators and the procedures to be used
    Identification/Preparation. As part of           for monitoring and evaluation. Systems for data
project preparation, diagnostic case studies         collection and plans for use of monitoring data should
using rapid rural appraisal methodologies are        be worked out in detail and be fully understood
useful to assess problems and potential in a         and agreed to by all parties. A detailed M&E
given agroecological area. These can be fol-         plan should be included in the Bank Project Ap-
lowed by more comprehensive, quantitative            praisal Document (see Box 5). Ideally, baseline
surveys to provide baseline data for selected        survey data should be available at appraisal. If
extension areas. Such baseline studies can then      it is not, the project M&E plan should provide
                                                                           M&E in the Bank Project Cycle         19

Table 4   Incorporating Evaluation into the Bank Project Cycle

     Stage of Traditional
         Project Cycle                 Evaluation-related Tasks                 Approaches and Methods
  Identification/Preparation   Develop and verify logic of project          Participatory needs assessment
                               hypothesis for investment impacts on         and diagnostic case studies using
                               CAS-related sector goal.                     rapid appraisal methodologies
                               Identify output, outcome, and impact         (findings can later be used in
                               indicators for the project.                  structuring surveys and special
                                                                            studies for M&E data collection).
  Appraisal/Negotiation        Identify targets for output, outcome, and    Client feedback.
                               impact indicators with established
                               baselines and controls.
                               Prepare detailed project M&E plans,
                               ♦ financing and staffing requirements
                                   for the management information
                               ♦ sources and detailed plans for
                                   baseline data collection; and
                               ♦ time-frame and key information
                                   requirements for follow-up studies
                                   and surveys.
  Implementation/              Use indicators and targets identified        Rapid appraisal, attitude surveys,
  Supervision                  during appraisal to monitor                  and use of panel or before/after
                               implementation.                              surveys for early assessment of
                                                                            outcomes and impacts.
                                                                            Beneficiary assessment.
  Completion                   Implement follow-up survey.                  Rapid appraisal and beneficiary
  Post-completion              Evaluate outcome and impact indicators       Comparison of means (for
                               for the project.                             randomization) and econometric
                                                                            methods combined with client
                                                                            feedback from qualitative

Source: Adapted from Ezemenari, Rudqvist, and Subbarao (1999).

explicit plans on how, when, and by whom such             mon feature of Bank-financed AKIS projects, but
baseline data will be collected.                          present an additional challenge to monitoring
     Implementation/Supervision. Indicators               and evaluation. They typically result in a large
identified during appraisal are used to monitor           number of diverse subactivities and there is a
implementation progress and feedback to sup-              need to assess operation of the grants program
port program management. Project manage-                  itself, the individual grants, and the combined
ment must provide routine reports on inputs               impact of the total program. Summary guidance
and outputs and complement this with infor-               on M&E for agricultural technology funds is
mation on progress from rapid rural apprais-              provided in Box 6.
als, attitude surveys, longitudinal case studies,              Completion/Post-Project. Project perfor-
and beneficiary assessments.                              mance evaluation and identification of lessons
     Agricultural Technology Funds (ATFs), in-            learned draw on follow-up case studies, ben-
cluding competitive grant programs, are a com-            eficiary assessments, and econometric impact
20       Monitoring and Evaluation for AKIS Projects: Framework and Options

 Box 5       Elements of a Project M&E Plan

 The PAD should summarize project M&E arrange-                   ♦   Baseline data: ideally should be available at ap-
 ments, including details on:                                        praisal, but, if not, arrangements for collection
                                                                     of baseline data should be spelled out.
 ♦       Who: what institutions are responsible for moni-        ♦   Capacity: the appraisal should assess current
         toring and evaluation; how responsibilities are             capacity for M&E in terms of established sys-
         divided; the numbers and qualifications of staff            tems, staffing, strengths, and weaknesses, and
         responsible for M&E functions.                              should detail training, technical assistance,
 ♦       What: what information and data are to be col-              and other inputs needed to strengthen M&E
         lected.                                                     capacity.
 ♦       How: what procedures and methodologies are              ♦   Project hypothesis: which is essentially embodied
         to be used to gather data; what sources will be             in the project logical framework.
         used; how will the project Management Infor-            ♦   Indicators: input, output, outcome, and impact
         mation System manage data.                                  indicators should be developed by component
 ♦       Why: how data will be used and by whom; how                 and with quantified targets by year over the
         will it feed into policymaking.                             project period.
 ♦       When: how frequently will data be collected and         ♦   Evaluation questions: any key issues for review
         reports prepared.                                           during the mid-term review or at any other point
 ♦       Where: what will be the responsibilities at differ-         during the project should be identified; these
         ent levels and in different organizations; how will         may include assumptions made but not well
         data flow between local and national levels.                tested during project appraisal.

     Box 6   Methodology for Performance Assessment of Agricultural Technology Funds
     Agricultural Technology Funds (ATFs) with portfo-               types of institutions undertaking research or ex-
     lios of many diverse subprojects present a special              tension, types of research being carried out, over-
     challenge for M&E systems. Such programs often                  all output of research or extension institutions,
     have a number of different objectives requiring dif-            and quality of research and extension activities.
     ferent indicators for impacts at the level of sub-          ♦   At the subproject implementation level. Implement-
     projects, program operations, and policy.                       ing institutions should be required to establish
                                                                     targets for inputs (extension visits or research
     ♦   At the program level. The management entity                 trials), outputs (farmers trained or research rec-
         needs indicators, targets, and a management                 ommendations), and outcomes (technology
         information system to track program implemen-               adoption, income increases). These should be
         tation performance (numbers of subprojects,                 considered in decisions on financing subprojects,
         quality of sub-projects, time required to approve           and implementing institutions should provide
         and start subprojects, and overhead costs). This            routine reports on progress against targets.
         information is useful to assess the efficiency of
         ATF operations and to characterize the overall               The ATF program management entity has two
         portfolio.                                              functions in subproject M&E. First, it must verify
     ♦   At the national policy level. Program financiers need   progress towards objectives through routine over-
         data from the ATF management information sys-           sight supplemented by routine audits and evalua-
         tem and special studies to track change in pro-         tions of randomly selected subprojects. Second, the
         grams or institutions. Progress on institutional        management entity must maintain a management in-
         reform objectives is measured by indicators, such       formation system to aggregate subproject data on
         as nongovernmental financing for the program,           program inputs, outputs, outcomes, and impacts.

studies, if appropriate. More importantly,            ful to refine program design and improve per-
project completion should not see the end of          formance.
monitoring and evaluation, but its establish-              Evaluation has generally referred to peri-
ment as an ongoing function of government             odic, ex post assessment of project hypotheses,
policy units and technology programs. A key           assumptions, performance, and impacts. Such
indicator for a project outcome may be the ex-        evaluations are useful because they force atten-
tent to which the institution’s monitoring and        tion on the performance and impact of program
evaluation system is established and utilized at      activities. They may assess an overall project or
project completion.                                   individual components or aspects of a project.
     “Evaluation” is actually a continuous pro-       In Bank-financed projects, periodic evaluations
cess of assessing information on a program’s          typically include the Mid-Term Review and the
performance and impact and, in Bank-financed          Implementation Completion Report (ICR). In all
projects, evaluation is an implicit objective in      evaluations, the importance of stakeholder par-
all supervision Missions. In the past, evaluation     ticipation is increasingly recognized.
was relegated to late stages of a project—often            There is a wealth of literature on evaluation.
after the Project Completion Report or Impact         One recent paper on evaluating project impacts
Audit or, at best, during the Mid-Term Re-            on poverty (Baker 1999) is relevant to AKIS
view. Timely evaluation provides input use-           project needs.

                              VII. Recommendations

Attention to a few simple recommendations             (using what procedures), by whom, when,
should significantly improve M&E in Bank              where, and why (how it will be used). The M&E
AKIS projects. However, developing M&E ca-            plan need not be long, but should: describe ar-
pacity is a long-term process and lies at the heart   rangements for baseline data collection; assess
of improving program management and build-            capacity for carrying out M&E; define indica-
ing effective institutions to promote economic        tors and targets; identify investments to
growth, reduce poverty, and conserve environ-         strengthen M&E capabilities; and identify key
mental resources.                                     assumptions or issues to be addressed in
                                                      project evaluations (for example, the Mid-Term
A. Describe Details of M&E System in                  Review). Data should generally be gender-
Project Appraisal Document                            disaggregated.

The PAD format requires some attention to             B. Request M&E Data from the Early Stages
M&E, but does not force a comprehensive de-           of a Project
scription and plan for an M&E system. As the
basis for a sound M&E plan, the PAD should            Good M&E plans often remain only plans and
include a comprehensive description of the pro-       are neglected as the myriad of implementa-
posed M&E system in a separate annex, includ-         tion details overwhelm the start-up phase of
ing details of what data are to be collected, how     a project. M&E data are not missed until they
22    Monitoring and Evaluation for AKIS Projects: Framework and Options

are needed for evaluation or management de-           D. Establish A Comprehensive Set of Key
cisions later in the project. Program manag-          Performance Indicators
ers and TTLs should give early and consistent
attention to M&E, beginning during project            A good set of hierarchically linked performance
launch and preparation of a project Opera-            indicators is critical to project M&E. Indicators
tional Manual, and continuing in all supervi-         can be selected from the menu of illustrative
sion missions.                                        indicators developed by AKIS (see Annex A)
     The format for project implementation re-        and should reflect the project impact hypoth-
ports should be agreed upon at the launch of          esis of cause-and-effect progressing from inputs
a new project, and the reports should be pre-         to outputs, outcomes, and impacts. To the ex-
pared at least annually (and possibly quar-           tent possible, indicators should have targets,
terly or semi-annually). These reports should         defined in terms of time, quantity, and quality
include updates on performance and impact             and be gender-disaggregated. Targets should
indicators being tracked by the project (rec-         be reviewed regularly and revised as appropri-
ognizing that some data will be available only        ate (probably not as often as every year, but
infrequently and that some indicators will not        perhaps every other year).
change until late in the project). If the project
M&E system (especially the internal MIS) is           E. Establish Project Baseline Data
not producing baseline information and regu-
lar reports, there is a problem. The early stage
                                                      In some cases, baseline data will be available
of the project is also the time to confirm that
                                                      from surveys or censuses before a program be-
data collection is disaggregated by gender
                                                      gins. This is ideal, but in practice this is rare and
and other key characteristics of the client
                                                      adequate data are often not available. When
                                                      baseline data are not available, the project
                                                      should provide for surveys or special studies
C. Develop Permanent M&E Capacity
                                                      to establish a baseline within the first year of
for Implementing Institutions
                                                      the project.
Project-specific M&E systems or supplemental
project M&E activities may occasionally be            F. Avoid Large Surveys
needed, but AKIS projects should generally base
M&E systems on permanent M&E units within             Though often potentially useful and sometimes
project implementing institutions. Development        necessary, extensive surveys are costly, time
of institutional capacity for ongoing M&E of          consuming, and—even when completed—often
research and extension programs is frequently         fail to provide desired information. Rapid ru-
a critical institutional development objective in     ral appraisals, case studies, and defined area
itself, and will require adequate investment in       surveys or censuses are generally more cost ef-
staffing (including training of social scientists)    fective and useful, but should ensure that
and system development. Such capacity build-          sample areas and informants represent and ad-
ing meets both immediate project requirements         dress issues of differential program impact on
and longer-term system needs. There will be           women and disadvantaged groups.
cases in which institutional development
projects require independent, project-specific        G. Use a Combination of Data
M&E arrangements to monitor institutional             Collection Methods
change, which would not be monitored by a
permanent M&E unit (such as, monitoring the           Project management information systems based
establishment of an M&E unit).                        on regular implementation reports should pro-
                                                                       References and Additional Readings        23

vide detailed data on inputs (extension staff),             H. Ensure Utilization of M&E Data
outputs (farmers trained), and initial outcomes
(technology adoption). Special studies—often                Effective M&E systems provide a wealth of in-
contracted from independent agencies—can                    formation that is often used for periodic project
assess quality of outputs (training), and quan-             evaluation, but underexploited for other uses.
tify and assess quality of outcomes (changes in             Routine reports from project management in-
knowledge, attitudes, practices, production,                formation systems should guide decisions on
and incomes). And, national level data col-                 project implementation and on refining plans
lected independently of the project can be                  for project activities. Analytical reports and
expected to reflect long-term (generally five               impact studies (that is, those demonstrating
to ten years or more) changes in project im-                impacts or identifying program constraints)
pact indicators.                                            should be available to policymakers to influence
                                                            investment plans and policy reforms.

              VIII. References and Additional Readings

Alex, G. 1998. Assessing Agricultural Research. Towards         World Bank, Operations Evaluation Department,
     Consensus on a Framework for Performance and Im-           Washington, D.C.
     pact Assessment. Agricultural Research and Ex-         Garcia, M., H. Alderman, and A. Rudqvist. 1999. Un-
     tension Group. Special Report No.6, World                  derstanding Poverty Reduction Impacts with Inno-
     Bank, Washington, D.C.                                     vative Monitoring and Evaluation: PREM Notes No.
Baker, J. 1999. Evaluating Project Impact for Poverty           32 (October 1999), World Bank, Washington, D.C.
     Reduction: A Handbook for Practitioners. World         Horstkotte-Wesseler, G., M. Maredia, D. Byerlee, and
     Bank, Washington, D.C.                                     G. Alex. 2000. Ex Ante Economic Analysis in AKIS
Babu, A., Y. Singh, and R. Sachdeva. 1997. “Estab-              Projects: Methods and Guidelines for Good Practice.
     lishing a Management Information System.” In               AKIS Thematic Group, Rural Development Fam-
     Improving Agricultural Extension: A Reference              ily. World Bank, Washington, D.C.
     Manual. Food and Agricultural Organization of          ISNAR. 1991. Guidelines for the Practitioner of IN-
     the United Nations, Rome.                                  FORM, An Information Management System for
Ballentyne, P., D. Horton, W. Peterson, K. Sheridan, B.         Agricultural Research.
     Uribe, and D. Gapasin. 1993. Monitoring and Evalu-         isnar/ publications/catalog/inform.htm. In-
     ation of Agricultural Research: A Sourcebook. Inter-       ternational Service for National Agricultural
     national Service for National Agricultural Research        Research (ISNAR), The Hague.
     (ISNAR), The Hague.                                    ISNAR. 2000. Listing of Research Monitoring and Evalu-
Casley, D. and K. Kumar. 1987. Project Monitoring               ation Publications.
     and Evaluation in Agriculture. World Bank, Wash-           publications/eval.htm. International Service for
     ington, D.C.                                               National Agricultural Research (ISNAR), The
Ezemenari, K., A. Rudqvist, and K. Subbarao. 1999.              Hague.
     Impact Evaluation: A Note on Concepts and Meth-        Kumar, K. ed. 1993. Rapid Appraisal Methods. World
     ods. Paper for World Bank Conference on Evalu-             Bank, Washington, D.C.
     ation and Poverty Reduction June 14-15, 1999.          Lecuit, L., J. Elder, C. Hurtado, F. Rantua, K. Siblini,
24    Monitoring and Evaluation for AKIS Projects: Framework and Options

    and M. Tovo. 1999. DeMIStifying MIS—Guide-            Salmen, L.F. 2000. The Voice of the Farmer in Agricul-
    lines for Management Information Systems in Social        tural Extension. AKIS Discussion Paper, World
    Funds. World Bank, Washington, D.C.                       Bank, Washington, D.C.
Maredia, M., D. Byerlee, and J. Anderson. 2000. Ex        Wood, S. and P. Pardey. 1997. Agroecological Aspects
    Post Evaluation of the Economic Impacts of Agricul-       of Evaluating Agricultural R&D: EPTD Discussion
    tural Research Programs: A Tour of Good Practice.         paper No. 23. International Food Policy Research
    Paper presented to a Workshop on “The Future              Institute (IFPRI), Washington, D.C.
    of Impact Assessment in the CGIAR: Needs,             World Bank. 1989. Operational Directive 10.70 on
    Constraints, and Options”, May 3-6, 2000.                 Project Monitoring and Evaluation. World Bank,
    CGIAR Technical Advisory Committee, FAO,                  Washington, D.C.
    Rome.                                                 World Bank. 1996. Lessons and Practices: Designing
Misra, D. 1997. “Monitoring Extension Programmes              Project Monitoring and Evaluation. Operations
    and Resources”. In Improving Agricultural Exten-          Evaluation Department. World Bank, Washing-
    sion: A Reference Manual. Food and Agricultural           ton, D.C.
    Organization of the United Nations. Rome.             World Bank. 2000. Tools for Gender-sensitive Monitor-
Murphy, J. and T. Marchant. 1988. Monitoring and              ing and Evaluation for Rural Development. Gender
    Evaluation in Extension Agencies: World Bank              and Rural Development Thematic Team. World
    Technical Paper No. 79. World Bank, Washing-              Bank. <
    ton, D.C.                                                 essd.nsf/gender/home>.
    Annex A. Illustrative Indicators for AKIS Projects

The following is a menu of possible goal, out-             at which a specific indicator is relevant may vary
come, output, and input indicators for technol-            depending on the specific project/program ob-
ogy system projects/programs. A limited                    jectives. (Note: Typical source for data is indi-
number of relevant indicators should be selected           cated as GDB = Global Database; CDB = Country
to meet the needs of specific projects/programs.           Database; PDB = Project Database; SS = Special
The level (especially for outputs vs. outcomes)            Studies; ER = Expert Review.)

I. Goal Level Indicators (typically not monitored by projects)

All AKIS Projects

                            Outcome Indicator                                Measurement          Source
  G Rural population below (national) poverty line                      %; number                GDB/
  G Rural population with incomes < $1 per day                          %; number                GDB/
  G Daily per capita dietary energy supply                              Calories/capita          GDB/
  G Malnutrition prevalence (rural and urban)                           %; number                GDB/
  G Agricultural value added/rural worker                               Value; % growth/         GDB/
                                                                         year                     CDB
  G Agricultural GDP/agricultural worker                                Value; % of national     GDB/
                                                                         GDP/worker               CDB
  G Agricultural value added                                            Value; % growth/         GDB/
                                                                         year                     CDB
  G Agricultural exports                                                Value; % growth/         GDB/
                                                                         year                     CDB
  G Freshwater withdrawal for agriculture                               Water                    GDB/
                                                                         use/agricultural GDP;    CDB
                                                                         total; % of total
  G Deforestation                                                       %per year; % change      GDB/
                                                                         in rate from 1990-95     CDB
  G Active community groups or farmers organizations                    Number; number of        GDB/
                                                                         members; total value     CDB
                                                                         of financial turn-over

26    Monitoring and Evaluation for AKIS Projects: Framework and Options

II.   Project Development Objective Outcome Indicators

Extension Projects/Institutional Development

                            Outcome Indicator                                     Measurement        Source
 G Farmer contribution to costs of extension, disaggregated by farm         % of total cost         PDB
      size, gender, age, farm type, and income
 G Extension cost per farmer, per hectare, or per unit value of             Cost per unit           PDB/SS
 G Farmer evaluation of extension staff, including proposals for            Ratings                 SS
      sanctions and promotions (by sex)
 G Extension strategy and objectives as reflected in government             Rating of quality and   ER
    policy statement                                                         effectiveness
 G Farmer requests for extension services (by sex)                          Number; %               PDB/SS
 G Farmer awareness of extension activities (by sex)                        % of farmers            SS
 G Frequency of extension staff supervision visits                          Weeks/visit             PDB/SS
 G Public sector extension expenditures                                     Total; % AgGDP          CDB/SS
 G Intensity of extension coverage                                          Families/agent or       PDB/SS
 G Efficiency of extension activity (demonstrations, training, field        Number/agent;           PDB/SS
      days)                                                                  cost/activity
 G Technical support services                                               Number of SMS per       PDB
                                                                             100 agents
 G Functional monitoring & evaluation systems                               M&E System Quality;     ER/SS/
                                                                             MIS data timeliness;    PDB
                                                                             Impact studies/100
 G Equitable services provision to women, minorities, and the poor          % of clients            PDB/SS
 G Gender balance of extension staff                                        % women                 PDB/SS
 G Ethnic balance or minority language capability of extension staff        % minority or           PDB/SS
                                                                             language capable
 G Agencies providing extension services (NGOs, universities,               Number                  PDB/SS/
      consulting firms)                                                                              CDB
 G Non-salary operational costs as a percent of total extension             % of total budget       CDB
 G Annual attrition rate or turnover of extension agents (by sex)           % of total staff        CDB
 G Extent of transfer of extension tasks and responsibilities to            % of total costs        PDB/SS
      private and farming sectors
 G Farmer representation on extension management committees (by             number; %               SS
      sex, ethnicity, farm size)
 G Surveyed farmer attitudes toward extension                               % positive responses    SS
(Note: Should also include some indicators of effectiveness of technology transfer, as below.)
                                                      Annex A. Illustrative Indicators for AKIS Projects   27

Extension Projects/Technology Dissemination

                         Outcome Indicator                                  Measurement         Source
G Adoption rates for new practices or technologies, including          % of all farmers        PDB/SS
    studies of rationales for adoption/non-adoption by sex of farmer
G Change in rate of degradation or use of natural resources (water,    Rate of use; area       PDB/SS
    forest, land)
G Adoption of conservation practices                                   Area; number of         PDB/SS
G Farm practices changed due to extension by sex of farmer             Number; % of            PDB/SS
G Change in productivity, risk, yield, costs, income or welfare as a   % change from base      SS
    result of new technology or practices, by sex of farmer
G Time-series or case studies of change in farmer knowledge,           % change from base      SS
    skills, attitudes, or understanding of technologies or practices
G Yield gap between farmers’ yields and on-farm trials                 % difference            SS
G Yield gap between national yields and neighboring countries          % difference            CDB
28    Monitoring and Evaluation for AKIS Projects: Framework and Options

Research Projects/Institutional Development

                          Outcome Indicator                                  Measurement            Source
 G Research strategy and objectives as reflected in government       Rating of quality and         ER
      policy statement                                                effectiveness
 G   Stakeholder participation in governance, funding, priority      % of funding or assessment    PDB/ER
      setting, and execution (by sex)                                 of participation
 G   Disciplinary, product & regional balance of research (social    % of total budget or          CDB/
      science, natural resource management, livestock, uplands)       scientist time allocated      PDB
 G   Institutions engaged in research execution                      Number                        PDB/SS
 G   Research scientists by level of qualification (by sex)          Number and qualification      CDB/SS
 G   Peer reviewed publications, patents, or dissemination           Number per year per           PDB/SS
      events (by sex)                                                 researcher
 G   Researcher productivity (by sex)                                Trials/scientist              PDB/SS
 G   Extent of on-farm research                                      Number of trials; % of        PDB/SS
                                                                      scientist time
 G Functional Monitoring & evaluation system                         Assessment of system;         PDB/ER/SS
                                                                      number of impact studies
 G   Effective research-extension linkages                           Assessment of systems         ER/PDB/SS
 G   Research projects under way                                     Number                        PDB/SS
 G   Research budget                                                 Total; % AgGDP                CDB
 G   Competitive funding (contract and grant) share of research      Total; % of total budget      PDB/ CDB
 G   Recurrent costs or personnel costs relative to total research   % of total budget             CDB
 G   Private financing of research by source                         Value; % of total             SS
 G   Ratio of professional to support staff and trained              Ratio                         PDB/SS
      technicians (by sex)
 G   Researcher or support staff job performance before and          Number and assessed           PDB/SS
      after training (by sex)                                         quality
 G Annual research staff turnover                                    % of all staff                CDB/SS
 G Laboratories equipped and functioning                             Number and quality            SS/ER
 G Factors reflecting positive staff morale and professional         % of staff                    SS
      activity (by sex)
 G Staff time devoted to research                                    % of time                     SS
 G Farmer representation on research management                      Number; %                     PDB/SS
      committees (by sex, farm size)
 G Extent of regional and international collaboration                % of time or budget ;         SS/PDB
                                                                      number of programs
(Note: Should also include some indicators of effectiveness of technology development, as below.)
                                                         Annex A. Illustrative Indicators for AKIS Projects         29

Research Projects/Technology Development

                            Outcome Indicator                                        Measurement          Source
 G Adoption of released technologies and management                             % of farmers adopting    SS
    recommendations (by sex)                                                     technology
 G Benefits of adoption of new technologies—e.g., impact on                     % change from base       SS/ER
 G Client satisfaction with new technology or management                        % positive response      SS
    recommendations (by sex)
 G Research impact on national policy formulation                               Number; significance     ER

Agricultural Education Projects/Institutional Development

                            Outcome Indicator                                        Measurement          Source
 G Faculty qualifications                                                       Number and               PDB/SS
 G Physical facilities                                                          Number or size           PDB/SS
 G Linkages to client groups and stakeholders of both sexes                     Number and               SS/PDB
 G Use of modern communications and information technologies                    % of staff using         SS/PDB
 G Involvement in national policy, research, and extension                      Assessment of            ER/SS/
    programs                                                                     involvement              PDB
 G Interinstitutional linkages                                                  Number and value         ER/PDB
 G Disciplinary balance in program, especially attention to private             % of faculty, courses,   SS/CDB
      sector and environmental issues                                            or time; assessed
 G Level of practical experience provided in curricula                          % of time                ER/SS/
 G Intellectual inbreeding—number of faculty with degrees from                  % of faculty             CDB/SS
    other institutions(by sex)
 G Level of extension and research involvement by faculty (by sex)              % of time spent or       SS/PDB/
                                                                                 expenditures             CDB
 G Length of time required to complete degree programs or obtain                Years                    CBD/SS
 G Annual attrition rates of faculty (by sex)                                   % of total staff         SS/PDB/
 G Cost of training programs per trainee                                        Value                    CDB/SS/
 G Training program costs recovered from payments by users                      % of total costs         CDB/SS/
(Note: Should include some indicators of effectiveness of training, as below.)
30    Monitoring and Evaluation for AKIS Projects: Framework and Options

Agricultural Education Projects/Training

                         Outcome Indicator                                 Measurement         Source
 G Students graduated, disaggregated by gender                        Number                  PDB/SS
 G Testing of program graduates to assess skills and competencies     Standardized exam       SS/ CDB
 G Employment rate of graduates (by sex)                              % unemployed after      CDB/SS
                                                                       various periods of
 G Job performance satisfaction of graduates (by sex)                 % satisfied with        SS
 G Supervisor assessments of skills and competencies of graduates     % of positive           SS
    (by sex)                                                           responses
 G Staffing of sector programs and institutions                       Number of jobs          SS/ER/
                                                                       unfilled or filled by   CDB

III. Project Output Indicators

Extension Projects/Institutional Development

                          Output Indicator                                  Measurement        Source
 G Regional/decentralized offices established                         Number                  PDB
 G Extension agency staffing (by sex)                                 Number and              PDB
 G Extension staff assigned in the field (by sex)                     %                       PDB
 G Extension service strategy and plan completed                      Assessed quality of     PDB/ER
 G Extension materials preparation unit established                   Size and quality of     PDB/ER
 G Availability of backstopping information (subject matter           Assessed quality;       PDB/SS
     specialist support) for extension agents                          Number of SMS/ 100
 G Extension agent training support (program schedules and            Number of staff         PDB/ER
     content)                                                          trained; training-      /SS
                                                                       days/ year; assessed
 G Extension programs tailored to specific needs of different types   Number and quality      PDB/ER
    and gender of farmers                                              of plans
 G Transport capacity (vehicle, bicycle)                              Number/ 100 agents      PDB
 G Computer support capacity                                          Number/ 100 agents      PDB
 G Cost per extension agent                                           Cost                    PDB
                                                      Annex A. Illustrative Indicators for AKIS Projects   31

Extension Projects/Technology Dissemination

                            Output Indicator                                Measurement         Source
 G   Demonstrations established                                        Number                  PDB
 G   Farmer training courses conducted                                 Number                  PDB
 G   Farmers trained, contacted, or visited (disagregated by sex)      Number                  PDB
 G   Cost per farmer served                                            Cost                    PDB
 G   Rate of turn-over of contact farmers (by sex)                     % per year              PDB
 G   Extension services provided to women farmers                      % of all services       PDB
 G   Mass media products disseminated                                  Number                  PDB
 G   Extension agent in-service training provided (by sex)             Number of person-       PDB
 G Extension agents trained (by sex)                                   Number                  PDB
 G Extension programs tailored to specific needs of different types    Number and quality      PDB/ER
      and gender of farmers                                             of plans
 G Frequency of face-to-face meeting with farmers (by sex)             Frequency               PDB/SS
 G Extension agent time in face-to-face meeting with farmers (by       % of extension agent    PDB/SS
      sex)                                                              time
 G Farmer group meetings/audiovisual shows/demonstrations              Numbers & number        PDB
                                                                        of participants
 G Surveyed farmer attitudes towards extension agents (by sex)         % positive responses    PDB/SS

Research Projects/Institutional Development

                             Output Indicator                             Measurement           Source
 G Research plans approved                                             Number                  PDB
 G Research plans financed                                             Number                  PDB
 G Peer-review mechanisms in place                                     Number                  PDB
 G M&E systems established including management information            Number                  PDB
 G Cost per researcher                                                 Cost/ researcher        PDB
 G Staff performance appraisal and promotion system revamped to        Number and quality      PDB
      reward productivity
 G Changes in orientation of research (i.e., to multidisciplinary,     % of total cost         PDB/SS
 G Collaborative research programs established with national and       Number and value        PDB
      international partners
 G Number of researchers trained (by sex)                              Number                  PDB
 G Journals/publications acquired                                      Number                  PDB
 G Research or support staff trained (by sex)                          Number                  PDB
 G Scientists linked to electronic information systems (by sex)        Number or %             PDB/SS
 G External management reviews                                         Number                  PDB/ER
 G Research-extension linkages established                             Assessed quality        PDB/ER
 G Regulatory and legal frameworks for research and technology         Assessed adequacy       PDB/ER
      importation established
32    Monitoring and Evaluation for AKIS Projects: Framework and Options

Research Projects/Technology Development

                          Output Indicator                              Measurement           Source
G New varieties released                                             Number                  PDB
G Extension recommendations published                                Number                  PDB
G Other research products completed research tools/methods           Number                  PDB
G Papers published                                                   Number                  PDB
G Collaborative research programs established with national and      Number and value        PDB
     international partners
G Cost per project, research paper, or technical recommendation or   Cost/ unit              PDB/SS
     technology developed
G Improved knowledge of farming systems                              Extent of knowledge     ER/PDB
G Improved knowledge of natural resources status and trends          Extent of knowledge     ER/PDB
G Improved research relevance through use of on-farm                 Extent of on-farm       PDB/SS/
     participatory methods                                            research                ER

Agricultural Education Projects/Institutional Development

                          Output Indicator                              Measurement           Source
 G Construction completed (laboratories, dormitories, classrooms)    Number and size         PDB
 G Labs equipped                                                     Number and quality      PDB
 G Faculty trained                                                   Number and              PDB
 G Postgraduate programs established                                 Number                  PDB
 G Curricula revised or developed                                    Number and quality      PDB/ER
 G Training materials prepared                                       Number                  PDB
 G Links to research programs developed                              Value or % of faculty   PDB/ER
 G Links to extension activities developed                           Value or % of faculty   PDB/ER
 G Linkages with agribusiness developed                              Assessed extent of      PDB/ER

Agricultural Education Projects/Training

                          Output Indicator                                Measurement         Source
G Graduates (by sex)                                                 Number                  PDB
G Cost                                                               Cost per graduate       PDB
                                                     Annex A. Illustrative Indicators for AKIS Projects   33

IV.   Input Indicators/All AKIS Projects

                           Input Indicator                               Measurement           Source
G Civil works completed                                               Number and Value        PDB
G Goods procured                                                      Value                   PDB
G Training completed                                                  Number of trainees      PDB
G Technical assistance provided                                       Person-weeks            PDB
G Curricula developed                                                 Number                  PDB
G Training programs conducted                                         Number                  PDB
G    Unit cost of inputs                                              Cost per unit           PDB
G    Quality of contacts with stakeholders                            Assessed quality        PDB/ER
G    Research or extension subprojects financed                       Number and value        PDB
G    Diagnostic reports on farmer constraints and priorities          Number                  PDB
G    Meetings of researchers and clients                              Number                  PDB
G    Scientific program reviews, including external reviews and       Number                  PDB/ER
      dialogues between researchers and clients, completed
G    Meetings of researchers with clients (by sex)                    Number and quality      PDB
G    Extension agent level of effort delivered (by sex)               Person-years            PDB
Annex B. Example of Routine Project Report Format:
     Key Indicators for PRODETAB (Brazil)
                                                                     Base    Planned    Actual   Project
                                                                    period   mid-term    mid-     end
 Key Quantitative Indicators: Outcomes

 1a: Percent of financing for the Competitive Grants System           0         5        23         15
 coming from private sector/beneficiaries
 1b. Number of public/private partnership contracts signed           175       200       239       250
 1c. State research contributions to competitive grants in funded     0         3        1.1        7
 projects (R$ m.)
 1d: Percent of EMBRAPA’s non-budget revenues through cost            7         12        7         15
 2a: Percent of resources from Competitive Grants System              16        25       45         33
 allocated to institutions other than EMBRAPA
 2b. Percent of projects lead by institutions other than              0         10       32         30
 EMBRAPA through competitive grants
 2d. Number of collaborating universities with EMBRAPA                40        48       42         60
 2e. Number of international collaborative agreements with
 2h. Scientific and extension publications originated from            0         80       61        400
 supported grants
 4a. Annual operating budgets to research on small farm              1.2       4.7       1.6       7.2
 development programs (R$ m.) by EMBRAPA for SNPA over
 its 1996 allocations for family farm development program
 4b. Annual operating budgets to research on natural resource        3.0       6.5       5.2       9.6
 management (R$ m.) by EMBRAPA for SNPA over its 1996
 allocations for natural resources program 1
 4c. Annual operating budgets to research on advanced                1.6       4.1       2.9       7.6
 technology development (R$ m.) by EMBRAPA for SNPA over
 its 1996 allocations for biotechnology program 1
 5a: Percent of EMBRAPA’s national operational budget                 0         10        7         30
 allocated to competitive grants
 5b. EMBRAPA support to state research systems (R$ million)           9         6         1         2

 Key Quantitative Indicators: Outputs

 1. Number of different institutions participating in approved
 •   EMBRAPA                                                                             35
 •   State research organizations                                                        13
 •   Universities                                                                        25
 2. Number of EMBRAPA staff trained                                   0         14       60         64
 3. Number of NRCs supported with infrastructure/equipment            0         5                   20
 4. Number of individuals given long-term training in IPR             2         4                    7
 5. Number of scientists placed on long-term assignments in           0         4                    6
 advanced foreign research institutions
 7. Number of state staff trained                                     0         25       12        100
 8. Number of SRCs supported with infrastructure/equipment            0         3                  10


To top