Document Sample
Indicators Powered By Docstoc
					    Monitoring and Evaluation performance

        Institute of statistics and applied
                    FACILITATOR M&E
                                        Performance Indicator
    •   Performance indicators are usually seen as numerical
        measures of achievement that are easy to collect and
    • A numerical measure of the degree to which the
        objective is being achieved.
           Illustration of Performance indicator
    •   If one of your objectives is to train people so they are
        able to gain employment then one performance indicator
        can be the proportion of trainees that gain employment
        after the training is completed.

                                      Introduction to Indicators
    •   Definition of indicators:
    “An indicator is a marker. It can be compared to a road sign
      which shows whether you are on the right road, how far
      you have travelled, and how far you still have to go in
      order to reach your destination.”
    •   Indicators refer to the information that would help us
        determine progress towards meeting project objectives.
        An indicator should provide, where possible, a clearly
        defined unit of measurement and a target detailing the
        quantity, quality and timing of expected results.

                              Indicators Cont.

    • Indicators are clues, signs, and markers as
     to how close we are to our path and how
     much things are changing. For example, if
     you are driving in a car and the gas gauge
     shows you are low on gas, it is not actually
     the gas you are looking at, but rather you
     are looking at an “indicator” of the amount of
     gas you have.

                     Identifying the Indicators for each Objective

•   Indicators are qualitative or quantitative criteria used to
    check whether the proposed objectives have been
• Qualitative indicators based on
    judgements/perceptions derived from subjective analysis
    (e.g. parents feel more capable of talking to their children
    about transactional relationships).
•   Quantitative indicators can be precisely measured
    (e.g. number of training sessions conducted, number of
    TASO materials distributed).

  The following questions can be helpful in
  selecting indicators:
  • Have the definitions of the indicators been tested and can
  objectives be measured accurately (operational) and
  • Will the indicators measure only what they are supposed
  to measure (valid)?
  • Are there areas of overlap in the content of the indicator
  with that of other indicators; is it specific, or is it too
  • Will the indicators be able to measure changes over time
  • What resources (human and financial) do the indicators
  require? (affordable, feasible)
  • Are there alternative measures that should be
  • Will multiple indicators be able to help clarify the results of
6 the primary objective?
                             Characteristics good indicators :
•   Relevant: directly linked to the project objectives.
•   Measurable: capable of being assessed.
•   Reliable: verifiable i.e. conclusions based on them
    should be the same if they are assessed by different
    people at different times.
•   Usable: people in the project should be able to
    understand them and use the information provided by
    them to make decisions or improve the project.
•   Cost–effective: results should be worth the time and
    money it costs to collect, analyse and apply them.

                     Characteristics good indicators

    • Timely: it should be possible to collect and
     analyse the data reasonably quickly i.e. in
     time to be useful for any decisions to be
    • Ethical: the collection and use of indicators
     should be acceptable to all involved,
     including those people providing the

    Characteristics of a good indicator:
• Produce the same results each time it is used to measure the
    same condition or event
•   Measure only the condition or event it is intended to measure
•   Reflect changes in the state or condition over time
•   Represent reasonable measurement costs
•   Be defined in clear and unambiguous terms
•   Indicators should be consistent with international standards and
    other reporting requirements.
• Indicators should be independent, meaning that they are non-
    directional and can vary in any direction. For instance, an
    indicator should measure the number of clients receiving
    counseling rather than an increase in the number of clients
    receiving counseling or The contraceptive prevalence rate should
    be measured, rather than the decrease in contraceptive
• Indicator values should be easy to interpret and explain, timely,
    precise, valid, and reliable.
• They should also be comparable across relevant population
9   groups, geography, and other program factors.
     Indicators of what?

                                   Are all indicators equal?
     Different types of indicators are not equal but linked to
      each other to reach the intended goals and objectives of
      a specific program

       How often are different indicators
 •   The frequency of reporting will depend on the place of
     the indicators within the M&E conceptual framework –
     taking into account a reasonable time frame for an
     expected change and program capacity for M&E. The
     following reporting schedules are suggested:

     Other Important Considerations

Quantitative and Qualitative Indicators:
•    Quantitative indicators are numeric and are presented as
     numbers or percentages.
•    Qualitative indicators are descriptive observations and can
     be used to supplement the numbers and percentages
     provided by quantitative indicators.
•    Examples include "availability of a clear, strategic
     organizational mission statement" and "existence of a
     multi-year procurement plan for each product offered."

                  Important of Indicators
• Indicators provide M&E information crucial for
 decision-making at every level and stage of program
• Indicators of program inputs measure the specific
 resources that go into carrying out a project or
 program (for example, amount of funds allocated to
 the health sector annually).
• Indicators of outputs measure the immediate results
 obtained by the program (for example, number of
 multivitamins distributed or number of staff trained).
• Indicators of outcomes measure whether the
 outcome changed in the desired direction
 and whether this change signifies program “success”
                                      What Is a Metric?
 •   A metric is an important part of what comprises an indicator. It is
     the precise calculation or formula on which the indicator is based.
     Calculation of the metric establishes the indicator’s objective value
     at a point in time. Even if the factor itself is subjective or qualitative,
     like the attitudes of a target population, the indicator metric
     calculates its value at a given time objectively.
 Here is an example:
 •   Indicator: Percentage of urban facilities scoring 85-100% on a
     Quality of Care Checklist
     Note that because this indicator calls for a percentage, a fraction is
     required to calculate it.
 Possible metrics:
 •   Numerator, or top number of the fraction: number of urban facilities
     scoring 85-100% on a Quality of Care Checklist.
 •   Denominator, or bottom number of the fraction: total number of
     urban facilities checked and scored.
 • Defining good metrics is crucial to the usefulness of any M&E plan
18 because it clarifies the single dimension of the result that is being
     measured by the indicator.
     Guidelines for Selecting Indicators:
•   Select indicators requiring data that can realistically be collected
    with the resources available.
•   Select at least one or two indicators (ideally, from different data
    sources) per key activity or result. Select at least one indicator
    for each core activity (e.g., training event, social marketing
    message, etc.).
•   select no more than 8-10 indicators per area of significant
    program focus. Use a mix of data collection sources whenever

                       Steps in Selecting Performance
                                       Indicators cont.

     •   Step 1. Clarify the results statements.
     •   Good performance indicators start with good results
         statements that people can understand and agree
         on. Carefully consider the result desired. Review the
         precise wording and intention of the strategic
         objective, strategic support objective, special
         objective, intermediate result, critical assumption, or
         result supported by partners. What exactly does it

              Step 1. Clarify the results statements.

     •   Avoid overly broad results statements. Sometimes
         objectives and results are so broadly stated it is
         difficult to identify the right performance indicators.
         Instead, specify those aspects believed to make the
         greatest difference to improved performance. For
         example, rather than using abroad results statement
         like "improved capacity" of a host country institution,
         clarify those aspects that program activities
         emphasize. For example, improved personnel
         recruitment process, or improved management

      Be clear about what type of change is implied. What
       is expected to change -- a situation, a condition, the
       level of knowledge, an attitude, a behavior? For
       example, changing a country's law about voting is
       very different from changing citizens' awareness of
       their right to vote, which again is different from their
       voting behavior. Each type of change is measured by
       different types of indicators.
      Also, clarify whether the change being sought is an
       absolute change, a relative change, or no change.
      Absolute changes involve the creation or introduction
       of something new.
      --Relative changes involve increases, decreases,
       improvements, strengthening or weakening in
       something that currently exists, but at a higher or
       lower level than is considered optimum.

                    Step 2. Develop a List of Possible

      There are usually many possible indicators for any
        desired outcome , but some are more appropriate and
        useful than others. In selecting indicators, don't settle too
        quickly on the first that come most conveniently or
        obviously to mind. A better approach is to start with a list
        of alternatives, which can then be assessed against a
        set of selection criteria.
      To create the initial list of possible indicators, tap the
        following sources:
     - internal brainstorming by the strategic objective team
     - consultations with experts in the substantive programe
     - Experience of other operating units with similar indicators.

                         Step 3. Assess Each Possible

     •   Assess each possible indicator on the initial list.
         Experience suggests using seven basic criteria for
         judging an indicator's appropriateness and utility.
         When assessing and comparing possible indicators,
         it is helpful to use a matrix with the seven criteria
         arrayed across the top and the candidate indicators
         listed down the left side. With a simple scoring
         scale, for example 1-5, rate each candidate
         indicator against each criterion. These ratings will
         help give an overall sense of the indicator's relative
         merit, and help in the selection process. However,
         apply this approach flexibly and with judgment,
         because all seven criteria may not be equally
                            Step 4. Select the "Best"
                            Performance Indicators.

     •   The next step is to narrow the list to the final
         indicators that will be used in the performance
         monitoring system. They should be the optimum set
         that meets the need for management-useful
         information at a reasonable cost. Be selective.
         Remember the costs associated with data collection
         and analysis. Limit the number of indicators used to
         track each objective or result to a few (two or three).
         Select only those that represent the most basic and
         important dimensions of our aims.

         Seven criteria for assessing performance
     • 1. DIRECT.
      A performance indicator should measure as closely as
      possible the result it is intended to measure. It should not be
      pegged at a higher or lower level than the result being
      measured. For example contraceptive prevalence rate is a
      direct measure of the result increased use of family planning
      methods . But number of service providers trained would NOT
      be a direct measure of the result improved service delivery.
      Just because people are trained does not necessarily mean
      they will deliver services better.

         Seven criteria for assessing performance
                      indicators cont.

     •   2. OBJECTIVE. An objective indicator has no
         ambiguity about what is being measured. That is,
         there is general agreement over interpretation of the
         results. It is both un dimensional and operationally
         precise. To be one-dimensional means that it
         measures only one phenomenon at a time. Avoid
         trying to combine too much in one indicator, such as
         measures of both access and Operational precision
         means no ambiguity over what kind of data would
         be collected for an indicator.

        Seven criteria for assessing performance
                     indicators cont.

     3. ADEQUATE.
     Taken as a group, a performance indicator and its companion
      indicators should adequately measure the result in question. A
      frequently asked question is "how many indicators should be
      used to measure any given result?" The answer depends on
     a) the complexity of the result being measured,
     b) b) the level of resources available for monitoring
         performance, and
     c) c) the amount of information needed to make reasonably
         confident decisions.
     • However, avoid using too many indicators. Try to strike a
       balance between resources available for measuring
       performance and the amount of information managers need to
       make reasonably well informed decisions.
                             Seven criteria for assessing
                                 performance indicators

         quantitative indicators are numerical (number or
         percentage of dollar value, tonnage, for example).
         Qualitative indicators are descriptive observations (an
         expert opinion of institutional strength, or a description
         of behavior). While quantitative indicators are not
         necessarily more objective, their numerical precision
         lends them to more agreement on interpretation of
         results data, and are thus usually preferable. However,
         even when effective quantitative indicators are being
         used, qualitative indicators can supplement the
         numbers and percentages with a richness of information
         that brings a program's results to life.

              Seven criteria for assessing
                performance indicators

     Is aggregating people-level program results by
     gender, age, location, or some other dimension
     is often important from a management or
     reporting point of view. Experience shows that
     development activities often require different
     approaches for different groups and affect
     those groups in different ways. Disaggregated
     data help track whether or not specific groups
     participate in and benefit from activities
     intended to include them. Therefore, it makes
     good management sense that performance
     indicators be sensitive to such differences.
                            Seven criteria for assessing
                                performance indicators

     •   6. PRACTICAL.
         An indicator is practical if data can be obtained in a
          timely way and at a reasonable cost. Managers
          require data that can be collected frequently enough
          to inform them of progress and influence decisions.
          USAID operating units should expect to incur
          reasonable, but not exorbitant, costs for obtaining
          useful performance information. A rule of thumb,
          given in the reengineering guidance, is to plan on
          allocating 3 to 10 percent of total program resources
          for performance monitoring and evaluation.

                           Seven criteria for assessing
                               performance indicators

     •   7. RELIABLE. A final consideration in choosing
         performance indicators is whether data of
         sufficiently reliable quality for confident decision-
         making can be obtained. But what standards of data
         quality are needed to be useful? The data that a
         program manager needs to make reasonably
         confident decisions about a program is not
         necessarily the same rigorous standard a social
         scientist is looking for. For example, a low cost
         minisurvey may be good enough for a given
         management need.


     • Thanks for your attention

       AS Days go , I learn more

                By: Bikokye Willy Kafeero
33                                          33

Shared By: