Monitoring Evaluation Unicef

Document Sample
Monitoring Evaluation Unicef Powered By Docstoc
					Monitoring & Evaluation in NIE
         Module 20




                           3-Nov-12   1
          Learning objectives
• Be familiar with the basic concepts and main
  characteristics of monitoring and evaluation
• Understand the differences between various
  kinds of evaluations
• Explain the different kinds of indicators
• Describe the very basics of a ‘log frame’
• Optional: Be familiar with the monitoring and
  evaluation of CMAM interventions

                                          3-Nov-12   2
3-Nov-12   3
The project cycle            Disaster




                                    ASSESSMENT
   EVALUATION




                Monitoring


                                    PROGRAMME
 IMPLEMENTATION                       DESIGN




                                           3-Nov-12   4
Monitoring &Evaluation




                         *
   M&E                           performance
                                      efficiency
                                                   outputs
               effectiveness
                                                                      appropriateness
                                               outcomes
           Quantitative indicators
                                                             Qualitative indicators
      impact             target               Logframes

                               DO NO HARM               assessment
coverage
                                      INPUTS
                                 connectedness
       accountability
                                        timeliness
            A WASP NEST………?                                                      *
           Definition
Monitoring
‘The systematic and continuous assessment of the
progress of a piece of work over time….’

‘To continuously measure progress against programme
objectives and check on relevance of the programme’

It involves collecting and analysing data/information

            It is NOT only about PROCESS

                                                        *
       Purpose of monitoring
• to document progress and results of project

• to provide the necessary information to
  Management for timely decision taking and
  corrective action (if necessary)

• to promote accountability* to all stakeholders
  of a project (to beneficiaries, donors, etc)


                                                *
Information collected for monitoring
must be:
• Useful and relevant
• Accurate
• Regular
• Acted upon
• Shared
• Timely


                                   *
Monitoring is an implicit part of an evaluation.

It is often done badly:
 – Routine data collection not done routinely!
 – Data collection done poorly
 – Information not processed/used in a timely manner
 – Focus only on process indicators and neglecting (lack
    of) preliminary impact


                                                           *
 Can you give examples of Monitoring
        in your current work?


For example
- From a CMAM programme?
- From a Micronutrient programme?
- From a General Food Distribution?
- From a Health programme?
- From a Livelihoods programme?



                                       *
                    Monitoring
• Monitoring compares intentions with results
• It guides project revisions, verifies targeting criteria
  and whether assistance is reaching the people
  intended.
• It checks the relevance of the project to the needs.
• It integrates and responds to community feedback
• It enhances transparency and accountability
            Difference between

Monitoring of


• Process/activities

• Impact/results




                                 *
The project cycle            Disaster




                                    ASSESSMENT
   EVALUATION




                Monitoring


                                    PROGRAMME
 IMPLEMENTATION                       DESIGN




                                                 *
Why would you
do an evaluation
of a programme?




                   *
      Definitions
Evaluation
The aim is to determine relevance and fulfilment of
objectives, as well as efficiency, effectiveness, impact
and sustainability of a project.


It involves the objective assessment of an
ongoing or completed project/programme, its
design, implementation and results.

                                                       *
There has been an increased
focus on evaluation of
humanitarian action as part
of efforts to improve quality and
standards




                                    *
             Evaluation

It aims to

– Improve policy and practice
– Enhance accountability




                                *
    Evaluations are done when /
              because:
– Monitoring highlights unexpected results
– More information is needed for decision making
– Implementation problems or unmet needs are identified
– Issues of sustainability, cost effectiveness or relevance
  arise
– Recommendations for actions to improve performance
  are needed
– Lessons learning are necessary for future activities
                  Evaluations
• Evaluation involves the same skills as assessment
  and analysis
• Evaluation should be done impartially and ideally
  by external staff
• Evaluation can also occur during (e.g. mid-term)
  and after implementation of the project


              One of the most important sources
              of information for evaluations is
              data used for monitoring
                                             3-Nov-12       20
                                                        *
           The OECD-DAC criteria
          Organisation for Economic Co-operation and Development


• The Development Assistance Committee
  (DAC) evaluation criteria are currently at the
  heart of the evaluation of humanitarian
  action.

• The DAC criteria are designed to improve
  evaluation of humanitarian action.



                                                                   *
                         Evaluation looks at

• Relevance/Appropriateness: Doing the right thing in the right way at
  the right time.
• Connectedness (and coordination): Was there any replication or gaps
  left in programming due to a lack of coordination?
• Coherence: Did the intervention make sense in the context of the
  emergency and the mandate of the implementing agency? Are their
  detrimental effects of the intervention on long run?
• Coverage: Who has been reached by the intervention, and where:
  linked to effectiveness?
• Efficiency: Were the results delivered in the least costly manner
  possible?
• Effectiveness: To what extent has the intervention achieved its
  objectives?
• Impact: Doing the right thing, changing the situation more
  profoundly and in the longer-term.

                                                                   *
                           Evaluation looks at


• Relevance/Appropriateness: Doing the right thing in the right way at the right
  time.
• Connectedness (and coordination): Was there any replication or gaps left in
  programming due to a lack of coordination?
• Coherence: Did the intervention make sense in the context of the emergency
  and the mandate of the implementing agency? Are their detrimental effects of
  the intervention on long run?
• Coverage: Who has been reached by the intervention, and where: linked to
  effectiveness?
• Efficiency: The extent to which results have been delivered in the least costly
  manner possible.
• Effectiveness: The extent to which an intervention has achieved its objectives –
• Impact: Doing the right thing, changing the situation more profoundly and in
  the longer-term.




                                                                               *
Example on General Food Distribution

• Relevance/Appropriateness: Doing the
  right thing in the right way at the right
  time.
  Was food aid the right thing to do, not
  cash?

• Connectedness: Are their detrimental
  effects of the intervention on long run?
  Did food aid lower food prices? Did local
  farmers suffer from that?
• Coverage: Who has been reached by the
  intervention, and where: linked to effectiveness?
  Were those that needed food aid indeed
  reached?

• Efficiency: Were the results delivered in the
  least costly manner possible?
  Was it right to import the food or should it
  have been purchased locally? Could the results
  have been achieved with less (financial)
  resources? Food aid was provided, would cash
  have been more cost-effective?
• Effectiveness: To what extent has the
  intervention achieved its objectives?
  Did food aid avoid undernutrition? (assuming it
  was an objective)

• Impact: Doing the right thing, changing the
  situation more profoundly and in the longer-
  term.
  Did the food aid avoid people becoming
  displaced? Did the people become dependent
  on food aid?
• Impact:
- Very much related to the general goal of the
  project
- Measures both positive and negative long-term
  effects, as well as intended and unintended
  effects.
  GFD: did it lower general food prices with long-term economic
  consequences for certain groups ? Were people that received food
  aid attacked because of the ration? (therefore more death…?)
- Need for baseline information!!!!
  (to measure results against….)
To evaluate projects well is a real skill!


         And you often need a team…




                                         *
       M&E in emergencies?




Any project without Monitoring and/or Evaluation is a
BAD project

                                                        *
*
          The “M” and the “E”…
                       Monitoring        Evaluation

Primary use of      Project           Accountability
the data            management        Planning (future
                                      projects)
Frequency of data   Ongoing           Periodic
collection
Type of data        Info on process   Info on effects
collected           and effects
Who collects the    Project staff     External
data                                  evaluators
    Evaluations in Humanitarian Context
• Single-agency evaluation (during/after
  project)
• There is an increasing move towards:
  – Inter-agency evaluations: the objective is to
    evaluate responses as a whole and the links
    between interventions
  – Real-time evaluations: carried out 8 to 12 weeks
    after the onset of an emergency and are processed
    within one month of data collection
            Real-time evaluations (1)
• WHY?
  Arose from concern that evaluations came too late to affect
  the operations they were assessing
• Various groups of organizations aim to undertake real-time
  evaluations
• Same purpose as any other evaluation
• Common characteristics:
   – Takes place during the course of implementation
   – In a short time frame




                                                          *
           Real-time evaluations (2)
• It is an improvement-oriented review; it can be regarded
  more as an internal function than an external process.
• It helps to bring about changes in the programme, rather
  than just reflecting on its quality after the event.
• A real-time “evaluator” is a “facilitator”, working with staff
  to find creative solutions to any difficulties they encounter.
• It helps to get closer to the people affected by crisis, and
  this enables to improve accountability to ‘beneficiaries’.




                                                              *
 Monitoring & Evaluation systems
• Main components of M&E systems:
  – M&E work plan for data collection and analysis,
    covering baseline, on-going M&E
  – Logical framework, including indicators and
    means/source of verification
  – Reporting flows and formats
  – Feedback and review plan
  – Capacity building design
  – Implementation schedule
  – Human resources and budget

                                               3-Nov-12   35
   Examples of data collection methods
                for M&E
          Quantitative Methods                    Qualitative methods
Administering structured oral or written   Semi structured interviews e.g. key
interviews with closed questions           informant
Population based surveys                   Focus group discussion
Reviewing medical and financial records    Observing
Completing forms and tally sheets          Case studies
Direct measurement (anthropometry,
                                           Mapping, ranking, scoring
biochemical analysis, clinical signs)
Lot quality assessment                     Problem sorting, ranking
Focus on INDICATORS
                Indicators
• An indicator is a measure that is used to
  show change in a situation, or the progress
  in/results of an activity, project, or
  programme.

• Indicators:
  – enable us to be “watchdogs”;
  – are essential instruments for monitoring and
    evaluation.
  – are objectively verifiable measurements
      What are the Qualities of a Good
                 Indicator?
•   Specific
•   Measurable                The Sphere Project provides the most
                              accepted indicators for nutrition and
•   Achievable                food security interventions in emergencies:
                              see Module 21.

•   Relevant
•   Time-bound
            And there is also the SMART initiative….
            Standardised Monitoring and Assessment in Relief and Transition
            Initiative - interagency initiative to improve the M&E of humanitarian
            assistance
               Types of indicators
Indicators exist in many different forms:

 Direct
                       Direct indicators correspond precisely to results at
                        any performance level.
    Indirect /
                       Indirect or "proxy" indicators demonstrate the
      proxy
                        change or results if direct measures are not feasible.

                           Indicators are usually quantitative measures,
Qualitativ
   e
                            expressed as percentage or share, as a rate, etc.
     Quantitativ           Indicators may also be qualitative observations.
           e


   Global /
standardised                     Standardised global indicators are
                                  comparable in all settings.
           Locally
          developed              Other indicators tend to be context specific
                                  and must be developed locally.
Impact


Outcom
   e

Output


 Input
Impact   Related to Goal




Outcom   Related to Objectives
         (or Purposes)
   e

         Related to Outputs
Output


 Input    Related to
          Activities/Resources
         Related to Goal         Malnutrition rates
Impact                           amongst young
                                 children reduced

                                 % of young children
Outcom   Related to Objectives   getting appropriate
         (or Purposes)
   e                             complementary food

                                 X number of mothers
         Related to Outputs
Output                           know about good
                                 complementary food and
                                 how to prepare that

                                 Nutritional education to
 Input    Related to             mothers on
          Activities/Resources   complementary food




                                                       *
            What is a Log Frame?
The logical framework or logframe is an analytical tool
used to plan, monitor, and evaluate projects.



                      ?   ?   ?


                  ?

                                            Victim of a log frame?
Log Frames

     IMPACT

   OUTCOME




     INPUTS
                   Impact



     Outcome      Outcome            Outcome



Output   Output    Output   Output   Output    Output
                                                             Impact


                                                             Outcome


                                                             Output
                                                             Output

                                                             Output




                                                    INPUTS
         Other terms that can be found in a logframe:

• The means of verification of progress towards achieving the
  indicators highlights the sources from where data is collected.
  The process of identifying the means of verification at this stage
  is useful as discussions on where to find information or how to
  collect it often lead to reformulation of the indicator.

• Assumptions are external factors or conditions that have the
  potential to influence the success of a programme. They may be
  factors outside the control of the programme. The achievement
  of a programme’s aims depends on whether or not assumptions
  hold true or anticipated risks do not materialise.



                                                           3-Nov-12    48
               logical framework for M&E
Project description   Indicators             Source / mean of      Assumptions / risks
                                             verification


Goal
                             If the OBJECTIVES are produced, then this should
                                       contribute to the overall GOAL
Objectives /
outcomes                       If OUTPUTS/RESULTS are produced, then the
                                      OBJECTIVES are accomplished

Deliverable outputs
                                   If adequate ACTIVITIES are conducted, then
                                        OUTPUT/RESULTS can be produced

Activities
                            If adequate RESOURCES/INPUTS are provided; then
                                        activities can be conducted
         Activities versus Results
Completed activities are not results.
• e.g. a hospital was built, does not mean that
  injured and sick people can be treated in the
  hospital, maybe the hospital has no water and the
  beds have not been delivered.

Results are the actual benefits or effects of
completed activities:
• e.g. Injured and sick people have access to a fully
  functional health facility.

                                                        *
Example
Another Example…




                   *
                         Key messages
• The monitoring of nutrition interventions in emergencies is an
  integral part of saving lives and maintaining nutrition status of the
  affected population.
• Successful monitoring systems allow for improvements in
  interventions in ‘real time’.
• Evaluations are important tools for learning, assessing interventions,
  comparing the costs of the interventions and their impact. Essential
  evaluation parameters are: effectiveness; efficiency;
  relevance/appropriateness; impact and coverage
• Involving communities in M&E places the affected population at the
  heart of the response, providing the opportunity for their views and
  perceptions to be incorporated into programme decisions and
  increases accountability towards them.
• A common mistake of designing M&E systems is creating a
  framework which is overly complex. Always make an M&E system
  practical and doable.
• The logical framework or logframe is an analytical tool used to plan,
  monitor, and evaluate projects.
Monitoring for CMAM interventions


• Types of monitoring, e.g.
  – Individual case monitoring,
  – Programme / activities monitoring




                                        3-Nov-12   56
    Individual monitoring for CMAM
• It is the basic follow up of cases in SFP / OTP /
  SC services:
  – Anthropometric / clinical assessment


• Tools for individual case follow up include:
  – Medical / nutrition and action protocols
  – Individual follow up card
  – Referral forms
  –…

                                               3-Nov-12   57
        Objectives of monitoring
            CMAM activities
• Assess service performance / outcomes
• Identify further needs
  – Support decision-taking for quality improvement
    (staffing, training, resources, site location,…)
• Contribute to the analysis of the general
  situation
  – Assessing the nutrition trends in the area



                                                 3-Nov-12   58
     Methods and tools for monitoring
          CMAM interventions
• Monthly / weekly reporting:
  • Reporting needs to be done per site (service unit)
    and compiled per area (district…) up to the
    national level

• Routine supervision

• External evaluations
  • Coverage surveys are one of the most important
    tools for evaluation of CMAM interventions


                                                 3-Nov-12   59
  Routine data collection for monitoring
          CMAM interventions
• Routine data is collected for specified time-periods:
   – Nb. of new admissions ,
   – Nb. of discharges (total and by category: cured, died,
     defaulted, non-recovered
   – Nb. of cases in treatment (nb. of beneficiaries
     registered at the end of the reporting time-period)

     Data on admissions should be disaggregated by
     gender

                                                  3-Nov-12    60
Category                  Criteria (Children 6 – 59 months)
                         MUAC <11.5 cm
                         or
New admissions for
                         W/H < -3 Z scores (WHO) or <70% of median (NCHS)
children 6 – 59 months
                         or
(or > 60 months but <130
                         Bilateral pitting oedema grade + or ++
cm height)
                         and
                         child is alert, has appetite, and is clinically well

Other new admissions      Carer refuses inpatient care despite advice


                          Child has previously defaulted and has returned to OTP
Returned Defaulter
                          (the child must meet admission criteria to be re-admitted).
                          A child is treated in OTP until discharge after meeting discharge
Readmissions/Relapses
                          criteria but relapses hence need for readmission
Transfer from inpatient
                          From in-patient care after stabilisation treatment
care (SC)

Transfer from OTP          Patients moved in from another OTP site


                                                                                  3-Nov-12    61
Category                Criteria (Children 6 – 59 months)
                        MUAC > 12.5cm and WFH > -2Z scores and no oedema for two
                        consecutive visits
Cured                    And
                        Child is clinically well

Defaulted               Absent for 3 consecutive visits


Died                    Died during time registered in OTP

                        Has not reached discharge criteria within four months of treatment
Non-Cured               Link the child to other programmes e.g. SFP. IYCF, GMP, targeted food
                        distributions

Transferred to SC       Condition has deteriorated and requires inpatient care


Transfer to other OTP   Child has been transferred to another OTP site


                                                                                3-Nov-12        62
         Monitoring of CMAM interventions:
          key indicators for SAM (Sphere)
• The proportion of discharges from therapeutic care should be:
   – Recovered > 75 %
   – Deaths < 10 %
   – Defaulter < 15 %
      They are primarily applicable to the 6–59 month age group,
      although others may be part of the programme.

• Distance: > 90 % of the target population is within less than one
  day’s return walk (including time for treatment) of the service /
  site.
• Coverage is > 50 % in rural areas, > 70 % in urban areas and >90
  % in camp situations

                                                            3-Nov-12   63
           Monitoring of CMAM interventions:
            key indicators for MAM (Sphere)
• The proportion of discharges from targeted SFP should be:
   – Recovered > 75 %
   – Deaths < 3 %
   – Defaulter < 15 %
   They are primarily applicable to the 6–59 month age group,
   although others may be part of the programme.

• Distance: > 90 % of the target population is within less than one day’s
  return walk (including time for treatment) of the programme site for
  dry ration SFP and no more than one hour’s walk for on-site wet SFP
• Coverage is > 50 % in rural areas, > 70 % in urban areas and > 90 % in
  a camp situation

                                                              3-Nov-12   64
    Additional data for monitoring CMAM
                interventions
Derived from routine
monitoring and other sources:      Sources of data:
• Average length of stay           • Registration books
• Average weight gain              • Individual follow up charts
• Relapse rate                     • Interviews and Focus group
• Distribution of admissions per     discussions
  type, per age, per origin…       • Observation, home-visits
• Causes of death                  • …
• Reasons for defaulting
• Investigation of non-recovery
  cases



                                                          3-Nov-12   65
        M&E for CMAM interventions:
                Supervision
Supportive supervision visits to sites are designed
to ensure / improve the quality of care offered by:
   • Identifying weaknesses in the performance of activities,
     taking immediate action and applying shared corrective
     solutions
   • Strengthening the technical capacity of health workers and
     motivating staff through encouragement of good practices
Supervisors and managers ensure that the
performance of activities and organization of the
services meet quality standards.

                                                        3-Nov-12   66
      Evaluation of SAM management
               interventions
• Effectiveness: programme performance with a strong
  focus on coverage
• Appropriateness: e.g. distribution and time of opening
  of treatment sites
• Connectedness: relates to the links with health system
  and shows levels of possible integration
• Cost-effectiveness has also been measured with
  various methods and showing high differences
  between contexts and different approaches


                                                 3-Nov-12   67
       M&E of CMAM interventions:
       population level assessments


• Community level assessment can be done
  through:
  – Repeated anthropometric surveys
  – Programme coverage




                                           3-Nov-12   68
     Evaluation of coverage for CMAM
• Coverage is one of the most important
  elements behind the success of the CMAM
  approach.
   – It is measured through studies using two main
     approaches:
      • The centric systematic area sampling (CSAS)
      • The Semi-Quantitative Evaluation of Access and
        Coverage (SQUEAC)
• Coverage should reach at least 90% of severe cases in
  camps situation, 70% in urban setting, 50% in rural
  setting (SPHERE standards)


                                                         3-Nov-12   69
     Evaluation of management of MAM
                interventions
• Same criteria as for all other interventions (relevance,
  efficiency, etc.)
• SFP evaluations are rarely shared, but evidence
  showed that defaulting and non-response are very
  common
• Needs for evaluating use of Ready-to-Use-
  Supplementary Food products in terms of efficiency:
  gain of weight, effect of defaulting, effect on easiness
  for beneficiaries, etc.



                                                    3-Nov-12   70

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:11
posted:11/3/2012
language:Unknown
pages:70