Your Federal Quarterly Tax Payments are due April 15th Get Help Now >>

Project Planning_ Monitoring and Evaluation - International AIDS by pengxiang

VIEWS: 6 PAGES: 53

									      Integrating Planning, Monitoring and
      Evaluation to Increase Effectiveness of
      your Interventions




Training organized on 12 December 2011 for HIDA (Ethiopia based NGO) by
      Laetitia Lienart , IAS Planning, Monitoring & Evaluation Expert
               PROGRAMME

          Introduction         



        Project planning    



   M&E scoping & framework
                        



Data management, analysis & use



        Proposal writing
INTRODUCTION
         M&E DEFINITION & ROLE
    M&E is a systematic process of collecting and analyzing 
                 information to accompany the
implementation of an action, project or programme and assess its 
               process, outcomes and impacts.
                                                 LEARNING /
                                                CONTINUOUS
ACCOUNTABILITY
                                               IMPROVEMENT


                          Institutional
                           Memory &
                           Evidence*




    M&E is not an audit nor does it seek to place blame.
         WHAT MAKES M&E
          SUCCESSFUL?
E   Ethical, Effective, Efficient

V   Valued

A   Accurate, Achievable, Accessible

L   Learning-oriented

U   Usable, Used, Useful

A   See above A

T   Timely, Transparent, Technically sound

I   Inclusive

O   Objective

N   Non-biaised
Another Success Key for M&E: Good Planning
                         Project 
                        planning 
                       (proposal)
       Reflecting 
         and                              M&E 
        making                           scoping
       changes




                                                  M&E 
    Reporting
                                               framework




            Information         Information 
           management            collection
             & analysis
PROJECT PLANNING
DEFINITION

§ Planning refers to the phase when you design your
  project. In other words, when you define its goal, objective, 
  expected results, activities you will have to implement to 
  deliver expected results and inputs/resources needed to 
  implement such activities. 

§ The way you design your project should follow a
  certain logic that shows in a clear way the links
  between all these elements. Such logic model is essential 
  for results-based management and for a successful M&E 
  which needs to have a clear understanding of what the 
  project is supposed to achieve and how.  


              Failing to plan is Planning to fail!
LOGIC MODEL

 § Also called ‘’theory of change’’ or ‘’results chain”
 § Various types of logic model exist and all have pros & 
   cons (just choose the one which is the best adapted to 
   your needs and available resources):
    ü Logframe
    ü Outcome mapping
    ü Outcome chain
    ü Social framework
    ü Etc.
THE RESULTS CHAIN
The logframe (project framework) is an expression of the “Results Chain”
 – the results you expect the project to achieve. The box below provides
       an example of a Results Chain and how it aligns with the LF.
        WHAT IS A LOGFRAME?

     A  table  giving  a  clear  and  synthetic  picture  of  the  project impact,
HERE outcome, outputs and inputs,  their  respective indicators and 

     sources of verification as well as assumptions.
            The  LF  process  helps  guide  the  planning  of  a  journey  from 
            where we are now, HERE, to where we want to go, THERE.
            A - Where do we want to be?
            What are our expected impact and outcome? 
            B - How will we get there?
            What are the outputs do we have to deliver? 
            C - What may stop us getting there?
            What are the risks and how can we manage them? 
            What assumptions are we making? 
            D - How will we know if we’ve got there?
            What are our indicators, baseline, milestones and targets? 
            What source of information (evidence) do we need? 
            E – What do we need to get there?
THERE       What detailed resources and related budget are needed? 
STEP 1 - DEFINE THE IMPACT (GOAL)


 § This  is  the higher order objective,  the longer term impact, that
   the project will contribute to. 

 § It  defines  the  overall  “big  picture” need or problem being
   addressed; it expresses the justification of what is planned.

                                  TIPS

    ü Have only one Impact/Goal statement.

    ü Example: Decreased AIDS mortality rate in the Nbiya region.
 STEP 2 - DEFINE THE OUTCOME
    (PURPOSE/OBJECTIVE)
§ This  describes  the specific and immediate results (outcomes) of  the 

  project. 

§ It should not be entirely deliverable, i.e. fully within the project manager’s 
  control. If it is deliverable, then it should be an Output*. 

                                       TIPS

ü Have only one Outcome. If you think you have more, then you may need 
  more than one logframe; or your multiple outcomes are in fact indicators of 
  a single outcome or lower outputs. 
ü Example: Improved access to HIV/AIDS treatment in the Nbiya region.
      STEP 3 - DEFINE THE
‘RESULTS/PRODUCTS’ OR OUTPUTS
 § The  Outputs  describe what the project will deliver in order to
   achieve the Outcome/Purpose.  They  are  the  results  that  the 
   project must deliver in the control of the project manager. 

 § Examples: a)  Medical  infrastructures  rehabilitated/reinforced  in  the 
   Nbiya  region;  b)  HIV/AIDS  awareness  of  Nbiya  region’s  inhabitants 
   raised; c) Health care workers of the Nbiya region adequatly trained 
   on HIV/AIDS; d) Partnerships between Nbiya’s government and drug 
   supplier(s) initiated; etc.   

 § Typically there are between 2 – 6 Outputs; any more than that and 
   the logframe will become over-complicated.
STEP 4 – DEFINE THE ACTIVITIES

§ The Activities describe what actions will be undertaken to achieve
  each output. 

§ Examples: needs  assessment,  recruitment  of  experts/consultants, 
  design of materials, development of training programme, selection of 
  participants/trainers/suppliers, organization of coordination meetings, 
  distribution  of  awareness  materials,  implementation  of  works, 
  procurement of equipment & supplies, etc.

§ Unlike  the  previous LF template,  activities  do  not  have  to  be 
  captured  in  the  LF. You  can  report  them  in  a  separate  table  and 
  depending on the donor requirements, specify for each of them the 
  milestones, risks and staff responsible (see example next slide).
ACTIVITIES LOG TEMPLATE
STEP 5 – TEST THE LOGIC FROM THE
       BOTTOM TO THE TOP

 § Use the IF/THEN test to check cause and effect,  reading from the

   bottom up: 


    Ø If we do these activities, then this output will be delivered. 


    Ø If we deliver these outputs, then this outcome will be achieved. 


    Ø If the outcome is achieved, then this will contribute to the impact. 
STEP 6 – UNDERTAKE A RISK
         ANALYSIS
                         IDENTIFY THE RISKS
§ Taking all the activities needed for Output 1, ask the question: ‘if we
  complete these Activities successfully, then what can stop us
  delivering Output 1?’ .  Repeat for all the other Outputs taking each 
  Output and its associated activities in turn.

§ Taking  all  the  Outputs  together,  ask  the  question:    ‘if we deliver all
  these Outputs successfully, then what can stop us achieving our
  Outcome?’  

§ Now ask the question: ‘if we are achieve our Outcome
  successfully, then what can stop us contributing to the Impact?’ 
     STEP 6 – UNDERTAKE A RISK
          ANALYSIS (cont.)
               ANALYSE AND RATE EACH RISK
§ What is its likely impact? high, medium or  low. 

§ What is its likely probability? high, medium or  low.  You may at this point 
  decide to hereafter disregard insignificant risks. 

§ Discuss & agree possible mitigation measures;  transfer them into the 
  Activities log template (i.e. extra activities).   

§ Example: 

   Ø risk: decrease in the # of health care workers affected to the Nbiya region 

   Ø mitigation measure/new activity:  advocacy/lobbying to the relevant 
     authorities & stakeholders involved in health care workers management 
     STEP 7 – FORMULATE THE
          ASSUMPTIONS
§ Assumptions  are what remains after the mitigation measures have 
  been  put  in  place  (even  if  mitigation  measures  are  successful,  it  is 
  unlikely you can remove the risk completely).

§ Assumptions are external factors which could affect the success of the 
  project but over which the project manager has no direct control. 

§ Example: ‘’Strong  commitment  and  continuous  support  of  the  Nbiya’s 
  authorities towards combatting HIV/AIDS‘’

§ Logic  test: once Activities have been carried out, and if the
  Assumptions at this level hold true, Outputs will be delivered. Test 
  the same logic for the upper levels. 
STEP 8 – FORMULATE INDICATORS

§ Once your logic model is established, it is important to think about which 
  information you will need  to collect  to measure your project 
  performance. 

§ Indicators are performance measures, which tell us what we are going
  to measure not what is to be achieved.

§ Indicators should be specific, usable and easy to measure. The basic 
  principle is that “if you can measure it, you can manage it’’.

                                    TIPS

    Do not start from scratch and use indicators that have been used &
    tested in your field, such as: 
    http://www.unaids.org/en/resources/presscentre/featurestories/20
    11/november/20111118reportingguidelines/
Indicators at impact level

§ Indicators should be “impact” measures. They  should only 
  state what will be measured – i.e. they should not include 
  elements of the baseline or target. 


            Under-five AIDS mortality rate falls by at least 5% 
            points by 2012 (to be avoided)


            Under-five AIDS mortality rate (recommended)
Indicators at outcome level

§ Outcome indicators demonstrate the changes which take 
  place as a result of the organization’s work, and show 
  progress towards meeting the outcome.
§ As with the Impact, indicators should not include elements of 
  the baseline or target. The rule is that each Indicator you 
  choose to measure your objectives must be verifiable by 
  some means. If not, you must find another indicator. 


           50% increase in ARV uptake rate in 
           targeted areas by 2012 


           ARV uptake rate in targeted areas 
Indicators at output level


§ Output indicators demonstrate the work the organization
  does and show progress towards meeting objectives. 

§ Best Practice suggests a maximum of three Indicators per 
  Output 

       Output indicators can be set for:
       ü Quantity: Number of HIV testing services
       ü Take-up: Number of people using HIV testing services
       ü Accessibility: Type of people using HIV testing 
         services
       ü Quality: Level of user satisfaction with HIV testing 
         services
       ü Cost
        
STEP 9 – DEVELOP BASELINE

All projects SHOULD have baseline data at Impact,
Outcome & Output level before being sent for approval

§ To make sure the information– in particular baseline data - 
  is robust, you might need to commission new analysis. 
  However, this should be avoided wherever possible: you
  are encouraged to draw data and analyses from
  existing sources (for example, from UNAIDS, a partner 
  government statistical office, etc.). 
§ What is important is that the data and analysis are current, 
  consistent and as accurate as can be reasonably achieved, 
  and disaggregated by sex where appropriate.
STEP 10 – MILESTONES & TARGETS
§ Indicators  are  a  means  by  which  change  will  be  measured 
  while milestones & targets are respectively mid-term and end 
  “goals”.  
§ Milestones are  intended  to  help  you  track  progress,  and 
  therefore  you  should  consider  the  specific  trajectory  of  your 
  project,  taking  into  account  all  relevant  factors,  including  the 
  sequencing  of  activities  and  the  release  of  data  from  the 
  source of monitoring information*.
§ Targets must be Specific, Measurable, Achievable, Relevant 
  and Time bound (i.e. SMART), thereby indicating the desired 
  result at the end of the project. 
§ In line with the guidance for Indicators and baselines, targets 
  must be included wherever baseline data is available and 
  should be disaggregated by sex where appropriate. 
STEP 11 – IDENTIFY THE SOURCE

 § When  you  choose  an  indicator,  it  is  important  to  decide 
   how  you  will  measure.  It  is  what  we  usually  refer  to  as 
   « Evidence » or « Verification ». 

 § This is a vital stage of the initial planning that is often 
   overlooked. 

 § It  should  be  considered when you formulate your
   indicators.
Some typical sources of verification


§ National and international statistics 

§ Project records, reviews and reports

§ Minutes of meetings and attendance lists 

§ Stakeholder feedback & focus groups 

§ Surveys and reports 

§ Newspapers, radio and TV recordings, photographs, satellite imagery 

§ External evaluation reports, training evaluation questionnaires, etc.
   STEP 12 – COMPLETE THE ROW
   LINKING THE INDICATOR,
   BASELINE, MILESTONES, TARGET
   & SOURCE
Indicator          Baseline            Milestone             Milestone             Target
                   2011                2012                  2013                  2014



Number of         0 Doctor            5 Doctors (2 F; 3 M)   10 Doctors (5 F; 5    15 Doctors (8 F; 7 
                  4 Nurses (1 F; 3 M) 8 Nurses (4 F; 4 M)    M)                    M) 
health                                                       12 Nurses (6 F; 6     20 Nurses (10 F; 10 
professionals at                                             M)                    M)
selected  
District 
Hospitals         Source: 
trained on HIV  • NGO training reports (quarterly)
testing           • Hospital records (annual)
          STEP 13 – DEFINE INPUTS 

§ This  level  indicates  the inputs  and  related costs needed to fund
  the project activities, as  well  as  the breakdown of costs by
  donors (in case the project is funded by several donors). 

§ It  also  shows  the staff who  will  be  involved  in  the  project 
  implementation, and if possible the proportion of their time allocated 
  to the project

Inputs ($)                            Costs        Donor 1         Donor 2
                                                   Share %         Share %
• Contract to rehabilitate hospital   US$5,000 50%                 50%
• ARVs                                US$3,000 20%                 80%

Inputs (HR)
1. Health advisor
2. Communications officer
LOGFRAME TEMPLATE
LOGFRAME TEMPLATE (cont.)
MAKING CHANGES TO THE LF

 § Logframes are dynamic, subject to change throughout 
  the active life of the project to which they refer. 



 § Changes to a logframe are normally made during a 
  formal review, or in response to circumstances, and 
  should be done in close consultation with all key
  stakeholders. 
RECOMMENDED WEBSITES TO
LEARN MORE ON THE LF

http://www.dfid.gov.uk/Documents/publications1/how-to-guid-rev
-log-fmwk.pdf

http://www.dfid.gov.uk/Documents/funding/gpaf/Logframe-how-
to-note2011.pdf

http://forexsoftwarebusiness.com/tags/guidance-on-using-the-
revised-logical-framework.html
                   EXERCISE

§ Subject: the  Nbiya  region  is  severely  affected  by  HIV/AIDS,  with 
  children and women among the most HIV infected populations. You 
  wish  to  apply  for  a  tender  launched  by  a  major  donor  in  order  to 
  address HIV/AIDS related health problems in that region.



§ Methodology:  each  working  group  is  expected  to  follow  all  the 
  above-presented steps to eventually design and present a complete 
  LF with 2 outputs.  Use the fact sheet as main background.
M&E SCOPING AND
  FRAMEWORK
M&E SCOPING
            Have a focus for your M&E because
             you cannot measure everything
§ Possible focus:
   ü Particular project or activity
   ü Particular target group
   ü Inputs, process, outputs, outcomes (changes) and/or impact
   ü Particular evaluation questions


§ Criteria to define your M&E scope:
   ü Info needed for further funding 
   ü You know exactly how the info will be used and for what
   ü Level of efforts required to collect the information (always consider 
     the info which is already being collected)
M&E FRAMEWORK

         Indicators and their sources of evidence
        constitute the basis of an M&E framework

§ Other elements that should be included:
  ü Staff responsibilities
  ü Resources needed and Budget
  ü Frequency of data collection
INFORMATION COLLECTION TOOLS

§ Sometimes, sources of evidence are not immediately 
  available and you will have to create a tool to collect the 
  required information (e.g. survey, interview, etc.).

                                  TIPS
   ü Turn the indicators into questions
      v Indicator: type of food eaten 
      v Question: how many times do you eat meat per week?


   ü Select the best tool to collect this information.
     Criteria for selection tools:
      v Reliability
      v Validity
      v Ease of collection & analysis
DATA MANAGEMENT,
ANALYSIS AND USE
§ Management:
   ü How to store information?
       v Paper
       v Simple database (e.g. Excel) or sophisticated database (e.g. Access)
       v Special database: standard or tailored to your needs (financial and 
         training requirements)
§ Analysis:
   ü Data cleaning and harmonization
   ü Coding if necessary
   ü Interpreting*
   ü Recommending
§ Use:
   ü Sharing results with staff, users and other key stakeholders
   ü Making decisions on what must we do differently to increase our
     success
PROPOSAL WRITING
Some tips that should increase your
chances to get money …


                 $ $ $ $ $ $
Process overview:
1. Draft                          3. Finalize                       5. Develop 
proposal:                         proposal:                         work plan: 
project team                      project team                      project team
                                  leader




     1                 2                 3               4                 5


                 2. Review                        4. Disseminate 
               draft proposal:                       proposal: 
                PME expert                         project team 
                                                      leader
            Key requirements


§ Involve all staff concerned and other stakeholders as needed

§ START  designing  the  project  framework  &  test  its  validity/logic  with 
  main stakeholders

§ Write the narrative, making sure it matches the project framework (LF)

§ Ensure consistency throughout the proposal*

§ Ask colleagues/partners to read it & adjust it if needed

§ Make sure all teams are properly informed of what they are supposed 
  to deliver as part of the new project, including reporting duties
      PROPOSAL STRUCTURE

§ Background and rationale

§ Project framework (Logframe)

§ Activities

§ Implementation/Coordination (mechanisms)

§ Risks

§ M&E

§ Resources & Budget

§ Reporting

§ Appendices (detailed budget, ToRs, etc.)
                           Background and rationale
                                         
This section should describe clearly:
ü The problem that the project aims to address (in other words, why you 
   propose this project) and if applicable, the extent to which the project 
   will contribute to the national targets and MDGs.
ü Which key stakeholders were involved in the design of this project.
ü If the project is a follow-up action of a previous project, intervention, 
   workshop or other event. In that case, provide an overview of this 
   previous project, intervention, workshop or other event (you may use 
   Appendixes to provide further information). 
ü The role of the organization in this project (leadership, partnership, 
   representation or affiliation function) and why it is strategically 
   positioned to lead or take part to this project (in other words, highlight 
   your comparative advantages/added values).
ü The extent to which the project fits into the organization’s strategy.
ü In which way(s) your key constituencies will be engaged in this project. 
                                         
           Project framework 


ü Insert the logframe

ü Because activities and inputs are described in a separate 
  section of the proposal, the logframe may only includes the 
  3 highest levels (i.e. Impact, Outcome and Outputs).
                                    Activities
                                           
This section should describe the main activities that will be implemented to 
deliver the expected results. 
                                           
                                      TIPS
ü Use bullet points (one per activity) or the Activities Log Table.
ü Match activities with expected outputs as stated in the LF. 
ü Don’t provide too many details (they will be part of the detailed project 
   work plan you will define at a later stage). For example, instead of 
   detailing all sub-activities you will implement to organize a training (such 
   as assess needs of participants, develop curricula, identify speakers, 
   book training venue and accommodation, etc.), just write as a main 
   activity: Organize a training on [topic] for [target group]. 
ü Once the proposal is approved, develop a project work plan or Gantt 
   chart (Excel sheet) with detailed activities and sub-activities.
                                           
                      Implementation/Coordination
                                        
This section should describe clearly: 

ü Who will be the project leader and what will be his/her 
  role/responsibilities.
ü Who will be member of the project team and what will be the 
  role/responsibilities of each member. 
ü Role/responsibilities of your Communication team (if not part of the 
  project team). 
ü Role/responsibilities of the staff who are not in the project team and do 
  not belong to the Communication team (only if relevant to the project). 
ü Mandate and composition of the project steering committee, advisory 
  group, working group or any other kind of consultative body set up for 
  this project. 
ü Coordination mechanisms put in place to create synergies and avoid 
  overlapping with other similar initiatives (only if relevant to the project). 
                                        
                                Risks
                                    
This section should describe any external factors (i.e. risks)
 that may have a negative impact on the project and cause its partial or 
total failure. 

It should also describe how you will monitor and mitigate them.
                                     
                                 M&E

This section should describe clearly:
ü Role/responsibilities of the project team.
ü Role/responsibilities of the M&E officer/team.
ü What the evaluation will focus on (i.e. process, outcomes and/or 
   impact). You may already specify your key evaluation questions:
        • What worked well? 
        • What did not work so well? 
        • What could be improved in the design phase? 
        • What could be improved in the implementation phase? 
        • What follow-up actions are needed to make project results 
           sustainable? 
                              M&E (cont.)

Other key evaluation questions:
        • Are the [items] delivered by the project used? 
        • Are the [items] delivered by the project used in the most 
           efficient and correct way? 
        • Who are the users? etc.

This section should also describe:
v Which M&E methodologies are envisaged (e.g. tracking records, 
   surveys, interviews, observations, desk reviews, etc.). 

                                    TIPS
ü Make sure the proposed methodologies will allow measurement of 
  all indicators listed in the project framework. 

ü If you plan an impact evaluation (i.e. assessment of changes 
  resulting from the project), make sure the budget will include 
  enough funds for this and include it in the project timeline (it should 
  be done at least 1 year after the end of the project).
                        Resources and Budget
                                       
This section should describe human resources that will be mobilized for 
the project and all necessary inputs (goods & services).
Provide if possible an outline of the budget (breakdown of total amount 
by budget line, including a certain % of the total budget for 1) overheads, 
2) M&E and 3) communications, if relevant).
                                       

                               Reporting
                                    
This section should specify:
    ü The frequency of progress and final report(s).
    ü The outline/structure of progress and final report(s) if possible.
    ü Try to use you organization’s standard budget lines whenever 
       possible.
Other suggestions to improve HIDA’s 
 visibility, improve work practices & 
              build networks
• Endorse the Code of Good Practice for 
  NGOs responding to HIV/AIDS 
  http://www.hivcode.org/ 
• Join relevant consortiums
• Take part to online discussions, groups, 
  communities of practice
• Subscribe to relevant email lists
• Attend relevant conferences, meetings, 
  workshops, etc.

								
To top