VIEWS: 2 PAGES: 17 POSTED ON: 8/11/2012
• Development Impact Evaluation • in Finance and Private Sector •1 Arianna Legovini Head, Development Impact Evaluation Initiative The World Bank Steps in Implementing an Impact Evaluation Workshop Development Impact Evaluation in Finance and Private Sector Dakar February 2010 With generous support from Gender Action Plan Steps Build capacity Feed results Set learning into policy agenda Design impact Analyze data evaluation Plan for IE Collect follow implementati up data on Roll out Conduct intervention baseline Step 1. Build capacity for IE Objectives: Become informed consumers of impact evaluation Set the learning agenda Use it as an internal management tool to improve program over time How Training Learning by doing Step 2: Set learning agenda Objective: Get answers to relevant policy and operational questions How? Dialectic discussion involving key policy makers and program managers Technical facilitation to structure framework of analysis Focus on few critical policy (what) and operational (how to) questions Discuss agenda with authorizing environment and constituencies Cont. 2: Questions Operational: design-choices of program ▪ Institutional arrangements, Delivery mechanisms, Packages, Pricing/incentive Management purpose Use random trials to test alternatives Measure effects on short term outcomes (months) ▪ take up rates, use, adoption Scale up better implementation modalities Policy: effectiveness of program Accountability purpose Use random assignment or next best method Measure effects medium to long term Scale up/down, negotiate budget, inform Step 3: Design IE Exploit opportunities: Will roll-out take time? Is the budget allocated insufficient to cover everyone? Are there quantitative eligibility rules? If the program has universal access, does it have imperfect take- up? Set scale: Pilot to try out an intervention Large scale w. representative sample: more costly, externally valid Large scale with purposeful sample: less costly, indicative Do power calculation to determine minimum sample size Cont. Step 3 Select “best” method for each of your questions Feasible Requires least assumptions Ethics No to deny access to something for which there is irrefutable evidence Test interventions before scale up when you have no solid evidence Step 4: Planning implementation Budget cost items ▪ Staff time (PROJECT FUNDS) and training (DIME) ▪ Analytical services and field coordination (DIME) ▪ Data collection (PROJECT FUNDS) ▪ Discussions and dissemination (shared) Timeline ▪ Use it to organize activities, responsibilities and work backwards to know when to start Team ▪ Government (program manager, economist/statistician); WB Project team (Task manager or substitute); Research team (Lead researcher, co-researchers, field coordinator); Data collection agency Step 5: Assignment to treatment and control The smallest unit of assignment is the unit of intervention Credit: individual or group SME services: enterprise Municipal registration system: municipality Create listing of treatment units assigned to the intervention and control units that are not Explain assignment to responsible parties to avoid contamination Step 6: Baseline data Quality assurance : IE team (not data collection agency) to Design questionnaire and sample Define terms of reference for data collection agency Train enumerators Conduct pilot Supervise data collection Do not collect data before your design is ready and agreed Cont. Step 6: Baseline data Contract data collection agency Bureau of Statistics: Integrate with existing data Ministry concerned: Ministry of Agriculture/Water Resources/Rural Development Private agency Analyze baseline data a feed back into program and evaluation design if needed Check for balance between treatment and control group: do they have similar average characteristics? Step 7: Roll out intervention Conduct intensive monitoring of roll-out to ensure evaluation is not compromised What if treatment and control receive the intervention? What if all the control group receive some other intervention? Step 8: Follow-up data Collect follow-up data with the same sample and questionnaire as baseline data At appropriate intervals Step 9: Estimate program effects Randomization: compare average outcomes for treatment and control group Other methods: Use relevant econometric analysis , test assumptions, check robustness Are the effects statistically significant? Basic statistical test tells whether differences are due to the program or to noisy data Are they significant in real terms? If a program is costly and its effects are small, may not be worthwhile Are they sustainable? Is the trajectory of results sustained? Step 10: Discuss, Disseminate and Feedback into policy Are you thinking about this only now? Discuss what are the policy implications of the results What actions should be taken How to present them to higher ups to justify changes/budget/scale up? Talk to policy-maker and disseminate to wider audience If no one knows about it, it won’t make a difference Make sure the information gets into the right policy discussions Real time discussions Workshops Reports Policy briefs Final step: Iterate What do you need to learn next?
Pages to are hidden for
"steps"Please download to view full document