Docstoc

The Evaluation Framework

Document Sample
The Evaluation Framework Powered By Docstoc
					        Mid-Term Evaluation of the EFA Fast Track Initiative




           The Evaluation Framework
Evaluation Team Guidelines on Process and Methodology




                          January 2009
             Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


                                              Contents
     Abbreviations and Acronyms ____________________________________________________________ 5
     Preface_______________________________________________________________________________ 7

1.         Introduction and Overview ________________________________________________________ 9
       Purpose and structure of this document ____________________________________________________ 9
       The Mid-Term Evaluation of the FTI _____________________________________________________ 9
         Objectives ________________________________________________________________________ 9
         Stages and Outputs ________________________________________________________________ 10
         Timetable _______________________________________________________________________ 11
       Challenges _________________________________________________________________________ 12
       Emerging good practice for joint evaluation of a global programme ____________________________ 13

2.          The Evaluation Process __________________________________________________________        15
       Structure and Components of the Evaluation ______________________________________________     15
       Team Structure and Responsibilities _____________________________________________________     15
       Quality Assurance ___________________________________________________________________         20
          Applicable Quality Standards ________________________________________________________      20
          Quality Assurance Panel ____________________________________________________________       20
          Other Dimensions of Quality Assurance________________________________________________      20
       Communications ____________________________________________________________________           20

3.          Evaluation Methodology _________________________________________________________         21
       General Approach ___________________________________________________________________          21
       High Level Evaluation Questions _______________________________________________________       23
       The Programme Logic ________________________________________________________________          23
       Working Paper Themes _______________________________________________________________          27
       The Role of Country Studies ___________________________________________________________       28
       Identifying and Involving Relevant Stakeholders ___________________________________________   29
       Cross-Cutting Issues _________________________________________________________________        29
       Literature Review and Data Management _________________________________________________       31

4.         Conducting Country Studies______________________________________________________          33
       Overview __________________________________________________________________________           33
       Country Study Outputs _______________________________________________________________         34
       Key phases of the country study work ____________________________________________________     35
       Stakeholder consultation and research____________________________________________________     38
       Team Roles ________________________________________________________________________           39

References _____________________________________________________________________________ 41

Annex A: Terms of Reference _____________________________________________________________ 43

Annex B: The GRPP Evaluation Sourcebook ________________________________________________ 55

Annex C: Quality Assurance Requirements__________________________________________________            61
    Quality Specifications of the TOR ______________________________________________________         61
       Evaluation Framework and Quality Standards (TOR ¶17-18) _______________________________       61
       Quality assurance and support (TOR ¶36) ______________________________________________        61
    DAC Evaluation Standards ____________________________________________________________            62

Annex D: Addressing the TOR's High Level Questions ________________________________________ 67

Annex E: Extended Logical Framework for the Evaluation of the FTI ___________________________ 73




file: FTI_Evaluation_Framework(Jan2009b).doc                                            Page 3 of 134
              Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


Annex F: Working Paper Topics __________________________________________________________ 99
    Overview __________________________________________________________________________ 99
    Working Paper 1: FTI Influence on Education Policy and Planning _____________________________ 99
    Working Paper 2: Financing EFA and Issues in PFM ________________________________________ 99
    Working Paper 3: Data Analysis and Monitoring for the FTI _________________________________ 101
       Current Data Sub-Stream: __________________________________________________________ 101
       Monitoring Sub-Stream ___________________________________________________________ 101
       Data Use and Analysis Sub-stream ___________________________________________________ 101
    Working Paper 4: Capacity Development (CD) ___________________________________________ 102
    Working Paper 5: FTI Governance and Aid Effectiveness ___________________________________ 103
    Working Paper 6: FTI and Fragile States and Fragile Partnerships _____________________________ 103

Annex G: Impact Evaluation Scoping Study ________________________________________________                                                          105
    Background and Time Schedule _______________________________________________________                                                           105
    Proposed Scoping Paper _____________________________________________________________                                                           105
    Oversight and Coordination___________________________________________________________                                                          106
    Deliverables _______________________________________________________________________                                                           106
    Follow-up ________________________________________________________________________                                                             107

Annex H: Selection of Country Cases ______________________________________________________                                                         109
    Rationale for Country Case Studies _____________________________________________________                                                       109
    Value of early decision ______________________________________________________________                                                         109
    Agreed Criteria for Case Study Selection ________________________________________________                                                      110
    Agreed parameters __________________________________________________________________                                                           110
    Proposal __________________________________________________________________________                                                            111

Annex I: Stakeholder Mapping ___________________________________________________________ 113

Annex J1: FTI Timeline (General) ________________________________________________________ 119

Annex J2: Trajectories of Country Involvement in FTI _______________________________________ 127

Annex K: Format for Interview Notes _____________________________________________________ 133



Boxes
            Box 2A: Components and Schedule of the Evaluation....................................................................... 16
            Box 3A: The DAC Evaluation Criteria .............................................................................................. 21
            Box 3B: Contribution Analysis .......................................................................................................... 22
            Box 3C: Pawson's Principles .............................................................................................................. 22
            Box 3D: High Level Evaluation Questions ........................................................................................ 24
            Box 4A: Role of the Pilot Country Studies ........................................................................................ 33
            Box 4B: Case Study Approach in Non-Endorsed Countries .............................................................. 35
            Box I: Template for identifying key country stakeholders ............................................................... 116

Figures
         Figure 3A: Concise Logical Framework for the Mid-Term Evaluation of FTI ...................................... 26
         Figure I: An Initial Stakeholder Map ................................................................................................... 115


Tables
            Table 4A: Phased preparation of country studies ............................................................................... 37




Page 4 of 134                                                                                                                printed: 04-Feb-09
           Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative



Abbreviations and Acronyms

CABRI           Collaborative Africa Budget Reform Initiative
CCI             Cross-cutting issue
CD              Capacity Development
CE              Cambridge Education
CF              Catalytic Fund
CSO             Civil Society Organizations
CSTL            Country Study Team Leader
DAC             Development Assistance Committee
DP              Development Partner
ECF             Expanded Catalytic Fund
EFA             Education for All
EMIS            Education Management Information System
EPDC            Education Policy and Data Centre.
EPDF            Education Program Development Fund
ESP             Education Sector Plan
ETF             Education Transition Fund
FF              Financing Framework
FTI             Fast Track Initiative
GBS             General Budget Support
GCE             Global Coalition for Education
GDP             Gross Domestic Product
GMR             Global Monitoring Report
GNI             Gross National Income
GRPP            Global and Regional Partnership Programs
H&A             Harmonisation and Alignment
HDI             Human Development Index
HIPC            Heavily Indebted Poor Country
HQ              Headquarters
IDA             International Development Association (World Bank)
IEG             Independent Evaluation Group
IIEP            International Institute for Educational Planning
IMF             International Monetary Fund
INGOs           International Non Governmental Organisations
IR              Inception Report
JAR             Joint Annual Review
JEGBS           Joint Evaluation of General Budget Support
JICA            Japan International Corporation Agenda
LAC             Latin America and Caribbean
LDG             Local Donor Group
LEG             Local Education Group
M&E             Monitoring and Evaluation
MDGs            Millennium Development Goals
MoE             Ministry of Education
MoF             Ministry of Finance
MoU             Memorandum of Understanding
MTEF            Medium-Term Expenditure Framework
NER             Net enrolment rate



file: FTI_Evaluation_Framework(Jan2009b).doc                                 Page 5 of 134
         Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


NGO             Non-governmental Organisation
NGO             Non-governmental Organisation
ODA             Official Development Assistance
ODI             Overseas Development Institute
OECD            Organisation for Economic Cooperation and Development
OECD DAC        OECD Development Assistance Committee
OPM             Oxford Policy Management
PER             Public Expenditure Review
PETS            Public Expenditure Tracking Surveys
PFM             Public Finance Management
PIU             Project Implementation Unit
PLHA            People Living with HIV&AIDS
PM              Partnership Meeting
PRSP            Poverty Reduction Strategy Paper
QA              Quality Assurance
QSDS            Quantitative Service Delivery Surveys
SAR             South Asia Region
SC              Strategy Committees
SPA             Strategic Partnership with Africa
SWAp            Sector-Wide Approach
TA              Technical Assistance
TL              Team Leader
TM              Technical Meetings
ToR             Terms of Reference
TT              Task Team
TTL             Task Team Leader
TW1             Team Workshop 1
UIS             UNESCO Institute for Statistics
UN              United Nations
UNESCO          United Nations Educational Scientific and Cultural Organization
UNGEI           United Nations Girls Education Initiative
UNICEF          United Nations Children‟s Fund
UPC             Universal Primary Completion
UPE             Universal Primary Education
WB              World Bank
WG              Working Group




Page 6 of 134                                                                     printed: 04-Feb-09
              Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative




Preface
The Fast Track Initiative (FTI) is linked both to the Education for All (EFA) goals and to the
Millennium Development Goals (MDGs). The FTI was launched in 2002, and has now been
running for half its expected lifetime. The FTI partnership has recognised the need to
evaluate whether it is achieving the goals it has set itself. The evaluation will provide an
opportunity for reform and change if necessary.

As stated in the Terms of Reference:
       The main purpose of the evaluation is to assess the effectiveness of FTI to
       date in accelerating progress towards achievement of EFA goals in
       participating countries, with particular attention to country movement towards
       universal primary completion (UPC). The evaluation will also assess FTI’s
       contributions to improving aid effectiveness at both the country and global
       levels.

The evaluation will draw lessons learned from the FTI‟s strengths and weaknesses and
make a series of recommendations to further improve future partnership programming and
effectiveness. Most importantly it is hoped that, as a result of this evaluation, progress
towards expanding and enhancing educational opportunities will be strengthened.

The evaluation is independent but is jointly supported by a consortium of donors. The
evaluation will take place between November 2008 and December 2009. The Evaluation
Oversight Committee (EOC) is made up of representatives from the donor community,
partner countries and civil society. It is coordinated by Joe DeStefano (email:
jdestefano@futureofschooling.org).

The evaluation team is a consortium of three companies Cambridge Education, Mokoro and
Oxford Policy Management (OPM). Core members of the evaluation team are:
Core Team Members         Area of Responsibility                   Email Address
Stephen Lister            Team Leader                              slister@mokoro.co.uk
                          Lead: Aid Effectiveness & Governance
Felicity Binns            Senior Project Manager                   felicity.binns@camb-ed.com
Georgina Rawle            Lead: Education Financing & PFM          georgina.rawle@opml.co.uk
Abby Riddell              Lead: Education Capacity Development     abby.riddell@googlemail.com
Roy Carr-Hill             Lead: Data and Monitoring & Evaluation   roycarrhill@yahoo.com
Eric Woods                Lead: Education Policy & Planning        ericwoods@clara.co.uk
Muriel Visser-Valfrey     Aid Effectiveness & Governance           mvisser@learndev.org
Catherine Dom             Fragile States & Partnerships            cdom@mokoro.co.uk
Anthea Sims Williams      Research Coordinator                     asimswilliams@mokoro.co.uk
Carlos Orte               Project Manager                          carlos.orte@camb-ed.com


For enquiries related to the evaluation please contact Anthea Sims Williams, the research
coordinator, at asimswilliams@mokoro.co.uk.

For regular updates about the evaluation and the most recent outputs please refer to the
evaluation website at: www.camb-ed.com\fasttrackinitiative.




file: FTI_Evaluation_Framework(Jan2009b).doc                                       Page 7 of 134
         Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative




Page 8 of 134                                                            printed: 04-Feb-09
           Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative



1.      Introduction and Overview
Purpose and structure of this document
1.1    This paper comprehensively describes the process and methodology for the mid-
term evaluation of the Education for All (EFA) Fast Track Initiative (FTI). It has three
intended audiences:
            first and foremost, it is a handbook for members of the evaluation team;
            second, it provides the description of methodology required by the Evaluation
            Oversight Committee (EOC);
            third, it provides transparency about the evaluation for any other interested
            parties.

1.2     The paper is arranged as follows:
            the present chapter summarises the requirements for the evaluation and the
            approach to conducting it;
            Chapter 2 describes the evaluation process, including the timetable. the structure
            of the evaluation team, the procedures for quality assurance and the
            communications strategy;
            Chapter 3 gives a detailed explanation of the methodology for the evaluation;
            Chapter 4 gives guidelines for the country case studies;
            a series of Annexes complement the main text.

1.3    Additional management and administrative issues are covered in a separate Work
Plan and Budget submitted to the EOC.

The Mid-Term Evaluation of the FTI
Objectives
1.4     The full Terms of Reference (TOR) are at Annex A.1 The core purpose and
objectives of the evaluation are succinctly described in the TOR as follows:
      The main purpose of the evaluation is to assess the effectiveness of FTI to date in
      accelerating progress towards achievement of EFA goals in participating countries, with
      particular attention to country movement towards universal primary completion (UPC).
      The evaluation will also assess FTI‟s contributions to improving aid effectiveness at both
      the country and global levels. The evaluation will assess the Initiative‟s added value,
      identify lessons learned from its strengths and weaknesses, and formulate
      recommendations for improved partnership programming and effectiveness.
      The objective of this evaluation will be to assess the current relevance, effectiveness,
      efficiency of the Fast Track Initiative. More specifically, it will assess:
             The relevance of FTI objectives and goals to country educational and
             development needs (relevance);
             FTI‟s contribution (value added) to country progress in accelerating action on
             EFA, in particular towards UPC, and related processes (effectiveness);
             FTI‟s contribution (value added) to improved resource mobilization and aid
             effectiveness (efficiency).




1
 The TOR are supplemented by an Issues Paper prepared by the EOC. Along with other key
documents, this is available on the evaluation's web site at www.camb-ed.com\fasttrackinitiative.



file: FTI_Evaluation_Framework(Jan2009b).doc                                           Page 9 of 134
          Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


1.5     The TOR further specify that the evaluation should cover the entire period of the FTI,
from its establishment in 2002 through September 2008, and the need to examine its effects
both at global and at country level.

1.6     The evaluation is to be forward-looking, with recommendations for the future direction
of the initiative, and a further important requirement is to develop a comprehensive
monitoring and evaluation framework that will, inter alia, facilitate future assessment of
impact.

Stages and Outputs
1.7     As envisaged in the TOR (Annex A, ¶24–27), the evaluation is divided into two
stages.
           Stage 1 (November 2008 – March 2009) is focused on gathering global level data
           addressing questions of FTI effectiveness and efficiency, relying principally on
           desk study to paint an initial picture of FTI outputs and outcomes (on resource
           mobilization, on improved policy and planning, and improved alignment,
           harmonization, ownership, on basic EFA indicators etc). It will also gather
           stakeholder perceptions of FTI effectiveness and efficiency, including its
           governance arrangements. Stage 1 will also review and assess the current
           monitoring and evaluation arrangements within FTI. Stage 1 also includes two
           pilot country studies.
           Stage 2 (March through December 2009) will be focused on gathering additional
           data to flesh out global perspectives on effectiveness and efficiency (with country
           examples and carefully constructed lessons learned) and addressing remaining
           questions of relevance. A further seven country case studies will be conducted,
           and will feed into a full report which will present overall findings, lessons learned,
           and recommendations which can be used by the partnership to change course as
           needed and improve relevance, effectiveness and efficiency. Stage 2 will also
           include the development of a framework for monitoring and evaluating the impact
           of FTI.

1.8    The principal outputs of the evaluation are specified as:
           A report establishing the evaluation framework (based on a refined FTI program
           theory) that will guide this formative evaluation. [i.e. the present document].
           A Preliminary Report, which the TOR (¶25) specify as follows:
           The main product of Stage 1 will be a preliminary report which will be used as part of
           the formative process of engaging the FTI Partnership in discussing global findings on
           FTI effectiveness and efficiency. It should provide evidence from existing reports,
           documents and studies of how the FTI has or has not contributed to the trajectory of
           indicators in FTI countries during the past decade, and include a first review of different
           stakeholder perceptions of the FTI. It should enable a comparison of trends on key
           indicators in FTI endorsed countries before and after endorsement, as well as in
           comparison to those endorsed at different times, or to countries not participating in the
           FTI. The report should present the findings of an initial review of the governance and
           management arrangements and the existing FTI monitoring and evaluation framework
           and approach. The preliminary report should also indicate how the consultant expects
           to continue to gather data during Stage 2 of the evaluation, including restating or
           revising the list of case studies that will be conducted and methodologies for systematic
           stakeholder sampling. [The Preliminary Report is also expected to draw lessons from
           the pilot country studies.]




Page 10 of 134                                                                        printed: 04-Feb-09
                              Chapter 1: Introduction and Overview


            Country Case Studies. The EOC has agreed that there will be nine full case
            studies, including the two pilots. Each will be written up as a free-standing report
            intended to be of value to country stakeholders.
            A Full Report which will synthesise all the main findings, conclusions and
            recommendations of the evaluation.


Timetable
1.9     The timetable envisaged in the TOR had to be adjusted because of delays in the
contracting process. Key dates are now as follows (see Chapter 2 and Box 2A for full
details):
Main study components                          Timing
Technical Proposal                             August 2008
Refinement with WB/EOC                         September 2008
FTI Meetings, Paris                            16-19 September 2008
Contract signature                             3 November 2008
Team Workshop 1 (Evaluation Framework)         25-26 November 2008
Final Evaluation Framework and Workplan        December 2008
FTI Meetings, Oslo                             11-14 December 2008
Literature and Data Review                     December 2008 – January 2008
Thematic Working Papers                        November 2008 – January 2009
                                               (internal deadline 18 January)
Pilot Country Studies                             Preparatory work on 2 pilot countries from
                                                  December;
                                                  Two weeks' field work in the second half of
                                                  January.
                                                  Internal deadline for full pilot country reports: 23
                                                  February 2009
                                                  submission to EOC 13 March 2009
Team Workshop 2 (Preliminary Report            12–13 February 2009
Stage)
Preliminary Report                             6 March 2009(English), 13 March 2009 (French)
FTI Meetings, Copenhagen                       21–23 April 2009
Additional Thematic Work                       March – July 2009
Main Country Studies
   Preparatory work (identifying teams            from December 2008
   including local consultants, preliminary
   data collection etc)
   Field work                                     mid-March – end June 2009
   Draft reports (internal)                       May – July 2009
   Full reports                                   June – August 2009
Team Workshop 3 (Synthesis Report)             August
Draft Full Report                              August 2009
FTI Meetings, Italy                            September 2009 (dates tbc)
Final Report                                   December 2009
Thematic Briefing Papers                       December 2009
Dissemination Events                           early 2010




file: FTI_Evaluation_Framework(Jan2009b).doc                                          Page 11 of 134
             Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


Challenges
1.10 The TOR highlight the challenging nature of this evaluation, and require an
explanation of how challenges will be addressed, and how limitations will be mitigated.

1.11 The methodology described in this handbook is based on an appreciation of the
conceptual difficulties in evaluating a programme that is intentionally diffuse in its effects.
Recognising the difficulties of establishing a clear counterfactual, the approach involves
systematic contribution analysis, and explicitly draws on the lessons of previous joint
evaluations, as well as the guidelines of the GRPP Sourcebook.2

1.12 Salient challenges and the ways the Evaluation Team will address them are as
follows:
     Challenge                                          Response
    The FTI is intentionally diffuse in its             Systematic use of contribution analysis
    effects. It seeks to influence progress             methodology (as described in Chapter 3).
    towards UPC through a variety of channels, in
    each case seeking to enhance and augment
    other activities and resources. Significantly its
    most visibly discrete component was called
    the Catalytic Fund. There is therefore a huge
    challenge in assessing what would have
    happened in the absence of FTI (the
    counterfactual) and thereby making reliable
    attributions of effects to the FTI.
    The FTI is part of global efforts to expand         The evaluation will economise on evaluation
    education (EFA), and needs to be understood         resources, focus efforts on critical issues and
    in that context; accordingly the potential          add value by drawing as much as possible on
    breadth of work is huge.                            existing data and making use of previous work
                                                        without duplicating it.
    It is necessary not only to review what the FTI     The contribution analysis approach is well
    has done (e.g. in its endorsed countries) but       suited to posing the why not question, and
    also to examine what FTI hasn't (yet)               the selection of country cases will include
    accomplished and why.                               countries that exemplify FTI's uncompleted
                                                        agenda.
    This evaluation is on a substantial scale, but      This reflected in systematic efforts to build on
    nevertheless resources are limited. For the         previous work, to focus on selected key
    most valuable results it is important not to        issues at both global and country level,
    spread resources too thinly, but to                 and to undertake a set of country studies that
    concentrate on tasks and issues that offer the      strikes the right balance between breadth and
    greatest potential for learning.                    depth.
    As stressed in the GRPP Sourcebook,                 The evaluation's response to the GRPP
    stakeholder involvement is a key ingredient         evaluation guidelines (Annex B) and the
    for successful evaluations. At the same time,       approach to stakeholder involvement
    in keeping with the philosophy of the FTI itself,   (Annex I) show the balance that will be
    it is important to minimise the transaction         sought.
    costs of the evaluation



2
 IEG (2007) Sourcebook for Evaluating Global and Regional Partnership Programs: Indicative
Principles and Standards. Independent Evaluation Group – World Bank. Washington DC.

Page 12 of 134                                                                            printed: 04-Feb-09
                               Chapter 1: Introduction and Overview


   Challenge                                        Response
  Scope for definitive quantitative analysis is     "Data" was one of the key gaps identified by
  limited by known issues in the availability and   the FTI, and the evaluation includes a
  quality of data (e.g. inconsistencies between     strong data workstream to make best use of
  country-level and global data series on           available data, assess progress in
  education, public finance and aid). These are     strengthening data, and develop an
  compounded by the short history of the FTI,       appropriate M&E strategy going forward.
  and its purposely evolving approaches.
  The adaptive approach of FTI presents a           It is appropriate that FTI should have evolved
  moving evaluation target.                         and adapted in the course of implementation.
                                                    The evaluation will focus on FTI's lesson-
                                                    learning and the appropriateness of its
                                                    adaptations. Although the formal starting
                                                    point of the FTI was 2002, the evaluation will
                                                    need to go further back in time in order to
                                                    understand properly the origins, context and
                                                    intentions of the FTI. This applies both at the
                                                    global level, and at country level, where it is
                                                    important to have a good understanding of the
                                                    dynamic processes that were under way at
                                                    the point when FTI effects, if any, began to be
                                                    felt.
  As noted in Samoff's review of the joint          The evaluation seeks to go behind the data,
  evaluation of basic education (Samoff 2005),      and beyond the technical explanations, to
  there is a tendency for evaluations to focus on   understand the interests and incentives
  technical issues and shy away from political      influencing stakeholders, at every level
  economy, yet this is fundamental to the           from the school up to the international political
  understanding of the FTI and its contexts.        arena.



Emerging good practice for joint evaluation of a global programme
1.13 The evaluation can draw on a growing body of experience in the joint evaluation of
global partnerships. The GRPP Sourcebook (IEG 2007) is an extremely valuable
compendium. Annex B provides a comprehensive summary of how the evaluation's
approach responds to the GRPP characteristics of the FTI. Other relevant references
include the (revised) DAC guidelines on joint evaluation (OECD DAC 2006b; see also OECD
DAC 2005), and specific lessons drawn from previous major evaluations – notably the joint
evaluation of aid to basic education (comments by Samoff 2005 on Netherlands Ministry of
Foreign Affairs 20032003), and the Joint Evaluation of General Budget Support (Note on
Approach and Methods, IDD and Associates 2007).

1.14 It is worth highlighting, from the JEGBS Note on Approach and Methods the
following "5 key lessons for a complex evaluation":
        1. Remember that success depends on a well-organised evaluation process as
           much as on a sophisticated methodology.
        2. Leave enough time for the evaluation itself: extra time refining methodology has a
           high cost if it cuts short the study proper.
        3. Draw as much as possible on existing studies and data. Start by thoroughly
           documenting the subject of the evaluation (the factual before the counterfactual).




file: FTI_Evaluation_Framework(Jan2009b).doc                                            Page 13 of 134
          Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


       4. Foster interaction among study teams and between the evaluation team and the
          client as the study proceeds. Workshops at key points in the study are
          invaluable.
       5. Present evidence and reasoning as transparently as possible. A common
          structure for country reports is an important element of quality assurance. (IDD
          and Associates 2007, Box 18)

1.15 The evaluation's work plan directly reflects these lessons, in its emphasis on
disaggregation, establishment of the factual, a standard approach across countries, and an
approach that mainstreams consultation with the EOC and stakeholders. The work plan
reflects the two stages defined in the TOR and will allow the detailed work plan for Stage 2
to reflect the findings from the preliminary analysis and the findings from the pilot country
studies. Chapter 2 explains the overall structure and process of the evaluation in detail.




Page 14 of 134                                                                printed: 04-Feb-09
              Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative



2.      The Evaluation Process
Structure and Components of the Evaluation
2.1    Box 2A below shows the schematic structure of the study, and the way it will be
broken down into components that systematically feed into each other, building towards a
coherent final synthesis report (accompanied by supplementary evaluation outputs). It
incorporates a revised timetable agreed with the EOC.

Team Structure and Responsibilities
2.2     Members of the core team and their roles are shown below.

Team Member                 Role

Stephen Lister              Team Leader
                            Lead: Aid Effectiveness & Governance
Felicity Binns              Senior Project Manager
Georgina Rawle              Lead: Education Financing & PFM
Abby Riddell                Lead: Education Capacity Development
Roy Carr-Hill               Lead: Data and Monitoring & Evaluation
Eric Woods                  Lead: Education Policy & Planning
Muriel Visser-Valfrey       Aid Effectiveness & Governance
Catherine Dom               Fragile States & Partnerships
Anthea Sims Williams        Research Coordinator
Carlos Orte                 Project Manager



2.3     The core team is supported by a larger panel of experts who will be drawn on for
research support and additional specialist inputs, and to conduct the country studies. Core
team members will also participate in the country studies. More details on task assignments
are included in the Work Plan and Budget document submitted to the EOC.




file: FTI_Evaluation_Framework(Jan2009b).doc                                   Page 15 of 134
                                   Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative

                                          Box 2A: Components and Schedule of the Evaluation




                      STAGE 1: NOVEMBER 2008 – MARCH 2009                                       STAGE 2: MARCH 2009 – JANUARY 2010

Main study components                      Timing                                                     Comment
1. Technical Proposal                     August 2008         This technical proposal's methodology and work plan have subsequently been elaborated
                                                              in consultation with the EOC. The study is organised under five main workstreams
                                                              addressing education policy and planning, financing, capacity, data and monitoring, and
                                                              governance and aid effectiveness.
2. Refinement with WB/EOC                  September          Results of the discussions and negotiations with EOC are incorporated in the full Project
                                                              Description now incorporated in the contract issued by the WB.
FTI Meetings, Paris                     16-19 September       DFID funding enabled Evaluation Team members (Stephen Lister, Felicity Binns,
                                                              Georgina Rawle, Abby Riddell, Roy Carr-Hill and Anthea Sims Williams) to attend Paris,
                                                              for participant observation at the FTI meetings plus negotiations with EOC.
Contract signature                        3 November          At negotiations, contract signature was anticipated to be no later than end-September, but
                                                              the eventual contract is dated 3 November, hence need to reschedule evaluation deadlines.
3. Team Workshop 1 (Evaluation          25-26 November        Held in Cambridge, UK, with active participation of EOC members (seven of whom
Framework)                                                    attended for one or both days). Focused on further refinement of evaluation
                                                              methodology, identification of countries for case study, and initial review of issues.




Page 16 of 134                                                                                                                       printed: 04-Feb-09
                                                              Chapter 2: The Evaluation Process

Main study components                             Timing                                                              Comment
4. Final Evaluation Framework                   8 December             This has two components:
and Workplan                                    (milestone)                 (a) As described in the proposal: It will document the full set of evaluation instruments that
                                                                                constitute the Evaluation Framework, together with detailed specification of tasks and activities.
                                                                                It will be a fully operational handbook for team members and management.
                                                                            (b) It will also provide a finalised workplan (specified in detail for Stage 1, and in
                                                                                outline for Stage 2) for approval by the EOC.
FTI Meetings, Oslo                             11-14 December          The EOC and the Evaluation Team Leader gave an update on progress and the
                                                                       Evaluation Framework. (Original intention was to present the Preliminary Report (#9) at
                                                                       this point.) Eric Woods and Anthea Sims Williams also attended to observe proceedings
                                                                       and to interview participants.
5. Literature and Data Review              December 2008 –             To provide basic data and documents for all workstreams and for the country studies,
                                            January 2008               together with more analytical literature search on a number of sub-topics to feed into the
                                                                       thematic working papers (#6).
6. Thematic Working Papers                 November 2008 –             Working Papers are aligned with the workstreams defined to share tasks across the core
                                             January 2009              evaluation team. Topics and authorsare:
                                                                                 FTI Governance / Aid Effectiveness (Muriel Visser and Stephen Lister) [to be
                                      (internal deadline 18 January)             treated as separate sub-topics]
                                                                                 FTI Influence on Education Policy and Planning (Eric Woods)
                                                                                 Financing EFA and Issues in PFM (Georgina Rawle)
                                                                                 Data Analysis and Monitoring for the FTI (Roy Carr-Hill)
                                                                                    o (this is interlinked with a Scoping Paper in Impact Evaluation to be
                                                                                        commissioned from 3ie)
                                                                                 Capacity Development (Abby Riddell)
                                                                                FTI and Fragile States and Fragile Partnerships (Catherine Dom)
                                                                       These are internal working papers but they will be shared with the EOC. The papers will
                                                                       include tentative assessments, especially at global level, and highlight key issues that need
                                                                       further investigation during Stage 2, especially through the country studies. The
                                                                       Preliminary Report (#9) will draw on these papers.




file: FTI_Evaluation_Framework(Jan2009b).doc                                                                                                                  Page 17 of 134
                                        Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative

Main study components                            Timing                                                          Comment
7. Pilot Country Studies                   Preparatory work on 2
                                           pilot countries from
                                           December;
                                           Two weeks' field work in     The pilot country studies will yield substantive country-level findings from two
                                           the second half of           contrasting FTI-eligible countries; they will also test prototype instruments and
                                           January.                     approaches to the country studies, and thereby strengthen the approach to the main set of
                                           Internal deadline for full   country studies to be conducted in Stage 2.
                                           pilot country reports: 23    Kenya and Cambodia have been selected as the two pilot cases.
                                           February
                                           submission to EOC
                                           (milestones) 13 March
8. Team Workshop 2 (Preliminary                  12-13 February         A second team workshop, involving all core team members (including the pilot country
Report Stage)                                                           study team members) to review the Working Papers and Pilot Country Study findings, and
                                                                        develop the key messages for the Preliminary Report.
9. Preliminary Report                         6 March (English)         As specified in the TOR. This will be in two parts:
                                              13 March (French)                  o a concise summary of findings thus far, and proposals for issues to be further
                                                 (milestone)                         investigated, including the main suite of country studies;
                                                                                 o a fully detailed workplan for Stage 2, for approval by the EOC.
FTI Meetings, Copenhagen                       21–23 April              Opportunity to discuss Preliminary Report and first 2 country studies.
10. Additional Thematic Work                   March– July              Further thematic work to be undertaken by the core steam during Stage 2 will be specified
                                                                        in the Preliminary Report and associated Stage 2 workplan. This will include a paper on
                                                                        specifying the future M&E framework (taking account of the findings from the impact
                                                                        evaluation scoping paper).
11. Main Country Studies

   Preparatory work (identifying           from December 2008
   teams including local consultants,
   preliminary data collection etc)
                                                                        Further country studies will be the major focus of Stage 2. The present workplan allows
   Field work                              mid-March – end June         for 7 country studies at this stage, making a total of 9 country studies including the pilots.
   Draft reports (internal)                May – July

   Full reports                            June – August
                                           (milestones)


Page 18 of 134                                                                                                                                      printed: 04-Feb-09
                                                              Chapter 2: The Evaluation Process

Main study components                             Timing                                                    Comment
12. Team Workshop 3 (Synthesis                    August               Team workshop to review findings from the full set of country studies and all other study
Report)                                                                components to date.
13. Draft Full Report                            August 2009
                                                                       The full draft synthesis report will be widely disseminated for discussion and feedback.
                                                 (milestone)
FTI Meetings, Italy                            September 2009
                                                                       Draft full report to be circulated in time for discussion at this meeting.
                                                  (dates tbc)
14. Final Report                               December 2009           Final Report, taking account of feedback.
15. Thematic Briefing Papers                   December 2009           Up to four Thematic Briefing Papers; details of topics etc to be agreed with the EOC in
                                                                       due course.
16. Dissemination Events                         early 2010            The TOR envisage three dissemination events. Details to be agreed with EOC in due
                                                                       course.




file: FTI_Evaluation_Framework(Jan2009b).doc                                                                                                        Page 19 of 134
          Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


Quality Assurance
Applicable Quality Standards
2.4    Ethical and other evaluation quality standards are highlighted in the TOR, and the
evaluation is required to adhere to the standards spelt out in the GRPP Sourcebook and the
OECD DAC guidelines. The TOR quality specifications and the key DAC standards are
reproduced in Annex C.

Quality Assurance Panel
2.5    Systematic quality assurance is integral to the evaluation team's working methods.
The evaluation is supported by an expert QA panel which comprises Christopher Colclough,
Nils Boesen, Juliet Pierce and Steve Packer.

2.6     The QA panel members will review draft and final outputs. They will provide succinct
written assessments of each document reviewed, following a standard pattern. Drafts of
intermediate and final outputs will be submitted to the appropriate QA expert(s), and their
comments taken into account, before formal submission to the EOC. The QA comments
themselves will also be shared with the EOC.

2.7   QA panel members comment from an independent perspective. They are not
members of the evaluation team and are not responsible for the team's conclusions.
However, they may be called upon for ad hoc advice, and may also be invited to some of the
team workshops.

Other Dimensions of Quality Assurance
2.8    The team structure, with clear role definition but networked collaboration across
workstreams, is also part of the concern for quality, as is the standardisation of approaches
to country studies etc. The proposed workshops have an important QA role.

2.9    The DAC and GRPP ethical standards for evaluation will be applied. These will be
incorporated in the task TOR and sub-contracts, as appropriate, for all members of the study
team.

2.10 The evaluation team respects the EOC's role in quality assurance, and recognises
the expertise and experience possessed by EOC members. Without compromising the
independence of the evaluation, the team will give EOC members (and other informed
stakeholders) the opportunity to comment on drafts prior to the formal submission of
deliverables.

Communications
2.11 The evaluation team has established a website which has three access levels: one
which is accessible to the general public and all FTI stakeholders, one which is accessible
only to members of the EOC and members of the evaluation team, and a final level for
storing private data of the nature that is generated by interviews and surveys.

2.12 The website will be a means of disseminating publicly available information that
results from the ongoing working papers and events of the evaluation, as well as functioning
as a central focus for members of the evaluation team. It supports the strategy for involving
relevant stakeholders that is described in Chapter 3.




Page 20 of 134                                                                printed: 04-Feb-09
           Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative



3.      Evaluation Methodology
General Approach
3.1   The methodological challenges facing the evaluation have been highlighted in
Chapter 1 (¶1.10–1.12). The present chapter provides a detailed guide to the methodology
adopted for this evaluation.

3.2   OECD DAC guidelines on evaluation are an important point of reference, and the
DAC evaluation criteria will be applied using the definitions shown in Box 3A.

                              Box 3A: The DAC Evaluation Criteria
The five DAC evaluation criteria are:
        Effectiveness: The extent to which the development intervention‟s objectives were achieved,
        or are expected to be achieved, taking into account their relative importance.
        Efficiency: A measure of how economically resources/inputs (funds, expertise, time, etc.) are
        converted to results.
        Relevance: The extent to which the objectives of a development intervention are consistent
        with beneficiaries‟ requirements, country needs, global priorities and partners‟ and donors‟
        policies.
        Impact: Positive and negative, primary and secondary long-term effects produced by a
        development intervention, directly or indirectly, intended or unintended.
        Sustainability: The continuation of benefits from a development intervention after major
        development assistance has been completed. The probability of continued long-term benefits.
        The resilience to risk of the net benefit flows over time.

Source: OECD DAC 2002 (Glossary of Key Terms in Evaluation and Results Based Management).


3.3    The TOR require the evaluation team : ... also explain how [it] expects to measure
FTI’s unique contribution or value added (see objectives 2 and 3), given the fact that FTI has
not been organized with an experimental design and does not lend itself easily to
counterfactual assessment (TOR ¶20).

3.4     We will address this primarily through contribution analysis. This allows a
pragmatic combination of qualitative and quantitative approaches, building up an overall
judgement from a variety of components. However it is not a "soft" option: its proper
application requires both diligence and rigour, which is especially important in this case, with
the need to synthesise findings from global level and from a series of country case studies.
Mayne's seminal paper (1999) explains the steps and iterations that are required, and
stresses the theory-based nature of the exercise. Box 3B below provides a synopsis, while
Pawson's principles (Box 3C) are also extremely relevant to a complex evaluation such as
this one.




file: FTI_Evaluation_Framework(Jan2009b).doc                                         Page 21 of 134
          Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


                           Box 3B: Contribution Analysis
The Idea of Contribution Analysis
       Based on the theory of change of the programme.
       Buttressed by evidence validating the theory of change.
       Reinforced by examination of other influencing factors.
       Contribution analysis builds a reasonably credible case about the difference a
       programme is making.
Steps in Contribution Analysis
1. Set out the attribution problem to be addressed.
2. Develop the postulated theory of change.
3. Gather existing evidence on the theory of change.
4. Assemble and assess the contribution story.
5. Seek out additional evidence.
6. Revise and strengthen the contribution story.
7. Develop the complex contribution story.
Source: adapted from Mayne (n.d.) PowerPoint.

                         Box 3C: Pawson's Principles
   1. Locate key programme components.
       Comprehensive scoping of programme activities and potential influences.
   2. Prioritise among programme components.
       Concentrate on elements of the programme theory that seem likely to have most
       influence, and on elements about which least is known.
   3. Evaluate programme components by sub-sets.
       Focus on identifiable components of the programme theory (then build up the overall
       picture).
   4. Identify bottlenecks in the programme network.
       Look for blockages and breaks in results chains (this is especially relevant for
       lessons to influence future adaptation and strengthening of the programme).
   5. Provide feedback on the conceptual framework.
       "The most durable and practical recommendations that evaluators can offer come
       from research that begins with theory and ends with a refined theory."


Source: adapted from Pawson (2006) Simple Principles for the Evaluation of Complex
Programmes. (In Public Health Evidence, Oxford.)


3.5  Contribution analysis provides the organising principle that links the different study
components and evaluation tools described in the rest of this chapter.




Page 22 of 134                                                                printed: 04-Feb-09
                              Chapter 3: Evaluation Methodology


High Level Evaluation Questions
3.6    The TOR require the evaluation to focus on a series of high-level evaluation
questions which are reproduced in Box 3D. The detailed questions supporting the High
Level Questions are described as a tentative list. (They are, in part, derived from a more
extensive set of questions in the Issues Paper which was a precursor to the TOR.)

3.7     Annex D maps these questions simultaneously onto (a) the five main workstreams of
the evaluation; (b) the DAC evaluation criteria (Box 3A above); and (c) the principal outputs
of the evaluation.

3.8   The programme logic described in the next section of this chapter also develops a
more extensive set of specific evaluation questions.

The Programme Logic
3.9      The Cambridge consortium's proposal included a first version of the FTI "programme
logic" to be investigated. This took the form of an overall "results chain" together with
disaggregated results chains at country and global level for five anticipated streams of
effects (four of which echo the "gaps" identified in the FTI framework):

           Education policy and planning
           Education finance
           Data and M&E
           Capacity
           Aid effectiveness.

3.10 The first Team Workshop (TW1) spent considerable effort on further analysis and
elaboration of the programme logic, and this has since been further developed by the core
team.

3.11 The purpose of the Logical Framework presented here is to provide both a theoretical
and a practical guide to the work of the evaluation. At a theoretical level it sets out clearly a
set of hypotheses to be tested. At a practical level, it lists relevant detailed evaluation
questions, together with main sources of evidence needed to answer them. It provides a
focus for each workstream's enquiry.

3.12 Following TW1 it was decided to consolidate the separate results chains into a single
matrix that would provide a common point of reference and also highlight the links and
interactions between the different streams of hypothesised effects. The result is presented
in two forms:

           A one-page concise version – the bird's-eye view – shown as Figure 3A below.
           The extended version – i.e. the practical guide to the evaluators. This is
           presented in full as Annex E.




file: FTI_Evaluation_Framework(Jan2009b).doc                                     Page 23 of 134
           Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


                          Box 3D: High Level Evaluation Questions

Objective 1: Assess the Relevance of FTI Objectives and Goals to Country Educational and
Development Needs
High Level Questions 1: Is what FTI aims to accomplish consistent with current needs and
priorities of FTI client countries? What can be done to ensure the relevance of FTI going
forward?
Related questions:
   a. How is FTI perceived and understood by various stakeholders and to what extent have
      varying interpretations of what FTI is and what it can accomplish influenced its effectiveness
      in any way?
   b. To what extent is FTI‟s focus on primary school completion relevant to the current needs of
      partner countries (and potential partner countries) and how has this emphasis been treated
      within the broader context of country EFA programming?
   c. Are FTI‟s intended contributions appropriate, given the partnership‟s objectives? Can they be
      expected to make a difference?
   d. How has the evolution of the FTI contributed to maintaining its ongoing relevance as a global
      partnership supporting EFA?


Objective 2: Assess FTI‟s Contribution to Country Progress in Accelerating Action on EFA and
Related Processes
High Level Questions 2: To what extent is the FTI accomplishing what it was designed to do,
namely, accelerating progress on EFA? Are FTI mechanisms and processes contributing
effectively to this? How can FTI mechanisms and processes be improved to maximize their
contributions going forward?
Related questions:
   a.   To what extent has FTI and its mechanisms/processes (e.g. Indicative Framework and the
        endorsement process) helped countries strengthen education policies, sector plans, and
        educational data/statistics?
   b.   Are FTI mechanisms and processes evolving appropriately to address the needs of different
        client countries (e.g. fragile states)?
   c.   Has policy making and planning become more open, transparent, and participatory in
        participating countries?
   d.   To what extent has FTI contributed to improved institutional capacity in partner countries
        (including those preparing for partnership); has the capacity building provided been
        sufficiently comprehensive to make a sustainable difference?
   e.   To what extent and how has FTI contributed to accelerated progress towards universal
        primary school completion by 2015?
   f.   How have FTI programs contributed to intermediate outcomes such as improvements in the
        grade 1 intake and gross enrolment ratios?
   g.   Has FTI itself supported other EFA objectives in partner countries and, if so, to what effect?
   h.   How has the FTI impacted government support for other EFA objectives?




Page 24 of 134                                                                       printed: 04-Feb-09
                               Chapter 3: Evaluation Methodology



Objective 3: Assess FTI‟s Contribution to Improved Resource Mobilization and Aid Effectiveness
High Level Questions 3: Has the FTI helped mobilize domestic and international resources in
support of EFA and helped donor agencies to adopt more efficient development assistance
strategies based on Paris Declaration ideals? How can FTI best help mobilize resources and
improve aid effectiveness going forward?
Related questions:
   a.   To what extent are FTI partners improving the alignment of their development assistance to
        country-owned plans?
   b.   Has FTI helped them to improve interagency cooperation and harmonization?
   c.   Has FTI contributed to the reduction in transaction costs, both for partner countries and
        partner agencies?
   d.   To what extent have partner countries and agencies been using innovative and flexible
        means of financial support and to what extent do those means of support demonstrably
        improve aid effectiveness?
   e.   Has FTI been influential in mobilizing increased funding levels and non-financial resources for
        education among partner countries and agencies?
   f.   To what extent has FTI assisted countries in closing their financing gaps for their EFA
        acceleration efforts? Has this been in the context of sustainable fiscal policy?
   g.   Has FTI succeeded in mobilizing additional resources in support of EFA? To what extent
        have any increases come from domestic financing? From international assistance? In
        countries where domestic financing in support of EFA has increased what are the prospects
        for sustaining those levels of funding?
   h.   Has FTI‟s own governance and management made a difference in how efficiently the Initiative
        works in furthering EFA goals? Namely, can the FTI governance bodies be considered
        legitimate? Are FTI decision-making processes open, transparent, inclusive and well
        founded? Does the Initiative have an adequate monitoring and evaluation framework, and is it
        creating regular, useful inputs to decision making? Are FTI processes (endorsement,
        processing of catalytic funds requests, triggering the flow of funds) consistent across
        countries, timely and efficient (for example, how much time does it take between FTI
        endorsement and the flow of FTI-triggered funding at the country level)?




file: FTI_Evaluation_Framework(Jan2009b).doc                                          Page 25 of 134
                       Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative



                  Figure 3A: Concise Logical Framework for the Mid-Term Evaluation of FTI
                                Level Zero – Entry Conditions




                                                                                                                  Externa
                                                                                                                  Factors
                                                                                                                  Feedba
                                                                                                                  Assum
                                                                                                                   ptions
                         (to establish the context/baseline prior to FTI)
                                                                                                             ◄=




                                                                                                                    and
                                                                                                                     ck




                                                                                                                      l
    Education              Education                                                          Aid
                                              Data and M&E              Capacity
  policy/planning           finance                                                      effectiveness
                                             Quality and use of
     Quality of          Adequacy of                                                     Extent to which
                                              data relevant for      Extent to which
  education policy     international and                                                aid for education
  and planning in
  relation to UPC
                       domestic finance
                       to meet EFA and
                                                setting and
                                                 monitoring
                                                                       capacity is
                                                                    adequate for EFA
                                                                                        is efficiently and
                                                                                            effectively
                                                                                                             ▲
                                                 education          and UPC targets
      and EFA             UPC targets                                                        provided
                                                 strategies

        ▼                     ▼                      ▼                     ▼                  ▼              ▲




                                                                                                                                                                                                    External influences (political, economic, other) at global and country level that affect the implementation and
                                        Level One – Inputs
                                      (FTI Inputs and Activities)                                            ◄=




                                                                                                                                                                                                        achievement of EFA and UPC goals. Assumptions and risks that affect the programme logic of FTI.
  Global advocacy                                                      Assessing
                       Assessing finance
      for UPC                                                           capacity        Efforts to improve
                       requirements and        Assessing data
     Support to            mobilising         requirements and
                                                                    requirements and      harmonisation
                                                                                                             ▲




                                                                                                                  Experience and learning feed back into subsequent inputs and secondary effects.
                                                                       supporting       and alignment of
    country-level        domestic and          addressing gaps
                                                                        capacity         aid to education
   education plans       external funds
                                                                      development

        ▼                     ▼                      ▼                     ▼                  ▼              ▲
                             Level Two – Immediate Effects
              (Effects on processes in education sector including role of aid)                               ◄=
  Education plans,     Education budget           Improved            Coordinated       More coordinated
   encompassing         process is more       collection of data    implementation of   international aid
  UPC targets, that
    meet quality
                        comprehensive,
                        transparent and
                                                  and better
                                                 information
                                                                      measures to
                                                                       strengthen
                                                                                           that is more
                                                                                          coherent with
                                                                                                             ▲
     standards              efficient              services              capacity       domestic efforts

        ▼                     ▼                      ▼                     ▼                  ▼              ▲
                        Level Three – Intermediate Outcomes
                (Changes in sector policy, expenditure and service delivery)                                 ◄=
                        Increase in total                                                   Aid that is
                                                                        Adequate
  Implementation of    funds for primary      Use of better data                             aligned,
  appropriate sector
       policies
                       education, better
                          aligned with
                                               to inform policy
                                                 and funding
                                                                       capacity to
                                                                    implement policy
                                                                                            adequate,
                                                                                         predictable and
                                                                                                             ▲
                                                                      and services
                         policy priorities                                                 accountable

        ▼                     ▼                      ▼                     ▼                  ▼              ▲
                                 Level Four – Outcomes
      (effects on quantity, quality, access and sustainability of primary education)                         ◄=
                 positive effects on availability of primary education and movement towards UPC target
                 positive effects on access and equity (including gender equity)
                 positive effects on learning outcomes
                                                                                                             ▲
                 sustainability of primary education provision and its quality

         ▼                   ▼                    ▼                   ▼                     ▼                ▲
                                  Level Five – Impact
  (long term personal, institutional, economic and social effects of expanded primary                        ◄=
                                        education)
                 enhanced learning, life skills and opportunities for individuals
                 stronger local and national institutions
                 personal and social benefits in education and other sectors (including health)
                                                                                                             ▲
                 economic growth due to increased human capital




Page 26 of 134                                                                                                    printed: 04-Feb-09
                               Chapter 3: Evaluation Methodology


3.13 The matrix follows a standard structure, in that it proceeds through successive levels
from inputs to impact. Note, however, that there are feedback loops between levels. FTI is
a continuing programme and each phase of activities is influenced by previous ones. Thus
successive levels of the framework are a logical sequence, but the chronological sequence
may involve successive rounds of inputs etc, and the design of the programme may change
over time. The overall time period that is relevant to the evaluation includes the formative
years of FTI, through its formal establishment and (as nearly as possible) up to the present.
For particular countries, the dates of first FTI inputs will vary, and it will be necessary to
record carefully what form they took, as FTI modalities have themselves been evolving.
(One of the strengths of this approach is that it allows for the possibility of different effects in
different countries; it will highlight cases where evaluators need to explain why things work
differently in different contexts.)

3.14 FTI represents only part of the overall efforts towards EFA and UPC. The evaluation
is using contribution analysis as an organising framework for its enquiry. This requires FTI to
be carefully set within the context of these overall efforts; their combined effect is assessed
as a prelude to assessing the particular contributions of FTI to the overall effects. "Level
Zero" is included to ensure that evaluators gather full information on the context and
situation prior to FTI inputs. This is particularly important in addressing the first set of High
Level questions (see Box 3D above) concerning the relevance of FTI, and of its focus on
UPC, in the context of different stakeholders' perspectives and different countries'
engagement with it.3 The five streams of hypothesised effects are separately tracked
through Levels 1–3 (inputs  immediate effects  intermediate outcomesoutcomes),
though interactions between the streams are also sought (there are overlaps between
questions, and the external factors/assumptions column highlights some of the interactions
between streams). Under "assumptions" it is important to check whether different
stakeholders and different countries operate with the same understanding of FTI objectives
and functions. If it is found that they do not, the consequences for efficiency, effectiveness
and sustainability should be explored for all relevant workstreams and evaluation levels.

3.15 At Level 4 (Outcomes) the emphasis is on understanding overall effects on the
quantity, quality, access and sustainability of primary education, then the extent to which
these effects can be explained by direct and indirect FTI inputs. Hypotheses and questions
at Level 4 are interlinked with hypotheses and questions at earlier levels. The enquiry at
Level 4 will establish what the outcomes ("results on the ground") have been; questions at
earlier levels will seek to establish whether and how FTI inputs may have contributed to
those results on the ground.

3.16 The mid-term evaluation of FTI will not attempt to assess the effects of FTI at Level 5
(Impact). However, an impact evaluation scoping study (see Annex G) will assess the
prospects, and make proposals, for monitoring and evaluating FTI impacts over the long
term.

Working Paper Themes
3.17 As shown in Box 2A (Chapter 2 above), a series of thematic working papers will feed
into the Stage1 Preliminary Report. The Working Papers (WPs) echo the main workstreams
that will run through the evaluation. The Working Papers will not be formal outputs of the
evaluation, but, as drafts, will facilitate discussion, consultation and feedback within and
beyond the team, while also feeding directly into the preparation of the Preliminary Report.


3
 Relevance of FTI is also a concern at subsequent levels of the framework. The same applies to the
existence, and possible influence, of differing perspectives on the FTI's objectives and role.


file: FTI_Evaluation_Framework(Jan2009b).doc                                        Page 27 of 134
            Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


They will be a key input to the second Team Workshop, along with the draft findings from the
pilot country studies.

3.18 Papers corresponding to each of the main workstreams will be complemented by
papers on two special topics: issues arising from fragile states and fragile partnerships (see
below), and a scoping paper on impact evaluation (see Annex GG).

3.19 Annex F gives the specifications for the WPs. Each will also be guided by the
detailed hypotheses and evaluation questions for each workstream that are shown in the
extended logical framework (Annex E). Annex D – the mapping of the TOR evaluation
questions onto the workstreams – is also relevant.

3.20     The literature review and data assembly exercises (discussed in the final section of
this chapter) will be a common resource for all workstreams and for the country studies.




The Role of Country Studies
3.21    The TOR (¶21) specified:

       Case studies are expected to be used in this evaluation as a means of developing
       greater insight into country-level processes, accomplishments, and problems, all in the
       context of each country, thus making a contribution to the lessons-learned part of the
       evaluation. Proposals must indicate that a priority will be placed on consulting equally all
       the key actors – government, civil society, and local funding agencies. Unlike desk
       studies the case studies can reach down to sub-national levels for informants and site
       visits to validate desk review findings and examine more details at the service delivery
       level. Case studies will be particularly useful in examining: a) questions about FTI
       relevance from various points of view; b) detailed questions about effectiveness, such as
       policy adjustments and the inclusiveness of the process; effects of capacity building,
       EFA outcomes at the school and community levels, etc), and c) detailed questions about
       efficiency (such as the functioning of the local donors group; any real evidence of donor
       agency changes in aid effectiveness; resource mobilization, and the sustainability of
       systems).

3.22    The consortium proposal noted:

       Country case studies are crucial to the evaluation and will dominate the evaluation's
       resources. It is not possible to draw a statistically representative sample, nor will
       aggregate pair-wise comparisons between countries (whether similar or dissimilar) be
       very revealing. As explained in our methodology, this has to be a theory-based
       evaluation in which causal relationships are carefully disaggregated and examined in a
       variety of contexts. The iterative contribution analysis approach then enables evaluators
       to make informed and transparent judgements about attribution.

3.23 The challenge therefore is to select a set of countries that enables FTI processes and
effects to be studied in a variety of contexts, so as to maximise the insights from the studies
and their contribution to the robustness of the evaluation's overall conclusions.


3.24 Annex H explains the rationale for the Stage 1 pilots (Kenya and Cambodia have
been selected) and for the case studies proposed for Stage 2. Chapter 4 provides
guidelines for undertaking the country studies.




Page 28 of 134                                                                         printed: 04-Feb-09
                                  Chapter 3: Evaluation Methodology


Identifying and Involving Relevant Stakeholders4
3.25 The GRPP Sourcebook highlights the importance of stakeholder participation
throughout the evaluation processes. This should enhance consensus building and
ownership of the evaluation process and its results, while also ensuring that the evaluators
draw on all relevant information sources.

3.26 At the same time it is recognized, that the quest for participation in evaluation of this
kind can put a heavy burden on stakeholders (Samoff, 2005). The FTI‟s philosophy is to
avoid imposing additional burdens and transaction costs on stakeholders. Therefore this
evaluation will strike a careful balance between the importance of participation and the need
to limit the demands on participants.

3.27 Accordingly, the evaluation will draw as much as possible on secondary sources and
on information that is available through regular government systems. Where practical, the
country research will piggy-back on regular events in the country calendar.

3.28     A key area of primary research, however, will be stakeholder interviews. Annex I
explains the evaluation's approach to stakeholder mapping and presents a general
stakeholder map for FTI. This will be supplemented by country-specific identification of
stakeholders and informants for the case studies. The country study teams will complete a
literature review and write a preliminary bibliography for the country before their country visit.
They will also write a draft country background section and a brief issues paper identifying
the areas of focus for the country concerned for review and verification in country. This will
ensure that interviews sought are well-targeted and focused in their topics of enquiry.

3.29 The evaluation's interviews will be the major unique set of data that it generates. The
evaluation team will adopt a strict protocol for writing up and sharing interview notes among
team members (while respecting informant confidentiality). Annex K shows the standard
format for interview notes that will be used.

3.30 It is important to give stakeholders the opportunity to engage with the evaluation and
its emerging findings. This is reflected in the structure of the evaluation (summarised in
Box 2A in Chapter 2). The Preliminary Report, the draft full report, and drafts of the country
studies will all be made widely available for comment, using websites and also direct
circulation to parties that are known to be interested. Final reports will include a note on
comments received and how they have influenced the final drafts. The evaluation team will
collaborate with the EOC to ensure useful debate on the evaluation topics at the various FTI
meetings scheduled in 2009.

3.31 At country level, it is anticipated that approaches will differ depending on country
specificities (including in terms of opportunities offered by on-going events on which the FTI
evaluation can piggy-back). Prior to country visits a stakeholder map will be produced and
populated specific to the country (see Annex I). In all cases the evaluation team will make
sure that stakeholders are informed ahead of the work in-country of the steps of the
processes and of the manner and moments at which stakeholders can provide critical inputs.

Cross-Cutting Issues
3.32 The evaluation needs to take account of a range of cross-cutting issues (CCIs)5
which may affect the FTI directly or indirectly. Gender and equity are mentioned in the TOR.
HIV/AIDS is not mentioned but is a critical factor in many countries. Disability, special needs

4
    See Annex I for an extended version of this discussion.
5
    Although neither the TOR nor the Issues Paper refer explicitly to cross-cutting issues as a category..


file: FTI_Evaluation_Framework(Jan2009b).doc                                             Page 29 of 134
          Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


and other vulnerabilities are important reasons for children being out of school. More
conventional CCIs, such as governance and human rights, may be an important influence on
the relationship between donors and partner countries (and are a core concern for the
evaluation's working paper on fragile states and partnerships).

3.33 The CCIs that will be most relevant will vary according to the theme and the country
concerned. It is important that key CCIs are not overlooked; at the same time, the
evaluation's resources are limited and enquiries need to be focused. Thus, for example, one
of the functions of the country issues papers (see Chapter 4) will be to identify any CCIs that
merit special investigation in a particular country.

3.34   Evaluation enquiries should consider, inter alia:

           Extent and manner in which cross-cutting issues have been mainstreamed in
           endorsed education sector plans.
           Quality of reporting on cross-cutting issues (globally and at country level in FTI
           documentation and in documentation on the education sector in general), i.e. are
           these issues comprehensively addressed in reports, are strategies identified to
           remedy issues, are these issues followed up from one report to the next?
           Extent and quality of debate and discussion around cross-cutting issues in global
           and country level forums (through review of documentation and specific questions
           in interviews).
           Quality of key indicators and targets related to gender, HIV&AIDS and other
           cross-cutting issues – globally and in country studies.
           Progress on key indicators and targets related to gender, HIV&AIDS and other
           cross-cutting issues – globally and in country studies.
           Attention to cross-cutting issues in endorsement guidelines, i.e. what specific
           steps and procedures have been included to ensure that cross cutting issues are
           assessed and become an integral part of the education plan, to what extent have
           these procedures been followed at country level, how is this reflected in the
           endorsed education sector plan.
           Attention to cross-cutting issues in the endorsement process itself (i.e. have
           cross-cutting issues been adequately discussed, were there any specific
           concerns on cross-cutting issues, were specific technical studies conducted, and
           are the results of these processes reflected in Education plans that adequately
           take account of these cross-cutting issues in terms of priorities, strategies,
           funding, and monitoring and evaluation arrangements?).
           Involvement – if any – of People Living with HIV&AIDS (PLHA) in endorsement
           processes and in education response in general.

3.35   In terms of evaluation approach:

           Ensure gender balance in the selection of informants whenever possible,
           including when interviewing end users (the target group). Ensure adequate
           representation of PLHA among those interviewed.
           Integrate specific questions on cross-cutting issues in the interview guidelines.
           Ensure gender (and geographical) balance in evaluation teams, and ability to
           cover the appropriate range of CCIs.




Page 30 of 134                                                                printed: 04-Feb-09
                               Chapter 3: Evaluation Methodology


           Include consultations with main (non-education) stakeholders in the HIV & AIDS
           response (National AIDS Council etc) to get their inputs on the quality of the
           education response. Idem for gender issues.
           Comprehensively report on the manner in which cross-cutting issues have been
           addressed in each of the main evaluation areas (relevance, efficiency and
           effectiveness), what achievements are evident and what areas require
           strengthening.




Literature Review and Data Management
3.36     The different workstreams and country study teams are supported by a common
pool of research resources to review documents and gather data.

3.37 For efficiency, early effort will focus on building up a common collection of relevant
documents and data (the project website will serve as an e-library for team members). The
output will be an annotated bibliography and guide to available data, which will be continually
updated as the evaluation proceeds.

3.38 The evaluation will draw systematically on existing data sources, including the
UNESCO Institute for Statistics (UIS) and the Global Monitoring Report (GMR), and the
OECD DAC series on aid flows, and on reviewing the strengths and limitations of these data
for the evaluation's purposes. The evaluation team is also drawing on the resources of the
Education Policy and Data Centre (EPDC) – see www.epdc.org.

3.39 Both literature and data searches will address country as well as global level, with an
initial focus on the pilot study countries and other planned case studies.

3.40 On documents, besides fully documenting the progress of FTI itself, the evaluation
will draw as much as possible on relevant previous reviews and evaluations, as well as
seeking documentary evidence of stakeholder positions prior to interviews.

3.41 Particular attention will be paid to existing and ongoing studies that are relevant to
the FTI evaluation themes. Relevant studies include, for example, the series of global and
country level evaluations of the Paris Declaration and studies of the use of country systems
(cf. Mokoro 2008a). Two ongoing studies of particular importance are the SPA-
commissioned case studies of sector budget support (ODI and Mokoro 2008), and an ODI
study entitled Achieving Universal Primary Education: Constraints and Opportunities:

      The purpose of the study is to identify and analyse the key factors that influence donors
      when deciding how best to allocate their efforts and funds to different development
      objectives, with a focus on primary education. It is intended to provide an informed basis
      for reflection and dialogue among aid agencies, business and civil society organisations.
      This study will explore strategic, operational and country level reasons for donor
      behaviour in supporting primary education as one of the MDGs. (ODI 2008)


3.42 Annex J1 provides a general timeline of FTI-relevant events. Country-specific
timelines will be included in each of the country reports.

3.43 Annex J2 shows how the engagement of different eligible countries with the FTI has
developed over time.




file: FTI_Evaluation_Framework(Jan2009b).doc                                         Page 31 of 134
         Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative




Page 32 of 134                                                           printed: 04-Feb-09
           Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative



4.     Conducting Country Studies
Overview
4.1    Country studies will be guided by the logical framework for the evaluation that is
summarised in Chapter 3 and fully specified in Annex E. The present chapter expands on
the objectives of the country studies, the process for conducting them and the outputs
expected.

4.2    The TOR (¶21) specified:
      Case studies are expected to be used in this evaluation as a means of developing
      greater insight into country-level processes, accomplishments, and problems, all in the
      context of each country, thus making a contribution to the lessons-learned part of the
      evaluation. Proposals must indicate that a priority will be placed on consulting equally all
      the key actors – government, civil society, and local funding agencies. Unlike desk
      studies the case studies can reach down to sub-national levels for informants and site
      visits to validate desk review findings and examine more details at the service delivery
      level. Case studies will be particularly useful in examining: a) questions about FTI
      relevance from various points of view; b) detailed questions about effectiveness, such as
      policy adjustments and the inclusiveness of the process; effects of capacity building,
      EFA outcomes at the school and community levels, etc), and c) detailed questions about
      efficiency (such as the functioning of the local donors group; any real evidence of donor
      agency changes in aid effectiveness; resource mobilization, and the sustainability of
      systems).

4.3      Chapter 2 (Box 2A) shows how the country studies fit into the overall structure of the
evaluation. Annex H explains the rationale for the selection of country case studies. The
aim of the two pilot studies during Stage 1 is to test the country study methodology, allowing
it to be refined before the main suite of seven further country studies takes place during
Stage 2 (see Box 4A).

                          Box 4A: Role of the Pilot Country Studies

The pilot country studies will test the logical framework itself, providing feedback on its
clarity, relevance and completeness. Beyond that, they will be a check on how much can
be accomplished in the time available, and what seem the most efficient methods of
proceeding. There will be particular attention to the balance between preparatory work and
work in-country, the identification of stakeholders and practical ways of interacting with them,
and the role of in-country field visits.
Perhaps most important, the pilots should yield a template for the structure of the later CS
reports, and a pattern for the process they will adopt.
The pilot country study team leaders will attend the second Team Workshop, in order to
provide feedback on the country study process as well as the pilot studies' initial findings.
Lessons will be incorporated in the work plan for Stage 2 of the evaluation.




4.4     The methodology described in Chapter 3 applies equally to the country studies. In
particular, the hypotheses summarised in Figure 3A and elaborated in Annex E will be tested
at both global and country levels. (Annex E indicates which evaluation questions are more



file: FTI_Evaluation_Framework(Jan2009b).doc                                           Page 33 of 134
           Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


pertinent at country and global levels.) Country studies willtake account of the different
perspectives of different stakeholders and consider the different streams of effects
(education policy and planning, education finance, capacity, data and M&E, aid
effectiveness) depicted in Figure 2A and reflected in the extended logical framework. They
will share the same concern to establish outcomes ("results on the ground") and to assess
whether and how FTI inputs may have contributed to those results. However, the country
study teams are quite small; team members will each need to cover more than one stream of
issues, and the teams will also need to be selective in identifying the issues to be followed
up, as it would not be practical to pursue exhaustively every evaluation question in Annex E.

Country Study Outputs
4.5     The output of each country study will be a free-standing final country report. This is
intended to be of value for all stakeholders at country level as well as contributing to the
overall evaluation.

4.6      The country study report will need to document how the FTI has interacted with and
contributed to country education processes. A first requirement for the country studies is
therefore to document what key changes have taken place in the education sector over time,
including trends in public expenditure and aid flows to the sector. A second task is to identify
where and in what ways FTI has made a specific contribution. A third task is to assess the
relevance, effectiveness and efficiency of these contributions. (In doing so, the country
studies will need to take account of the possibility that different stakeholders may have
operated with different interpretations of the FTI's objectives and role.) Finally, the reports
will need to present evidence of the outputs and outcomes of these interventions. The
studies will need to place the education sector firmly within the overall country context and
will also need to tease out to what extent and in what ways FTI has had an impact on the
education sector specifically.

4.7     In line with the contribution analysis approach, the reports will be structured as
follows:
           a) Introductory: purpose and methodology of the report.
           b) Overview of basic education in [country]. In the context of EFA objectives
              (and UPC in particular) what progress has been made; what is the current
              status and what are the prospects for meeting the 2015 targets? How have
              domestic and foreign resources, and different stakeholders, contributed?
              This review is to be set in the context of broader national strategy and
              performance, and the overall context of aid flows and systems for aid
              management.
           c) Review of FTI-related inputs and activities, documenting carefully what
              distinct inputs can be attributed to FTI and assessing the contributions
              (positive or negative) that were made in terms of education policy and
              planning, education financing, data and M&E, capacity development, and
              overall aid effectiveness.
           d) Summary of conclusions and recommendations.
           e) Annexes: for data and for detailed supporting information.

4.8     More detailed country report outlines will be developed with the pilot study teams in
January 2009, and the pilot studies will provide the pattern for the subsequent country
reports. Box 4B notes the special features involved in the study of non-endorsed countries.



Page 34 of 134                                                                  printed: 04-Feb-09
                            Chapter 4: Conducting Country Studies


               Box 4B: Case Study Approach in Non-Endorsed Countries
As noted in ¶1.12 above, it is necessary not only to review what the FTI has done (e.g. in its
endorsed countries) but also to examine what FTI hasn't (yet) accomplished and why.
Hence it is expected that two non-endorsed countries will be included among the case
studies (see Annex H on the choice of country case studies).
The contribution analysis methodology is well-suited to asking why not? questions about
FTI's role, and the structure of country reports on endorsed and non-endorsed countries will
be similar. In the non-endorsed countries there will be the same concern to establish the
context (the then and now of basic education performance and the contributions aid has
made). Beyond that there will be careful documentation of how the country has featured in
FTI aspirations and actual FTI contacts; assessment of any FTI inputs (EPDF, influence on
donor architecture etc), consideration of plans and prospects for future FTI roles and inputs;
and considerations of the relevance of FTI (in its various formulations) to country needs. All
this will be used to inform forward looking judgements about whether and how FTI could be
relevant in future.


Key phases of the country study work
4.9    The country studies will involve four overlapping phases:
          Phase 1: Preparation for the country study (including initial issues paper)
          Phase 2: The country visit
          Phase 3: The country aide memoire (which partially overlaps with the country visit
          itself)
          Phase 4: The country report.


4.10 The final product of the country visits is the country report. The country aide memoire
is an intermediary product and will be an important building block for the final report. (Note
that the aide memoire is an initial summary of country findings: it represents the (preliminary)
views of the evaluation team and it is circulated for information and to elicit comments.
Country endorsement is not sought.)

4.11 The evaluation is based on the assumption that the country visits will build on a
substantial amount of preparatory work. This will include a specific review of documentation
on the country itself, to produce the following preliminary products:
           A series of annexes on the country and education context, including:
              o A time line of FTI events at country level.
              o A summary of Basic Education Indicators for the country.
              o A summary of public expenditure and aid flows to the education sector.
              o An analysis of EFA-FTI Catalytic Fund Allocation for the country (where
                  appropriate).
              o An analysis of EFA FTI Education Program Development Fund allocation
                  for the Country (where appropriate).
              o An analysis of education sector expenditure 2002–2008.
              o A preliminary stakeholder map of key education stakeholders (donors,
                  civil society, MoE, other sectors) indicating nature and type of involvement
                  of stakeholders.



file: FTI_Evaluation_Framework(Jan2009b).doc                                    Page 35 of 134
          Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


           A narrative which outlines the country and education sector context and how it
           has evolved over time.
           A brief issues paper (internal to the team in the first instance), indicating which
           are the areas of focus for the country study.

4.12 This issues paper will draw on the preliminary documentary review by the team to
highlight what issues are already relatively clear from the prior analysis, which issues will
need to be further addressed, and of these which will be the subject of examination at
country level as they will add most value to the evaluation. This particularly important
because of the evaluation‟s broad scope, and the need therefore to be selective. Among
other aspects, the issues paper will identify which cross-cutting issues require special
attention (see Chapter 3, ¶3.32–3.35).

4.13 The work in country will concentrate on the key areas identified in the issues paper.
For efficiency reasons, and to reduce the burden that may be associated with the
consultation process, the engagement of stakeholders will be sought through a variety of
means, including stakeholder interviews (face to face, at a distance, combined, individual),
short questionnaires, thematic discussions, feedback on sub-products (e.g. country aide
memoires), e-consultations, workshops, field visits, and collection of additional
documentation to further develop the report. This process will minimize transaction costs and
ensure that the various preliminary products of the evaluation which will be available are
used to maximum effect.

4.14 At country level the approach to engaging with stakeholders will vary depending on
country specificities (including in terms of on-going events which the evaluation may take
advantage of). The team leader will be responsible for identifying opportunities and for
deciding on which approaches to take. In all cases the evaluation team will make sure it
informs stakeholders ahead of time of the work in country, of the steps of the process, and
the manner and moments in which stakeholders can contribute.

4.15 Table 4A below summarizes the phases of the country studies. It provides an
overview of the key steps for each phase and specifies what the expected outputs are at
each stage.




Page 36 of 134                                                                  printed: 04-Feb-09
                                Chapter 4: Conducting Country Studies


                        Table 4A: Phased preparation of country studies
                           Phase                                     Outputs
                                     Phase 1: Preparation
Key steps:                                                           Outputs for phase 1:
   Familiarization based on the evaluation's literature review       1. Programme for the country visit.
   and data assembly                                                 2. Complete draft set of annexes.
   Reading of the thematic working papers (as available).            3. A preliminary bibliography for the
   Reading of the country specific documentation on the                 country.
   evaluation website.                                               4. A draft background section to the
   Preparatory work on developing the annexes to the report             report outlining the overall context, the
   based on available data (to be completed in country with             FTI factual (what FTI did), and how
   additional data).                                                    the context and the FTI has evolved
   Preparation of a first draft outlining the overall context, the      over time.
   FTI factual and how FTI has evolved over time.                    5. A brief issues paper identifying the
   Preparation based on the analysis above of a brief issues            areas of focus for the country
   paper outlining key issues of focus for the country study            concerned.
   work.
   Identification and briefing of local consultants.
   Initial contact with local education group on arrangements
   for the study and request for any additional documentation.
   Preparation of a list of interviewees (stakeholder map) in
   consultation with the lead donor and the local consultant and
   setting up of the meetings.
   Preliminary decisions on the various ways in which data will
   be collected from the stakeholders and inclusion in the
   program of work.
   Sending out of emails to the country stakeholders (this will
   need to be coordinated with the lead donor) alerting about
   the study and the process.
   Preparation of the field visit, including the site visits.

                                       Phase 2: The country visit
Key steps:                                                           Outputs for phase 2:
   Initial briefing session for the main stakeholders in country     1. Interview and field visit notes for all
   on the purpose of the study (including information about the         interviews, focus groups, observations.
   country aide memoire).                                            2. List of supplementary documentation.
   Interviews with selected stakeholders, focus groups and field     3. Draft aide memoire.
   visit as defined in the country visit program, including
   attention to cross-cutting issues.
   Collection of additional data and documentation as per gaps
   identified during the field visit preparation.
   Drafting of preliminary version of the country aide memoire.
   Debriefing to the key stakeholders in country.
   Follow-up interviews by phone as necessary (for key
   stakeholders who were not in country).
                                 Phase 3: The country aide memoire
Key steps:                                                           Output for phase 3:
   Circulation of the country aide memoire to all stakeholders       1. No separate output but the aide
   interviewed with a request for inputs.                               memoire plus comments received will
   Drafting of the country report following the agreed outline          feed into the country report.
   and taking into account the inputs received on the country
   aide memoire.




file: FTI_Evaluation_Framework(Jan2009b).doc                                                   Page 37 of 134
            Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


                           Phase                                                Outputs
                                    Phase 4: The country report
Key steps:                                                       Output for phase 4:
   Preparation of a first draft of the country report in         1. Final country report which integrates
   accordance with outline provided.                                comments received from evaluation
   Circulation of the draft report for comments                     team and selected stakeholders
   Finalization of the draft report based on comments received



4.16 The approach to stakeholder consultation is described in Annex I. This includes a
template for identifying key stakeholders in each study country. The country team will need
to select from these a manageable number for formal interviews (bearing in mind that some
relevant informants may no longer be in the country, and may need to be contacted by
telephone or e-mail).


Stakeholder consultation and research
4.17 During the country visits the key evaluation issues will be examined using a
combination of interviews, site visits and supplementary data collection.

4.18 Interviews can be conducted individually or collectively, at the discretion of the team
leader. Formal interviews will be written up in a standard format, so as to form a body of
evidence that will be available (in confidence) to all evaluation team members. A standard
template for interview notes will be used (see Annex K).

4.19 The team should consult a representative group of stakeholders including the
Ministry of Education (various levels), other government ministries (especially finance and
planning), the lead donor, other donors, civil society representatives, teachers, students and
parents. The collective (focus group) interviews may be most appropriate for the
„operational and beneficiary levels‟ (teachers, parents, pupils, district staff). Focus groups
with sub-groups of stakeholders (donors, education ministry staff) may also be helpful at the
end of the study for discussion of emerging findings. E-consultation, thematic meetings and
feedback by selected stakeholders on drafts (for example of the country aide memoire) can
also be considered.

4.20 Where possible the team should take advantage of existing events so as to reduce
the burden of the evaluation at country level. The team leader will be responsible for
deciding on priority issues which will be examined at country level (this will be indicated in
the brief country issues paper which was mentioned above) and for establishing which
methodology is best for each interview and whether the interviews will be conducted
together by members of the team or by each member individually.

4.21 Site visits will be undertaken to a selection of sector institutions and service
providers, including at local level where possible. It is suggested that the field visits take up
about two days out of the total time (two weeks) allocated to the field work. They are,
however, essential to the overall study and should be given due attention. The country lead
donor and the local consultant will be essential in providing advice on the best locations and
in facilitating the preparations for the field visit.




Page 38 of 134                                                                         printed: 04-Feb-09
                            Chapter 4: Conducting Country Studies


4.22 Supplementary data collections may include overall country statistics (on poverty
etc.), education and financial statistics over time, education reports and evaluations of
reference to the study, tracking studies, etc. While much of the relevant data will be
assembled in advance, the country visit will need to fill gaps and resolve queries.


Team Roles
4.23 Across country studies team composition may vary according to the size of the study
etc. Roles for team members will be adapted to their particular skills and country
requirements. In outline:

   o   The CS team leader (CSTL) has overall responsibility for managing the CS team and
       delivering the report. S/he works under the overall direction of the evaluation team
       leader (Stephen Lister).
   o   Each team will have one or two additional international consultants, and at least one
       national consultant.
   o   Allocation of roles (e.g. which workstreams will be covered by whom) will be agreed
       with the CSTL.
   o   National consultants (and, where available, international consultants resident in the
       country) will play a special role in study preparation – liaising with in-country
       stakeholders, helping to collect data and prepare an itinerary ahead of the field work,
       etc., and following up as necessary after the country visit.
   o   The central evaluation team also has research resources (coordinated by Anthea
       Sims Williams) and will assist in compiling country-level information and material.




file: FTI_Evaluation_Framework(Jan2009b).doc                                   Page 39 of 134
         Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative




Page 40 of 134                                                           printed: 04-Feb-09
           Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative



References
Bruns, B., A. Mingat, and R. Rakotamalala (2003), Achieving Universal Primary Education
     for All by 2015: A Chance for Every Child. Washington, D.C.: World Bank.
Buse, K. (2005), Education for All – Fast Track Initiative: Review of the governance and
     management structures. London, Overseas Development Institute.
Buse, K. (2007), Education for All – Fast Track Initiative: An Embodiment of the Paris
     Declaration on Aid Effectiveness? Report prepared for Global Programs and
     Partnerships (GPP), World Bank, as a component of a multi-country study on Global-
     Country Integration of Global Programs.
Education for All – Fast Track Initiative (2004), Accelerating progress towards quality
    universal primary education: Framework.
,(2008), Guidelines for Capacity Development in the Education Sector.
Education for All – Fast Track Initiative Secretariat. (2007), Quality Education for All
    Children: Meeting the Challenge. Annual Report 2007.
Education for All Global Monitoring Report. (2007), Education for All by 2015: Will We Make
    It?
EFA-FTI Co-Chairs (2005), The Future of the Education For All – Fast Track Initiative.
    Unpublished.
Foster, M (2005), Costing the Education MDGs: A Review of the Leading Methodologies.
     Report at a meeting of the Global FTI Partnership, Croydon, UK.
IDD & Associates (2007), Joint Evaluation of General Budget Support: Note on Approach
     and Methods. (February 2007) Glasgow: DFID.
IEG (2007), Sourcebook for Evaluating Global and Regional Partnership Programs:
    Indicative Principles and Standards. Independent Evaluation Group – World Bank.
    Washington DC.
Mayne, J. ,(1999), Addressing Attribution Through Contribution Analysis: Using Performance
    measures Sensibly. Office of the Auditor General of Canada.
Mayne, J. (n.d.), Using Contribution Analysis to Address Cause-Effect Questions: Theory
    and Concepts. Powerpoint presentation.
Mokoro and CIPFA (2008), Stocktake on Donor Approaches to Managing Risk when Using
    Country Systems. Jeremy Cant (CIPFA), Rebecca Carter and Stephen Lister (Mokoro),
    May 2008.
Mokoro Ltd. (2008a), Putting Aid On Budget Synthesis Report. (April 2008).
Mokoro Ltd. (2008b), Good Practice Note: Using Country Budget Systems. (June 2008).
ODI (2008), Achieving Universal Primary Education: Constraints and Opportunities [study
     description; contact - Dr. Liesbet Steer].
ODI and Mokoro (2008), Sector Budget Support in Practice: Inception Report (Final Draft,
     21st November 2008)
Netherlands Ministry of Foreign Affairs (2003), Local Solutions to Global Challenges:
     Towards Effective Partnership in Basic Education - Final Report. Joint Evaluation of
     External Support to Basic Education in Developing Countries. Netherlands Ministry of
     Foreign Affairs.




file: FTI_Evaluation_Framework(Jan2009b).doc                                     Page 41 of 134
          Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


OECD DAC (2002), Glossary of Key Terms in Evaluation and Results Based Management
   [in English, French and Spanish]. Paris: OECD DAC (DAC Working Party on Aid
   Evaluation).
OECD DAC (2005), Joint Evaluations: Recent Experiences, Lessons Learned and Options
   for the Future. DAC Evaluation Network Working Paper, prepared by Dr Horst Breier.
   Paris: OECD.
OECD DAC (2006a), DAC Evaluation Quality Standards, Paris, March 2006.
OECD DAC (2006b). Guidance for Managing Joint Evaluations. DAC Evaluation Series.
   Paris: OECD.
Pawson (2006, Simple Principles for the Evaluation of Complex Programmes. In Public
    Health Evidence, Oxford.
Samoff, J. (2005), Imaginative Innovations and Muddling Through in Joint Evaluations:
    Observations Drawn from Local Solutions to Global Challenges: Towards Effective
    Partnership in Basic Education. Presentation at Joint Conference of the Canadian
    Evaluation Society and the American Evaluation Association.




Page 42 of 134                                                            printed: 04-Feb-09
             Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative



Annex A: Terms of Reference
External Evaluation of the Education for All – Fast Track Initiative

Final Terms of Reference dated 8 July 2008


BACKGROUND
1.       The Fast Track Initiative Partnership has established an Evaluation Oversight
Committee to commission an evaluation of the Initiative‟s work at the approximate midpoint
between the inception of the FTI and the target date for achieving the EFA goals the
Initiative was designed to support. The EOC is therefore seeking a qualified firm or group of
firms to conduct a formative mid-term evaluation of the FTI‟s relevance, effectiveness and
efficiency, provide recommendations designed to improve the initiative going forward, and
create a monitoring and evaluation framework for continued assessment of its processes,
outcomes and impacts.

FTI Objective, Goals and Assumptions
2.      The Education for All – Fast Track Initiative (EFA-FTI) is an evolving partnership of
developing and donor countries and agencies having the main objective of “accelerating
progress towards the core EFA goal of universal primary school completion (UPC), for boys
and girls alike, by 2015.”6 It was established in 2002 by 22 bilateral donors, development
Banks and international agencies, prompted by the 2000 Dakar World Forum on Education,
which yielded both the current EFA goals and a commitment to increased financial support
for basic education.7 Also, as an outgrowth of the 2002 Monterrey Consensus, FTI was
designed as a compact, linking increased financial support and country improvements in
“policy performance and accountability for results.”8

3.     According to its Framework document (2004), the FTI‟s major contributions to
accelerated UPC is through the following, which also captures some of the assumptions
behind the goals:
       Sound sector policies in education: The FTI assumed that one of the best ways
       to accelerate UPC and contribute to EFA progress at the country level was through
       the strengthening of sector policies and use of benchmarking;
       More efficient aid for primary education: the FTI assumed that countries could
       focus more attention and resources on service delivery if aid partners coordinated
       better among themselves and adopted more efficient, harmonized approaches to
       support along lines that were eventually codified in the Paris Declaration (2005);9
       Sustained increases in aid for primary education: the FTI‟s call for more
       sustainable support assumed that longer-term and predictable aid to education would

6
   Education for All – Fast Track Initiative (2004), Accelerating progress towards quality universal
primary education: Framework, p. 3.
7
 The Dakar Forum communiqué stated that “No countries seriously committed to Education for All
will be thwarted in their achievement of this goal by lack of resources.”
8
  EFA-FTI Framework., p. 4. The FTI is considered to be the first global initiative to operationalizes
the Monterrey Consensus and has become an example for other sectors.
      9
          OECD/DAC (2005), Paris Declaration on Aid Effectiveness: Ownership, alignment,
          harmonization, managing for results, and mutual accountability. High Level Forum, February
          28-March 2.



file: FTI_Evaluation_Framework(Jan2009b).doc                                            Page 43 of 134
             Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


          allow countries to plan better and take the needed bold steps forward;
          Adequate and sustainable domestic financing for education: most increases in
          financing would still come from domestic sources and these needed to be substantial
          and sustainable to allow for breakthroughs in service delivery; the FTI looked for
          evidence of increased and sustainable funding for education in a country‟s poverty
          reduction strategy, medium-term expenditure framework, or other country
          statements, as appropriate;
          Increased accountability for sector results: the FTI acknowledged the boost to
          effectiveness provided by an open and transparent reporting of progress and key
          sector outcomes against a set of appropriate indicators.

Through such contributions to country progress on EFA goals, the FTI aspired to help
countries close four gaps: financial, policy, capacity and data.10

Guiding Principles
4.       The 2004 Framework set forth the following as guiding principles of the Fast Track
Initiative:
         Country-ownership: the FTI is a country-driven process, with the primary locus of
         activity and decision-making at the country level;
         Benchmarking: the FTI encourages the use of indicative benchmarks (the FTI
         Indicative Framework), locally adapted, to stimulate and enlighten debate over
         policies, to facilitate reporting of progress on both policies and performance, and to
         enhance mutual learning among countries on what works to improve primary
         education outcomes;
         Support linked to performance: The FTI is intended to provide more sustained,
         predictable and flexible support to countries that have demonstrated commitment to
         the goal of UPC, adopted policies in full consideration of a locally adapted FTI
         Indicative Framework, and have a need for, and the capacity to use effectively,
         incremental external resources;
         Lower transaction costs: The FTI encourages donor actions to provide resources
         to developing countries in a manner which minimizes transaction costs for recipient
         countries (and for the agencies themselves);
         Transparency: The FTI encourages the open sharing of information on the policies
         and practices of participating and donors alike.

The FTI Mechanisms and Processes
5.       During the early phase of the FTI, low-income countries having Poverty Reduction
Strategy Papers (PRSPs) and strong Education Sector Plans were invited to submit
proposals for FTI support, with their proposals being reviewed and endorsed at the global
level. As FTI evolved it opened the partnership to all low-income countries (as determined by
IDA eligibility) and based the endorsement process on a review of country programs and
priorities articulated in an existing education sector plan. Also, instead of being undertaken
at the global level, the process is now managed at the country level by Local Education
Groups – involving government, civil society and the development agencies having
educational support programs in the country. To establish a common process for endorsing
national education sector a common framework and assessment guidelines were adopted in
2005.

6.       FTI endorsement has also involved review of a country‟s commitments on a common
set of indicators as set out in the FTI “Indicative Framework.” This comprised a set of policy

10
     See www.education-fast-track.org, “About FTI.”


Page 44 of 134                                                                  printed: 04-Feb-09
                                 Annex A: Terms of Reference


and financing parameters considered “indicative” of good progress towards UPE.11
Benchmarks on these indicators have been provided, based on average figures from a set of
“on-track” countries for universal primary school completion,12 but potential FTI partner
countries have been encouraged to set their own target levels according to local preferences
and constraints and to use these to encourage dialogue among country-level stakeholders.
Ultimately, accountability in FTI is based on an annual review of progress on Indicative
Framework commitments as well as other nationally developed indicators agreed to during
the endorsement process.

7.      In 2003 and 2004 two new financial instruments (or Trust Funds) were established
within the FTI: the Catalytic Fund (CF), and the Education and Program Development Fund
(EPDF). The Catalytic Fund was set up to provide grant financing over a 2 to 3 year period
to “endorsed” countries with a limited number of donors. The CF is managed by the World
Bank but country executed. As of February 2008, funders had pledged US$1.3 billion to the
CF. The Fund had disbursed US$218 million to 18 countries as of December 2007. With
the recent change in the CF to allow longer term financing (up to 3 year periods through
2015) and access by all endorsed countries having substantial financing gaps even after
domestic funding mobilization, it is anticipated that the fund will need to grow substantially in
coming years.

8.      The Education and Program Development Fund (EPDF) was established in 2004 to
provide eligible countries access to grant financing for capacity building (e.g., analytic work
for planning and budgeting or training) and to support cross-country learning experiences. In
particular, the Fund aims to increase the number of low-income countries with sound
education sector programs, to strengthen country capacity to develop policies through a
broad-based consultative process, to improve and share knowledge of what works, to
strengthen donor partnerships and harmonization at the country level, and to strengthen
partnerships with regional networks and institutions. The EPDF had disbursed US$15
million (of US$24.2 million committed) to over 60 countries as of December 2007. As in the
case of the CF, it is managed by the World Bank.


FTI Governance and Management
9.       The FTI has a two-tiered governance and decision making structure, consisting of
global and country-level entities. At the global level the ultimate authority is the full FTI
Partnership, which includes FTI countries, donor agencies and NGOs, lead by co-chairs
(bilateral agencies) one from a G-8 country and the other a non-G-8 donor, which serve for
one year on a rotating basis. The Full Partnership meets biannually (prior to 2006 annually)
to deliberate progress and set strategic policies and directions. The chief executing body of
the FTI is its Steering Committee; it implements Partnership policies and makes operational
decisions. It originally had eight members: six development agencies plus one each from a
developing country and civil society, but as of 2006 two more seats have been added for
developing countries and two for civil society organizations. A small FTI Secretariat, hosted
and legally managed by the World Bank (see staffing chart in Annex B), provides
management and technical support to the Partnership, promotes communication at all levels,
and provides support to the Steering Committee and to the FTI trust funds (CF and EPDF).
Between September 2004 and March 2008 the Secretariat disbursed around US$3.5 million

11
   Based on research reported in Bruns, B., A. Mingat, and R. Rakotamalala (2003), Achieving
Universal Primary Education for All by 2015: A Chance for Every Child. Washington, D.C.: World
Bank. For a display of the Indicative Framework and related benchmark values see EFA-FTI (2004),
Framework.
12
   See Bruns et al., above.


file: FTI_Evaluation_Framework(Jan2009b).doc                                     Page 45 of 134
          Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


for its services, or about US$1 million per year (see Annex C). At the country level FTI
operates through a Local Education Group, consisting of government officials, civil society
groups and local donor agency representatives. The donor agencies join to form a Local
Donor Group, chaired by a “coordinating agency.” The Local Donor Group serves as a
primary link with the country on FTI operational issues, makes endorsement and funding
decisions, mobilizes funding needed for endorsed sector plans, and promotes aid
effectiveness.

Evolution of the FTI
10.     As a global partnership established to operationalize the Monterrey Consensus in the
education sector and accelerate action on EFA, the FTI had no precedents; thus, it has been
both exploratory and experimental. Consequently, the partnership has evolved considerably
over its almost six years of existence. For example, originally partner countries were
selected by invitation on the basis of the strength of their poverty reduction and sector plans
and a record of good governance. Eventually the Initiative evolved, first to include all low-
income countries having poverty reduction and sector plans and finally, even more
inclusively, to include fragile states for which different requirements and approaches are still
being worked out. There also has been an evolution in the endorsement process: at first
countries were invited to submit FTI proposals which were reviewed globally; more recently
they have simply been required to submit their current (or updated) sector plan for review
with the review being at the country (not global) level. The Catalytic Fund as well has
evolved from a bridging fund for a few countries to a source of funding for almost all FTI
endorsed partners. Over the years the FTI has formed various task forces to examine and
recommend changes in its processes or mechanisms, and as a result some of the ways the
FTI conducts business have changed and continue to change through a variety of processes
and backed by varying amounts of analytical work and consultations.

PURPOSE AND SCOPE OF THE EXTERNAL EVALUATION
11.    This evaluation is being commissioned by the FTI Partnership through its Steering
Committee. The Steering Committee has appointed an Evaluation Oversight Committee
(EOC) to manage and assure the independence of this evaluation. The evaluation is being
conducted at the mid point between FTI‟s founding in 2002 and the culmination date of the
Education for All goals (2015), and thus will be mostly formative in nature.

12.      The main purpose of the evaluation is to assess the effectiveness of FTI to date in
accelerating progress towards achievement of EFA goals in participating countries, with
particular attention to country movement towards universal primary completion (UPC). The
evaluation will also assess FTI‟s contributions to improving aid effectiveness at both the
country and global levels. The evaluation will assess the Initiative‟s added value, identify
lessons learned from its strengths and weaknesses, and formulate recommendations for
improved partnership programming and effectiveness. Given the fact that the FTI is a global
program in the “established” stage (see GRPP Sourcebook),13 the evaluation will focus on
the issues considered appropriate for this status: resource mobilization, strategy, inputs,
activities, and outputs, as well as a reanalysis of program design, governance and
management structures, and recommendations for improvements as the Initiative moves
forward. Some EFA outcomes will also be a focus since this is the main realm where FTI is
expected to add value, but only in countries that have been in the partnership long enough to
show results.

13.    The scope of the evaluation should cover the entire period of FTI‟s existence, from its

13
  World Bank, Independent Evaluation Group (2007), Sourcebook for Evaluating Global and
Regional Partnership Programs: Indicative Principles and Standards.


Page 46 of 134                                                                 printed: 04-Feb-09
                                 Annex A: Terms of Reference


founding in 2002 through September 2008, and assess the evolution of its goals and
processes over that period with the objective of identifying lessons-learned to guide
recommendations for the future direction of the Initiative. Given the fact that FTI processes
and programs are at both the international (global) and national level, the scope will include
both international and country program and progress. Some data will be compiled or
collected on all 35 FTI endorsed countries (see Annex E for a list of countries) mostly from
existing archives held by FTI, the countries and development agencies. More in-depth data
will be collected on case study countries only. The Initiative has not been established long
enough to allow for an evaluation of impact, but a comprehensive design for FTI monitoring
and evaluation, including an evaluation of impact and relevant baseline data, should be one
of this evaluation‟s outputs.

14.   The estimated budget provided is an indication only, and consultants are free to
propose their own estimates.

OBJECTIVE OF THE EXTERNAL EVALUATION
15.      The objective of this evaluation will be to assess the current relevance, effectiveness,
efficiency of the Fast Track Initiative. More specifically, it will assess:
      The relevance of FTI objectives and goals to country educational and development
      needs (relevance);
      FTI‟s contribution (value added) to country progress in accelerating action on EFA, in
      particular towards UPC, and related processes (effectiveness);
      FTI‟s contribution (value added) to improved resource mobilization and aid
      effectiveness (efficiency);
Additionally, the evaluation will develop a comprehensive framework for monitoring and
evaluation of the FTI, building on the evaluation framework created for this evaluation, which
will help frame future monitoring and evaluation efforts, including the assessment of FTI
impact. The comprehensive framework should indicate baseline data needed for the
evaluation of FTI impact, much of which will be collected during this evaluation.

AUDIENCES FOR THE EVALUATION
16.     The main audiences for the evaluation will be the full FTI partnership (partner
country actors at the national and sub-national levels, and representatives from participating
development agencies and civil society organizations), the FTI Steering Committee (which
commissioned the study), and the FTI Secretariat.

EVALUATION FRAMEWORK AND QUALITY STANDARDS
17.     This evaluation will be expected to apply three of the five DAC evaluation criteria for
evaluating development assistance, relevance, effectiveness, and efficiency. Because it is
primarily a formative evaluation it will not be expected to fully apply the other two, impact and
sustainability. However, as mentioned above, it will be expected to provide guidance for the
assessment of the impact of the FTI at a later stage and to gather baseline data for such an
assessment at both the global and country levels (e.g. in case study countries). Also, even
though it is too early to assess the sustainability of FTI beyond its implicit closing time (at
EFA culmination in 2015), issues of sustainability at the country level should be addressed in
this evaluation, for example, in examining the sustainability and predictability of domestic
funding for primary or basic education and the effects of capacity building efforts on long
term institutional capacity.

18.    As an evaluation of a global or regional partnership program (GRPP) this evaluation
should also be oriented by the GRPP evaluation Sourcebook (see footnote 6). This



file: FTI_Evaluation_Framework(Jan2009b).doc                                     Page 47 of 134
           Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


Sourcebook sets quality standards for global program evaluations, including, for example,
the need for an examination and critique of the global programs existing monitoring and
evaluation framework, the legitimacy of the organization‟s governance structure, conflict of
interest issues, and the appropriate kinds of issues to address given the development stage
of the global program. The proposals for this evaluation should show evidence of a clear
understanding of Sourcebook recommendations, and in particular should demonstrate how
consultants intend to comply with the Sourcebook’s quality and ethical standards.

 QUESTIONS TO BE ADDRESSED
19.    In the following, each evaluation objective will be followed by a set of “high level
questions” and further elaborated by a tentative set of detailed questions, drawing from the
Evaluation Oversight Committee‟s “Issues Paper,” which synthesizes inputs and suggestions
from a wide-range of FTI stakeholders and evaluation experts.14

Objective 1: Assess the Relevance of FTI Objectives and Goals to Country Educational and
Development Needs

High Level Questions 1: Is what FTI aims to accomplish consistent with current needs
and priorities of FTI client countries? What can be done to ensure the relevance of FTI
going forward?

Related questions:
 a. How is FTI perceived and understood by various stakeholders and to what extent have
      varying interpretations of what FTI is and what it can accomplish influenced its
      effectiveness in any way?
 b. To what extent is FTI‟s focus on primary school completion relevant to the current
      needs of partner countries (and potential partner countries) and how has this emphasis
      been treated within the broader context of country EFA programming?
 c. Are FTI‟s intended contributions appropriate, given the partnership‟s objectives? Can
      they be expected to make a difference?
 d. How has the evolution of the FTI contributed to maintaining its ongoing relevance as a
      global partnership supporting EFA?

Objective 2: Assess FTI‟s Contribution to Country Progress in Accelerating Action on EFA
and Related Processes

High Level Questions 2: To what extent is the FTI accomplishing what it was
designed to do, namely, accelerating progress on EFA? Are FTI mechanisms and
processes contributing effectively to this? How can FTI mechanisms and processes
be improved to maximize their contributions going forward?

Related questions:
 a. To what extent has FTI and its mechanisms/processes (e.g. Indicative Framework and
      the endorsement process) helped countries strengthen education policies, sector
      plans, and educational data/statistics?
 b. Are FTI mechanisms and processes evolving appropriately to address the needs of
14
    Early formulations of evaluation questions were compiled by the Evaluation Oversight Committee
in the form of an “Issues Paper,” which was shared with stakeholders in Tokyo (FTI Technical Meeting
April 21-25 2008) and Maputo, Mozambique (ADEA biennial meeting, May 5-9, 2008) as a means of
stimulating and gathering feedback and new inputs. The feedback is reflected in the revised Issues
Paper which is in the TOR Annex.


Page 48 of 134                                                                    printed: 04-Feb-09
                                 Annex A: Terms of Reference


     different client countries (e.g. fragile states)?
  c. Has policy making and planning become more open, transparent, and participatory in
     participating countries?
  d. To what extent has FTI contributed to improved institutional capacity in partner
     countries (including those preparing for partnership); has the capacity building
     provided been sufficiently comprehensive to make a sustainable difference.
  e. To what extent and how has FTI contributed to accelerated progress towards universal
     primary school completion by 2015;
  f. How have FTI programs contributed to intermediate outcomes such as improvements
     in the grade 1 intake and gross enrollment ratios?
  g. Has FTI itself supported other EFA objectives in partner countries and, if so, to what
     effect?
  h. How has the FTI impacted government support for other EFA objectives?


Objective 3: Assess FTI‟s Contribution to Improved Resource Mobilization and Aid
Effectiveness

High Level Questions 3: Has the FTI helped mobilize domestic and international
resources in support of EFA and helped donor agencies to adopt more efficient
development assistance strategies based on Paris Declaration ideals? How can FTI
best help mobilize resources and improve aid effectiveness going forward?

Related questions:
  a. To what extent are FTI partners improving the alignment of their development
      assistance to country-owned plans?
  b. Has FTI helped them to improve interagency cooperation and harmonization?
  c. Has FTI contributed to the reduction in transaction costs, both for partner countries
      and partner agencies?
  d. To what extent have partner countries and agencies been using innovative and
      flexible means of financial support and to what extent do those means of support
      demonstrably improve aid effectiveness?
  e. Has FTI been influential in mobilizing increased funding levels and non-financial
      resources for education among partner countries and agencies?
  f. To what extent has FTI assisted countries in closing their financing gaps for their EFA
      acceleration efforts? Has this been in the context of sustainable fiscal policy?
  g. Has FTI succeeded in mobilizing additional resources in support of EFA? To what
      extent have any increases come from domestic financing? From international
      assistance? In countries where domestic financing in support of EFA has increased
      what are the prospects for sustaining those levels of funding?
  h. Has FTI‟s own governance and management made a difference in how efficiently the
      Initiative works in furthering EFA goals? Namely, can the FTI governance bodies be
      considered legitimate? Are FTI decision-making processes open, transparent,
      inclusive and well founded? Does the Initiative have an adequate monitoring and
      evaluation framework, and is it creating regular, useful inputs to decision making? Are
      FTI processes (endorsement, processing of catalytic funds requests, triggering the
      flow of funds) consistent across countries, timely and efficient (for example, how much
      time does it take between FTI endorsement and the flow of FTI-triggered funding at
      the country level)?




file: FTI_Evaluation_Framework(Jan2009b).doc                                 Page 49 of 134
           Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


METHODOLOGY
General considerations
20.      Details on evaluation methodology will be developed by the bidders in their proposals
(with final details being agreed to during negotiations). It should take cognizance of the data
routinely collected from all FTI endorsed countries on EFA indicators (see Annex D), any
previous evaluations and studies, and progress reports at the global and country level. Key
actors should be identified at the international level (e.g, FTI Steering Committee and
Secretariat members and key headquarters officials of partner agencies and CSOs) and the
country level (government, CSO and agency decision makers), and a procedure established
for selecting, as a balanced set of key informants, comparable groups of actors from each
agency, organization and country. Proposals should include the firm‟s initial understanding
of the program theory15 underpinning FTI (including differences among stakeholders) and a
description of how the firm will refine its view of the program theory in the first weeks of the
evaluation. This view should be reflected in a framework (conceptual and operational) for
this formative evaluation. Subsequently, the consultant should create a more comprehensive
monitoring and evaluation framework, covering both this mid-term evaluation and proposed
future M&E efforts, including the evaluation of FTI impact.

20 [sic].       Proposals should also include the appropriate treatment of gender and other
equity considerations, both in terms of assessing the relevance, effectiveness and efficiency
of FTI, as well as in the data collection and analytical approaches of proposed cross country
and case study methodologies. They should also explain how the firm expects to measure
FTI‟s unique contribution or value added (see objectives 2 and 3), given the fact that FTI has
not been organized with an experimental design and does not lend itself easily to
counterfactual assessment (see “Limitations”).

21.     Case studies are expected to be used in this evaluation as a means of developing
greater insight into country-level processes, accomplishments, and problems, all in the
context of each country, thus making a contribution to the lessons-learned part of the
evaluation. Proposals must indicate that a priority will be placed on consulting equally all the
key actors – government, civil society, and local funding agencies. Unlike desk studies the
case studies can reach down to sub-national levels for informants and site visits to validate
desk review findings and examine more details at the service delivery level. Case studies
will be particularly useful in examining: a) questions about FTI relevance from various points
of view; b) detailed questions about effectiveness, such as policy adjustments and the
inclusiveness of the process; effects of capacity building, EFA outcomes at the school and
community levels, etc), and c) detailed questions about efficiency (such as the functioning of
the local donors group; any real evidence of donor agency changes in aid effectiveness;
resource mobilization, and the sustainability of systems).

22.    Proposals should create and explain a sampling design for case studies representing
a wide range of countries and optimizing possibilities for contrast and comparison.
Proposals should also specify the kinds of informants that should be sampled in each case
and the sampling methods. Proposals should include between 8 and 16 strong case
studies. Bidders will be expected to create their own country selection framework, with the

15
   By “program theory” this TOR means the major assumptions and guiding principles behind FTI
programs and processes, including the pathways along which FTI support has been expected to
contribute to desired outputs and outcomes. This does not necessarily mean laying out a logical
framework for FTI, but at least there should be an elaboration of the assumed results chain, from FTI-
inspired inputs, to processes, to outputs, outcomes and impacts. It should be noted that different
stakeholders have held different assumptions about FTI and its program theory, and these have
changed overtime. This will need to be acknowledged and dealt with by the bidders.

Page 50 of 134                                                                      printed: 04-Feb-09
                                 Annex A: Terms of Reference


provision that they should consider parameters such as:
        Country geographical region (e.g., Africa/Non-Africa)
        Time of entry into FTI (e.g., early v. late)
        FTI endorsed v. countries invited into FTI but not endorsed
        Countries on-track as well as off-track for achieving UPC by 2015
        Size of country
        Etc.

23.    Prospective consultants should indicate which countries they would include on their
case study list, what procedure they used in selecting the countries, how they would select
informants and site visit locations within countries, and how their selections would allow the
evaluation to address the evaluation questions set forth in these terms of reference.

24.     After reviewing the existing FTI approach to monitoring and evaluation and
developing a refined view of program theory (see paragraph 19), the consultant will propose
a framework for this mid-term evaluation, and, ultimately, a comprehensive M&E framework
for FTI. The comprehensive framework should include a proposed design for the evaluation
of FTI impact, including the baseline data that will need to be collected, plus estimates of
cost and level of effort to conduct it. Some criteria for impact evaluation might be changes in
the human resource bases in FTI countries, and, at the global level, the long-term impact of
FTI on donor effectiveness as defined throughout these terms of reference, including the
impact of FTI as a global partnership designed to exemplify the new aid architecture.

Stages of the study
Stage 1: September to December 2008. Focused on gathering global level data addressing
questions of FTI effectiveness and efficiency, relying principally on desk study to paint initial
picture of FTI outputs and outcomes (on resource mobilization, on improved policy and
planning, and improved alignment, harmonization, ownership, on basic EFA indicators etc).
It will also gather stakeholder perceptions of FTI effectiveness and efficiency, including its
governance arrangements. One option during Stage 1 would be the piloting of the case
study methodology in one country. Stage 1 will also review and assess the current
monitoring and evaluation arrangements within FTI.

25.    Within the first 30 days of Stage 1, the consultant will submit to the Evaluation
Oversight Committee for review its evaluation framework for this mid-term evaluation,
incorporating its refined view of FTI program theory, and an indication of how it will go about
creating a comprehensive M&E framework for FTI.

26.     The main product of Stage 1 will be a preliminary report which will be used as part of
the formative process of engaging the FTI Partnership in discussing global findings on FTI
effectiveness and efficiency. It should provide evidence from existing reports, documents
and studies of how the FTI has or has not contributed to the trajectory of indicators in FTI
countries during the past decade, and include a first review of different stakeholder
perceptions of the FTI. It should enable a comparison of trends on key indicators in FTI
endorsed countries before and after endorsement, as well as in comparison to those
endorsed at different times, or to countries not participating in the FTI. The report should
present the findings of an initial review of the governance and management arrangements
and the existing FTI monitoring and evaluation framework and approach. The preliminary
report should also indicate how the consultant expects to continue to gather data during
Stage 2 of the evaluation, including restating or revising the list of case studies that will be
conducted and methodologies for systematic stakeholder sampling. If the case study
methodology will have been piloted in Stage 1, then the preliminary report should be


file: FTI_Evaluation_Framework(Jan2009b).doc                                     Page 51 of 134
          Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


structured to include lessons learned from that pilot and specific statement of how the
consultant will use the lessons from the pilot to improve the conduct of the additional cases
to be undertaken in Stage 2.

27.      Stage 2 – December 2008 through September 2009. Focused on gathering
additional data to flesh out global perspectives on effectiveness and efficiency (with country
examples and carefully constructed lessons learned) and addressing remaining questions of
relevance. Insights from completed case studies will be compiled and analyzed and used
together with Stage 1 preliminary findings for the full report, presenting overall findings,
lessons learned, and recommendations which can be used by the partnership to change
course as needed and improve relevance, effectiveness and efficiency. Stage 2 should also
include the development of a framework for monitoring and evaluating the impact of FTI,
including development of a manageable set of indicators for tracking progress towards
sustainable impact and baseline data from case study countries on those indicators.
Stakeholders should be consulted for input regarding their perspectives on the long-term
impact of their education investments supported through FTI and likely indicators of progress
towards that impact. Stage 2 products include a progress report due in March and
scheduled to coincide with the technical meetings of FTI, as well as the draft full report due
in July and final full report due in September. Stage 2 will also produce up to 4 thematic
briefing papers (e.g., one covering a proposed monitoring and evaluation framework,
including impact study indicators and baseline data), and a condensed version (synopsis) of
the full report.

DELIVERABLES
28. The consultant will be expected to deliver reports and briefings as specified below:
        Attendance at FTI meetings in Paris in September 2008
        Within 30 days of signing the contract, a report establishing the evaluation
        framework (based on a refined FTI program theory) that will guide this formative
        evaluation.
        A Preliminary Report, including an executive summary, in English and French to be
        submitted no later than the end of November 2008.
        Attendance at a December „08 FTI meeting to present/discuss the preliminary report
        and gather stakeholder input/feedback on it.
        Stage 2 progress report to be submitted no later than March „09
        Attendance at Spring ‟09 FTI meeting to present/discuss the progress report
        Individual case studies, including executive summaries of each
        Draft Full Report, including a comprehensive M&E framework no later than July „09.
        Attendance at Fall ‟09 FTI meeting to present/discuss draft Full Report and gather
        stakeholder input/feedback on the report
        Final Full Report, including an executive summary in English and French
        Attendance at November ‟09 High Level Meeting to present/discuss the Full Report
        Up to four short thematic briefing papers targeting policy makers. The themes for
        these papers to be discussed with the EOC and FTI Partnership (at draft full report
        stage)
        Commitment to participate in/lead 3 dissemination events based on the final report
        and/or thematic briefing papers

29.    The consultant will also be required to provide the FTI Partnership the raw data on
which the reports/products are based, with the additional stipulation that it cannot
independently use the data to generate its own reports and publications.




Page 52 of 134                                                                printed: 04-Feb-09
                                 Annex A: Terms of Reference


 LIMITATIONS
30.     It is acknowledged that there are certain limitations which will constrain the power
and generalizability of this evaluation effort. First, time constraints and overall project
design/ history have limited the possibility of employing an experimental or quasi
experimental design, meaning it is not possible to rule out certain biases in the findings. In
addition, the evaluation is faced with difficult questions of attribution: are positive changes
towards EFA goals a result of FTI support or would they have happened anyway? Another
problem is endemic to quickly evolving initiatives: a study may painstakingly ascertain the
contribution of a certain intervention only to find the intervention is evolving. Thus, it is
sometimes difficult to know what to evaluate: original design features or what the
intervention has been transformed into.

31.    Proposals should include a further elaboration of the limitations inherent in executing
these terms of reference and, more importantly, indicate what specific measures the
consultant will take to overcome or mitigate the identified limitations.

CONSULTANT PROFILE AND QUALIFICATIONS
32.     An evaluation group or team (Consultant), selected through competitive proposal
submission, will carry out the evaluation. Given the wide range of themes and the
compressed time schedule for the evaluation, it is anticipated (but not required) that the
Consultant will consist of a consortium of institutions. Also, given the fact that there will be
data collection and case studies in developing countries, it would be desirable that all or
many of the Consultant‟s institutions have experience in conducting evaluations/ studies in
such countries, and include one or more developing country institutions. The Consultant
should include personnel with advanced professional background (at the doctoral level for
senior researchers and master‟s degree level for research assistants) and extensive
experience in: 1) education issues in developing countries, particularly at the basic education
or EFA level; 2) the study and/or evaluation of basic education initiatives and reforms in
such countries; 3) assessment of education outcomes (e.g., primary school access,
completion and learning gains) at the national and sub-national level; 4) conducting financial
analysis of domestic and international spending on basic education (changes in
commitments and modalities, predictability, disbursement, etc.); 5) evaluation of institutional
development and change at the national and global levels; 6) the management and conduct
of case studies in diverse developing country settings (need for staff fluency in at least
French and English); and 7) the use of alternative evaluation methodologies, quantitative
and qualitative, and the capacity to synthesize various strands into a concise report including
actionable recommendations for the future. The team leader should have extensive
experience managing complex multi-country evaluations of development assistance.

33.      The organization of the evaluation will be the responsibility of the Consultant and
should be specified and explained clearly in the proposal. Program objectives, high level
questions and related evaluation questions are to be based on the points made in these
Terms of Reference (including the “Issues Paper” in Annex A), but the Consultant is given
some latitude in refining, prioritizing and combining evaluation questions, and deciding how
to collect and analyze data and present the results. The candidate consultant‟s proposals
will be the technical basis for selection. Before a formal contract is issued a negotiation will
take place between the candidate consultant and the EOC in which specific design features,
work plans, timelines, and deliverables will be agreed upon.




file: FTI_Evaluation_Framework(Jan2009b).doc                                     Page 53 of 134
          Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


ROLES AND RESPONSIBILITIES
34.     The evaluation is being commissioned by the FTI partnership, through its Steering
Committee. The Steering Committee has appointed an Evaluation Oversight Committee
(EOC) which operates independently with the express purposes of a) defining the terms of
reference for the evaluation, b) managing the tendering process and selection of an
evaluation consultant, c) managing the implementation of the terms of reference, and d)
assuring the overall quality and theoretical and methodological rigor and integrity. The
consultant will therefore report to the EOC. All reports, products and deliverables will be
submitted to the EOC for technical review and feedback. The EOC will determine the
acceptability of all deliverables. The consultant will communicate with the EOC regarding
their ongoing work and will report any deviations from the expected timely execution of the
terms of reference or from the negotiated and agreed upon work plan.

35.    The EOC is working through the FTI Secretariat and the World Bank‟s procurement
department to contract for this evaluation. World Bank staff in the FTI Secretariat will
oversee the execution of the contract to ensure compliance with World Bank procurement
and contract procedures and regulations.

QUALITY ASSURANCE AND SUPPORT
36.     As an external evaluation, it is important that the consultant be able to pursue its
work plan and make its conclusions without interference. However, there is a need for
quality assurance, and this will be exercised both by the Evaluation Oversight Committee
and by the consultant itself. The roles of the EOC with regard to quality assurance will
include: a) selecting the consultant based on an objective and unbiased review process, b)
making sure, during contract negotiations with the consultant, that the proposed evaluation
design and work plan are consistent with the TOR, realistic, and consistent with professional
standards for the evaluation of global partnerships (see section on Evaluation Framework
and Quality Standards), c) reviewing progress as indicated in the second quarter progress
report; d) reviewing the preliminary and full reports, assuring that they are methodologically
sound and rigorous, that conclusions are clear and supported by evidence, and that
recommendations follow from the conclusions; and e) assuring that gender issues are
adequately addressed by the evaluation and that females are represented on the evaluation
team. Concerning quality assurance measures to be taken by the consultant, these should
be described in the proposals submitted by the candidate firms, including the treatment of
gender issues. Where the EOC quality assurance measures reveal that quality standards
are not met, it can request that the consultant make changes and revisions. Deliverables will
only be submitted to the FTI Steering Committee after they are certified by the EOC to have
met the above mentioned quality standards.

37.    The final report should be of publishable quality and will be published by the FTI
Partnership.

PROPOSAL SPECIFICATIONS
38.    Proposals must be submitted in English both electronically and in hard copy (10
copies). The narrative portion of the technical proposal must not exceed 35 pages, single
spaced, 12 point font, with margins of 1 inch. Annexes can comprise up to another 35
pages.




Page 54 of 134                                                                printed: 04-Feb-09
                                        Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative



Annex B: The GRPP Evaluation Sourcebook
The first two columns of this matrix are drawn from the Sourcebook for Evaluating Global and Regional Partnership Programs (the second
column is condensed). The third column shows how the evaluation team will address these issues in the FTI Mid-Term Evaluation. This annex
was included in the proposal but has been updated following work done at the team workshop in November 2008. The GRPP sourcebook is
highly relevant to an evaluation of this complexity and breadth.

GRPP Feature                                General Implications                       Implications for this FTI mid-evaluation
GRPPs are programmatic partnerships         Identifying various categories of          An initial mapping of nature and type of involvement of stakeholders,
involving multiple donors, partners, and    stakeholders early to ensure appropriate   which reflects progression/changes over time at the different levels of
other stakeholders who have varying         degree of consultation and participation   intervention, has been developed for all evaluation levels (global,
interests. Decision making and              during the evaluation.                     regional, country). Country study teams will draw on the general
accountability is joint at the governance                                              stakeholder map when preparing a systematic list of organisations
level                                                                                  and individuals for the country study team to contact. (See Annex I in
                                                                                       this volume.)
                                            Assessing continued relevance of the       This assessment requires a good understanding of adaptations of the
                                            program to main stakeholders on both       FTI over time, as well as possibly changing demands by different
                                            the supply and demand side.                stakeholders. Careful timelines of key environmental and policy
                                                                                       changes/conventions will be developed. The current draft of the
                                                                                       global timeline is included as Annex J. A country-specific time line
                                                                                       will be prepared in the preliminary stages of each country case study.
                                            Assessing legitimacy and effectiveness     For each of the case study countries a map of the FTI approval
                                            of governance and management               process will be developed from initial policy work throughout
                                            arrangements, including communication      evaluation period, and issues of legitimacy and relevance arising in
                                            to various stakeholders.                   different stages identified through documentary analysis and
                                                                                       interviews. In addition, FTI's global governance arrangements will be
                                                                                       carefully reviewed as stipulated in the TOR. Mapping of the FTI
                                                                                       approval process will take place for each country study before the
                                                                                       field work is started, so as to allow well-focused discussion of
                                                                                       governance and management issues.
                                                                                       The FTI's global governance arrangements will be carefully reviewed
                                                                                       as an input to the Preliminary Report.




file: FTI_Evaluation_Framework(Jan2009b).doc                                                                                                 Page 55 of 134
                                        Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative

GRPP Feature                                General Implications                       Implications for this FTI mid-evaluation
                                            In assessing governance and                We will examine how FTI modalities have been shaped by donor
                                            management take account of fiduciary       fiduciary and accountability concerns, and assess the
                                            standards of donors and trustees as well   (positive/negative) influences that result. In doing so we will draw on
                                            as financial management, reporting,        very recent work by Mokoro on good practices in the use of country
                                            compliance with donor requirements.        budget systems (for SPA and CABRI, see Mokoro 2008a and 2008b),
                                                                                       and on donor approaches to risk in the use of country systems (for
                                                                                       the OECD DAC Joint Venture on PFM, see Mokoro and CIPFA
                                                                                       2008). The financial workstream will be responsible for leading in this
                                                                                       area.
GRPPs are global or regional in scope       Soon after the launch of the programme,    FTI M&E arrangements will be carefully and critically reviewed and
working in different socio-political        management needs to establish a            an assessment made of the adequacy of the arrangements vis-à-vis
contexts and at multiple levels             results-based M&E framework.               the purpose and scope of the programme. Selected comparisons with
                                                                                       M&E arrangements for other similar GRPPs will inform this analysis
                                                                                       and will also feed into the identification of suggested areas for
                                                                                       strengthening. A scoping paper on the options for impact evaluation
                                                                                       and their implications for M&E (see TOR at Annex G) will feed into
                                                                                       the Preliminary Report, and will enable ideas to be tested with
                                                                                       stakeholders long before the launch of the final report.
                                            In GRPP evaluations validity will be a     For FTI a particular challenge is adequate sampling through (and
                                            function of decisions on evaluation        within) country studies. Statistically representative samples are not
                                            scope, coverage, and sampling to           feasible, and inter-country comparisons will need to be made at the
                                            ensure adequate representativeness and     level of subcomponents in the results chain. The selection of country
                                            thus validity of findings.                 case studies is designed to cover a range of contexts and FTI
                                                                                       experiences (see Annex H). Fieldwork for two pilot country studies in
                                                                                       Kenya and Cambodia is due to take place in January 2009.
                                                                                       Methodological lessons from the pilots will be discussed in depth at
                                                                                       the second team workshop in February and will inform the refinement
                                                                                       of the country study approaches in Stage 2.
                                            Given the scope and levels, GRPP           Regarding time frame, the overall duration of the study, and the
                                            evaluations require a longer time frame    periods allowed for feedback on initial findings appear adequate.
                                            and larger budget to achieve sufficient    Despite the slippage on contract agreement at the start of the
                                            level of data collection and stakeholder   programme adequate time has been built in, and deadlines have
                                            participation.                             been moved back to reflect this. Stakeholder participation will be
                                                                                       encouraged during and after the country visits through the use of
                                                                                       internet based discussion boards.




Page 56 of 134                                                                                                                              printed: 04-Feb-09
                                                         Annex B: The GRPP Evaluation Sourcebook

GRPP Feature                                  General Implications                             Implications for this FTI mid-evaluation
Most GRPPs are housed in international        GRPPs should have an evaluation policy           The final report will include recommendations on future M&E
organizations with separate governing         that guarantees independence and is              approaches.
bodies but with managers who are              agreed upon by the governing body.
employees of the organization                 Evaluations should be commissioned by
                                              and report to a governing body (not              Taken into account in evaluation terms of reference, and role of
                                              management) and be conducted through             Evaluation Oversight Committee.
                                              competitive bidding.

The results of GRPPs are the joint            Assessing effectiveness of a GRPP                The Logical Framework set out in Annex E will provide a central
product of global/regional and country        requires consideration of the programme          reference point for workstream leaders and country study teams.
level activities and of parallel activities   results chain at all levels, ideally with        Consideration has also been given to the sources of evidence for
financed by other development agencies        measurable indicators and a                      each level of the results chain.
                                              representative sample of activities at all
                                              levels.
                                              Attribution is often particularly difficult to   As described in methodology section of this report, we will address
                                              discern in the case of a GRPP.                   this by the careful application of contribution analysis in all
                                                                                               components of the evaluation.
The programme usually evolves over            The purpose, objectives, scope and               Evaluation approach will ensure that documentary resources,
time, based on availability of financing      design of an evaluation need to take into        interviews and country analysis cover the full evaluation period and
and usually does not have a fixed end-        account the maturity of the programme.           the full range of developments in a representative manner. A detailed
point.                                        Evaluation are usually mid term and              inventory will be done of prior thematic assessments and
                                              often planned to build on each other             'evaluations' (such as the work done by various task forces over time)
                                              sequentially.                                    and careful consideration given to how the recommendations from
                                                                                               these products are reflected in the evolution of FTI.
                                                                                               We will also (a) take note that the FTI is of variable maturity at
                                                                                               country level and (b) explore the "pre-natal" stages at both global and
                                                                                               country level.
                                              GRPP evaluations should include an               For FTI – given its sponsors' declared aim to ensure that not-viable
                                              assessment of sources and uses of                country plans for UPC are frustrated by lack of funds – this element is
                                              funds and the resource mobilisation              fundamental. This is reflected in the prominence we will give to
                                              strategy.                                        financing workstream, and to issues of fungibility and predictability of
                                                                                               financing.




file: FTI_Evaluation_Framework(Jan2009b).doc                                                                                                          Page 57 of 134
                                        Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative

GRPP Feature                                General Implications                       Implications for this FTI mid-evaluation
                                            In a mature programme, it may be           In the FTI's case, it is clear that requirements for international support
                                            necessary to assess strategies for         to primary education will not end in 2015, and our considerations of
                                            devolution, exit, or alternative           sustainability will reflect this.
                                            organizational and financing
                                            arrangements.
Governance and management are multi-        Assessing governance should include        Principles of an organizational review will be applied in this evaluation
layered and decision making complex.        review of respective roles of the          including a careful comparison between the formal set up and its
Continuity may be an issue because of       governing body and management in           evolution over time and the manner in which this has been
rotating membership.                        various decision making processes.         implemented in practice. Global governance will be an important topic
                                                                                       for the Preliminary Report.
                                            Evaluators need to ascertain any           The TOR draw attention to a number of such changes which require
                                            changes in membership criteria for the     assessment, again taking account of both formal and informal
                                            governing body or changes in actual        aspects. Regular contact with the FTI Secretariat will enable the
                                            representation.                            evaluation team to remain abreast of changes, which are continuing.
                                            Feedback processes and dissemination       The TOR already take some account of ultimate dissemination
                                            plans for evaluation products need to be   requirements. Workshops will take place at key stages,
                                            defined before the evaluation to include   complemented by a study website which can serve as a platform for
                                            all relevant stakeholders.                 a communications strategy. At country level, we will facilitate
                                                                                       feedback by circulating draft country aide memoires with key findings
                                                                                       for comment at the conclusion of in-country work. There will also be
                                                                                       country level websites which will be connected to the main evaluation
                                                                                       site.
The decision on activities to support are   Criteria and processes for allocating      As noted in the TOR, this is an area where FTI has evolved, and the
made through programmatic process           resources and choosing activities are      evaluation will assess this evolution as well as the relevance of
rather than fixed in advance as in a        important ingredients of relevance and     current arrangements and criteria. The evaluation will dynamically
discrete project.                           effectiveness.                             capture criteria and processes for decision making at country and
                                                                                       global level by reviewing key documentation, interviewing partners
                                                                                       who were directly involved and through discussions with other key
                                                                                       informants and stakeholders.
                                                                                       The interaction between donor allocation processes and country
                                                                                       budgeting processes is very important and will receive special
                                                                                       attention.




Page 58 of 134                                                                                                                                printed: 04-Feb-09
                                                      Annex B: The GRPP Evaluation Sourcebook

GRPP Feature                               General Implications                         Implications for this FTI mid-evaluation
GRPPs are typically externally financed    Evaluations need to take account of          Again, the financial analysis is crucial for FTI. It is necessary to
with little capacity to earn income from   sources and uses of funds and the            investigate whether there has been additionality/catalysis by FTI
their own resources. Total financing       relationship of the resource mobilisation    (donor-wise as well as country-wise), and to review the overall
depends on individual donors' funding      strategy to scale, effectiveness and         coherence of separate decision processes regarding funding.
decisions.                                 efficiency of the programme.
                                           Evaluations need to take account of the      The programme theory and contribution analysis approach allow for
                                           fact that causality may flow in both         such two-way effects to be investigated. We have highlighted the
                                           directions - i.e. strategy for resource      importance of feedback loops in the-programme logic.
                                           mobilisation influences effectiveness and    In looking at FTI's monitoring and reporting, we will bear in mind the
                                           efficiency, and achievement of results       possibility of biases introduced by the exigencies of resource
                                           impacts on resource mobilisation.            mobilisation.
                                           In a mature programme alternative            For FTI it is necessary to consider both its direct (CF) and indirect
                                           financing arrangements may need to be        influences on funding in relation to alternative domestic and
                                           assessed.                                    international sources of funds for primary education. Looking ahead,
                                                                                        the continuing relevance of FTI beyond 2015 needs to be considered.
GRPPs take several years to set up due     Analysis of costs and benefits should        The evaluation design includes review of documentation (where
to the need to reach consensus and         ideally factor in start up costs that were   existent) and detailed interviews with stakeholders involved in the set
establish the legal framework and          incurred prior to legal establishment as     up of the FTI programme. The evaluation period will therefore extend
governance arrangements. Sunk costs        well as costs by the convening partners.     back to before 2002 to include the key activities, events and costs
are relatively high at initial stages.                                                  related to the start up of the programme.
                                           Evaluations should assess                    As well as reviewing the direct administrative costs of the FTI, both
                                           administrative costs relative to activity    for its overall governance and for the Secretariat, we will carefully
                                           costs and note any actual or expected        assess its influence on overall transaction costs. (In this, as in many
                                           economies of scale.                          other aspects, there may be contrasting experiences in different
                                                                                        countries from which useful lessons may be drawn.)
GRPPs are diverse in size, age, sectoral   Standards for evaluation of GRPPs are        Taken into account in evaluation Terms of Reference, this evaluation
focus and objectives and in the type of    necessary to ensure credibility.             framework also explicitly addresses both the GRPP and the DAC
activities supported (knowledge,                                                        evaluation standards (see Annex C).
technical assistance, investments)         Evaluation design, scope, coverage and       This evaluation framework endeavours to reflect the EOC purposes
                                           methodology may differ according to the      as set out in the TOR. However it is also careful to focus the
                                           governing body's purpose in conducting       evaluation in a realistic way (avoiding the temptation of offering more
                                           the evaluation at a given point in time.     than is realistic, and of spreading resources too thinly across too wide
                                                                                        a set of objectives in our approach to country studies).




file: FTI_Evaluation_Framework(Jan2009b).doc                                                                                                    Page 59 of 134
                 Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative




Page 60 of 134                                                                            printed: 04-Feb-09
           Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative



Annex C: Quality Assurance Requirements
This annex reproduces the quality assurance specifications from the TOR, together with the
applicable OECD DAC evaluation quality standards.


Quality Specifications of the TOR
Evaluation Framework and Quality Standards (TOR ¶17-18)
This evaluation will be expected to apply three of the five DAC evaluation criteria for
evaluating development assistance, relevance, effectiveness, and efficiency. Because it is
primarily a formative evaluation it will not be expected to fully apply the other two, impact and
sustainability. However, as mentioned above, it will be expected to provide guidance for the
assessment of the impact of the FTI at a later stage and to gather baseline data for such an
assessment at both the global and country levels (e.g. in case study countries). Also, even
though it is too early to assess the sustainability of FTI beyond its implicit closing time (at
EFA culmination in 2015), issues of sustainability at the country level should be addressed in
this evaluation, for example, in examining the sustainability and predictability of domestic
funding for primary or basic education and the effects of capacity building efforts on long
term institutional capacity.

As an evaluation of a global or regional partnership program (GRPP) this evaluation should
also be oriented by the GRPP evaluation Sourcebook [IEG 2007]. This Sourcebook sets
quality standards for global program evaluations, including, for example, the need for an
examination and critique of the global program's existing monitoring and evaluation
framework, the legitimacy of the organization‟s governance structure, conflict of interest
issues, and the appropriate kinds of issues to address given the development stage of the
global program. The proposals for this evaluation should show evidence of a clear
understanding of Sourcebook recommendations, and in particular should demonstrate how
consultants intend to comply with the Sourcebook’s quality and ethical standards.


Quality assurance and support (TOR ¶36)
As an external evaluation, it is important that the consultant be able to pursue its work plan
and make its conclusions without interference. However, there is a need for quality
assurance, and this will be exercised both by the Evaluation Oversight Committee and by
the consultant itself. The roles of the EOC with regard to quality assurance will include:
a) selecting the consultant based on an objective and unbiased review process, b) making
sure, during contract negotiations with the consultant, that the proposed evaluation design
and work plan are consistent with the TOR, realistic, and consistent with professional
standards for the evaluation of global partnerships (see section on Evaluation Framework
and Quality Standards), c) reviewing progress as indicated in the second quarter progress
report; d) reviewing the preliminary and full reports, assuring that they are methodologically
sound and rigorous, that conclusions are clear and supported by evidence, and that
recommendations follow from the conclusions; and e) assuring that gender issues are
adequately addressed by the evaluation and that females are represented on the evaluation
team. Concerning quality assurance measures to be taken by the consultant, these should
be described in the proposals submitted by the candidate firms, including the treatment of
gender issues. Where the EOC quality assurance measures reveal that quality standards
are not met, it can request that the consultant make changes and revisions. Deliverables will
only be submitted to the FTI Steering Committee after they are certified by the EOC to have
met the above mentioned quality standards.



file: FTI_Evaluation_Framework(Jan2009b).doc                                     Page 61 of 134
          Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


DAC Evaluation Standards
The ethical requirements of the DAC evaluation standards are incorporated in the GRPP
principles cited in the Evaluation TOR (quoted above). However, all team members should
be familiar with the full DAC Evaluation Quality Standards (OECD DAC 2006a, available at
www.oecd.org/dataoecd/30/62/36596604.pdf). These are also quoted in part below.

1. Rationale, purpose and objectives of an evaluation
       1.1 The rationale of the evaluation
       Describes why and for whom the evaluation is undertaken and why it is undertaken
       at a particular point in time.
       1.2 The purpose of the evaluation
       The evaluation purpose is in line with the learning and accountability function of
       evaluations. For example the evaluation‟s purpose may be to:
                  - Contribute to improving an aid policy, procedure or technique
                  - Consider a continuation or discontinuation of a project/programme
                  - Account for aid expenditures to stakeholders and tax payers
       1.3 The objectives of the evaluation
       The objectives of the evaluation, specify what the evaluation aims to achieve.
       For example:
                  - To ascertain results (output, outcome, impact) and assess the
                     effectiveness, efficiency and relevance of a specific development
                     intervention;
                  - To provide findings, conclusions and recommendations with respect to
                     a specific policy, programme etc.

2. Evaluation scope
       2.1 Scope of the evaluation
       The scope of the evaluation is clearly defined by specifying the issues covered, funds
       actually spent, the time period, types of interventions, geographical coverage, target
       groups, as well as other elements of the development intervention addressed in the
       evaluation.
       2.2 Intervention logic and findings
       The evaluation report briefly describes and assesses the intervention logic and
       distinguishes between findings at the different levels: inputs, activities, outcomes and
       impacts. The report also provides a brief overall assessment of the intervention logic.
       2.3 Evaluation criteria
       The evaluation report applies the five DAC criteria for evaluating development
       assistance: relevance, efficiency, effectiveness, impact and sustainability. The criteria
       applied for the given evaluation are defined in unambiguous terms. If a particular
       criterion is not applied this is explained in the evaluation report, as are any additional
       criteria applied.
       2.4 Evaluation questions
       The questions asked, as well as any revisions to the original questions, are
       documented in the report for readers to be able to assess whether the evaluation
       team has sufficiently assessed them.

3. Context
       3.1 The development and policy context
       The evaluation report provides a description of the policy context relevant to the
       development intervention, the development agency‟s and partners‟ policy documents,
       objectives and strategies. The development context may refer to: regional and
       national economy and levels of development. The policy context may refer to:
       Poverty reduction strategies, gender equality, environmental protection and human
       rights.

Page 62 of 134                                                                  printed: 04-Feb-09
                          Annex C: Quality Assurance Requirements


       3.2 The institutional context
       The evaluation report provides a description of the institutional environment and
       stakeholder involvement relevant to the development intervention, so that their
       influence can be identified and assessed.
       3.3 The socio-political context
       The evaluation report describes the socio-political context within which the
       intervention takes place, and its influence on the outcome and impact of the
       development intervention.
       3.4 Implementation arrangements
       The evaluation report describes the organisational arrangements established for
       implementation of the development intervention, including the roles of donors and
       partners.

4. Evaluation methodology
       4.1 Explanation of the methodology used
       The evaluation report describes and explains the evaluation method and process and
       discusses validity and reliability. It acknowledges any constraints encountered and
       their impact on the evaluation, including their impact on the independence of the
       evaluation. It details the methods and techniques used for data and information
       collection and processing. The choices are justified and limitations and shortcomings
       are explained.
       4.2 Assessment of results
       Methods for assessment of results are specified. Attribution and
       contributing/confounding
       factors should be addressed. If indicators are used as a basis for results assessment
       these should be SMART (specific, measurable, attainable, relevant and time bound).
       4.3 Relevant stakeholders consulted
       Relevant stakeholders are involved in the evaluation process to identify issues and
       provide input for the evaluation. Both donors and partners are consulted. The
       evaluation report indicates the stakeholders consulted the criteria for their selection
       and describes stakeholders‟ participation. If less than the full range of stakeholders
       was consulted, the methods and reasons for selection of particular stakeholders are
       described.
       4.4 Sampling
       The evaluation report explains the selection of any sample. Limitations regarding the
       representativeness of the evaluation sample are identified.
       4.5 Evaluation team
       The composition of evaluation teams should posses a mix of evaluative skills and
       thematic knowledge, be gender balanced, and include professionals from the
       countries or regions concerned.

5. Information sources
        5.1 Transparency of information sources
        The evaluation report describes the sources of information used (documentation,
        respondents, literature etc.) in sufficient detail, so that the adequacy of the
        information can be assessed. Complete lists of interviewees and documents
        consulted are included, to the extent that this does not conflict with the privacy and
        confidentiality of participants.
        5.2 Reliability and accuracy of information sources
        The evaluation cross-validates and critically assesses the information sources used
        and the validity of the data using a variety of methods and sources of information.




file: FTI_Evaluation_Framework(Jan2009b).doc                                    Page 63 of 134
          Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


6. Independence
       6.1 Independence of evaluators vis-à-vis stakeholders
       The evaluation report indicates the degree of independence of the evaluators from
       the policy, operations and management function of the commissioning agent,
       implementers and beneficiaries. Possible conflicts of interest are addressed openly
       and honestly.
       6.2 Free and open evaluation process
       The evaluation team is able to work freely and without interference. It is assured of
       cooperation and access to all relevant information. The evaluation report indicates
       any obstruction which may have impacted on the process of evaluation.

7. Evaluation ethics
       7.1 Evaluation conducted in a professional and ethical manner
       The evaluation process shows sensitivity to gender, beliefs, manners and customs of
       all stakeholders and is undertaken with integrity and honesty. The rights and welfare
       of participants in the evaluation are protected. Anonymity and confidentiality of
       individual informants should be protected when requested and/or as required by law.
       7.2 Acknowledgement of disagreements within the evaluation team
       Evaluation team members should have the opportunity to dissociate themselves from
       particular judgements and recommendations. Any unresolved differences of opinion
       within the team should be acknowledged in the report.

8. Quality assurance
       8.1 Incorporation of stakeholders’ comments
       Stakeholders are given the opportunity to comment on findings, conclusions,
       recommendations and lessons learned. The evaluation report reflects these
       comments and acknowledges any substantive disagreements. In disputes about
       facts that can be verified, the evaluators should investigate and change the draft
       where necessary. In the case of opinion or interpretation, stakeholders‟ comments
       should be reproduced verbatim, such as in an annex, to the extent that this does not
       conflict with the rights and welfare of participants.
       8.2 Quality control
       Quality control is exercised throughout the evaluation process. Depending on the
       evaluation‟s scope and complexity, quality control is carried out either internally or
       through an external body, peer review, or reference group. Quality controls adhere to
       the principle of independence of the evaluator.

9. Relevance of the evaluation results
       9.1 Formulation of evaluation findings
       The evaluation findings are relevant to the object being evaluated and the purpose of
       the evaluation. The results should follow clearly from the evaluation questions and
       analysis of data, showing a clear line of evidence to support the conclusions. Any
       discrepancies between the planned and actual implementation of the object being
       evaluated are explained.
       9.2 Evaluation implemented within the allotted time and budget
       The evaluation is conducted and results are made available in a timely manner in
       relation to the purpose of the evaluation. Un-envisaged changes to timeframe and
       budget are explained in the report. Any discrepancies between the planned and
       actual implementation and products of the evaluation are explained.
       9.3 Recommendations and lessons learned
       Recommendations and lessons learned are relevant, targeted to the intended users
       and actionable within the responsibilities of the users. Recommendations are
       actionable proposals and lessons learned are generalizations of conclusions
       applicable for wider use.


Page 64 of 134                                                                printed: 04-Feb-09
                          Annex C: Quality Assurance Requirements


       9.4 Use of evaluation
       Evaluation requires an explicit acknowledgement and response from management
       regarding intended follow-up to the evaluation results. Management will ensure the
       systematic dissemination, storage and management of the output from the evaluation
       to ensure easy accessibility and to maximise the benefits of the evaluation‟s findings.

10. Completeness
      10.1 Evaluation questions answered by conclusions
      The evaluation report answers all the questions and information needs detailed in the
      scope of the evaluation. Where this is not possible, reasons and explanations are
      provided.
      10.2 Clarity of analysis
      The analysis is structured with a logical flow. Data and information are presented,
      analysed and interpreted systematically. Findings and conclusions are clearly
      identified and flow logically from the analysis of the data and information. Underlying
      assumptions are made explicit and taken into account.
      10.3 Distinction between conclusions, recommendations and lessons learned
      Evaluation reports must distinguish clearly between findings, conclusions and
      recommendations. The evaluation presents conclusions, recommendations and
      lessons learned separately and with a clear logical distinction between them.
      Conclusions are substantiated by findings and analysis. Recommendations and
      lessons learned follow logically from the conclusions.
      10.4 Clarity and representativeness of the summary
      The evaluation report contains an executive summary. The summary provides an
      overview of the report, highlighting the main conclusions, recommendations and
      lessons learned.”

(Quoted from OECD (2006) DAC Evaluation Quality Standards pp.4-8.)




file: FTI_Evaluation_Framework(Jan2009b).doc                                  Page 65 of 134
         Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative




Page 66 of 134                                                           printed: 04-Feb-09
                                      Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative



Annex D: Addressing the TOR's High Level Questions
The following table shows how the questions posed in the TOR will be addressed by the different workstreams, and the different outputs, of the
evaluation.
                                                                               Which of the standard DAC
                                                 Which workstream will be
                                                                               criteria are most relevant in Which reports will answer
Terms of Reference Question                     contributing most towards
                                                                                assessing the answer to               this question?
                                                 answering the question?
                                                                                       this question?
High Level Questions 1:                                                                                          Country Studies
Is what FTI aims to accomplish consistent with         Education Policy               Relevance                  Final Report (initial discussion in
current needs and priorities of FTI client                                                                       Preliminary Report)
countries?

What can be done to ensure the relevance of FTI                                       Relevance
                                                       All                                                       Final Report
going forward?                                                                        Sustainability

  a. How is FTI perceived and understood by
     various stakeholders and to what extent have                                                                (Preliminary Report)
                                                       Education Policy               Relevance
     varying interpretations of what FTI is and what                                                             Country Studies
                                                       Aid Effectiveness/Governance   Effectiveness
     it can accomplish influenced its effectiveness                                                              Final Report
     in any way?
  b. To what extent is FTI‟s focus on primary
     school completion relevant to the current
                                                       Education Policy                                          Country Studies
     needs of partner countries (and potential                                        Relevance
                                                       Finance                                                   Final Report (initial discussion in
     partner countries) and how has this emphasis                                     Sustainability
                                                       Data and M&E                                              Preliminary Report)
     been treated within the broader context of
     country EFA programming?
  c. Are FTI‟s intended contributions appropriate,
                                                                                      Relevance                  (Preliminary Report)
     given the partnership‟s objectives? Can they      All workstreams
                                                                                      Effectiveness              Final Report
     be expected to make a difference?
  d. How has the evolution of the FTI contributed
                                                                                                                 (Preliminary Report)
     to maintaining its ongoing relevance as a         Aid Effectiveness/Governance   Relevance
                                                                                                                 Final Report
     global partnership supporting EFA?




file: FTI_Evaluation_Framework(Jan2009b).doc                                                                                     Page 67 of 134
                                        Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


                                                                                         Which of the standard DAC
                                                         Which workstream will be
                                                                                         criteria are most relevant in   Which reports will answer
Terms of Reference Question                              contributing most towards
                                                                                          assessing the answer to             this question?
                                                          answering the question?
                                                                                                 this question?
High Level Questions 2:                                                                                                  (Preliminary Report)
To what extent is the FTI accomplishing what it          Education Policy                Effectiveness
                                                                                                                         Country Studies
was designed to do, namely, accelerating                 Finance                         Efficiency
                                                                                                                         Final Report
progress on EFA?                                         Capacity Development

Are FTI mechanisms and processes contributing                                            Effectiveness                   Country Studies
                                                         Aid Effectiveness/Governance
effectively to this?                                                                     Efficiency                      Final Report

How can FTI mechanisms and processes be                                                  Effectiveness
                                                         All workstreams
improved to maximize their contributions going                                           Efficiency,                     Final Report
                                                         Aid Effectiveness/Governance
forward?                                                                                 Sustainability
  a. To what extent has FTI and its
      mechanisms/processes (e.g. Indicative
                                                         Education Policy
      Framework and the endorsement process)                                             Effectiveness
                                                         Finance                                                         Country Studies
      helped countries strengthen education                                              Efficiency,
                                                         Capacity Development                                            Final Report
      policies, sector plans, and educational                                            Sustainability
                                                         Data and M&E
      data/statistics?

  b.   Are FTI mechanisms and processes evolving
                                                         Aid Effectiveness/Governance,
       appropriately to address the needs of different                                   Relevance                       (Preliminary Report)
                                                         Education Policy
       client countries (e.g. fragile states)?                                           Effectiveness                   Final Report
                                                         Capacity Development
  c.   Has policy making and planning become more
                                                                                         Relevance
       open, transparent, and participatory in           Education Policy                                                Country Studies
                                                                                         Efficiency
       participating countries?                          Aid Effectiveness/Governance                                    Final Report
                                                                                         Effectiveness
  d.   To what extent has FTI contributed to
       improved institutional capacity in partner
                                                                                         Relevance                       (Preliminary Report)
       countries (including those preparing for
                                                         Capacity Development            Effectiveness                   Country Studies
       partnership); has the capacity building
                                                                                         Sustainability                  Final Report
       provided been sufficiently comprehensive to
       make a sustainable difference?




Page 68 of 134                                                                                                                          printed: 04-Feb-09
                                                           Annex D: Addressing High Level Questions


                                                                                        Which of the standard DAC
                                                           Which workstream will be
                                                                                        criteria are most relevant in   Which reports will answer
Terms of Reference Question                                contributing most towards
                                                                                         assessing the answer to             this question?
                                                            answering the question?
                                                                                                this question?
  e.    To what extent and how has FTI contributed to
                                                           Data and M&E
        accelerated progress towards universal                                          Effectiveness                   Final Report
                                                           Finance
        primary school completion by 2015?
  f.    How have FTI programs contributed to
        intermediate outcomes such as improvements         Education Policy             Effectiveness                   Country Studies
        in the grade 1 intake and gross enrolment          Data and M&E                 Efficiency                      Final Report
        ratios?
  g.    Has FTI itself supported other EFA objectives      Education Policy                                             Country Studies
                                                                                        Effectiveness
        in partner countries and, if so, to what effect?   Data and M&E                                                 Final Report
                                                                                        Relevance
  h.    How has the FTI impacted government                Education Policy                                             Country Studies
                                                                                        Efficiency
        support for other EFA objectives?                  Finance                                                      Final Report
                                                                                        Sustainability
High Level Questions 3:
Has the FTI helped mobilize domestic and
international resources in support of EFA and                                           Relevance                       (Preliminary Report)
helped donor agencies to adopt more efficient              Aid Effectiveness, Finance   Effectiveness                   Final Report
development assistance strategies based on
Paris Declaration ideals?
                                                                                        Relevance
How can FTI best help mobilize resources and               Finance
                                                                                        Efficiency                      Final Report
improve aid effectiveness going forward?                   Aid Effectiveness
                                                                                        Sustainability
   a.   To what extent are FTI partners improving the                                   Relevance                       (Preliminary Report)
                                                           Aid Effectiveness
        alignment of their development assistance to                                    Efficiency                      Country Studies
                                                           Finance
        country-owned plans?                                                            Effectiveness                   Final Report
                                                                                        Relevance
   b.   Has FTI helped them to improve interagency                                                                      Country Studies
                                                           Aid Effectiveness            Efficiency
        cooperation and harmonization?                                                                                  Preliminary Report
                                                                                        Effectiveness
   c.   Has FTI contributed to the reduction in
                                                                                                                        Country Studies
        transaction costs, both for partner countries      Aid Effectiveness            Efficiency
                                                                                                                        Final Report
        and partner agencies?




file: FTI_Evaluation_Framework(Jan2009b).doc                                                                                           Page 69 of 134
                                        Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


                                                                                       Which of the standard DAC
                                                        Which workstream will be
                                                                                       criteria are most relevant in   Which reports will answer
Terms of Reference Question                             contributing most towards
                                                                                        assessing the answer to             this question?
                                                         answering the question?
                                                                                               this question?
  d.   To what extent have partner countries and
       agencies been using innovative and flexible                                                                     (Preliminary Report)
                                                        Finance                        Efficiency
       means of financial support and to what extent                                                                   Country Studies
                                                        Aid Effectiveness              Effectiveness
       do those means of support demonstrably                                                                          Final Report
       improve aid effectiveness?
  e.   Has FTI been influential in mobilizing
                                                                                                                       (Preliminary Report)
       increased funding levels and non-financial       Finance
                                                                                       Effectiveness                   Country Studies
       resources for education among partner            Data and M&E
                                                                                                                       Final Report
       countries and agencies?
  f.   To what extent has FTI assisted countries in
                                                                                                                       (Preliminary Report)
       closing their financing gaps for their EFA                                      Effectiveness
                                                        Finance                                                        Country Studies
       acceleration efforts? Has this been in the                                      Sustainability
                                                                                                                       Final Report
       context of sustainable fiscal policy?
  g.   Has FTI succeeded in mobilizing additional
                                                                                                                       (Preliminary Report)
       resources in support of EFA? To what extent      Finance
                                                                                       Effectiveness, Efficiency       Country Studies
       have any increases come from domestic            Data and M&E
                                                                                                                       Final Report
       financing? From international assistance?
       In countries where domestic financing in
       support of EFA has increased what are the                                                                       Country Studies
                                                        Finance                        Sustainability
       prospects for sustaining those levels of                                                                        Final Report
       funding?
  h.   Has FTI‟s own governance and management
       made a difference in how efficiently the
                                                        Finance
       Initiative works in furthering EFA goals?                                                                       (Preliminary Report)
                                                        Capacity Development           Relevance
       Namely, can the FTI governance bodies be                                                                        Country Studies
                                                        Aid Effectiveness/Governance   Efficiency
       considered legitimate? Are FTI decision-                                                                        Final Report
       making processes open, transparent,
       inclusive and well founded?
       Does the Initiative have an adequate
                                                        Data and M&E                   Efficiency
       monitoring and evaluation framework, and is                                                                     (Preliminary Report)
                                                        Capacity development           Sustainability
       it creating regular, useful inputs to decision                                                                  Final Report
                                                        Aid Effectiveness/Governance   (Impact)
       making?



Page 70 of 134                                                                                                                       printed: 04-Feb-09
                                                         Annex D: Addressing High Level Questions


                                                                                      Which of the standard DAC
                                                         Which workstream will be
                                                                                      criteria are most relevant in   Which reports will answer
Terms of Reference Question                              contributing most towards
                                                                                       assessing the answer to             this question?
                                                          answering the question?
                                                                                              this question?
      Are FTI processes (endorsement, processing
      of catalytic funds requests, triggering the flow
                                                         Education Policy
      of funds) consistent across countries, timely                                   Relevance
                                                         Finance                                                      (Preliminary Report)
      and efficient (for example, how much time                                       Efficiency
                                                         Capacity Development                                         Final Report
      does it take between FTI endorsement and
                                                         Aid Effectiveness
      the flow of FTI-triggered funding at the
      country level)?




file: FTI_Evaluation_Framework(Jan2009b).doc                                                                                         Page 71 of 134
                 Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative




Page 72 of 134                                                                            printed: 04-Feb-09
           Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative



Annex E:
Extended Logical Framework for the Evaluation of the FTI
1.     This Annex is the detailed version of the Logical Framework for the evaluation.

2.      The matrix follows a standard structure, in that It proceeds through successive levels
from inputs to impact. Note, however, that there are feedback loops between levels. FTI is
a continuing programme and successive phases of activities are influenced by previous
ones. Thus successive levels of the framework are a logical sequence, but the chronological
sequence may involve successive rounds of inputs etc, and the design of the programme
may change over time.

3.      The overall time period that is relevant to the evaluation includes the formative years
of FTI, through is its formal establishment and (as nearly as possible) up to the present. For
particular countries, the dates of first FTI inputs will vary, and it will be necessary to record
carefully what form they took, as FTI modalities have themselves been evolving. "Level
Zero" is included to ensure that evaluators gather full information on the context and
situation prior to FTI inputs. This is particularly important in addressing the first set of High
Level questions (see Box 3D) concerning the relevance of FTI, and of its focus on UPC, in
the context of different stakeholders' perspectives and different countries' engagement with
it.16

4.      FTI represents only part of the overall efforts towards EFA and UPC. The evaluation
is using contribution analysis as an organising framework for its enquiry. This requires FTI to
be carefully set within the context of these overall efforts; their combined effect is assessed
as a prelude to assessing the particular contributions of FTI to the overall effects. The five
streams of hypothesised effects are separately tracked through Levels 1–3 (inputs 
immediate effects intermediate outcomes), though interactions between the streams are
also sought (there are overlaps between questions, and the external factors/assumptions
column highlights some of the interactions between streams). Under "assumptions" it is
important to check whether different stakeholders and different countries operate with the
same understanding of FTI objectives and functions. If it is found that they do not, the
consequences for efficiency, effectiveness and sustainability should be explored for all
relevant workstreams and evaluation levels.

5.       At Level 4 (Outcomes) the emphasis is on understanding overall effects on the
quantity, quality, access and sustainability of primary education, then the extent to which
these effects can be explained by direct and indirect FTI inputs. Hypotheses and questions
at Level 4 are interlinked with hypotheses and questions at earlier levels. The enquiry at
Level 4 will establish what the outcomes ("results on the ground") have been; questions at
earlier levels will seek to establish whether and how FTI inputs may have contributed to
those results on the ground.

6.     The mid-term evaluation of FTI will not attempt to assess the effects of FTI at Level 5
(Impact). However, an impact evaluation scoping study (see Annex G below) will assess the
prospects, and make proposals, for monitoring and evaluating FTI impacts over the long
term.

16
  Relevance of FTI is also a concern at subsequent levels of the framework. The same applies to the
existence, and possible influence, of differing perspectives on the FTI's objectives and role.



file: FTI_Evaluation_Framework(Jan2009b).doc                                       Page 73 of 134
          Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


7.      The FTI's Indicative Framework proposed benchmark values for a number of key
performance indicators. Country-level evaluators should pay special attention to the role
and relevance of the Indicative Framework and gather available information on trends in the
indicators it proposed.

8.     The letters G and C are used in the matrix below to indicate whether the detailed
questions in the second column apply mainly to the global or to the country level of enquiry.




Page 74 of 134                                                                printed: 04-Feb-09
                                                           Annex E: Extended Logical Framework for the Evaluation


      Main Evaluation                             Information Required (detailed questions)                              Sources of evidence                    External factors/
       Hypotheses                                                                                                                                                Assumptions
                                                                              Level ZERO – PRE-FTI
                                                             (Establishes the context and starting point prior to FTI)
0.1. EDUCATION POLICY AND PLANNING
OVERALL HYPOTHESIS: Education sector policy and planning at global and country levels was lacking; development partners were not supporting country processes
effectively or efficiently
a. International development        Pre-FTI, what was state of progress towards EFA goals (G and C)                  Documents (global):                  External factors which are
    agendas and country             Pre-FTI, what fora existed for global policy and planning? How effective         Background documents on:             important to understanding
    development frameworks          were they? To what extent did they support country level processes? (G)          -     global education strategy,     different contexts over time and
    giving insufficient priority to How did donor agendas influence global policy and planning? What was the               from Jomtien onwards.          across countries include:
    education and also to primary   balance of donor support: between education and other sectors? between           -     Millennium aid commitments  Political and governance
    sub-sector.                     sub-sectors within education (Global – aggregate and individual donors)          -      PRSP approaches                   factors:
b. Education sector policy in       How did key actors conceptualise “a credible plan” post Dakar? (G and C)         -     donor and CSO strategies           o Election cycles
    need of strengthening           What main changes had happened in the education sector in the years prior              (general and education)            o National elites
    globally and at country level   to FTI? (G and C)                                                                -     evaluations of basic               o Stability/fragility
c. Stakeholder involvement in       What influence did the HIPC agenda and PRSPs have on education strategy?               education                          o Corruption
    education sector patchy         Was the PRSP process complementary to EFA planning? Was UPC a priority?          -     origins and intentions of FTI      o Institutional capacity
d. Education planning not taking    Which stakeholders had most influence on the setting of strategic priorities? Documents (country):                              and reforms
    account of key cross-cutting    (G and C)                                                                        National strategies and plans,        Economic factors:
    issues such as gender, HIV,      Did an education sector plan exist before FTI? (If so what was the process      including PRSP                           o global trends
                                    for drafting it, agreeing it, implementing it, monitoring and evaluating it; for Education policy and programme           o national economic
    equity, etc.
                                    taking account of failures and weakness and addressing them? Did it              documents                                      performance
                                    encompass the full sector or were some sub-sector elements omitted? Was          Donor group records, SWAp             Aid environment:
                                    it well established before FTI became a factor? C)                               documents etc                            o aid and foreign policy
                                    How consultative was the process of macro-economic planning and policies         Donor documentation with                       agenda of donors
                                    been? (C)                                                                        analysis on policy and                   o global commitments
                                    How consultative was the process of education policy making and planning? implementation (for country and                       on aid effectiveness
                                    (C)                                                                              for education sector)                    o aid dependency
                                                                                                                                                              o aid modalities
                                    Did education plans take account of key cross-cutting issues such as gender,
                                                                                                                      Interviews:                          Other factors e.g.
                                    HIV, equity, disability? (G and C)                                               Global actors in setting strategy        o HIV and AIDS
                                                                                                                     for early days of FTI.                   o Disasters
                                                                                                                     Country actors and observers for      Cross-currents from other
                                                                                                                     the period ( ministries of               sectors and sub-sectors
                                                                                                                     planning, finance and education,     NB. These factors are relevant,
                                                                                                                     civil society , donor partners, etc) in different degrees, across all
                                                                                                                                                          the workstreams.



file: FTI_Evaluation_Framework(Jan2009b).doc                                                                                                                            Page 75 of 134
                                                Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative

      Main Evaluation                          Information Required (detailed questions)                           Sources of evidence                  External factors/
       Hypotheses                                                                                                                                        Assumptions
0.2. EDUCATION FINANCE
OVERALL HYPOTHESIS – Domestic and external financing constraints were impeding progress towards UPC
a. Overall funding to education     What had been the level of funding to the education sector, and basic     Documents (global):                  Assumption that finance is a
    fell short of needs, including  education, over time from domestic and external resources? (G and C)      International data on aid flows       binding constraint implies
    needs for basic education       How were decisions on funding to the education sector made prior to FTI?  and public spending patterns          that other constraints (e.g.
    (globally and at country level) (G and C)                                                                 Global education financial            implementation capacity)
b. External funding for             How were stakeholders collaborating in decisions around funding and was   datasets (UIS, DAC)                   can be overcome.
    education insufficient and      this based on a review of achievements/progress in the sector? (G and C)
    unpredictable;                  How predictable was funding for basic education? (G and C)                Documents (country):
c. Education not getting            Did a costed ESP exist? To what extent was it used to guide domestic and  MTEFs
    sufficient domestic funding     external resource allocation to education? (C)                            Education sector plans
    priority and/or insufficient    How integrated were key elements of the budget process in the education   Education sector multi-year
    priority for primary education  sector—strategic planning, budget preparation, monitoring and reporting?  financial frameworks
d. No well-costed, and              (C)                                                                       Education sector budgets
    prioritised education sector    What information was available on funding patterns and how accessible was Expenditure monitoring reports
    plan (ESP) in place, and/or     this information (G and C)?                                               End-of-year financial reports
    financial planning and          How comprehensive were the education budget and expenditure reports?      Donor financial reports and
    management (costing,            What role did the MoE play in compiling information on domestic and       strategy documents
    budgeting and monitoring) in    external funding (C)? What role was played by expenditure and financial   Education sector review
    the education sector was        monitoring reports in prospective education planning? (C)                 documents
    weak                            Which external partners were providing the bulk of the resources and      Studies on education finance and
e. Funding levels not adjusted      through what modalities? (G and C)?                                       planning (e.g. PERs, global
    to deal with major challenges                                                                             reports on aid-flows)
                                    What funding was on-budget and off-budget? (C)
    to UPC such as the impact of                                                                              Reports on financial implications
                                    To what extent were specific financial provisions were made to deal with
    HIV and AIDS, conflict,                                                                                   of cross cutting issues (gender,
                                    major challenges affecting the education sector such as HIV and AIDS,
    gender inequality, and                                                                                    HIV, etc.)
                                    conflict, gender and equity, disability (C and G)?
    reaching disadvantaged
    groups                                                                                                    Interviews (country):
                                                                                                              MoE finance and planning staff
                                                                                                              Ministry of Finance
                                                                                                              Ministry of Planning
                                                                                                              National AIDS Authority
                                                                                                              Key external stakeholders e.g.
                                                                                                              donor partners, civil society
                                                                                                              Relevant TA personnel




Page 76 of 134                                                                                                                                                printed: 04-Feb-09
                                                        Annex E: Extended Logical Framework for the Evaluation


      Main Evaluation                           Information Required (detailed questions)                           Sources of evidence                  External factors/
       Hypotheses                                                                                                                                         Assumptions
0.3. DATA AND M&E
OVERALL HYPOTHESIS: Monitoring and evaluation of education sector progress is not prioritized, was weak and fragmented and inadequately linked to overall global and
national development plans
a. Global progress on education    What was the situation regarding education data and M&E prior to FTI         Documents:                           Assumption that obstacles
    goals insufficiently monitored activities? (G and C) In particular...                                       Household surveys                     to evidence-based decision-
b. Agreed upon indicators of       How frequently were the World Education Reports and other UNESCO             Education statistics reports over     making are technical rather
    progress that allow for        international statistics referred to, and used in, policy discussions? (G)   time                                  than political or cultural.
    country comparisons and        How had data collection and analysis in education evolved over time? (G      EFA GMRs
    time trends not in place       and C)                                                                       UIS Reports
c. Monitoring and evaluation of    How was data shared? And to what extent and at what moments was there
    progress in the education      joint discussion around data? Which fora existed for discussion of education Interviews:
    sector in need of              analysis? How functional were these? Who was involved and at what levels? GLOBALLY
    strengthening                  (G and C)                                                                    Education Analysts in 1990s in
d. Decision-making not             What was the capacity for data collection and analysis like prior to FTI?(C) UNESCO / WB
    sufficiently informed by       Which key indicators have been used over time to monitor progress? Were      Consultants promoting EMIS
    evidence                       any targets set? Did these data capture progress on cross-cutting issues     systems
e. Data not adequately             (HIV, gender, equity, disability etc.)? (G and C)                            IIEP staff
    capturing progress on key      Were there any incentives/procedures for stakeholders to pay attention to    Donors, including former
    cross cutting issues (HIV,     data in their decision making? If so which ones and to what extent did they  education specialists in
                                   work? (G and C)                                                              multilateral and bilateral agencies
    gender, etc.)
                                   What resources were expended over time on data collection and analysis?      IN CASE STUDY COUNTRIES
                                   (C)                                                                          MoE Planning staff at central and
                                   Was there an EMIS in place? (C)                                              decentralized levels
                                                                                                                Ministry of Planning
                                   Was data informing decision making systematically? In what ways? (C)
                                                                                                                Ministry of Finance
                                                                                                                National Bureau of Statistics
                                                                                                                Consultants providing support on
                                                                                                                EMIS
                                                                                                                Donors, including former
                                                                                                                education specialists in
                                                                                                                multilateral and bilateral agencies




file: FTI_Evaluation_Framework(Jan2009b).doc                                                                                                                    Page 77 of 134
                                                 Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative

      Main Evaluation                            Information Required (detailed questions)                             Sources of evidence                    External factors/
       Hypotheses                                                                                                                                              Assumptions
0.4. CAPACITY DEVELOPMENT
OVERALL HYPOTHESIS: Poor capacity was hampering the development and implementation of education plans and programmes and affecting progress on achieving UPC
a. Weak capacity affects various     What were the main capacity constraints on the sector? Which areas of the     Reports:                           Special influences of external
    levels (e.g. planning,           sector were most affected? (sub-sectors, activities, institutional levels,    Analysis of capacity bottlenecks   factors on capacity include:
    budgeting, financial             geographical regions) (G and C)                                               from general sector and country    -    salaries
    management, education            How did the DPs approach capacity development (including the capacity of      reports.                           -    public service structure and
    service delivery, M&E.           DPs themselves)? (G and C)                                                    Institutional and governance            reform
b. Capacity development not          How well was capacity development linked to planning and priority activities  reports.                           -    governance and economic
    receiving sufficient attention   in the sector? (C)                                                            Reports of capacity assessments         management
    in planning and budgeting        What major capacity development efforts were in place prior to FTI for        in the civil service and the
    globally and nationally          planning staff as well as for teachers and other important areas of the plan? education sector                   Also relevant over time are
c. Monitoring of outcomes and        (C and G)                                                                     Donor reports                      changing international
    impact of capacity               Who was responsible for these? How were priorities determined? How were Capacity development plans               approaches to the analysis of
    development needing              modalities decided? How effective were these efforts? (C)                     Sector plans                       capacity, and changing concepts
    improvement                      How was the outcome and impact of the capacity development being              WB IEG Report on CD                of good practice in TA and
d. Capacity development efforts      tracked in general? (C)                                                       JICA TC Report                     capacity development.
    fragmented and insufficiently    Was there prior to FTI a capacity building plan for the sector? If so, did it
    linked to key sector priorities; address cross-cutting issues such as gender, HIV/AIDS, disability, equity     Interviews:
    not addressing cross-cutting     issues? (C)                                                                   Relevant MoE staff
    issues such as gender,                                                                                         Ministry of Planning
    HIV&AIDS and equity issues;                                                                                    Ministry of Finance
    not integrated with                                                                                            Key donor task managers over
    institutional and                                                                                              time
    organisational development                                                                                     Key responsible consultants/TA
                                                                                                                   over time
                                                                                                                   Training institutions (private and
                                                                                                                   public)
                                                                                                                   Universities




Page 78 of 134                                                                                                                                                      printed: 04-Feb-09
                                                       Annex E: Extended Logical Framework for the Evaluation


      Main Evaluation                          Information Required (detailed questions)                         Sources of evidence                 External factors/
       Hypotheses                                                                                                                                     Assumptions
0.5. AID EFFECTIVENESS
OVERALL HYPOTHESIS: Aid to basic education was fragmented, fell short of needs and was insufficiently targeted to global and national priorities
 a. Aid provided in a fragmented  What had been the history of aid to the education sector over time? What    Reports:                           Need to appreciate that the
    manner and guided mostly by   key bottlenecks existed in coordination, harmonisation and alignment prior  Background literature on aid        international aid
    individual donor priorities   to FTI? (G and C)                                                           effectiveness and new aid           effectiveness debate
 b. Planning taking place at the  Which donors have been involved? Which donors have been most                modalities.                         continued to evolve during
    level of individual           influential and in what way? (G and C)                                      Reports of annual sector reviews    the genesis and
    stakeholders rather than      What levels of funding were being provided? What have been the trends in    Individual donor reports            implementation of the FTI.
    collectively                  terms of modalities and priority areas of support? (G and C)                Multi-donor aid reviews             Important to take account
 c. Aid distribution between      How have key stakeholders coordinated and reviewed progress? Have any       Selective comparison of FTI with    of changing perceptions and
    countries is sub-optimal      external assessments of aid effectiveness to the sector (or more broadly    other GRPPs.                        pressures over time.
    (donor orphans and financing  been conducted? What main changes had taken place in this respect over
    gaps)                         time? Were there in-country donor agreements? (C)                           Interviews:
 d. Aid distribution between      What role has the MoE or/and other ministries taken over time in            MoE senior staff (especially from
    education sub-sectors is also coordinating aid to the education sector? How effective has this been? (C)  planning) and former relevant
    sub-optimal                   What was the situation like with respect to external aid to the sector when staff
f. External funding modalities    FTI came in? (C)                                                            Ministries of finance/planning
    for education are inefficient What sector wide programs and approaches were in place or in preparation Key education stakeholders over
    (weakness in alignment and    (SWAps)? (C)                                                                time (globally and nationally)
    harmonisation)                What motivated country stakeholders to go for FTI? Were there different     Civil society
 e. Mechanisms for discussion     motivations for different stakeholder groups? Did any tensions arise as a   Teacher unions
    around aid effectiveness poor result? Were all main stakeholder groups able to participate? (C)
    and insufficiently            What was the purpose of FTI understood to be? Did this change over time
    consultative                  (e.g. for different cohorts of endorsed countries)? (G and C)
 f. Insufficient attention to
    cross-cutting issues that
    affect the education sector




file: FTI_Evaluation_Framework(Jan2009b).doc                                                                                                                Page 79 of 134
                                                   Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative

      Main Evaluation                             Information Required (detailed questions)                                Sources of evidence                    External factors/
       Hypotheses                                                                                                                                                  Assumptions
                                                                          Level ONE – ACTIVITIES / INPUTS
                             (Relates to activities and inputs which were provided in support of basic education, globally and at country level)
1.1. EDUCATION POLICY AND PLANNING
OVERALL HYPOTHESIS: FTI processes and activities supported the strengthening of policy and planning for EFA and UPC at global and country levels.
a. Policy experiences across            What FTI activities were undertaken in relation to education policy and      Reports:                              Government and key
    countries periodically shared       planning? In what ways were education policy and planning modified as a      Education policy documents                stakeholders committed to
    and used to inform decision         result? (G and C)                                                            Education review reports                  promoting EFA
    making by FTI partners              What specific contributions were made by EPDF in support of education        Reports and documents of the FTI  (For most FTI countries)
b. Existing policy framework for        policy development and planning? (C)                                         Secretariat for specific countries        PRSP process in place and
    the education sector                What was the process for preparing for endorsement? (C)                      Hazel Bines World Vision report           constitutes a reference for
    reviewed by all stakeholders        What changes took place to the plan before it was presented to FTI? Was it   Education’s Missing Millions              planning
    with guidance from MoE and          improved as a result of this process? In what ways? Was the development of (2007)                                  Education sector plan
    designated lead donor in            the indicative plan in keeping with what the country would have done itself, EDPF concept notes. EPDF                  available as a starting point
    order to achieve UPE                possibly in a different manner? (C)                                          proposals, activity reports               for more detailed planning
c. Education sector policy and          Who initiated the review process of the education sector as part of the FTI                                        Commitment by key
    implementation plan costed          engagement? What was the quality of the process? (C)                         Interviews:                               partners to a participatory
d. Studies and consultations            Was there a lead donor? How was the donor selected? To what extent was       MoE directors of planning                 approach for fine-tuning the
    conducted to strengthen             this influenced by the political economy in the donor group? (C)             Senior MoE staff in charge of             education plan.
    weak areas of policy and            What was the decision making process in the donor group? (C)                 policy                                Funding and capacity in
    planning                            Who were the key stakeholders involved? How was their participation          FTI Secretariat and Steering              place to conduct additional
e. [Participation] Country PRSP         sought? Did their participation change over time? (C)                        Committee members                         studies as a basis for an
    reviewed in a participatory         What were the main policy contestations? Were additional studies             Consultants supporting the MoE            improved sector plan
    manner, examining in                conducted to develop particular policy areas? By whom and with what          and FTI                               At least one donor willing
    particular priority for             consultation? (C)                                                            WB Regional Managers                      and sufficiently resourced
    education within the overall                                                                                                                               (manpower) to take on the
                                        How effective were the reviews? (C)
    development framework                                                                                            Field visits:                             role of lead donor for the
                                        What role did the indicative framework of FTI play in shaping country
f. [CCI] Policy framework                                                                                            To educational settings to assess         FTI process
                                        policy? (C)
    addresses cross-cutting                                                                                          implementation of policies.;
                                        To what extent, if any, did global discussions around education policies
    issues such as gender,                                                                                           stakeholder perceptions of FTI-      Test the assumption that
                                        translate into country level action? (C)
                                                                                                                     related activities                   different actors share the same
    HIV&AIDS and equity                 What specific efforts were made to ensure that cross-cutting issues were
                                                                                                                     Schools, colleges, Policy institutes understanding of FTI objectives,
                                        adequately addressed in policy and planning? How successful were these
                                                                                                                                                          mechanisms and obligations.
                                        efforts? (G and C)




Page 80 of 134                                                                                                                                                           printed: 04-Feb-09
                                                          Annex E: Extended Logical Framework for the Evaluation


      Main Evaluation                            Information Required (detailed questions)                             Sources of evidence                    External factors/
       Hypotheses                                                                                                                                              Assumptions
1.2. EDUCATION FINANCE
OVERALL HYPOTHESIS – The FTI process put in place a complete set of activities aimed at identifying financial requirements and mobilising domestic and external funding
to support EFA and UPC globally and at country level.
 a. Support to education sector    What FTI financial inputs were provided, and what FTI activities were         Documents:                            Financial resources and
    strategic planning: costing    undertaken to encourage the financing of basic education? (G and C)           FTI secretariat records e.g.              technical capacity available
    and modelling studies          Did the FTI strategy for addressing financing gaps change significantly over  minutes, supporting documents             for conducting additional
    conducted                      time? Have donor expectations of financing considerations been consistent Country level FTI documents e.g.              studies
 b. Comprehensive financial        over the FTI process period? (G and C)                                        Initial info. from FTI to LEG;        Donors make available data
    framework (FF) developed       How were the global and country level FTI processes and criteria              Govt letter to LDG; fiduciary             on activities, financial
    consistent with ESP; financial developed? G(G)                                                               assessment; mapping of current            commitment and modalities
    gap quantified                 What measures were taken to mobilise funds at global level for the CF and     and future aid; EPDF application;         of support
 c. Medium-term budget             EPDF? G(G)                                                                    ESP appraisal report; CF program      EPDF and CF resources
    prepared consistent with ESP   How and by whom was the education sector financial framework put              proposal                                  provided in a timely manner
    financial framework            together? Had such exercises been conducted pre- FTI? (C)                     LEG & LDG minutes                         to fund and support
 d. Financial management and       o Did the MoE consult/work with other ministries, notably MoF, in             ESP & financial framework                 selected activities
    monitoring arrangements             producing the FF? Was a medium-term budget for education prepared        Education sector medium-term          Commitment and
    reviewed; instruments and           based on the ESP FF? (C)                                                 and annual budgets, expenditure           participation by other
    processes developed            o What was the consultative process on proposed financial allocations?        reports, financial monitoring             government departments
 e. External funds mobilised at         Which stakeholders participated? (C)                                     reports                                   (especially finance and
    global level to support the    o Were relevant cross-cutting issues taken into account? (C)                  Donor budgets and reports                 planning)
    CF/EPDF                        How did the FTI indicative framework influence the development of the FF?                                           Bottlenecks in financial
 f. Country applications           Did its primary education focus affect financial planning for other           Interviews:                               systems identified in a
    submitted for the CF and       subsectors? C(C)                                                              FTI secretariat                           timely manner and
    EPDF as appropriate            Was all external donor support to education included in the FF? C(C)          MoE finance and planning staff            addressed
 g. Information and lessons        What instruments and processes to strengthen financial management and         Ministry of Finance                   Coherent resource planning
    shared between countries on    monitoring of the education sector were developed? G(G and C)                 Ministry of Planning                      takes place at sector level,
    FTI processes                  Did external partners share information and lessons on funding modalities     National AIDS Authority                   or is adequately articulated
                                   for education? G(G and C)                                                     LDG/LEG; Key non-govt                     with national planning,
                                                                                                                 stakeholders e.g. civil society, and      budgeting and aid
                                   What did the process of: (i) seeking endorsement of the ESP; (ii) applying to
                                                                                                                 others involved in the LEG                management.
                                   the CF; (iii) applying to the EPDF, involve? C(C)
                                                                                                                 Consultants and other providers      Test the assumption that
                                   What activities or processes were put in place to promote information
                                                                                                                 of external technical support        different actors share the same
                                   sharing and lessons learning between countries on FTI processes? (G)
                                                                                                                                                      understanding of FTI objectives,
                                   To what extent did FTI-supported activities add to or enhance pre-existing
                                                                                                                                                      mechanisms and obligations.
                                   sector processes? (G and C)




file: FTI_Evaluation_Framework(Jan2009b).doc                                                                                                                         Page 81 of 134
                                                Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative

      Main Evaluation                           Information Required (detailed questions)                           Sources of evidence                  External factors/
       Hypotheses                                                                                                                                         Assumptions
1.3. DATA AND M&E
OVERALL HYPOTHESIS – FTI recognized the importance of data systems for decision making and supported activities which aimed at ensuring strong data collection,
reporting and feed back to decision making
 a. Indicators agreed for            What data-related activities did FTI directly undertake or indirectly support? GLOBALLY                         Bottlenecks in data
    monitoring progress towards      (G and C)                                                                      Documents                         management identified and
    UPC and other EFA goals          Which indicators were selected to monitor progress in quality of data          GMR                               a process designed to
 b. Strength and weaknesses of       systems? How was this carried out, according to whose design and under         FTI Indicative Framework and      address these
    data management assessed         what management? Were any benchmarks/targets defined? Were cross-              progress reports                 Stakeholders committed to
    and appropriate mechanisms       cutting issues included? (G and C)                                                                               ensuring that data systems
    put in place for continuous      What reporting mechanisms around data were put in place? (G)                   Interviews                        get sufficient funding and
    quality improvement.             What capacity building has taken place to strengthen data systems? (G and      FTI Secretariat                   are prioritized
 c. Capacity built for designing,    C)                                                                             Consultants promoting EMIS       Staff in place in key
    maintaining and updating         What priority activities were rolled out under FTI to address constraints on   systems                           positions for data
    data systems                     data collection and analysis? (G and C)                                        GMR staff                         management at MoE and
 d. Indicators of quality of data    Was EPDF used for strengthening data systems, and if so what activities        UIS staff                         decentralized levels
    systems agreed upon              were conducted? What funds were made available under EPDF and were             IIEP staff                       EPDF funding provided in a
    between stakeholders,            these commensurate to the problem? Were there any issues in accessing                                            timely manner for capacity
    monitored regularly and used     EPDF funding and support? Was EPDF funding used at all to strengthen           CASE STUDY COUNTRIES              building needs in data
    in decision making on            capacity on data collection and processing around cross-cutting issues? (G     Documents:                        management
    priorities for improving data    and C)                                                                         Government and MoE progress      Technical support available
    management                       What is status of EMIS in FTI and other countries? (G and C)                   reports and annual reviews       Mechanisms for periodic
                                     Do pre-endorsement documents include an analysis of gaps in data and           Donor documentation               review of data established
                                     progress in filling them? (G and C)                                            EPDF progress reports             and used
                                     Do JARs - or equivalent – include an analysis of the progress in improving                                      Cultural and political
                                     data systems and gaps and (i) how do these compare between FTI and non-        Interviews:                       obstacles to evidence based
                                     FTI countries and (ii) how have these changed in FTI and non-FTI countries     Senior MoE and other planning     decision making overcome
                                     over this decade? (G and C)                                                    staff
                                                                                                                    Relevant TA personnel
                                     Which indicators were selected to monitor progress? How were these
                                     determined? Were cross-cutting issues included? (C)
                                                                                                                    Field work:
                                                                                                                    Field visits to local education
                                                                                                                    offices




Page 82 of 134                                                                                                                                                 printed: 04-Feb-09
                                                        Annex E: Extended Logical Framework for the Evaluation


      Main Evaluation                          Information Required (detailed questions)                           Sources of evidence                  External factors/
       Hypotheses                                                                                                                                        Assumptions
1.4. CAPACITY DEVELOPMENT
OVERALL HYPOTHESIS – FTI promoted a harmonized approach to capacity development and supported capacity development in areas which constitute a priority for
achieving UPC
a. Stakeholders jointly identify   What activities related to capacity development did FTI directly undertake  Documentation:                     Country stakeholders have a
    capacity gaps and agree on     or indirectly support? (G and C)                                            EPDF reports                        shared vision of capacity
    plans to address these         What was the process by which capacity development needs were identified FTI regional reports                   development and are
b. Capacity development plans      (under whose initiative, who managed it, and to what extent is capacity     Needs assessment                    committed to ensuring that
    are periodically updated to    assessed more broadly in the public sector and not just the education       documentation                       it gets priority attention
    reflect changing needs and to  sector)? (C)                                                                Capacity development plans         Capacity gaps clearly
    ensure mutual accountability   Have the FTI Capacity Development Guidelines been used, to what effect,     Sector development plan             identified and a process for
c. Capacity development is         with whose involvement? (C)                                                 Aide memoires                       addressing these developed
    conducted in accordance        Is there a capacity development plan? How was it developed? Is it used? (C) Hazel Bines World Vision report    EPDF funding provided in a
    with needs of all              To what extent is there a shared perception of capacity development by the Education’s Missing Millions 2007    timely manner and in line
    stakeholders, in line with     MoE, the donor community and other country stakeholders? (C)                                                    with the identified needs
    their respective roles and     How often is the capacity assessment updated,? Do the updates show          Interviews:                        Trained staff stay on the job
    cognizant of institutional and changes in the plan and activities? (C)                                     Key stakeholders, including         (i.e. incentives in place to
    organisational constraints.    How have regional advisory groups for EPDF influenced the CD assessment,    donors and lead donor               keep them there)
d. Proposals submitted to the      its planning, implementation processes? (G and C)                           Task managers                      System able to cope with
    EPDF for specific capacity     To what extent have country stakeholders contributed towards identifying    Universities and other training     loss of manpower through
    development priorities         approaches to developing capacity and rolling these out? (C)                institutions                        HIV and AIDS
e. Monitoring and evaluation of    What levels and types of funding were made available for capacity           Private training providers         Systems in place for
    capacity development           development? From what sources? (G and C)                                   WB regional education managers      periodically reviewing
    activities carried out and                                                                                 TA/hired consultants for capacity   capacity gaps and for
                                   Was effective support provided to build capacity in financial management?
    reports on results/impact                                                                                  development                         revising plans for capacity
                                   (C)
    produced, disseminated and                                                                                                                     development
                                   What proposals were submitted for funding by the EPDF? What financing
    acted upon.
                                   and support was received? How effective was this? How was effectiveness
f. Cross-cutting issues
                                   assessed? (G and C)
    integrated across capacity
                                   To what extent have relevant cross-cutting issues been included in capacity
    development plans and
                                   development efforts? How important is capacity development in these
    activities and progress and
                                   areas to stakeholders? Has any funding been made available specifically for
    included in reporting.
                                   these priorities? (G and C)
                                   How are the outcomes and impact of capacity development efforts being
                                   monitored? By whom? To what extent does monitoring inform future
                                   priority setting? In what ways? (G and C)




file: FTI_Evaluation_Framework(Jan2009b).doc                                                                                                                   Page 83 of 134
                                                 Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative

      Main Evaluation                            Information Required (detailed questions)                             Sources of evidence                   External factors/
       Hypotheses                                                                                                                                             Assumptions
1.5. AID EFFECTIVENESS
OVERALL HYPOTHESIS – FTI resulted in a review of current practice to promote better coordination, harmonisation and alignment among stakeholders in education at
global and country levels.
a. Coordination mechanisms            What activities did FTI directly undertake or indirectly support to promote  Documentation:                      Donors committed to
    reviewed and strengthened         greater aid effectiveness? (G and C)                                         Document on aid management              improved coordination,
b. Mechanisms put in place for        What were the governance arrangements for FTI, and how did they change       and coordination, including             harmonization and
    ensuring that the FTI             over time? (G and C)                                                         multi-sector forums and                 alignment
    endorsement process is an         How have global efforts at coordination impacted at country level? (G and C) programmes (e.g. PRSCs)             MoE takes leadership of
    integral part of the country      What changes in coordination among stakeholders have taken place in the      Annual education reviews                efforts to promote
    process for education review      education sector? To what extent can these changes be attributed to FTI?     Overall aid effectiveness studies       coordination in the
    and approval                      (C)                                                                          MOF, MoE and donor group                education sector
c. Specific efforts made to           How and to what extent were key bottlenecks in coordination,                 statistics on donor support to the  MoE willing to demand
    include additional                harmonisation and alignment addressed through FTI? (C)                       sector                                  accountability by all
    stakeholders to ensure            Did stakeholders adopt more harmonised and aligned approaches? How did                                               stakeholders
    adequate representation of        this come about? (G and C)                                                   Comparison with other GRPPs         Approaches adapted to
    key groups                        To what extent were changes in aid management for the education sector                                               country systems of
d. Stakeholders regularly             linked to overall changes in aid management? (G and C)                       Interviews:                             decentralisation.
    provided with updated             How is aid effectiveness monitored? How is information on aid effectiveness  MoE and other ministry staff        Donor incentives to reform
    information on aid flows and      shared, and how does this inform further decision making? (G and C)          responsible for coordinating            their ways of doing
    aid effectiveness              How is adherence to the Paris principles monitored as part of FTI? (G and C)    external support to the sector          business.
e. Monitoring country progress                                                                                     Key donors and other               Test the assumption that
    and identifying constraints to                                                                                 stakeholders over time             different actors share the same
    further progress (e.g. poor                                                                                    Ministries of Planning and         understanding of FTI objectives,
    accountability, concerns                                                                                       Finance                            mechanisms and obligations.
    about financial management,
    etc.)
f. Specific actions identified to
    address constraints to aid
    effectiveness e.g. capacity,
    strengthening of public
    financial management, etc.
g. Aid harmonisation and
    alignment efforts take
    account of key cross-cutting
    issues.




Page 84 of 134                                                                                                                                                      printed: 04-Feb-09
                                                         Annex E: Extended Logical Framework for the Evaluation


      Main Evaluation                           Information Required (detailed questions)                            Sources of evidence                   External factors/
       Hypotheses                                                                                                                                           Assumptions
                                                                    Level TWO – IMMEDIATE EFFECTS
                                          (Relates to the effects of the activities/inputs on processes in the education sector )
2.1. EDUCATION POLICY AND PLANNING
OVERALL HYPOTHESIS – Adoption of education policies and plans which reflect strategic priorities for achieving UPC and draw support from all key stakeholders
a. Improved understanding of        What changes have occurred in donor policies to commit them to a stronger Documentation                          Donor agendas and
    priority strategies and actions emphasis on support to EFA? (G and C)                                          FTI reports                        commitments reflect (and
    needed globally and at          What increases are discernable in the prioritisation of UPC –related policies  Media reports                      are adjusted to) priority for
    country levels to reach EFA     and plans? C(C)                                                                Reports of WB Annual Meetings      basic/primary education
    and UPC goals is reflected in   Has there been an increase in media/civil discussion of education and in the Global Monitoring Reports           Gov and MoE committed to
    operational plans               media generally? To what extent was open public debate fostered by FTI?        Reports of Annual Sector Reviews   inclusive and participatory
b. National plans, including        (C)                                                                            and Joint Review Missions          approach to education
    PRSP, reflect greater priority  How coherent are the PRSP, the national development plan, and the              Thematic Studies and reports       planning
    for (basic) education sector    education sector plan? Has this changed over time? (C)                         EPDF annual and activity reports  Wider stakeholder
c. Increased commitment to          What was produced as a result of the planning process under FTI? What did      In-country Civil Society reports   participation strengthens
    EFA and UPC reflected in        the policy framework entail? Did a pre-existing policy framework exist and if                                     sector policy and
    donor policies                  so was it altered in response to FTI? (C)                                      Interviews                         performance.
d. Involvement of a wider group     Did the education plan presented to FTI have any parliamentary oversight at    FTI Secretariat
    of stakeholders in meaningful   all? Or oversight by other ministries? C(C)                                    CS Stakeholders
    policy dialogue around basic    What emphasis has been given to the full extent of EFA, i.e. including early   Education International and in-
    education                       childhood, Adult and Basic (ABE) and non-formal, as opposed to over-           country teacher unions
e. Specific strategies and          emphasis on formal primary schooling at the expense of the other               WB Regional directors-
    priorities identified and       elements? (G and C)                                                            Members of FTI Steering
    adopted for key cross cutting   What policies for key cross-cutting issues have been put in place and how      Committee, etc
                                    strongly are they reflected in plans (Annual Work Plans and other?) (C and     MoE directors of planning
    issues in the sector (gender,
                                    G)                                                                             Senior MoE staff in charge of
    HIV&AIDS, equity)                                                                                              policy
                                    What are the mechanisms for key stakeholders to make their contributions
                                    in formulation of policies, plans and in periodic (annual) reviews, especially Ministries of Finance and
                                    with respect to EFA? C(C)                                                      Planning
                                                                                                                   Consultants supporting the MoE
                                    What has been the role of the EPDF in generating and supporting a
                                                                                                                   and FTI
                                    deepening commitment to EFA? (regional and country)
                                                                                                                   NGOs and INGOs
                                    To what extent is stakeholder engagement ongoing? C(C)
                                    To what extent have the changes at this level resulted from the FTI activities
                                    and inputs identified at Level 1? (C and G)




file: FTI_Evaluation_Framework(Jan2009b).doc                                                                                                                      Page 85 of 134
                                                 Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative

      Main Evaluation                            Information Required (detailed questions)                            Sources of evidence                   External factors/
       Hypotheses                                                                                                                                            Assumptions
2.2. EDUCATION FINANCE
OVERALL HYPOTHESIS – Education sector budget process is more transparent, comprehensive, integrated and efficient
a. Education budget cycle          Is the education sector budget comprehensive and consistent with the ESP       Documentation:                       Government staffing in
    processes strengthened;        FF? C(C)                                                                       ESP, FF & monitoring &                place at critical levels of the
    clear links between financial  What financial monitoring and review processes for education are in place at implementation framework                system
    monitoring and reporting,      national and sub-national levels? How comprehensive are they? Who is           Education budgets and                External partners
    strategic planning, and        involved? What is the information used for? C(C)                               preparation documentation             able/willing to provide data
    budget preparation             Are financial monitoring reports used in: (i) overall education sector review  Expenditure monitoring reports        on financial commitments
b. Strong financial management     processes; (ii) education planning and budget preparation processes? C(C)      End-of-year financial reports
    systems in place to facilitate Are effective processes in place for sharing information on education          Donor financial reports and
    comprehensive monitoring of    funding between key government departments and external partners? C(C)         strategy documents
    education sector funding       Are there any systems in place for tracking public expenditure to school and   Studies on education finance and
c. Improved coordination and       student level? C(C)                                                            planning (e.g. PERs)
    information sharing between    What financial monitoring processes for education are in place at global       Education sector review
    key government departments     level (including CF/EPDF) monitoring)? What is the information used for?       documents
    and external partners on       G(G)                                                                           LDG minutes
    education funding              How did external partners decide on the amount of funding to commit to         FTI CF/EPDF financial reports
d. Greater clarity among           the education sector? Are decisions linked to information from a               Global education financial
    external partners on resource  performance monitoring process? (G and C)                                      datasets (UIS, DAC)
    requirements; planned          Did external partners decide to adopt more harmonised and aligned
    funding increasingly linked to modalities? Why did they take these decisions? (G and C)                       Interviews:
    improved performance in the    What modality was agreed for grants from the CF? Was this modality             MoE finance and planning staff
    sector                         harmonised with others used by external partners in the sector? How            Ministry of Finance
e. Decisions by external           aligned was this modality? (C)                                                 Relevant parliamentary
    partners to adopt more                                                                                        committees
                                   Can MoF clarify distinct aid flows into education? Are these as expected? (C)
    aligned financing modalities.                                                                                 LDG
                                   What changes have there been in the proportion of the education budget
f. Agreement on harmonised                                                                                        Key non-government
                                   allocated to EFA in its widest sense: specifically to primary? (G and C)
    modalities for external                                                                                       stakeholders e.g. civil society,
                                   To what extent have the changes at this level resulted from the FTI activities
    funding and mechanisms for                                                                                    others involved in the LEG
                                   and inputs identified at Level 11? (G and C)
    enhancing efficiency and                                                                                      FTI secretariat staff who work on
    reducing transaction costs                                                                                    the CF/EPDF; CF fund managers
g. Agreement on a harmonised
    and aligned modality for CF




Page 86 of 134                                                                                                                                                     printed: 04-Feb-09
                                                         Annex E: Extended Logical Framework for the Evaluation


      Main Evaluation                           Information Required (detailed questions)                            Sources of evidence                   External factors/
       Hypotheses                                                                                                                                           Assumptions
2.3. DATA AND M&E
OVERALL HYPOTHESIS – Data collection on education progress improved, shared with stakeholders and capacity built to inform decision making around policies and plans
for achieving UPC
 a. Improved frequency and         How has data collection and analysis in education evolved over time? (G and Documents                             Staff and systems in place
    quality of reporting on        C)                                                                               Review of EMIS systems based on   for data collection at all
    progress towards EFA and       Can countries report any changes in data collection and management               published articles (G and C)      levels of the education
    UPC                            decision-making on the basis of data at national and district levels? (C)        Review of a sample of JARs from   system
 b. Improved understanding of      Is there any evidence of use of data for management at sub-national levels? FTI and non-FTI countries (G and      Mechanisms for sharing of
    areas of progress and of gaps  (G and C)                                                                        C)                                information fully
    that need to be addressed to   Has there been a qualitative improvement in data collection and                  Search for published or grey      operational and linked to
    achieve EFA, including on      dissemination? If yes, in what way? (G and C)                                    literature on sub-national        decision making structures
    cross-cutting issues affecting If implemented, are indicators of quality of data systems monitored              analyses and classification by    internally (within the sector)
    the sector                     regularly> (G and C)                                                             contributing authors (G and C)    and externally (MoF,
 c. Improved capacity for data     Is data more widely available and more regularly distributed? (G and C)                                            donors, etc.)
    use and management at          What evidence is there for increased capacity of data analysis at various        Interviews                       Staff in charge of
    various levels of education    levels of system (G and C)                                                       Directors of Planning (C)         management and decision
    systems reflected in quality   What is coverage of non-formal education, private sector data, etc.              Main Providers(s) of Non-State    making (at all levels of the
    reporting, and flow of         Did FTI contribute, directly or indirectly, towards filling perceived data gaps? Education (C)                     system) has sufficient
    information.                                                                                                                                      capacity and incentives to
                                        Can the FTI Secretariat indicate any improvements in education data in
 d. Key intermediate indicators                                                                                                                       use data as a basis for
                                        FTI countries? (G)
    and targets for measuring                                                                                                                         decision making
                                        Can education planners identify any improvements in education data
    progress on EFA and UPE                                                                                                                          External partners decision
                                        and if so, how far do they attribute this to FTI or other sources of
    agreed upon among                                                                                                                                 making processes data
                                        support? (C)
    stakeholders, including                                                                                                                           driven
                                   To what extent have the changes at this level resulted from the FTI activities
    indicators and targets on key                                                                                                                    Accountability mechanisms
                                   and inputs identified at Level !? (G and C)
    cross-cutting issues                                                                                                                              in place which are data
                                                                                                                                                      driven
                                                                                                                                                     Cultural and political
                                                                                                                                                      obstacles to evidence based
                                                                                                                                                      decision making overcome.




file: FTI_Evaluation_Framework(Jan2009b).doc                                                                                                                      Page 87 of 134
                                                 Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative

      Main Evaluation                            Information Required (detailed questions)                             Sources of evidence                   External factors/
       Hypotheses                                                                                                                                             Assumptions
2.4. CAPACITY DEVELOPMENT
OVERALL HYPOTHESIS – Support to the education sector oriented towards developing capacity in agreed-upon priority areas of the education sector plan – and
increasingly harmonised.
 a. Key stakeholders aware of          Have the key stakeholders come together to harmonise and align capacity        Documents                          Donor agendas prioritize
    main capacity gaps and             development according to a commonly understood approach to needs               EPDF and other donor capacity       capacity development of
    committed to providing             identification and choice of funding, CD modalities and management? (C         development evaluations             partner countries and
    increasingly harmonised,           and G)                                                                         ‘Project’ documents on capacity     sectors
    necessary financial and            Is CD integrated in the education sector plan? Is it costed? (C)               development and reviews            Donor funding sufficiently
    technical support in context       Have additional funds been committed to CD as a result? (C and G)                                                  flexible to allow for
    of institutional and               Has a capacity building plan been produced in the context of EFA and UPC?      Interviews                          adjustments based on
    organisational constraints         What is the quality of this plan and of the strategies?                        Key stakeholders, including         specific capacity needs
 b. Capacity development needs              Have contextual issues been taken into account in drafting the CD plan?   donors and lead donor              Stakeholders periodically
    adequately costed and                   Does the plan address institutional development and organizational        Task managers                       held accountable for
    integrated in the Education             environment issues? (C)                                                   Universities and other training     progress on capacity
    Sector Plan                        What is the ownership of the plan, who manages it and who monitors it?         institutions                        development
 c. Increase in resources              Has there been wider participation of other stakeholders in defining the       Private training providers         EPDF funding made
    devoted to capacity                approach to and development of CD plan? (C)                                    WB regional education managers      available in accordance with
    development in context of          Has the ‘niche’ for EPDF activities been identified – against other CD         TA/hired consultants for capacity   priority needs and in a
    EFA and UPC                        funding? To what extent does the EPDF mechanism influence whether CD           development                         timely manner
 d. Funding and human                  needs and resource requirements are embedded in the sector plan or not?                                           Government staffing and
    resourcing shifts at level of      (G and C)                                                                                                          human resource
    different stakeholders (MoE,       Have donor representatives/ lead/ task team leader with appropriate                                                development plans in place
    donors, etc.) reflect capacity     qualifications and experience been appointed to deal with ongoing                                                  and in line with key
    gaps and needs globally and        developments? (C)                                                                                                  priorities of the education
    at country level                                                                                                                                      sector plan
                                       Have appointments and post descriptions of those undertaking CD
 e. Increasing attention to                                                                                                                              Government civil service
                                       management within the MOE been made? (C)
    capacity development of                                                                                                                               mechanisms address factors
                                       Has such an approach paid sufficient attention to cross-cutting issues such
    external stakeholders such as                                                                                                                         which impact negatively on
                                       as gender, HIV&AIDS and equity – and included a concern for disability (G
    NGOs and donors                                                                                                                                       staff retention
                                       and C)
 f. Capacity development
                                   To what extent have the changes at this level resulted from the FTI activities and
    includes attention to cross-   inputs identified at Level !? (G and C)
    cutting issues such as gender,
    HIV&AIDS and equity




Page 88 of 134                                                                                                                                                     printed: 04-Feb-09
                                                          Annex E: Extended Logical Framework for the Evaluation


      Main Evaluation                            Information Required (detailed questions)                          Sources of evidence                  External factors/
       Hypotheses                                                                                                                                         Assumptions
2.5. AID EFFECTIVENESS
OVERALL HYPOTHESIS – Partners modify their way of operating globally and at country levels to ensure better coordination, harmonisation and alignment
a. Greater clarity on mutual       What instructions have headquarters of donor agencies, bilateral and           Documents                           Country and leadership on
    commitments and time           multilateral, issued to their country teams to apply Paris Declaration         Literature related to Rome and       donor coordination strong
    frames among partners          harmonisation and alignment, etc. Which of their procedures, e.g.              Paris Declarations; monitoring of   Donor HQ, regional and
b. Increased aid flows globally    commitment to conduct bilateral reviews and evaluations, have they             the implementation of the Paris      country levels interact and
    and at country levels linked   changed? (G)How have country and DP behaviour changed at country level         Declaration                          collaborate effectively to
    to education performance       in terms of:                                                                   Studies on use of country            address bottlenecks to aid
c. Aid commitments that are              mechanisms for dialogue, alignment and harmonisation use of country      systems                              effectiveness
    longer-term and more                 systems?                                                                 Literature on SWAps and budget      Donors held accountable for
    predictable                          aid modalities and approaches (budget support, SWAps etc)                support (with special reference to   progress on coordination,
d. Improved reporting of           agreed performance indicators and systems for joint review? (C)How have        education sector)                    harmonisation and
    stakeholders' financial and    approaches to aid effectiveness in the education sector (especially basic      Donors' general and country          alignment at country and
    technical contributions to the education) evolved in comparison to other sectors? (C)                         reports and studies                  global levels
    sector                         How effective has been the coordination between the education sector and       Comparison with other GRPPs         Mechanisms for reviewing
e. Increased use of aid            national systems for planning, budgeting and aid management? (C)                                                    progress globally and at
    modalities that are more       How effective has been the coordination between central and local levels?      Interviews                           country level in place and
    harmonised and aligned         (C and G)                                                                      Participants in global and country   effective in making
                                   To what extent have the changes at this level resulted from the FTI activities level aid management and             decisions.
                                   and inputs identified at Level !? (C and G)                                    coordination, from all relevant
                                   To what extent have the governance arrangements for FTI (a)met relevant        phases of FTI evolution and
                                   criteria of legitimacy, representativeness, efficiency, effectiveness, etc;    implementation.
                                      (b) affected FTI's influence, positively or negatively? (C and G)




file: FTI_Evaluation_Framework(Jan2009b).doc                                                                                                                     Page 89 of 134
                                                 Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative

      Main Evaluation                            Information Required (detailed questions)                              Sources of evidence                 External factors/
       Hypotheses                                                                                                                                            Assumptions
                                                                Level THREE: INTERMEDIATE OUTCOMES
                                                         (Changes in sector policy, expenditure and service delivery)
3.1. EDUCATION POLICY AND PLANNING
OVERALL HYPOTHESIS: Implementation of improved sector policies and plans, with support from all key stakeholders
a. Implementation of national      What overall difference has FTI made to national strategies as reflected in,    Documentation:                      PRSP process operational
    plans, including PRSP, that    e.g. PRSPs? (G and C)                                                           GMRs etc                            Strong MoE participation in
    reflect greater priority for   To what extent have strengthened sector policies and plans been                 Education policy and planning        the PRSP process
    education sector               implemented? What have been the limits and constraints on                       documents over time                 Strong horizontal planning
b. Implementation of               implementation? (C)                                                             MOUs between donor groups            mechanisms with other key
    comprehensive education        How has the prioritisation of EFA and UPC affected the balance within the       and governments                      government ministries
    sector policies ands plans     education sector and across sectors? (G and C)                                  FTI status reports, and research    Strong, independent media
    designed to achieve UPC        Is the education sector plan operational? Are policy commitments being          Global Monitoring Reports           Fora for public participation
c. Implemented plans include       implemented? are budgetary commitments being funded? (C)                        Academic literature (CREATE,         in education at all levels in
    clear measures to address      To what extent are the activities of all stakeholders (including donor          especially)                          place
    issues such as HIV&AIDS,       partners) aligned with the government plan? (C)                                 Reports of EPDF Events and          Systems for government
    gender inequity, geographical  Is there increased public understanding of education policy? (C)                activities                           accountability to Parliament
    disparities                    What is the legal framework within which FTI works? Are the legal and                                                in place and functional
d. All key stakeholders engaged    education policies aligned or contradictory? C(C)                               Interviews:                         Donor agendas reflect
    with and support national      Is there parliamentary oversight at all? Or oversight by other ministries? (C)) Relevant parliamentary               priority for basic/primary
    policies and plans for UPC.    To what extent can changes at this level be attributed to FTI inputs and        committees                           education
e. Mechanisms established for      activities? (G and C)                                                           Ministers and Senior MoE            Donor agencies able to
    policy monitoring and review,                                                                                  officials over time                  flexibly respond to changing
    including indicators on cross-                                                                                 Donor representatives                priorities, including in terms
                                                                                                                   Other stakeholders                   of budget allocations
    cutting issues
                                                                                                                   Media representatives               Strong FTI governance
                                                                                                                   FTI SC members                       mechanisms.

                                                                                                                    Inventory:
                                                                                                                    Compile an inventory of policy
                                                                                                                    and planning document which
                                                                                                                    show change over time




Page 90 of 134                                                                                                                                                     printed: 04-Feb-09
                                                        Annex E: Extended Logical Framework for the Evaluation


      Main Evaluation                           Information Required (detailed questions)                           Sources of evidence                    External factors/
       Hypotheses                                                                                                                                           Assumptions
3.2. EDUCATION FINANCE
OVERALL HYPOTHESIS: Domestic and external funding allocations to the education sector increased and better aligned with policy priorities
a. Costed ESP reflecting           better policy alignment..                                                     Documentation:                       Government adopts targets
    priorities, including cross-       Is there an ESP with a credible FF available? What specific financial     ESP, FF & monitoring &                for education spending and
    cutting issues, and sources of     provisions are included to deal with major challenges affecting the       implementation framework              is held accountable on
    funding – available and            education sector such as HIV and AIDS, conflict, gender inequality and    Education budgets                     progress
    endorsed by all stakeholders       reaching disadvantaged groups? (C)                                        End-of-year financial reports        MoE shows leadership and
b. Domestic and external               What parts of government finalised and approved the ESP? Did the Ministry Donor financial reports               commitment in working
    funding allocations for            of Finance approve the FF? Which donor partners endorsed the ESP and FF? Studies on education finance and       with other government
    education increased at             (C)                                                                       planning (e.g. PERs)                  ministries
    country and global levels      increased and more appropriate funding                                        Education sector review              Donors held accountable for
c. Domestic and external               Is a comprehensive annual financial report available for the education    documents                             progress on harmonisation
    funding allocations shifted to     sector? Has the quality of financial reporting improved? (C and G)        FTI CF/EPDF financial reports         and alignment
    reflect priorities in ESP          Have domestic and external funding allocations for education increased?   Global education financial           Donors adopt multi-year
d. External funding modalities         Have allocations increased beyond what would have been expected from      datasets (UIS, DAC)                   commitments
    increasingly harmonised and        past trends? What role has the CF played in any increase? (C and G)       Studies on use of country systems    Development budgets of
    aligned                            Have patterns of domestic and external funding allocations changed? Are                                         donor governments
e. Funds allocated and                 shifts consistent with ESP priorities?C (C and G)                         Interviews:                           adjusted to reflect priority
    disbursed efficiently by the       Are increasing amounts of external funding being allocated on-budget? (C  MoE                                   for basic education
    CF/EPDF, if the country has        and G)                                                                    Ministry of Finance                  Economic growth
    met the relevant criteria          Has the duration of external funding commitments to education increased?  Ministry of Planning                 CF made available in
f. CF/EPDF grants allocated as         (C and G)                                                                 LDG/LEG                               accordance with identified
    per needs identified at            To what extent have donor partners been using common arrangements and Key non-government                        needs and in timely manner
    country level                      flexible means of financial support? (C and G)                            stakeholders e.g. civil society,     Country and sector level
g. Annual financial monitoring         What was the process for application and approval of the CF like? Were    others involved in the LEG            financial management,
    reports available and of good      there any significant delays? Why? C(C)                                   FTI secretariat                       reporting and accountability
    quality                                                                                                      CF/EPDF fund managers                 strengthened
                                       Are FTI processes (endorsement, processing of CF requests, triggering the
                                       flow of funds) consistent across countries? (C)
                                       Has the FTI CF and EPDF been distributed according to its criteria? How
                                       equitably has it been distributed? (G)
                                       To what extent can changes at this level be attributed to FTI inputs and
                                       activities? (G and C)




file: FTI_Evaluation_Framework(Jan2009b).doc                                                                                                                      Page 91 of 134
                                               Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative

      Main Evaluation                          Information Required (detailed questions)                         Sources of evidence                  External factors/
       Hypotheses                                                                                                                                      Assumptions
3.3. DATA AND M&E
OVERALL HYPOTHESIS - Reports of education sector progress against agreed upon indicators inform decision making and funding allocations to the sector
a. Improved understanding of       Has there been a qualitative improvement in data collection and            Documents                         External institutions/
    areas of progress and of gaps  dissemination? If yes, in what way? (G and C)                              Review of EMIS systems based on    stakeholders (e.g. private
    that need to be addressed to   If implemented, are indicators of quality of data systems monitored        published articles (G and C)       institutions) effectively
    achieve EFA, including on      regularly? (G and C)                                                       Review of a sample of JARs from    involved in sectoral
    cross-cutting issues affecting Is data more widely available and more regularly distributed? (G and C)I   FTI and non-FTI countries (G and   coordination efforts and
    the sector                     s there any evidence of use of data for management at sub-national levels? C)                                 willing to share data
b. Key intermediate indicators     (G and C)                                                                  Search for published or grey      Management systems in the
    and targets for measuring      What evidence is there for increased capacity of data analysis at various  literature on sub-national         sector link data generation
    progress on EFA and UPE        levels of system (G and C)                                                 analyses and classification by     and analysis to decision
    agreed upon among              What is coverage of non-formal education, private sector data, etc? (G and contributing authors (G and C)     making at key levels of the
    stakeholders, including        C)                                                                                                            system
    indicators and targets on key  To what extent can changes at this level be attributed to FTI inputs and   Interviews                        Human resources in place,
    cross-cutting issues           activities? (G and C)                                                      Directors of Planning (C)          trained, and motivated at all
c. Improved frequency and                                                                                     Main Providers(s) of Non-State     levels of the education
    quality of reporting on                                                                                   Education (C)                      system
    progress towards EFA and                                                                                                                    Fora for consultation and
    UPC                                                                                                                                          decision making among
d. Improved capacity for data                                                                                                                    external stakeholders in
    use and management at                                                                                                                        place and functional
    various levels of education                                                                                                                 FTI Governance
    systems reflected in quality                                                                                                                 arrangements favour
    reporting, and flow of                                                                                                                       accountability and decision
    information.                                                                                                                                 making
                                                                                                                                                Political and cultural
                                                                                                                                                 obstacles to evidence based
                                                                                                                                                 decision making are
                                                                                                                                                 overcome.




Page 92 of 134                                                                                                                                              printed: 04-Feb-09
                                                           Annex E: Extended Logical Framework for the Evaluation


      Main Evaluation                             Information Required (detailed questions)                              Sources of evidence                   External factors/
       Hypotheses                                                                                                                                               Assumptions
3.4. CAPACITY DEVELOPMENT
OVERALL HYPOTHESIS - Key capacity gaps in the sector addressed ensuring good quality policy, planning, budgeting, implementation and reporting in priority areas of the
education plan
a. Implementation of capacity           What evidence is there that capacity in the sector overall has improved?         Documentation:                     Donor agendas prioritize
    and institutional                   And in what areas? (C)                                                           EPDF and other donor capacity       capacity development of
    development plan outlining          Is a CD plan being implemented? How and by whom is implementation                development evaluations             partner countries and
    priorities for each sub-sector      being monitored? (C)                                                             ‘Project’ documents on capacity     sectors
    and for different levels of the     Who decides what activities/modalities and levels of CD will be carried out? development and reviews                Donor funding sufficiently
    education plan                      Have any evaluations of CD activities been carried out which demonstrate                                             flexible to allow for
b. Capacity development plan            their effectiveness? (C)                                                         Interviews:                         adjustments based on
    linked to UPC is                    Are funding agency CD activities better aligned with country CD plans? (C)       Key stakeholders, including         specific capacity needs
    implemented and monitored           Have there been improvements in terms of capacity development                    donors and lead donor              Stakeholders periodically
c. Mechanisms in place to               approaches over time? (G and C)                                                  Task managers                       held accountable for
    assess effectiveness of             Has the EPDF helped address key capacity gaps? What use has been made of         Universities and other training     progress on capacity
    capacity development efforts        EPDF compared to FTI core funding for capacity building? (C and G)               institutions                        development
    in terms of quantitative and        What changes are evident pre- and post- FTI/EPDF in terms of priority            Private training providers         EPDF funding made
    qualitative gains for the           setting, resource allocation and monitoring of CD in the sector? (C)             WB regional education managers      available in accordance with
    system                              Has it resulted in funding agencies better aligning their activities with the CD TA/hired consultants for capacity   priority needs and in a
                                        plan over time? (C)                                                              development                         timely manner
                                        What changes have there been among the donor community in terms of                                                  Government staffing and
                                        professional capacity as well as staffing? (G and C)                                                                 human resource
                                                                                                                                                             development plans in place
                                        Has policy around CD evolved? In what ways? Is this reflected in assessment
                                                                                                                                                             and in line with key
                                        and evaluation? (G and C)
                                                                                                                                                             priorities of the education
                                        Are CD plans being periodically assessed and updated? (C)
                                                                                                                                                             sector plan
                                        Is there better accountability on the impact of CD? (C and G)
                                                                                                                                                            Government civil service
                                    To what extent can changes at this level be attributed to FTI inputs and
                                                                                                                                                             mechanisms address factors
                                    activities? (C and G)
                                                                                                                                                             which impact negatively on
                                                                                                                                                             staff retention




file: FTI_Evaluation_Framework(Jan2009b).doc                                                                                                                           Page 93 of 134
                                               Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative

      Main Evaluation                          Information Required (detailed questions)                          Sources of evidence                 External factors/
       Hypotheses                                                                                                                                      Assumptions
3.5. AID EFFECTIVENESS
OVERALL HYPOTHESIS – Improved predictability of aid flows to the sector and greater accountability of external stakeholders on their commitments to promoting UPC
a. Qualitative improvement in     Is there greater predictability of funding with increasing volumes of funds Documents                           Country and leadership on
    dialogue around education at  released in accordance with agreed schedules and as part of annual and      Literature related to Rome and       donor coordination strong
    country and global levels and multi-annual arrangements? (C and G)                                        Paris Declarations; monitoring of   Donor HQ, regional and
    greater awareness of          Is there a growing amount of funding allocated and channelled through       the implementation of the Paris      country levels interact and
    constraints which need to be  common funding mechanisms? (C and G)                                        Declaration                          collaborate effectively to
    addressed                     Is there increased use of country systems for aid delivery? (C and G)       Studies on use of country            address bottlenecks to aid
b. External funding modalities    Are there indicators of improved accountability of funding to education and systems                              effectiveness
    increasingly harmonised and   of progress towards aid effectiveness in the sector? (C and G)              Literature on SWAps and budget      Donors held accountable for
    aligned                       Is there more effective dialogue among stakeholders in the sector, with     support (with special reference to   progress on coordination,
c. Growing amount of money        respect to EFA and UPC targets and generally? (C and G)                     education sector)                    harmonisation and
    allocated and channelled      To what extent can changes at this level be attributed to FTI inputs and    Donors' general and country          alignment at country and
    through common funding        activities? (C and G)                                                       reports and studies                  global levels
    mechanisms; reduction in      To what extent have the governance arrangements for FTI (a) met relevant    Comparison with other GRPPs         Mechanisms for reviewing
    parallel programmes           criteria of legitimacy, representativeness, efficiency, effectiveness, etc;                                      progress globally and at
d. Greater predictability of      (b) affected FTI's influence, positively or negatively? (C and G)           Interviews                           country level in place and
    funding with increasing                                                                                   Participants in global and country   effective in making
    volumes of funds released                                                                                 level aid management and             decisions.
e. Improved accountability of                                                                                 coordination, from all relevant
    funding to education and of                                                                               phases of FTI evolution and
    progress towards aid                                                                                      implementation.
    effectiveness in the sector




Page 94 of 134                                                                                                                                              printed: 04-Feb-09
                                                         Annex E: Extended Logical Framework for the Evaluation


      Main Evaluation                           Information Required (detailed questions)                            Sources of evidence                  External factors/
       Hypotheses                                                                                                                                          Assumptions
                                                                          Level FOUR: OUTCOMES
                                           (Effects on quantity, access, quality and sustainability of basic education)
4.A.    EFFECTS ON THE QUANTITY OF BASIC EDUCATION PROVIDED                               Note: from here on streams of effects at previous levels are expected to converge.
OVERALL HYPOTHESIS – FTI has helped to expand the provision of basic                      But different workstreams are still looking at characteristically different evidence; it
education.                                                                                should be fairly evident which workstream will be the principal source of evidence for
                                                                                          each question.
a. The provision of basic      Trends in numbers and proportion of children enrolled in and completing
   education services has      primary education. (G and C)
   increased.                                                                                                    Same documentary and interview
                               Expenditure on basic education, in total and as percentage of all education
b. FTI has contributed to this                                                                                   sources of evidence as at previous
                               expenditure and of all public expenditure. (G and C)
   increase in provision.                                                                                        levels.
                               Other indicators of inputs and service provision (teachers, teacher training,
                                                                                                                 Particular role for interviews in
                               text books and learning materials, physical facilities) (C and G)
                                                                                                                 helping to assess extent to which
                               Contribution of external aid to the financing of public expenditure. (G and C)
                                                                                                                 FT has contributed to results.
                               Direct contributions of FTI to expenditure on UPC. (G and C)
                               Indirect influences of FTI on patterns on domestic and external expenditure
                               on basic education (G and C)
4.B.    EFFECTS ON ACCESS TO BASIC EDUCATION
OVERALL HYPOTHESIS – FTI has helped to expand access to basic education.
a. There is greater access to  Overall enrolment rates at country level. (C)                                                                          Demand as well as supply
   basic education.            Enrolment and completion rates by gender, geographical , social and ethnic        Same documentary and interview          for basic education
b. FTI has helped to ensure    groups within the country. (C)                                                    sources of evidence as at previous
   greater access to basic     Global patterns of enrolment and completion. (G)Direct contributions of FTI levels.
   education.                  in expanding access to primary education. (G and C)                               Particular role for interviews in
                               Indirect influences of FTI on policies and strategies that have expanded          helping to assess extent to which
                               basic education (G and C)                                                         FTII has contributed to results.




file: FTI_Evaluation_Framework(Jan2009b).doc                                                                                                                      Page 95 of 134
                                                Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative

      Main Evaluation                           Information Required (detailed questions)                          Sources of evidence                   External factors/
       Hypotheses                                                                                                                                         Assumptions
4.C.     EFFECTS ON THE QUALITY OF BASIC EDUCATION
    OVERALL HYPOTHESIS – FTI has helped to maintain and improve the quality of basic education
a. Basic education is of         Data on proxies of education quality (pupil/teacher ratios, teacher                                                 
   sufficient quality to provide qualifications, completion rates etc (UPC) (G and C)                           Same documentary and interview
   lasting benefits to those     Data on trends in learning outcomes. (G and C)                                 sources of evidence as at previous
   completing the primary cycle. Trends in continuation to higher levels of education. (G and C)Direct          levels.
b. FTI has had a positive        contributions of FTI in maintaining and improving the quality of basic         Particular role for interviews in
   influence on the quality of   education. (G and C)                                                           helping to assess extent to which
   basic education.              Indirect influences of FTI on policies and strategies that have maintained     FTII has contributed to results.
                                 and improved the quality of basic education. (G and C)

4.D.    EFFECTS ON THE SUSTAINABILITY OF BASIC EDUCATION
    OVERALL HYPOTHESIS – FTI has helped to secure improvements in provision, access and quality of basic education which are sustainable.
a. Improvements in provision ,. Aggregate and unit costs of basic education (G and C)
   access and quality of basic  Balance between education sub-sectors, and between education and other
   education are sustainable.   sectors (G and C)
b. FTI has helped to ensure the Prospects for domestic financing of basic education (future trajectory of
   sustainability of these                                                                                                                        population growth
                                costs vs. domestic revenues)? (taking account of costs of provision for "hard
   improvements.                                                                                              Same documentary and interview  economic growth
                                to reach" children) (C and G)
                                                                                                              sources of evidence as at previous  sustained demand for
                                Is there adequate capacity to sustain and improve primary education
                                                                                                              levels.                              primary education (in terms
                                provision? (C)
                                                                                                              Particular role for interviews in    of pupils claiming places, in
                                Are there adequate data and systems to monitor education performance e
                                                                                                              helping to assess extent to which    terms of political support
                                and adapt policy and expenditures in light of experience? (C)
                                                                                                              FTII has contributed to results.     for the prioritisation of UPC)
                                Are there effective mechanisms to maintain relevance, adequacy and                                                aid commitments up to and
                                effectiveness of aid to the sector? (C and G)Direct contributions of FTI                                           beyond 2015
                                towards ensuring the sustainability of progress towards UPC. (C and G)
                                Indirect influences of FTI on policies and strategies that have ensured the
                                sustainability of progress towards UPC. (G and C)




Page 96 of 134                                                                                                                                                 printed: 04-Feb-09
                                                       Annex E: Extended Logical Framework for the Evaluation


      Main Evaluation                          Information Required (detailed questions)                         Sources of evidence                  External factors/
       Hypotheses                                                                                                                                      Assumptions
                                                                          Level FIVE – IMPACT
                                               (The long term benefits derived from more children completing primary education)
NOTE: The mid-term evaluation of FTI will not attempt to assess the effects of FTI at this level. However, a scoping study will assess the prospects, and make proposals, for
monitoring and evaluating FTI impacts over the long term.
            enhanced learning, life skills and opportunities for individuals
            stronger local and national institutions
            personal and social benefits in education and other sectors (including health)
            economic growth due to increased human capital




file: FTI_Evaluation_Framework(Jan2009b).doc                                                                                                                 Page 97 of 134
                 Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative




Page 98 of 134                                                                            printed: 04-Feb-09
           Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative



Annex F: Working Paper Topics
Overview
1.1      The Working Papers (WPs) to be prepared during Stage 1 echo the main
workstreams that will run through the evaluation. The Working Papers will not be formal
outputs of the evaluation, but, as drafts, they will facilitate discussion, consultation and
feedback within and beyond the team, while also feeding directly into the preparation of the
Preliminary Report. They will be a key input to the second Team Workshop, along with the
draft findings from the pilot country studies.

1.2    Papers corresponding to each of the main workstreams will be complemented by
papers on two special topics: issues arising from fragile states and fragile partnerships (see
WP6 below), and a scoping paper on impact evaluation (see Annex G).

Working Paper 1: FTI Influence on Education Policy and Planning
1.3    Topics to be addressed will include:
           o   balance between various EFA goals and MDGs
           o   issues in quality and expansion, including the nature of trade-offs
           o   scope and influence of the FTI basic indicator approach, and its strengths and
               weaknesses
           o   incentive issues at country level and now they affect education planning and
               policy processes (e.g. in the different perspectives of education and finance
               ministries, and different administrative levels of the education system)
           o   policy, planning and implementation for basic education in the wider context
               of sector and national planning approaches (e.g. SWAps, PRSPs)
           o   appropriate time horizons for planning, and policy and planning capacity
               development workstream
           o   the degree of integration of CF processes and activities with overall sector
               planning etc.

Working Paper 2: Financing EFA and Issues in PFM
1.4    Topics to be addressed will include:
       Aggregate analysis of FTI finance (in context of overall financing for education)
          trends in level of pledges and disbursements to FTI by donor;
          trends in aid flows for education by donor, country, and education subsector;
          comparative context of EFA financial gap estimates and other political pledges to
          education;
          distribution of FTI commitments and disbursements across countries (equity
          analysis);
          analysis of reasons for apparent inequities (particularly institutional and political
          context); and
          operational efficiency of disbursement procedures overall and by country.
       Country-level analysis of FTI finance (in context of overall financing for
       education, and with special focus on endorsed countries)




file: FTI_Evaluation_Framework(Jan2009b).doc                                   Page 99 of 134
           Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


           trends in overall public budget by sector, by source of financing (domestic or
           external with particular attention to aid modalities);
           trends in share of education budgets allocated to primary education;
           trends in domestic financing for education (in many countries this will be
           complicated by additionality issues, depending on aid-modalities);
           trends in external financing for education by aid-modality (again in many
           countries this will be complicated by additionality issues);
           financial sustainability of the education budget;
           FTI in the context of financial frameworks underpinning sector plans (financial
           gaps, FTI indicative framework); [probably mainly case study work]
           Links between sector plan financial frameworks, education budgets, releases and
           spending patterns (during the case studies, existing survey data may help to
           shed light on the beneficiary receipt of funds and financial equity issues);
           [probably mainly case study work]
           FTI financial management processes at country level and how they fit into public
           sector financial management processes (particularly education budget cycle
           processes), including issues of predictability, timeliness [probably mainly study
           work]
1.5   This workstream will be carefully coordinated with the workstreams on
data/monitoring and on aid effectiveness. Financial and other analysis of the EPDF will be
come under the capacity development workstream.

1.6     The working structure of the paper is shown below. The initial focus will be on
global-level issues.

WP2: Finance and PFM Issues for the FTI

       1       Introduction
               1.1    Background on FTI and CF
               1.2    Objective of this working paper
               1.3    Methodology for this working paper
       2       Education financing at global level
               2.1    Volume of education finance
               2.2    Distribution of education finance between FTI eligible countries
               2.3    Education financing processes
               2.4    FTI contribution to level and process changes in education financing
       3       Education financing at country level
               3.1    Volume of education finance
               3.2    Equity in education financing
               3.3    Future education financing
               3.4    Education sector budget process
               3.5    FTI contribution level and process changes in education financing
       4       Conclusions and issues arising




Page 100 of 134                                                                   printed: 04-Feb-09
                                Annex F:Working Paper Topics


Working Paper 3: Data Analysis and Monitoring for the FTI
1.7     The FTI has as one of its objective to tackle the data gap usually described in terms
of the lack of coverage of sub-sectors such as non-formal education, although it is also
recognised that there are several weaknesses in the administrative data reported for the
mainstream formal sub-sector.

1.8    This paper will be in three parts: a current data sub-stream, a monitoring sub-stream
and a data use and analysis sub-stream

Current Data Sub-Stream:
1.9    This will include:
       (a) A review of the current state of knowledge about progress towards all the EFA
           goals, with a specific focus on primary education, drawing on GMR and any
           additional data immediately available from UIS and the international household
           surveys.
       (b) A discussion of the appropriateness of the quantitative educational indicators of
           the indicative framework and in particular the UPC indicator as part of a
           performance based framework. For example, do they correspond correctly to
           policy goals? Is there any danger that they may have generated perverse
           incentives?
       (c) An examination of the availability and quality of data from 2002 to date for those
           indicators with a specific focus on UPC.

Monitoring Sub-Stream
1.10 The review in the previous sub-stream will have highlighted some of the weaknesses
in data at global and country levels. The focus in this section will be on:
       (a) The existing administrative monitoring systems and efforts and possibilities for
           addressing those data weaknesses; and in particular on the development of
           EMIS systems from the end of the 1990 to present.
       (b) The use of household surveys for monitoring educational levels achieved, and of
           the use of assessment surveys for monitoring educational outcomes
       (c) A review of FTI efforts to address the data gap;
       (d) Development of guidelines for investigating this issue in the country case studies.

Data Use and Analysis Sub-stream
1.11 It is well known that, without use and analysis, little attention is paid to the quality of
data entered into the database system so that the data „decays‟. It is therefore important to
ensure the institutionalisation of the use and analysis of data. This section will therefore
review:
       (a) approaches to use and analysis of data at global and national levels, both in
           terms of actual practice of the FTI secretariat or other global bodies or by
           national entities, and in terms of what could be done with the data
       (b) the extent to which available data actually is used to inform management and
           planning at sun-national levels in countries




file: FTI_Evaluation_Framework(Jan2009b).doc                                    Page 101 of 134
            Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


1.12 In addition, this section will review FTI's own data management and M&E strategy.
This work will be closely coordinated with work on the education financing Working Paper,
and the impact evaluation scoping study (see Annex G) will form part of the same
workstream.

Working Paper 4: Capacity Development (CD)
1.13 "Capacity" is one of the four gaps that the FTI attempts to tackle. The EPDF is a
discrete FTI instrument, and will be thoroughly reviewed under this workstream. The review
will go beyond relevance and efficiency issues, to include questions of effectiveness,
selection of CD areas, modalities, and changes in the EPDF's governance.

1.14 However, FTI's influence on capacity development is much broader than the EPDF,
and the evaluation team will adopt a broad perspective in addressing capacity development
issues. The unpacking of „capacity development‟ into the three constituent parts, the
individual, the organisation, and the institutional framework, effectively undermines the view
that donor agencies can merely provide technical, gap-filling solutions to recipient countries‟
capacity development requirements. Thus, the questions raised in the evaluation of FTI will
be informed by this three-pronged approach that builds additionally on OECD guidance, that
“Capacity building would be ineffective so long as it was not part of an endogenous process
of change, getting its main impulse from within.”17

1.15 In addition to a thorough preliminary review of the EPDF, this paper will focus on the
treatment of CD issues in the country studies.18 The country studies should illuminate the
sustainability of capacity development advances, given the choices made, and unpackage
them. The studies will require technical initiatives to be understood in the context of a
political economy analysis of stakeholder interests and how these affect the education sector
and FTI in particular.

1.16 A review of the recent literature19 points to the following factors related to successful
capacity development. This checklist (to be refined and elaborated in the working paper) will
be used as a point of reference for the investigation of FTI‟s contribution to improved and
sustained institutional capacity in partner countries.
            Coordinated programmes aligned (cf. Paris Declaration Indicator 4 a)
            Reduction in PIUs (Paris Declaration Indicator 6)
            3-pronged approach: individual, organisation, and institutional framework
            (enabling conditions)
            Endogenous – donor-supported, not donor-driven
            Pooled finance?
            CD as core goal, not collateral objective
            TA strategy – reduce fragmentation, synchronise individual, organisational, and
            institutional changes
            Untied aid
            International TA, when no national TA, managed nationally, including phase-out
            Best fit, not best practice – contextualise

17
   Organisation for Economic Cooperation and Development, The Challenge of Capacity Development: Working
Towards Good Practice, OECD, Paris, 2006, p.15.
18
   And it is expected that the workstream leader will participate in one of the pilot studies.
19
   Background case studies of Guyana and Bangladesh carried out for the forthcoming UNESCO Policy Paper on
Capacity Development, IIEP, Paris.


Page 102 of 134                                                                        printed: 04-Feb-09
                                Annex F:Working Paper Topics


           Political/socio-economic and technical/functional analyses
           Incentives of recipients and providers
           Demand-led, transparency, accountability
           Commitment and leadership of recipient country
           Professional development plan/strategy
           Integrate sectoral with overall public service reform
           Include diaspora in CD planning
           High technical calibre and inter-personal skills of TA
           Responsiveness to, and under direct management control of recipient
           Quality assurance overseer/honest broker.


Working Paper 5: FTI Governance and Aid Effectiveness
1.17 FTI governance has been an area of continuing concern and innovation throughout
the existence of the FTI. The Oslo meetings in December 2008 were seen by many
participants as a watershed in defining a revised governance framework going forward. FTI
governance will be treated as a separate sub-topic within this workstream, and the working
paper will carefully chart the evolution of FTI governance at both global and country levels,
as a basis for identifying the main issues for retrospective assessment and possible
recommendations going forward.

1.18   The aid effectiveness strand of this Working Paper will:
           Set the FTI within the context of aid effectiveness debates and assumptions
           during the FTI's conception and subsequently.
           Document the aid effectiveness objectives and intentions of the FTI (noting
           possibly different perspectives of different actors), and spell out the ways in which
           the FTI agenda was distinctive (not least by comparison with other global aid
           programmes).
           Discuss practical criteria for assessing FTI's possible contributions to greater aid
           effectiveness and assess the availability of robust evidence at global and country
           levels.



Working Paper 6: FTI and Fragile States and Fragile Partnerships
1.19 Promoting UPC in fragile states is already an important topic for FTI. An FTI working
group has most recently prepared a draft "Progressive Framework" for FTI support to fragile
states. This will be a continuing concern, and an important issue for this evaluation's
forward-looking recommendations.
1.20 However, fragility is not static; previously stable states may experience turmoil (e.g.
Kenya), and events can occur in strong states that make partnership fragile even while the
state remains effective (Ethiopia). This paper will take stock of FTI's work on fragile states
and its emerging approach. It will relate this to broader aid management issues when
partnerships are fragile, and, in particular it will highlight how this issue can best be
addressed in the country studies. WP6 will be supported by a broader literature review of
how aid agencies have addressed issues of "fragility", both generally and in relation to the
education sector. Outlines of the main paper and the literature review are shown below.




file: FTI_Evaluation_Framework(Jan2009b).doc                                   Page 103 of 134
            Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative



FTI and Fragile States and Fragile Partnerships
       1.       Introduction
       2.       FTI engagement with Fragile States
                2.1.    Emergence of the „fragile states‟ issue in the context around the FTI
                2.2.    The Fast Track Initiative and Fragile States
                2.3.    Fragile States/fragility in other FTI workstreams
                2.4.    FTI and individual fragile states
       3.       Issues facing FTI and Fragile States
                3.1.    Issues for the compact at the global level
                3.2.    Country level issues

FTI and Fragile States and Fragile Partnerships: Background Literature Review
       1.       Introduction
       2.       Definition of Fragile States and/or fragility
                2.1.    Definition of the „population‟ of fragile states
                2.2.    Categories of fragile states
                2.3.    Recent evolution in thinking about fragile states
                2.4.    Moving out of fragility
       3.       Donor engagement and aid effectiveness in fragile states
                3.1.    Donor engagement in fragile states
                3.2.    Frameworks, categories and „business models'
                3.3.    Relevance of the Paris Declaration and the PRSP-based approach
                3.4.    State-building, state‟s functions, and measurement of progress
                3.5.    Allocating aid to Fragile States
                3.6.    Aid approaches and instruments in Fragile States
       4.       Education and fragility
                4.1.    Importance of education with regard to fragility and resilience
                4.2.    Opportunities and challenges for education reform in fragility
       5.       Donor engagement in education in fragile states
                5.1.    Rationale for donor engagement in education in fragile states
                5.2.    Approaches and instruments to support education in fragile contexts
                5.3.    Funding education in fragile contexts




Page 104 of 134                                                                      printed: 04-Feb-09
           Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


Annex G: Impact Evaluation Scoping Study
These are the Terms of Reference for a paper commissioned from Howard White and the
3ie organisation.


Background and Time Schedule
1.      A consortium of Cambridge Education, Mokoro and Oxford Policy Management is
undertaking the Mid-Term Evaluation of the EFA Fast Track Initiative (FTI). The overall TOR
for the evaluation include specific requirements on the subject of impact evaluation:
        Additionally, the evaluation will develop a comprehensive framework for monitoring
        and evaluation of the FTI, building on the evaluation framework created for this
        evaluation, which will help frame future monitoring and evaluation efforts, including
        the assessment of FTI impact. The comprehensive framework should indicate
        baseline data needed for the evaluation of FTI impact, much of which will be
        collected during this evaluation. (TOR, ¶15)

2.     The overall approach and methodology for the evaluation is set out in a revised
Evaluation Framework, which provides the background for the scoping study described in
the present TOR. The scoping study will be closely coordinated with the work of the Data
and M&E Workstream of the Evaluation.

3.     The scoping study will contribute to the first stage of the evaluation, and feed into its
Preliminary Report. This means that:
           A first draft of the Scoping Study should be delivered no later than 23 January
           2009.
           Its principal author should attend the Evaluation Team's workshop in Cambridge
           on 13 February 2009
           The final draft of the Scoping Study, incorporating responses to comments on the
           first draft, should be delivered by 20 February 2009.

Proposed Scoping Paper
4.     Impact evaluation of FTI could proceed in two possible ways.
          One would be a cross-country analysis, comparing progress on FTI-indicators in
          FTI countries and non FTI countries. However, such an approach would face
          many deep conceptual problems, and would be unlikely to pass current external
          peer review standards. The possibilities and limits of such an approach will be
          reviewed separately under the Data and M&E workstream. They do not form part
          of the proposed scoping study, but the scoping study author will review and
          comment on this work.
          The second approach to analyzing the impact of FTI, deep country case studies,
          is a far more promising approach. The scoping paper will set out the possibilities
          of such an approach, set out a template for conducting such evaluations, and
          identify a selection of countries where the prospects for such evaluations are
          most promising. It will also spell out the implications of such work for the overall
          M&E strategy of the FTI (e.g. the requirements for establishing baseline data and
          adapting future surveys).

5.      The country-level impact studies would adopt a theory-based approach which traces
effects of the FTI through from inputs to educational outcomes in FTI countries. Each



file: FTI_Evaluation_Framework(Jan2009b).doc                                   Page 105 of 134
           Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative

country case study would therefore be heavily contextualized, taking account of the political
economy of education programs in developing countries as a means of identifying the role of
the FTI in changing policies and programs, including resourcing levels. Having identified the
changes in education policies and outputs sought by the FTI, the studies would model their
impacts on outcomes, allowing for possible endogeneity in program placement. The
programme theory embodied in the Evaluation Framework will be the starting point in
identifying elements that are susceptible to rigorous impact evaluation.

6.     The scoping study report will:
          Briefly review existing impact studies of education programs in developing
          countries, assessing both knowledge and the state of the art regarding impact
          evaluation approaches. This review will inform the modelling strategy for the
          impact studies.
          Review existing data sources in FTI countries (administrative data and household
          surveys), to assess the scope for impact studies in each country. Surveys which
          link household and school facility data are of particular importance. Information
          will be collected on planned surveys, including possible piggy-backing to enhance
          data collection. (This work will be coordinated with the Mid-Term Evaluation's
          work under the Data and M&E workstream.)
          Explain the feasible approaches to impact evaluation for the FTI, setting out the
          advantages and disadvantages of the various alternatives, and present a detailed
          concept note for a single country case study, including an estimated budget for
          implementing the study.
          Propose a short-list of 8-10 countries for the impact studies, in the expectation
          that eventual deep country case studies would be done in 5-6 of these countries.

7.       The report will consist of a concise main text supported by more detailed annexes. It
will include an Executive Summary.

Oversight and Coordination
8.     The Team Leader for the Mid-Term Evaluation of the FTI is Stephen Lister. He will
oversee the study and any variation in the proposed work will require his approval. The
consultant should also liaise with Roy Carr-Hill as leader for the Data and M&E workstream,
Georgina Rawle (workstream leader for Education Financing), Luca Pellerano (impact
evaluation specialist) and Anthea Sims Williams (research coordinator). Study outputs will
also be reviewed by the evaluation team's quality assurance panel and the Evaluation
Oversight Committee.

Deliverables
9.     Deliverables for this study are:
           Comments on the draft Workstream Paper on Data and M&E (late January 2009).
           Draft Scoping Paper on Impact Evaluation for the FTI (23 January 2009).20
           Participation in the evaluation team workshop on 13 February 2009.
           Final draft Scoping Paper on Impact Evaluation (20 February 2009).




20
  Members of the evaluation team will also provide informal comments on a working draft of the
scoping paper as soon as it is available.

Page 106 of 134                                                                   printed: 04-Feb-09
                          Annex G: Impact Evaluation Scoping Study


Follow-up
10.    Possible further inputs during Stage 2 of the overall evaluation will be negotiated
separately.




file: FTI_Evaluation_Framework(Jan2009b).doc                                  Page 107 of 134
         Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative




Page 108 of 134                                                          printed: 04-Feb-09
           Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative



Annex H: Selection of Country Cases

Rationale for Country Case Studies
1.     The TOR (¶21) specified:
       Case studies are expected to be used in this evaluation as a means of developing
       greater insight into country-level processes, accomplishments, and problems, all in
       the context of each country, thus making a contribution to the lessons-learned part of
       the evaluation. Proposals must indicate that a priority will be placed on consulting
       equally all the key actors – government, civil society, and local funding agencies.
       Unlike desk studies the case studies can reach down to sub-national levels for
       informants and site visits to validate desk review findings and examine more details
       at the service delivery level. Case studies will be particularly useful in examining: a)
       questions about FTI relevance from various points of view; b) detailed questions
       about effectiveness, such as policy adjustments and the inclusiveness of the
       process; effects of capacity building, EFA outcomes at the school and community
       levels, etc), and c) detailed questions about efficiency (such as the functioning of the
       local donors group; any real evidence of donor agency changes in aid effectiveness;
       resource mobilization, and the sustainability of systems).

2.     The consortium proposal noted:
       Country case studies are crucial to the evaluation and will dominate the evaluation's
       resources. It is not possible to draw a statistically representative sample, nor will
       aggregate pair-wise comparisons between countries (whether similar or dissimilar)
       be very revealing. As explained in our methodology, this has to be a theory-based
       evaluation in which causal relationships are carefully disaggregated and examined in
       a variety of contexts. The iterative contribution analysis approach then enables
       evaluators to make informed and transparent judgements about attribution.

3.      The challenge therefore is to select a set of countries that enables FTI processes and
effects to be studied in a variety of contexts, so as to maximise the insights from the studies
and their contribution to the robustness of the evaluation's overall conclusions.

Value of early decision
4.      The fieldwork for the main country studies is now timetabled for March–June 2009,
and will not proceed before EOC have agreed the detailed Stage 2 work plan and budget.
However, it will be valuable to identify the countries early so that preliminary data and
documentation can be assembled, strong study teams and local consultants be identified,
and begin liaison and planning with in-country stakeholders.




file: FTI_Evaluation_Framework(Jan2009b).doc                                  Page 109 of 134
           Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


Agreed Criteria for Case Study Selection
5.      From the TOR (¶22):
      Bidders will be expected to create their own country selection framework, with the
      provision that they should consider parameters such as:
               Country geographical region (e.g., Africa/Non-Africa)
               Time of entry into FTI (e.g., early v. late)
               FTI endorsed v. countries invited into FTI but not endorsed
               Countries on-track as well as off-track for achieving UPC by 2015
               Size of country
               Etc.

6.      From the Cambridge consortium proposal:
                                        Key selection factors
         Country                 FTI engagement          Education factors
         characteristics
         Region                  Endorsed, expected or       Degrees of whether on/off track
                                 eligible                    for UPC
         Population              Longevity of                EFA index
                                 relationship
         Government type         Receipt of CF               Trends in NER
         Aid dependency          Receipt of EPDF
         Stability/Fragility21



Agreed parameters
7.      It has already been agreed with the EOC:
            That the budget can accommodate 9 country case studies in total.
            It is expected that not more than 2 non-endorsed countries would be included in
            the sample.
            Different countries require different scale of evaluation effort, and cost. The
            agreed total of 9 countries was based on 2 "large", 3 "medium", and 4 "small"
            studies.

8.       The Evaluation Team also noted that the study would look closely at, and report on,
FTI experiences in a number of countries besides those selected for full country case studies
with field visits. (A relevant factor here is the availability of existing studies to draw on.)




21
  Using the OECD Development Assistance Committee's list of fragile states, 2005; but in the
knowledge that a number of countries not categorised as such exhibit degrees of fragility which will
enable the gathering of insights for evolving FTI policy in this respect

Page 110 of 134                                                                      printed: 04-Feb-09
                         Annex H: Selection of Country Case Studies


Proposal
9.     Taking all these factors into account, the evaluation team proposes to undertake the
country case studies shown in the following table:
  Scale (in terms of              First choice                        Fringe (for special
  evaluation resources)                                               desk review)
  Large [2]                       Pakistan
                                  Nigeria (focus on expected                 Ethiopia
                                  states)                                     Malawi
  Medium [3]                      Kenya (pilot),                               Mali
                                  Mozambique,                                Moldova
                                  Ghana                                      Rwanda
  Small [4]                       Cambodia (pilot)                           Uganda
                                  Yemen                                      Vietnam
                                  Burkina Faso                               Zambia
                                  Nicaragua


10.    Of these, Kenya and Cambodia will be the pilot case studies, with field work
beginning in January 2009.

11.    The list of "special desk review" countries is not definitive. As with the main case
study countries, we have sought a varied sample, while taking advantage of the evaluation
team's previous familiarity with most of those proposed. Several of them have been the
subject of relevant previous studies, and three (Rwanda, Uganda, Mali) are the subject of an
ongoing SPA study of education sector budget support (ODI and Mokoro 2008).

12.     The Evaluation Team will not attempt a full report on each country. Findings will be
based mainly on desk reviews, although these countries will be borne in mind when
conducting general interviews. Findings are likely to appear in two forms. First, a
background paper will summarise experiences of these countries (perhaps alongside the
main case studies) against a restricted set of the evaluation questions. Design of a useful
summary matrix will take account of experience with the pilot country studies. Secondly,
there may be certain episodes in these countries' experiences that add depth to the
evidence gained from the full case studies and could appear as additional illustrations in the
evaluation reports. The approach to the desk review countries will be spelt out in more detail
in the work plan for Stage 2.

13.     Liberia (which was earlier considered as a possible case study) will receive attention
in the review of fragile states issues.

14.     The final table, below, maps the first choice countries (in bold) onto the summary of
FTI status provided as Annex E of the TOR. (Annex J2 provides a detailed summary of
each country's FTI status over time.) The selected case study countries are subject to
review and confirmation as a part of the Stage 2 work plan and budget to be submitted to the
EOC in March 2008.




file: FTI_Evaluation_Framework(Jan2009b).doc                                  Page 111 of 134
            Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


        Countries with FTI Endorsed Sector Plans and Estimated Schedule of Endorsements
                        (as of December 2007) [source: Annex E of the TOR]
Countries with endorsed plans by year of        Countries expecting               Other FTI eligible
endorsement                                     endorsement                       countries
                                                    In 2008           In 2009
2002      Burkina Faso         Mauritania       Bhutan            Angola          Afghanistan
          Guinea               Nicaragua        Burundi           Bangladesh      Cote d’Ivoire
          Guyana               Niger            Central African   Comoros         India
          Honduras                              Chad              Congo, Dem      Indonesia
2003      The Gambia           Vietnam          Haiti             Rep of          Kiribati
          Mozambique           Yemen, Rep. of   Malawi            Congo, Rep of   Myanmar
2004      Ghana                Ethiopia         Papua New         Eritrea         Nepal
2005      Kenya                Moldova          Guinea            Guinea-Bissau   Nigeria (other states)
          Lesotho              Tajikistan       Uganda            Lao PDR         Pakistan
          Madagascar           Timor-Leste      Vanuatu           Nigeria (3-4    Somalia
                                                                  states)
2006      Albania              Mali             Zambia                            Sri Lanka
                                                                  Solomon
          Cambodia             Mongolia                                           Sudan
                                                                  Islands
          Cameroon             Rwanda                                             Zimbabwe
                                                                  Tanzania
          Djibouti             Senegal                            Togo
          Kyrgyz Republic                                         Tonga
2007      Benin                Sierra Leone
          Georgia              Sao Tome &
          Liberia              Principe




Page 112 of 134                                                                       printed: 04-Feb-09
           Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative



Annex I: Stakeholder Mapping
The Stakeholder Role in Evaluation
1.      The GRPP Sourcebook highlights the importance of stakeholder participation
throughout the evaluation processes. Participation in the evaluation process is expected to
add value (IEG 2007, p21). Properly done (i.e. in all of the key steps and phases of the
evaluation process) stakeholder participation will promote learning of all parties involved and
ensure that the views and concerns of all partners become part and parcel of the process.
Ultimately, participation will enhance consensus building and ownership of the evaluation
process and its results, and will contribute to embedding the evaluation in on-going work,
rather than seeing it as separate and distant from the implementation process. This is
particularly important in the case of formative evaluations such as this one.

2.      The evaluation team recognises that the substantial engagement of stakeholders at
this important mid-term point in programme will certainly benefit the quality of the processes
and results as FTI further expands and consolidates. It will also be an important precursor
for the planned implementation of new M&E procedures following the evaluation.
Stakeholders that are involved in an evaluation are more likely to respond positively and to
engage with the resultant process of change that may follow.

3.      At the same time it is recognized, that the quest for participation in evaluation of this
kind can put a heavy burden on stakeholders (Samoff, 2005). The FTI‟s philosophy is to
avoid imposing additional burdens and transaction costs on stakeholders. Therefore this
evaluation will strike a careful balance between the importance of participation and the need
to respect the reality on the ground. In order to achieve such efficiency the country study
teams will complete a literature review and write a preliminary bibliography for the country
before their country visit. They will also write a draft country background section and a brief
issues paper identifying the areas of focus for the country concerned for review and
verification in country.

4.       In line with GRPP Sourcebook guidelines, the EOC is expected to provide the
evaluation team with a “clear stakeholder map, including roles and responsibilities of those
identified” (IEG 2007, p25). However, to allow for detailed planning of the evaluation
processes, and in particular to support the development of the methodology, an initial
stakeholder map was included in the proposal, and a revised version appears as Figure I
below. (The revisions reflect comments at the team workshop in November.)

5.       This map seeks to distinguish between partners and participants. In the absence of
a definition of these terms in the Terms of Reference, this evaluation proposes to follow the
GRPP suggested definition of the term partners as “those who convene and govern the
program” i.e. all stakeholders who are members of governing, executive or advisory bodies
(IEG 2007 p25). These are indicated by the oval shapes in the tentative stakeholder map. By
default the term participant therefore concerns all other stakeholders (government, donor,
civil society) who are involved or affected by the implementation without being directly
involved in its governance. This covers a very broad range of individuals and institutions
ranging from donors who are not directly involved or contributing to FTI to parents/guardians
and pupils at community level.

6.      It is acknowledged, however, that in some cases the distinction between partner and
participant may not be sufficiently clear. A case in point is the Ministry of Education itself
which at senior level is involved in governance and is therefore a partner but at more



file: FTI_Evaluation_Framework(Jan2009b).doc                                    Page 113 of 134
           Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


decentralized levels (implementers, technical staff, working groups, schools, teachers and
pupils) deserves the label of participant. It is similarly important to be aware of differing
interests and incentives within donor agencies and other organisations. The evaluation team
will take these factors into account, and ensure that interviews cover an appropriate
spectrum within as well as across organisations.

Engaging with stakeholders throughout the evaluation
7.      For efficiency reasons, and to reduce the burden that may be associated with the
consultation process, the engagement of both partners and participants will be sought
through a variety of means, including interviews (face to face and at a distance, combined
and individual), short questionnaires, thematic discussions, feedback on sub-products (such
as the country aide memoires), e-consultations, workshops, etc.

8.      Workshops are planned at three key stages of the process (see main proposal
Box 2A). For the first (focusing on the evaluation framework), participation of the EOC itself
is key. For the subsequent workshops, at Preliminary Report and Draft Full Report stages,
direct participation by the EOC would not be appropriate, given the need to ensure the
independence of the evaluation findings. However, it is important to ensure wide feedback
on drafts and the evaluation team will discuss with the EOC how best to address this.

9.      At country level, it is anticipated that approaches will differ depending on country
specificities (including in terms of opportunities offered by on-going events on which the FTI
evaluation can piggy-back). Prior to country visits a stakeholder map will be produced and
populated specific to the country (see Box I below). In all cases the evaluation team will
make sure that stakeholders are informed ahead of the work in-country of the steps of the
processes and of the manner and moments at which stakeholders can provide critical inputs.

10.     In addition, the country aide memoires and country final reports will document in a
detailed manner how consultation was strived for, what was achieved and where there may
have been gaps, so that these can be taken into account in the interpretation of the findings
and in follow-up to the evaluation. It is important in this context to highlight the purpose of
the country aide memoires. These will be produced within 10 days after the country visit and
will contain preliminary findings and conclusions, presented in a succinct form (5-6 pages
per country). The stakeholders who were consulted during the country work will all receive a
copy and will be asked to provide frank comments and feedback. In this manner, issues of
contention or areas which need to be further developed will be pointed out at an early stage.

11.     Box I below will be used as a template for each country study team. This has been
put into table format in order to simplify completion and allow the addition of a temporal
element. The table and the map above are deliberately vague putting the onus upon the
country teams to establish what an accurate stakeholder map would look like in their country
of study. It will not be relevant to fill every box for every country; instead this matrix should
be used as a framework and altered where relevant.




Page 114 of 134                                                                 printed: 04-Feb-09
                                                                                                   Annex I: Stakeholder Mapping
                                                                                             Figure I: An Initial Stakeholder Map
                                                                                                                                        FTI
         International Level
                                                                                                                                    Partnership
         Influence: Ministers in                                                                                                      Forum                                Parliament and
         other countries, G8, OECD
                                                                                 FTI                                                                                       specialised
         DAC
                                                                              Secretariat                FTI Steering Committee                                            Committees
                                                                                                         (incl. rep. from donors,
                           External Partners                                                             countries and civil
                                                                                                                                                        Min. of Planning
                           Multilateral, bilateral, NGOs, private                                        society)
                           sector, religious organizations (in
                                                                                                                                 Min. of Finance
                           each organization and as relevant at                                                                                                       Min of Social Welfare
                           different levels – HQ and locally – as
                           well as combining education
                           specialists and general staff or staff                                                                                                                  Min. of Health
                           from other related areas such as
                           health, youth, HIV and AIDS)                                                         Ministry of Education
                                                                                                                including senior, middle level      Representatives and                     Min. of
                                                                            Representatives and                 managerial and technical            participants in other                   Labour
                                                                          participants in education             staff (Planning, Finance,           multi-sectoral
                                                                              coordination and                  Human Resources,                    coordination structures       Other Government
                                                                           implementation forum ,               Curriculum, Primary                 e.g. around PRSP,             Departments
                                                                        including the lead donor and            Education, Teacher Training         MTEF, etc.).
                                                                         FTI coordination structures,           etc)
                       Organizations/individuals                        Development Partner Group.
                       providing technical
                       support (local, regional,                                                                                                                                   Opposition political parties
                       international) to the sector,                                                                                                                               (also at regional level in
                       education coalitions                                                                                                                                        large countries).
                                                                                                                 Provincial/ Regional
                                                                                                                 Education Staff
                                                                                                                 (managerial and
                                                                                                                 technical)
                                                                                                                                                                                   Chambers of Commerce
                       Prominent local individuals                                                                                                                                 and other business
                       with a non-sector view on                                                                                                                                   associations
                       education e.g. politicians,                                                               District/ Local Education
                       local leaders, etc.                                                                       Staff (managerial and                                             Teachers
                                                                                                                 technical)                                                        Unions

                                                                                                                                                                                   Youth
                                                       Other community                                           Public schools as well                                            groups
                                                       level service                                             as community &
                                                       providers (health,                                        religious institutions                                            PLHA Groups
                                                       social welfare)
                                                                                                                                             Teachers & other
                                                                                                                                             staff (managerial                     PTAs etc
                                                       Other community                                                                       and technical)
                                                                                            Parents/guardians
                                                       level influential
                                                                                            Education Staff                                                                        Women‟s Groups
                                                       groups (religious,
                                                                                            (managerial and         Pupils (in and out of
                                                       political, etc.)
                                                                                            technical)              school)
                                                                                                                                                                                    Other organized national,
                                                                                                                                                                                    regional and local groups




file: FTI_Evaluation_Framework(Jan2009b).doc                                                                                                                                                                      Page 115 of 134
                                         Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative




                                                  Box I: Template for identifying key country stakeholders
Stakeholders                                       Before FTI                          During FTI endorsement      Present
                                                   Key stakeholder + contact details   process                     Key stakeholder +Contact detail
                                                                                       Key stakeholder + contact
                                                                                       details
International Level e.g. Ministers, G8,                                                                            s
OECD
External Partners e.g. Representatives and
participants in education coordination and
implementation forum, including the lead
donor and FTI coordination structures,
Development Partner Group.
Education Stakeholders
Representatives and participants in other
multi-sectoral coordination structures e.g.
PRSP, MTEF
Provincial/ Regional Education Staff
(managerial and technical)
District/ Local Education Staff
Public schools as well as community and
religious
Teachers and other staff (managerial and
technical)
Pupils (in and out of school)
Teachers Unions
Parent Teacher Associations (PTAs)
Organizations/individuals providing technical
support (local, regional, international) to the
sector, education coalitions
Prominent local individuals with a non-sector
view on education e.g. politicians, local
leaders, etc.




Page 116 of 134                                                                                                                         printed: 04-Feb-09
                                                             Annex I: Stakeholder Mapping




Stakeholders                                   Before FTI                          During FTI endorsement      Present
                                               Key stakeholder + contact details   process                     Key stakeholder +Contact detail
                                                                                   Key stakeholder + contact
                                                                                   details
Representatives and participants in
education coordination and implementation
forum: e.g. the lead donor and FTI
coordination structures, Development
Partner Group.
Government Stakeholders
Ministry of Planning
Ministry of Finance
Ministry of Health
Ministry of Labour
Other relevant Government Departments
Parliament and specialised committees
Opposition political parties
Civil Society Stakeholders
Women‟s Groups
Youth groups
People Living with HIV/AIDS (PLHA) groups
Local NGOs
Business stakeholders
Chambers of Commerce and other business
associations




file: FTI_Evaluation_Framework(Jan2009b).doc                                                                                       Page 117 of 134
                  Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative




Page 118 of 134                                                                            printed: 04-Feb-09
                                                   Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


Annex J1: FTI Timeline (General)
Date          General Aid Context                         Key Developments in Education               FTI Main Meetings and Events   FTI Endorsements and Expenditure
  pre 1999                                                March 1990 World Conference on
                                                          Education for All, in Jomtien, Thailand
                                                          adopted the World Declaration on
                                                          Education for All, which stated that all
                                                          have a right to education. The conference
                                                          recognised the setbacks experienced in
                                                          the 1980's by many South nations and
                                                          made a commitment to meeting basic
                                                          learning needs of every citizen.


                                                          Convention against discrimination in
                                                          education, adopted by UNESCO in 1960.
                                                          Set out that there should be no
                                                          discrimination in access or quality of
                                                          education.
       1999                                               Education For All (EFA) Assessment
                                                          1999-2000, involving six regional
                                                          conferences revealed that the EFA
                                                          agenda had been neglected.
       2000   United Nations Millennium Summit in         World Education Forum, 164
              2000, 189 world leaders signed up to try    governments, adopted the Dakar
              and end poverty by 2015 when they           Framework for Action in which they
              agreed to meet the Millennium               promised to commit the necessary
              Development Goals.                          resources and effort to create a
                                                          comprehensive and inclusive education
                                                          system for all.
       2001   G8 Meeting - Genoa, Italy. July 2001: G8    Aid to basic education grew consistently
              countries establish an EFA Task Force, to   between 2001-2004.
              be led by Canada




file: FTI_Evaluation_Framework(Jan2009b).doc                                                                                                           Page 119 of 134
                                                    Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative

Date          General Aid Context                           Key Developments in Education                FTI Main Meetings and Events                   FTI Endorsements and Expenditure
       2002   Monterrey Consensus, forged at the            Education for All (EFA) Amsterdam,           The Fast-track Initiative (FTI) was            2002 Burkina Faso, Guinea, Guyana,
              International Finance and Development         Netherlands. April 2002: Developing          established. It was started by 22 bilateral    Honduras, Mauritania, Nicaragua, Niger
              Conference, March 2002, Monterrey,            countries and their external partners        donors, development banks and                  are endorsed by FTI.
              Mexico. The consensus commits rich            agree at a Dutch-World Bank sponsored        international agencies active in supporting
              nations to boost trade and aid                conference on broad principles for scaling   education in low-income countries. The
              opportunities for countries with sound        up EFA efforts; the Netherlands commits      FTI is co-chaired on a rotating basis by
              policies.                                     135 million Euro to set the process in       one G8 and one non-G8 donor. It is
                                                            motion.                                      supported by a steering committee, and a
                                                                                                         secretariat housed in and managed by the
              G8 Washington, DC USA. April 2002: The
                                                                                                         World Bank.
              Development Committee endorses the            EFA Global Monitoring Report was
              proposed EFA Action Plan and approves         established to monitor progress towards
              the Fast Track Initiative (FTI), amid         the six EFA goals.                           FTI Launch Washington DC, USA. July
              overwhelming support from the                                                              2002: 18 countries are invited to join FTI
              international community.                                                                   and become eligible for policy and
              G8 Kananaskis, Canada. June 2002:                                                          financial support, based on two criteria:
              agreement to significantly increase                                                        having in place a PRSP and an effectively
              bilateral assistance for the achievement of                                                implemented education sector plan. Five
              EFA and to work with bilateral and                                                         other countries with half the world's out of
              multilateral agencies to ensure                                                            school population are also invited to
              implementation of FTI.                                                                     receive technical and analytical
                                                                                                         assistance to help create an enabling
                                                                                                         environment (policies, capacity) for
                                                                                                         scaling up EFA efforts.
                                                                                                         FTI Donors Meeting, Brussels, November
                                                                                                         2002: Donors meet and endorse seven
                                                                                                         country programs, and agree to ensure
                                                                                                         that adequate funding for these programs
                                                                                                         would be secured.




Page 120 of 134                                                                                                                                                              printed: 04-Feb-09
                                                                          Annex J1: FTI Timeline (General)
Date          General Aid Context                        Key Developments in Education                   FTI Main Meetings and Events                    FTI Endorsements and Expenditure
       2003   Rome Declaration on the harmonisation of   FTI Donors Meeting - Paris, March 2003:         FTI Donors Meeting - Paris, March 2003:         March 2003 - Gambia, Mozambique,
              aid, Rome, Feb 2003. The development       Donors agree on modus operandi for FTI          Donors agree on modus operandi for FTI          Vietnam and Yemen endorsed.
              community committed to work towards        that is country driven, secure funding for      that is country driven, secure funding for
              aligning its assistance around country     the seven countries and agree on an             the seven countries and agree on an
                                                                                                                                                         April 2003 CF country allocations agreed
              development priorities and to harmonize    operating framework for FTI.                    operating framework for FTI.
                                                                                                                                                         for 2003-2005 to Yemen (US$20m), Niger
              donor policies and priorities around
                                                                                                                                                         ($21m), Nicaragua ($14m), The Gambia
              country systems
                                                         The FTI Catalytic Fund (CF) was                 The FTI Catalytic Fund (CF) was                 ($8m), Guyana ($8m), Mauritania ($9m)
                                                         established. It aims to provide transitional    established. It aims to provide transitional
                                                         grants over a maximum of 2-3 years to           grants over a maximum of 2-3 years to
                                                         enable countries lacking resources at           enable countries lacking resources at
                                                         country level but with FTI endorsed             country level but with FTI endorsed
                                                         education sector plans to scale up the          education sector plans to scale up the
                                                         implementation of their plans.                  implementation of their plans.


                                                         FTI Partnership Meeting Oslo Meeting,           FTI Partnership Meeting Oslo Meeting,
                                                         November 2003: Ministers and senior             November 2003: Ministers and senior
                                                         officials from the first FTI countries, Civil   officials from the first FTI countries, Civil
                                                         Society and donors meeting together for         Society and donors meeting together for
                                                         the first time. Discussion of the definition,   the first time. Discussion of the definition,
                                                         modalities, instruments, and governance         modalities, instruments, and governance
                                                         of the FTI partnership. Agreement that FTI      of the FTI partnership. Agreement that FTI
                                                         should be opened to all low-income              should be opened to all low-income
                                                         countries.                                      countries.




file: FTI_Evaluation_Framework(Jan2009b).doc                                                                                                                                  Page 121 of 134
                                    Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative

Date          General Aid Context        Key Developments in Education       FTI Main Meetings and Events                 FTI Endorsements and Expenditure
       2004                                                                  Education Programme Development Fund         2004 Ghana and Ethiopia are endorsed
                                                                             (EPDF) was established in November           by FTI.
                                                                             2004 as a funding window under the FTI
                                                                             to support low income countries improve      Yemen – Grant agreement was signed in
                                                                             the quality and sustainability of their      April 2004 for US$ 10m for 2004. A
                                                                             education sector planning and program        disbursement of US$3.0m was made on
                                                                             development.                                 10 August 2004.
                                                                                                                          The Gambia – The grant agreement for
                                                                             FTI Partnership Meeting, Nov 2004,           $4m allocation was signed on 11
                                                                             Brasilia, Brazil, third meeting of the FTI   November 2004.
                                                                             partnership. There was agreement on the      Niger - The 2003 allocation of US$ 5m
                                                                             FTI Framework document and the need          was disbursed in March 2004.
                                                                             for more formal Assessment Guidelines.
                                                                                                                          Nicaragua - The CF agreement was
                                                                                                                          signed on 17 August 2004; US$ 3.5m was
                                                                                                                          disbursed on 27 September 2004.
                                                                                                                          Mauritania - Two grant agreements (one
                                                                                                                          for the 2003 BNPP US$ 5m allocation, the
                                                                                                                          other for the 2004 US$ 2m CF allocation)
                                                                                                                          were signed on 23 April 2004. US$ 3.5
                                                                                                                          million were disbursed on 31 July 2004.
                                                                                                                          Guyana - The Trust Fund Administration
                                                                                                                          Agreement was signed on 28 June 2004.
                                                                                                                          The Catalytic Fund grant agreement was
                                                                                                                          signed on 28 September 2004.




Page 122 of 134                                                                                                                                 printed: 04-Feb-09
                                                                           Annex J1: FTI Timeline (General)
Date          General Aid Context                         Key Developments in Education               FTI Main Meetings and Events                FTI Endorsements and Expenditure
       2005   March 2005, Paris Declaration, was          The flow of aid to basic education          FTI Donors Technical Meeting, in            Madagascar, Lesotho, Moldova,
              endorsed by over one hundred Ministers,     declined for the first time since 2001.     Washington, DC, September, 2005. The        Tajikistan, Timor Leste and Kenya
              Heads of Agencies and other Senior                                                      main objective of the meeting was to        endorsed by FTI in 2005.
              Officials. Who committed their countries                                                review comments received on the co-
                                                          The gender parity goal set for 2005 has
              and organisations to continue to increase                                               chairs consultation paper, and to define
                                                          been missed. Only 59 out of 181 countries                                               In 2005, a total of 28 country programmes
              efforts in the harmonisation, alignment                                                 the agenda for the Beijing Partnership
                                                          with data have no gender disparities in                                                 received EDPF funds totalling $4.9 million.
              and management aid for results with a set                                               Meeting in November.
                                                          primary and secondary education. Of
              of monitorable actions and indicators.
                                                          these 59 only 3 eliminated gender
                                                                                                                                                  Catalytic Fund disbursed $4m to Ghana,
                                                          disparities between 1999 and 2005.          UK and Norway are the only donors that
                                                                                                                                                  $24.2 to Kenya, $4m to Guyana and $6m
              UN World Summit New York, September                                                     give to the EPDF Trust fund in 2005.
                                                                                                                                                  to Madagascar in 2005. The following
              2005: delegates were accused of
                                                                                                                                                  allocations were pending at the end of
              producing a 'watered-down' outcome
                                                                                                      FTI Partnership Meeting, Beijing,           2005; Yemen, $10m, Niger, $8m,
              document which merely reiterates existing
                                                                                                      November 2005: the FTI Appraisal            Nicaragua, $7m, Gambia, $4m,
              pledges.
                                                                                                      Guidelines are modified in accordance       Mauritania, $2m, Ghana, $4m,
                                                                                                      with UNGEI recommendations and there        Madagascar, $4m.
                                                                                                      is discussion about FTI's role in fragile
                                                                                                      states.
       2006   Committee on the Rights of the Child        Educational Roundtable, held during         FTI Donors Technical Meeting, Moscow,       Mongolia, Cameroon, Djibouti, Kyrgyzstan
              (41st session), Geneva, Switzerland.        World Bank/IMF Annual Meetings,             Russia, March 2006. The main objective      and Rwanda all CF eligible are endorsed
                                                          September 2006, Singapore. The meeting      of the meeting was to review the            in 2006. Albania, Cambodia, Mali and
                                                          focused on the progress that Finance        agreements made at the Beijing              Senegal are also FTI endorsed.
                                                          Ministers from developing countries have    Partnership Meeting in November 2005.
                                                          made in preparing long term plans to        CF Strategy Meeting took place in           Allocations made to five newly endorsed
                                                          achieve the education millennium            Moscow in March 2006                        countries through the CF Djibouti,
                                                          development goals.
                                                                                                      FTI Donors Technical Meeting, Brussels,     Lesotho, Moldova, Tajikistan and Timor-
                                                                                                      Belgium, October 2006. The main             Leste.
                                                                                                      objective of the meeting was to advance
                                                                                                      the work of the Initiative in view of the
                                                                                                                                                  Disbursements made by the CF in 2006:
                                                                                                      decisions to be made at the upcoming
                                                                                                                                                  Djibouti ($3m), Ghana ($10m), Guyana
                                                                                                      Partnership Meeting in Cairo, and
                                                                                                                                                  ($4m), Lesotho ($1.8m), Madagascar
                                                                                                      involved discussions regarding
                                                                                                                                                  ($11m), Mauritania ($1m), Moldova
                                                                                                      governance and the expanded financing
                                                                                                                                                  ($0.25m), Nicaragua ($3.5m), Tajikistan
                                                                                                      mechanism.
                                                                                                                                                  ($3.1m), Gambia ($4m), Timor Leste
                                                                                                       In 2006 addition of two more seats for     ($1.5m), Yemen ($10m).
                                                                                                      developing countries and two more for
                                                                                                      civil society organisations to the FTI
                                                                                                      Steering Committee.




file: FTI_Evaluation_Framework(Jan2009b).doc                                                                                                                            Page 123 of 134
                                                     Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative

Date          General Aid Context                         Key Developments in Education                FTI Main Meetings and Events                   FTI Endorsements and Expenditure
       2007   Committee on the Rights of the Child        Keeping our Promises on Education, May       The FTI donors Technical Meeting, Bonn,        2007 Benin, Sierra Leone, Liberia are
              (45th Session).                             2007, Brussels, organised by the EC, the     Germany, on May, 2007: review of               endorsed by FTI.
                                                          UK and the World Bank. The objective         progress since Cairo, and discussion of
                                                          was to seek concrete proposals and           ways to strengthen country-level
                                                                                                                                                      Cameroon: The grant agreement for
                                                          commitments for action to deliver on the     processes.
                                                                                                                                                      US$22.5m was signed in May 2007 and
                                                          promise to give all the world's children a   EPDF Committee Meeting, Bonn, May              the first tranche of US$11.25m was
                                                          full primary education by 2015.              2007 at which the monitoring and               disbursed in May 2007. Mongolia: The
                                                                                                       evaluation task team was set up .              grant agreement for Mongolia was signed
                                                          In Oct 2007, the German Federal Ministry     The Steering Committee met five times in       in March 2007. The first tranche of
                                                          for Economic Cooperation and                 2007: January (audio), March (audio),          US$3m was disbursed on 10 May 2007.
                                                          Development organised an international       May (audio). The expanded Steering             Rwanda: The grant agreement for
                                                          forum on “Capacity Development for           Committee met for the first time in May        US$26m was signed in May 2007 and the
                                                          Education for All: Putting Policy into       22-25 (Bonn, Germany), and finally in DC       first tranche of US$13m was disbursed in
                                                          Practice.” Participants recommended          in September.                                  May 2007.
                                                          more strategic use of the EPDF to support    Concept Note for the Expanded Catalytic
                                                          capacity development activities, and to      Fund (ECF) was finalised in April 2007.        The following countries received CF
                                                          harmonise and align donor support for        This enabled CF money to be given over         allocations in Bonn, in May 2007: Benin
                                                          technical assistance and capacity            a longer period than the original 3 years to   ($76.1m), Cambodia ($57.4m), Mali
                                                          development in all low-income countries.     countries with FTI-endorsed plans but          ($8.7m), Mauritania ($14m), Mongolia
                                                          They also recommended that the FTI           insufficient domestic or external aid          (allocated funds in Cairo), Mozambique
                                                          partnership consider how to work more        resources. The CF guideline for allocation     ($79m) and Sierra Leone ($13.9m).
                                                          effectively with UNESCO, the World Bank,     based on donor orphan status was
                                                          and other agencies.                          replaced with a new guideline based on
                                                                                                       the size of the funding gap as a proportion
                                                                                                       of existing donor funding.                     .
                                                                                                       CF Committee (CFC) meeting on 10
                                                                                                       December 2007 in Dakar (Senegal).




Page 124 of 134                                                                                                                                                             printed: 04-Feb-09
                                                                      Annex J1: FTI Timeline (General)
Date          General Aid Context                        Key Developments in Education      FTI Main Meetings and Events                    FTI Endorsements and Expenditure
                                                                                            September 19, 2007 in Washington, the           At EPDF Bonn meeting, bridge funding
                                                                                            Catalytic Fund (CF) Committee decided to        was approved for E. Asia and Pacific
                                                                                            clarify the rules regarding countries'          Regions totalling $US 500,000.
                                                                                            eligibility to receive CF funding. It was       In Washington, Sept 2007 Special
                                                                                            decided that only IDA low-income                Committee Meeting approved US$7.5m
                                                                                            countries would be eligible.                    for 4 regions
                                                                                            At its September 2007 meeting in
                                                                                            Washington, the EPDF Committee
                                                                                                                                            The following four countries became new
                                                                                            decided that EPDF could be used to fund
                                                                                                                                            CF recipients at the CF Committee
                                                                                            thematic activities, either global or cross-
                                                                                                                                            Meeting in Dakar (Senegal) in December
                                                                                            regional. It also decided that funds could
                                                                                                                                            2007, with commitments totalling US$270
                                                                                            be used to finance activities at other levels
                                                                                                                                            million: Ethiopia, Guinea, Sao Tome and
                                                                                            of the education system, as long as clear
                                                                                                                                            Principe, and Senegal.
                                                                                            analytical links were made with the
                                                                                            overarching objective of Universal Primary
                                                                                            Education.
       2008   September 2008, Accra summit on aid                                           FTI Donors Technical Meeting, in Tokyo,         Eight countries projected to have their
              effectiveness, donor countries have                                           April 2008. The main items on the agenda        education sector plans endorsed in the
              agreed to end the fragmentation of aid.                                       were to promote a holistic approach to          calendar year 2008 : Bhutan, Burundi,
                                                                                            EFA, to continue the discussions on             Central African Republic, Chad, Haiti,
                                                                                            strengthening country level processes,          Malawi, Uganda, and Zambia.
              Donors agreed to donate half of aid
                                                                                            mobilising additional resources for
              directly to governments of low-income
                                                                                            education. The Steering Committee met
              countries, rather than to individual
                                                                                            twice in April 2008, to update of progress
              projects. Donors will also give earlier
                                                                                            since the December Dakar meetings, a
              statements of their giving, so that
                                                                                            report on strengthening country level
              recipients can draw up balanced budgets.
                                                                                            processes, a discussion on governance
                                                                                            and management issues,and presentation
              Donors have also agreed to coordinate                                         of the FTI Secretariat work plan and
              aid better.                                                                   budget, as well as of the external
                                                                                            evaluation of FTI.


                                                                                            The Steering Committee and the Trust
                                                                                            Fund Committees met September, 2008,
                                                                                            at UNESCO's Headquarters, Paris,
                                                                                            France.




file: FTI_Evaluation_Framework(Jan2009b).doc                                                                                                                      Page 125 of 134
                  Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative




Page 126 of 134                                                                            printed: 04-Feb-09
                                               Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


                                        Annex J2: Trajectories of Country Involvement in FTI




file: FTI_Evaluation_Framework(Jan2009b).doc                                                                            Page 127 of 134
                  Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative




Page 128 of 134                                                                            printed: 04-Feb-09
                                               Annex J2: Trajectories of Country Involvement in FTI




file: FTI_Evaluation_Framework(Jan2009b).doc                                                          Page 129 of 134
                  Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative




Page 130 of 134                                                                            printed: 04-Feb-09
                                               Annex J2: Trajectories of Country Involvement in FTI




file: FTI_Evaluation_Framework(Jan2009b).doc                                                          Page 131 of 134
                  Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative




Page 132 of 134                                                                            printed: 04-Feb-09
              Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative




Annex K: Format for Interview Notes
Interview notes will be written up in a standard format to facilitate sharing, collating and
searching the data that they represent. Interviewees' confidentiality will be respected and
notes will not be shared beyond the evaluation team. The format is designed to strike a
balance between standardisation and flexibility (given that the interviews are only semi-
structured conversations).


Date (e.g.2008-12-15): Interviewee Name/Organisation
Filename: Use the following format for the file name, running elements together with no spaces:
date(yyyy-mm-dd)_name_organisation. (e.g. 2008-12-15_Bermingham_FTIsecretariat)


GENERAL
Date:                      Location of Interview:                             Evaluation Team Members Present:



INTERVIEWEE(S)
Name: [gender?]            Designation: [organisation, job title,             Contacts: [phone / e-mail, etc]
                           etc]



NOTE TAKING
Name:                      Date completed:                                    For admin use only: on website?


BACKGROUND AND LINKS TO FTI
Interviewee's general background; Nature and dates of interviewee involvement in FTI


TOPICS
Record responses by topic with clear headings, not necessarily in chronological sequence of
discussion. Make clear when a direct quote is recorded. Add headings and sub-headings as needed.


Narrative
Story of engagement with FTI (by individual / organisation / country).



Perceptions of FTI
Purposes and intentions of FTI, Perceptions of its success/weaknesses.


Analysis
Interviewees views (if any) on main FTI themes.
Financing Gap

Education Policy and Planning




file: FTI_Evaluation_Framework(Jan2009b).doc                                                       Page 133 of 134
           Team Handbook for the Mid-Term Evaluation of the Fast Track Initiative


Capacity Development

Data and M&E

FTI Governance

Aid Effectiveness

DATA/DOCUMENTS PROVIDED/RECOMMENDED
Seek full references for documents not already in study library.


OTHER PROPOSED FOLLOW-UP
e.g. other interviewees recommended / proposals on consultation and dissemination




Page 134 of 134                                                                     printed: 04-Feb-09

				
DOCUMENT INFO