Development Cooperation Ireland Evaluation of the Development Cooperation Ireland Multi

Development Cooperation Ireland Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 Front Cover photo: Dr. Tara Shine, 2 Shepard Boys, Tigray 2004. Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 Synthesis Report October 2005 ACRONYMS DCI DFID GNP HAPS HDI HIPC IHL IMF LDCs LDOs MAPS MDGs NGO ODA OECD PAEG PDED PMC PPA PPMG PRGF PRSPs UN UNAIDS UNDP Development Cooperation Ireland Department for International Development (UK) Gross National Product HIV/AIDS Partnership Scheme Human Development Index Heavily Indebted Poor Country Initiative International Humanitarian Law International Monetary Fund Least Developed Countries Local Development Organisations Multi-Annual Programme Scheme (Development Cooperation Ireland) Millennium Development Goals Non-Governmental Organisation Overseas Development Assistance Organisation for Economic Cooperation and Development Projects’ Appraisal and Evaluation Group (Development Cooperation Ireland) Policy Development and Evaluation Directorate Partnership Monitoring Committee Partnership Programme Agreement Programme Planning and Monitoring Group Poverty Reduction and Growth Facility Poverty Reduction Strategy Papers United Nations United Nations Programme on HIV/AIDS United Nations Development Programme page 2 Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 CONTENTS Executive Summary 1. Introduction 2. Origins and Background to MAPS 3. Synthesis of Key Points from the MAPS Agency Studies 3.1. Partnership Relationships between MAPS NGOs and Development Cooperation Ireland 3.2. The Nature of the Partnership with Southern NGOs 3.3. Organisational Strengthening 3.4. Southern Partnerships/Impact 3.5. Cross-Cutting Issues – mainstreaming. 4. Summary of the Comparative Study 4.1. Introduction to the Comparative Study 4.2. Background to Official Agency Support for NGOs 4.3. Types of Block Grant or Framework Scheme 4.4. Criteria for Choice of Agencies 4.5. Monitoring and Evaluation of Funding Schemes 4.6. Mechanisms for Managing Funding Schemes 4.7. Learning From the Different Approaches 5. Criteria for Selection for a Future MAPS Programme 6. Analysis against MAPS objectives 6.1. The First Stated Objective 6.2. The Second Stated MAPS Objective 6.3. The Third of the Initially Stated Objectives 6.4. Accountability 6.5. Monitoring and Evaluation 6.6. Implications for Non MAPS NGOs 6.7. Other Implications 6.8. Summary 7. Conclusions and Recommendations 7.1. General 7.2. Ouality Aspects 7.3. Institutional Arrangements Annex 1 Evaluation of Development Cooperation Ireland’s Multi-Annual Programme Scheme 2003–2005: Abbreviated Terms of Reference 1. Introduction 2. Purpose and Scope 3. Initial Objectives/Purpose of Evaluation 4. Methodology 5. Output 6. Management Structure 7. Selection Criteria Annex 2 DCI Management Response to the Evaluation of the Multi-Annual Programme Scheme – 2005 7.1 General 7.2 Quality Aspects 7.3 Institutional Arrangements 4 7 8 9 9 10 12 14 17 20 20 20 21 22 22 23 23 24 26 26 27 28 29 31 32 33 33 34 34 36 38 41 41 41 41 42 42 42 43 44 44 48 50 www.dci.gov.ie page 3 EXECUTIVE SUMMARY 1. The Multi-Annual Programme Scheme (MAPS) was an innovative concept, with a series of interlocking features, each representative of international best practice. Its objectives were such that, if fully implemented, the Scheme would constitute a very effective framework for coherent and predictable development work by its partners. At the functional level, MAPS NGO field staff and Southern partner staff would be aware of best practice methodologies and would be mainstreaming the cross-cutting issues in every programme being funded by MAPS. At the management level, all the programmes would be operating in compliance with coherent strategy plans, and impact and outcomes assessment would be ongoing. Some elements of this conceptual framework have been implemented in the two and a half year period since MAPS began. Whilst all agencies have made strides relative to their starting positions in 2002, the opportunities afforded by MAPS to work programmatically and creatively have not yet been maximised, and the potential of the Scheme as a force for programmatic transformation and the continual quest for best practice has not yet come to fruition. The evaluation team readily acknowledges that the objective of working with different levels of partnership in a programmatic mode is understandably a long-term process which is not realistically attainable within a three year ‘pilot’ phase; indeed ‘MAPS I’ was expected by its architects to involve a good deal of ‘learning by doing’ for all parties involved, and that has turned out to be the case. In the opinion of the evaluation team, there is scope for all of the MAPS NGOs to progress further and faster down the programmatic road, by constantly focusing on key features such as macro-level, strategic impact, long-term perspective, and a more refined partnership-based mode of operation. 3. The evaluation team finding is that the movement towards realisation of the key elements indicates the validity of MAPS, both as a partnership vehicle and as a strategic funding mechanism. MAPS does contribute towards a framework for predictability and coherence. A critical element of MAPS lies in the move from a project to a programme approach based on a strategic plan. While the proposals from the MAPS NGOs contained a strategic focus, the work to be funded reflected both the ongoing policies of the agencies as well as a move from project to programme. The process involved in this paradigm shift turned out to require more elucidation, and more implementation, than was possibly envisaged at the outset, but it is ongoing. The Scheme has been found to be broadly achieving the objectives of conferring greater security and predictability of funding on the participating NGOs, thereby providing scope for more strategic planning of programmes of development assistance interventions. Flexibility was repeatedly mentioned by the MAPS NGOs as a significant merit of the Scheme, in that (i) MAPS NGOs were free to include programmes in their initial MAPS submissions, and (ii) were subsequently free to vire funds between budget heads up to a limit of 20%. That said, the partner NGOs differ from each other in how they use this flexibility. There is agreement regarding the reduction in transaction costs and scale-of-funding advantages conferred by MAPS. The contribution of MAPS to lowering the unit cost of managing the €117 million allocated is significant, taking into account the combined costs incurred both within each MAPS NGO and within Development Cooperation Ireland. To have successfully disbursed the same amount through Block Grants or (to an even greater extent) through the NGO Co-Financing Scheme, 2. 4. page 4 Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 would have involved much greater costs for both the donor and the recipients. Conversely, without a mechanism such as MAPS, the same level of management capacity across the six partners could not have coped with the equivalent volume of aid activity. 5. There has been some reluctance to innovate within MAPS, or to put forward for MAPS funding, programmes seen by MAPS NGOs as potentially risky. Whilst Development Cooperation Ireland had indicated its openness to this (recognising the value of an NGO approach), it had not been articulated or ‘pushed’ very hard by both parties. The outcome has been that, in general, the MAPS NGOs have adopted a rather conservative stance, partly because of concerns surrounding accountability, but partly also to ensure that such an unprecedented source of stable, long-term funding would not be jeopardised in any way. The finding of the evaluation team is that the spirit of partnership between Development Cooperation Ireland and the MAPS NGOs has, in general, advanced during MAPS I. No instance arose in which a MAPS NGO felt pressure from Development Cooperation Ireland to act outside its own visions and strategies, nor has there been pressure to standardise. On the contrary, there has been a strong commitment overall, within Development Cooperation Ireland, to avoid a ‘one-size-fits-all’ approach. The evaluation team has found therefore that the diverse visions, cultures, priorities and emphases of the five NGOs are preserved and valued within MAPS, that this has given the venture significant strength and that there is a real desire to maintain this. With regard to policy coherence between Development Cooperation Ireland and the MAPS NGOs, two processes are taking place. The first aims to increase policy coherence on the ground. This is the basis on which the MAPS partners seek to implement a poverty focus and to mainstream the three primary cross-cutting issues of HIV/AIDS, gender and environment. Some progress has been made in this regard, although to varying degrees across the agencies; for example, progress on environment has in most instances been negligible. The second process is more strategic and is about the move from a project to a programme approach. In this process, the policy preferences of Development Cooperation Ireland for stronger strategic planning, programming and attention to macro-level outcomes and impact are intended to be implemented by the MAPS NGOs. Our finding is that there has been movement in this direction by all partners and that this movement has been accelerated as a result of MAPS-related processes, but that there remains some distance to go towards fulfilment. It would seem that, at the outset of MAPS, assumptions were made about agencies’ capacity in this regard, but that the monitoring processes (in the agencies and in Development Cooperation Ireland) were generally not adequate to identify shortcomings. The long-term aspiration implicit in MAPS that a ‘programme ethos’ should percolate through to Southern partner NGOs is considered to be at a very incipient stage indeed. The potential for fruitful effort in this regard in future phase(s) of MAPS is therefore considerable. 8. At present, the lack of standardisation across the five MAPS NGOs on presentation of financial data complicates the preparation of breakdowns on how MAPS funding is or has been spent, although a recent (January 2005) request for information broken down by country will improve the Development Cooperation Ireland overview. Some of the key recommendations emerging from the Evaluation are as follows: a) That there be a second phase of MAPS for a further period of five years duration, with the possibility of phasing prospective participating NGOs into the Scheme as and when they fulfill the MAPS Guidelines requirements, and with the proviso that a mid-term evaluation be built in. In relation to this, if necessary, a ‘bridging’ period for the existing MAPS NGOs should be created in the event that preparations for MAPS II are not yet complete by the end of 2005. 6. 9. 7. www.dci.gov.ie page 5 b) That a restricted call for proposals be issued for participation in MAPS II, based on the agreed eligibility criteria (suggestions for such criteria are given in Section 5 of this Report). Applicant NGOs will need to demonstrate their capacity and capability to be able to apply, manage, monitor and communicate on the use of MAPS funds at the levels of outcomes and impact; c) That, for MAPS NGOs, most of the existing funding schemes be subsumed into MAPS and that a similar rationalisation happen for those NGOs in the Block Grant and NGO Co-financing Schemes. It is recommended that these changes be made as soon as possible, and not be postponed until after the planned decentralisation of Development Cooperation Ireland. d) That the criteria for the number of countries to which NGOs apply MAPS funds be based on their ability to manage the funding according to MAPS Guidelines. e) That a clearer, crisper, more policy-focused set of MAPS Guidelines be formulated, which inter alia would safeguard against over-dependence on a single income source, by requiring that each agency participating in MAPS should maintain a ratio between the annual total of its funding from all Development Cooperation Ireland funding schemes (excepting those designed to assist victims of sudden-onset emergencies) on the one hand, and funds raised from all other sources (excluding sudden-onset emergency appeals) on the other; this ratio not to exceed 70:30 in favour of the former. f) Use of external (i.e. out-sourced) technical support to strengthen the capacity of the Civil Society Section to monitor effectively and evaluate an expanded MAPS programme with a clear focus on outcomes and impact. This approach will also provide Development Cooperation Ireland with valuable external perspectives. g) That clear terms of reference for the Partnership Monitoring Committee structure be devised, distinguishing clearly between management and learning, and emphasising the former. h) That, in MAPS II, Partnership Monitoring Committees meet only twice annually but focus exclusively on management issues, reporting and accountability; and in addition that all partners come together for a twice annual plenary meeting, with facilitation, to share learning on MAPS’ thematic and policy issues, to harvest best practice, to clarify the expectations of Development Cooperation Ireland, to address evaporation, to exchange outcomes on mainstreaming of cross-cutting issues, and to engage in programme level learning and thematic discussion. i) For the next phase of MAPS, that all agencies comply with a summary presentation of their expenditure data according to an electronic format standard template, on the basis that this: ■ would not impose major new burdens on the MAPS NGOs; ■ would meet Development Cooperation Ireland’s legitimate needs; ■ would make greater use of accounting IT; ■ would fit in with the various different systems in use in MAPS NGOs; ■ would seek to address questions of Development Cooperation Ireland’s other information needs; ■ would comply in format with DAC Code requirements. page 6 Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 1. INTRODUCTION 1.1. This report summarises the six reports of the five MAPS-funded NGOs and Development Cooperation Ireland which together constitute the end-product of a wide-ranging external evaluation commissioned by Development Cooperation Ireland of its Multi-Annual Programme Scheme (MAPS). This Scheme was first introduced toward the end of 2002; it involved five participating agencies (Christian Aid Ireland, Concern, GOAL, Self Help Development International and Trócaire) and, at the time of writing (July 2005), MAPS is approaching expiry of its initial three-year phase of operation. The evaluation assignment was conducted between late January and late June 2005 by a consultant team comprising INTRAC (UK) and Annesley Resource Partnership (Ireland). The purpose of the MAPS evaluation was to assess the progress of the Scheme, focusing primarily on relevant strategic and operational issues and examining performance from the perspectives of (a) international standards of best practice, (b) effectiveness, (c) efficiency and (d) impact, with particular emphasis on partnership. The exercise was designed to look mainly at organisational structures and systems, their changes/developments since the inception of the programme both horizontally and vertically, and to establish any evidence of operational changes. Whilst most of the evaluators’ time was devoted to information gathering and in-depth bilateral consultations with key personnel in Development Cooperation Ireland and in the five participating agencies in Ireland, field visits were also undertaken in five selected countries to see at first hand some illustrative examples of the outworking of MAPS at ground level. 1.3. This Synthesis Report draws together the key findings and conclusions of the six agency studies (the five MAPS-funded NGOs and Development Cooperation Ireland), and brings in comparative information from similar schemes throughout Europe to make recommendations on how the MAPS funding scheme could be taken forward. The authors would like to acknowledge warmly the excellent cooperation shown at all times by the staff of Development Cooperation Ireland and by the five agencies concerned, both at headquarters and in the field. They greatly facilitated the work of the evaluation team and were always generous with their time. 1.4. 1.2. www.dci.gov.ie page 7 2. ORIGINS AND BACKGROUND TO MAPS 2.1. Ireland has a long history of support for work with the poorest and most marginalised communities around the world through Irish missionary and non-governmental organisations. NGOs are recognised as important partners to Development Cooperation Ireland through their mode of working, strong partnerships with local civil society groups and commitment to influencing policy and advocacy. Over the last decade, there has been a steady increase in the amount of support that the Irish Government has given to development work, with a focus on the Millennium Development Goals and substantial increases in funding support to the work of NGOs. The increases in development support must also be seen against a background of rationalising and minimising costs in the civil service through reducing staff numbers, and a policy of decentralisation. Support for NGOs has been directed through a number of different funding/co-funding schemes. These schemes have been projectfocused and short-term, typically annual. The weakness with such approaches is that they tend to emphasise activities rather than outcomes and impact, and do not easily enhance the development of a stronger and more vibrant civil society. At the same, time they are timeconsuming for both Development Cooperation Ireland and the NGOs. As part of its drive for excellence and effectiveness and recognising the role that NGOs have to play, Development Cooperation Ireland has looked at ways of improving how it relates to NGOs and addresses these issues. 2.3. In the year 2000, Development Cooperation Ireland started to examine the potential for new arrangements and relationships with the NGO sector in Ireland. During the period 2000–2003, negotiations and dialogue took place between Development Cooperation Ireland and the five selected NGOs1, culminating (in 2003) in the establishment of a Multi-Annual Programme Scheme (MAPS) which involved three-year funding agreements to a total of €117 million. At the heart of this arrangement was a move to a new relationship between the donor and the participating NGOs, based on multi-annual strategic planning and partnership. Through MAPS, development assistance is delivered to a wide range of sectors (including health, education, HIV/AIDS, sustainable livelihoods) in more than 40 countries, including those where the official bilateral aid programme has a formal presence. The principle of partnership between Development Cooperation Ireland and the participating five NGOs is fundamental to the programme. This is manifested through the Partnership Monitoring Committees whose function is to help establish good relationships and a balance between learning and accountability (in the broadest sense). 2.4. 2.2. 1 Christian Aid Ireland, Concern, GOAL, Self Help Development International and Trócaire. page 8 Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 3. SYNTHESIS OF KEY POINTS FROM THE MAPS AGENCY STUDIES 3.1. Partnership Relationships between MAPS NGOs and Development Cooperation Ireland 3.1.1. The partnership that Development Cooperation Ireland has with the MAPS-supported NGOs has involved the development of trust which, from both stakeholders’ perspectives, has enabled them to use substantially greater levels of funding than under previous schemes. This is a clear benefit that emerges also from the Comparative Study (see Annex 1). An example from the evaluation: The quarterly Partnership Monitoring Committee (PMC) meetings with Development Cooperation Ireland served to enhance the relationship and level of dialogue between Trócaire and Development Cooperation Ireland...The review meeting in June 2004, to critique the first Annual Report, was positive and useful in providing constructive feedback on the report and on the progress of the scheme in general.2 3.1.2. The Comparative Study brings out the point that the development of trust3 is a key factor in the partnership relationship between Development Cooperation Ireland and MAPS NGOs. It is also a distinctive element in a number of funding schemes which see that procedural simplicity (allied to cost-saving) needs to have an adequate basis of trust for it to be effective. It is significant that a good level of trust has developed over such a short space of time. The key factors in developing this trust are: ■ Regular contact. The process of managing MAPS through the quarterly Partnership Monitoring Committees has clearly been a significant factor. ■ Commitment. It is important to recognise that the Government puts a high priority on working with NGOs, perceiving them to have a relevance, and complementarity with its own work; this has provided a foundation of openness and willingness to enter into dialogue. Commitment is also seen in the time that both parties have put into the process.4 ■ Access. Development Cooperation Ireland and the MAPS NGOs have many opportunities for informal dialogue and discussion, which adds depth to the relationship. The partnership process is not limited to specific moments, activities or channels of communication. 3.1.3. With time and resource pressures facing all stakeholders, the trust that has been developed is a value that needs to be nurtured. In the next round of MAPS, it will be essential to establish how that trust can be maintained for the current MAPS NGOs and how it can be developed for any new NGO partners. Two specific issues need to be taken into account: 2 3 4 Trócaire Year 2 MAPS Report to Development Cooperation Ireland; March 2005, p. 7. Alan Fowler refers to such relations as ‘authentic partnerships’, which he suggests are based on trust and commitment; shared beliefs, values, or culture; accepted standards of legitimacy, transparency, and accountability; and common approach to gender issues (Fowler, 1997, Striking A Balance: A Guide to Enhancing the Effectiveness of NGOs, p.109). An example of this is the commitment of time and resources that the Civil Society Section of Development Cooperation Ireland invested in supporting MAPS NGOs in their preliminary work for GOAL through MAPS I, specifically through the development of a strategic plan. www.dci.gov.ie page 9 ■ The planned decentralisation of Development Cooperation Ireland to Limerick and any expansion of the Scheme will mean that the proximity of NGOs to Development Cooperation Ireland will be lessened and that access will be more time- and resourceconsuming. ■ Whether the current arrangements for individual agency Partnership Monitoring Committee meetings each quarter can be reconciled with a new phase of MAPS in which the number of MAPS NGOs may be greater, but staffing levels in Development Cooperation Ireland remain static. 3.1.4. Partnership needs trust. In MAPS, this has been growing, partly as the result of innovations such as the Partnership Monitoring Committees. Continued development of trust requires ‘nosurprises’ relationships which can: ■ tell the bad news, of objectives not achieved, as well as the good; acknowledge in a mature way the weaknesses in the approach of one’s own agency; ■ provide straightforward and comprehensive reporting; question stereotypical perceptions; ensure accurate accounting; ■ have realistic joint objective-setting at the levels of both policy and programme; develop clearer understanding of each other’s constraints; ■ develop ongoing good use of the Partnership Monitoring Committee structure; including constructive joint work on non-contentious issues such as the mainstreaming of environment; ■ further partnership activities outside the direct framework of the Partnership Monitoring Committees; share experience (e.g. in Southern partnering or in programming); ■ move toward jointly agreed definitions of key concepts; acknowledge history, and move on; and develop understanding of why different issues are seen by different partners as having different levels of importance. 5 3.2. The Nature of the Partnership with Southern NGOs 3.2.1. For this evaluation of MAPS, the evaluation team proposed using a ‘Partnership Continuum’ from Fowler5 as a way of trying to understand the types of partnership relationships which each of the MAPS-funded NGOs had with their Southern partners. In practice, this continuum was not as helpful as had been envisaged when it became clear that the NGOs developed partnerships with different types of Southern NGOs – small, large, well established, and so on, and that the types of partnerships varied according to: ■ the capacity of the Southern partners; ■ the degree of engagement with the partners (usually but not always related to the first point); ■ the administrative support mechanism put in place by the MAPS NGO. 3.2.2. The findings from the evaluation show that the five MAPS NGOs had different approaches to and understandings of partnership. They were also very different in that Christian Aid and Trócaire came from a strongly ‘non-operational’ background, whilst GOAL has traditionally been operational. Although Development Cooperation Ireland did not give an explicit definition of partnership, the MAPS Guidelines drafted in 2001 gave an overview of how it understood its purpose: The aid relationship is perceived as a partnership built on mutual respect and parity of esteem. It is vitally important that developing countries have a sense of ‘ownership’ of aid-supported programmes and that the people who are the intended recipients should have an opportunity to participate in the process as far as possible. Fowler’s Partnership Continuum proposed different types of partnership relationships with the rights that would be expected of Northern and Southern NGOs in those relationships (Fowler, 1997, Striking A Balance: A Guide to Enhancing the Effectiveness of NGOs, Table 7.1, p.164). page 10 Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 3.2.3. Whilst not all agencies have a ‘Partnership Policy Statement’, they articulate partnership in their major documents. A summary of the key points of each organisation’s approach and understanding is given below: ■ Trócaire has a strong partnership-based approach as a distinctive keynote of its profile. It is non-operational and has established a field office presence overseas in order to facilitate the cultivation of deeper and more effective partnerships. Over the years, Trócaire had accumulated more experience than its peers as a practitioner of partnership; even in emergency settings Trócaire works in a partnership mode more often than not. ■ SHDI. The partnership approach to joint working is a key principle of SHDI, and this has remained constant both before and during MAPS. Within SHDI, partnership in the South takes the form of the trilateral model (comprising village communities, local government organs, and SHDI). Partnership is seen as essential to building and strengthening existing capacity. ■ GOAL was primarily an implementing organisation which is increasingly moving into development work in partnership with different local organisations. It sees capacitybuilding of its local partners as an essential element of its work. It does not yet have a partnership policy. ■ Christian Aid. Partnership is one of Christian Aid’s three fundamental principles. Christian Aid is essentially a non-operational organisation that works with and through partners. It sees itself as having a model of partnership which both informs and reinforces its engagement in effective advocacy. The importance of the relationships between Christian Aid and its partners is based in part on the use of reciprocal learning and exchange. ■ Concern works with and through partners. It aims to have a close relationship but not to dominate. The majority of Concern’s partners are small-to medium-sized. Concern has a draft policy and guidelines which have nine partnership principles: Compatibility, Mutual Respect, Shared Understanding, Responsibility, Flexibility, Transparency, Learning, Mutual Influence, Accountability. 3.2.4. From the evaluation a number of common characteristics of the principles of partnership emerge: ■ Flexibility; ■ Transparency and accountability; ■ Advocacy; ■ Mutual influence and reciprocal learning; ■ Capacity-building; ■ Long-term relationship. These key principles link with those from Fowler’s Partnership Continuum and show that the MAPS NGOs are primarily focused around partnership as ‘programme’ or ‘institutional’ support. 3.2.5. Whilst the MAPS NGOs are defining what partnership means for them, it is clear that there is a need also for Development Cooperation Ireland to develop its own partnership policy which will articulate what it sees as the key elements of partnerships for its engagement with NGOs, whether through MAPS or other funding mechanisms. 3.2.6. As partnership is seen as a central part of MAPS, it is essential that the key elements of what those partnerships should include are articulated in the MAPS Guidelines. This is an area that could be included in a meeting of the agencies involved in the current round of MAPS, which could draw up a clearer statement of partnership characteristics and purpose for the MAPS Guidelines. www.dci.gov.ie page 11 3.3. Organisational Strengthening 3.3.1. As we have seen, the core MAPS requirements include a focus on strategy, a shift from a projectbased to a programme-based approach, working through and with Southern partners, and mainstreaming a number of cross-cutting issues. The extent to which MAPS NGOs had independently adopted these principles before the advent of MAPS differs significantly between them, resulting now in an uneven pattern in terms of policy coherence. Key issues identified in relation to NGOs’ operationalisation of a programmatic approach include: ■ no clear added value, over and above organising projects in a thematic way; ■ evaporation from headquarters to field; ■ weak focus on outcomes and impact with a corresponding tendency to report on activities and outputs; ■ the need for a stronger link to policy dialogue and advocacy. Whilst all agencies have made strides relative to their starting positions in 2002, the opportunities afforded by MAPS to work more programmatically, creatively and imaginatively have not yet been maximised, and the potential of the Scheme as a force for programmatic transformation and the continual quest for best practice has not yet come to fruition. The evaluation team readily acknowledges that the objective of working with different levels of partnerships in a programmatic mode is understandably a long-term process which is not realistically attainable within a three-year ‘pilot’ phase; indeed, MAPS I was expected by its architects to involve a good deal of ‘learning by doing’ for all parties involved and that has turned out to be the case. In the opinion of the evaluation team, there is scope for all of the MAPS NGOs to progress further and faster down the programmatic road, by constantly focusing on key features such as macro-level, strategic impact, long-term perspective and a more refined partnership-based mode of operation, whilst at the same time building internal and external coherence within their programmes, on an agency-wide basis. 3.3.2. The evaluation team has found many strengths in the organisational approach adopted by Development Cooperation Ireland, whose Civil Society Section carries the prime responsibility for MAPS. These include a wide flexibility in understanding that the MAPS NGOs have quite different histories, value systems and management cultures; a high level of energy and commitment to MAPS and to making it work successfully (evidenced, for example, in the quality of exchanges within the Partnership Monitoring Committees); a focus on the impact of MAPS, as well as the details; and a willingness to supplement the Partnership Monitoring Committees with other modalities (such as benchmarking, monitoring visits, accounting systems review, and so on). 3.3.3. Weaknesses in the Development Cooperation Ireland approach include the fact that there are now too many grant schemes, much smaller than MAPS in budgetary terms, but more labour intensive, which are being administered (for the most part) by the Civil Society Section, a section with a very limited staff complement. With MAPS, these other schemes and additional duties on top, this small unit is overloaded. The forthcoming decentralisation of Development Cooperation Ireland may complicate matters further. 3.3.4. One requirement which is now evident is that a set of MAPS progress indicators (together with supporting objectives) is needed. While the Partnership Monitoring Committees are a joint operation both in design and practice, there is no framework against which MAPS partners can assess whether particular MAPS NGOs are operationalising particular objectives satisfactorily or not. The finding of the evaluation team, overall, is that this first phase of MAPS is succeeding, but in an incremental way. In such page 12 Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 circumstances, best practice calls for the agreement of verifiable indicators and milestones by which they will be arrived at, even in respect of less tangible or measurable variables. It is essential to fashion the tools for identifying and tracking change in MAPS. 3.3.5. Amongst the MAPS NGOs, it is clear that there is an appetite for the Scheme and the organisational approach to it offers a number of strengths. These include the MAPS Co-ordinator positions which have provided continuity, attention to detail and the possibility for learning to accumulate and to be programmed in, as well as giving Development Cooperation Ireland in each MAPS NGO a permanent focal point for the Scheme. 3.3.6. The MAPS NGOs have brought strength to the working of the Partnership Monitoring Committees and have played a constructive and developmental role in their movement toward engaging the thematic as well as the management-related questions. And they have also brought to MAPS the breadth and sheer volume of their overseas activity, providing a very wide canvas on which MAPS can test its effectiveness, for example in mainstreaming the cross-cutting issues. 3.3.7. Weaknesses on the MAPS NGO side of the partnership include, firstly, policy evaporation, i.e. dilution of the policy goals when they reach (or sometimes fail to reach) field level. A second weakness in the process is mainstreaming the cross-cutting issues, in particular a general reluctance to engage fully with environment as an issue. A critical issue here is the fact that there were different understandings amongst the five MAPS NGOs of the degree of mainstreaming that would be expected (specifically with regard to environment). A third problem was staff turnover, most notably within GOAL and Development Cooperation Ireland prior to and since the inception of the Scheme. Having said this, it is acknowledged that it would have been unrealistic to expect MAPS in its fully-fledged form to be universally embedded within three years across a large number of programmes of very different kinds and scales, in a number of countries, involving a wide range of Southern partnership models by five MAPS NGOS, each with its own unique personality and culture. 3.3.8. In the context of prospects for a second and expanded phase of MAPS , the requirements for strengthened organisation include: ■ Training/familiarisation in both Ireland and a few strategically-selected field locations in MAPS core principles such as strategy, planning, Southern partnering, programming, mainstreaming, etc. ■ A new structure for the Partnership Monitoring Committees involving a reduction in bilateral meetings from four to two annually (with a focus on management and accounting issues), and two annual, externally facilitated, thematic round tables, for all MAPS NGOs and Development Cooperation Ireland combined. ■ A schedule of two joint monitoring field visits per MAPS NGO within a five-year MAPS II (timed where possible to coincide with MAPS field training or familiarisation, to keep costs lower). ■ Provision of external (i.e. out-sourced) technical support to strengthen the capacity of the Civil Society Section to monitor effectively and evaluate an expanded MAPS programme (replete with the lateral learning and outcome/ impact assessment features being recommended in this evaluation). It is thus envisaged that, in respect of each MAPS NGO, the Civil Society Section would draw together a team comprising the responsible Assistant Principal and Desk Officer, the MAPS Officer in the relevant MAPS NGO, and a ‘retained’ external expert whose professional profile would match the main sectoral focus of the MAPS NGO concerned. Each MAPS NGO would have one such dedicated team, to manage ongoing dialogue on policy and practice; to facilitate and arrange training and familiarisation exercises; to participate in joint field monitoring visits (together with representative(s) of the www.dci.gov.ie page 13 relevant NGO); to support both bilateral (Partnership Monitoring Committee) and multi-lateral (thematic) MAPS meetings; and to provide any benchmarking services required. ■ Continued refinement/improvement within MAPS NGOs themselves of the internal processes of strategic planning, policy development, programmatic and organisational development and change. 3.4.3. The findings from the evaluation show that the five MAPS NGOs have a variety of approaches to the relationships they have with their Southern partners. Key elements common to all are trust and a mutual value/respect. This is developed through interactive involvement between the MAPS NGOs and their Southern partners. 3.4.5. Whilst partnership is valued, it is time consuming and resource intensive to develop and nurture. The MAPS NGOs address this primarily through having field offices in the countries or regions they work in to provide support (though some agencies, such as Trócaire, are critically examining the cost/benefit implications of this structure). It is clear that the quality of partnership between the MAPS NGOs and local partners is influenced by the particular administrative mechanism through which the NGO maintains contact with that partner. 3.4.6. Capacity-Building For all of the MAPS NGOs, capacity building was seen to be an essential element of strengthening their Southern partners and supporting them to be effective. This was carried out in a number of different ways: GOAL, for example, regarded capacity building as focused on strengthening organisational processes, and management systems, and building the external links or networks of their partners; by contrast, Concern’s approach was very much to make capacity building ‘needs-led’. 3.4.7. A major difference between MAPS funding and block grants is that the former is multi-annual. Partners saw that this gave them added flexibility and stability – as foreseen in the MAPS objectives. GOAL found that, because they were still receiving MAPS funding on an annual basis, their partners’ stability was not in fact enhanced. (GOAL receives annual funding in MAPS because of difficulties which took place in its strategy planning). But overall, it is clear that having guaranteed funding over a number of years has important benefits for Southern partners, enabling them to plan more effectively and, critically, to enable them to retain their staff. 3.4. Southern Partnerships/Impact 3.4.1. Has MAPS funding actually strengthened the local partners of the MAPS- funded NGOs to be more effective in their work? It is important to reiterate a point made in the Inception Report, that this evaluation occurs near the end of the three-year cycle of MAPS funding. It is therefore rather early to be looking for impact, which is seen as longer-term and focused on change processes. However, it is relevant to look for ‘signs’ of impact and to do this we have focused on four key areas: partnership, capacity building, the transition from a project or activity focus to a programmatic or impact-oriented perspective, and monitoring and evaluation. 3.4.2. Partnership In assessing the impact of MAPS on Southern partners, an essential area is that of partnership. Partnership is considered important in helping to improve aid effectiveness and create a more collaborative framework for implementation of development programmes, in enhancing donor coordination and behaviour, and in building more local ownership of the development process. Partnership is a key principle that has been used to define the relationship between the more progressive donor NGOs and their implementing partners for over thirty years, and is also now being used by governments to define their funding relationships with civil society agencies, hence its centrality to MAPS. page 14 Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 3.4.8. There are clearly signs of MAPS funding having an impact on the work of Southern partners. One example is the PICOT (Partners in Conflict Transformation) network that Christian Aid Ireland supports in Sierra Leone which illustrates how networking is being used effectively, as was noted in the Partner Report: In Sierra Leone, the creation of the PICOT network (bringing together the three Christian Aid Ireland partners working on Conflict Transformation, two of whom are MAPS-funded partners) is providing these partners with the opportunity for learning, sharing and developing collaborative approaches to peace building work as well as creating linkage at national and regional levels. PICOT has been instrumental in convening national CSO forums for discussion and the sharing of experience as well as development of an advocacy agenda. This is an excellent example of how MAPS has been used to confer additionality over and above the core work programmes of the individual agencies, whilst in the process deepening the capacity and broadening the impact of those same agencies. 3.4.9. Project to Programme Transition A central feature of MAPS has been the focus on moving from a project to a programmatic way of working. This has been one area where the MAPS NGOs and their Southern partners have encountered the most difficulty, mainly in understanding the differences between the two approaches. In most cases, Southern partners and NGO field staff would understand a programme to be a collection of projects, grouped by theme or location, whilst Christian Aid Ireland saw that it would also include the ability to take a longerterm perspective. Some of the problems seem to revolve around the fact that projects would be seen as small, one-off activities which would not fit with an understanding of a programme as something bigger and longer-lasting. The MAPS Guidelines indicate that the key elements of a programmatic approach are that it is multi-annual, that it should make an impact on macro issues and that MAPS NGOs should be seen to be linking their work to the broader arena of policy dialogue and advocacy. Looking critically at the above elements it is clear that a programmatic approach to working has to demonstrate a ‘value added’ in comparison to a project approach. This point links straight back to the weakness of the monitoring and evaluation systems which have tended to collect data at the level of activities and outputs without clearly articulating what value they are trying to add at the level of outcomes and impact. 3.4.10. The lack of clarity in this area and the fact that it is a central aspect of MAPS would suggest that it is an important area both for Development Cooperation Ireland and MAPS-funded NGOs to address. It would be helpful for Development Cooperation Ireland to think through what its understanding of a programmatic approach is and what characteristics of that approach are essential elements of work that it supports through MAPS. 3.4.11. Monitoring and Evaluation The monitoring and evaluation of MAPS both by Development Cooperation Ireland and the MAPS-funded NGOs are weak; a point recognised by all of the key stakeholders. The findings of the evaluation team are that, whilst a number of agencies had monitoring and evaluation systems in place, these tended to focus on lower level outputs and activities rather than on outcomes and impact. This relates in part to a ‘project’ approach or mentality which would focus on activities rather than outcomes and impact or the bigger purpose/issues. This situation is gradually being addressed. An example of this is Trócaire’s Project Appraisal and Monitoring Form (PAMF), which incorporates baseline information on the situation analysis prior to the planned intervention and identifies appropriate indicators of performance and achievement; these tools facilitate subsequent impact assessment and make evaluations at later stages in the project cycle more meaningful. www.dci.gov.ie page 15 3.4.12. A number of the comments on the difficulty with monitoring and evaluation have focused around the need for appropriate indicators. In conversations with field staff and Southern partners, it was very clear that they understood the difference between assessing outputs, outcomes and impact, and could think of appropriate qualitative indicators and examples of data that could be used to assess outcomes and impact. The difficulty was in developing indicators themselves, especially when there was a need to develop proxy indicators – for more sensitive issues and areas. It is important to recognise that good indicators need good objectives (at the levels of outcome and impact) and that the problem of weak monitoring and evaluation often shows that there is not a clear enough logical link between the different levels of an area of work. 3.4.13. The issue of development of appropriate indicators for outcomes and impact that can help to assess changes in attitudes and practice is difficult for most organisations and it is not surprising that the MAPS NGOs have found this difficult both for themselves and their partners. This is an area where support and/or training could most helpfully be given. The key to the effective use of any monitoring and evaluation system is for it to be kept as simple as possible. This includes keeping the number of indicators down to a minimum. SHDI have begun to address this by holding training sessions for field staff on impact assessment, as part of their staff capacity-building process. The matrix below is of key importance as it illustrates the link between the different levels of work. The diagram clarifies the difference between outputs, outcomes and impact: Point of measurement Outputs Outcomes What is measured Effort Effectiveness Indicators Implementation of activities Use of outputs and sustained production of benefits Difference from the original problem situation (Fowler, 19976). Impact Change 6 Fowler, 1997, Striking A Balance: A Guide to Enhancing the Effectiveness of NGOs, Table 7.1, p. 164. page 16 Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 3.5. Cross-Cutting Issues: Mainstreaming 3.5.1. The experiences of the MAPS NGOS in mainstreaming the three areas of HIV/AIDS, gender and environment share some common features, but in all cases are mediated by the organisational culture, interpretations, policymaking processes and priorities. All of the MAPS NGOS have a wider list of cross-cutting issues than those of Development Cooperation Ireland itself, some with as many as seven. The evaluation has brought out the difficulty of mainstreaming in terms of achievement of objectives. The findings from the evaluation are in line with the analysis from the Comparative Study, which shows that when governments take a broad approach to funding, as the Department for International Development (UK) does through its Partnership Programme Agreements, it is not possible to relate specific project work, expenditures or outputs to the external funds, as such the funding is prone to being fungible. In order to mainstream HIV/AIDS, for example, it is necessary that the organisation funded has its own clearly articulated objectives. 3.5.2. In all organisations, mainstreaming is acknowledged to be work in progress; of the three cross-cutting areas under MAPS, most progress is evident in HIV/AIDS, with rather less progress apparent in the area of gender. Environment tended to be the most overlooked (except for one agency for whom environment is part of their main raison d’être). The increase in mainstreaming HIV/AIDS activities is not specifically linked to MAPS funding; however, the increase in dedicated staff capacity for mainstreaming at different levels of the agencies’ programmes is directly correlated with the advent of MAPS funding. In general, there was a correlation between the progress made towards mainstreaming gender and HIV/AIDS under MAPS by those agencies which also demonstrated a corporate commitment to impact assessment. 3.5.3. This first phase of MAPS reveals that the organisations have concentrated on development of policy on issues to be mainstreamed, such as gender. The prioritisation has varied, but most MAPS NGOs now have mainstreaming policies, albeit rather recent. In most of the organisations, significant work has been done in the area of policy development and, to varying degrees, this has involved country offices and field offices; as might be expected, the greater the breadth of such participation, the lower the risk of policy evaporation. These iterative processes were evident in Concern, Trócaire and SHDI. A downside of the iterations is that policy development can take a long time and, although field staff awareness is raised about the need for a policy, they may not immediately take action as they wait for a ‘definitive policy’ to be finalised. 3.5.4. In most of the fields visited by the evaluation team, agency staff and partners alike face a challenge in understanding and operationalising mainstreaming. An ‘evaporation’ problem was identified in terms of gaps in understanding between head office, field office, programme and sub-programme or project, and in terms of achieving a consistent level of understanding and approach to the methodology of mainstreaming. Some field offices have been developing their own initiatives, carrying out gender audits and recruiting gender officers and HIV/AIDS officers. It is too early to say what the impact of these might be but it is probable that these initiatives will contribute to organisational learning around the cross-cutting issues. All the organisations need to consider how they can address the ‘how to’ elements if they are to really address change in practice. 3.5.5. Although there is some monitoring of women’s participation in the field, it was not clear that this was leading to a more strategic approach to gender or the idea of changing power relationships which currently mediate the way project benefits can be enjoyed. The view from the field appears to be that gender is more easily tackled than equality. The gender policy www.dci.gov.ie page 17 document of Development Cooperation Ireland has a list of 11 key points on what is involved in mainstreaming (p.9). These could helpfully be used in the next round of MAPS as a basis for ensuring that, if mainstreaming is to be taken seriously by all parties, what is involved is clearly understood. 3.5.6. The agencies’ efforts to mainstream raise some interesting questions; firstly, in relation to crosscutting issues in the context of working with and through partners and, secondly, whether there needs to be a more nuanced and strategic approach to implementing an ever-increasing list of critical issues. There would appear to be a perennial challenge for all agencies about how to mainstream with partners without appearing to be overbearing or undermining mutual respect. The differing rates and degree of mainstreaming and individual organisations’ experiences provide an interesting opportunity for learning between Development Cooperation Ireland and the NGOs, and horizontally between the NGOs. The approach to mainstreaming has maintained the MAPS philosophy of respecting their partners’ own strategic priorities and offering support where necessary. Future MAPS Guidelines could be more explicit about this while also agreeing milestones with the individual organisations, on which they are monitored. 3.5.7. HAPS vs MAPS as a Mechanism for Mainstreaming HIV/AIDS All the MAPS NGOs (except SHDI) have been in receipt of HIV/AIDS Partnership Scheme (HAPS) funds as well as MAPS and, in many of the organisations, HIV/AIDS was found to be more effectively addressed than the other two MAPS cross-cutting issues. HIV/AIDS is considered core to MAPS and, although there has been a separate funding window, it was included in the terms of reference of this evaluation. Attributing any of the changes in HIV/AIDS to MAPS per se is difficult. Most organisations have commented that the more prescriptive nature of the HAPS policy on mainstreaming has helped them create capacity, and focus on this critical issue. In some organisations, there was confusion at the outset as to whether those in receipt of HAPS could include HIV/AIDS in their MAPS proposal (or vice-versa). In consequence, a pragmatic distinction appears to have been made, in that substantive activities are supported through HAPS funds, whereas some mainstreaming activities, such as staff development, are more likely to have been funded under MAPS. However, this observance is by no means consistent across the agencies, and there is consensus among MAPS NGOs that henceforth HAPS should be conflated into MAPS. A high level of confidence was expressed that HIV/AIDS occupied such a position of prominence in the agencies’ thinking and action that it no longer needed the ‘crutch’ of a dedicated funding window. The evaluation team found itself in agreement with the viewpoint, and recommends that the next round of MAPS have HIV/AIDS mainstreamed so that there is no need for a separate HAPS funding mechanism. This would ease the administrative burden of managing two separate but linked schemes. Points made earlier need to be borne in mind that effective mainstreaming needs clear objectives. These need to be for both agencies receiving MAPS funding and the Civil Society Section of Development Cooperation Ireland. 3.6. Lesson Learning 3.6.1. MAPS was seen as a portal to shared learning, dialogue and developing mutual understanding between Development Cooperation Ireland and the MAPS NGOs, and laterally among the MAPS NGOs concerned. Unfortunately, there appears to have been no obvious vehicle or forum associated with the MAPS process that enabled the MAPS NGOs to come together to engage with Development Cooperation Ireland staff in wider policy debates. Equally, there was no obvious mechanism associated with the MAPS process to encourage mainstreaming ‘best practice’ or to share lessons learned among the MAPS partners. Each agency, in some locations of its operations, has gained useful insights into mainstreaming cross-cutting issues, proceeding on an increasingly programmatic basis or working page 18 Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 through advanced models of Southern partnering. But the richness of learning and experience inherent in these instances of better practice is not being captured or harvested, worked into case studies, or presented in a learning context to the full MAPS partnership. 3.6.2. To some extent, responsibility for this weakness is shared between MAPS NGOs and Development Cooperation Ireland, as none of the agencies appear to have fully developed learning processes in their organisation. Learning is not a stand-alone product; it has to emerge from specific learning processes. These processes are critical if learning is to emerge from an interface between different levels of the organisation and is linked to what is referred to elsewhere as a general weakness in monitoring and evaluation. 3.6.3. There has been some degree of shared learning between Development Cooperation Ireland and individual MAPS NGOs on institutional and thematic issues as part of the quarterly bilateral Partnership Monitoring Committee meetings. However, there has been little lateral exchange of learning between the MAPS NGOs. The relative dearth of such interaction was even more striking at field level. Whilst there have been occasional meetings of MAPS NGO personnel at country level, these have tended to be in countries where Development Cooperation Ireland itself maintains a mission (e.g. in Tanzania and Ethiopia7). In Sierra Leone (where a Development Cooperation Ireland office has just recently opened), there has been virtually no round-table gathering of MAPS NGO representatives, nor was there even regular contact between two MAPS NGOs (Trócaire and Christian Aid) who were both utilising MAPS to support one local partner simultaneously (albeit for distinct activities). MAPS may have helped to facilitate some additional dialogue between sectoral specialists in the partner NGOs at head office level, but it is likely that this would have happened anyway through existing contacts (especially relating to emergency and relief interventions) or through the Dóchas interagency network. 7 3.6.4. If all the stakeholders continue to want learning from both the MAPS process and content of their work, this could be prioritised and focused on areas of common interest which have emerged during the course of this evaluation as a starting point for a joint learning agenda. Development Cooperation Ireland and Trócaire are promoting the meetings amongst the MAPS NGOS in Ethiopia but it is not clear whether there exists among all MAPS NGOs in Ethiopia a strong shared vision about their purpose. Incidentally, there is an intention to expand this round-table forum to include DFID and Partnership Programme Agreements organisations, e.g. Oxfam. www.dci.gov.ie page 19 4. SUMMARY OF THE COMPARATIVE STUDY 4.1. Introduction to the Comparative Study 4.1.1. A comparative study of the funding schemes of a number of other European official agencies was included as a part of the terms of reference in order to gain a wider perspective on how the approach taken by Development Cooperation Ireland with MAPS compares to other schemes and to draw lessons from them for the future development of MAPS. The Comparative Study has been carried out as a desk review of the literature on seven European official agencies. The full study is attached as appendices to this report. This section summarises the main points and draws out the issues and potential lessons for MAPS. 4.2.2. Funding through NGOs has changed over the years, with reductions in funding as support has been targeted directly towards Southern NGOs. Whilst there had been concern that this would weaken the position of Northern NGOs, the fact that it has not underlines the importance of the domestic political agenda. 4.2.3. One of the most significant changes has been in the way in which NGOs have been funded by official agencies. In part, this has been as a response to a need to reduce the transaction costs of funding, as civil services across Europe try to save on costs through an overall reduction in staff numbers. 4.2.4. The vast majority of official agency funding schemes started by providing grants for specific projects on a co-funded basis. Co-funding often meant a 50/50 funding split between the official agency and an NGO. Co-funding schemes typically involved quite small proposals and took a lot of administrative effort to manage. As a result, as they grew in size and numbers they began to give way to other schemes, the most common being some kind of block funding or framework agreement. 4.2.5. An area of continuing concern has been the question as to whether official agencies should continue to fund ‘operational NGOs’ – the argument being that it is neither sustainable, nor good developmental or capacity-building practice for international NGOs to engage directly in development programmes rather than supporting local organisations to take over this role. 4.2. Background to Official Agency Support for NGOs 4.2.1. Official agencies fund NGOs for a number of different reasons: ■ As a way of channelling further resources directly to poor people rather than through the State; ■ As a way of funding personnel sending agencies (volunteers); ■ Support for publicly popular agencies; ■ As a way to gain public support for the overall aid budget through development education, bringing NGO and official agency interests in alignment and ensuring the NGOs lobby for more aid (0.7% campaigns are a prime example); ■ Promoting the countries’ interests and a positive image of the donor country. page 20 Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 4.3. Types of Block Grant or Framework Scheme 4.3.1. The terms ‘block grant’ or ‘general support’ have been used to describe different approaches to funding by the official agencies. A difficulty which the authors of the Comparative Study struggled with was that there is no agreement on the meanings of terms such as ‘framework agreement’ and ‘partnership agreement’. The outcome is that terms to describe the different approaches/schemes vary between official agencies and even in different documents relating to one agency’s scheme. 4.3.2. Block grants were generally an amount of money given for a set period, normally one year, within which projects would be reported on in detail. Reporting would either be on an individual project basis or some form of consolidated report. 4.3.3. Framework or partnership agreements would be general support to an agency based on an overall organisational assessment and sometimes agreed outcomes. The level of negotiation over these types of support would vary greatly: ■ formalised reviews of lists of projects; ■ agreeing country priorities; ■ thematic considerations; ■ past performance or track record; ■ numbers of domestic members; ■ formal evaluation procedures. Of these framework agreements, the DFID Partnership Programme Agreements (PPAs) probably impose the least onerous requirements in terms of control, with reporting based on general outcomes which would be set in general strategic terms. 4.3.4. The annual reports for this type of framework agreement refer to outcomes or strategic objectives but strictly speaking do not correlate them to the degree one would expect for example in a log frame. It is for this reason that the official agency (DFID in this case) needs to have trust and confidence in the funded NGOs. This is relevant for the development of MAPS as the partnership aspect of the process has assisted the development of a good degree of trust and understanding between Development Cooperation Ireland and the MAPS-funded NGOs. 4.3.5. An issue with this type of framework approach is that it is not possible to relate specific project work, expenditures or outputs to the external funds provided as such funding is entirely fungible. Experience has shown that the use of such funds does not encourage the mainstreaming of priority themes, geographic focus or issues. This is clearly an issue for Development Cooperation Ireland which has prioritised the mainstreaming of HIV/AIDS, gender and the environment as critical issues to be addressed. Experience from other agency schemes highlights that if it is necessary to mainstream issues then it is essential that the funded agencies themselves articulate those issues as specific priority areas with their own sets of outcomes and strategic objectives. 4.3.6. Adaptations of Framework/Partnership Schemes MAPS is an example of a scheme which has characteristics from various framework approaches. In essence, it has a lot of similarity to DFID’s PPAs, with a focus on outcomes and impact and longer-term support, whilst at the same time reporting on projects using MAPS money, which would appear to be a carryover from the block grant approach. 4.3.7. Institutional Funding Agreements A further development of general support are institutional funding agreements (the Netherlands is an example of this approach) where an overall budget is agreed by parliament and then divided amongst the co-funded agencies on a proportional basis. An innovation from this type of scheme is the development of ‘thematic funding’, which is perhaps closer to the MAPS and PPA approaches. www.dci.gov.ie page 21 4.4. Criteria for Choice of Agencies 4.4.1. A controversial issue in most of these schemes centres around the choice of agencies for inclusion/special treatment. Many of them share the common tendency to be accused of a lack of transparency and a failure to set clear criteria for those who should be included or excluded. A common criterion is ‘track record’ or previous involvement in other official agency schemes, for example involvement of MAPS-funded agencies in block grant funding was a criterion for their consideration for MAPS funding. 4.4.2. The confusion or lack of clarity over the criteria for involvement in different funding schemes can lead to them becoming ‘frozen in time’ with the same agencies being supported year-on-year whilst those outside aspire to join the partnership. 4.5.3. A more recent proposal, now under consideration by DFID, involves an evaluation function being built into the Partnership Programme Agreements, whereby representatives of the particular NGO, staff of the official agency and an external representative would jointly evaluate the quality of an agency’s use of DFID funding, against outcomes. The results of this evaluation would help to make the decision in following submissions. 4.6. Mechanisms for Managing Funding Schemes 4.6.1. Contracting Out. One response to the need for official agencies to reduce their costs is for them to contract out various aspects of the management of funding schemes. This is done in several ways: ■ Private contractors are used to deal with different aspects of the funding scheme. For example, DFID use a consultancy firm to carry out the appraisal of new proposals to their Challenge Fund. Subsequent grant management is then carried out in-house by DFID. In some instances, a Resource Centre would review and comment on proposals within their area of expertise. Individual consultants are used in a number of situations to monitor and accompany an agency. A danger is that the consultants can lose their independence. There is a need therefore with such approaches to renew and circulate the consultants involved. Umbrella/membership/framework agencies are used particularly where there are a large number of smaller NGOs. Finland and Sweden are two countries have adopted this approach. The umbrella agencies carry out the proposal management and follow-up for the official agency. Most umbrella agencies would be based around a common interest (church, trade union, etc.). Whilst umbrella agencies have a 4.5. Monitoring and Evaluation of Funding Schemes 4.5.1. Building on the previous point, the issue of monitoring and evaluation of the various schemes has been fraught with difficulties. The Dutch developed a system of programme evaluations which would take key themes and then identify geographic areas in which to investigate how the co-funded agencies worked in those themes. This has potential interest for the next round of MAPS as it could help to address the issue of learning and appropriate and useful monitoring and evaluation systems. 4.5.2. Swedish Sida found it difficult to evaluate their support for NGO work as they had created too many links in the chain, with their system of umbrella organisations making it virtually impossible to evaluate impact. This is clearly an argument for keeping the structure as well as the criteria of a co-funding scheme as simple as possible. ■ ■ page 22 Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 number of advantages they also have important disadvantages and costs. Two of the main problems are difficulties in monitoring and evaluation (the umbrella agencies do not have the capacity to carry this out) and possible conflict of interest between grant managers in an umbrella organisation and their members. ■ Lines of accountability and criteria for contractors to use need to be between the contractor and the donor; ■ Minimise budget lines. Too many budget lines can be confusing, expensive to administer and counterproductive. Too few lines may exclude new ideas and groups and may make it difficult to mainstream priorities (gender, environment and so on); ■ Need for an organisational assessment. Without such an assessment it is difficult to judge or defend some of the decisions made to include agencies in this sort of scheme; ■ Ensure that there is a clear and appropriate monitoring and evaluation system linked to clear objectives and indicators. Several existing schemes seem to have sacrificed quality and monitoring and evaluation (which of course are all interrelated) in the face of the need to reduce transaction costs as well as to ensure a certain political profile for funding. This is a shortterm saving as future investigators will query why some decisions were made on the basis of such a weak assessment of agencies’ capacities and the impact of their work. 4.7.3. To summarise the lessons from the Comparative Study, it is clear that the better schemes have managed to ensure clarity over choices of allocation of funds; a commitment to quality through participatory development, monitoring and evaluation; and a focus on strategic goals which are relevant to the needs of poor people in developing countries. 4.7. Learning From the Different Approaches 4.7.1. A Single Model of Good Practice or a Set of Principles? It is clear from the Comparative Study that there is no single model which can be applied to the needs of Development Cooperation Ireland as it moves to develop MAPS. There are, however, a number of principles which can be placed against the political and fiscal realities faced by donors. The most important constraint at the moment is the restriction on expanding their capacity through more staff whilst also needing to cut back on costs (often through a process of decentralisation). The following principles are helpful as they highlight a number of the issues that have come up in the evaluation of the MAPS process. 4.7.2. Principles for Choice of System/Approach: ■ Clear criteria to choose agencies for framework type programmes, and for funding decisions; ■ That framework/partnership/block grant schemes need indicators of success. If these are around an agency’s overall performance, is it clear how they will be measured? If around specific outcomes are these specific enough to be identified for the purposes of monitoring and evaluation? ■ Mechanisms need to allow for changes to avoid making allocations which may become out of date, constrain new entrants and make it difficult to reduce funding to less relevant agencies; www.dci.gov.ie page 23 5. CRITERIA FOR SELECTION FOR A FUTURE MAPS PROGRAMME The criteria for allocation of funding in MAPS I were quite broad. It was recognised by all concerned that (a) MAPS was so innovative that it would necessarily involve some degree of ‘learning by doing’, and (b) that one of its core characteristics of flexibility would have been negated, the more prescriptive its Guidelines were. To a large extent therefore, a leap of faith was required by both parties (especially Development Cooperation Ireland) in order to get the Scheme up and running on the basis of a fairly open set of Guidelines, at least for a pilot phase, subject to this joint evaluation taking place after two full years of operation. This resulted in a certain lack of clarity/transparency as to the basis on which funding decisions had been made by Development Cooperation Ireland, and – from the NGOs’ perspectives – widely differing assumptions as to the standards which would be applied in using MAPS funding (e.g. on mainstreaming of environment). 5.1. It should be noted that the decision on whether or not an applicant NGO joins MAPS is taken by the Programme Appraisal and Evaluation Group (PAEG) in response to a submission containing a detailed fully costed programme. The criteria suggested here will not therefore lead to becoming a MAPS partner, but relate to the prior stage of being invited to make a submission. They are: Focus of Work: the agency is engaged in development programmes in the poorer countries (in general, within the bottom 50 of the Human Development Index) with a clearly defined poverty focus working with and through Southern partners; Charity Registration: the agency is registered as a charity or a non-profit organisation in Ireland for a period of at least five years prior to its MAPS application; Financial Accounting: satisfactory outcome to an independent Financial Systems Analysis of the agency and furnishing of audited accounts for each of the three financial years prior to its MAPS application; (iv) Funding Base: to safeguard against overdependence on a single income source, each agency should undertake to maintain a ratio between the annual total of its funding from all Development Cooperation Ireland funding schemes (excepting those designed to assist victims of sudden-onset emergencies) on the one hand, and funds raised from all other sources (excluding sudden-onset emergency appeals) on the other; this ratio not to exceed 70:30 in favour of the former; Capacity: the scale of the agency’s existing programme should provide sufficient critical mass for MAPS and the agency should demonstrate capacity to link their work with wider policy and advocacy work and to implement MAPS development criteria in issues such as poverty focus, mainstreaming and Southern partnering; Support: the agency demonstrate professional capacity to give operational effect to its strategic plans: to implement programmes rather than projects; to report outcomes and impact rather than activities and outputs; clearly defined partnership focus and approach. In the case of international NGOs, sufficient programming authority and expertise resides in the Irish entity to ensure it has the capacity to generate the level of policy coherence necessary for MAPS; Experience: previous significant experience and relationship with Development Cooperation Ireland over a five-year timescale providing a foundation for both the agency and Development Cooperation Ireland to go forward on the basis that the high level of partnership and trust implicit in MAPS will apply; Distinctiveness: in the case of international NGOs, the MAPS funds remain distinct and fully traceable; with their outcomes distinct and clearly visible, and when Development Cooperation Ireland dialogues with the Irish entity, it is in contact with the real decisionmakers on issues of MAPS expenditures; (v) (vi) ■ ■ ■ (vii) (i) (viii) (ii) (ix) (iii) page 24 Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 (x) (xi) Coherence: the existing programme of the agency demonstrates a significant degree of coherence with the MAPS priorities; Reporting: the agency has the capacity to report at the levels of outcomes and impact and is willing to report in formats required by the DAC Codes (of the Development Assistance Committee of the Organisation for Economic Co-operation and Development). The MAPS workshop which reviewed a draft of this Synthesis Report criticised a lack of transparent criteria for MAPS I, not only on the question of selection of NGOs to become MAPS partners, but also on the relative size of financial allocations to the agencies selected. The evaluation team considered this and suggests that the key factor in criteria for selection and financial allocations for MAPS, is performance8 against MAPS priorities and policies; that this can reasonably be assessed by Development Cooperation Ireland on the basis of a written application, strengthened by a (subjective) view of the previous quality of partnership. In making an assessment, the ‘history’ of involvement with an NGO is an important basis on which trust in its capacity and capability can be assessed – albeit subjectively. This is an approach taken by DFID, which re-examines its relationship over many years with each applicant and decides to what extent that partnership has been characterised by an adequate level of trust. The danger in this case is of the basis for allocation decisions becoming out of date and not taking enough account of changing situations, a point made in the Comparative Study. The new proposal, now under consideration by DFID, which involves an evaluation function being built into the Partnership Programme Agreements with a joint evaluation of the quality of an agency’s use of DFID funding against outcomes, is a potentially helpful approach as it builds on the trust and applications made by the NGOs. The results of this evaluation would help to make the decision in following submissions and would address the problem of funding allocations being outdated. 5.6. Given the timeframe, it is doubtful that highly detailed assessments pre-allocation, which would accurately demonstrate the technical capacity and ability of MAPS II applicant NGOs, could realistically be completed (given the likely MAPS I/MAPS II timeframe). A possible criterion for allocation therefore is the scale of an NGO’s nonemergency income. This is not an indicator of capacity or commitment to either strategy or particular policies. But it provides a basic comparator and can reasonably be used in conjunction with more relevant variables such as commitment to strategy, a programmatic approach, or mainstreaming the cross-cutting issues if a method of comparative (or competitive) measurement of these variables can be agreed. If necessary, a simple weighted scoring grid can easily be devised and, to balance the subjectivity with objectivity, an external assessment can be undertaken, with recommendations as to specific allocations. 5.2. 5.3. 5.7. 5.4. 5.5. 8 Performance can be reviewed against previous use of MAPS funding, use of block or co-funding or previous track record of an agency. www.dci.gov.ie page 25 6. ANALYSIS AGAINST MAPS OBJECTIVES MAPS has three stated objectives against which the evaluation team has approached its analysis of the Scheme to date. This analysis proceeds from a generic and independent perspective on the six-member partnership rather than from a standpoint which is confined to any of its individual constituents. extent) through the NGO Co-financing Scheme, would have involved much greater costs for both the donor and the recipients. Conversely, without a mechanism such as MAPS, the same level of management capacity across the six partners could not have coped with the equivalent volume of aid activity. 6.1.3. There has been a reluctance to innovate within MAPS I or to put forward for MAPS funding those programmes seen by MAPS NGOs as potentially risky, even though Development Cooperation Ireland had indicated its openness to some risk-sharing if the potential benefits are commensurate with the risk. In general, the MAPS NGOs have adopted a rather conservative stance, partly because of concerns surrounding accountability, but partly also to ensure that such an unprecedented source of stable, long-term funding would not be jeopardised in any way. 6.1.4. This first stated objective of MAPS also refers to the MAPS NGOs’ ‘own programme framework of visions and strategies’. The finding of the evaluation team is that, in this context, the objectives are being successfully achieved. No instance arose in which a MAPS NGO felt pressure from Development Cooperation Ireland to act outside its own visions and strategies, nor has there been pressure to standardise. On the contrary, there has been a strong commitment overall, within Development Cooperation Ireland, to avoid a ‘one-size-fits-all’ approach. The evaluation team has found therefore that the diverse visions, cultures, priorities and emphases of the five are preserved and valued within MAPS, and this has given the venture significant strength. 6.1.5. The third part of this first stated objective refers to the promotion of flexibility within the MAPS NGOs’ framework of vision and strategies in so far as these are in line with the Irish Government’s policy of development 6.1. The first stated objective is: To facilitate receipt by NGOs of Development Cooperation Ireland funding in such a way as to promote flexibility within their own programme framework of visions and strategies, in so far as these visions and strategies are in line with the Irish Government’s development cooperation policy. 6.1.1. This objective is broadly being achieved; flexibility was repeatedly mentioned by the MAPS NGOs as a great merit of the Scheme, in that (i) MAPS NGOs were free to include programmes in their initial MAPS submissions, and (ii) were subsequently free to transfer funds between budget heads up to a limit of 20%. That said, the partner NGOs differ from each other in how they use this flexibility. For example, whilst they have all used MAPS funding to scale up their work, some more than others have invested in capacity building within their programmes. 6.1.2. There is broad agreement also regarding the reduction in transaction costs and scale-offunding advantages conferred by MAPS. The contribution of MAPS to lowering the unit cost of managing the €117 million allocated is significant, taking into account the combined costs incurred both within each MAPS NGO and within Development Cooperation Ireland. To have successfully disbursed the same amount through block grants or (to an even greater page 26 Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 cooperation. These issues of policy coherence are elaborated in the MAPS Guidelines. Two processes are taking place. The first involves actual on-the-ground functional and sectoral policy of Development Cooperation Ireland, and this is the basis on which the MAPS partners seek to implement a poverty focus and to mainstream the three primary cross-cutting issues of HIV/AIDS, gender and environment. Some progress has been made in this regard, although to varying degrees across the agencies; for example, progress on environment has in some instances been negligible. 6.1.6. The second process is less sectoral and more strategic. In this process, the policy preferences of Development Cooperation Ireland for stronger strategy planning, programming and attention to macro-level outcomes and impact are intended to be transferred to the partners. Our finding is that there has been movement in this direction by all six partners and that this movement has been accelerated as a result of MAPS-related processes, but that there remains some distance to go towards fulfilment. It would seem that, at the outset of MAPS, assumptions were made about agencies’ capacity in this regard, but that the monitoring processes (in the agencies and in Development Cooperation Ireland) were generally not adequate to identify shortcomings. The ulterior aspiration implicit in MAPS that a ‘programme ethos’ should percolate through to Southern partner NGOs is considered to be at a very incipient stage indeed. The potential for fruitful effort in this regard in future phase(s) of MAPS is therefore considerable. 6.1.7. The key obstacle which has hampered progress from a project focus to a programmatic basis is inadequate understanding of precisely what is involved. Agency staff (and Southern partners, with even greater force) are long accustomed to thinking and working in the frame of projects. For many years, and indeed decades, this construct was perceived as adequate for most purposes. The programme focus is younger, newer and involves not only the comprehensive taking-onboard of different macro concepts (how the national anti-malaria campaign is going, when one is fighting the disease in a cluster of four isolated villages), but also the sudden abandonment of information described for years by the donors as of critical importance (precisely how much was spent on seed and fertiliser, or exactly how many people attended the MotherChild Health Clinic in a given week?). Understanding (and use) of the programmatic approach is increasing throughout MAPS; but more time, structured familiarisation and clearer definitions are needed. 6.2. The second stated MAPS objective is: To permit NGOs with a proven capacity to act in a predictable and coherent framework in so far as funding from Development Cooperation Ireland is concerned, thereby allowing longer-term relations to be established with partners in the South. 6.2.1. All the NGOs invited to participate in MAPS were perceived by Development Cooperation Ireland to be well-established agencies with proven capacity (although no independent evaluations were carried out to put this assumption to the test). The question as to whether MAPS has permitted them to act in a predictable and coherent framework can be considered from two perspectives. 6.2.2. Firstly MAPS in its design was an innovative concept overall, with a series of interlocking new features, each representative of international best practice. Its objectives were such that, if fully implemented, the Scheme would constitute a very effective framework indeed for coherent and predictable development work by the partners in the Scheme. At the functional level, MAPS NGO field staff and Southern partner staff would be aware of the best practice methodologies and would be mainstreaming the cross-cutting issues in every programme being funded by MAPS. At the management level, all the programmes would be operating in compliance with coherent strategy plans, and impact and outcomes assessment would be ongoing. www.dci.gov.ie page 27 6.2.3. Some elements of this conceptual framework are now in place. Although each year, since the inception of MAPS, has brought the totality of its operations and programmes incrementally closer to broad objectives in the MAPS Guidelines, reality – not surprisingly – falls some distance short of full attainment. In any event, the evaluation team finding is that the movement towards realisation of the concept vindicates the validity of MAPS per se. 6.2.4. Secondly, a defining feature of MAPS is that it is multi-annual (notwithstanding the restricted application of this in the case of GOAL). This should allow, undoubtedly, for greater predictability and coherence by a MAPS NGO in designing and implementing its programmes, while strongly facilitating the paradigm shift from projects to programmes. It also facilitates effective capacity transfer in the field, and ‘managed’ exit where and when this is indicated. The multiannual attribute of MAPS is also intended to permeate to Southern partners, ‘allowing longerterm relations to be established with partners in the South.’ The evaluation found that this is being achieved more by exception than by rule and that, in general, MAPS NGOs are not yet passing the full benefits of MAPS down the line to their Southern partners. Some MAPS NGOs were imposing on their partners an unduly tight reporting frame, irrespective of the capacity or track record of the latter, which illustrates that they were still adhering to an operational/ management approach to the partnership continuum. Thus, the degree to which partnership is passed on to Southern partners would seem to correlate to the degree to which partnership is a goal of the individual MAPS agencies. 6.2.5. Multi-annual funding needs to be of sufficient duration to fit agencies’ strategic planning cycles and processes. Five-year plans are common and our recommendation is that in MAPS II a partner NGO remains so for five years, although every agency need not begin and end on the same date. This will also support both a meaningful mid-term evaluation and an impact assessment at the end of five years, when it should be possible to catalogue significant change. Bringing agencies into the Scheme at different times will also lessen the burden on the Civil Society Section at the beginning and end of the five-year period; and expanding the MAPS duration to five years will provide further scope to allow, in the words of this second MAPS objective, ‘longerterm relations to be established with partners in the South’. 6.3. The third of the initially stated objectives is: To enhance and strengthen dialogue and mutual learning between Development Cooperation Ireland and the NGOs in matters relating to strategies, approaches and the adoption of best practice. 6.3.1. The evaluation team’s finding is that the mechanisms for dialogue are weak and that a future MAPS would benefit greatly from their strengthening. It would be wrong, however, to assert that no efforts have been made by all the players. The Partnership Monitoring Committees are seen as a significant contribution to dialogue as well as to reporting and accounting and, particularly over the period since mid-2004, it has been possible to devote more time to thematic questions. In addition, six-way meetings were held in the earlier period of MAPS 9. It is significant that, in these meetings, participation by the MAPS NGOs was strong and that the Technical Section of Development Cooperation Ireland was strongly represented. Topics included evaluation methodologies, optimising on the role of the Partnership Monitoring Committees, monitoring, learning and accountability, environment, tracking change, joint evaluations, audit requirements, the role of monitoring and evaluation in policy development, and so on. 9 There were three meetings of all partners in 2003 (on 20 May, 1 July and 17 October). page 28 Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 6.3.2. Whilst participation in meetings and debates can be regarded as a necessary condition for institutional learning, it is clearly not a sufficient one; what is more important is the outcome, in terms of what lessons are transmitted back to and absorbed by the organisations. A difficulty here appears to have been a lack of clarity or objectives from the side of Development Cooperation Ireland as to what learning processes MAPS should engender. There have been no publications on the ‘state of the art’ of development praxis of Irish NGOs which are part of MAPS. Learning appears to have been included in the objectives because it was laudable, but the processes in the MAPS NGOs were not sufficiently geared up for this and there was no evidence that MAPS has really changed this beyond random encouragement. 6.4.3. A review of the annual financial reports of the MAPS NGOs for 2003 highlights a number of issues which this lack of clear guidance raises: ■ Not all partners provide breakdowns of expenditure by country;10 ■ HIV/AIDS is not always reported if it is funded under HAPS. More significantly, a number of MAPS NGO figures do not take account of other sources of Development Cooperation Ireland funding; ■ Some MAPS NGOs (but not all) show the percentages of their own funding alongside those of MAPS, Development Cooperation Ireland and other bilateral and multi-lateral sources; this is helpful. Some do not show other sources at all or give very little detail; ■ There is no information on gender expenditure, although disaggregation of data is part of Development Cooperation Ireland’s gender policy. Overall, on the basis of the accounts submitted, it is very difficult to gain an overview of how MAPS is being used or to compare data between the MAPS NGOs. 6.4.4. Development Cooperation Ireland needs to have confidence that partner NGOs’ accounting procedures and systems are effective, credible and reliable. There are five different accounting systems in the five MAPS NGOs, which complicates the delivery of satisfactory accounting in a manner which is easily intelligible to Development Cooperation Ireland. In some cases, to quote one example reported to the evaluation team, the MAPS NGO considers the money to have been ‘spent’ once it leaves Dublin. 6.4.5. Aside from the accountability requirement resting on Development Cooperation Ireland, some additional issues have surfaced relating to financial reporting in MAPS. One is that computerisation of accounting needs to be taken further to avoid wasteful repetitious data entry. 6.4. Accountability 6.4.1. The MAPS Guidelines (2001) gave a very brief outline of reporting requirements for both financial and narrative reports, with no specific system developed or suggested (although they acknowledged the need to maintain a balance between learning and accountability). Early meetings of the Partnerhip Monitoring Committees dealt with clarifications on what was appropriate and adequate in order to maintain a balance between working with the systems that NGOs had already in place and meeting the needs of Development Cooperation Ireland. 6.4.2. Development Cooperation Ireland receives annual narrative reports from each MAPS NGO. There has been some improvement in the quality of these reports, most notably from Christian Aid and Concern during the lifetime of MAPS. The reports are used by Development Cooperation Ireland as support for accounting for its distribution and use of MAPS funds, and this in itself is valuable; otherwise, they form the basis for discussion at meetings of the Partnership Monitoring Committees. 10 Development Cooperation Ireland requested that MAPS NGOs send details of expenditure by country in Years 1 and 2 of MAPS. www.dci.gov.ie page 29 A second is that Development Cooperation Ireland is annually required to submit a report which analyses its expenditures under ‘DAC Codes’ to allow international comparisons to be made by the Development Assistance Committee of the Organisation for Economic Co-operation and Development. Accounting information from the MAPS NGOs is not normally presented in this DAC Code format, resulting in a major transcription effort using a lot of Civil Society Section staff time. 6.4.6. There is a delicate balance to be struck between Development Cooperation Ireland prescribing more comprehensive accounting and reporting requirements for MAPS, on the one hand, and on the other, preserving that very characteristic of MAPS which is most cherished by the recipients, namely (in the words of the MAPS Guidelines) the need ‘to facilitate receipt by NGOs of Development Cooperation Ireland funding in such a way as to promote flexibility within their own programme framework of visions and strategies’. Development Cooperation Ireland would understandably like a standardised format of presentation of accounts. Would compliance be enforced? Would it cover MAPS only, or more? If MAPS only, it could then in fact create a disjointed system within the finance office of a MAPS NGO, with one system for that agency’s MAPS funding and other systems for everything else. 6.4.7. That said, Development Cooperation Ireland and the MAPS NGOs share an absolute need for accountability requirements to be clearly agreed and met. For Development Cooperation Ireland, these are public funds, taxpayers’ money, voted by Dáil Éireann and fully subject to the requirements, firstly, of the Department of Foreign Affairs as the accounting authority and secondly of the Comptroller and AuditorGeneral. The MAPS NGOs, on the other side of the partnership, are without exception registered charities for whom financial probity is imperative. There is a coincidence of interest in adherence to good accounting practice, which is precisely the reason for the prudent decision by Development Cooperation Ireland at the outset of MAPS to commission an independent financial systems analysis in each of the five prospective MAPS NGOs. This procedure should be followed for future new entrants into MAPS. 6.4.8. On the accountability side, there is already welcome movement toward the kind of guidelines required for a MAPS II. The 2003 MAPS Guidelines did not specify a particular format for financial reporting. Agencies were allowed to use the existing reporting formats, with some dialogue in early Partnership Monitoring Committee meetings on how they could be improved. At present, nevertheless, the lack of standardisation across the five MAPS NGOs on presentation of financial data complicates the preparation of breakdowns on how MAPS funding is or has been spent, although a recent (January 2005) request for information broken down by country will improve the Development Cooperation Ireland overview. For the next phase of MAPS, it is recommended that all agencies should comply with a summary presentation of their expenditure data according to an electronic format standard template, on the basis that this: ■ would not impose major new burdens on the MAPS NGOs; ■ would meet Development Cooperation Ireland’s legitimate needs; ■ would make greater use of accounting IT; ■ would fit in with the various different systems in use in MAPS NGOs; ■ would seek to address questions of Development Cooperation Ireland’s other information needs; ■ would use a format compatible with the DAC Codes requirement. 6.4.9. A wider concept of accountability could also embrace assessment of efficiency and effectiveness of implementation, and it is arguably in these areas that the greater risk lies. The risks incurred by the Scheme in this context were the following: page 30 Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 ■ There would be insufficient familiarity with concepts such as partnership, mainstreaming or the programmatic approach, to permit their universal deployment in MAPS within reasonable timeframes; in particular, understanding among all six stakeholders of the projects-to-programmes transition (rightly entitled a paradigm shift) would be superficial and unsatisfactory, consisting primarily in the aggregation of clusters of existing similar projects (‘more of the same’). ■ The quality of the inter-agency relationships between the MAPS NGOs would prove inadequate for a partnership of the kind that MAPS requires; NGOs have always competed, and have not always held each other in the highest of esteem. In the event, this has not been a problem (and indeed, as suggested elsewhere in the evaluation, the relative income security provided by MAPS may have been conducive to more collaborative approaches). ■ High staff turnover, and in particular the staffing constraints which apply in Development Cooperation Ireland and, in particular, in the Civil Society Section (insufficient numbers of staff, large volume of different responsibilities for each in the different grant schemes, very limited previous development experience, relatively high turnover, limited experience in managing a partnership-based scheme such as MAPS, etc.) would significantly affect the possibilities of the stated objectives being achieved. Although apprehensions existed at the outset of MAPS about the risks inherent in it, the overall view of the evaluation team is that the risk factors have been sufficiently well managed, have been worth taking, and (in general), have not destabilised the underlying sense of trust and confidence which is key to the satisfactory implementation of a partnership arrangement of the MAPS scale and scope. 6.5. Monitoring and Evaluation 6.5.1. Monitoring and evaluation of MAPS both by Development Cooperation Ireland and the MAPS-funded NGOs are weak, a point recognised by all of the key stakeholders. The findings of the evaluation team are that, whilst a number of agencies had monitoring and evaluation systems in place, these tended to focus on lower level outputs and activities rather than outcomes and impact. This situation is gradually being addressed. An example of this is Trócaire’s Project Appraisal and Monitoring Form (PAMF), which incorporates baseline information on the situation analysis prior to the planned intervention, and identifies appropriate indicators of performance and achievement; these tools facilitate subsequent impact assessment and make evaluations at later stages in the project cycle more meaningful. 6.5.2. A number of the comments on the difficulty with monitoring and evaluation have focused around the need for appropriate indicators. In conversations with field staff and Southern partners, it was very clear that they understood the difference between assessing outputs, outcomes and impact, and could think of appropriate qualitative indicators and examples of data that could be used to assess outcomes and impact. The difficulty was in developing indicators themselves, especially when there was a need to develop proxy indicators – for more sensitive issues and areas. 6.5.3. The issue of development of appropriate indicators for outcomes and impact that can help to assess changes in attitudes and practice is difficult for most organisations and it is not surprising that the MAPS NGOs have found this difficult both for themselves and their partners. This is an area where support and/or training could most helpfully be given. The key to the www.dci.gov.ie page 31 effective use of any monitoring and evaluation system is for it to be kept as simple as possible. This includes keeping the number of indicators down to a minimum. SHDI has begun to address this by holding training sessions for its field staff on impact assessment as part of the staff capacity building process. 6.5.4. For Development Cooperation Ireland, monitoring involved reviewing the reports submitted by the agencies and carrying out a monitoring visit to one of each agency’s countries of operation (with the exception of Christian Aid Ireland where planned visits failed to materialise due to extraneous factors). The present system of monitoring is based on reporting progress at agency level. This clearly presents advantages from a basic viewpoint in administering the scheme. But there is little or no assessment of the value added of taking a programmatic approach. Development Cooperation Ireland in effect can be seen as treating MAPS as four large multi-annual and one large annual block grant, with NGOs being encouraged to move towards assessing their outcomes and impact but little by way of similar movement on the part of Development Cooperation Ireland. The gap here is that Development Cooperation Ireland did not develop a set of objectives and indicators for themselves on which to measure the progress and outcomes of the Scheme. 6.5.5. If MAPS I is indeed a pilot scheme, with hopes of consolidation in the period to follow, the evaluation team finds that a new, clearer, more incisive set of objectives will be required, together with a supporting framework of indicators with which to assess its impact and contribution. 6.6. Implications for Non-MAPS NGOs 6.6.1. The evaluation team contacted the non-MAPS NGOs in a variety of ways.11 The main finding to emerge was that, although the five agencies then in the block grant scheme were invited to apply for MAPS I, and although the three agencies now (June 2005) in the block grants would wish to be members of a future MAPS, our recommendation is that the issue of membership of MAPS should be decoupled, both in practice and perception, from the question of participation in the block grants. Flexibility should be preserved both for block grant NGOs which in the future may not wish to join a MAPS and for Development Cooperation Ireland to select ‘candidates’ on more appropriate criteria than simple involvement in the block grant scheme. 6.6.2. Other findings emerging from these consultations were as follows12: ■ There is a view among the non-MAPS NGOs that there was little transparency regarding the criteria used in selecting agencies for MAPS membership; there is a need for Development Cooperation Ireland to follow clear criteria in any future MAPS; our proposals in Section 5 above are intended to offer a starting point in this regard. ■ While some non-MAPS NGOs would wish to be MAPS partners, it is wrong to suggest that MAPS has created a form of league table in which its partners are in the first division, with other NGOs relegated to second-class citizenship; the view taken by the non-MAPS NGOs is more mature than this. 11 12 These comprised: ■ A briefing session (on 26/01/2005) between the Director of Dóchas (the Irish Association of Non-Governmental Development Organisations) and the evaluation team. ■ A focus group meeting between an evaluation team consultant and eight non-MAPS NGOs at the Dóchas offices on 16/03/2005; the objective was to brief non-MAPS NGOs on the MAPS evaluation; an opportunity for individual meetings was offered. ■ Individual meetings (in response to requests) were subsequently held with ActionAid on 14/04/2005 and Oxfam on 15/04/2005. At the request of the evaluators, a meeting was also held with World Vision on 20/05/2005. In all three cases, the Chief Executives participated, together with colleagues. Specific findings relating to some of the agencies met are included in appendix three to this Report. page 32 Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 ■ There is considerable interest on the part of non-MAPS NGOs in reaping what they see as some of the practical benefits of MAPS-type schemes, in particular multi-annual funding and programmatic rather than project-based funding. ‘A smaller MAPS for smaller agencies’, a mini-MAPS or a ‘MAPS Lite’ were suggested. The evaluation team would endorse their view that both block grants and NGO co-financing support could become multi-annual. Trócaire has operated on an all-Ireland basis since the beginning, its trustees being the Roman Catholic Bishops, whose dioceses work throughout both jurisdictions. The evaluation team’s recommendation is that Development Cooperation Ireland (in questions related to MAPS) simply continue the open and generous policy it has followed overall for several decades, in effect not differentiating between agencies in Ireland whose centre of gravity lies either side of the border. 6.7. Other Implications 6.7.1. The Existing Partners A question which arose during the evaluation concerned whether, in the event of a MAPS II, any of the five existing MAPS NGOs which wished to be a partner in it would be required to re-apply. In the view of the evaluation team they should, for several reasons. First, there is an argument for a ‘level playing field’ with new candidate agencies. This point is borne out by the findings from the Comparative Study which show that this would address the issue of continuing legitimacy. Second, selection for a MAPS II (if the qualifications recommended by the evaluation are accepted) will be on the basis of a set of more transparent criteria than were discernible for MAPS I; in keeping with the spirit of partnership, these proposed criteria should be the subject to a consultative process prior to being formalised. Third, the quality of engagement in MAPS I for MAPS NGOs will itself be a criterion (and in the block grant scheme for those NGOs in that scheme). 6.7.2 Northern Ireland Another question which arose during the evaluation concerns Northern Ireland. The MAPS NGOs vary in this respect. GOAL and Concern originated in the Republic, and spread into Northern Ireland (and elsewhere). In contrast, Christian Aid’s profile in Northern Ireland was traditionally more pronounced than in the Republic. Self Help Development International does not promote itself in Northern Ireland. 6.8. Summary Several key figures within the MAPS partnership believe MAPS to be an exercise in budget support. Others speak of ‘managed’ budget support, in which the funds come with strings. Others speak of programme support, whereby the Irish Government’s official development assistance programme examines the fairly extensive programmes of substantial NGOs, identifies the degree of policy cohesion and ‘buys in’ accordingly. The evaluation team has no difficulty with any of these terms as applied to MAPS, leaning possibly toward the programme support definition. But the key finding in this context is that, whatever the terminology, the support provided through MAPS is of a sophisticated kind, promoting the adoption and opertationalisation of an entire range of best-practice approaches, from strategy planning to a programmatic approach, to high-quality Southern partnering, to HIV/AIDS, to gender, to sharing the learning, to environment, to tight and focused management, to mainstreaming as a concept. Conversely from the MAPS NGO viewpoint, the Scheme offers substantial funds, a large measure of freedom in expenditure and the opportunity to learn from partner MAPS NGOs. Moreover, as the Comparative Study annexed to this report shows, the Scheme went considerably beyond that of any international analogues in relative scale and scope and, in this sense, was innovative and groundbreaking in nature. www.dci.gov.ie page 33 7. CONCLUSIONS AND RECOMMENDATIONS In drafting the recommendations, the evaluation team recognises that the majority are focused on Development Cooperation Ireland as the owner, originator and managing authority of MAPS. At the end of the day, the management and responsibility for the shape, focus and running of MAPS is theirs. However, an essential element of MAPS is that of partnership with the NGOs involved in the Scheme. These NGOs also have responsibility for engaging proactively and supporting the development and refinement of MAPS as an appropriate funding mechanism. relocation of Development Cooperation Ireland’s headquarters; capacity constraints of the Civil Society Section of Development Cooperation Ireland; promoting an improved flow of learning; and ensuring the right time lapse for effective programme reviews in the future. Whilst in theory, staggered commencement dates for different NGOs could give rise to some complications, these would be outweighed by two considerations: it would be unfair to hold up MAPS II for compliant agencies until possible sticking points in the case of one or two applicants were fully resolved; having to conduct simultaneous compliance checks and negotiations with a multiplicity of agencies by a fixed uniform deadline would place undue strain on an already overloaded Civil Society Section in Development Cooperation Ireland. (i) 7.1. General 7.1.1. MAPS, in both its design and its outworking as a partnership vehicle and strategic funding mechanism, has been innovative (in the sense of being in the forefront of international donor–NGO partnership practice) and is judged to have been a risk worth taking. It has generated a corpus of learning which, if factored into a MAPS II, has the capacity to provide greater security, to support progress towards achievement of the stated objectives and, at the same time, to effectively absorb incremental increases in overall expenditure. (ii) Recommendation 2: That the second MAPS programme be of five years duration, with participating NGOs phasing into the Scheme on different dates as and when they fulfilled the MAPS Guidelines requirements and with the proviso that, a mid-term evaluation and an impact assessment after five years be built in. Implementor(s): Development Cooperation Ireland. Recommendation 1: There be a second phase of MAPS involving a larger number of agencies. Implementor(s): Development Cooperation Ireland. 7.1.2. The evaluation team considered the relative merits of three to five years for the duration of MAPS II. Factors include the prospective page 34 Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 Recommendation 3: That a restricted call for proposals be issued for participation in MAPS II, based on the agreed eligibility criteria (suggestions for such criteria are given in Section 5 of this Report). Applicant NGOs will need to demonstrate their capacity and capability to be able to apply, manage, monitor and communicate on the use of MAPS funds at the levels of outcomes and impact. Implementor(s): Development Cooperation Ireland, current and prospective MAPS NGOs. Recommendation 4: That Development Cooperation Ireland plan for incremental increases in the overall MAPS budget, in anticipation of (i) additional NGOs meeting the eligibility criteria for admission, and (ii) some increase in the allocations to existing MAPS NGOs, consistent with the need presented elsewhere in this report for a focus on consolidation and qualitative deepening. Implementor(s): Development Cooperation Ireland. 7.1.3. The evaluation team considered whether expenditure should increase significantly for MAPS II, and took account of: ■ the effects of inflation; ■ potential increase in the number of participating NGOs; ■ the expenditure consequences (for MAPS if not for the overall Development Cooperation Ireland budget) of rolling all the other grant schemes (for MAPS NGOs) into MAPS; ■ the question of what is financially appropriate to MAPS moving from an inception to a consolidation phase; ■ increased capacity of some MAPS NGOs since 2002/3 for implementation of MAPS Guidelines and achievement of MAPS objectives on an increased scale; ■ the possible consequences of the ‘70:30’ recommendation in respect of the ratio of Development Cooperation Ireland funding to other income. 7.1.4. Flexibility is one of the defining characteristics of MAPS, something which is highly appreciated by the participating NGOs and which should be maintained in the future. However, the existing Guidelines are lacking in clarity and comprehensiveness is some key areas and have had to be supplemented by memoranda on (for example) the Annual Report format. Recommendation 5: The Guidelines for MAPS II should be re-drafted, so that they are clearer, more comprehensive and more policyfocused. This work needs to be carried out collaboratively with the MAPS NGOs and including Dóchas. The new Guidelines should address the following: ■ The criteria for eligibility for participation in MAPS (see Section 5 of this Report); ■ Provision for a ‘liquidity ratio’ whereby participating NGOs would be expected to generate a specified proportion of their total income in the form of unrestricted funds (the evaluation team would recommend a figure of 30%); ■ A clearer definition of the nature and degree of expected policy coherence and complementarity between Development Cooperation Ireland and MAPS NGOs in utilising MAPS funds, especially in Development Cooperation Ireland programme countries (but ensuring, at the same time, that NGO autonomy for advocacy purposes is fully respected). www.dci.gov.ie page 35 ■ The financial accountability requirements (including summary reporting format); ■ The format for the (narrative) Annual Report, which will need to be impact oriented. This will be based on the need for NGOs in MAPS to have developed objectives and indicators at outcome and impact level; ■ A clearer statement on how and when MAPS NGOs must justify their activation of the ‘20% flexibility’ clause to transfer funds between country programmes, substitute some MAPS activities for others, and report on it later; ■ Description of mechanisms for lateral learning within the MAPS ‘family’. This potentially needs to be separated out as two pieces of work. One is how Development Cooperation Ireland and the NGOs will engage in lateral learning and the other is how instead the NGOs engage with each other (potentially as part of the round-table with Development Cooperation Ireland but not restricted to it)? There also needs to be a requirement for networking, engagement and learning in the countries in which MAPS funding is used. If NGOs cannot provide that commitment then they need to think again about allocating MAPS funds to such a country; ■ A dispute resolution mechanism; ■ Satisfactory outcome of an independent financial systems review. It is also recommended that the Guidelines indicate mechanisms to ensure that the aspiration for regular institutional dialogue involving all MAPS partners in Development Cooperation Ireland programme countries is translated into operational effect and not be dependent on individual goodwill and interest alone. Implementor(s): Development Cooperation Ireland, current MAPS NGOs together with Dóchas. 7.1.5. The evaluation team considered the wisdom of having an upper limit on the number of countries in which MAPS funds could be deployed. The arguments in favour included concentrating the effort on a smaller area to facilitate greater effectiveness and impact evaluation over time, and (conceivably) the development of closer linkages with the remainder of the programme of Development Cooperation Ireland. The principal arguments against concentration are that this could be seen as undue interference by Development Cooperation Ireland in the programmes of the MAPS NGOs and that the number of operational countries should be related to the overall capacity of the NGO concerned, rather than some arbitrary figure. Recommendation 6: That no limit be placed on the number of countries in which MAPS operates. Applicant NGOs should be conscious of the danger of diluting MAPS resources too thinly; consequently, the number of countries proposed to be covered by a given NGO in MAPS II should depend on the ability of participating NGOs to use the funding effectively. Implementor(s): Development Cooperation Ireland and MAPS NGOs. 7.2. Quality Aspects 7.2.1. This Report has highlighted a number of areas relating to programme quality in which existing MAPS practice could be significantly improved, in order to remedy problems, such as: ■ the evaporation effect whereby MAPS parameters, once agreed and clearly understood in Dublin, lose some of this definition when transferred to MAPS NGO field offices, programmes, Southern partners, out-lying districts, sub-programmes, etc. Mainstreaming is a major example. page 36 Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 ■ insufficient impact indicators and benchmarks to provide the basis for wider-angle evaluation of MAPS against its objectives over time; ■ inadequate field experience, development knowledge or familiarity with NGO cultures and constraints on the part of staff of Development Cooperation Ireland (a deficiency which is likely to worsen as a result of substantial staff turnover following decentralisation); ■ differing interpretations of key MAPS policies, objectives or activities; ■ insufficient mechanisms for, or activity around, exchange of learning among the partners; ■ insufficient opportunity for individuals on the staff of Development Cooperation Ireland and the MAPS NGOs to work together in building the partnership. Recommendation 9: That one country monitoring visit of approximately 10 days duration be undertaken by the Civil Society Section to each MAPS NGO every two to three years, to include the appropriate Desk Officer, a development specialist (either internal or external) and the MAPS Coordinator for the MAPS NGO in question. Recommendation 10: That a plenary workshop for all partners be convened early in MAPS II to (a) examine more closely what the programmatic approach really means and to share inter-agency experiences and (b) work on indicators. Recommendation 7: That prospective MAPS NGOs (for the next round of MAPS) formulate impact indicators appropriate to their proposed programme objectives as part of their submissions to MAPS II. Implementor(s): prospective participating NGOs in MAPS II. Recommendation 8: A strengthened MAPS support function of ‘teams’ be established by and in Development Cooperation Ireland (Civil Society Section), using its own staff and resources as far as possible and complemented, as needed, by external specialists on a fixed-term consultancy basis. If the consultancy path is to be followed, it should be emphasised as a facilitative role, and should not be allowed to put distance between the MAPS NGOs and Development Cooperation Ireland. Recommendation 11: That joint training/familiarisation workshops be arranged both in Ireland (especially to induct a newlyappointed cadre of Development Cooperation Ireland staff post-decentralisation) and in a few strategically-selected field locations in MAPS core principles such as strategy, planning, Southern partnering, programming, mainstreaming, etc. In the case of the training events which take place overseas, these would involve the MAPS NGO, Development Cooperation Ireland and Southern partner field staff, and should be synchronised with MAPS field monitoring visits in the interests of cost effectiveness. Implementor(s) for 8–11 above: Development Cooperation Ireland and the full MAPS partnership. www.dci.gov.ie page 37 7.3. Institutional Arrangements 7.3.1. The mechanisms and opportunities for learning exchange and for cross-fertilisation in MAPS need to be substantially improved. MAPS NGOs and Development Cooperation Ireland need to engage more proactively. Although the proportion of time devoted by the Partnership Monitoring Committees to thematic issues has increased, their role still mixes learning with management. There is a need to deepen the MAPS NGO–Development Cooperation Ireland dialogue both within and beyond the Partnership Monitoring Committees, thereby addressing divergences, policy evaporation, and sharedlearning constraints. Using the benefit of the learning generated by MAPS to date, it is considered that the objective and role of the Partnership Monitoring Committees should now be re-stated more clearly. This is understood as a shared responsibility of the NGOs and Development Cooperation Ireland. Recommendation 13: That one joint monitoring field visit per MAPS NGO take place every two to three years (timed where possible to coincide with MAPS field training or familiarisation, to keep costs lower); each visit to involve the relevant Civil Society Section support team and a senior representative of the relevant MAPS NGO. Recommendation 12: Provision of external (i.e. out-sourced) technical support to strengthen the capacity of the Civil Society Section to monitor effectively and evaluate an expanded MAPS programme (replete with the lateral learning and focus on outcomes and impact assessment features being recommended in this evaluation). In respect of each MAPS NGO, the Civil Society Section should draw together one ‘support team’, comprising the responsible Assistant Principal and Desk Officer, the MAPS Coordinator for the NGO concerned and a ‘retained’ external expert whose professional profile would match the main sectoral focus of that NGO. These support teams would manage ongoing dialogue on policy and practice, facilitate and arrange training and familiarisation exercises, participate in joint field monitoring visits, support both bilateral (Partnership Monitoring Committee) and multi-lateral (thematic) MAPS meetings and undertake any benchmarking activity required. Recommendation 14: That in MAPS II, Partnership Monitoring Committees meet only twice annually on management issues, reporting and accounting; in addition, that all partners come together for a twice annual plenary meeting, with facilitation, to share learning on MAPS thematic and policy issues, to harvest best practice, to clarify the expectations of Development Cooperation Ireland, to address evaporation, to exchange outcomes on mainstreaming of cross-cutting issues, and to engage in programme-level learning and thematic discussion. Implementor(s): full MAPS partnership. 7.3.2. The number of separate schemes under which Development Cooperation Ireland allocates funds to the NGOs has grown over the years, each scheme having its own justification, meeting its particular need and implementing its own set of conditions. There has been discussion on this subject in Development Cooperation Ireland for some time. There is now a sense of urgency that this matter needs to be addressed. The opportunity now exists and both the MAPS NGOs and Development Cooperation Ireland now wish to merge as many of these schemes as possible. The evaluation team found that the multiplicity of schemes increases the managerial burden primarily on the Civil Society Section staff (this section handles most but not all of the schemes), and generates considerable and partly page 38 Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 duplicated extra work for the MAPS NGOs. There is an additional problem of some indistinct demarcations between the schemes. This issue is not mentioned in the terms of reference. But it was raised by the MAPS NGOs and, in particular, by Development Cooperation Ireland staff. To the extent that delivering more grant finance with the same level of staff is a MAPS outcome, the question of rationalising the other schemes is directly apposite. Recommendation 15: That for MAPS NGOs, most of the existing funding schemes be subsumed into MAPS and that a similar rationalisation be made for those NGOs in the Block Grant and NGO Co-financing Schemes. It is recommended that these changes be made as soon as possible and not be postponed until after the planned decentralisation of Development Cooperation Ireland. In making this recommendation, the team is aware of the complexities involved and of the volume of work that will be involved in making the changes smoothly and ensuring continued access by all NGOs to the supports previously available. On the other hand, the change, once made, will benefit all stakeholders. Implementor(s): Development Cooperation Ireland. 7.3.3. Between the scheduled end-date for this evaluation (30 June 2005) and the scheduled end-date for MAPS I (31 December 2005), there may be insufficient time for Development Cooperation Ireland to absorb the evaluation reports, decide which recommendations are to be adopted, begin implementation of them, decide on which NGOs might be invited to apply to become partners in MAPS II, invite those selected, re-state objectives and MAPS Guidelines, receive applications/programmes, negotiate around these with individual NGOs, submit agreed applications to the Projects Assessment and Evaluation Group, transmit the outcomes, prepare and agree new contracts and pay the first annual instalments. Recommendation 17: That all the processes above be undertaken as quickly as possible but, in the event of delays, that a ‘bridging’ period of no more than six months be created in order to maintain confidence in the reliability of MAPS and to maintain the momentum of existing collaboration between MAPS NGOs and partners in-country. Implementor(s): full MAPS partnership. Recommendation 16: A set of terms of reference for the Partnership Monitoring Committees be developed, which should distinguish clearly between management and learning. Implementor(s): Development Cooperation Ireland and the full MAPS partnership. 7.3.4. Policy development is an essential support to MAPS for both Development Cooperation Ireland and the NGOs involved. For Development Cooperation Ireland, work was begun, but not completed, on a comprehensive civil society policy document by Development Cooperation Ireland. Recommendation 18: That all MAPS partners (including Development Cooperation Ireland) redouble their efforts on policy and strategy development so that all programmes operate on a clear policy basis. Implementors: Development Cooperation Ireland and the full MAPS partnership. www.dci.gov.ie page 39 Recommendation 19: That all MAPS partners recognise that policy development is an essential support to MAPS, and urge them to re-double their efforts in this area. The task of bringing the Development Cooperation Ireland Civil Society Policy document to fruition is a matter of priority. Implementor(s): Development Cooperation Ireland and the full MAPS partnership. page 40 Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 ANNEX 1 Evaluation of Development Cooperation Ireland’s Multi-Annual Programme Scheme 2003–2005: Abbreviated Terms of Reference 1. Introduction 3. Initial Objectives / Purpose of Evaluation In 2003, a Multi-Annual Programme Scheme (MAPS) was established by Development Cooperation Ireland. At the heart of this arrangement was a move to a new relationship between the donor and the participating NGOs based on multi-annual strategic planning and partnership. The three-year programme entered into agreements totalling €117 million with five participating NGOs: Christian Aid Ireland, Concern World Wide, GOAL, Self Help Development International and Trócaire. Given the size and potential complexity of the study, it is suggested that the evaluation be broken into two different levels: i. General level: examine the general understanding of objectives and implementation of MAPS; Management level: examine overall programming and operational issues brought about by the new scheme and recommend how HAPS could be integrated into future MAPS structures. ii. 2. Purpose and Scope The purpose of the evaluation is to assess the progress of the Scheme. It should focus primarily on policy, strategic and operational issues relevant to the Scheme and examine performance from the perspectives of international standards of best practice. In doing so, it will be necessary to evaluate this in terms of relevance, efficiency, effectiveness, impact and sustainability, with particular emphasis on partnership. The exercise will look mainly at organisational structures and systems, their changes/developments since the inception of the programme both horizontally and vertically and establish any evidence of operational changes. While it is not anticipated that the programme is sufficiently established yet to have achieved sustained impact at beneficiary level, field work may include visits to and meetings with local partners. General Level (a) Under the criteria of relevance, effectiveness and coherence, is MAPS facilitating its partners to deliver more coherent and strategic approaches to achieving both the MDGs, Development Cooperation Ireland policies and strategies and their implementation? Issues to be examined include the following: Clarity of objectives, Policy implementation, Impact, Innovation, Learning. (b) Is MAPS concretely producing a more substantive relationship between Development Cooperation Ireland and its NGO partners? Issues to be examined include the following: nature of partnership, communication and cooperation efforts, accountability, implications for non MAPS NGOs. www.dci.gov.ie page 41 Management Level Under the criteria of efficiency, effectiveness and sustainability, has MAPS contributed to more effective management? Issues to be examined include the following: capacity-building – has MAPS changed the way partners plan and implement their programmes? information flow and reporting – is MAPS likely to obtain an efficient and effective delivery on objectives? Are criteria for funding allocations to and within the partner NGOs coherent and transparent? 5. Output The consultancy reports consist of an inception report, one synthesis study and six component reports, one for each partner (five NGOs and Development Cooperation Ireland). The synthesis report will cover the main findings and include recommendations for phase 2 set in the context of best international practice and standards. Recommendations would also be made around the integration of the HIV/AIDS Partnership Scheme (HAPS) into that phase. The report should be primarily addressed to Senior Management in both Development Cooperation Ireland and the partner NGOs. Other audiences will include programme staff in each partner organisation and the general public. The report should contain clear and concise conclusions and recommendations in relation to the issues raised in Section 3. All partners would have the opportunity to make comments on both their respective draft reports and the synthesis report. The full report will be the responsibility of the consultant. While the Evaluation Unit and the Steering Committee (see below) will have responsibility for assessing quality, nevertheless, the tender submission should outline procedures for the internal quality control of the evaluation and report. 4. Methodology The exercise will be undertaken in January–June 2005 and will consist of an Inception Phase (3 weeks), Study Phase (5 weeks) and Field Phase (7 weeks). There will be two country visits per NGO – one major and one follow-up visit. The report writing phase will consist of: initial report writing (5 weeks), workshop to discuss draft report – findings and recommendations for phase 2 (2 days). Final report (1 week). Total time : 22 weeks approximately (in terms of overall timeframe). Tenderers will need to determine the professional input/number of consultancy days required to carry out this evaluation. Proposed Team Composition: ■ Team Leader (with extensive experience in NGO management, donor relationships and partnership concepts); Management Specialist; Social Development Specialist. 6. Management Structure ■ ■ A Steering Committee involving Development Cooperation Ireland (Civil Society Section the Evaluation and Audit Unit) and representatives from the NGO partners will be responsible for the key decisions and overall management of the study. For each of the individual NGO partner studies undertaken, a specific Management Committee will be appointed. The Evaluation and Audit Unit will have responsibility for the day to day management of the evaluation. page 42 Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 7. ■ ■ Selection Criteria Skills and experience: Proven capacity and experience in development strategy, planning, programmatic approaches and management, particularly in an NGO environment; Strong understanding of NGO and donor relationships in MAPS type schemes; Strong understanding of the concepts and practice of partnership, particularly in a development context; Extensive evaluation experience ■ ■ ■ ■ ■ Proposed methodology Balance and composition of team reflecting knowledge of best international standards and practices with regards to the objectives of the terms of reference Track record of consultancy firm in carrying out similar evaluations Availability within the timeframe outlined Cost ■ ■ ■ NOTE: Proposed team members should not include any personnel involved in the design and preparation of the MAPS scheme. www.dci.gov.ie page 43 ANNEX 2 DCI Management Response to the Evaluation of the Multi-Annual Programme Scheme – 2005 7.1 General Finding 7.1.1. MAPS, in both its design and its outworking as a partnership vehicle and strategic funding mechanism, has been innovative (in the sense of being in the forefront of international donor – NGO partnership practice) and is judged to have been a risk worth taking. It has generated a corpus of learning which, if factored into a MAPS II, has the capacity to provide greater security, support progress towards achievement of the stated objectives and at the same time, to effectively absorb incremental increases in overall expenditure. Recommendation 1. There should be a second phase of MAPS. Management Response We agree Management Action Develop a new set of guidelines (including eligibility criteria) based on the recommendations from the MAPS evaluation by end October 2005. Finding 7.1.2. The evaluation team considered the relative merits of three to five years as the duration of MAPS II. Factors include prospective relocation of Development Cooperation Ireland’s HQ, capacity constraints of the Civil Society Section of Development Cooperation Ireland, promoting an improved flow of learning, and ensuring the right time lapse for effective programme reviews in the future. Whilst in theory staggered commencement dates for different NGOs could give rise to some complications, these would be outweighed by two considerations: (i) (ii) it would be unfair to hold up MAPS II for compliant agencies until possible sticking points in the case of one or two applicants were fully resolved; having to conduct simultaneous compliance checks and negotiations with a multiplicity of agencies by a fixed uniform deadline would place undue strain on an already overloaded Civil Society Section in Development Cooperation Ireland. Management Response We agree with the proposal of a fiveyear term. However, it might be preferable to have a shorter initial MAPS for new partners, say three years. On the timing for entry into MAPS II, Development Cooperation Ireland will be flexible with participating agencies. Management Action Design five-year MAPS II for existing MAPS NGOs, with the possibility of a three-year programme for new partners. Discuss with compliant agencies their timing requirements and agree timetable for staggered entry into MAPS II. Recommendation 2. That the second MAPS programme be of five years duration, with participating NGOs phasing into the Scheme on different dates as and when they fulfilled the MAPS Guidelines requirements, and with the proviso that, a mid-term evaluation and an impact assessment after five years be built-in. page 44 Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 Recommendation Management Response We agree with the proposal for an impact assessment on agencies who have participated in MAPS I and II after five years. Other evaluations will be carried out as and when required. Management Action 3. That a restricted call for proposals be issued for participation in MAPS II, based on the agreed eligibility criteria (suggestions for such criteria are given in Section 5 of this Report). Applicant NGOs will need to demonstrate their capacity and capability to be able to apply, manage, monitor and communicate on the use of MAPS funds at the levels of outcomes and impact. We agree. Only participating MAPS I agencies, which comply with the agreed eligibility criteria, will be invited to submit proposals for MAPS II from late 2005. Additional agencies, which comply with agreed eligibility criteria, will be invited to submit proposals in 2006 and to enter MAPS II in 2007. Finding 7.1.3. The evaluation team considered whether expenditure should increase significantly for MAPS II, and took account of: ■ ■ ■ ■ ■ the effects of inflation; potential increase in the number of participating NGOs; the expenditure consequences (for MAPS if not for the overall Development Cooperation Ireland budget) of rolling all the other grant schemes (for MAPS NGOs) into MAPS; the question of what is financially appropriate to a MAPS moving from an inception to a consolidation phase; increased capacity by some MGNOs since 2002/3 for implementation of MAPS guidelines and achievement of MAPS objectives on an increased scale; the possible consequences of the ‘70:30’ recommendation in respect of the ratio of Development Cooperation Ireland funding to other income Recommendation 4. That Development Cooperation Ireland plan for incremental increases in the overall MAPS budget, in anticipation of (i) additional NGOs meeting the eligibility criteria for admission, and (ii) some increase in the allocations to existing MAPS NGOs, consistent with the need presented elsewhere in this report for a focus on consolidation and qualitative deepening. Management Response In the light of increased ODA funding, we agree that an increase in the allocations to existing MAPS NGOs is justified, but also agree that the focus should be on consolidation and qualitative deepening. Management Action Decide MAPS II level of funding in overall Development Cooperation Ireland budget allocation discussions at Senior Management Level (DCD). www.dci.gov.ie page 45 Finding 7.1.4. Flexibility is one of the defining characteristics of MAPS, something which is highly appreciated by the participating NGOs and which should be maintained into the future. However, the existing Guidelines are lacking in clarity and comprehensiveness in some key areas and have had to be supplemented by memoranda on (for example) the Annual Report format. Recommendation 5. The Guidelines for MAPS II be redrafted, so that they are clearer, more comprehensive and more policyfocused. This work needs to be carried out collaboratively with the MAPS NGOs and include Dóchas. The new Guidelines should address the following: ■ Management Response We agree on the necessity of redrafting the Guidelines for MAPS II. Management Action Development Cooperation Ireland will draft the Guidelines following consultation with MAPS partners and others as appropriate. The criteria for eligibility for participation in MAPS (see Section 5 of this Report); Provision for a ‘liquidity ratio’ whereby participating NGOs would be expected to generate a specified proportion of their total income in the form of unrestricted funds (the evaluation team would recommend a figure of 30%); A clearer definition of the nature and degree of expected policy coherence and complementarity between Development Cooperation Ireland and MAPS NGOs in utilising MAPS funds, especially in Development Cooperation Ireland programme countries (but ensuring at the same time that NGO autonomy for advocacy purposes is fully respected); The financial accountability requirements (including summary reporting format); ■ We agree that the criteria for eligibility for participation in MAPS II should be based on those set out in the MAPS evaluation report. We agree with the proposal for a ‘liquidity ratio’ in order to preserve the independence of participating NGOs. The Guidelines will address the issues raised by the evaluation and incorporate them along the lines proposed by the team. ■ Participating MAPS NGOs share much in common with Development Cooperation Ireland in terms of policy approach. We agree. Consultations with NGOs for the purposes of developing the Guidelines will examine the issue of policy coherence and complementarity ■ The current format for the annual report is considered useful by the MAPS NGOs. page 46 Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 Recommendation ■ Management Response Management Action Some adjustment in the current annual report format may be necessary for MAPS II. The format for the (narrative) Annual Report, which will need to be impact oriented. This will be based on the need for NGOs in MAPS to have developed objectives and indicators at outcome and impact level; A clearer statement on how and when MAPS NGOs must justify their activation of the ‘20% flexibility’ clause to vire funds between country programmes. substitute some MAPS activities for others, and report on it later; Description of mechanisms for lateral learning within the MAPS ‘family’. This potentially needs to be separated out as two pieces of work. One is how Development Cooperation Ireland and the NGOs will engage in lateral learning and the other is how the NGOs engage with each other (potentially as part of the round table with Development Cooperation Ireland but not restricted to it). There also needs to be a requirement for networking, engagement and learning in the countries in which MAPS funding is used. If NGOs cannot provide that commitment, then they need to think again about allocating MAPS funds to such a country. A dispute resolution mechanism; Satisfactory outcome of an independent Financial Systems Review. We agree. The Guidelines for MAPS II will be clearer on the use of the ‘flexibility clause’ on the issue of moving funds, including consideration of lowering the percentage threshold. We agree. We think it important that NGOs provide a commitment to networking, engagement and learning in countries where MAPS is applied. Equally, we would encourage MAPS NGOs to document best practice for learning purposes between all MAPS NGOs and Development Cooperation Ireland; inter alia this would provide useful input to joint PMCs. ■ ■ The joint PMC on thematic issues will be one of the key fora for lateral learning with MAPS partners. Development Cooperation Ireland will further encourage and support learning initiatives proposed by MAPS NGOs as appropriate. ■ ■ DCI recognises that differences may arise in the context of interpretation of Guidelines, etc. However, DCI must reserve the right to determine the Guidelines in accordance with its priorities and accountability standards. We agree. This will apply to new entrants and other MAPS NGOs following a recommendation from Development Cooperation Ireland’s Audit Unit. We agree. DCI is committed to dialogue with its MAPS partners and will continue its current practice of ad hoc meetings to try to resolve any differences. All new MAPS II entrants will be subject to an independent financial systems review. The Guidelines will be explicit about the dialogue necessary at programme country level. It is also recommended that the Guidelines indicate mechanisms to ensure that the aspiration for regular institutional dialogue involving all MAPS partners in Development Cooperation Ireland programme countries is translated into operational effect and should not be dependent on individual goodwill and interest alone. www.dci.gov.ie page 47 Finding 7.1.5. The evaluation team considered the wisdom of having an upper limit on the number of countries in which MAPS funds could be deployed. The arguments in favour included concentrating the effort in a smaller area to facilitate greater effectiveness and impact evaluation over time, and (conceivably) the development of closer linkages with the remainder of the programme of Development Cooperation Ireland. The principal arguments against concentration is that this could be seen as undue interference by Development Cooperation Ireland in the programmes of the MAPS NGOs and that the number of operational countries should be related to the overall capacity of the NGO concerned, rather than some arbitrary figure. Recommendation 6. That no limit be placed on the number of countries in which MAPS operates. Applicant NGOs should be conscious of the danger of diluting MAPS resources too thinly; consequently, the number of countries proposed to be covered by a given NGO in MAPS II should depend on the ability of participating NGOs to use the funding effectively. Management Response We agree. Development Cooperation Ireland does not wish to be prescriptive in this regard but recalls the evaluation team’s recommendation on the need for consolidation and qualitative deepening. Management Action Encourage participating agencies to focus MAPS resources in a manner that results in greater effectiveness and demonstrable impact. 7.2 Quality Aspects Finding 7.2.1. This report has highlighted a number of areas relating to programme quality in which existing MAPS practice could be significantly improved, in order to remedy problems, such as; ■ the evaporation effect whereby MAPS parameters, once agreed and clearly understood in Dublin, lose some of this comprehension when transferred to MAPS NGO field offices, programmes, Southern partners, out-lying districts, sub-programmes, etc; mainstreaming is major example; insufficient impact indicators and benchmarks to provide the basis for wider-angle evaluation of MAPS against its objectives over time; inadequate field experience, development knowledge or familiarity with NGO cultures and constraints on the part of staff of Development Cooperation Ireland, (a deficiency which is likely to worsen as a result of substantial staff turnover following decentralisation); differing interpretations of key MAPS policies, objectives or activities; insufficient mechanisms for or activity around exchange of learning among the partners; insufficient opportunity for individuals on the staff of Development Cooperation Ireland and the MAPS NGOs to work together in building the partnership. ■ ■ ■ ■ ■ page 48 Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 Recommendation 7. That prospective MAPS NGOs (for the next round of MAPS) formulate impact indicators appropriate to their proposed programme objectives as part of their submissions MAPS II. 8. A strengthened MAPS support function of ‘teams’ be established by and in Development Cooperation Ireland (Civil Society Section), using its own staff and resources as far as possible, and complemented as needed by external specialists on a fixed-term consultancy basis. If the consultancy path is to be followed, it should be emphasised as a facilitative role, and should not be allowed to put distance between the MAPS NGOs and Development Cooperation Ireland. 9. That one country monitoring visit of approximately 10 days duration be undertaken by Civil Society Section to each MAPS NGO every two-three years, to include the appropriate Desk Officer, a development specialist (either internal or external) and the MAPS Coordinator of the MAPS NGO in question. 10. That a plenary workshop for all partners be convened early in MAPS II to (a) examine more closely what the programmatic approach really means and share inter-agency experiences and (b) work on indicators. Management Response We agree. Management Action The programme appraisal process will consider the quality of the impact indicators as a key determinant in assessing the proposals We agree but recognise the difficult capacity constraints within which Development Cooperation Ireland operates and also the shortage of qualified consultants to take on these roles on an ongoing basis. The decentralisation process to Limerick will place an additional strain on our ability to meet this recommendation. Design a monitoring process which facilitates a continuing close relationship between the MAPS NGOs and the Civil Society Section but that also helps to build the capacity of Development Cooperation Ireland staff. Use will be made of external consultants as necessary. We agree but this will continue to be extremely difficult unless staff numbers increase proportionately with the number of MAPS partners. Monitoring visits will be planned and carried out in close collaboration with the MAPS NGO concerned and the TOR of any external consultants used will emphasise the importance of facilitation and capacity building for Development Cooperation Ireland Staff. We agree with a workshop to examine more closely the meaning of a programmatic approach. We feel it is important to identify the indicators to measure the programmatic impacts and outcomes of MAPS II as early as possible. At an agreed time in 2006, we will host a thematic workshop for all MAPS partners to explore and reach a common understanding of what a programmatic approach is and share inter-agency experiences. A separate workshop will be held involving the following Sections in Development Cooperation Ireland: Technical, Civil Society and Evaluation and Audit and representatives from the MAPS partners to identify jointly a set of benchmarks and indicators to measure the programmatic impacts and outcomes of MAPS II. www.dci.gov.ie page 49 Recommendation 11. That joint training/familiarisation workshops be arranged in both Ireland (especially to induct a newlyappointed cadre of Development Cooperation Ireland staff postdecentralisation) and a few strategically-selected field locations in MAPS’ core principles such as strategy, planning, southern partnering, programming, mainstreaming, etc. In the case of the training events which take place overseas, these would involve MAPS NGO, Development Cooperation Ireland and Southern partner field staff, and should be synchronised with MAPS field monitoring visits in the interests of cost effectiveness. Management Response Training and familiarisation of the newly appointed cadre of Development Cooperation Ireland staff post-decentralisation will be carried out as part of Development Cooperation Ireland’s induction and training programme. MAPS workshops will focus on thematic learning on an inter-agency basis. They will not be used for familiarisation purposes. This will be the responsibility of each participating organisation. Management Action Best practice will be the subject of all Joint PMCs. Training and familiarisation will be the responsibility of each agency, including Development Cooperation Ireland, for its own staff. Staff resources permitting, we will seek opportunities for joint staff training. 7.3 Institutional Arrangements Finding 7.3.1. The mechanisms and opportunities for learning exchange and for cross-fertilisation in MAPS need to be substantially improved. MAPS NGOs and Development Cooperation Ireland need to engage more proactively. Although the proportion of time devoted by the Partnership Monitoring Committees to thematic issues has increased, their role still mixes learning with management. There is need to deepen the MAPS NGO/Development Cooperation Ireland dialogue both within and beyond the Partnership Monitoring Committees, thereby addressing divergences, policy evaporation, and shared-learning constraints. Using the benefit of the learning generated by MAPS to date, it is considered that the objective and role of the Partnership Monitoring Committees should now be re-stated more clearly. This is understood as a shared responsibility of the NGOs and Development Cooperation Ireland. page 50 Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 Recommendation 12. Provision of external (i.e. outsourced) technical support to strengthen the capacity of the Civil Society Section to monitor effectively and evaluate an expanded MAPS programme (replete with the lateral learning and focus on outcomes and impact assessment features being recommended in this evaluation). In respect of each MAPS NGO, Civil Society Section should draw together one ‘support team’, comprising the responsible Assistant Principal and Desk Officer, the MAPS Co-ordinator of the NGO concerned and a ‘retained’ external expert whose professional profile would match the main sectoral focus of that NGO. These support teams would manage ongoing dialogue on policy and practice, acilitate and arrange training and familiarisation exercises, participate in joint field monitoring visits, support both bilateral (Partnership Monitoring Committee) and multilateral (thematic) MAPS meetings, and undertake any benchmarking activity required. 13. That one joint monitoring field visit per MAPS NGO take place every two to three years (timed where possible to coincide with MAPS field training or familiarization, to keep costs lower); each visit to involve the relevant Civil Society Section support team and a senior representative of the relevant MAPS NGO. Management Response In general, we agree with the proposition that the capacity of the Civil Society Section should be strengthened and that specific teams should be established, linked to each MAPS NGO. The use of external consultants will be considered, if necessary, but where possible the use of internal specialists is preferable. Management Action The composition of each team will be decided following internal Development Cooperation Ireland discussions and based on the outcome of the overall rationalisation of funding schemes currently being undertaken in the Civil Society Section. Use will be made of external consultants, as necessary, to complement internal specialists. We agree with the joint monitoring field visit and the involvement of the relevant Civil Society Section team and a senior representative of the relevant NGO. Other events, which may take place during such visits, will be decided on a case by case basis. A calendar of joint monitoring field visits will be prepared following consultation with relevant partners. www.dci.gov.ie page 51 Recommendation 14. That in MAPS II, Partnership Monitoring Committees meet only twice annually on management issues, reporting and accounting; in addition, that all partners come together for a twice annual plenary meeting, with facilitation, to share learning on MAPS’ thematic and policy issues, to harvest best practice, to clarify the expectations of Development Cooperation Ireland, to address evaporation, to exchange outcomes on mainstreaming of crosscutting issues, and to engage in programme level learning and thematic discussion. Management Response We agree. Management Action We will develop a calendar of PMCs and joint plenary meetings following consultation with the MAPS NGOs. cf para. 16. Finding 7.3.2 The number of separate schemes under which Development Cooperation Ireland allocates funds to the NGOs has grown over the years, each scheme having its own justification, meeting its particular need and implementing its own set of conditions. There has been discussion on this subject in Development Cooperation Ireland for some time. There is now a sense of urgency that this matter needs to be addressed. The opportunity now exists and both the MAPS NGOs and Development Cooperation Ireland now wish to merge as many of these schemes as possible. The evaluation team found that the multiplicity of schemes increases the managerial burden primarily on the Civil Society Section staff (this Section handles most but not all of the schemes),and generates considerable and partly duplicated extra work for the MAPS NGOs. There is an additional problem of some indistinct demarcations between the schemes. This issue is not mentioned in the Terms of Reference. But it was raised by the MAPS NGOs and, in particular, by Development Cooperation Ireland staff. To the extent that delivering more grant finance with the same level of staff is a MAPS outcome, the question of rationalising the other schemes is directly apposite. page 52 Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 Recommendation 15. That for MAPS NGOs, most of the existing funding schemes be subsumed into MAPS and that a similar rationalisation be made for those NGOs in the Block Grant and NGO Co-Financing Schemes. It is recommended that these changes be made as soon as possible and not be postponed until after the planned decentralisation of Development Cooperation Ireland. In making this recommendation, the team is aware of the complexities involved and of the volume of work that will be involved in making the changes smoothly and ensuring continued access by all NGOs to the supports previously available. On the other hand, the change once made will benefit all stakeholders. 16. A set of Terms of Reference for the Partnership Monitoring Committees should be developed, which distinguish clearly between management and learning. Management Response This will be examined. Management Action The rationalisation of funding schemes is underway and will be completed by 11 November. MAPS NGOs will be eligible to apply for grants for sudden onset emergencies but to no other funding schemes in Development Cooperation Ireland. We agree. These TOR will be developed following consultation with the MAPS partners in early 2006. Finding 7.3.3 Between the scheduled end-date for this evaluation (30 June 2005) and the scheduled end-date for MAPS 1 (31 December 2005), there may be insufficient time for Development Cooperation Ireland to absorb the evaluation reports, decide which recommendations are to be adopted, begin implementation of them, decide on which NGOs might be invited to apply to become partners in MAPS II, invite those selected, re-state objectives and MAPS Guidelines, receive applications/ programmes, negotiate around these with individual NGOs, submit agreed applications to the Projects Assessment and Evaluation Group, transmit the outcomes, prepare and agree new contracts and pay the first annual instalments. www.dci.gov.ie page 53 Recommendation 17. That all the processes above be undertaken as quickly as possible, but in the event of delays, that a "bridging" period of no more than six months be created in order to maintain confidence in the reliability of MAPS and to maintain the momentum of existing collaboration between MAPS NGOs and partners in-country. Management Response We agree for existing MAPS NGOs. For new entrants, however the process of preparation will begin during 2006 with a view to a start date of January 2007. Management Action A priority is to finalise the criteria and guidelines for MAPS II. Additionally, we will request all MAPS NGOs to submit a concept note and timeline for submission of completed programme proposals by the middle of November 2005. A bridging period will be made available for those NGOs which require it. All final proposals should be submitted by end-March 2006. Finding 7.3.4. Policy Development is an essential support to MAPS for both Development Cooperation Ireland and the NGOs involved. For Development Cooperation Ireland work was begun but not completed on a comprehensive Civil Society policy document by Development Cooperation Ireland. Recommendation 18. That all MAPS partners (including Development Cooperation Ireland) redouble their efforts on policy and strategy development so that all programmes operate on a clear policy basis. 19. That all MAPS partners recognise that Policy Development is an essential support to MAPS, and urge them to re-double their efforts in this area. The task of bringing the Development Cooperation Ireland Civil Society Policy document to fruition is a matter of priority. Management Response We agree Management Action MAPS II will provide support to MAPS NGOs for policy and strategy development. Development Cooperation Ireland will continue its own policy development across the programme and as part of the White Paper Process. Development Cooperation Ireland’s Civil Society Policy will be completed as a matter of urgency. We agree page 54 Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 www.dci.gov.ie page 55 page 56 Evaluation of the Development Cooperation Ireland Multi-Annual Programme Scheme 2003-2005 designed by www.reddog.ie www.dci.gov.ie Development Cooperation Ireland Department of Foreign Affairs Bishop’s Square Redmond’s Hill Dublin 2 Tel Fax Email Web +353 1 408 2000 +353 1 408 2880 dci@dfa.ie www.dci.gov.ie The paper used in this publication contains 30% recycled Mill Broke