Get documents about "SELF- EVALUATION REPORT
Shared by: 2c2A5Z
-
Stats
- views:
- 14
- posted:
- 3/17/2012
- language:
- English
- pages:
- 58
Document Sample
SELF- EVALUATION REPORT
THE MAPA PROJECT
!!! DRAFT !!!
last changed: May 21, 2002
This draft is sent out to participants in the project for feed-back and consultation.
Abbreviations:
MAPA: Method of Applied Planning and Assessment – methodology for participatory
planning and evaluation, developed initially for the Portuguese Development Co-operation by
Professor Ulrich Schiefer (ISCTE Lisbon); related to the models promoted by the German
Development Co-operation (GTZ), Swiss Development Co-operation (SDC), Institute for
Development Studies (IDS - UK).
OSI: Open Society Institute
IEP: Institute for Educational Policy of the OSI (became Education Support Programme after
January 2002).
ESP: Education Support Programme.
INTRODUCTION
The MAPA project was designed to assist education programmes from Soros foundations and
education spin-off NGOs to acquire skills in participatory evaluation and planning. The
project began in September 2000 and lasted eighteen months, initially with central training
(directly organised by the ESP), and ending totally devolved to the network. Moreover, in
countries, the methodology spread to schools, public administration, ministries, other NGOs
and even businesses. The final evaluation meeting for the project was cancelled due to the
departure of the Project Manager (Lucinia Bal), following restructuring of the ESP.
This report was produced by the Project Manager in order to conclude the project, with on-
line consultation of beneficiaries and of invited experts, as listed below:
1
- the MAPA-L list (“Circle for Planning and Evaluation”), which includes all staff and
experts who participated in the MAPA training workshops directly organised by the ESP
(that is, the "first generation" of facilitators and trainers);
- the initial trainers: Ulrich Schiefer and Reinald Doebel;
- managers of OSI departments, who had network-programme staff trained within the
MAPA project, and who have been in touch with the Project Manager during the
development of the project;
- one of the two experts who reviewed the MAPA handbook, Ms. Marlene Laeubli, from
the Swiss government, who had been invited to our real workshop to be held in Budapest
- now cancelled.
This report was compiled with information arriving from the network until April 24, 2002. It
will be updated with any information arriving during the month of May, and will be posted on
the MAPA website http://www.osi.hu/iep/MAPA/MAPA_main.htm, to assist any planned
follow-up.
This self-evaluation exercise offers a shared conclusion to a common project. It is intended to
help the exchange of experience in the network, and to model the practices it preached.
Additionally, it can be shared with other donors/institutions in order to draw attention to
examples of good practice in the OSI network.
THE PROJECT – SHORT DESCRIPTION
At the end of the Evaluation Conference organised by the IEP (now ESP) in Budapest in
January 2000 for education directors and co-ordinators from the OSI network (report at
http://www.osi.hu/iep/Workshops/eval/eval_frame.htm), participants requested assistance ‘to
introduce evaluation as a strategic tool’ (50%), and ‘to learn more about planning and
evaluation’ (77%).
After research lasting several months, the IEP concluded that capacity building in
participatory approaches to planning and evaluation would represent a good answer to the
requests from the conference. The MAPA methodology was selected (the participatory
approach developed by Professor Ulrich Schiefer, initially for the Portuguese Development
Co-operation), in order to pilot the training (first training workshop in September 2000). An
invitation was launched to the network in July-August 2000, and eight foundation
representatives came to the first event of what was planned to be ‘an attempt to introduce in
the network the use of evaluation as good practice.’ Within the overall IEP strategy, this
capacity-building in evaluation was aimed to support the sustainability of foundations and
especially of the spin-off NGOs, and to respond to the requests from the network.
The training workshops used simulation of planning and evaluation events as main training
methodology. The simulation was backed up by “input sessions” where the theoretical parts
were discussed.
The team that met in September 2000 (eight foundation representatives and four IEP staff)
used as simulation exercise the planning of a project. The project was meant to offer a
solution to the main problems identified from the question: ‘What are the problems you see in
your organisation and in the network in relation to project planning and evaluation.’ At the
end of the workshop, participants from the network requested the implementation by the IEP
of the project they had designed (file with the project plan attached as Appendix 2). From this
date on, the history of the MAPA project is documented on the web, at
http://www.osi.hu/iep/MAPA/MAPA_main.htm.
The defined goal of the project, was:
2
‘It will strengthen the coherence of the network; ensure sustainability (through capacity to
attract other donors); will improve the quality of programmes (which will maximise impact
on education systems); will enhance learning capacity of organisations.’
The specific objective was:
‘By the end of 2001, 30% of the network educational programmes and spin-offs have the
understanding and use a standardised method for project planning and evaluation developed
in a participatory way and integrated into the strategy.’
The short narrative of project activities:
The project set out to train a) how to plan and evaluate using a participatory approach (that is,
train facilitators of participatory planning and evaluation workshops), and b) trainers of such
facilitators. Together, participants in the MAPA project developed quality-assurance
guidelines for the training workshops, and adapted them continuously. A certification system
and an on-line support group have been set up, in order to make sure that everyone applying
the methodology has enough support to do it well.
Several activities were implemented in parallel, directly by the project manager: a) on-line
support (information bulletins sent to the MAPA-L list, answers to questions, collection of
country data, set up and maintenance of the website), b) organisation of training workshops;
c) support of project evaluations; d) support for in-country training events; e) publication of a
handbook in English and Russian. In addition, activities were initiated in countries, managed
directly by the trainees, and with minimal or no support from the project manager.
Besides the initial (September 2000) training workshop, two more training workshops have
been organised in Budapest, in December 2000 and February 2001, one in Tbilisi (in co-
operation with OSGF Georgia), and one in Kiev (in co-operation with IRF Ukraine). All these
workshops brought together colleagues from several countries. The first three workshops
have been in English, the third in Russian, the fourth in English and Russian, with on-line
traislation. Staff and experts have been trained from 16 countries (Albania, Azerbaijan,
Bulgaria, Czech Republic, Georgia, Kazakhstan, Kosovo, Latvia, Moldova, Mongolia,
Poland, Russia, Serbia, Slovakia, Ukraine, Uzbekistan), while foundations from the remaining
12 countries from the region did not take part in the project. These five workshops were
planned as the starting point of a cascaded training to be extended in countries, independently
of "the central co-ordination", but possibly with support from the IEP in terms of advice and
provision of a tutor. This cascading actually happened and continues to happen, involving
already several hundreds of participants in country workshops over the whole network (more
information in the indicators below).
The expected results of the project (according to the initial plan):
1. At least 30% of foundations and spin-off NGOs (percentage calculated over no. of
countries) have a relevant number of people trained in this method
achieved: staff or experts from 57% (16 out of 28) of the countries of the region
received training – while records exist of systematic follow-up in 50.0% (14 out of
28) of these countries
2. All foundations (NFs) and spin-off NGOs (SOs) [trained] have received copies of manual
of process in enough numbers to distribute within organisation
handbooks have been sent to all foundations and spin-off NGOs that received training: not
all confirmed receipt (a few handbooks are still available if needed, and the pdf file is on the
MAPA webiste)
3
3. All willing and interested NFs and SOs have received requested support for planning
workshop
almost achieved: the request for support from Russia is currently in discussion;
other requests (indicators below) have been answered
4. Communication system available to all NFs and SOs to exchange standardised project
information
achieved: the website was created after the September 2000 workshop, and regularly
updated until the end of January 2002; the MAPA-L list was created and used until today
(in my own view, I could have been more active on the communication side, which started
with intensive exchanges, and got somewhat tired during the past months: more persuasive
facilitation is needed until the on-line groups take off)
5. After 1 year, an evaluation meeting of all institutions applying the method has been
organised
no: the on-line meeting is happening instead
6. All willing and interested NFs and SOs have received requested support for evaluation
workshop
almost achieved: the request for support from Russia is currently in discussion;
other requests (indicators below) have been answered
7. To have a group of trained trainers
achieved: training is happening with funding from other donors, or from the OSI,
in countries; moreover newly-trained trainers facilitated the Tbilisi and Kiev
workshop.
Assumptions for the project to succeed (according to the initial plan):
for the GOAL:
1. The network structure continues to exist.
2. Continuation of network-wide standard of planning and evaluation methods
3. Alternative sources of funding exist
for the OBJECTIVE:
4. The method will be accepted by NFs and SOs
5. The method will be encouraged by top levels (OSI)
6. network-wide, the organisations see the benefit in practice
7. strategy development and project planning have to be competitive
8. Top level of NFs will accept the method
for the RESULTS
9. 30% of participating organisations are willing to use the method
10. There is enough goodwill and space from the rest of the network to allow this to move
forward
11. Organisations see the benefits of this method
12. Organisations are in a culture that can apply this method
for ACTIVITIES
13. 30% organisations are interested, and will take part in workshops
14. All people wanting to become trainers master the language of the workshop
15. Organisations will allow trainers to work as facilitators
16. have enough facilitators available
17. have capable authors to write handbook
18. some organisations which have applied the method take part in the evaluation meeting
Budget
4
Workshop 1 (Budapest, September 2000) USD 10,000
Workshop 2 (Budapest, December 2000) USD 14,000
Workshop 3 (Budapest, February 2001) USD 15,000
Workshop 4 (Tbilisi, May 2001) USD 15,000
Workshop 5 (Kiev, November 2001) USD 8,000 (including external
evaluation)
Support for country workshops USD 3,000
Handbook USD 15,000
Handbook editing and translation USD 5,000
Handbook publication USD 3,000
Web publishing in-kind, Project Manager
Web content in-kind, Project Manager
Maintenance of electronic list in-kind, Project Manager
On-line support in-kind, Project Manager, Experts
TOTAL USD 88,000
ACHIEVEMENT INDICATORS (from the initial plan)
- for the GOAL:
1. Number of NFs and SOs using the method for strategy planning
5: Albania, Azerbaijan, Kosovo, Georgia, Moldova (possibly exit strategies for other
mega-projects)
2. Number of NFs and SOs using the method for project planning/evaluation
12 countries – (in most of these countries the method spread to several spin-off NGOs.
Countries are: Albania (2 organisations), Azerbaijan (foundation and network spin-off
NGOs), Bulgaria (Kristin continues working with MAPA with CRS, where she is in charge
with education projects), Czech Republic (educaiton spin-off), Georgia (components of
EDP), Kosovo (Kosova Education Centre, which also supported the local foundation),
Moldova (2 organisations), Mongolia (components of EDP), Russia (components of EDP;
workshop to take place in June), Ukraine (foundation), Slovakia (with the Pedagogic
Institute); Uzbekistan (2 organisations))
3. Number of projects planned with MAPA and accepted by other donors
(not enough data: Kosovo (9 projects), Slovakia (? textbook project, Pedagogic
Institute), Azerbaijan (2), Ukraine (?preparation of thematic policy papers),
Uzbekistan (2 projects), Bulgaria (1 project))
Specific projects:
Azerbaijan:
- Debate-Internet Project
- Teacher Training Centre for Innovative Methods
Bulgaria
- Monitoring and Evaluation of the WB Loan Project for Education in Bulgaria
Kosovo:
- Reading and Writing for Critical Thinking 2001 (funded by KFOS)
- Quality Education Program 2001 (funded by Save the Children UK)
- Didactic Centers in Kosovo (funded by the Swiss Agency for Development and
Cooperation)
- Education Forum (funded by KFOS)
- Interactive Learning (funded by UNICEF)
5
- Training in Project Planning and Evaluation (funded by Kosovar Foundation
for Civil Society)
- Reading and Writing for Critical Thinking 2002 (funded by KFOS)
- Reading and Writing for Critical Thinking 2002/03 (funded by the Canadian
International Development Agency)
- School Management (funded by KFOS)
- Women Literacy Project (funded by UNICEF)
- Step-By-Step in Primary Education (funded by Soros Foundation Network)
Moldova:
- School Self-Evaluation Project 2002, EC Pro Didactica, Moldova (funded by
Soros Foundation Moldova)
Uzbekistan:
- set-up of a resource centre
- development of the Civic Education Project (Tashkent Public
EducationCenter).
4. Number of projects submitted to other donors
at least: (Albania (AEDP, RWCT), Azerbaijan, Kosovo, Moldova)
Specific Projects:
Azerbaijan:
- Education Information Centre
Kosovo:
5 projects
- School Based Management and Administration (rejected by UNICEF in 2001)
- Life Skills (rejected by UNICEF in 2001)
- Child Friendly School Initiative (submitted to UNICEF in 2002)
- Quality Education Program 2002 (submitted to Save the Children UK in 2002)
- Didactic Center in Mitrovica (submitted to DANIDA in 2002, approved by
Austrian Youth Red Cross for co-funding)\
Moldova:
- School Self-Evaluation Project (EC Pro Didactica, Moldova). The project has
been planed and developed to be submitted to the EU TEMPUS 2002.
5. Number of planned transnational projects
(not enough data: I know about co-operation between Albania and Kosovo, also in
the SEE, between Albania, Kosovo and Moldova)
Specific projects:
a)Kosovo
3 Projects:
- Interactive Learning (funded by UNICEF, partner: Albanian Reading
Association)
- School Based Management and Administration (rejected by UNICEF in 2001,
partner: AEDP-Albania)
- Life Skills (rejected by UNICEF in 2001, partner: Search for Common Ground
in Macedonia)
b) planned and submitted by the Project Manager (proposed for Stability Pact funding):
I.M.PACT – “Introducing Methodology in the Stability Pact” – regional training in
participatory evaluation methodologies (involves Albania, Kosovo, Moldova)
6. (New indicator) Number of projects planned/evaluated with international co-operation
6
(New indicator, resulting from a new set of activities developed within MAPA, and
many of the trainees followed-up in countries.)
Specific Projects:
Azerbaijan:
- planning of the strategy for Educational Information Centre (with colleagues
from Moldova)
Kosovo:
- Didactic Centers in Kosovo (funded by the Swiss Agency for Development and
Cooperation, planned with KulturKontakt-AUSTRIA)
- Interactive Learning (funded by UNICEF, planned with Albanian Reading
Association)
- School Based Management and Administration (rejected by UNICEF in 2001,
planned with AEDP)
- Life Skills (rejected by UNICEF in 2001, planned with Search for Common
Ground in Macedonia)
Slovakia:
- planning of the national textbook project: together with a colleague from the
education spin-off in the Czech Republic
Ukraine:
- evaluation of the project of Development of the National Doctrine (participants
from UNDP, World Bank, colleagues from foundations and spin-off NGOs in
Uzbekistan, Russia, Moldova, Mongolia)
7. Number of funded transnational projects
(not enough data: Kosovo (1))
8. Number of innovations adopted by education systems [and linked to MAPA]
In constructing the indicator, I considered innovations the introduction of new practices in
the education systems. Instead of counting them, I listed the experiences as reported from
countries.
Albania:
Vladimir Pasku: “I have been invited by the Institute of Pedagogical Studies to
train Local Educational Authorities of 4 districts on MAPA issues. During the
summer, it is planned to do the same with groups of school directors. I have been
deeply involved as trainer of school self-evaluation issues. We had 14 pilot schools
and I was surprised to know that 70% of these schools are using the MAPA,
(Problem Analysis). The school staff loves the way MAPA invites them to
participate.”
Saimir Repishti: “We had two training sessions with the core group involved on doing the
WHITE PAPER on Education for Albania. This core group is composed of 23 prominent
people in the field of education and is led by the Deputy Minister of Education that
participated in workshop actively.”
Azerbaijan:
Elmina Kazimzade, Abdullahad Bagirov, Vitalie Popa: The strategy for an
Education Information Centre was planned with participation of all main
stakeholders. The participatory training was extended to teachers and education
administrators, who use the methodology. In addition, the OSI staff and regional
NGOs have been also trained.
7
Bulgaria:
Kristin Razsolkova: the set-up of monitoring indicators for the national reform was
prepared for the Ministry of Education, using MAPA. THe monitoring and
evaluation of the WB Loan Project for Education in Bulgaria was planned with
MAPA.
Czech Republic:
Petr Vrzacek: evaluated the debate for national education reform, using similar
democratic processes – participation.
Kosovo (Dukagjin Pupovci):
- 40 staff of the Ministry of Education, Science and Technology where trained in
MAPA in two groups. The simulation projects were: Building the in-service
teacher training system in Kosovo and Qualification of school primcipals. It is
expected that the methodology will help people understand and accept principles
of co-operation with non-governmental entities, what would represent a very
significant advancement in the work of the Ministry.
- 36 leading administrators of the Kosovar cultural institutions were trained in
MAPA.
Moldova:
Igor Padure: participatory planning for rural associations (SCA);
Vitalie Popa: use of MAPA in self-evaluation of schools. The schools that are
involved in the School Self-Evaluation Project (3 pilot schools and 12 schools that
are involved in the project beginning with 2002) plan and evaluate their activities in
a participatory way as wel as 5 County Educational Authorities used MAPA for
strategic planing.
Mongolia
Erdenejargal Perenlei: activities have taken place, but too little data exists for this
report.
Russia
Elena Golovko: MAPA workshop prepared to support networking of regional teams
engaged in education development projects.
Slovakia
Jana Huttova, Petr Vrzacek: The textbook reform was planned by the Pedagogic
Institute, using MAPA.
Ukraine
Natalia Shablya: For the first time, a national policy project was evaluated and the
follow-up planned using MAPA (and with the help of an external evaluator).
Uzbekistan:
Zulfiya Tursunova, Igor Pogrebov: participatory development of the plans for a
resource centre;
Igor Pogrebov: evaluated the Civic Camp project;
Igor Pogrebov: used participation in the development of a country-wide Civic
Education Project.
8
- for the OBJECTIVE
Indicators 9 and 10 were linked to the administrative procedures of the category A process –
which changed in the meantime. Project plans do not need to be submitted any more.
9. Number of NFs and SOs submitting their programme plans in this logframe format
(inlcuding evaluation) by the end of 2001
10. The fit between strategy and programme
11. Percentage of plans produced with active participation in a workshop
(this indicator is redundant with the number of projects planned using MAPA –
number 2)
12. Level of satisfaction of stakeholders about participation in a workshop
Evaluation results (Lickert scale 1-5, 1 very poor, 5 outstanding)
a) for first Budapest workshop (September 2000):
INDICATOR
relevance of methods and techniques 4.55
relevance of content for work 4.90
variety of content 4.22
usefulness 4.83
applicability 4.42
completion of programme 3.80
general participant satisfaction 4.91
general satisfaction with process 4.58
satisfaction with facilitation 4.36
adequacy of trainers 4.60
follow-up plans exist 3.25
clarity of communication 4.13
internal logic of workshop 4.45
adequate pace 3.91
level of commitment to new method 4.10
level of understanding of the process 4.20
participant capacity to deliver workshops 3.89
TOTAL 4.31
b) second Budapest workshop (December 2000)
INDICATOR
relevance of methods and techniques 4.58
relevance of content for work 4.40
variety of content 3.78
usefulness 4.33
applicability 4.42
completion of programme 3.00
general participant satisfaction 4.30
general satisfaction with process 4.10
satisfaction with facilitation 4.08
adequacy of trainers 4.62
follow-up plans exist 4.00
clarity of communication 4.00
internal logic of workshop 3.73
adequate pace 3.82
9
level of commitment to new method 4.40
level of understanding of the process 4.18
participant capacity to deliver workshops 4.09
TOTAL 4.11
c) third Budapest workshop:
INDICATOR
efficiency 3.38
new knowledge and skills 4.13
clarity 4.08
interactivity 3.85
satisfaction of participants 3.53
performance of trainers 3.90
meeting expectations 3.47
working atmosphere 4.00
logistics of workshop 4.83
efficacy of workshop 4.00
will the method work 4.36
balance between content parts 3.23
TOTAL 3.90
d) fourth workshop (Tbilisi):
Indicator
Conditions for conducting the seminar 4,75
New information 4,33
Acquiring full information about the MAPA method 3,80
Acquiring new skills 3,50
Atmosphere of the seminar 5,00
Methodological completeness 3,46
Level of participants' active participation 4,27
Efficiency of discussions 4,23
Technical equipment 4,17
Living conditions 4,80
Professional level of trainers 4,92
Handouts / Manuals 4,14
Outside atmosphere 5,00
Seminar organisation 4,38
Tutor assistance 4,31
Adequacy of issue discussed 3,92
Assumptions on sustainability of seminar 3,92
Average rating 4,05
10
e) fifth workshop (Kiev)
Indicator
Stakeholder participation 3.62
Correspondence between workshop methods and
goals
– training 3.77
- evaluation of the project 4
Level of organization 4
Correspondence of results to expectations 3.57
The new character of the information 4.10
Interaction of participants 3.77
Understanding of MAPA methodology 3.69
The actuality of the information 4.75
Intention to use MAPA in the future 4.46
Overall rate 3.97
Feed-back from Kosovo:
We have not done any evaluation of MAPA planning workshops, but what we have
done is evaluation of MAPA training workshops. Participants filled out evaluation
sheets and graded various aspects with marks from 1-5, 1 being the lowest, and 5
the highest mark. Below are the average marks:
Knowledge and skills obtained in the seminar 4.32
Clarity of presentation 4.64
Cooperation of trainers with participants 4.63
Quality of the trainers’ work 4.69
Quality of the program of training 4.71
Working environment 4.55
Working conditions 4.33
To what extent are your expectations met 4.49
Importance of MAPA methodology for quality planning and evaluation 4.85
Average 4.53
Some comments from Kosovo workshops:
I find this seminar very important for my job. (Seminar with the Ministry staff, 8-12 April
2002)
I would prefer to have a ready made project planned with MAPA and learn onj it. More
clarity in presentation. These topics are new for us. (Seminar with the Ministry staff, 25-29
March 2002)
After two years of participation in trainings I finally feel that the gap is filled. I think it is
very important to keep organizing this kind of training. Thank you! (Seminar with leading
NGOs, 29 November – 3 December 2001)
11
Please keep the sessions shorter, because after 3 pm there is decrease in the level of
participation. (Seminar with NGOs from Education and Culture, 8-12 November
2001)
This methodology should be spread in our institutions, and particularly among political
parties. (Seminar with NGOs from Education and Culture, 8-12 November 2001)
I saw better trainings than this one. The simulation topic was not very convenient for me.
(Seminar with women NGOs, 11-15 October 2001)
(N.A. – comments and processing of the above results in a following draft)
13. Number of ongoing projects revised / redefined by using the standard participatory
evaluation
Albania: replanning of the teacher education project,
Azerbaijan: Debate project
Georgia: evaluation and replanning of the national textbook development project;
Ukraine: evaluation of the National Doctrine project and planning of follow-up.
Moldova: MAPA was used for the redefinition of the activities of Moldova Microfinance
Aliance (MMA) for year 2001
14. Number of planning/evaluation workshops in countries
Albania:
Azerbaijan: 3
Bulgaria: at least 2
Georgia:
Kosovo: 13
Moldova:
- one planning workshop within MMA (see information under indicator no. 13)
- one planning workshop (School Self-Evaluation Progect)
- five strategic planning workshops for County Educational Authorities (within the
activities of School Self-Evaluation Progect)
- the participatory methodology will be used for the evaluation of the 2001-2002 activities
of the 3 schools involved in the pilot stage (2001) of School Self-Evaluation Progect
Mongolia:
Uzbekistan: 3
15. Number of training workshops in countries
Albania, Azerbaijan (at least 7 training workshops: The participatory training was
extended to teachers and education administrators. In addition, the OSI staff and
regional NGOs have been also trained.), Kosovo: 7 full workshops (5 days each) –
115 participants; 2 “fast-track” workshops (2 days each) – 66 participants, Moldova
(02-06 April 2002, training workshop (3 days) for NGO’s representatives from the
network of the Public Health Program, Soros Foundation Moldova), Mongolia,
Uzbekistan (2 workshops).
16. Quality of the handbook
The handbook has been reviewed by experts from the Swiss government (Ms.
Marlene Laeubli) and specifically from the Swiss Development Co-eopration (M.
Benoit Girardin). Most review comments have been inserted in the printed version –
with the remaining ones planned to be included in a second version. This second
version wishes to include further comments made by the trainers from the network
12
(Vitalie Popa, Dukagjin Pupovci) and by the authors – following the experiences of
these eighteen months.
The review is attached as Appendix 9
Feed-back from the Albanian-language translation prepared in Kosovo: The
Handbook was translated to Albanian, and has been used for the workshops. Each
participant gets a copy of the handbook at the end of a workshop.
The Russian translation was sent out to the network for feed-back.
- for ACTIVITIES
18. Number of people trained
Trained during the workshops in Budapest, Tbilisi and Kiev: 57
Albania: 3
Azerbaijan: 6
Bulgaria: 1
Czech Republic: 1
Georgia: 6
Kazakhstan: 1
Kosovo: 1
Latvia: 1
Moldova: 3
Mongolia: 2
Poland: 1
Russia: 4
Ukraine: 12
Uzbekistan: 5
Slovakia: 1
Sebia: 2
IEP: 7
from the network (49) or work with the network (8)
out of which, during the 18 months of the project, 9 people I know of (18%)
left the network (including myself); 4 have been in touch with me and
possibly use MAPA.
All trained staff received certificates according to the MAPA certification system
(for facilitators and trainers). The certificates have been distributed in November
2001.
19. Number of manuals distributed to the network:
317
20. Number of organisations that use the information system
Number of requests for support or advice: on e-mail:
66 (unrelated to the 5 workshops organised from Budapest)
21. Number of organisations that requested training workshops (from Budapest)
5: Azerbaijan, Georgia, Kosovo (assistance with tutor), Russia, Ukraine
(at least 6 organisations developed the training workshops alone: Albania,
Azerbaijan (again), Kosovo, Moldova, Mongolia, Uzbekistan)
22. Number of organisations that received training workshops
in countries - 4: for Azerbaijan, the training happened in Georgia; Russia not yet
answered
in Budapest: 16
23. Number of organisations that requested evaluation workshops
13
4: Bulgaria, Georgia, Ukraine, Russia
(plus 4 organisations mentioned to have developed their own evaluation
workshops: Albania, Czech Republic, Georgia, Moldova)
24. Number of organisations that received evaluation workshops
3: plus a 4th, Russia, currently developed in co-operation with the initial trainers
25. Number of organisations participating in the evaluation meeting
evaluation on-line – 10 answers, all from the network; 15 participants from
the network sent info during the year, 3 IEP staff provided feed-back during
the year
26. Number and quality of trained trainers
9 – the number was calculated according to the certification system: trainers
need to go through two workshops, and then train in the company of a tutor
(counted from the workshops managed with IEP involvement, plus the
information received about training in countries – Albania, Azerbaijan,
Kosovo, Moldova, Mongolia, Uzbekistan);
quality appreciated very high: indicator resulting from direct or indirect feed-back
on the outcomes of training workshops – not only in terms of workshop evaluation,
but also in terms of follow-up.
MORE INDICATORS
The human dimension, the real achievements, are better expressed by stories coming from
countries, and in letters describing personal experiences. The changes any project wishes to
achieve are ultimately at the personal level.
In discussions with several participants in the project other important outcomes have been
highlighted:
- there is a deeper understanding of democratic behaviours: one of our colleagues
mentioned the debate he had about the participatory procedures launched to develop
options, AFTER the problems and their causes have been determined hierarchically: this
procedure was now easy to identify as hierarchical rather than participatory;
- there is spill-over to public administration and the education systems, with important
partnerships developed as a consequence of participatory events;
- the workshops managed to reunite conflicting parties (during the last weeks, Serb
participants attended workshops organised in Kosovo);
- there are new partnerships developed between foundations in different countries, or
between different spin-offs from the network.
At the end of the day, we aim to change people, not only procedures. Reports of personal
experiences point to this type of impact.
MAPA had a very important personal meaning for me, the Project Manager, as it was one
important way in which I felt that my action matched my values and the statements I was
making – about partnerships, networking, development, sustainability.
Issues reported about MAPA:
“Sometimes the use of MAPA is not viewed very positively. In many cases,
international NGOs and funding agencies manage to raise funds on two pages
14
rationales, and the last thing they would like to do is to fund projects planned in a
participatory way. They think that they know better what a country needs, and do not
feel the need to promote the participation of the people in the process.”
(MAPA trainer from the network)
CONCLUSION
This project started with one project manager and grew into many projects, in many countries,
so it is in fact the self-evaluation for all of us who used participatory techniques. Therefore, as
soon all participants involved in the on-line evaluation will agree on the final text, I hope it
can be a product that will help your own self-evaluation, and activities.
THANK ALL OF YOU A THOUSAND TIMES, FOR YOUR SUPPORT, FOR THE MANY
GOOD EXPERIENCES AND EXCHANGES THAT HAPPENED DURING THE
UNFOLDING OF THIS PROJECT.
Lucinia Bal
Budapest, May, 2002
15
Appendix 1:
Project Concept – L. Bal, October 2000
Concept paper: support in project implementation
Objective
The objective of this project, in line with the network education strategy, is to provide
sustainable support to national foundations, in project implementation. The support
we envisage consists in building capacity for planning, monitoring, evaluation, within
the organisations managing the project. These skills will not only assist the program
management activities, but will provide access to funding from other sources, as
most major donors now request proposals produced using the standardised
methodology.
The activities we started to develop in order to achieve the objective consist in
training, and in tutored activities. The workshops combine training for facilitators in
participatory planning and evaluation, with the acquiring of the standardised
methodology. In addition, the trainees are prepared to also become trainers. Each
workshop lasts five full days. The full training cycle comprises two workshops, and a
series of tutored planning and evaluation activities.
The training will be backed up by expert support with in-country planning and
evaluation procedures. The development of the skills within the network has been
designed as a cascaded scheme, together with foundation representatives, during
the first planning workshop of the series.
Content
The two techniques acquired through the training are MAPA, for the participatory
planning and evaluation of projects, and LOGFRAME, which is the technique to
represent the internal structure of a project.
MAPA (Method for Applied Planning and Assessment) is an operationalised method
for participatory planning and evaluation. The MAPA procedures support the
development of operational goals, objectives, results and measurable indicators, with
the participation of stakeholders, creating ownership and fostering transparency of
the project. MAPA has the strength of using parallel procedures for the planning and
evaluation of projects, allowing the replanning of a project during any of its
implementation phases. This replanning can be used to construct monitoring
indicators, to answer to changes of the environment, or to investigate the existing
logic of the project.
Technically, MAPA consists in the description of a process consisting in a series of
workshops and follow-up activities. It defines specific skills for the facilitation of these
workshops, defines the outputs and interactions of the process. The MAPA training
develops skills for the production of the desired output and the participation of
stakeholders.
LOGFRAME (logical framework) is the framework which helps construct the internal
logic of the project. It assists the process of defining operational goals, objectives,
activities for projects and prompts towards the research of assumptions.
Furthermore, it helps the understanding of the interaction among these elements by
16
placing them in a logical sequence. It also connects the elements of the planning
process with the implementation itself, by assisting the development of measurable
indicators for the monitoring of the initial state and further progress of the project.
Technically, logframe consists in a matrix where all elements of a project need to be
arranged, and where logical inferences are defined among the cells of the matrix.
Process
The full training course consists in two workshops. These workshops use two
synergic techniques in order to develop the planning and evaluation skills. On one
side, the content is delivered and discussed, on the other side, the team performs an
actual planning or evaluation of a project, thus simulating the two MAPA planning /
evaluation workshops, including the reflection, research and feed-back activities
developed between the two workshops. The results of the planning/evaluation
process are developed in a logframe matrix.
The first workshop, held in September 2000 and organised to pilot this training,
brought together staff from foundations and from the IEP, to plan together a project
aimed at solving the needs identified as a response to the following research
question:
"What are the problems you see in the network in relation to project planning
and evaluation?"
The further development of this training scheme and the cascading within the
network are following the plan outlined at this initial workshop.
Skills
The skills to be acquired at the end of the full training session (two workshops), are:
- facilitation of planning and evaluation workshops using the MAPA methodology
- use of logframe in the planning, implementation, monitoring and evaluation of
projects
- potential training skills for MAPA and logframe.
At the end of the first workshop, participants can facilitate MAPA participatory
planning /evaluation workshops together with a tutor, and understand the logframe
technology (that is, away from the flawed understanding of "placing goals and
objectives in a matrix", but as “participatory development of a project in a logical
sequence”).
By the end of the second workshop, trainees master the logframe technology, the full
MAPA planning /evaluation procedures and can act as facilitators. They can also
train together with a tutor, and after two tutored training activities, they can act as
trainers.
Outcomes
The training builds capacity both within foundations and spin-offs, and within the IEP,
at the project implementation levels (integration of projects into strategies, planning,
monitoring, evaluation).
17
The use of the standardised methodology, if correctly applied, has been effective in
supporting the increase of efficiency, effectiveness and adequacy of projects, the
improvement of communication in the implementation process and with the funders,
the development of partnerships. These results have been reported by major donor
agencies, which currently use similar approaches to planning and development. The
methodology allows project managers to routinely use monitoring and evaluation as
management tools. The use of standards which is now internationally required is
important in attracting other funds.
The added training of trainers element is aimed at the spread of the methodology
within the network, but also at the creation of capacity in the foundations for the
provision of skills which become demanded in their own countries, in the process of
transition.
18
Appendix 2: Project description (Logframe)
(attached Excel file)
19
Appendix 3: Report September Workshop
"Integrated Project Planning and Evaluation"
workshop
Budapest, September 7-13, 2000
Aims
The workshop is part of the IEP effort to respond to the request of several
foundations to acquire the skills of participatory (and standardised) planning and
evaluation methodologies. This methodology will not only assist the project
management activities of foundations, but will provide access to funding from other
sources, as most major donors now request proposals produced using the
methodology.
Description
The workshop combined training for facilitators in participatory planning and
evaluation with the acquiring of the standardised methodology. Six foundation
representatives and a group of IEP staff have been trained during September 7-13
(participant list attached - Annex 1). The trainers have been Ulrich Schiefer,
professor at the Higher Institute for Management Studies in Lisbon, Portugal, and
Reinald Doebel, from the Institute of Sociology in Muenster, Germany.
The workshop lasted five full days.
The evaluation of the workshop was made for criteria set both by participants, and by
the facilitators, with marks shown in the table below (on a five-step Lickert scale
weighed 1 to 5, as follows: 5 eq. ++ (very good), 4 eq. + (good), 3 eq. 0 (satisfactory),
2 eq. - (poor), 1 eq. -- (very poor)):
INDICATOR
relevance of methods and techniques 4.55
relevance of content for work 4.90
variety of content 4.22
usefulness 4.83
applicability 4.42
completion of programme 3.80
general participant satisfaction 4.91
general satisfaction with process 4.58
satisfaction with facilitation 4.36
adequacy of trainers 4.60
follow-up plans exist 3.25
clarity of communication 4.13
internal logic of workshop 4.45
adequate pace 3.91
level of commitment to new method 4.10
level of understanding of the process 4.20
participant capacity to deliver workshops 3.89
TOTAL 4.31
The workshop had an overall rating of 4.31, with no marks below 3 (scale from 1 to
5). The highest marks have been obtained for participant satisfaction, and for
20
relevance of the content and materials for their work. The most critical mark (3.25)
reflected participant concern about the possible follow up. Apart from the rating, the
comments in response to the evaluation questions have been gratifying both to
participants, and to the trainers.
Content
In the current context, by "project" we understand a project for social intervention (not
an investment project, which requires a different methodology). A project is
characterised by a timeframe, runs activities which produce results, the results need
to contribute to the achievement of an overall purpose, is influenced by the
environment, requires resources, must be co-ordinated, and its progress must be
reflected in measurable indicators, with reliable sources.
Logframe represents a visualisation technique for the internal structure of a project;
it is a logical framework which reflects and helps construct the internal logic of a
project. The filling in of the logframe matrix - which is the planning of the project
itself - must be achieved in a participatory manner - and the methodology used for
this is MAPA (Annex 4). MAPA has the strength of using parallel procedures for the
planning and evaluation of projects, allowing the replanning of a project during any of
its implementation phases.
Logframe arranges the goal, objective, activities and assumptions of a project in a
logical sequence. Annex 2 describes the logframe matrix with the OSI terminology,
with the logical inferences. The assumptions take into consideration the environment.
The intervention logic describes the project, as it develops logically from the goals,
objectives, and activities (with results). There should be one logframe matrix per
objective, this is why the matrix mentions just one project purpose (objective). The
achievement of results, objective and goals are described by measurable indicators,
the description of which must also be included in the matrix at the moment of
planning of the project.
The goal, objective, results, must be expressed as affirmative statements, including a
timeframe, and a measurable target (avoiding general statements such as "improve
education" - but replace them with clear statements, such as: "By the end of 2005,
have in place a quality assurance system for general education.")
The indicators must be measurable (either qualitative, or quantitative) and must have
sources. An indicator without a source is not valid. There may be cases where the
project must include activities to collect data for its indicators.
The activities must be planned in detail, using the operational plan (Annex 3).
The last lines of the matrix, covering the results and activities, are the responsibility
of the Project Manager, who may reallocate resources within activities, so that he
produces the results, on condition s/he stays within the agreed budget frame. The
Project Manager may not change the objectives. Reciprocally, the project manager
should be allowed to organise his or her work without interference from the
organisation, as long as monitoring results are satisfactory, and the budget frame is
respected.
The basic characteristics of a project, reflecting the actual change it achieves, and
the cost of it, are:
21
- the efficiency of the project, measuring the rate result / input (at what cost are the
results obtained - where "input": represents the sum of resources used in the project)
- the effectiveness of the project, measuring the rate purpose / input (the extent to
which the project manages to actually achieve its purpose, with the given costs)
- the adequacy of the project, comparing the intervention logic to the assumptions
(the extent to which the project is adequate to its environment).
All these indicators describing the project can be calculated from the logframe matrix.
The methodology has a record of helping improve these characteristics.
When filling the operational plan, where there are activities depending on each other,
care must be taken to insert time buffers in order to allow for possible delays.
Activities needing the same resources need to be planned at different times. Critical
sequences of activities will need special attention (the operational plan highlights
them). Usually, the projects we implement so not need more complicated tools (such
as the pert graphs0 for the planning of activities.
The project needs to be monitored - that is, register what happens at all levels (also
with the assumptions). The monitoring has different time-scales for the different
levels of the registering: for instance, activities may need to be monitored weekly,
results monthly, objective (purpose) yearly, and the goal, during the evaluation
procedure. Care must be taken when monitoring, that the indicator is not placed
before the objective (that is, the attainment of an indicator may hide the non-
achievement of an objective - for instance, the production and installation of the
planned length of fences by a factory, which planted the fences in the middle of
nowhere, in order to declare the objective of planting that length of fences achieved:
just that the fences should have been of use to someone).
The goal and objective (purpose) of a project are the result of a diagnostic, or
problem analysis (needs assessment, and participatory discussion of the problem).
The assumptions must be subjected to a risk analysis, at least using the simplest
method of "probability versus impact quadrants" (Annex 5).
When analysing the environment, variables must be checked, which influence the
project. Common errors are a) the use of assumptions as reality; and the other
extreme b) the use of too many assumptions (information not necessary must be
ignored).
Process
The workshop used two synergic techniques in order to develop the planning and
evaluation skills. On one side, the content was delivered and discussed, on the other
side, the team performed an actual planning of a project.
The project started with a needs-analysis (diagnostic), triggered by the question:
"What are the problems you see in the network in relation to project planning
and evaluation."
An initial session of brainstorming for answers to this question resulted in the
description of a first set of problems. An important idea of the method is that there
must be no ownership of the problems, and no debate around them. Each member of
22
the group received then a number of "votes" to attribute to problems he or she
considered more relevant. After this first sharing and individual analysis of problems,
a second session of brainstorming produced a new set of problems, added to the
initial one. The second round of "voting" for problems was organised. The voting
procedure allowed each member of the group to individually analyse all the
problems.
Further analysis of the problems was made by the team by organising the problems
into categories ("clouds"), and then attempting to make a strategic choice of what
would be the main problem to be tackled in each cloud. The other problems in the
cloud were arranged in a causal relationship with this main problem: either causes, or
effects.
At this point of the planning process, all members of the team had a wider view of the
answers to the initial question. They had become familiar with the perspectives of
different stakeholders (in this case, comprised by foundation staff, local network
programme staff, IEP staff), and could see the areas with problems ("clouds") and
the causal relationships. (Annex 6 contains the list of problems identified, with the
number of "votes".)
Based on this knowledge, another major strategic decision was made, on the main
problem to tackle with a project.
The logframe methodology was used for the planning from this stage on. The
participants in such a planning workshop do not need to master the technology: the
facilitators are the ones guiding the thinking along the logical framework. At the end
of the workshop, the result is a commonly-agreed plan, with clear objectives, results,
and indicators to help the programme managers and funders monitor it.
The project answering to the question of solving problems with planning and
evaluation is presented in Annex 7. The following workshop will finalise the plan by
adding the operational details.
The following workshop will also model an evaluation, and demonstrate the link
between projects (programmes) and strategy. Besides finalising the project they are
now part of, participants will take part in the evaluation of the strategy development
process, and replan any further stages, if need will be acknowledged.
The following workshop is also aimed at modeling the parallel processes of planning
and evaluation used in the MAPA methodology, allowing the replanning of a project
during any phase of its implementation.
An important element of the methodology is that the participatory planning or
evaluation workshops are complex and energy-consuming. For this reason,
facilitators must always work in pairs. The workshop itself modeled the interaction
between the two facilitators.
After having planned the project, the initial problems have been revisited, and
checked whether they were answered by it. the group considered that most of the
problems have been taken into account - and either influenced the "assumptions"
field, or were addressed by activities.
Another exercise meant to strengthen the project was made by participants who have
been requested to change roles, and judge the project from the perspective of a
donor. More, the participants have been requested to criticise the project as heavily
23
as they could, from all perspectives. The question was: "why do you consider this
project is not worth funding". The brainstorm on drawbacks of the project pointed to
the need of changing the goal and adjusting the objective. the internal logic of the
project was not attacked, and proved solid.
Skills
At the end of the first seminar, participants can facilitate participatory planning
workshops together with a tutor, and can use correctly the logframe technology (that
is, away from the common and flawed understanding of "placing goals and objectives
in a matrix"). The correct use of logframe is within participatory workshops, having
the role of focusing the thinking of a group which is working together on a given task.
The logframe should not be the result of the work of a lonely planner.
Outcomes
Three evaluation and replanning workshops will happen until the end of December.
The first one, replanning a textbook development programme, brought a meaningful
contribution to the programme, solving important problems. Four of the six foundation
participants in the first workshop have answered that they have started to use (or
planned to use) the technology already.
The Council of Europe approached the IEP with the request to share the proposal for
a similar cascaded training for NGOs and institutions in south-east Europe. This
proposal may be discussed with participants at the following seminar, and will be
developed jointly if foundations have an interest in becoming involved in these
activities.
LB - September 2000
24
Appendix 4: Report December Workshop
"Integrated Project Planning and Evaluation"
workshop II
Budapest, December 6-12, 2000
Aims
The workshop is part of the IEP effort support the full cycle of strategy development,
which includes the evaluation and re-planning phases. It offers an operationalised
methodology and tools to integrate the strategy development loop. This methodology
will not only assist the project management activities of foundations, but will provide
access to funding from other sources, as most major donors now request proposals
produced using the methodology. It is therefore meant as an important support for
sustainability.
Description
The workshop combined training for facilitators in participatory planning and
evaluation, with the acquiring of the standardised methodology. It continued the
training started in September, as described in Annex 2 (concept paper for the
training). Ten foundation representatives and a group of eight IEP staff, including the
IEP senior management, have been trained during December 6-12 (participant list
attached - Annex 1). The trainers have been Ulrich Schiefer, professor at the Higher
Institute for Management Studies in Lisbon, Portugal, and Reinald Doebel, from the
Institute of Sociology in Muenster, Germany.
The workshop lasted five full days.
The evaluation of the workshop was made for criteria set both by participants, and by
the facilitators, which were developed as an evaluation exercise at the September
workshop. Evaluation results are shown in the table below (on a five-step Lickert
scale weighed 1 to 5, as follows: 5 eq. ++ (very good), 4 eq. + (good), 3 eq. 0
(satisfactory), 2 eq. - (poor), 1 eq. -- (very poor)).
INDICATOR
relevance of methods and techniques 4.58
relevance of content for work 4.40
variety of content 3.78
usefulness 4.33
applicability 4.42
completion of programme 3.00
general participant satisfaction 4.30
general satisfaction with process 4.10
satisfaction with facilitation 4.08
adequacy of trainers 4.62
follow-up plans exist 4.00
clarity of communication 4.00
internal logic of workshop 3.73
adequate pace 3.82
level of commitment to new method 4.40
level of understanding of the process 4.18
participant capacity to deliver workshops 4.09
TOTAL 4.11
25
The workshop had an overall rating of 4.11, with no marks below 3 (scale from 1 to
5). The highest marks have been obtained for the adequacy of trainers, and for
relevance of the content and materials for the work of the participants. The most
critical mark (3.00) reflected participant concern about the extent of content to be
covered. We have been aware of the difficulty of presenting in detail all tools, and
have requested the preparation of a handbook adapted to the needs of the
foundations and spin-offs (first draft of the handbook was distributed at the end of the
workshop for feed-back). We hope that this handbook, together with the follow-up
exchanges, will support the mastering of the entire content.
A rewarding element in the evaluation was the important increase in the rating for
follow-up plans, as compared to September (up to 4.00, from 3.25). We think this
reflected a relationship of trust, an ongoing dialogue among the participants in the
first training session, and most of all, a response to the sharing of experience by
foundations which already tested the method. The discussion accompanying the
evaluation offered not just feed-back, but also suggestions for the follow-up. The
representatives of foundations have suggested the set-up of a support mechanism, a
quality-control loop for the application of the methodology, and an the set-up of active
communication channels to support the exchanges among users of the methodology
and immediate access to support. All these suggestions have been taken into
account in the planning of the network support activities to follow up the training.
Content
The content delivered contained MAPA, logframe, and a battery of tools needed for
project planning and evaluation.
MAPA (Method for Applied Planning and Assessment) is an operationalised method
for participatory planning and evaluation. The MAPA procedures support the
development of operational goals, objectives, results and measurable indicators, with
the participation of stakeholders, creating ownership and fostering transparency of
the project. MAPA has the strength of using parallel procedures for the planning and
evaluation of projects, allowing the replanning of a project during any of its
implementation phases. This replanning can be used to construct monitoring
indicators, to answer to changes of the environment, or to investigate the existing
logic of the project.
Technically, MAPA consists in the description of a process consisting in a series of
workshops and follow-up activities. It defines specific skills for the facilitation of these
workshops, defines the outputs and interactions of the process. The MAPA training
develops skills for the production of the desired output and the participation of
stakeholders.
LOGFRAME (logical framework) is the framework which helps construct the internal
logic of the project. It assists the process of defining operational goals, objectives,
activities for projects and prompts towards the research of assumptions.
Furthermore, it helps the understanding of the interaction among these elements by
placing them in a logical sequence. It also connects the elements of the planning
process with the implementation itself, by assisting the development of measurable
indicators for the monitoring of the initial state and further progress of the project.
Technically, logframe consists in a matrix where all elements of a project need to be
arranged, and where logical inferences are defined among the cells of the matrix.
26
OTHER TOOLS: tool for establishing criteria for the integration of programmes into
strategies, tree model, Ishikawa model, Eisenhower model, risk analysis matrix,
target group analysis, stakeholder analysis, gender analysis, visualisation, facilitation,
needs assessment (diagnostics) tool, GANTT diagram, operational plan for
evaluation, survey.
A first draft of the handbook to cover this content was handed out to participants, at
the end of the workshop. The handbook will comprise detailed descriptions of all
tools and techniques. References and copies of supplementary materials have been
distributed to participants.
Process
The workshop used two synergic techniques in order to develop the planning and
evaluation skills. On one side, the content was delivered and discussed, on the other
side, the team performed the simulation of the evaluation of a project (rather, a set of
activities in which all foundations and the IEP have been involved, comprised under
the name of "Strategy Development and Review Process").
The project started with a needs-analysis (diagnostic), triggered by the question:
"What are the issues you see in the development and review of education
strategies."
The question itself raised debate as to the breadth of the question: some participants
wished to include the strategy implementation process into the question, as a part of
the loop of implementing change. The term strategy itself was discussed, so as to
distinguish "strategy" as a deployment of forces to achieve a goal, from "strategy
paper" - which is a document. The clarification of terms helped develop a common
agreement on the breadth of the question.
Due to the complexity of the problem the simulation did not produce the full
operational plan for the evaluation. It was noted that the simulation needed to reflect
problems of interest to all participants, to be as close to reality as possible, in order to
be a valuable training process.
The development of criteria for the integration of projects into the strategy was
simulated using the education strategy offered by one of the participants.
The two simulation processes brought back to attention the project developed in
September, of supporting the introduction of an integrated methodology for project
planning and evaluation into the network. New activities have been added to the
project, and the objective itself was further clarified.
Outcomes
The most important moment of the workshop was the sharing of experience of
participants who are now in their second training workshop, and who already started
using the method in their foundations. These participants presented their results, and
also offered suggestions for the support to be further provided by the IEP. They
proposed the integration of a quality-assurance loop in the monitoring of the
programme in order to ensure good use of the methodology, involvement of
foundation executive directors in the process, set up of a technical support
mechanism. Participants also suggested the development of an intensive two-week
training course, possibly as summer school, and the sharing and discussion of
samples of materials (proposals, evaluation reports, plans of projects, etc.).
27
LB - Dec. 2000
28
Appendix 5: Report February Workshop
"Integrated Project Planning and Evaluation"
workshop III
Budapest, February 7-13, 2001
Aims
The workshop offers an operationalised methodology and tools to integrate the
strategy development loop (strategic planning - project planning end evaluation - re-
planning). This methodology compliant with international standards, is not only meant
to assist the project management activities, but also to support access to funding
from other sources and the development of a common language of development. It is
therefore meant as support for sustainability and networking.
Description
The workshop combined the training for facilitators in participatory planning and
evaluation, with the acquiring of the standardised methodology. It represents one of
the two training workshops needed for full training, as described in Annex 2 (concept
paper for the training). Thirteen representatives of foundations, of two OSI-funded
organisation, of one partner organisation and a group of two IEP staff, have been
trained during February 7-13, 2001 (participant list attached - Annex 1). The trainers
have been Prof.Dr. Ulrich Schiefer, professor at the Higher Institute for Labour and
Business Studies (ISCTE) in Lisbon, Portugal, and Dr. Reinald Doebel, from the
Institute of Sociology of the University of Muenster, Germany.
The workshop lasted five full days, plus a half-day meeting for the discussion of a
case study, proposed by one of the participants.
The evaluation of the workshop was made for criteria set by participants, developed
as an evaluation exercise. Evaluation results are shown in the table below (on a five-
step Lickert scale weighed 1 to 5, as follows: 5 eq. ++ (very good), 4 eq. + (good), 3
eq. 0 (satisfactory), 2 eq. - (poor), 1 eq. -- (very poor)).
INDICATOR
efficiency 3.38
new knowledge and skills 4.13
clarity 4.08
interactivity 3.85
satisfaction of participants 3.53
performance of trainers 3.90
meeting expectations 3.47
working atmosphere 4.00
logistics of workshop 4.83
efficacy of workshop 4.00
will the method work 4.36
balance between content parts 3.23
TOTAL 3.90
29
The workshop had an overall rating of 3.90, with no average marks below 3 (scale
from 1 to 5). The highest marks (apart from the logistics) have been obtained for the
adequacy of the method to participant purposes ("will the method work"), and for the
assimilation of new content and skills. The most critical mark (3.23) reflected
participant concern about the balance between content parts, with suggestions for
specific changes. As the training is content-loaded, and this workshop represents the
first part of the whole training for almost all participants, this concern was somewhat
expected. Nevertheless, we plan to include participants suggestions for changes in
the further workshops. In doing this, we will have to take care of the balance of
interests and needs of participants (for instance, participants in the previous
workshop encouraged us to aim for obtaining the full products of the simulation
exercises, rather than point out to ideas, and to have a detailed exercise on
indicators - which in turn seemed a bit lengthy to some of the current participants).
We have been aware of the difficulty of presenting in detail all tools, and have
requested the preparation of a handbook adapted to the needs of the foundations
and spin-offs (the second draft of the handbook was distributed to participants,
having incorporated the feed-back of the first round of consultation, and included
most annexes). We hope that this handbook, together with the follow-up exchanges,
will support the mastering of the entire content.
For the previous similar training workshops, individual marks given by participants
have been grouped (all marks in the range 3-5, with exceptional 2). This time,
although the overall workshop mark is relatively the same, the marks have been
polarised, with a higher number of maximum marks, and at the same time one-two
participants awarding the lowest marks for several criteria. This suggested a great
difference in individual needs, most probably resulting from different expectations.
The group was least homogeneous this time. The evaluation result is suggesting the
need for a stronger homogeneity of the participant group, in terms of a common
starting point and awareness of working methods, not in terms of organisational
belonging. Participant suggestions of initiating e-mail discussions on the topic, prior
to following workshops, will be taken into account in order to strengthen this point.
A formative evaluation exercise was performed mid-way through the workshop, as
trainers and organisers noticed these possible differences, mainly manifested
through different levels of participation in group exercises. The formative evaluation
had a less participative form than the one prescribed by trainers during previous
workshops (consisting in the defining of criteria and marking by participants, without
direct interaction with the trainers and organisers), and lacked the dialogue element
of previous evaluations. We believe that due to this missing element, the feed-back
loop did not have the expected effect (formative and final evaluation results are
roughly identical, as evaluation marks have been requested by mail from participants
who left early in the day - in order to maintain the representation allowing
comparison). This pointed out to the need to enrich the pure methodological training
with interactive elements of open dialogue.
Content
The content represented a battery of tools needed for project planning and
evaluation.
MAPA (Method for Applied Planning and Assessment) is an operationalised method
for participatory planning and evaluation. The MAPA procedures support the
development of operational goals, objectives, results and measurable indicators, with
the participation of stakeholders, creating ownership and fostering transparency of
the project. MAPA has the strength of using parallel procedures for the planning and
evaluation of projects, allowing the re-planning of a project during any of its
30
implementation phases. This re-planning can be used to construct monitoring
indicators, to answer to changes of the environment, or to investigate the existing
logic of the project.
Technically, MAPA consists in the description of a process consisting in a series of
workshops and follow-up activities. It defines specific skills for the facilitation of these
workshops, defines the outputs and interactions of the process. The MAPA training
develops skills for the production of the desired output and the participation of
stakeholders.
The skills requested by MAPA are:
a) facilitation of participatory workshops;
b) knowledge of attached planning tools, such as LOGFRAME.
LOGFRAME (logical framework) is the framework which helps construct the internal
logic of the project. It assists the process of defining operational goals, objectives,
activities for projects and prompts towards the research of assumptions.
Furthermore, it helps the understanding of the interaction among these elements by
placing them in a logical sequence. It also connects the elements of the planning
process with the implementation itself, by assisting the development of measurable
indicators for the monitoring of the initial state and further progress of the project.
Technically, logframe consists in a matrix where all elements of a project need to be
arranged, and where logical inferences are defined among the cells of the matrix.
OTHER TOOLS: tools for problem analysis, risk analysis matrix, target group
analysis, stakeholder analysis, visualisation techniques, facilitation techniques,
implementation plan diagram, organisational landscape diagrams.
A first draft of the handbook to cover this content was handed out to participants, at
the end of the workshop. The handbook will comprise detailed descriptions of all
tools and techniques. References and copies of supplementary materials have been
distributed to participants.
The structuring of the succession of use of the MAPA tools is taught through
simulation exercises.
Sequence of MAPA steps: MAPA is based on the Project Cycle management
(PCM). For the planning process, the purpose of the MAPA workshops is to start with
some potential problems, and possibly with an idea to tackle them, and to end with a
detailed plan for dealing with the identified problems. The whole process must be
achieved with the input of representatives of most (if possibly all) stakeholders.
Planning steps:
1. The first step in the process, is the diagnostic step. This identifies the needs for the
area at focus. The aims of this first step are:
a) to possibly identify all problems viewed from all stakeholder perspectives, for the
area at focus;
b) to familiarise all participants with the identified problems and perspectives.
As the scope of the first step is very broad, it is essential that the initial question,
which identifies the area at focus, be very clear to all participants, and well thought-
through in advance. The question, once stated, will be discussed with participants for
clarification, and then written down. It is possible that the broadness of the area at
31
focus is unknown to the facilitators of the process. In this case, and alternative
technique, or identifying the question itself together with the participants might be
used. The result should be equally clear and understandable to all participants.
The answer to this question will be a list of problems, as seen by participants. In
order to provide equal representation of all participants, the problems are listed on
identical cards, size A5, one card-one problem, in capital letters. The problems
equally identified by any previous needs-assessment or diagnostic exercise will be
fed into this process, equally written on identical cards.
The problems identified as an answer to the initial questions will be displayed and
read one by one. Participants will be requested to rank them ("validate them"), in the
order of the importance. This exercise is useful in familiarising the participants not
only with the problems, but also with the perspectives of the others: they may often
acknowledge that other participants identified issues more relevant than their own.
Participants will most likely remember more details about the initial question, after
having read the views of the others. Therefore, the facilitator will initiate a second
round of collecting problems on cards, and a second ranking exercise. This exercise
marks the end of step one. At the end of this step, the group provided an answer to
the triggering question, and the answer is displayed in the form of cards, in a random
order, and with different rankings awarded by every participant.
It can be seen that for this step of the process, the presence in the room of experts in
the area of focus is most welcome, as their input will add "feasibility issues" and other
"expert issues" to the diagnostic process.
It is useful to mention to participants that all problems, not just the highest ranking,
will be taken into account. This is mandatory for a truly participatory process.
Nevertheless, the order of further dealing with problems will always start with the
highest ranking ones.
2. The second step of the process, is the organisation of the identified problems into
categories, and further into more structured data structures, in order to make them
approachable.
The identification of categories can be performed either by grouping problems which
"seem related in a well-identified way", or by brainstorming for categories and then
grouping problems around the identified categories.
It is advisable that the process of grouping the issues is closely monitored by the
facilitator not only by way of assisting the group to find its centre and to make the
decisions, but also by having in mind the need to obtain a manageable number of
categories (advisable between three and eight), and that the categories are clear and
if possible, with little overlapping. In case the facilitator does not seem confident to
have an input in this direction, it is advisable that s/he invites an expert to assist the
group in this endeavour (as participant).
In case the problems identified can be relatively easily placed in a cause-effect
relationship, most methodologies advise the further organisation of problems in each
category ("cloud") into cause-relationship trees. The set of problems may suggest
that all clouds are placed into one tree. It is clear at this point, that the facilitator
needs project management skills, in order to assess the feasibility of issues to be
tackled by a project, and to trace back to this moment in time, the organisation of
32
problems to assist the development of feasible projects. While the facilitator is not yet
confident in this task, it is again advisable to have a tutor, or an expert in the room.
Problems which seem to be isolated, will be placed into a special category,
provisionally labeled as "treasure box."
Development handbooks mention the clear impossibility to design "perfect projects" -
and the ownership and agreement reached through participation may overcome
many of the imperfections. Nevertheless, it is important that the result (planned
project) be feasible and relevant to the ones who took part in its planning. Therefore,
while carefully managing the participation of all members of the group in the process,
the facilitator will have to have this aim in mind all the time, and be aware both of the
time constraints, and of the methodological requirements of each step of the planning
process.
Alternative tools rather than the tree may be used to structure the problems (fish
diagram, Eisenhower model). The tree was chosen as main tool because there can
be straightforward translation of problems into objectives in the case of simple areas
of focus. This is not the case with more real-life problems tackled with social
intervention projects.
3. The third step of the process, and probably the most difficult one, as it is not
methodology-driven, is the one to identify goals and objectives for the project. This
step includes an active support from the facilitator to the group to achieve its
outcome, while the facilitator will have to avoid "driving the group" or "making
decisions on behalf of the group.".
It is almost impossible, in a real and complex situation, to achieve a "simple tree",
with one core problem, and all causes grouped below, which allows for an "automatic
construction of the tree of objectives" - therefore use almost-automatic
methodological tools in identifying objectives.
The participants need to be assisted to focus on what they see as the core problem
to be solved - and to decide whether this can be approached through a project they
wish to plan, or is a wider aspiration. In the first case, this core problem is
transformed into a project objective - else, it becomes a provisional goal (to which the
project will aim, but not fully achieve). The core problem may be not yet among the
problems identified by participants, but described by them at this point.
4. The fourth step of the process, is the creation of the logical framework of the
project planned to achieve the identified goal. Again, automatic translation from the
problem-tree to the objective tree (see PCM handbook), and then into the logframe
matrix, is rarely feasible. Nevertheless, this methodology may support the thinking in
the construction of the project.
Rules in describing each element of the matrix are advised, in order to support the
further planning steps: objectives must be expressed as sentences with active-tense
verbs, time frames and target values, etc. (see MAPA handbook).
This step, similar to the previous two, again requests expertise in project
management, in order to assist the group in developing a realistic project. As noticed
throughout the planning steps (and it will be similarly complex for evaluation), there is
a great complexity of skills and a high level of attention and involvement needed for
the MAPA facilitators. The methodology therefore requests facilitators of MAPA
33
workshops to work in pairs, in order to provide feed-back, support, and to exchange
the rhythm of facilitation during the process.
5. It is imperative that at the end of the process, a "mopping-up exercise" is
performed, in order to clarify for participants the way all initial problems have been
taken into account - and to make sure that in fact, every problem was tackled in
some way. In case this has not happened, a discussion will be launched to see in
which way these problems can be approached (within assumptions, supplementary
activities, etc.).
6. If there is time available, a new exercise, the "kill the project" approach, can be
attempted with participants. This exercise can assist in the identification of possible
weaknesses of the project, or of potential threats. If internal structural discrepancies
of the logframe are noted, the logic has to be revisited. Participants can also be
encouraged to find ways to tackle external threats (redefining the project goal,
accompanying the project with specific lobbying activities, etc.).
Evaluation steps:
MAPA designs the evaluation steps in a parallel sequence to the planning steps, in
order to allow for possible re-planning of a project, irrespective of its planning
methodology.
1. The evaluation questions must be clearly identified in advance, and addressed to
the group at the start of the session, in order to lead to the identification of evaluation
criteria. For a formative evaluation, these questions should identify whether the
project is still on track, and whether the environment is still the same. The responses
to the evaluation questions are collected similarly to the problems from the planning
process
2. The second step groups the identified problems into categories, which will become
the identified criteria (goals) to be evaluated. Added criteria coming from an external
evaluator, or pre-planned, can be fed into the process at this point.
3. The third step consists into the identification of indicators for each criterion. The
facilitator will have to assist the group in the understanding of the meaning of
process-result indicators, as well as of input-output-impact indicators. This should be
done without entering the technical slang, by pointing to the elements needed to be
evaluated, and to the moments in time.
4. For each indicator, the group will have to identify a source, form in which it is
available, method of collection and analysis, and assign the task. This exercise is
useful, as it will point to the complexity of the task for obtaining each indicator, and to
the feasibility of the task. At the same time, this will lead to the creation of an activity
plan for the actual production of indicators.
Bibliography: mainly MAPA Handbook, but also GTZ handbooks
(http://www.gtz.de/pcm/download/english/zopp_e.pdf and for more handbooks,
http://www.gtz.de/pcm/deutsch/instrumente.htm), SDC handbooks
(http://194.230.65.134/dezaweb2/root/Publications/Zentrale_Dienste/Publikationen/St
rategisches_Controlling/D_Methodologie_und_Entwicklungszusammenarbeit_2000.a
sp), EC handbook
(http://europa.eu.int/comm/europeaid/evaluation/methods/pcm.htm)
Process
34
The workshop used two synergic techniques in order to develop the planning and
evaluation skills. On one side, the content was delivered and discussed, on the other
side, the team performed the simulation of the planning of a project followed by the
simulation of an evaluation.
The planned project was triggered by the following question:
"What problems do you see in your organisation, related to project planning
and
evaluation."
The question led to a set of problems (Annex 3), initially grouped into seven
categories, resulting into two main problem trees (knowledge-and-skills-related
problems, and management-related problems, apart from environmental problems).
The group decided that two parallel projects needed to be run in order to tackle the
problems, and produced a logframe plan for the project dealing with the knowledge-
and-skills-related problems (Annex 4).
The evaluation simulation was triggered by the following question:
"What problems do you see in your strategy development process?"
The question led to a set of problems (Annex 5), and their grouping led to the
identification of twelve evaluation criteria. The indicators developed for these criteria
are listed in Annex 6 - and if the participants find them useful, they may use them in
their own strategy planning work (keeping in mind the fact that the indicators have
been obtained by analysing "a virtual strategy development process" - and that this
one most probably did not include many of the real problems that their own
organisation faced).
Outcomes
The most important moment of the workshop was the sharing of experience of
participants who already started using the methodology. The presentations of Friday
afternoon led to a supplementary session on Saturday afternoon, where the group
focused on an individual strategy in order to provide advice for further planning. This
moment was considered a valuable experience by all members of the workshop, and
a proposal was made by participants to repeat similar events.
The participants provided invaluable suggestions for the follow-up of the workshop:
a) the creation of an e-mail list of the participants in the MAPA workshop, to act as
"planning and evaluation circles" within the network (implemented by the IEP this
week);
b) the creation of a glossary of development terms, with definitions, to be added to
the handbook, and translated into languages of the region (started by the handbook
developers, and soon to be discussed on the listserv);
c) the organisation of similar workshop in other countries (included in the cascading
plans, to be attached as Annex 7);
d) the on-line discussion of a real project, which to then become the simulation topic
for a following workshop;
35
e) the on-line "start of the workshop", for new participants, aimed at familiarising
them with the techniques and with the content;
f) the sharing with participants of "the logistics list" for materials (the handbook
contains details in this respect - I also added the list we used since September, as
Annex 8)
g) the clarification of participant profile prior to the workshops (group work, detailed
syllabus)
h) the development of an intensive two-week training course, possibly as summer
school
i) the sharing and discussion of samples of materials (proposals, evaluation reports,
plans of projects, etc.)
j) the presentation of projects and strategies of foundations and spin-offs, for
discussion and advice.
The results of this workshops are considered by the organisers a major and
rewarding step towards closer networking and good bases for co-operation. This is
becoming more and more a strength of the network of Soros foundations and related
organisations.
LB - February 2001
36
Appendix 6. May 2001 Report
Report on the
MAPA Training Workshop
Tbilisi, May 31 - June 5, 2001, Open Society Georgia Foundation
Aims
The workshop was part of the IEP effort support the full cycle of strategy
development, which includes the evaluation and re-planning phases. It offered an
operationalised methodology and tools to integrate project planning into the
development of strategy. This methodology is intended to assist the project
management activities of foundations, but will also ease access to funding from other
sources, as most major donors now request proposals containing elements which
form essential parts of the methodology. Its use should therefore contribute to the
sustainability of the work of foundations and spin-offs.
Description
This training workshop was the first one offered by IEP outside of Budapest and with
Russian as the medium of instruction, and thus represented the first step in
“cascading” the training in a standardized methodology of participatory planning and
evaluation into the network of foundations – as envisaged in earlier training-cum-
planning workshops in Budapest (see reports on the September 2000, December
2000, and February 2001 workshops on this website) and following the agenda sent
out prior to the workshop (see Annex 1). The 15 Participants came from Azerbaijan,
Georgia, Kazakhstan, Ukraine, and Uzbekistan (participant list attached - Annex 2).
The trainers were Igor Padure (Micro-Finance Alliance, Moldova) and Vitalie Popa
(Pro-Didactica Centre, Moldova), backed up by Reinald Doebel (Institute of
Sociology, University of Muenster, Germany.) as tutor.
As in previous workshops, the content delivered covered the MAPA method, the
“logical framework” (logframe) for a structured complete representation of a project
(see annex 5 for the logframe developed during this workshop and the handbook for
a detailed explanation), and a number of additional tools needed for project planning
and evaluation (problem analysis (see annex 3 and 4 for “problem clouds” and an
exemplary “problem tree,” stakeholder analysis, institutional landscape, risk analysis,
participation, collaboration with experts).
MAPA (Method for Applied Planning and Assessment) is an operationalised method
for participatory planning and evaluation. The MAPA procedures support the
development of operational goals, objectives, results and measurable indicators, with
the participation of stakeholders, creating ownership and fostering transparency of
the project. MAPA has the strength of using parallel procedures for the planning and
evaluation of projects, allowing the replanning of a project during any of its
implementation phases. This replanning can be used to construct monitoring
indicators, to answer to changes of the environment, or to investigate the existing
logic of the project. For details see the handbook available for download on this
website.
37
Technically, MAPA consists in a series of workshops and follow-up activities with
clearly defined activities, interactions, and outputs. The MAPA training develops
skills for the production of the desired outputs and the participation of stakeholders.
For this reason, the process of the training workshop is an alternation between
structured inputs and the simulation of the planning of a real project (during this first
of a series of two workshops: the second will be devoted to evaluation; see also the
section on Evaluation and Follow-Up)
A brief chronology of the workshop
The Tbilisi workshop had to cope with the difficulty of more than one third of the
participants arriving only in the afternoon of Thursday (May 31, the scheduled first
day of training): the train from Azerbaijan only arrived in the afternoon of that day.
Therefore, all introductory speeches (by Shalva Dundua on behalf of OSGF as the
host of the workshop, Natalya Shablya on behalf of IEP, the institution which initiated
MAPA trainings and continues to support them, and the tutor and trainers) were very
brief. To make up for time lost, starting time in the morning was advanced to 9.00
a.m. and the training continued longer in the afternoons of Friday and Saturday.
As a start-up for the training and a preparation for the diagnostic phase of the project
simulation exercise – which begins with a collection of problems and therefore has a
negative touch for many people especially from this cultural region of the world -
participants were then asked to turn to the person they knew least and tell them
about the greatest achievements of their organisation. The listener in this exercise
then presented these achievements to the group. As predicted by the trainer’s earlier
experience with this exercise this “warm-up” provided the group with a sense of
achievement as a stabilizing force against which the collection of problems could
proceed.
Before that, the first input section of the workshop started with a brief discussion of
the methods used in the participants’ organisation for project planning. It turned out
that some had started from a needs analysis while others had followed prescribed
formats from other organisations. Against this backdrop of diversity the trainer then
clarified the meaning of the terms “project,” “planning,” “evaluation,” and especially
the essential aspects of all planning and evaluation: “effectiveness,” “efficiency,”
“impact,” “relevance,” and “sustainability.”
Then participants went into the diagnostic phase of the project simulation
(“Introducing a standardized method of planning and evaluation in the network”). The
guiding question for the collection of problems was:
“What problems does your organisation (and you personally) experience with the
planning and evaluation of projects?” The personal aspect in the question was
emphasized in order to avoid that participants would restrict their forwarding of
problems to those which were deemed “officially acceptable” from the point of view of
their respective organisations.
Before breaking for dinner (which deserves special praise - as all the meals organized
by host) the wall reserved for that purpose was almost filled and the problems were
“validated” with adhesive dots (as part of the MAPA methodology, see also the
handbook available on this website and distributed to participants at the end of the
workshop).
38
Friday morning participants again went through these problems and – with the help
of participants as volunteer facilitators – collected additional problems with respect to
the planning and evaluation of projects, which were again visually “validated.” In this
way the discussion among the trainers whether it would be preferable to leave the
collection of problems intact as one stage, or to break it into two parts instead, was
resolved by the circumstances of dinner waiting for participants who had already
worked longer than anticipated ...
The main “teaching” input on Friday concerned the relationship between “problem
analysis” (as a kind of “needs assessment” done in a participatory manner with all
participants of a real planning workshop) and the following stage of defining goals
(“objectives” – i.e. “Goal,” “Purpose,” “Results,” and ”Activities” in the logframe
parlance). It was made clear that in international practice the “objectives” must be
linked to and aimed at solving real problems in real situations.
The rest of the day was devoted to grouping the all the problems into “problem
clouds” (see annex 3) and to structure the problems of at least one of these problem
areas into a “problem tree” defining cause-and-effect relationships between these
problems (see annex 4 for the problem tree). As an alternative model for
transforming problems into objectives, the “Eisenhower model” (see the handbook
available on this website for an explanation) was discussed. As most of the other
“teaching inputs” this one also raised a controversial discussion between workshop
participants: while some questioned the inherent “logic” of such a model, others
emphasized the practical appeal of this (and similarly simplified) models.
Saturday
The “input” of Saturday morning covered the following topics:
The importance of linking planning and evaluation
The importance of stakeholder participation
The co-operation with experts
The treatment of differences of interest
An overview over the logframe, and particularly its “intervention logic”
Participants then turned towards constructing the logical framework for the simulated
project “Enhancing the Efficiency in Planning and Evaluation.” This formulation of an
Overall Goal was suggested by the trainers and initiated a fruitful discussion among
participants: who was the intended target group of this “project” – the participants of
the workshop, their organisations, or the network as a whole? As a result of this
discussion, participants came up with a number of alternative formulations (for the
final one see annex 5), before proceeding to discuss the project purpose which
intially focussed on the development of appropriate planning and evaluation skills (for
the final formulation see annex 5).
As an additional guide for the discussion about objectives, the trainer presented the
characteristics for a SMART goal:
Specific
Measurable
Achievable
Relevant
Time-frame
After clarification of the intervention logic (defining results and activities – with
discussions about these levels providing clarity about the usefulness and definition of
39
these levels) participants moved to define objectively verifiable indicators and
sources of verification. This task could not be finished despite the lengthening of the
working day to make up for the loss of time on the first day. Nontheless, participants
all felt they had come together as a group and achieved an important step towards
laying out a clear plan for the simulated project.
Against this backdrop, the excursion through Tbilisi and to historical sites in the
vicinity of Tbilisi on Sunday was both pleasurable and informative. The discovery of
the antiquity and creativity of Georgian culture somehow gave the workshop a
background of unexpected depth and significance – a non-measurable “quality”
appreciated enormously by all participants, as they expressed in their toasts (in
Georgian fashion) during the farewell dinner on Monday evening (see also the final
evaluation).
On Monday morning the work on indicators and sources of verification was
completed, before moving to clarify the assumptions about external conditions which
the simulated project would need in order to succeed. The participants showed their
professionalism in readjusting already stated objectives and adding new ones
(including the appropriate indicators and sources of verification) when the risk
analysis (the first major “input” for Monday, for details see the handbook available on
this website) showed that this was necessary. This successful reshuffling
demonstrated both the logic of the logframe itself and the understanding participants
had gained of it.
To end the day, the Mopping Up Exercise (as part of the MAPA methodology, see
also the handbook) provided participants with an opportunity to return to the long list
of problems collected at the beginning of the workshop and to check if all of them had
been considered in the project plan as represented in the completed logframe. The
only discussion arose about the “differences in payment between local and foreign
experts” – which was concluded by an agreement that this “problem” simply belongs
to the general conditions under which organizations have to plan their projects. As
one of the participants expressed in a toast during the farewell dinner that evening: “It
feels so good to find that all those many problems could be taken off the wall” –
because they had all been acknowledged in one form or another in the logframe.
Tuesday morning still had a long agenda to be completed before the evaluation of
the workshop:
A discussion about participation and the role of facilitation
The “Kill the Project” Exercise
The beginning of an operational plan using the Gantt schedule
It turned out that the importance of stakeholder participation to some felt more like an
additional burden for an already difficult task (i.e. of planning and imlementing
projects) while others had positive first-hand experiences with this approach. When it
came to defining the necessary qualities for facilitation, however, all participants
quickly joined in putting together the following list of characteristica:
ability to listen
patience
stressing the essential
not to force one’s own opinion (on the other participants)
communicability
open-mindedness
sense of humour
40
organizational skills
group management
friendly disposition
state ground rules
distinct pronounciation
Despite some (expected) initial resistance to now turn against their own brainchild
(the completed logframe), participants accepted to engage in the "Kill the Project”
Exercise. They accepted the advantage of using this exercise as a means to make a
project proposal less vulnerable to (later) outside criticism – and came up with the
following list of criticism (which did result in reformulations, added assumptions, and
even in added results (and indicators plus sources of verification) in the logframe as
given in annex 5):
Political instability
Project does not contribute to the goal of increasing the efficiency and
sustainability of projects (run by the organizations)
Quality of the handbook is not guaranteed (which resulted in an added indicator
on the utilization of the handbook)
Quality of the trainings is not guaranteed (which resulted in an additional indicator
as well)
Lack of sources (which resulted in the addition of a survey as a result)
An operational plan does not exist (it was clarified that this would be the next step
in the simulation exercise)
What sort of evaluation should we carry out? (Which resulted in a reformulation of
a result)
There are no guarantees for sustainability (which prompted the trainer to explain
the importance of achieving a “critical mass” of organizations, people, and
projects using the new method)
As the logframe was still empty in the Means and Costs cells of the matrix, the time
had come to involve participants in the beginnings of elaborating an operational plan
using the “Gantt” schedule (for details see the handbook). The principle quickly
became clear in trying to break down the activity of organizing 4 trainings for 60
people of 20 % of organizations into subactivities, giving timeframes to these
subactivities, defining the necessary human resources, the necessary material inputs
(equipment), finances, and clarifying the responsibilities. Thus, it was demonstrated
that the elaboration of an operational plan can also be done in a participatory manner
involving all present participants and visualizing the result on the wall – in the same
manner as the logframe itself.
The main part of the seminar was concluded by additional “inputs” on the
establishment of a monitoring and evaluation system for projects, and on the use of
the “rainbow model” for the analysis of the “institutional landscape.” Finally, a general
discussion on the use of planning methodologies and participation in international
contexts revealed that some of the participants had actually already had considerable
experience before coming to the workshop. And at least one participant could report
about about a foreign company’s misuse of the participatory method: when called to
participate in a workshop with participatory exercises, the participants had not been
aware that the results would be used by the company for the real planning of their
own activities.
41
Evaluation and Follow-Up
In the afternoon of Tuesday, the seminar was evaluated using criteria set by the
participants themselves and the results are shown in the table below. The first five
columns give the number of participants who put their adhesive dots in the respective
cells, while the last column gives the result of a calculation based on a five-step
Lickert scale weighed 1 to 5 (as indicated in the top line)
Indicator Very Satisfied Neutral Dissatisfied Very
satisfied Dissatisfied
5 4 3 2 1
Conditions for 9 3 4,75
conducting the
seminar
New information 4 8 4,33
Acquiring full 8 2 3,80
information about the
MAPA method
Acquiring new skills 7 7 3,50
Atmosphere of the 12 5,00
seminar
Methodological 6 7 3,46
completeness
Level of participants' 3 8 4,27
active participation
Efficiency of 3 10 4,23
discussions
Technical equipment 4 6 2 4,17
Living conditions 12 3 4,80
Professional level of 12 1 4,92
trainers
Handouts / Manuals 3 10 1 4,14
Outside atmosphere 13 5,00
Seminar organisation 5 8 4,38
Tutor assistance 4 9 4,31
Adequacy of issue 2 8 3 3,92
discussed
Assumptions on 1 9 2 3,92
sustainability of
seminar
Average rating 4,05
The workshop had an overall rating of 4.05, with no marks below 3 (scale from 1 to
5).
The highest marks revealed the participants’ satisfaction both with the
professionalism of the trainers – and for the atmosphere of Tbilisi and (implicitly) the
care of the hosts. The lower marks for “acquiring new skills” probably reflects the
participants already high level of profesionalism in general and of project planning in
particular, while the other points evaluated as not as highly positive all seem to refer
42
to the fact that the workshop had not fulfilled some participants initial expectations: to
be familiarised with particular methods for the evaluation.
This was confirmed in the concluding round of open discussion, when some
participants again stressed the need for concrete methods for needs assessment,
evaluation, and monitoring – which some felt to be not as easy as simply recording
actual events on a template using the operational plan: the need for a more complete
management information system for was emphasized.
Opinions were divided concerning the need for more “structured input” – versus the
more open discussion engendered by the simulation exercise – and for more
comprehensive handouts. Some would have liked more information on the
development and history of the MAPA method itself, while others would have
preferred more complete information before the beginning of the workshop –
especially about the simulation exercise itself.
What became clear again was the different level of previous experience of the
participants: while some found the method presented extremely useful for their
present work (remarking that projects they had to deal with should have been
planned according to this method in the first place), others felt more strongly about
the lack of completeness in this workshop: they felt that “planning” had taken too
much precedence over “evaluation” when that had been their main interest.
Overall, participants felt that experienced facilitators would be able to conduct a real
planning workshop on the basis of the information of the training seminar. This
information, one of the participants remarked, would need to be “used” – and not just
“read”: “There is a point when you need to put down the books and do something
with what you have learned. And this is what we should do now with what we have
learned here about MAPA).
The trainers also made clear that the feeling of incompleteness was indeed
appropriate, as a second training workshop would focus on evaluation, rather than on
planning (which also answered the question about the amount of training needed for
the professional application of MAPA). And they made clear that participants’
remarks would be taken seriously in the planning of future workshops – to further
improve MAPA, which one participant likened to “an electric drill as compared to a
hand drill.”
Outcomes
As mentioned before, the training workshop achieved its goal of providing
participants with the skills to conduct a real planning workshop using the MAPA
methodology – as was expressed by participants themselves.
The workshop also confirmed the validity of the collaboration between the trainers
and a tutor (which went on unnoticed in late evening discussions about the
sequencing of training steps for the following day). In that sense, the workshop
educated the trainers and the tutor as much as the participants.
At the same time, both these discussions and participants’ remarks (including the
logframe result of an elaborated training programme) pointed to the necessity to
further elaborate the training methodology. It was felt that not much additional work in
43
this direction would be necessary to enable the trainers to continue such trainings on
their own and without further tutor assistance.
One concrete idea which emerged from these discussions is the idea to use the
follow-up workshop exclusively for evaluation – and to take a real project for the
evaluation exercise, possibly including real stakeholders. While this would need more
extensive preparation, it would not only give participants a more “real-life” feeling, but
at the same time provide the chosen project’s managers with some real experience
in evaluation. Given the high level of professionalism of the participants of this
workshop, such a hybrid between “training in evaluation” and “real evaluation” might
lead to a result comparable to a professional outside evaluation.
44
Appendix 7: November 2001 Report – Kiev
Report on the training workshop
for participatory evaluation techniques (MAPA)
in Kiev, Ukraine (November 1 – 6, 2001)
by
Reinald Döbel
Institute of Sociology
University of Muenster
Scharnhorststr. 121
48151 Muenster
Germany
45
Table of Contents
Executive Summary
.............................................................................................................. 3
Major Conclusions
................................................................................................................. 5
Contents and Sequence of Workshop
.................................................................................... 6
Workshop Evaluation
.......................................................................................................... 14
Appendix 1: Results of Working Group on “Policy Analysis as part of policy making”
... 16
Appendix 2: Working Group on Information Campaign
.................................................... 17
Appendix 3: Interview Schedule
......................................................................................... 18
Appendix 4: Table of Indicators for the Completion of the Evaluation produced
by Workshop Participants
.............................................................................. 20
Appendix 5: List of Participants
........................................................................................ 22
46
Executive Summary
This was the first workshop to combine the evaluation of a real project with he
participatory MAPA methods with training in these methods. Therefore the workshop
had to both provide sufficient information on the project to be evaluated to the
workshop participants and to train them in the tools to evaluate this project. The
workshop had been requested by the International Renaissance Foundation (IRF) of
Ukraine and was co-financed by the Institute for Educational Policy (IEP) of the Open
Society Institute (OSI) in Budapest as part of their evaluation strategy, by the
Mongolian Soros Foundation and Educational Center Pro-Didactica (spin off of the
Soros Foundation in Republic of Moldova), who sent over experts to support inter-
foundation co-operation and evaluation efforts, by national foundations who
supported the travel and accommodation of their staff attending the workshop.
The project which the IRF requested to be evaluated during the workshop was the
Educational Policy Support Program laid out in the Policy support proposal of
December 25, 2000 (prepared by Thomas B. Timar, Natalya Shabla, and Alla
Seletska); Appendix 1. Despite the difficult task of both informing participants about
the project and training them in the assessment of this information, the workshop
evaluation was rather positive with an average score of 3.97 and score of 4.46 (of a
maximum of 5) for the intention to use MAPA methods in the future (for details see
the section of “Workshop Evaluation”).
Participants became increasingly interested in understanding the complex project to
be evaluated – and at the same time they also appreciated to an increasing extent its
contribution to the educational system. The main reason for their increasingly positive
attitude was the participation of a number of real stakeholders of the project in this
training seminar. These provided sometimes unexpected information about events
and effects of the project which led to increasing appreciation of the achievement of
the project (see the following section on Conclusions)
The main lesson drawn by the trainers and the IRF staff at the end of the seminar
was twofold:
1) The usefulness of the MAPA methodology is more readily understandable if
applied to the evaluation of a real case, rather than the planning of a virtual
project: the ability to deal with a „real life“ situation is more convincing than
the logical coherence of a set of procedures applied to a fictitious case.
2) In the future, such training events should involve more joint IEP/trainer/project
staff pre-workshop planning in order to better structure the sequencing of
training and evaluation sections and to prepare better instructional and
information materials for workshop participants
In addition to its positive training results, the workshop was successful in establishing
an evaluation strategy for the Policy Support Project which participants and
stakeholders particularly from Ukraine found credible and hoped to see implemented.
It was therefore agreed between the trainers, the project staff and the IEP
representative to hold a final MAPA evaluation workshop after collection of the data
required by the table of indicators produced by the workshop (see appendix 6). A
second precondition for this workshop would be the completion of a preliminary
evaluation report by the main trainer, based on this information.
47
Major Conclusions
The main evaluative conclusions of the workshop concerning the Policy Support
Project were:
The project successfully managed to coordinate various donors (World Bank,
UNDP, IRF, British Council) in support of the development of a sound education
reform strategy;
The project conducted a series of successful workshops according to its defined
goals which contributed to capacity building within the Ukrainian education
community;
The project also contributed to capacity building among local researchers who
were mobilised to provide a sound analytical base for policy decision in the area
of educational reform;
These analytical studies - as well as other mechanisms of eliciting comments on
the development of the national “doctrine” on education reform - systematically
influenced the formulation of this important document;
The project provided a focus for increased public and expert discussion about the
reform strategy. In the words of one stakeholder: the project “energized” the
whole education sector. This was later confirmed in an independent interview
with Vladimir Spivakowsky, one of the participants of the Advisory Group brought
together by the Policy Support Project.
48
Contents and Sequence of Workshop
Rather early in the workshop it became clear that not all participants had a clear
understanding of the dual purpose of the workshop and therefore (and because of their
generally busy schedule) had not studied the project materials sent to them via e-mail.
Therefore the sequencing of information events and methodological inputs had to
adapt to the participant’s growing understanding of both the project and the purpose
as well as methodology of evaluation, in particular of a MAPA evaluation.
First Day:
The morning started with a welcome to participants and a brief overview over the
training course. The objective of following sequence was to start with participant’s
understanding of evaluation as brought to the workshop.
They were asked to divide into three groups and discuss positive experiences with
evaluation. A sample of these stories was brought back to the plenary sessions to
clarify the meaning of a “good evaluation” and to develop a list of criteria for such
“good evaluation” as a basis for the discussions of the workshop: as criteria to be
applied to the evaluation of the policy support project during the workshop. At this
point a controversial discussion developed over the notion of what “good evaluation”
means: and evaluation following proper procedures, or an evaluation delivering a
positive statement on the project evaluated? Does “good evaluation” refer to the
quality of the evaluation or to the quality of the thing evaluated? For the time being,
this discussion deflected from the purpose of collaboratively developing a list of
criteria for an evaluation following a “good procedure” – which was then produced the
following morning (see below).
This was followed by an overview of the MAPA sequence of planning and evaluation
workshops, including a brief presentation of the logframe matrix.
The afternoon was dedicated to a presentation of the main lines of the Educational
Policy Support Program by project manager Lilia Ostapenko. This presentation was
deemed necessary as most participants had not found the time to read the materials
distributed via e-mail in advance of the workshop. On the basis of the presentation,
we then worked out the basic structure of the project, according to three main goals:
1) Introduce informed policy making through policy analysis (analytical studies)
2) Support Ministry with development of concept paper
3) Information Campaign to inform wider public
Some participants were not happy with the difficulties encountered when trying to
understand this particular project and expressed their wish to be simply trained in the
“tools” to do an evaluation – while some others appreciated precisely this “hands-on”
approach as an expression of “how life is.” It became clear that not everybody had
taken the announcement seriously that this training workshop would at the same time
be a “real evaluation” – because the “tools” cannot be understood independently of
the situation. As a result of this discussion, and realising that this task would at the
present time overtax the participant’s capacities, in the evening trainers and IRF staff
grouped all project activities under the three main goals – instead of leaving this task
to participants, as planned beforehand.
Second Day:
49
Taking some participant’s dissatisfaction of the previous day into consideration, the
second day started with clarifying the concept of evaluation. Going back to “criteria of
a good evaluation” participants developed the following list on the basis of the
question:
WHAT IS EXPECTED OF AN EVALUATION?
- To what extent the results correspond to the goals
- To what extent the project is significant, actual and predictable
- How effective is the project
- To get feedback
- To what extent the activities are adequate to the goal
- How effective were the money spent
- To what degree the project satisfied every stakeholder
- What is the impact of the project
- How the project influenced the image of the organization
- To what extent the result correspond to the mission of the foundation
- How well are the elements of the project balanced against / amongst each
other
- Is the timeframe real
- How good the human resources were identified
- How good was the situation analysis
- To what extent were the risks taken into account
- Is it possible to evaluate the impact
- What is the degree of innovation of the project
The trainers then suggested to condense this list into basic questions which ANY
evaluation has to take care of:
MAIN QUESTIONS FOR EVALUATION
Did the project achieve its goals / objectives?
Was the goal the right one?
Were the means to achieve the goal properly applied?
Were the means the right ones?
What other effects did the project have?
Participants agreed that these questions did indeed take care of all the points
mentioned in the above list. One additional basic issue was proposed as important of
all evaluations: consideration of “strengths and weaknesses” of a project /
programme.
With these goals in mind, participants in the afternoon went into working groups
corresponding to the three main goals, attempting to evaluate the degree to which
the project activities had taken place as planned, had achieved the results aimed for,
and had actually contributed to “their” respective main goal. As each working group
was composed of stakeholders of the project, project staff, and trainees from other
countries, this activity provided participants with a good impression of the discussions
at any MAPA evaluation session. The fundamental evaluation questions were
reformulated for the working groups in the following manner:
Did the activities take place as planned? If not, why not?
Did the activities achieve their particular purpose? If not, why not?
Are there questions which could not be clarified?
Who could answer these questions?
50
(If time permits:) What was the contribution to the higher goal / objective?
This activity went on into the morning of the third day, when the results were
presented in a plenary session. One of the interesting aspects of the discussions in
the groups was the “revelation” of one of the members of the study groups that these
groups had indeed substantially (and systematically) influenced the content of the
“Concept Paper” produced by the government – as shown by the inclusion of the
new chapter on equal access not present in the first draft ( unofficial draft sent to
international consultants for review).
Third Day:
The day started with the presentation of the group results of the previous day, with
particular emphasis on the additional questions identified by the participants. Two
examples of the tables produced by the working groups are given in Appendix 1 and
Appendix 2.
The discussions had been left more “open” on purpose in order to introduce to
participants the idea of checking results against plans as one of the core concepts of
any evaluative activity. With the lists of activities checked in this manner, participants
were ready for the more abstract concept of “indicators” as one of the tools of
achieving this check. It was stressed that there is a need for indicators at all levels: at
the result (“output”) level, at the purpose (“outcome”) level, and finally also on the
level of the impacts on the project on the wider environment. The higher the level, the
more abstract and more difficult the measurement of results (outcomes, impacts)
becomes – and is often impossible to measure directly. Hence the need for “pointers”
– i.e. indicators.
Explaining indicators with reference to the example of the “indicator” on cars (which
simply “indicates” into which direction the car will go next) an attempt was made to
demystify this concept of indicator – and to point to the necessity to think about the
appropriateness of each and every indicator with reference to the stated result
(output, objective, goal, outcome, impact). One of the examples discussed at length
later with respect to the goal of initiating a public discussion about education reform
in Ukraine was the echo in the mass media and the public reaction to media reports.
It was finally accepted that a table with the number of reports appearing in different
media, in different months, and in different regions, compared to the number of
reactions (letters to the editor, phone calls to radio or TV stations) would give an
adequate “shortcut” for assessing the “initiation of public discussion.”
One of the walls of the meeting room had been prepared to allow for the construction
of a matrix showing for each goal and sub-goal (objective, result, outcome, impact)
the indicator, the location of the data, the method of data collection, the method of
presentation, the time frame, and the responsibility (see also MAPA handbook p.
127, Fig 12).
This scheme was worked through in the whole group for two objectives, before
participants again split into working groups to complete the matrix for all relevant
goals and sub-goals (objectives, results, outcomes, impacts). The attempt to clearly
think through indicators while keeping in mind the practicality (rather than the
scientific “validity”) of indicators proved to be a difficult exercise –which was not
completed on this third day of the workshop (Saturday).
Fourth Day:
51
The discussions between the trainers had resulted in the realisation that the goals
and structure of the Policy Support Project had not become sufficiently clear to all
participants. As qualitative interviewing is one of the most important tools of
evaluation, it was decided to stage a “focus group interview” (with two trainers, two
project staff and the representative from IEP) in front of the group. This was intended
to demonstrate both a more appreciative attitude (rather than a “critical” and
“inquisitive” one) in a qualitative interview – and to clarify the project goals and
structure. The interview schedule is shown in Appendix 3, and with the exception of a
minor detail the structure of the project as indicated in the schedule proved correct.
Further, the main goals of the project were narrowed down to two:
1) To assist the Ministry of Science and Education in the formulation of an education
policy which takes into account the existing problems (as identified by experts
before the start of the project and studied in more detail in the “Study Groups”
initiated by the project) and
2) To help the Ministry to make the decision process more “open” (i.e. democratic)
by taking into account the views of other stakeholders (referred to in the original
project proposal as the “initiation of a public debate” and implemented as an
information campaign and the establishment of an “advisory group”)
These two goals (rather than the three previously mentioned ones) were confirmed
as correct both by the project staff – and by the representative of the World Bank
whose presence as one of the more important “stakeholders” was very welcome for
this workshop.
The working groups then reconvened to complete their unfinished task of
enumerating the indicators for the evaluation of the project. Indicators which project
staff could actually collect in the time frame indicated by the arrival of another expert
mission in December and with the methods specified by the participants of the
workshop. While the groups – as expected by the trainers – differed on details, they
were largely in agreement about the most important indicators. One of the interesting
aspects of these discussions was the strong preference for quantitative, rather than
qualitative indicators – which were deemed less “objective.” And yet, one of the
stakeholders confirmed their importance by simply stating that he – being a member
of a Ministry group working on the concept paper – had changed his whole attitude
and outlook as a result of attending one of the workshops organized by the project.
Which in turn had influenced his further work in the Ministry.
Fifth Day:
The Working groups reached a consensus agreement on the indicators to use for the
completion of the evaluation of the Policy Support Project of IRF by presenting the
results of their discussions during the previous day, and also agreeing together with
project staff on the feasibility of collecting the required information in the given time
frame. Thus, the main result of the workshop was a plan for the completion of the
evaluation, arrived at in a participatory manner with the presence of stakeholders
(members of the study group, the Education Board of IRF, of the ministry, other
professional educators, and of course “outside evaluators” represented by the
representatives of Soros Foundations from other countries (see list of participants,
Appendix 5)
In addition, this last day was enlivened by a discussion about the unplanned aspects
of the project. The following list of topics gives an indication of the overall positive
effects the project had in the eyes of all present stakeholders. Indeed, it proved
52
difficult to find any negative aspects. The only agreed one was the raising of
expectations which may later be disappointed – and the possibility of widening
existing inequalities (see: “social contradictions were reinforced”).
OTHER EFFECTS OF THE PROJECT
- The pedagogical community became more active
- A new plan of training of the students of medical higher education institutions
developed
- Cultivation of hope
- Social contradictions were reinforced
- Bigger influence on donors
- First workshop on evaluation of the project conducted
- Unexpected supplementary expertise of the concept paper
- Participation of new “unplanned” donors
There was strong agreement on the “activation” of the education community in
Ukraine – something which a later individual interview with Vladimir Spivakowsky
(one of the more prominent members of the “Advisory Group”) confirmed. An effect,
moreover, which had been produced by the synergic effects of organizing expert
workshops and involving the mass media in their presentation.
The strongest “unplanned outcome” was reported by a member of a medical
institution: participation in the project had led to a reformulation of the training in
higher medical education institutions, with particular emphasis on the standardization
of quality control.
Further, this last day saw a discussion on where the project might go in its second
phase. This discussion was initiated by the trainers’ attempt to clarify their
understanding of the meaning of “systemic impact” in the overall scheme of the work
of the Soros Foundations. The main point of this discussion was to point out the need
to evaluate the “systemic impact” of projects as a lasting contribution towards the
value of “open society” in all the countries where Soros Foundations are active. In
addition to opening the discussion for a consideration of the place of the MAPA
methodology in this overall scheme, the following topics were discussed, but not
resolved:
THE FUTURE OF THE PROJECT
- the problems identified by the analytical studies should be taken into account
Priorities:
- to elaborate the concept of monitoring of quality of education
- equal access to education
- Development of the plan of the resolution of the existing problems
- To establish a policy group
Finally the workshop went into the (so far) shortest workshop evaluation (as the other
tasks had left little time.
53
Workshop Evaluation
As usually done in MAPA workshops, participants produced the indicators for the
workshop themselves and then rated them on a Lickert scale of 1 – 5 (5 "very good",
4 "good", 3 "OK", 2 "poor", 1 "very poor"), to produce the evaluation table below.
Indicator 5 4 3 2 1 avr.
Stakeholder participation 10 1 2 3.62
Correspondence between workshop methods
and goals
– training 3 4 6 3.77
- evaluation of the project 1 10 1 4
Level of organization 4 6 2 1 4
Correspondence of results to expectations 2 5 6 1 3.57
The new character of the information 2 7 1 4.10
Interaction of participants 11 1 1 3.77
Understanding of MAPA methodology 1 7 5 3.69
The actuality of the information 9 3 4.75
Intention to use MAPA in the future 9 2 1 1 4.46
In addition they gave valuable suggestions on how future “hands-on” trainings in
evaluation could be improved – before they all insisted on the value of this training for
their future work (as indicated in the table quoted: “Intention to use MAPA has an
overall score of 4,46, with nine participants giving it the highest mark). Some
participants had to leave early because of other commitments, but made sure to
inform the trainers about their appreciation of the methodology offered by the
workshop. Project and IRF staff also expressed their satisfaction about the
usefulness of the feedback and input from participants. The average score for the
workshop is 3.97 (average with respect to other MAPA workshops, but with the
highest minimal score, with all scores above 3.5, and with a very encouraging high of
4.46 for the intention to use participatory evaluation techniques in the future. The
highest score is 'for the actuality of information, 4.75, meaning a highlighting of the fit
between the content and type of training, and the actual needs of foundations. The
relatively low score of 'correspondence of results to expectations', 3.57 is a
consequence of the general expectation for clearly structured learning according to a
fixed syllabus, rather than the “hands-on” learning through actually doing an
evaluation as practised in this workshop.
54
Appendix 8: Review of the MAPA handbook
Appraisal of the MAPA Project : Practical Guide to Integrated Project Planning and
Evaluation
Marlène M. Läubli Loud Benoit Girardin
Evalutaion Management and Resource Centre Swiss Agency for
Swiss Federal Office of Public Health Development and
Co-operation
In the first instance, we would very much like to commend your production. The idea
of ensuring that monitoring, evaluation planning are integral components of project
planning is highly commendable. For those of you in the Development and Co-
operation field, this is probably mundane, but within the health field, this is certainly
not always the case. It is really most helpful and efficient we believe, to plan these
functions together.
Congratulations also on the Appendix section. It offers some most useful guidelines
and practical tools for organising the activities of both the training and evaluation
workshops. I personally particularly liked the inclusion of the Gender analysis
instrument since this is an aspect all too easily forgotten in the designing and
implementation of projects as well as evaluations!
In your Annex 17, Monitoring and Evaluation System, we felt that you didn’t
sufficiently emphasise the need to keep one’s eyes open for the “unintended
effects”. An additional paragraph with some examples would be useful to bring
this point home. With regard to the list of essential points any evaluation must
check : the list appears to be more in line with the concerns of monitoring or
controlling – I personally feel that evaluation has to be more reflective – it should
look towards answering such questions as what measures seem to work best, for
whom, under what conditions and at what cost? To help external evaluators
therefore, one could ask project managers to add to the monitoring activities,
some reflection on Why (e.g. things took longer / shorter than planned) What
solutions were found and put into place, and what were the consequences.
Similarly, we couldn’t see anything in the planning tools provided for monitoring
activities that suggest the need to pick up information of “results” in terms of
“outputs” (goods and services produced) and how many? Nor indeed, was there
any discussion of “Self evaluation”. Is this on purpose ?
Now, to the actual text of the Manual, both M. Girardin and myself found it quite
difficult to read from cover to cover because it was repetitive (naturally enough to
push the message!) and a little dull to read (maybe some more pictures, jokes,
graphics etc??)
In the introduction, we were disappointed that in the section on the “Limitations of the
participatory method” there was no discussion about group dynamics and the need to
be vigilant to the possibility of ”domination” of one group/an individual’s ideas or
“manipulation” by dominant members of the group. There is also the problem of
representation / if all critical parties are represented, you can end up with a large
group which is difficult to handle. “Participatory” can easily become “manipulative”.
55
I was quite amused on page 13 to read that “In addition to logical coherence,
transparency….. the PAP-PROJECT provides a clear and standardised format
and a clear and coherent language, a language that is internationally understood
by project planners, evaluators….In the world of evaluators in any case, there is
little agreement on what evaluation is, let alone an absolutely agreed,
standardised glossary of terminology! Further on this point, M. Girardin
stressed however that since the OECD-DAC Working Party is about to issue a
Glossary on evaluation, a thorough check should be carried out to ensure a
match between theirs and
We also felt that even at the planning stage, the document should clarify the need
to make a clear delineation between the different level of project achievements:
outputs (goods / services, tangible things) outcomes (changes happening within
the target group) , and the “impacts”, that is what happens at a wider level than
that at which the projects is aimed : outputs, objectives, goals.
Your section on Evaluation in the Overview section of the MAPA Project (p. 24 in our
version) starts off with the statement that Any evaluation is meant to determine to
what extent a project has reached its explicitly formulated goals”. As mentioned
above, the major aim of evaluation should be to show the effects and consequences
of the project’s activities, and how these were achieved in relation to the project’s
contextual setting. Discussions on achieving pre-fixed objects or not is to more
relevant to the work of controlling or of evaluation limited to effectiveness or even
auditing. However, the rest of the text in this section tends to go towards this way of
thinking, so it is really more the original statement that is rather misleading.
Evaluation should also address the issues of sustainability and relevance. Values
underlying the evaluation process should deserve a larger and more explicit
handling. Think of values such as responsibility, accountability, transparency,
openness for learning out of mistakes and errors, efficiency…
Programme perspective is too modest. The emphasis is too much on projects,
whereas more and more NGOs are operating programmes. Programme evaluation
requires some additional specifics compared with project evaluation
Lograme is nicely presented, but one could expect some words on logframe
limitations, for instance the trend to overemphasise results that are easily
measured.
Appendix 10 on risk analysis could also address the issues of scaling down the
probability or reducing the influence of risk factors
Finally, we feel the whole of the text part could benefit from some more “livening up”
with some good concrete examples, some illustrations, graphics and a few jokes!! Its
“entertainment value” is a bit low.
56
57
58
Related docs
Other docs by 2c2A5Z
