CDS support for M&E capacity building within AAV: A summary
Rick Davies1 and Ger Clarke, Hanoi, 22nd September, 2001
Inputs were provided to AAV in September, 2001 through the following meetings: With Emma Woodford and Ngoc Anh to discuss the CSP III Monitoring Framework, on Monday 17th With Ha Tinh programme staff on Tuesday and Wednesday With senior staff of the AAV Hanoi office on Thursday and Friday With Lai Chau programme staff on Friday
1. The Ha Tinh programme Operationalising ALPS. The Ha Tinh planning, monitoring and evaluation procedures have recently been reviewed and revised in association with a planned attempt to operationalise ALPs within the Vietnam programme. Two workshops have been held to explain ALPS concepts to AAV staff. One of the outputs has been a tabulation of six attributes of all the current planning, monitoring and evaluation procedures (what, how, when, who, product, constraints and challenges). This is a useful document. It includes information on what is new, as well as old practices which will be continued. Information on what procedures have been removed or significantly changed would also be useful. It appears that the sector specific Technical Reports will no longer be under taken by external experts, and may be replaced by partners’ own self-assessments. Informal assessment of the new procedures. Ha Tinh staff expressed the view that the revised system was more time consuming than the pre-existing system. This is despite the intention stated within the ALPS Notes that it should be less time consuming. One reasons given for this result is that now everyone has to be involved, all AAV staff and all the stakeholders. Assessing planning, monitoring and evaluation procedures. The Notes to Accompany ALPS point out that there are at least six ways in which ALPS is expected to differ from the previous APRS (ActionAid Planning and Reporting System). These are in effect six performance criteria for the new system. On Wednesday Ha Tinh staff were asked to matrix rank all 21 procedures in terms of these criteria, and then to explain why the position of the highest and lowest ranking procedure on each criteria. This matrix ranking provides AAV with a base line measure of the qualities of the system at present, against which changes in the system could be noted by repeating the matrix ranking in a year or so’s time, and periodically thereafter. A copy of the matrix ranking data has been given to AAV. Advocacy starts at home? Three separate staff groups undertook the matrix ranking exercise. In all three groups the quarterly, six monthly and annual reporting to PACOM were ranked lowest on all six performance criteria. Clearly this is one area where there should be a change in practice from the past, if ALPS
1
Social Development Consultant, Cambridge, UK. Email rick@shimbir.demon.co.uk Web pages at http://www.swan.ac.uk/cds/rd/rd1.htm and http://www.mande.co.uk/news.htm
1
is to be operationalised in AAV. But so far no changes have been made or proposed to any of these three procedures. The general attitude amongst Ha Tinh staff seems to be that this reporting requirement was a given that could not be challenged in any way. This seems ironical in the extreme, given that advocacy is now expected to be one of AAV’s tools of change. If AAV cannot undertake effective advocacy work this immediate arena, then where else can AAV expect to be effective? AAV staff should examine each PACOM reporting procedure in the light of the six criteria and see if they can identify any ways of improving the procedures which might be acceptable to PACOM. More awareness raising. When staff were asked to explain the reasons for placing the highest and lowest ranking procedures on each criteria the explanations given were simple, and verging on the simplistic. Participation was largely defined in terms of numbers of individuals and types of people. The concepts of downward accountability, transparency and participation were barely distinguished, the first two appearing largely as other interpretations of participation. When asked to identify which criteria were easiest and most difficult to rank, participation was identified as the easiest and the fifth criteria (“More inclusive…”) being the most difficult. If the Ha Tinh staff are to operationalise ALPS they will need more assistance with differentiating what these six criteria are all about. The existing manual is of some use but it has its own limitations. The main text of the “Notes to accompany ALPS” refers to five main areas of practice that should be embodied in ALPS, and only three of these are the same as the six forms of difference between ALPS and APRS noted on page 7. More guidance from regional or headquarters offices would be useful. Before then the reasons given by each group for its highest and lowest rankings should be documented in more detail, beyond the notes taken by EW and RD. Differentiate responsibilities. The procedures with the highest overall ratings, and the highest ratings on participation, were the District Workshop (initiated by DFID) and the Annual Programme Review (initiated by AAV two years ago). Both of these are special events and involve large-scale participation, relative to other procedures. At times it seems that AAV is taking a stew approach to participation. The more people, and types of people that are involved in a review process the better, seems to be the belief. The alternate approach is to identify who is responsible to know what in each setting, and then how to ensure that they do so. At present the “who” column in the tabulation of planning, monitoring and evaluation procedures simply refers to “Partners, Communities” in nine of the 21 procedures. This recommendation is especially relevant now that AAV is moving towards commune-based planning, but will still also be working on a sectoral basis as well. Clearer identification of M&E responsibilities may help ensure that plans are implemented as expected. Bring in, or build on? In deciding how to operationalise ALPs AAV has an important choice to make, between bringing in partners into AAV’s procedures, or ensuring that AAV’s procedures build on partners own procedures. For the latter to happen AAV will need to invest in building partners M&E capacities. So far AAV seem to prefer the former approach. In the Tuesday meeting Ha Tinh staff showed surprisingly little knowledge of partners planning, reporting and evaluation procedures, especially those of government departments. The idea of
2
using government systems was view with almost complete skepticism. When asked about how the separate AAV systems would be sustained after AAV pulled out it was argued that they hoped that the government would take on board and retain some of the more participatory features of AAV’s procedures (i.e. imitate AAV practices). But this strategy seems to be in outright contradiction with the skepticism expressed about the possibility of influencing government systems through much more direct attempts at M&R capacity building. Recommendation: Senior AAV staff should investigate the extent to which AAV is using parallel systems of planning, monitoring and evaluation in Ha Tinh, and identify where there are possibilities of making more use of partners pre-existing systems, with M&E capacity building support being provided wherever necessary. This should be done before further steps are taken to operationalise ALPs. The Group Based Assessment of Change. This impact assessment procedure was implemented in 1998 and in 1999. However the data collected in 1999 has not yet been analysed. The main reason seems to be that no one in AAV thinks it is very important. This is surprising since it is AAV’s only systematic approach to measuring changes in peoples lives which provides a comparable data set over different years, which can be dis-aggregated down to relatively small units, and which can be easily related to the scale and type of AAV inputs received. The Participatory Poverty Assessment, though of value, does not produce the same type and quality of data, at the same low cost. When asked what they wanted to do with the GBAC staff expressed the desire to cease using it. This is despite the fact that they had ranked above 13 other procedures in terms of the ALPS criteria. Assessing achievements in relation to objectives. According the NA, the main means now being used to assess the achievements of the Ha Tinh programme is the Annual Programme Review, the most recent of which was held in August and September this year. The results have not yet been written up, so it is difficult to see to what extent information has emerged which is useful in assessing the achievements as formally defined in the either the Long Term perspective or in the HTPAP Logical Framework for Cam Loc.
2. The CSP III Monitoring Framework AAV has recently developed a monitoring framework for their CSP III. This is a positive development. Most country level strategy documents do not get monitored or evaluated on a systematic basis. Concept paper: After discussions with Emma Woodford and Ngoc Anh on Monday 17th RD developed a concept paper on how to take the existing Monitoring Framework forward2. Testing. The approach was tested on Thursday. The focus was on a number of ranking exercises designed to give the strategy paper more structure, and to help identify how PASPs related to the CSP and to identify where data was available
The CSP III Monitoring Framework: Comments and Suggestions
2
3
on the prescribed indicators. A disk copy of the information collected has already been provided to AAV. Further work of the same kind was undertaken by AW on the Friday afternoon (in respect to the organisational objectives). Other applications: The concept paper should be shared with the AAV regional office. It seems likely that the regional office could use the same approach to monitor the achievement of the regional strategy. Similarly, the same approach might also be useful in monitoring the PASPs, where these are being implemented through partners. This is most likely to be the case where partners do have their own objectives and plans, and are not just co-opted extensions of AAV. Related methods. Two types of non-indicator based approaches to monitoring were explained, using an outcomes table put them in context: Most Significant Changes monitoring, for unexpected outcomes Most Significant difference monitoring, for outcomes whose meaning is not agreed upon.
3. Other issues discussed with senior management staff Organisational learning. A short presentation was given on the subject of organisational learning. In the process it addressed the following queries raised by participants: How to document lessons learned and integrate this process into a programme? Should the focus be specifically on Learning Forums, or more broadly on organisational learning? How to ensure that individual learning transfers into organisational learning Is AAV an learning organisation or not? How to balance the use of learning versus reporting systems? Recommendation: The proposed use of Learning Forums within AAV should be given a more specific focus than that proposed at present. They should be the setting where the participants identify which lessons learned should be documented and exported, to other sections of AAV or to other organisations beyond AAV. Report writing. Staff asked for advice on report writing. A short presentation was given which focused on the structuring of information in reports, the importance of taking a reader’s perspective, and bottom-line indicators of the usefulness of reports. M&E in the Lai Chau programme. The Programme Coordinator asked for advice on how to connect the very specific indicators identified within the animal husbandry project, with the wider objectives in the PASP and CSP. This can be done by treating these indicators as part of the partners own plans. The development of the partners planning capacity should in turn be included as part of AAV’s PASP in Lai Chau. He also asked for advice on developing their new PASP, although advice and plans have been provided within AA which should be sufficient. The additional advice that was given focused on methods which help
4
ensure some continuity between the previous programmer strategy, while allowing some further specialisation and innovation. These turned out to be of limited use since there was little in the way of a previously documented strategy which spelled out a hierarchy of objectives. Documents available mainly focused on means and methods of work.
5