Monitoring and evaluation of capacity development by 0D35aA1

VIEWS: 7 PAGES: 20

									   Monitoring and
evaluation of capacity
    development
Guidance for AusAID PNG program staff
               Linda Kelly
               12/20/2008
                                  Monitoring and Evaluation of Capacity Development
                                                                     December 2008




Acronyms
BEDP       Basic Education Development Program

CDS        Community Development Scheme

CPP        Churches Partnership Program

ECDPM      European Centre for Development Policy Management

L&JS       Law and Justice Sector

M&E        Monitoring and Evaluation

MSC        Most Significant Change

PE         Participatory Evaluation

PNG        Papua New Guinea

SNS        Sub-National Strategy

TA         Technical Assistance

TSSP       Transport Sector Support Program

QAE        Quality at Entry

QAI        Quality at Implementation




                              2
                                                                               Monitoring and Evaluation of Capacity Development
                                                                                                                  December 2008




Contents
  Acronyms .......................................................................................................................................... 2
  Introduction ...................................................................................................................................... 4
  A note for senior managers .............................................................................................................. 5
  Guidance for program managers ..................................................................................................... 7
  Detailed guidance for capacity development monitoring.............................................................. 10
     Design and Intent ....................................................................................................................... 10
     Implementation .......................................................................................................................... 13
         The purpose of capacity development ................................................................................... 13
         The capacity development model .......................................................................................... 13
         Monitoring over time ............................................................................................................. 18
     Reporting and program development ........................................................................................ 19
     Evaluation and completion......................................................................................................... 20




                                                                           3
                                                           Monitoring and Evaluation of Capacity Development
                                                                                              December 2008




Introduction
    Capacity development, as both a ‘means’ and an ‘end’ features strongly in the AusAID PNG
program. The stock take of capacity development work in the AusAID program in PNG1
discusses a range of the strengths and limitations of the current approaches. In particular the
review notes that the monitoring of current capacity development is generally poor and unable
to provide the evidence base required to ensure good understanding of the value and outcomes
of current approaches.

   The purpose of this paper is to use the lessons and examples from the PNG program to
develop some guidance about good quality monitoring, evaluation and reporting for capacity
development. It is directed at program managers, who have responsibility to both oversee
implementation of programs and provide accountability to AusAID.

    The guidance does not provide a complete overview of the AusAID approach to capacity
development either in PNG or the wider aid program. Capacity and capacity development are
defined from an AusAID perspective in current policies and other guidance2. This guidance does
not attempt to provide the rationale for why capacity development is important in the PNG
program or what types of programs should utilize capacity development. While it addresses the
process of monitoring from the start of design it is not intended as guidance about design.

    The guidance focuses in particular on the questions for the program manager as she or he
assess the quality of the monitoring system and as they manage that system to meet the twin
aims of accountability and effective program implementation. It includes a brief note for senior
managers as well as more detailed guidelines for implementers.

    Monitoring requires attention to four areas. These include:

    1. A clear understanding of the intent of the program and therefore what will be monitored
       and why.
    2. A detailed explanation of the way data will be collected and the implications of those
       methods for the particular context and program.
    3. Clear explanation of how the ‘sense-making ‘ or analysis of the program will be
       undertaken.
    4. Explanation of the reporting processes and how these will meet both learning and
       accountability requirements of a particular program.

    This guidance is structured to address these four areas of good monitoring. It is presented
through the program delivery cycle of design, implementation, reporting and evaluation.




    1
       AusAID 2008, “PNG Country strategy: Information on Capacity Building Frameworks by Sector 2007-08”,
internal document.
     2
       http://www.ausaid.gov.au/hottopics/topic.cfm?ID=5397_541_5662_3417_8129


                                                       4
                                                      Monitoring and Evaluation of Capacity Development
                                                                                         December 2008




A note for senior managers
    Capacity development forms a major part of many aid interventions undertaken in PNG.
AusAID needs to know more about what is working and what not and why. While every
program will need to develop specific monitoring which meets the particular needs of that
program, experience and recent research suggests that there are some general considerations
which should be given to all programs. This guidance is developed to identify the specific
considerations which would improve monitoring plans so that AusAID is more informed about
the progress of capacity development in a program.

    Effective monitoring and evaluation of capacity development will start with the design
process. The design must be clear about the capacity development approach it will use and how
this particular approach fits with the broader rationale and objectives of the program. It needs
to be able to identify the risks of particular types of capacity development approaches within
PNG and how these will be monitored throughout the life of the program. In line with this, there
are particular questions which should be addressed at the Quality at Entry stage of the design to
ensure the new program will be established in a way which allows for effective monitoring.

    At implementation the monitoring framework needs to fulfill AusAID accountability
requirements but given the dynamic and developing nature of effective capacity development,
the framework also has to provide for ongoing learning and development within the program.
For this reasons the framework can be expected to go beyond simple checking against predicted
outcomes. It will need to give attention to outcomes and to process. It will need to both assess
what has occurred as well as explore why this has, or has not, happened.

    In order to meet this dual purpose, in the complex setting of PNG, the monitoring will need
to be able to meet a number of requirements. These include:
         The engagement of many different stakeholders in data collection and data analysis
         The consideration of influences and issues beyond the program control
         The interaction of culture, values and power in a situation. In particular how this is
            experienced differently for men and women.
         How the program is able to develop and change to address new learning

    This range of inclusions within the monitoring framework will produce detailed information
and analysis. While the monitoring framework is unlikely to be able to unravel all the possible
combinations of factors which influence capacity development in any one situation, but it must
be established in a way that enables critical reflection and consideration of internal and external
factors so that the progress of capacity development is able to be understood within context. It
is unlikely that this will be captured within the reports made available to AusAID but it is
essential that this process is at the basis of those reports. This ensures good quality judgment
can be made about progress.

    Local engaged program staff bring particular insight and value to this type of monitoring.
Their understanding of the local context and culture mean that they are able to assess the how
well the monitoring approach is looking to broader factors such as culture, power, gender and
values. They are also able to contribute insight and content to the analysis process; raising
questions, interrogating data and generally checking on the validity of findings and
assumptions.




                                                  5
                                                      Monitoring and Evaluation of Capacity Development
                                                                                         December 2008

    Freeing them to be able to play some active role within the high level analysis in this way
will ensure that they are both more reliably informed about the progress of the program and
also that they have some critical view of the quality of the monitoring and how well it is meeting
the information needs of both AusAID and the program itself. The following more detailed
guidance is developed with this view of the active and engaged program manager.




                                                  6
                                                     Monitoring and Evaluation of Capacity Development
                                                                                        December 2008




Guidance for program managers

Design
    While monitoring is not undertaken until after a design is implemented, the quality of the
design is very important for the quality of the monitoring that can subsequently be undertaken.
Therefore the guidance about monitoring and evaluation (M&E) for capacity development in
programs in PNG has to start with some attention to issues at the design stage.

    There are several key questions to consider for the Quality at Entry (QAE) stage. These
include:

        Does the design explain the intent of the capacity development for this program?
        Does the design explain what capacity development approach is being used in the
         program and is this explanation based on a clear rationale and lessons learned from
         other PNG programs?
        Does the design identify the risks of using that capacity development approach in the
         particular PNG context?
        If the capacity development approach includes technical assistance (TA), is attention
         given in the design to the particular factors which determine if TA is likely to be
         effective? Is the monitoring framework able to keep giving attention to these same
         issues throughout the life of the program?
        Is the design flexible enough to allow for changes in the approaches to capacity
         development and also changes in the way it is monitored and assessed?

Implementation
    The monitoring of capacity development requires that the program manager take a more
active and engaged role than is typical for some other aid work. The program manager needs to
participate in the monitoring. That does not mean collecting data or writing reports. It means
being part of some of the process of ‘making sense’ of the information (the analysis process).
This will require the program manager to give some time to the program, time enough to
understand what it is doing and time to participate in key meetings and discussions.

   There are key questions which should be addressed at Quality at Implementation (QAI).
These include:

        Does the monitoring provide information for learning as well as accountability?
        Does the monitoring focus on process as well as outcomes?
        Are the ‘voices’ of different stakeholders, especially those of women and the less
         powerful people, included in the data collection and analysis processes?
        Does the monitoring address factors related to values, culture and power?
        Does the monitoring contribute additional and independent information beyond the
         boundaries of the program and the changes expected by the program?
        Does the monitoring build ownership and control by the stakeholders?
        Does the monitoring capture unexpected and unplanned changes?

   While it is important to review the monitoring it is also important that program staff protect
   an agreed monitoring approach from constant changes. Don’t ask for new information or
   new approaches until the current plan has been able to produce data over time. Resist


                                                 7
                                               Monitoring and Evaluation of Capacity Development
                                                                                  December 2008

attempts by others to interfere until you have had time to assess the quality of the data
being produced by the monitoring.




                                           8
                                                     Monitoring and Evaluation of Capacity Development
                                                                                        December 2008




Reporting
    The program manger should be concerned with the quality of reporting that they receive.
But given the focus on learning that is required for good capacity development monitoring, it is
also important that the program manager assess how well the reporting leads to program
learning and improvement.

   There are some key quality requirements for reporting for capacity development which the
program manager can check. These include:

           Is the reporting providing accountability to all stakeholders? This may require
            different methods for different stakeholders groups?
           Are reporting and communication needs transparent and respectful, and do they
            support the capacity development approach?
           Does the reporting link directly back into the program management? Is it leading to
            changes in the program?

Evaluation
    The program manger is expected to reflect upon any programs at least once each year as
part of the quality review process undertaken in AusAID. This allows the opportunity to go
beyond monitoring to ask evaluative questions. In addition some programs will also undertake
formal reviews and evaluation as part of the ongoing M&E process.

     Evaluation needs to be focused around judgment about the sustainable outcomes which
have been built through capacity development. This will require consideration of a wide range
of factors, in particular how the external environment is likely to support or not the changes
which have occurred.

   Methods for evaluation need to have the same approaches as those for monitoring. They
should be inclusive of different voices, able to promote ownership by local stakeholders and
enable analysis and learning about changes across multiple levels and types of influences



    More detailed explanation of these various stages of program management are
included in the next section.




                                                 9
                                                          Monitoring and Evaluation of Capacity Development
                                                                                             December 2008




Detailed guidance for capacity development monitoring

Design and Intent
     Good quality monitoring requires that the intent of the capacity development in a program
is clear to all stakeholders from the beginning of the program. This requires that the analysis
and the design of the program give attention to the capacity development approach and
expected outcomes.

    That does not necessarily mean capacity development will appear as a major objective in
the design document. But it does mean that the capacity development approach and rationale
will form a significant part of the design proposal3. This requires attention to the following
areas.

    Means or end?
    Most programs in PNG appear to understand capacity development both as a means and
also as an outcome. That is programs want to build the capacity of people or organizations in
order to achieve tangible outcomes like improved delivery of basic services or more
representative and effective government. At the same time the programs often describe capacity
development as an end in itself, with the idea that sustainable outcomes can only be achieved if
people have the capacity to take change forward without external assistance. This mixed
approach seems to fit well with how people understand capacity development in PNG. Research
suggests that people understand capacity development when it provides a clear and obvious
link to what is important for their well being or that of their community.

     While this approach is sensible from the perspective of sustainable development for PNG, it
can create some tensions in understanding about how to monitor capacity development. Should
the focus be on long term service delivery outcomes or shorter term changes in capacity? The
answer seems to be the monitoring must address both. That is it must provide information
about the outcomes of capacity development as well the way in which that capacity
development is contributing to the overall objectives of the program.

    At a minimum, it is important for the program design to identify and address the possible
tension between the two areas of means and end and explain how this will be managed and
monitored within the program.


        The PNG Churches Partnership program
            The PNG Churches Partnership program (CPP) clearly understood that the intent of
        the program was to improve the role of the churches in delivering services in PNG and
        also in helping to create a demand for better service delivery. Building the internal
        capacity of the church was understood to be required step towards that intent.
        Therefore capacity development became one of the three outcomes areas for the
        program. It was both a means and an end of the program.

    Complement and coordinate

    3
     Bolger, J (2008) “ Pacific Choice, Learning from Success”, Capacity Development Series, Asian
Development Bank.

                                                        10
                                                             Monitoring and Evaluation of Capacity Development
                                                                                                December 2008

    In PNG few programs have capacity development as their only strategy for change.
Generally capacity development sits as one process among many within a program. Good
quality designs explain how the capacity development approach will complement and fit with
the other program interventions.

    Further, many programs funded by AusAID and other donors in PNG use capacity
development as a major strategy. It is common for people in government departments, NGOs,
churches and communities to be the recipients of capacity development from more than one
source at the same time. The program design will need to explain how the capacity development
will coordinate with other interventions and seek to utilize and work with those interventions.
The monitoring plan can then be developed based on the expected interactions and
coordination within and between programs.

   Capacity development approach
   There are several models and approaches to capacity development which have been
adapted into programs in PNG. Such models provide entry points for a new program. It is
important that the design outline the details of the particular capacity development approach
and how it has been adapted to suit the particular context and group of stakeholders.

    Information should be provided in the design about how this approach it is expected to
contribute to change. This provides a starting point for the monitoring plan. However, every
approach to capacity development brings with it certain risks. The monitoring plan needs to
give attention to these risks and the limitations of any approach within the particular context of
this program.



       Two different approaches to capacity development
            Particular attention needs to be given in the PNG context to the differences between an
       approach to capacity development that focuses on the problem or gaps in the situation versus those
       that work from an analysis of existing strengths. It appears that ‘gap based’ approaches to capacity
       development have been overused in the PNG context due to a failure of analysis and limited
       development experience. While such approaches can be successfully applied in some situations, they
       run a risk of underestimating the existing capacity in a situation and thereby undermining that
       capacity. If they are used uncritically and with insufficient attention to the wider context, they can fail
       to utilize the existing incentives for change and improvement.

            Strength based models offer considerable advantages over the more narrowly focused problem
       based models. They encourage a more systems oriented approach to understanding a situation, they
       uncover unexpected and unknown opportunities, they build upon the best of what has worked before
       and are more likely to enable people to experiment with further development opportunities. At the
       same time there are risks with this type of capacity development approach. It can underestimate the
       structural and environmental limitations on people and on what they can achieve. It can gloss over
       differences between people and how outcomes might benefit one group over another. It can
       disempower people and communities if the program fails to meet the expected outcomes.



     The use of technical assistance (TA) for capacity development is very common in PNG and
there has been considerable study about this area4. Recent research5 has shown that there are
critical aspects of TA that determine if it is effective in capacity development. These include:


   4
       http://www.ausaid.gov.au/hottopics/topic.cfm?ID=5397_541_5662_3417_8129
   5
       Land (2007)

                                                           11
                                                          Monitoring and Evaluation of Capacity Development
                                                                                             December 2008

      Ownership and leadership by the host organization
      Clearly defined and genuine demand
      Flexible and realistic design which takes good account of the external context
      TA personnel who have process as well as technical skills.

   These factors need to be addressed in the design and in the ongoing monitoring.

    Flexible and responsive design
    Any capacity development approach is only the starting point. Good quality capacity
development changes and develops over time as relationships and program understanding also
develop. Both the design and the monitoring plan need to allow for such development and be
able to change with the program.


 The Transport Sector Support Program experience:
      The TSSP is based on a flexible design process. Its approach to capacity development has been
 deliberately slow and cautious. It has made use of the Staged Model of Capacity Building, but has
 adapted the model to the PNG context, changing the language, orientating it to start with a focus on
 positive strengths and focusing on work groups rather than individuals.
      The program has seen the model as a starting point or an entry point into its capacity
 development work. It allows for a practical focus on the functions people have, which is a relatively
 safe and tangible place to being the exploration of capacity. The model is simple and easily understood
 by all the stakeholders. At the same time the model has needed adaptation and is expected to only be
 a starting point. The plan is to move from this focus to analyse why people are unable to fulfil their
 assigned functions and what other areas need to change.




   http://www.ecdpm.org/Web_ECDPM/Web/Content/Download.nsf/0/483DE3933D95FDC3C125736B002B
4AC6/$FILE/Synthesis%20report%20TA%20DP78e_FINAL.pdf


                                                       12
                                                        Monitoring and Evaluation of Capacity Development
                                                                                           December 2008




Implementation

The purpose of capacity development
    Capacity development is hard to do well. It requires adaptation into each new situation,
especially in PNG6 where people, organizations and communities can be so different from one
location to another. The evidence from PNG suggests that capacity development is an evolving
and dynamic process and good programs learn about how to develop capacity as they proceed.

    Program monitoring should support this by having a strong emphasis upon learning as well
as accountability. The monitoring will be contributing to the development of capacity through
improvement of the program.

   Learning should be a mutual process with assessment directed at all stakeholders, including
AusAID and other donors. For programs that have a high reliance on TA the mutual learning
component is particularly important7.

   In order to promote learning the monitoring needs to include both attention to outcomes
and to process. That is, the monitoring plan must ask, what is happening? But must also seek to
understand why?


The capacity development model
    Depending upon the capacity development approach in the program, different approaches
to monitoring will be developed. As noted in the section on design, the monitoring should also
give attention to the limitations and risks inherent in the particular capacity development
approach and monitor for these as well as for the expected outcomes or types of change.

    In addition there are several features which should be included in all monitoring for
capacity development. The program manager can assess the quality of the monitoring
framework by checking attention to these issues. They include:

   1. Multiple data collection tools
   In PNG people bring different perspectives on change and what it means based on their age,
gender, cultural and clan background, their power or position within an organization or
community and also from their past experience of capacity development and other change
processes.



    6
      Brinkerhoff, 2007
    http://www.ecdpm.org/Web_ECDPM/Web/Content/Navigation.nsf/index2?readform&http://www.ecdp
m.org/Web_ECDPM/Web/Content/Content.nsf/80ba021853007405c1256c790053145c/c56df8709c249bbcc1
2573060028676c?OpenDocument
    7
      drawing on research that included PNG, Land reports,
    “If TA personnel are to focus on capacity development, then appropriate performance indicators that
    recognise and incentivise process dimensions are required. Methodologies that focus on learning rather
    than just on accountability can help derive lessons on how change happens and how TA personnel can
    contribute. These approaches have been shown to motivate stakeholders towards more productive co-
    operation, and to yield better mutual understanding.”Land, 2007, pg ix
    http://www.ecdpm.org/Web_ECDPM/Web/Content/Download.nsf/0/483DE3933D95FDC3C125736B002B
4AC6/$FILE/Synthesis%20report%20TA%20DP78e_FINAL.pdf


                                                      13
                                                       Monitoring and Evaluation of Capacity Development
                                                                                          December 2008

    Therefore it is critical that the monitoring plan utilize data gathering techniques that ensure
these different perspectives or different voices are ‘heard’. That means the monitoring plan
needs to have a range of data collection techniques that will increase the participation and input
from different people8. Relying only on reports from advisors will not be sufficient to gain a
realistic understanding of capacity development.

   Multiple data collection tools should be a feature of the monitoring plan. In addition, given
the particular difficulties women have in PNG, their voice or experience should be given
additional attention through the data collection tools.

    Similarly the sense making or analysis of the data needs to be undertaken from more than
one perspective. The information about progress in capacity development needs to be analyzed
by different stakeholders, drawing on their various experiences and world views to make sense
of what the observed changes mean for the program. This includes different perspectives from
women as well as men. The analysis process should draw from the formal monitoring data but
also allow room for additional information from informal data sources. This is an opportunity
for the AusAID program manager to engage with the program monitoring, in participation in the
analysis process.


 The Basic Education Development Project experience
      The BEDP is an action-learning based community process. Its outcomes are focused around
 strengthening community participation at school levels and strengthening the school systems for
 managing and maintaining its own infrastructure.
      For monitoring the program includes multiple monitoring tools, including school based reviews and
 community self review, using the MSC approach. This data collection tool builds the communities’
 appreciation of what they have achieved, and why it might be important. It provides a way for many
 people in the community to have a voice about what has changed and why.
      The program particularly seeks to include women and to include their assessment of the changes.
 Based on the assumption that women are particularly informed when it comes to services for their
 children, the program seeks to listen in particular to women’s assessments of the changes.


    2. Attention to different levels of change
    There are many models and approaches to capacity development outlined in the literature
and research. Common to most is the view that capacity development needs to be understood
across many levels of change9. These include change for individuals, for organizations or
systems and change in the wider environment. In PNG consideration also needs to be given to
networks, or the ways in which people relate together.

    The monitoring plan will likewise have to give attention to change across several levels and
influences within a program. While it might not be possible to unravel all the possible influences

    8
      There are a range of tools that purport to enable the measurement of capacity. See for example
http://www.capacity.org/en/publications/a_brief_review_of_20_tools_to_assess_institutional_capacity
    A test of the value of these tools is the degree to which they enable the inclusion of many different
perspectives.
    9
      See:
    DAC, 2006,
    http://www.oecd.org/dataoecd/4/36/36326495.pdf
    Bolger, J (2008) “ Pacific Choice, Learning from Success”, Capacity Development Series, Asian
    Development Bank.
Morgan and Baser, 2008,
http://www.ecdpm.org/Web_ECDPM/Web/Content/Download.nsf/0/AE807798DF344457C1257442004750D6
/$FILE/08-59B_Baser_Morgan.pdf

                                                     14
                                                             Monitoring and Evaluation of Capacity Development
                                                                                                December 2008

upon a situation it is important that the monitoring be designed to enable stakeholders to
understand which of the influences is important in any particular situation.

   In particular in PNG experience suggests that there are factors related to culture10, values
and power which are critical to effective capacity development. All monitoring plans should at a
minimum address these factors which include:
    Is the leadership effective and committed to the change being sought?
    Are the relationships between the implementers and the counterparts, based on trust,
       mutual regard and a shared view of what needs to change and how?
    Are there incentives in place for change and are these being taken up? If not why?
    How is the change different for men and women?

    3. Broad based monitoring
    In addition the monitoring needs to be able to go beyond attention to program activities, to
investigate the way in which the wider environment enables or hinders change in a particular
situation. Ideally the monitoring needs to have some independence from the program so that it
can ask questions about the interaction between the program and other factors. These
questions include:

              What change is taking place and why?
              How is this being influenced by the wider environment, including issues of culture,
               gender, power and the values people hold, as well as the actions of other
               organizations or systems?
              What does this mean for this program?
              How should the program adapt and change to address these enabling or
               constraining factors?


  The Churches Partnership Program experience:
       The CPP is based on a flexible design with three broad outcome areas which included improvement of
  the capacity of churches in PNG.
       While the focus is on the church as a whole the program takes account of change in individuals as well as
  changes in the wider environment. The program is also concerned with the values and ways of relating that
  people bring into the program. Recent review of the program concluded that it has the potential to define a
  PNG approach to capacity development, based on PNG values and understanding or how organizations
  should operate (Rhodes, 2008).




    10
         A study on the PNG health sector,
     It suggests that the system in PNG is a complex of competing and occasionally complementary policy
objectives, institutional arrangements, relationships, incentive systems, and political interests, some of which
support efforts to strengthen sector capacity and improve performance, and others which can undermine it. The
study also points to the importance of Papua New Guinean culture, traditions and diversity as factors influencing
organisational behaviour, stakeholder collaboration, and even the perceived legitimacy of the state.( Bolger, et al,
2005,
http://www.ecdpm.org/Web_ECDPM/Web/Content/Navigation.nsf/index2?readform&http://www.ecdpm.org/Web_
ECDPM/Web/Content/Content.nsf/0/82828882ADEB0C8AC1256FB1005D44C4?Opendocument


                                                           15
                                                               Monitoring and Evaluation of Capacity Development
                                                                                                  December 2008



     The Law and Justice Sector program experience:
         The L&JS is a flexible program working with a range of law and justice sector agencies and community
     organisations. It has a long term commitment to developing PNG leadership and ownership of capacity
     development.
         While the program has a comprehensive monitoring process which ranges across several government
     departments and other organisations it also takes a sector wide perspective on understanding change and
     progress. This has enabled it to target capacity development at national level as well as in traditional justice
     systems and non-government stakeholders.
         To try to assist with making sense of the sum of these interactions the program has sought to experiment
     with monitoring based on the ECDPM model of capacity development (Morgan and Baser, 2008). While this
     process is currently still being developed it is an attempt to use a sophisticated and comprehensive model of
     capacity development to make sense of the multiple levels of intervention and influence in the sector wide
     program.


    4. Contribution to ownership
    Research shows that for capacity development to be effective it needs to be participative
and, as far as possible, owned and managed by the stakeholders in the program. The monitoring
needs to support this approach. This means that the monitoring plan should, as far as possible,
use existing systems for data collection and analysis11. Where these systems are still under
development the monitoring process should include attention to supporting and improving
these systems.


     The Sub-National Strategy experience:
          The SNS program works across several levels of the GoPNG as well as across locations and
     departments to try to improve the sub-national PNG Government administration. Data is collected
     through multiple strategies and from multiple sources.
          In particular the data collection relies upon government systems as far as possible. For the PPII
     component of the program these include the existing provincial monitoring undertaken by the
     Department of Provincial and Legal Government Affairs and the annual provincial reports to National
     government.
          Including these processes as central to the AusAID monitoring enhances their legitimacy and also
     supports the Government of PNG ownership over the PPII program. It focuses attention to how these
     monitoring processes should be improved.
          In many cases the available information also challenges the information that is gathered from the
     technical advisors and AusAID staff, providing a different perspective and analysis about a situation. This
     difference increases the overall validity of the program assessments.


    Existing Government of PNG systems tend to focus on outcomes and provide useful
information about the change that has taken place. They are less likely to provide information
about process. Additional monitoring systems are often required to provide this information.

    Where new systems for data collection need to be developed they should be based on
participatory approaches that will engage stakeholders in the monitoring process. The methods
should also promote accountability to all stakeholders, not just to AusAID.




11
  Watson, 2006,
http://www.ecdpm.org/Web_ECDPM/Web/Content/Download.nsf/0/4EB26B200266AED5C12570C1003E2
8A2/$FILE/Watson_M&E%20of%20capacity%20and%20CD_2006_DP58B.pdf


                                                            16
                                                       Monitoring and Evaluation of Capacity Development
                                                                                          December 2008



 The Community Development Scheme experience
 The CDS developed tool called Participatory Evaluation (PE). This is a simple participatory exercise
 which encourages the community (or project participants) to stori individual and group experiences
 from CDS funded activities. It separates the men and women (and/or leaders, youth etc as relevant) so
 each group can discuss how the project processes delivered the activities and outputs and what benefits
 and outcomes came from them. At the start of the PE, review is made of the following:
         The outputs
         The project delivery processes
         Achievements and successes
         Difficulties encountered
         Contributing factors

 It then moves on to discuss what benefits and outcomes flowed from these outputs and processes, and
 what were the most significant changes or impacts experienced through the project; both the positive
 and negative changes.


    5. Using exploratory monitoring approaches
    Capacity development models are useful ways to get started: they provide a framework for
action and for engagement with individuals and organizations. Yet they need to be treated with
some caution. Models are only an approximation of how change might happen. The monitoring
plan will need to understand and utilize the capacity development approach in the program as
well as monitor beyond that approach. This includes being able to identify and explore
unexpected outcomes and changes.

   For this type of monitoring it is unlikely that fixed and predictive M&E approaches will suit
most programs. Exploratory monitoring approaches, based on flexible methodologies are
generally more suitable for monitoring of capacity development process and outcomes.

   Examples of exploratory monitoring approaches include Most Significant Change (MSC) and
Outcome Mapping. There are other approaches which share the qualitative and exploratory
nature of these approaches. Key features include:

     •      Structured interaction and reflection between stakeholders.
     •      The approaches are not concerned primarily with quantitative measurement or analysis,
            but with getting people focused on what has changed.
     •      They rarely make reference to detailed, predetermined outcome indicators, but are
            more likely focus on what has actually happened
     •      These approaches usually involve dissemination of information about 'what happened'
            and lead people to think and ask ‘why’?
     •      They attempt to demystify and de-professionalise M&E and allow clients - including the
            most vulnerable - to have a voice in periodic reflection on achievements and learning to
            date.
     •      They therefore develop capacities for analysis, debate and consensual decision making
            among stakeholders and the staff of the organisations.12




     12
      Based on Watson,(2006),
http://www.ecdpm.org/Web_ECDPM/Web/Content/Download.nsf/0/4EB26B200266AED5C12570C1003E2
8A2/$FILE/Watson_M&E%20of%20capacity%20and%20CD_2006_DP58B.pdf


                                                    17
                                                   Monitoring and Evaluation of Capacity Development
                                                                                      December 2008


   6. A note about monitoring technical assistance
       As noted in the section on design, recent research and experience in PNG has now led to
   a more clear understanding of the use of TA as a capacity development tool. There are
   certain features which are more likely to ensure TA is effective in both helping to develop
   capacity and also in contributing to program outcomes.

        A first consideration is the objective of using TA. Where TA is being used to substitute
   for lack of capacity then the focus on capacity development is a secondary consideration. In
   these situations capacity development may be a side benefit of the intervention or may not
   happen at all. There is a focus on how well the task is completed and the general ownership
   of the outcomes, and this is where the monitoring should focus. Capacity development
   change is not likely to be strongly featured in the monitoring plan, although the well
   established risk that such TA might undermine existing capacity needs to be carefully
   monitored during the life of the program.

       If the TA is intended to develop capacity, then there are several factors which should be
   given attention during monitoring. Progress in these areas is likely to suggest that the TA is
   more able to contribute to longer term sustainable change. They include:

       •   How well the use of the TA fits within a broader analysis of how change happens in
           the situation, especially giving attention to the complex nature of sustained change
           within the public service in PNG?
           How is the program maintaining attention to these wider issues?
           How easy it is for the TA and their counterparts to make sense of these wider
           considerations of social, political and cultural constraints to change and what they
           are able to achieve within that context?
       •   How well does the TA or the use of TA fit within the local systems, including systems
           of informal decision making, communication, networking and values?
       •   How is the government of PNG ownership of the TA being supported and improved
           by the program? How is the TA being more integrated into existing government
           structures, including those of performance appraisal, accountability and
           management?
       •   Are the relationships between the TA and their counterparts based on reciprocity,
           mutual respect, transparency and loyalty? Has the structure of the program
           supported the development of this type of relationships?


Monitoring over time
    Capacity development takes a long time. It is important to allow the capacity development
process to change and develop over time, and also to expect that the monitoring will likewise
develop and change during the program life. However introducing new ideas, methods,
questions too soon into processes which have been agreed by stakeholders, can be very
damaging and distracting and can undermine capacity development and monitoring systems.
The program manager should be very careful to ensure the capacity development approach and
the monitoring has sufficient time and space to prove its worth.




                                                 18
                                                   Monitoring and Evaluation of Capacity Development
                                                                                      December 2008




Reporting and program development
Multiple accountabilities
   The information about capacity development is important for learning in the program. And
people cannot change unless they have information about their situation. For these reasons it is
very important that the assessment and understanding generated by the monitoring about
capacity development is available to all stakeholders.

    In order to ensure information is accessible it may have to be presented in different formats
for different audiences. Concise reports are useful to AusAID, but other audiences may prefer
longer more detailed information with examples and stories or information presented verbally
where there is opportunity for discussion and clarification.

Transparency
    Effective capacity development requires a respectful approach to sharing of information.
That means assessment and analysis of change needs to be honestly shared with all the
stakeholders. At the same time relationships are very important in PNG and showing respect for
others is very important in communications. Reporting on sensitive or negative assessments of
change should be undertaken carefully, using the relationships already established through the
capacity development process. Sometimes such processes are handled more effectively by PNG
counterparts.

Program change and development
    Capacity development monitoring needs to provide learning for program management.
While some information will be important for accountability, including AusAID accountability,
the monitoring should be primarily tailored to provide accessible and timely information for the
program implementers and other stakeholders. There should be clear systems for ensuring that
the program is developing based on the assessments that come from monitoring.

    As a result of the new information there should be ongoing change and development in the
capacity development of a program. It is critical that lessons learned are reflected in new
strategies and actions under the program.




                                                 19
                                                   Monitoring and Evaluation of Capacity Development
                                                                                      December 2008




Evaluation and completion
    Focus on sustainable change
    Evaluation provides the opportunity to understand the value of the program and what has
been achieved through this approach to aid delivery. For the capacity development it provides
an important opportunity to understand what sustainable change has occurred and why this
has been able to happen in this situation. The evaluation approach should give particular
attention to this area.

     Evaluation methods
     There are several evaluation methods which can be applied either during or at the end of a
program. In line with the previous comments about capacity development it is important that
these methods are able to include different views and perspective, can take into account the
changes at different levels and the different influences of culture, gender, age and power in PNG.
It is also important that the evaluation continue to build ownership and capacity of local
stakeholders to analyze and change their own situation.

    Similar to the monitoring process it is unlikely that simple extractive methods of evaluation
will be sufficient for this approach. Exploratory and participative methods will be more in line
with what is required.




                                                 20

								
To top