Evaluating Partnerships by yaoyufang


									Evaluating Partnerships

 Presentation to the DURAS
   Project closing seminar
       June 12, 2008

    ILAC Institutional Learning & Change
• Defining “partnership”
• Overarching organizing framework
• Indicators and methods

           ILAC Institutional Learning & Change
Part 1: Defining Partnership

       ILAC Institutional Learning & Change
    The nature of partnerships
• Partnerships are not contracts
• Partners come together because each partner
  believes that they can derived real or perceived
• And that the benefits will outweigh the costs or
  efforts involved in the coming together and
  working together
• Drivers of collaboration: incentives,
  disincentives and obligations
• Understanding what motivates the partners to
  operate in partnership from the perspective of
  the partners is an essential component of
  partnership evaluation
              ILAC Institutional Learning & Change
  Another word about partnerships
• Partnership is not a gift: all participants are there
  to gain benefits to forward their goals
• Partnership is not based on conditionality:
  conditions cannot be imposed rather participants
  define their own conditions together
• Partnership is not a “principle-agent”
  relationship: the more powerful cannot prescribe
  the terms as if in an employment contract
• Partnership is not simply a team activity: while
  there are some common interests, there may
  also be some divergent interests

               ILAC Institutional Learning & Change
           Spectrum of participation
• Conventional research: “scientists” make the decisions
  alone without organized communication with “farmers.”
• Contractual: scientists contract with farmers for their
• Consultative: scientists make decisions alone but with
  organized communication with farmers.
• Collaborative: decision-making authority is shared
  between farmers and scientists based on organised
  communication between the two groups. Neither party has
  the right to revoke or override the joint decision.
• Collegial: farmers make decisions collectively either in a
  group process or through individual farmers who are in
  organized communication with scientists.
• Farmer experimentation: farmers make the decisions
  without organized communication with scientists.

                  ILAC Institutional Learning & Change
 Functional (improving adoption or some other
               goal achievement)
 Empowerment (changing the power balance)

Participatory research (discovering together the
Participatory learning (learning about something
        known by discovering it for yourself)

            ILAC Institutional Learning & Change
Part 2: An organizing framework

        ILAC Institutional Learning & Change
    Simple                    Complicated Complex
  Following a Recipe          A Rocket to the Moon                 Raising a Child
                               •   Formulae are critical •        Formulae have only a
The recipe is essential           and necessary                  limited application

Recipes are tested to         •   Sending one rocket         •   Raising one child
assure replicability of            increases assurance            gives no assurance of
later efforts                      that next will be ok           success with the next

No particular                 •   High level of              •   Expertise can help
expertise; knowing how             expertise in many              but is not sufficient;
to cook increases                  specialized fields +           relationships are
success                            coordination                   key
Recipe notes the              •   Separate into parts        •   Can’t separate parts
quantity and nature of             and then coordinate            from the whole
“parts” needed
                               •   Rockets similar in         •   Every child is unique
Recipes produce                   critical ways
standard products                                             •   Uncertainty of
                               •   High degree of                 outcome remains
Certainty of same                 certainty of outcome
results every time
                                          M.Q. Patton May, 2008
                          ILAC Institutional Learning & Change
          Simple (Known)

Inter-relationships between elements:

• Tight, centralised connections
• Anyone can see the things the way they
• Very simple linear cause and effect
• Everyone knows the right answer within
     the current context

                    MQ Patton May, 2008
           ILAC Institutional Learning & Change
 Complicated (Knowable)
Inter-relationships between elements:

•Relationships are loose but still
clustered around a central core
•Cause and effect is non-linear
•Relationships can be modelled and
•An expert would know the right answer(s)

                     Michael Quinn Patton
         ILAC             May, 2008
                Institutional Learning &    Change
  Inter relationship between elements:

•Difficult to tell where the core is within wide
      range of relationships (core may not
•Cause effect difficult to understand in
      current setting
• Situation understandable only in retrospect
• Not predictable

                      MQ Patton May, 2008
             ILAC Institutional Learning & Change
Close to
                           Zone of

           Close to             Certainty   Far from

                      MQ Patton May, 2008
    ILAC Institutional Learning & Change
             Types of interventions
Simple intervention           Complicated or complex
Single causal strand.         Multiple simultaneous causal
Intervention is sufficient to    strands required to produce
   produce the impacts           the impacts
Universal mechanism.          Different causal mechanisms
Intervention is necessary to     operating        in    different
   produce the impacts           contexts
Linear causality,             Recursive,       with   feedback
   proportional impact           loops,         leading        to
                                 disproportionate impact at
                                 critical levels
Pre-identified outcomes       Emergent outcomes
                ILAC Institutional Learning & Change
            Simple            Complicated             Complex
Deciding    Likely to be      Likely to differ, May be
impacts     agreed            reflecting        emergent
                              different agendas
Describing More likely to Evidence needed Harder to plan
impacts    have           about multiple  for given
            standardised      components              emergence
Analysing   Likely to be      Causal packages Unique, highly
cause       clear             and non-linearity contingent
            counter-                            causality
Reporting   Clear             Complicated             Uptake requires
            messages          message                 further
               ILAC Institutional Learning & Change
 Traditional evaluation approaches
• Cost benefit analysis
• Logical framework
• SMART criteria (specific, measurable,
  achievable, realistic, time bound)
• Organizational assessment approaches
• But these may miss the evolutionary
  nature of partnerships and the complexity
  within the partnership and between the
  partners and external forces

            ILAC Institutional Learning & Change
       Developmental Evaluation: Emerging
            Approach for complexity

Evaluation processes support program, product, staff
  and/or organizational development.
The evaluator is part of a team whose members collaborate
  to conceptualize, design and test new approaches in a
  long-term, on-going process of continuous improvement,
  adaptation and intentional change.
The evaluator's primary function in the team is to elucidate
  team discussions with evaluative questions, data and
  logic, and facilitate data-based decision-making in the
  developmental process.
Capacity to learn might be more relevant than specific
                                                        MQ Patton May, 2008

                 ILAC Institutional Learning & Change
Part 3 Indicators and Methods

       ILAC Institutional Learning & Change
           What to evaluate
• Results of the partnership
• Quality of the partnership itself (degree of
  ownership, levels of participation, range of
  partners) and changes over time
• Short term outcomes
• Longer term outcomes and sustainability
  of them
• Empowerment or learning
• Capacity for learning
            ILAC Institutional Learning & Change
      Proven benefits from PR*
• Increased the benefits and is more effective at
  reaching women and the poor
• Improves research efficiency
• Leads to more acceptable varieties and
  accelerates adoption
• Leads to changes in costs that do not lower cost
  benefit ratios and may improve these

*with careful research goals, targeting of environments and
  selection of user communities

                ILAC Institutional Learning & Change
Partnership Program Indicators
• Benefits derived from the partnership to each partner over
• Evolution of outcomes
• Transactions costs
• Changed outcomes (behaviors, relationships, actions and
• Sustainability of actions and relationships
• Time to adoption/change point
• Local adaptation of technologies

Partnership Process Indicators
• Evolving relationships over time (social networks)
• Transparency of relationships and operations
• Degree of sharing of resources/costs
• Quality of decision making and processes
• Empowerment and learning
                ILAC Institutional Learning & Change
                     Actor network maps
                Farmer Research Groups in Honduras

(i) 1996                        (ii) 2003

     Source: Douthwaite 2004.
                        ILAC Institutional Learning & Change
Organizational Learning Capacity =
Ability to adapt and respond to the
            next situation
• Supportive learning environment
   – Psychological safety, openness to new ideas, time for
• Learning processes and practices
   – Experimentation, information collection, analysis,
     education & training
• Leadership that reinforces learning

Experimentation, risk taking, participatory decision
  making, dialogue, interaction with external

              ILAC Institutional Learning & Change
                 Ethical issues
• Is it ethical, especially within a partnership or
  empowerment project to conduct evaluation so that
  partners are excluded from opportunities to learn and
  benefit from the evaluation process?
• Cognitive dissonance: Can you work in collaboration and
  embrace those values but adopt a different set of values
  (expert driven and external-objective) for evaluation work?
• Are partners rights and values being respected?
• Are power imbalances addressed in the evaluation
• Are disempowered being protected or threatened by the
  evaluation process?
• Is information ownership respected?
• Are people likely to be put at risk by the evaluation
  process (confidentiality and prudence)
                ILAC Institutional Learning & Change
          Attribution issues
• To what extent could or should the
  evaluation of partnerships attempt to
  attribute to individual partners effects or
• To attempt to do so affect the trust needed
  within the partnerships?
• To what extent must you compromise
  methods and rigor in the interest of the

            ILAC Institutional Learning & Change
        Qualitative data analysis
Information gathered in non numeric form, may include:
• Qualitative data analysis is an iterative process

• Get to know the data: read and re read the text
• Identify common themes or patterns that emerge from the data
   – Emergent categories
   – Pre determined categories (sometimes coming from theory or literature)
• Organize the information into coherent groups
• Triangulate information from different sources to confirm themes and
  patterns or deviations from themes or patterns from across data
• Compare the findings from different analysts and resolve conflicts
• Manage the data as data: coding, confidentiality, archiving

                   ILAC Institutional Learning & Change
 ALPS: accountability, learning and
         planning system
• 3 Core elements: Principles, attitudes & behaviors and
  organizational policies and practices
• Accountability to target groups
• Strengthened commitment to women’s rights and gender
• Requires a constant analysis and action on power
• Aims to simplify reporting requirements at the same time
  emphasizing critical engagement, mutual learning and
  downward accountability
• Requires transparency and proactive sharing of

               ILAC Institutional Learning & Change
            Other methods
• Participatory Impact Pathway Analysis
  (before and during)
• Social Network Analysis (before and after)
• Outcome mapping (before, during and
• Contribution analysis (after)
• Horizontal evaluation (during and after)

            ILAC Institutional Learning & Change

To top