Documents
Resources
Learning Center
Upload
Plans & pricing Sign in
Sign Out

Tools review

VIEWS: 5 PAGES: 36

									                            Guidelines No. 2
Improving the Impact of
 Microfinance on Poverty

        Action Research     A Review of Impact Assessment Tools
             Programme




                            September 2001



                            Anton Simanowitz




                                                                           Imp-Act
                                                      The Institute of Development Studies,
                                          at the University of Sussex, Brighton BN1 9RE, UK
                             Tel: +44 (0) 1273 873733, Fax: +44 (0) 1273 621202/691647
                                       Email: Imp-Act@ids.ac.uk     Web: www.Imp-Act.org
                                                                                     1
               Charitable Company no.877338 limited by guarantee and registered in England
CONTENTS

Introduction: What is this Tools Review? __________________ 3
 What is a tool? __________________________________________ 3
 What is an impact assessment tool? _________________________ 3
 So what will this review do?________________________________ 3
 Getting the mix right _____________________________________ 4
 Start with analysis! ______________________________________ 4
Section One: Review of Types of Tools ____________________ 5
 1. Quantitative/Qualitative and Participatory tools ______________ 5
 2. Quantitative Tools _____________________________________ 6
  2.1 Surveys ________________________________________________________ 6
  2.2 Participatory tools ________________________________________________ 6
  2.3 Impact monitoring/management information systems ____________________ 6
 3. Qualitative Tools ______________________________________ 8
 4. Participatory Tools (see MSA tool-kit) ______________________ 8
 5. Market Research Tools (see MSA tool-kit) ___________________ 9
 6. Impact Monitoring (see discussion under Quantitative tools) ____ 9
 7. Participatory Monitoring Systems _________________________ 9
 8. Organisational Assessment Systems _______________________ 9
 9. IA systems ___________________________________________ 9
Section Two: Detailed Annex of Tools ____________________ 11
 Index ________________________________________________ 11
 1. Tool Kits ____________________________________________ 12
  1.1 AIMS Practitioner Tool Kit & Guidelines for Microfinance Impact Assessments _ 12
  1.2 MSA Tool-Kit ___________________________________________________ 13
  1.3 IKM Framework, PlaNet Finance ____________________________________ 16
 2. Operational Systems __________________________________ 18
  2.1   Small Enterprise Foundation, South Africa ____________________________ 18
  2.2   Impact Audit, Cetzam, Zambia _____________________________________ 20
  2.3   Impact monitoring systems, Workers Bank of Jamaica ___________________ 21
  2.4   Progress Tracking, Freedom from Hunger _____________________________ 23
  2.5   Remeans/Net-worth Test: CARD Philippines/Grameen replicants ___________ 25
  2.6   Annual Impact Tracking System, Enterprise Works Worldwide, Eric Hyman ___ 26
 3.   Participatory monitoring systems _______________________ 28
  3.1 Internal Learning System, Helzi Noponen _____________________________ 28
  3.2. Participatory Impact Monitoring and Evaluation System (PIMES), Sinapi Aba
  Trust, Ghana ______________________________________________________ 30
  3.3. Participatory Monitoring System, Christian Commission for Development in
  Bangladesh (CCDB) _________________________________________________ 32
 4. Institutional analysis __________________________________ 33
  4.1. MicroAssess, PACT, USA __________________________________________ 33
  4. 2. Development Audit, EDA, India ____________________________________ 35



                                                                                    2
             A Review of Impact Assessment Tools


Introduction: What is this Tools Review?

A review of impact assessment tools sounds pretty straight-forward. All that is
needed is to look through the impact assessment work done by different
organisations and consultants and to describe and evaluate the tools that they
have used. But the reality of looking for tools is more complex.

What is a tool?

To start with we have to define what we mean by tools. An impact assessment
tool is a mechanism of obtaining the answer to certain questions or revealing
certain information about impact that we are looking for. So the simplest tool is
a single question asked. Where the question is asked in a methodical and
thought out way, with the questioner knowing why they are asking the question,
and having some notion of how the answer given will relate to the bigger picture
or context in which the question is asked, then this can be defined as a tool.

There are many so-called “innovative” tools.          However, these tools are
essentially just ways of asking the simple question in a “better” way ie. in a way
that is more likely to result in accurate answers, or in a way which targets the
information given to a specific information need. For example, participatory
tools were developed in order to allow respondents a greater ability to
understand the research process and thereby give answers that are more likely
to be thought through. The AIMS client drop-out tool provides a structure for an
interview that is directed towards learning about specific issues. Neither tool is
unique and a skilled researcher is likely to incorporate elements of these tools
and approaches and others, without necessarily defining these as specific tools.
The key issue is to approach research in a flexible and open way, and with a
good grounding in the issues to be investigated. Tools then provide the
researcher with ideas to assist in the planning and development of an interview
process, they are not a rigid formula that needs to be applied in the same way in
all contexts.

What is an impact assessment tool?

Information is gathered in different types of research for different purposes.
There are standard research methods that are taught in Universities and other
institutions (Social research methods; market research methods; anthropological
research methods; economic research methods etc). There is nothing unique
about impact assessment that sets it aside from other research. It is the
process of defining the research questions (in this case impact) which
determines which mix of methods or tools are appropriate for answering the
questions, and answering them in an appropriate way.

So what will this review do?

So if impact assessment tools are simply a way of packaging existing research
methods a tools review could be a mammoth task. At one extreme this could be
a review of all research methods ever used! At the other could be a review of
only named packages of tools that have been used for microfinance impact
                                                                                3
assessment. Even if the focus is just on impact assessment this does not
necessarily need to be limited to microfinance, since there is a lot of overlap and
relevance of impact assessment conducted for other economic and social
development programmes. This review sets out to be somewhere in-between.

Section one gives a broad overview of research methods that can be used for
impact assessment. Only a very brief outline is given of each method or tool as
it is assumed that readers can refer to other sources of information for a more
detailed understanding.

Section two is an annex of specific packages or systems of tools as applied to
microfinance organisations.    This review aims to give an overview of the
objectives behind the tools, a summary of the different components, and some
pointers as to the strengths and issues in their application. The list covered is
not exhaustive, but rather aims to give a sense of the range of approaches
available. Links are given to specific documents or organisations should the
reader wish to learn more about a specific tool or approach.

It is important in looking at the tools to bear in mind that they are a specific mix
of methods that have been developed to meet specific needs and contexts.
When applying tools to other contexts or objectives it is necessary to think
through how they might need to be adapted.

Getting the mix right

There is no one way which is correct to do IA. The skill is in choosing the right
mix of methods. The selection of tools is a balancing act between choosing
something that will reliably collect the data needed and meet other objectives of
the IA within the constraints imposed by the organisation and its context. For
example, there may be a commitment to involve clients in the IA process, but
this is limited by the demands placed on clients and the MFI in terms of time,
skills, or resources required for data collection or analysis.

Each impact assessment develops its own “tool-kit” of methods for finding out
the information that has been prioritised. The mix of tools and how they are
applied very much depends on the objectives of the impact assessment and the
people applying it. Designing an impact assessment study or system is about
selecting a mix of methods which can complement each other, blending to give a
holistic picture. The right “package” is determined by the purpose of the study,
the use that will be made of the information (eg. what is the relative balance
between the need for a description of patterns compared to an understanding of
reasons and the audience, the degree of rigour required, the skills and capacity
of the staff implementing the IA, and the resources available.

Start with analysis

The starting point in any impact assessment should be analysis! A major
problem, particularly in quantitative research, is that lot of the data collected is
not used at all. Thus, when designing and conducting research, one has be
prudent in the use of resources by putting thought into why the data is being
collected. Part of the tools selection process therefore, should be to look at the
data a particular tool will produce and to think about how this will be processed
and analysed.

                                                                                  4
Section One: Review of Types of Tools1

1. Quantitative/Qualitative and Participatory tools

Quantitative methods are useful for getting broad descriptions of a situation,
how it has changed or measured impacts. They can give rigorous information
that can assist with analysing descriptions of patterns, for example what
changes are associated with what characteristics or inputs. They are useful in
supporting conclusions drawn from more in-depth study but on their own do not
lead to understanding of why changes have occurred.

Qualitative methods are useful for understanding the reasons for events
described in research or impact assessment. Because they generally are more
in-depth and time-consuming than quantitative tools, qualitative tools generally
do not gather enough information for general patterns to be described
statistically.

There is a tendency to think that participatory methods are qualitative, and
where quantitative data is required a survey is required. However, many
participatory methods can provide quantitative data, and if implemented well
can do so in an objective and rigorous way. The real distinction therefore should
be between conventional and participatory methods to do qualitative and
quantitative research (see diagram below).




1 Note where literature already exists describing a particular tool or set of tools this is referenced
rather than being re-written in this document.
                                                                                                    5
2. Quantitative Tools

General references

   DfID EDIAIS website http://www.enterprise-impact.org.uk/ir-
    toolbox.htm#Quantitative
    A detailed description of various aspects of impact assessment, with a
    document describing quantitative methods.

2.1 Surveys

Quasi-experimental design or modelling

References

1. Roche, Chris, “Impact Assessment for Development Agencies: Learning to
   Value Change”, Oxfam (1999)
2. Barnes, Carolyn & Sebstad, Jennefer, “Guidelines for Microfinance Impact
   Assessments”        AIMS        (March      2000),      available      at
   http://www.ids.ac.uk/cgap/static/1786.htm
3. SEEP/AIMS Tools – Survey tool; detailed step by step guide in the design,
   implementation      and     analysis     of    an     impact      survey.
   http://www.ids.ac.uk/cgap/static/1646.htm

2.2 Participatory tools

Participatory methods can be used in a number of ways to collect quantitative
data.

   Rapid survey methods: Participatory methods can be used to quickly
    gather a lot of information from a large number of people.               Many
    participatory tools, for example matrix ranking, generate quantitative results,
    which can quickly give detailed information about community level or
    household level patterns. Community meetings may be used to ask people to
    vote on certain issues, thus giving a quantitative picture of the views of the
    community. Wealth ranking data may be used to give a quantitative view of
    the relative wealth of households in a community.

   Participatory monitoring (see below): Participatory methods may be used
    to collect simple quantitative data on a regular basis.

Resources

   “Impact Assessment Using Participatory Approaches: ‘Starter Pack’ and
    Sustainable Agriculture in Malawi”; Rowland Chirwa with KWERA
    Development Centre, ODI www.odi.org.uk/agren/publist.html or direct link to
    paper in PDF format
    http://www.odi.org.uk/agren/papers/agrenpaper_112.pdf

2.3 Impact monitoring/management information systems

   Client entry data: All MFIs collect some data about their clients on a one-
    off or regular basis. Client entry data (“member information forms”) can be

                                                                                 6
    used to collect base-line data which can be used to track change through
    impact surveys or monitoring. The type of data that can be collected
    includes: household members (disaggregated by gender, age, level of
    education), sources of income, sources of credit and savings, housing
    condition, income and expenditure data (difficult to do accurately) etc. It is
    important that the process of selecting what data to collect and how to collect
    it in an accurate way is carefully thought through. The processing/storage
    and analysis of the data also needs to be planned.

   Management information systems:             Many MFIs have management
    information or loan tracking systems that collect data which can be useful for
    impact assessment. Data collected may include some of the following: (loan
    size, length of time in the programme, repayment record, savings amounts
    and regularity, business value etc).      Where this data is linked to an
    understanding of client livelihoods and impact much of this data may act as
    useful impact indicators. For example accumulation of savings may be a
    good indicator of reducing vulnerability. Irregularity of loan repayment or
    savings may indicate financial difficulties and over-payments indicate
    success.

   Impact monitoring systems: Once an organisation has reviewed existing
    base-line information and information which is collected on a regular basis, it
    may consider collecting additional impact information on a regular basis.
    Data on a small number of indicators may be collected over time, thus
    providing longitudinal quantitative panel data on these clients. This data may
    form the basis for quantitative impact analysis, as well as providing timely
    information for management.

Impact monitoring collects a small amount of impact information on a regular
basis from a sample or all of an MFI’s clients. A sample may be chosen if the
primary purpose is to develop a quantitative understanding of patterns of
change amongst clients. Where the purpose is to facilitate greater interaction
and understanding between loan officer and clients the MFI may choose to
include all clients in the monitoring.

Impact monitoring is generally very constrained by lack of time and resources,
since it is usually an additional task for loan officers. The data that can be
collected is therefore very limited. It is therefore vital that qualitative research
is first conducted to determine which indicators will provide the most accurate
reflection of the type of impact information that the organisation wishes to
gather. The most common mistake with impact monitoring is to select indicators
that are not fully grounded in an understanding of the processes they seek to
monitor. This is also a problem in other quantitative survey work where
insufficient understanding goes into the development of indicators, and the
indicators are not rigorous in their usefulness.

The most commonly used form of impact monitoring is to collect a small amount
of data as part of the loan application process, or integrated into other routine
field staff activities. For example, at SEF impact monitoring data is collected at
several different points by different staff (loan application, centre leadership
elections, centre evaluations). The form that the monitoring takes depends very
much on the objectives of the process. Where the primary aim is to collect data
then a simple questionnaire form may be administered by the loan officer.

                                                                                  7
Where a more qualitative understanding is required more participatory methods
may be used to facilitate greater interaction between the loan officer and the
client.

3. Qualitative Tools

General Resources

   DfID EDIAIS website http://www.enterprise-impact.org.uk/ir-
    toolbox.htm#Qualitative
    A detailed description of various aspects of impact assessment, with a
    document describing qualitative methods.

   Focus group discussions

See training material in MicroSave-Africa tool-kit

   Oral histories

   Participatory methods (see participatory tools)

   In-depth “case-study” research looking at a range of impact questions or
    focused on specific issues (e.g. savings, empowerment, social capital, asset
    building, business development, intra-household relationships etc)

   Specific tools (e.g. to understand reasons for client drop-out, AIMS
    Empowerment tools etc)

   Participant observation

4. Participatory Tools (see MSA tool-kit)

General Resources

   DfID EDIAIS website http://www.enterprise-impact.org.uk/ir-
    toolbox.htm#Participatory
    A detailed description of various aspects of impact assessment, with a
    document describing participatory methods.

Drama/role plays                             Livelihood diagrams/mapping
Pocket charts                                Flow diagrams
Visual methods                               Force field analysis
Story with a gap                             Time-lines and calendars
Voting systems                               Transect walks
Card sorting methods                         Key informants
Causal diagrams                              Life histories
Venn/Chapati diagrams                        Ranking – pairwise, wealth
Daily activity charts                        Seasonal calendars
Mapping - body mapping, health,              Time lines
historical, land use, mobility,              Etc, Etc
resource, social, village
Matricies



                                                                              8
5. Market Research Tools (see MSA tool-kit)

   Designed around specific questions e.g. client satisfaction, loan use, business
    development
    Eg. AIMS tools (empowerment, client satisfaction etc); MSA tools

6. Impact Monitoring (see discussion under Quantitative tools)

   Data collected on an on-going basis as part of normal operations. Usually
    integrated into existing loan-officer work patterns and management
    information. May be a combination of quantitative and qualitative data, but
    generally a very limited number of simple indicators collected.

7. Participatory Monitoring Systems

   Participatory monitoring systems are a specific type of impact monitoring
    where a great emphasis is placed on the collection and analysis of data and
    learning by clients themselves. Clients are involved in the process of
    selecting indicators to monitor and will monitor and often act on the data
    themselves. This data can then be passed onto the MFI staff and also form
    part of a more formal impact monitoring system.

8. Organisational Assessment Systems

   Organisational assessment systems look beyond the impacts of an
    organisation on its clients and attempt to look at the overall functioning and
    effectiveness of the organisation. The impact on clients is therefore seen in
    the wider organisational context, reflecting mission, organisational culture,
    operational practices, organisational learning and changes systems, feedback
    mechanisms from staff to management and clients to staff etc. This type of
    assessment provides a much more holistic picture of impact and is
    particularly useful for organisations wishing to look at how to improve their
    operational systems as well as products and services in order to maximise
    their impact and achievement of mission.

9. IA systems

   An impact assessment system integrates a number of different components
    in a way that is both institutionalised on an on-going basis and that provides
    rigorous information that meets the needs of the organisation’s stakeholders.

    David Hulme2 http://www.ids.ac.uk/cgap/static/28.htm outlines a framework
    for effective and credible institutionalised impact assessment systems.

   “Impact measurement tools that focus on improving, rather than proving,
    may incorporate some of the tools already in use, but will utilize them
    differently. Impact measurement tools that get used by management on a
    regular basis will take a different focus.


2
 Impact assessment methodologies for microfinance: A review; Assessing Impact of
Microenterprise Services (AIMS), Management Systems International / USAID (1997)



                                                                                   9
   They will be incorporated into the regular routine of data collection in an
    organization.

   Staff members will collect and analyze the data.

   The data collected and analyzed will remain consistent over time and
    location, allowing for trend analysis and comparisons of different
    geographic areas.

   They will include a point of comparison with people in the same
    community who do not receive loans

   The results will be reviewed on a regular (at least annual) basis and
    become a key part of the planning process.

   The analysis will seek to provide input to the question "How can we
    improve the positive impact and promote transformation?" rather than the
    question "Was there any impact?"

   Outside experts-ideally local ones-will be brought in to verify the accuracy
    of the data, make recommendations for improvements in the collection
    process, and help in developing control group comparisons.”




                                                                             10
Section Two: Detailed Annex of Tools

Index

1 Tool-kits
  1.1 AIMS Practitioner Tools and Guidelines for Microfinance Impact
     Assessments
  1.2 MSA Market Research and PRA tool-kits
  1.3 IKM Framework

2 Operational Systems
  2.1 Small Enterprise Foundation, South Africa
  2.2 Impact Audit, Cetzam, Zambia
  2.3 Impact Monitoring Systems, Workers Bank of Jamaica
  2.4 Progress Tracking, Freedom from Hunger
  2.5 Remeans/Net-worth Test: CARD Philippines/Grameen replicants
  2.6 Annual Impact Tracking System, Enterprise Works World-wide, Eric
     Hyman

3 Participatory Monitoring Systems
  3.1. Internal Learning System, Helzi Noponen
  3.2. Participatory Impact Monitoring and Evaluation System (PIMES), Sinapi
      Aba Trust, Ghana
  3.3 Participatory Monitoring System, Christian Commission for Development
     in Bangladesh (CCDB)

4 Institutional Analysis
  4.1 MicroAssess, PACT, USA
  4.2 Development Audit, EDA, India




                                                                          11
1. Tool Kits

1.1 AIMS Practitioner Tool Kit and Guidelines for Microfinance Impact
Assessments
 http://www.ids.ac.uk/cgap/static/1646.htm
 http://www.ids.ac.uk/cgap/static/1786.htm

Objectives

   Set of five tools that practitioners can use to gather information about their
    programmes – information that is useful for impact assessment and market research.

Outputs

   The five tools provide impact information about client satisfaction, empowerment,
    businesses and impact. The tools are a mix of a quantitative survey and semi-
    structured tools aimed at understanding specific questions.

Description

5 Practitioner tools

    a. Household surveys
    b. Exit survey- a tool allowing MFI to understand why clients are leaving the
       program and to determine what strategies are appropriate to address
       desertion.
    c. Loan/savings use tool– a tool investigating in detail the way of using the
       loan by micro-clients, useful for both IA and MR activities.
    d. Focus group discussions on client satisfaction- a tool assessing client
       satisfaction and identifying ways of product refinement and new product
       development
    e. Focus group discussions on empowerment – a tool explaining complex
       issues of client empowerment due to participation in microfinance
       activities.

Manual on Microfinance impact assessment

    a. A cost effective, useful and credible IA approach
    b. Dual focus on demonstrating the program’s impact and identifying ways to
       improve it that satisfies both the accountability expectations of donors
       and the relevance of managers.
    c. Comprehensive guidance to planning and implementing an IA, including
       adapting tools to specific assessment goals, training staff to apply the
       tools and analysing data using Epi-Info statistical software.
    d. Clear explanations of indicators used, as well, as informative discussions
       of measures tested and rejected by the SEEP team.




                                                                                   12
Application

Key Strengths
 Provide a sound conceptual framework on which impact can be
  understood
 Provide clear thinking of the process of impact assessment and the
  methodological issues that need to be addressed
 Provide a simple set of tools that can be used as a starting point in impact
  assessment by organisations
 Provide some guidance as to how each organisation needs to think
  through the impact assessment process for themselves.

Issues
 The tool-kit is fairly limited in scope, with the bulk of the manual focused
   on the survey tool. This survey is an example of a survey, but it is
   essential that it is adapted to local conditions and objectives in its use.
 The approaches in the qualitative tools can be easily adapted to suit other
   contexts and questions.


1.2 MSA Tool-Kit

Objectives

   Provide MFIs with qualitative market              research    skills    to   increase
    understanding of the needs of their clients.

Outputs

The methods outlined in the tools kit can be used for a number of different applications.
These include, but are not limited to:

   Developing new products and modifying old ones

   Understanding clients      and    their   perceptions    of   the      MFI   and   its
    services/products

   Developing/refining marketing programmes

   Analysis of clients' risks/vulnerability opportunities and how people use
    (formal and informal sector) financial services

   Understanding the "financial landscape", or environment, within which the
    MFI is operating

   Analysing problems such as drop-outs and growing trends loan default

   Impact assessment and evaluation

   Analysis of relative depth of outreach



                                                                                       13
   Detecting fraud/rent-seeking

   Running strategic planning/staff meetings

Description

   Included in the program is a workshop, followed by in-field training and
    practical product development to help staff in implementation and to assist in
    market research activities. The course also provides guidelines on how to
    analyse results to identify common trends and recurring themes and to
    triangulate evidence.

Focus-group discussions

   Preparing for Focus Group Discussions
   Designing a Discussion Guide

Group Moderation

   The Qualities of a Good Moderator
   Getting Started
   Managing the Discussion Process
   Improving Participation

Analysing and Presenting FGD Results

   Techniques of Analysing FGD Results
   Reporting FGD Results

PRA Tool-kit

   Seasonality Analysis of household income, expenditure, savings and
    credit is used to obtain information on seasonal flows of income and
    expenditure, and the demand for credit and savings services. This analysis
    also provides insights into some of the risks and pressures faced by clients
    and how they use MFIs’ financial services to respond to these.

   Seasonality Analysis of migration, casual employment and
    goods/services provided by the poor looks at the availability of cash to the
    people in the community. This has important implications for their ability to
    make regular savings and loan repayments.

   Life-cycle Profile to determine which of the events require lump-sums of
    cash;     to    examine   the    implications    of   these    for   household
    income/expenditure; to establish current coping mechanisms; and then
    finally to discuss how access to MFI financial services can help the household
    respond to these.

   Venn\Chapati         Diagram    allows     analysis of financial service
    groups/organisations within the community and to understand more about
    the social capital accumulated by participants.



                                                                               14
   Simple Ranking can be used to explore a wide variety of issues when an
    understanding of the relative importance/desirability etc. is needed.

   Relative Preference Ranking is used to see how clients and potential
    clients perceive the financial service providers and components of the
    financial services they provide.

   Pair-wise Ranking is used to examine in detail how clients and potential
    clients compare and contrast critical components of financial services, and
    why those elements are important for them.

   Simple Wealth Ranking provides a rapid way of segregating a community
    into three basic categories, and is useful in situations where there are many
    households in a community. This is useful for targeting. This exercise can
    also be useful in impact assessment, and for examining the socio-economic
    characteristics of people who chose to join (or don’t join) the MFI and also
    those who leave or whose accounts become dormant.

   Detailed Wealth Ranking provides an understanding of in what way and
    why rich people are wealthy and the poor are poor, and a ‘ranking’ of the
    households in the village, from the most wealthy to the least wealthy, as
    seen by the members of the community.

   Cash Mobility Mapping provides an understanding of where the community
    goes to acquire or spend cash (markets, waged labour, co-operatives,
    informal financial organisations etc.) and to lead into discussions of which
    financial service institutions they trust or value and why.

   Time Series of sickness, death, loss of employment, theft, natural
    disaster etc. (this year, last year, 5 and 10 years before) provides an
    opportunity to learn from the community about how it views change overtime
    in various areas related to a series of crisis.

   Time Series of asset ownership is useful in determining what “productive”
    and “protective” assets are valued the most and thus the potential for
    designing or refining corresponding financial products including leasing,
    contractual savings deposits (e.g for housing, education, health insurance
    etc).

   Financial Services Matrix is useful in determining which financial services
    are used by which socio-economic or socio-cultural strata of society and why,
    and thus the potential for designing or refining appropriate financial products.

   Financial Sector Trend Analysis is useful in determining which financial
    services have been used over time by which socio-economic or socio-cultural
    strata of society, and thus for understanding the changes in the
    use/availability of a variety of financial services over time.




                                                                                 15
Application

Key Strengths
 The MSA tool-kit is a very useful adaptation of widely used qualitative and
  participatory research methods to a microfinance market research
  context.

Issues
 It is important to recognise in the use of these tools that they are generic
   approaches that can be used creatively, and adapted and applied a wide
   range of research questions.
 MSA places great emphasis on the quality of training in the use of these
   tools, and this is essential to ensure that credible results are produced
   from their application.
 It may necessary to combine these tools with other methods, and
   certainly with a clear conceptual framework to use these tools for impact
   assessment rather than market research.

1.3 IKM Framework, PlaNet Finance

Objectives

   The IKM is a manual which outlines a process whereby organisations can
    develop a set of techniques, combining impact assessment and market
    research into a client-centred monitoring system useful for institutional
    management.

Outputs

   The IKM tools and techniques used depend strongly on the context of the
    organisation, and the information requirements that are identified. It is
    widely acknowledged that it is important to use multiple methods and
    triangulation in client level assessments. That is why IKM proposes to its
    clients a shopping list of quantitative, qualitative and Participatory
    Rapid Appraisal (PRA) tools that could be applied within the MFI with
    regard to its needs. The tools tested during IKM activities can be then
    incorporated into the Client-Focused Learning System.

Description

The IKM toolkit consists of two main kinds of tools: 1) supporting the planning
and design stage and 2) assessment tools.

Tools supporting the planning and design stage

1) Base Questionnaire – is a standard questionnaire sent to MFI before the
   planning and design stage. It collects background institutional information as
   well as consist of open-ended questions about needs, problems and plans of
   the institution. The IKM Base Questionnaire allows the IKM consultant to start
   participatory planning and design with essential knowledge about the
   institution.


                                                                                 16
2) Institution Form – during the planning and design stage more
   comprehensive information about the institution is collected. The IKM
   consultant follows this semi-structured questionnaire based on the GIRAFE
   rating methodology to gather structural data, historical performance data,
   management styles and practices, the services provided financial outcomes
   over at least the past three years. Then his task is to identify burning areas
   that need further investigation on the client level.
3) Context Appraisal Forms – these forms are semi-structured questionnaires
   with open-ended questions providing the IKM consultant with fundamental
   knowledge about the area of intervention. They support the IKM consultant in
   terms of understanding local socio-economic context, local criteria of poverty
   appraisal, recent events in the community and local political and juridical
   environment. They are useful for planning and design stage as well as for
   data analysis.
4) Market Supply Form – this form investigates the current situation of supply
   of microfinance services on the local market. It is an indispensable tool for
   MR activities.

Assessment Tools

The IKM assessment tools can be divided into 3 main categories:

1) Quantitative sample surveys

   Impact Survey (two versions simple and extended) – instrument assessing
    the impact of MFI on their clients.
 Client Household Survey - gathering more in-depth data on some of the
    key variables useful both for IA and MR.
A. Marketing Survey - instrument allowing micro-finance practitioners to get to
   know their clientele thus improving their products and performance. This tool
   assesses client satisfaction, client needs and the ways of using the loan.
B. Market Demand Survey – instrument allowing to get to know better the
   demand for microfinance services of both existing and potential clients of the
   MFI.
C. Drop-outs Monitoring Tool – instrument allowing to monitor the clients who
   leave the program (what groups are leaving, what are the reasons, etc.)

2) Qualitative tools based on focus groups and semi-structured
   interviews

3) Participatory Rapid Appraisal (PRA) toolkit (MSA tool-kit)

4) Other data sources: existing MIS; client database; interviews with
   loan officers




                                                                              17
Application

Key Strengths
 The strength of the framework is in its contextualisation of impact
  assessment within a broader organisational framework, and the discussion
  of the planning and design stages, and the overall impact assessment
  process.

Issues
 The assessment tools presented in the IKM are largely based on the AIMS
   practitioner tools and the MSA tool-kit, thus the IKM does not offer much
   in terms of specific tools.
 This is a useful process resource.



2. Operational Systems

2.1 Small Enterprise Foundation, South Africa
 http://www.ids.ac.uk/cgap/static/517.htm

Objectives

SEF developed its impact assessment system to increase understanding of the
factors that lead to improvements or decline of members’ livelihoods, and
thereby to improve the practice of the organisation.

The intention was to develop an internal learning process involving clients and
staff at all level which can highlight successes and problems and lessons learned
and improve practice. The system therefore was designed to yield useful results
rapidly, although it has both short term and longer-term components. It is
designed to be implemented, analysed and used by SEF Operational and
Development staff, without the requirement for significant outside support, and
thereby fit within a resource constrained context.

Outputs

SEF''s system integrates both impact monitoring (IM) and assessment (IA). IA
provides the basis of understanding and the wider picture, but an occasional
“snapshot” is not sufficient to ensure that impact is managed not just assessed.
IM provides simple, quick information which can improve learning and
understanding, but, integrated into the MIS provides management with a tool for
improving impact.

Impact monitoring thus serves two goals:
1) To supplement the occasional nature of impact assessment with continuous
data, thus strengthening the impact assessment
2) To provide immediate operational benefit through indicators of problems
which may lead to reduced impact, or of success. These are built into the MIS at
the level of client, group, centre, loan officer, branch or zonal manager




                                                                               18
Description

Livelihood Case-studies

Detailed qualitative interviews with a sample of 60 clients, interviewed 9-
monthly, develops an understanding of the role of credit and savings in
household economies. Recall methods examine changes since joining the MFI.
Discussions using visual participatory methods explore significant variables that
impact on member's households and their businesses.

Impact Monitoring and MIS System

Existing MIS data: savings amounts/regularity; repayment regularity; meeting
attendance; business value; loan size.

Information collected from clients on each loan application: A facilitated
discussion between client and loan officer allows for self-analysis and
assessment of client vulnerability by loan officer. Quantitative data collected on
following indicators: relative income/expenditure; housing; food; education;
informal savings; business diversification

Information collected from centres periodically:            Other group-based
opportunities used to gather client satisfaction and empowerment information.

Market research tools

Drop-out monitoring tool: This tool takes a different approach to the AIMS client
exit tool, and attempts to understand underlying rather than superficial reasons
for drop-outs. Clients are not asked directly why they left the programme,
rather a focus group meeting followed by individual interviews explores four
areas of the clients life and interaction with the programme (satisfaction with
MFI services; group/centre relationships; business success; personal issues). It
is assumed that reasons for drop-out will relate to one or more of these areas.

Business profiles: This tool explores the dynamics of particular business types
including: market; sales pattern over time (daily, weekly, monthly, annually);
business location and suitability to the environment; stock and turn-over
patterns; profitability; marketing strategies; business combinations; support
needed for the business success; strengths, weaknesses and vulnerability of
business type.

In-depth research

Research is conducted into issues in response to issues arising from operations
and from the impact assessment and monitoring work. Such studies have
included: Savings and Client financial transaction costs.




                                                                               19
Application

Key Strengths
 The SEF system is a good example of an system developed for internal
  organisational needs, that uses existing information and triangulation to
  provide credible information.
 SEF’s system combines in-depth research, quantitative analysis of MIS
  data, and qualitative market research-type work., and the detailed
  conceptual and livelihood case-studies provide the understanding that
  allows the simple monitoring indicators to be interpreted and effectively
  used.

Issues
 A key issue in the systems application and usefulness is the need to
   change the culture of the organisation and interaction with clients, to one
   where vulnerability and impact is a central focus. Clients and staff at all
   levels are involved.
 Although this system is relatively low cost in terms of staff time and out-
   side inputs, there is a need for a strong institutional commitment and for
   diligent implementation and monitoring for it to be successful.


2.2 Impact Audit, Cetzam, Zambia
 http://www.ids.ac.uk/cgap/static/1836.htm

Objectives

The Impact Audit system used by CETZAM for assessing its services is primarily
client centric. It is aimed at improving existing services to clients and also
towards being competitive in the market and ultimately achieve better outreach
as well as sustainability.

It helps managers in decision-making by providing periodic information on client
experiences and perspectives about the services, especially about those that
influence client decisions in continuing to use the services. This feeds back as
useful learning for the growth of the organisation.

Outputs

The system provides information on the following aspects that helps the
management in making client focussed decisions:
1. Assessment outreach by type of clientele and its match with targets.
2. Comparative information about clients through different periods of time and
   through trend analysis.
3. An understanding of client perspectives depending on client profiles,
   performance of loan officer, business sector, etc.
4. Assessment of goal achievement, i.e. employment generation and socio-
   economic progress of clients.




                                                                                 20
Description

Impact Audit is integrated with existing information systems of the institution as
much as possible, to make it an ongoing feature as well as a part of the
standard functioning of the institution.       Information is collected largely
quantitative though some qualitative data is also gathered at the routine
meetings. The methods used are:

1. Means test - a tool which determines the potential client profile through
   socio-economic and demographic details. It helps in ranking poverty levels
   of the prospective clients and in assessing their eligibility for the services.

2. Impact Survey - compares existing one-year old clients with a newly
   enrolled clients.

3. Exit survey - builds on the impact survey and explores in depth into client
   satisfaction to understand how many clients exit, when and why.

4. Qualitative methods - case study interviews record basic changes in
   household circumstances, business activities, experiences of the services
   through open ended generative questions.


Application

Key Strengths
 The Cetzam system combines a number of different mechanisms to
  effectively use opportunities to gather information and to increase the
  credibility of results through triangulation.
 It provides a good example of an integrated system that can provide
  useful information to managers and also has the potential to provide
  credible impact information. It is a fairly costly system that demands
  expertise, particularly in the analysis, but this would be no more
  demanding or costly than the AIMS impact survey.

Issues
 Staff are involved in the IA process, but this is not necessarily a primary
   focus of the system in the way it is at SEF – a separate impact monitoring
   officer collects most of the data rather than loan officers.


2.3 Impact monitoring systems, Workers Bank of Jamaica
http://www.ids.ac.uk/cgap/static/25.htm

Objectives

The Client Profiling System (CPS) - a computer based client monitoring system
that is integrated into the loan application and review process. It develops a
database that provides information about impact on clients at the business as
well as household level for purposes of assessing objectives achievement and
progress of an organisation.



                                                                                21
The system is cost-effective and a reliable process that reports on the impact of
client participation in the programme by tracking changes on some of the key
indicators.

Outputs

The system provides for a systematic monitoring of the client profiles and
progress at the enterprise level. It provides quick answers management
questions. It also enables to process applications on an ongoing basis, takes in
account the varying duration of loans and allows for time between each
measurement point to assess change or progress.

Description

This is an impact monitoring system that is incorporated into the loan application
system. Clients apply for a loan through an interview with a loan officer, with the
loan officer entering the data directly into a computer. The data requested in the
interview includes information needed to assess the loan as well as information
on the client's income, assets, patterns of expenditure, etc. The entire interview
process takes about 20 minutes. When clients reapply for loans, the data is
collected again. This allows for tracking changes in these indicators over the
time that the clients borrow from the Workers Bank.

System development:

The monitoring system is developed through three stages.

   1. Firstly it involves discussions with all stakeholders to develop preliminary
      monitoring indicators.
   2. This is followed by the second stage of refining of indicators through
      further discussions, a review of existing data collection procedures of the
      institution, and a final assessment of indicators through a comparative
      examination of international experience to establish their robustness.
   3. The last stage involves the establishment of the monitoring system which
      includes revision of the loan application forms, installation of the database
      system, developing data analysis plans and standardisation of indicators.

The monitoring system generates details on the following indicators:

1. Household Indicators: provides information on the economic status of clients
   using income levels, household assets including durable goods, savings and
   fixed assets and finally expenditure on food. Details of household savings are
   not included in the set of monitoring indicators but the value is used as an
   indirect measure of household welfare.

2. Enterprise Indicators: provides information on enterprise performance by
   looking at sales turnover, assets and personnel employed.

3. Relationship of Indicators with objective: the quantum and frequency of loans
   disbursed by gender and the creation of new opportunities for women show
   the level of achievement of the objective.



                                                                                22
Methodology and methods:

1. A fixed response format questionnaire is used to elicit information about the
   total household income and the income range that clients can be categorised
   into.
2. The World Bank Living Standards Measurements Survey questionnaire format
   is used to identify durable goods owned by the family to allow comparisons of
   results.
3. Data on household savings is taken from accounts with financial institutions,
   bonds and certificates of deposits, home savings and informal savings
   programme.
4. Data on food expenditure is collected through the loan application form and
   results compared with the World Bank Living Standards Measurements
   Survey data.
5. Details on sales value is collected for various periods (high, low and medium)
   and then annualised.
6. Information on fixed assets of the enterprise is taken by listing assets and
   then assessing current value taking into account the depreciation.
7. The USAID – National Survey of Small and Microenterprises questionnaire
   format is used to collect employment details which are then disaggregated by
   ownership, paid workers, etc., to look at gender sensitivity.

Application

Key Strengths
 The system has good use of existing information sources and is integrated
  into existing systems. The type of data collected are mostly quantitative,
  and the monitoring system is essentially a simplified survey in terms of its
  design and application. Good use if made of existing information sources
  to make comparisons with other poverty and enterprise survey data.
 Storing information in computer database allows for a wide variety of
  reports comparisons between various input factors (loan size, business
  type subsequent impact (changes in assets, education, health, etc.))
 Good use of existing information and systems
 Comparison to existing data sources and surveys to give reference points

Issues
 This is a system designed to provide client and enterprise level data.
   Data is collected by loan officers as part of the loan application process,
   but the sense is that this is primarily a data collection exercise, with little
   focus on discussion and learning between client and loan officer.

2.4 Progress Tracking, Freedom from Hunger
 info@freedomfromhunger.org

Objectives

Developed by Freedom from Hunger, Progress Tracking aims to provide impact
information for purposes of internal proving and improving impact.

Progress Tracking is a performance monitoring system to help provide managers
and field staff timely feedback on the successful attainment of goals through


                                                                                     23
information on “poverty levels of clients, the quality and appropriateness of
services offered, staff and institutional performance and progress towards social
impact of goals.

    Multiple stakeholders but primarily for internal use
    Analysis of programme activities for decision making
    Routine system
    By building on and rationalising existing systems should not increase staff
     work-load or costs

Outputs

Progress Tracking provides managers with timely evidence of:

1)   Sound financial performance of product units (branches)
2)   Quality of field-agent performance, especially facilitation of learning sessions
3)   Client satisfaction and loyalty
4)   Penetration of the potential market and depth of outreach
5)   Progress towards social impact.

Description

Progress Tracking uses cost-effective methods that can be directly embedded
into the programme’s regular operations, such as sampling, simple dichotomous
‘yes/no’ indicators, and non-formal ranking and rating exercises and uses
existing opportunities to collect, analyse, disseminate and incorporate learning.

Methodologies include:

1. Financial Performance

2. Depth of Outreach
     a. Client Poverty Assessment Tool –short survey of randomly selected
        clients completed by field agents to better profile clients vulnerability
        and experience and assign a relative Poverty Score.

3. Supervision and Service Quality
     a. Spot Audits – unannounced visits of filed agents’ Credit Associations
        are conducted.
     b. Education Facilitation Assessment - unannounced visits to ensure
        quality

4. Impacts
     a. Mini-Surveys - to small number of clients during visits to Credit
        Associations to assess member knowledge, practice and information
        sharing. (May adapt Lot Quality Assurance Samplings (LQAS)
        technique)
     b. Guides for Qualitative Interviews with Clients – in-depth
        interview guides with a small number of randomly selected clients and
        followed with annual or semi-annual follow-up interviews. These
        interviews will be shared at regular monthly staff meetings.



                                                                                   24
5. End of Cycle PLA Exercises
     a. Participatory Learning and Action (PLA) exercises conducted by
        field agents with members toward the end of the loan cycle to
        determine client satisfaction and gain feedback about trends,
        education topics, and other areas of improvement.
     b. Food Security, Vulnerability and Coping Strategies - to identify
        seasonal and periodic factors. (Similar to a research conducted by
        MicroSave)
     c. Child Health and Nutrition – meet with women to try and determine
        child health and nutrition over the past (5 years) and try and
        determine positive and negative trends in the community.

6. Client Satisfaction
      a. Client Satisfaction Group Discussion
      b. Drop-Out Inquiry

Application

Key Strengths
 This is a comprehensive impact monitoring system that has the potential
  to provide a lot of quite in-depth knowledge about clients and impact.
 The system uses existing information and is built into existing work
  patterns and is therefore potentially integrated into the organisation.
 Use of participatory methods and other forms of interaction between staff
  and clients incorporates a learning agenda into the system.

Issues
 The system is still in its pilot phase and there are a lot of issues which will
   need to be resolved in terms of the ease of its application – time
   requirements; data processing and analysis; staff skills; usefulness and
   use of information produced.

   NB stress need for specific management orientation – “Field-Agent
    Centred Management”, to send right signals to field staff, and to facilitate
    internal learning processes.

2.5 Remeans/Net-worth Test: CARD Philippines/Grameen replicants

Objectives and Outputs

As part of the poverty-targeting methodology of CARD and other Grameen
replicants a Remeans Test is administered with all potential clients. This looks at
members’ assets, income and housing. CARD repeats this test with clients on
their third loan cycle to track changes in members situation.

Description

10 to 15 minute interview is conducted by the loan officer to assess the eligibility
of potential clients to join the programme. This focuses on the value of their
productive assets. These assets include agricultural land owned and/or operated,
farm equipment (including buildings) and machinery, large farm animals,
transport vehicles, and stocks of goods for sale, etc. The total value of


                                                                                    25
household assets is estimated, and compared with the locally relevant cut-off
that distinguishes non-poor from poor households and the poor from the
poorest.

In-house tracking of institutional and program performance is done through the
Operations Update. Among the indicators/ratios being tracked are the
following:

Application

Key Strengths
 This is a simple system for collecting basic base-line information about all
  clients as part of normal operations, and then looking for changes in the
  situation of clients over time.

Issues
 The system has not been formalised for impact assessment. Useful base-
   line information is collected which gives scope for assessing change over
   time. There is a need, however, to strengthen the base-line information
   collected, the systems for maintaining base-line information, the rigour of
   the repeat test, and the analysis of information produced.
 It is also important to determine how often the test should be repeated
   and if it should be with all clients. Also it is important that it is conducted
   with drop-outs as well as continuing clients, otherwise there will be a very
   distorted picture.

Note CARD has stopped using the Remeans test in favour of AIMS tools.

2.6 Annual Impact Tracking System, Enterprise Works Worldwide, Eric
Hyman
 (hymane@enterpriseworks.org)

Objectives

Annual Impact Training System (ITS) aims to provide timely information on
impact, using a standard set of indicators, for use with projects providing small
and micro-business development services.            It helps improve internal
management systems and also outreach.

Outputs

The computerised tracking system enables comparison and aggregation of the
following indicators across projects, programmes, geographic regions and the
portfolio as a whole –

   outreach and impact,
   sustainability of impact,
   funding and diversification,
   cost effectiveness

It is also possible to monitor projects for impacts on the environment and
natural resource base.


                                                                                     26
Description

The following methods and sources are used to collate information and assess
impact:

      a. Regular financial and progress reports are used to generate
         information on economic impact.
      b. Quantitative data is used primarily but some qualitative data is
         gathered.
      c. Longitudinal data on outreach to analyse impact across time.
      d. Focuses mainly on clients but also looks at groups of non-clients to
         understand reasons for non-participation
      e. Methods and frequency of data collection are reconsidered and revised
         yearly.
      f. Compilation of the impact information in a computerised database.
      g. Regional spot checks and review of filed work submissions
      h. Cost-benefit analysis (NPV, IRR)

The system focuses on clients with annual information collection. There is some
assessment of impact on certain groups of non-clients, but this does not take
place on an annual basis.


The indicators used of assessment include:

Outreach - focuses in clients and also in the indirect effect on associated
members and therefore looks at participants in the enterprise (economic
participants, producer participants and consumer beneficiaries).

Economic Impact - total monetary benefits are calculated by including net
producer income gains and producer cost savings.

Sustainability - of outreach and impact is judged by private sector investment
by enterprises for fixed or working capital and cost recovery from assisted
enterprises.

Funding Diversification - uses different measures of funding like co-operative
agreement funding, direct funding, instrumental leveraging, etc. to assess
diversification.

Cost Effectiveness - Uses cost-effectiveness ratios based on previous years to
compare cumulative benefits to cumulative costs and cost benefit is calculated
using net present value and internal rate of return.




                                                                            27
Application

Key Strengths
 Primarily a programmatic monitoring system, but with applications for
  individual organisations.
 Well defined indicators.

Issues
 The ITS has been implemented with standard indicators for all projects,
   but it could be adapted to suit the differing needs of individual
   organisations.
 This is a system for information management and analysis, and is flexible
   enough to allow input of data from different sources. The tools for
   gathering data are not prescribed in the system.

3.   Participatory monitoring systems

3.1 Internal Learning System, Helzi Noponen
 http://www.ids.ac.uk/cgap/static/1501.htm

Objectives

ILS was designed to assist poor women borrowers and NGO staff to think
critically about micro-finance program impact and processes and make strategic
changes in a decentralised, participatory manner.

Outputs

Participants use the system at each organisational level in a development
program, from the poor participant, to participant groups, field officers, and
programme managers. ILS users at each level, especially poor women
borrowers, are the first to learn about program impact and performance, and
alter plans as a result. They are not only data gatherers but they are also the
data analysts, planners, documenters, and trainers.

The medium of the system is the ILS pictorial diary. It is in many ways a survey
questionnaire in pictorial format, which participants fill out on their own. The
process of filling in the diary generates self-analysis, discussion and learning.

The system is thus internal, decentralised, integrated monitoring, evaluation,
management and training system that can be used for monitoring and assessing
client level changes, and for decision making at all levels, including clients, field
workers, and donors.

ILS has been successfully field tested among five different NGOs in India.
Participants, including very poor and illiterate women, are able to keep an
impact diary, recording changes in their situation over time on programme
impact and process. The use of ILS diaries by very poor and illiterate members,
in particular, seems to have a strong catalytic effect of increasing women’s
confidence and motivation to improve their situation. Women have also used the


                                                                                  28
diary to analyze their situation, set priorities and make future plans. NGO staff
have also benefited from learning about their members’ lives.

Description

The Integrated Learning System (ILS) is based on a pictorial diary compiled by
the clients, who feed the information into a summary diary kept by the
microcredit self-help group, then in turn to the area organiser, the programme
manager, and finally the intermediary provider of funds. At each level, the
participants collect data, assess change, analyze the reasons for change, alter
their strategies based on their learning, and document progress and share
learning.

The structure is flexible so that different elements can be implemented at
different levels depending on the objectives of the MFI, their commitment and
the resources available.

Sample Questions:
          Have I improved my shelter?
          Have I improved my production processes?
          Have I improved access to health care?
          Am I satisfied with my conditions?
          (asked through accompanying pictures)

The ILS can be adapted by using the Stakeholder Analysis Manual to determine
appropriate domains of inquiry and indicators, and then hiring a local artist to
adapt the drawings. The implementation cost of the ILS is essentially in the staff
training and processing time, transportation costs to travel to training sessions,
and the cost of printing or photocopying diaries and manuals. (Three-year client
diaries cost about 40¢ each). Because this is an integrated system, the essential
requirement is commitment by the NGO to staff involvement at all levels.




                                                                               29
Application

Key Strengths
 An integrated system that is incorporated into the ongoing data collection
  work of
 clients, loan officers, and managers
 Provides useful information to all decision-makers in the process,
  including clients
 Based on pictures, so it can be used with illiterate clients

Issues in application
 Quite complex and potentially costly in terms of time. Needs a lot of
   commitment, but great potential benefits in terms of client empowerment
   and MFI understanding.
 It is still a system in development and has yet to be fully implemented in
   an MFI.
 If the MFI is not interested in “improving” – finding and fixing poor
   performance – or in building client understanding and problem-solving
   skills, then doing ILS is a waste of time, resources and staff effort. A one-
   time, external impact study is a more appropriate option if the goal of the
   evaluation is instead to “prove”.
 The MFI must also have a culture of “organisational learning” unafraid of
   examining problems and poor performance in a constructive manner and
   finding solutions. ILS is wasted if the NGO managers are intolerant of
   mistakes and lagging results.

3.2. Participatory Impact Monitoring and Evaluation System (PIMES),
Sinapi Aba Trust, Ghana

Objectives

The Participatory Impact Monitoring and Evaluation System (PIMES) aims to
assess project outputs and measure impact on a systematic and continuous
basis.  It is primarily a tool to track changes and assess organisational
performance and the element of participation attempts to bring in ownership and
responsibility towards the project from all stakeholders.

Outputs

The system is useful for managers as well as the beneficiaries as it provides
easily accessible and reliable information through which:
 The progress of activities being implemented can be tracked
 Problems within the project and any deviations from objectives can be
   detected
 The relevance, efficiency and effectiveness of interventions be determined
   systematically
 Future development can be planned

Description




                                                                                   30
The system uses a methodology, which promote interaction between
stakeholders and integrates both quantitative and qualitative techniques for the
process.

The General framework of the System:

1. Identification of activities - Areas where intervention contributes best to
   poverty alleviation viz. savings mobilisation, credit delivery, training and
   capacity building, etc. are promoted by improving outreach, fund allocations,
   performance standards etc. by developing partnerships, interaction and
   communication between stakeholders.
2. Development of indicators - The direct benefits of the activities (outputs) lead
   to measures of impact and the indicators which bear relevance to
   transformation within context of poverty alleviation are carefully delineated
   and categorised.       Indicators include socio-economic and demographic
   information of beneficiaries, institutional efficiency, sustainability of
   programmes and participation components.
3. Data requirements - Relevant quantifiable information provides details of
   quantum of credit, quantity and type of institutional inputs, enterprise related
   issues of productivity, sustainability etc. which are coded and a scoring
   system is used to assess and compare levels.
4. Methods of data collection - Quantitative methods include questionnaires,
   data sheets which, are administered at institutional and community level for
   information on enterprise performance.          Qualitative methods like key
   informant interviews, focus group discussions, etc. are used to assess levels
   of participation, social relations, self esteem, gender equity, empowerment
   etc.
5. Agents for data collection - Different categories of members undertake data
   collection at each level depending on their skills and responsibilities.

The Operationalisation of the system:

The initial planning exercises focus on the beneficiaries and include:

1. Community Animation - sensitising the community to project objectives,
2. Group Orientation - training programmes along with group activities to
   enable group formation.
3. Project Design - development of local project package as well as project
   specific evaluation systems relevant to the local environment which is
   reviewed and agreed on by all stake holders.

The data collection mechanism is then established by:

1. Data definition and generation - the categories of data are defined to include
   existing situation (demographic, socio-economic details of beneficiaries,
   current skills and performance levels of enterprises) and the expected
   changes with respect to outputs and impacts (enterprise performance,
   outputs, sustainability etc.). Details of institutional processes are also
   included.
2. Data generation - quantitative and qualitative methods employed as detailed
   above.



                                                                                31
The data analysis follow a 'before - after' framework to assess impact at all the
levels - the beneficiaries, the community and the institution.

The results are utilised to find tune decision making and implementation
processes across levels and are communicated through meetings, directives,
workshops, circulars, etc.

Application

Key Strengths
 Participatory system involving clients and staff in analysing progress, and
  developing improvements.
 Develops an on-going information and organisational learning system
 Combines quantitative and qualitative data
 Has the potential to develop detailed credible impact information

Issues
 PIMES is still in the development and piloting stages. It is not yet clear
   how easy it is to implement or effective it is in generating credible
   information.

3.3. Participatory Monitoring System, Christian Commission for
Development in Bangladesh (CCDB)

Objectives

The Participatory Monitoring System implemented by CCDB was an attempt to
identify and assess differences of perceptions and approaches within
organisation and between organisation and beneficiaries. CCDB is a NGO that
offers development assistance through its Peoples Participatory Development
Program (PPRDP). They offer three forms of assistance: group savings and
credit facilities, grant assistance, and skills training.

Outputs

The PMS is helpful in following ways:

   The rating scale provides insight into differences between types of
    participation within the organisation.
   Provides knowledge of diversity of perceptions
   Reports back significant changes that occur at all levels
   Allows revision of the monitoring system.

Description

PMS does not use “indicators” in the traditional sense. The “indicatorless”
reporting approach involves staff who are requested to report the most
significant changes over the last period and explain why they have chose these.

The implemented PMS program consists of 9 steps:




                                                                                32
          a. The selection of domains of change to be monitored (people’s lives,
             participation, and sustainability of institutions and their activities)
          b. The reporting period – where monthly changes were recorded
          c. The selection of participants - includes beneficiaries, different
             categories of project staff and donors.
          d. Phrasing the question – obtaining descriptive and explanatory
             answers.
          e. The structure of participation – workshop to plan the
             implementation       of    the     monitoring      system.      (iterated
             variation/evolutionary algorithm)
          f. Feedback - within the institution for development of dialogue.
          g. Verification - to ensure authenticity of information

Optional steps:

          h. Quantification – the extent to which changes identifies as most
             significant in one zone have taken place in other locations.
          i. Monitoring the monitoring system

Local interviews or “stories” used to add local context to collected data.

Application

Key Strengths
 Participatory methodology is useful in eliciting
 Useful for strengthening communication within institutions and with
  beneficiaries.

Issues
 It is unclear how PMS can be used for assessing socio-economic impact on
   beneficiaries

4. Institutional analysis

4.1. MicroAssess, PACT, USA

Objectives

MicroAssess is a cost-effective assessment tool that measures socio-economic
and institutional impact of micro-finance programmes. The objective of the tool
is basically targeted towards institutions to help them complement financial
sustainability assessments of programmes with institutional sustainability
measures and thus strengthen institutional commitments towards poverty
reduction.

The tool uses a combination of techniques to judge institutional performance
across three levels – the household level to see changes in poverty indicators,
the group level to assess development of communities and at the institutional
level to gauge sustainability.

Outputs



                                                                                   33
It provides institutions with –
     Quantifiable and measurable indicators of institutional development
     Information which helps in overcoming operational problems
     Standardised and comparable measures of institutional performance
     Pointers for sustainable development
     A knowledge of client needs
     Leverage in negotiations with governments, donors and partners

Description

MicroAssess is essentially a participatory process that is facilitated by the
institution but involves all stakeholders. The various components involved in the
process are:

1. Assessment and planning:
     a. Self Assessment – Developed from the Organisational Capacity
        Assessment (OCA) methodology and tries to show a link between
        financial and institutional indicators in an objective and accurate
        manner. This allows evaluation of institutional progress individually
        and comparatively. Comparability of results is possible with the use of
        master tool that is developed in consultation with other similar
        institutions and after extensive review to test its applicability and
        acceptance.      This master tool standardises core capacities of
        institutions. In addition to this, it is uses self assessment methods and
        a standardised review of portfolio quality which are combined to
        develop an Institutional Sustainability rating for the institution.

      b. Community Assessment – The self assessment procedure is extended
         to client-managed cohorts to help in monitoring change and progress
         at the individual client level. This also incorporates the element of self-
         learning for the clients about their own capacities and growth.

2. Training and technical assistance:
This includes programmes to mentor and train institutional stakeholders for
improving existing internal management practices and external relations with
clients. It also helps develop a comprehensive computerised management
information system which assists in co-ordination of institutional sustainability
indicators with financial sustainability indicators. The tool therefore integrates
the assessment process effectively into the regular functions of the institution.

3. Participatory Monitoring, Evaluation and Reporting:
A combination of the Institutional Sustainability rating, client impact monitoring,
portfolio review with participatory methods of regular monitoring of a sample of
clients envisages client-focussed and client-designed services.

4. Network Promotion:
The standardised measures contribute to adoption of standards and codes of
conduct across institutions. It also helps development of partnerships with the
private sector as the tool helps in establishing credibility of institutions.




                                                                                 34
Assessment

Key Strengths
 MicroAssess is very comprehensive and inclusive
 The participatory nature of the process gives good opportunity for
  personal and organisational learning

Issues
 The assessment process is likely to be quite challenging to the
   organisation, and particularly to management. There needs to be a
   commitment to seeing the process through and acting on the results.
 Cost and time commitment
 Impact data is not central to the assessment, but one of many areas. It
   may therefore be difficult to produce credible impact data for outside
   stakeholders from this assessment, and it is likely that it would have to be
   combined with other methods.
 This is a system which is still under development



4. 2. Development Audit, EDA, India

Objectives

Development Audit is aimed at improving development planning and increasing
stakeholder accountability. It does this by engaging both 'strong' and 'weak'
stakeholders in a dialogue and thereby contributing towards the assessment of
institutional objectives. The tool is an adaptation of the Social Audit exercises
undertaken for the corporate sector.

The tool is designed in such a manner that it ensures audit quality, includes
views of weaker stakeholders, generates reliable results which are again
communicated and verified with all the stakeholders. The responsibility of
undertaking this auditing process is entirely on the organisation which
undergoes it and is therefore not donor driven.

Outputs

Development Audit provides reliable facts about programmes that are owned by
all stakeholders and the output of the process is that it brings about:
1. Clarity and sharing of objectives by different stakeholders
2. Greater awareness of various effects of the programme
3. Increased sense of responsibility
4. Greater gender awareness
5. Better and improved data collection and recording systems
6. Transparency in accounts
7. Joint target setting by all stakeholders

Description

Development Audit requires involvement of all stakeholders and is facilitated by
the institution. It involves a series of steps in the regular auditing cycle:


                                                                                  35
1. Scoping study - involving the development and planning of the auditing
   process.

2. Programme description - by reviewing records, activities and outreach of the
   institution

3. Stakeholder identification - which involves identifying stakeholder groups at
   all levels including the target group, those implementing the programme,
   women, funders/supporters, wider society, policy makers, etc.

4. Consultations - with each group that is identified to understand their
   expectations and priorities

5. Setting Indicators - appropriate indicators for each stakeholder group are set
   through group discussions

6. Information collection - through simple 'before' and 'after' questions which
   are quantifiable

7. Target setting - to measure change measured on each indicator then
   compared with the target for that indicator which all the stakeholders
   considered 'achievable' through the programme. This is then quantified
   through a scoring system.

8. Effectiveness scoring - reflects the priorities of stakeholders and compare
   effectiveness of programme along different parameters.

9. Costs review - compares programme costs with the effects and the findings
   are discussed jointly by all stakeholders

10.Dissemination - of the findings and their assessment for future action.

Methods used through out these steps include PRA, Group Discussions and
Questionnaires. It also includes a cost benefit analysis.


Application

Key Strengths
 Involving weaker stakeholders in the process and analysis

Issues
 Need for organisational commitment and potential for process to highlight
   tensions between stakeholders
 System is not set up to provide detailed impact information, but could be
   easily combined with other methods to give credible impact information.
 Importance of stakeholder process in defining indicators and desirable
   impact targets.




                                                                              36

								
To top