Docstoc

Module 4

Document Sample
Module 4 Powered By Docstoc
					                                                                                                     Who has seen the wind?
Module 4: Your Indicators                                                                              Neither you nor I.
                                                                                                But when the trees bow their heads,
                                                                                                 We know the wind is passing by.

                                                                                                       -Written by the poet
                                                                                                    Christina Georgina Rossetti


Overview

   Consider your Evaluation questions, and audiences, so that you         Choose at least one indicator for each objective.
    can develop and choose indicators that best address the questions      Make sure the indicators you select, adapt or create are valid
    you have.                                                               (relevant to the program’s local context), reliable (understood the
   Create locally relevant indicators for your program objectives          same way by different people), and feasible (relatively easy and
    (what does your objective- or change you are aiming for- look           affordable to measure).
    like in your setting?)                                                 Understand what a Logical Framework (Log Frame) is, and start
         o Consider existing international SRHR indicators.                 to use it.
   Decide which indicators will be numerical and which will be
    non-numerical.




                                                                                                    STEPS    Module 4: Your Indicators   Page 1
Table of Contents

Key Concepts                                                                 Worksheet: Selecting the most Relevant Denominator

Step 1: Focusing your M&E – Activities and Objectives                 Task 4: Factors that can Influence your Participant Outcomes
                                                                      Task 5: Write your Indicators
Step 2: Developing Locally Relevant Indicators                                Worksheet: Selecting the Most Relevant Indicator
Task 1: What do you see? Find out what your objective looks like in   Step 3: Creating Your Logical Framework
        your program's setting                                        Task 1: Fill in Columns 1 and 2 of the Logical Framework
Task 2: What would success look like in your setting?                         Worksheet: Logical Framework
        Group Exercise: What would success look like?                 Summary
        Worksheet A: Indicators                                       Tips
Task 3: Your Evaluation Needs and Your Indicators: Numerical,         References
        non-numerical, or both                                        Log Frame




                                                                                                 STEPS    Module 4: Your Indicators   Page 2
Key Concepts

Components of Complex Social Concepts (CSC) - measurable
aspects of the complex, multi-faceted social concepts that the         Locally Relevant Indicators – measures that capture the way
objectives of most SRHR programs target. Some of the examples are      behaviors and concepts related to SRHR look in the local setting.
healthy sexuality, human rights, gender equity, quality of care, and
women’s empowerment.                                                   Monitoring Indicators – tell you how the program was actually
                                                                       implemented. They are reflections of the activities you identified.
Evaluation Indicators – measure the results identified in your
Causal Pathway. You will always measure immediate results, but         Reliable – stable so that if different people measure them or the
whether you measure intermediate and final results depends on your     same person does it at different times close together, you get the
level of resources. For more details, see Tips: Deciding How           same result. Your indicators need to make sense and be understood
Far to Go.                                                             the same way by the program’s population and stakeholders.

Feasible – relatively easy to measure in the local setting.            Valid – accurate representations (or reflections) of the concept you
                                                                       want to measure. Since the concepts and behaviors related to SRHR
Indicators – measures of concepts, events or behaviors. They are not   programs look different in different settings, you will need to choose,
the concepts, the events or behaviors themselves but rather            adapt or develop ones that accurately capture what appears in the
reflections of them.                                                   program’s population.




                                                                                                   STEPS    Module 4: Your Indicators   Page 3
Step 1: Focusing your M&E – Activities and Objectives

Now that you have developed your program (and your causal               components. You need to think about your goals for your M&E –
pathway, see Module 3: Your Objectives) you need to decide which        what do you need to know and why? If you would like some
activities and objectives you want to focus your monitoring and         guidance in thinking about your main evaluation questions and
evaluation on. You may not have the resources to evaluate your          audiences please see Module 1: Getting Started
entire intervention, especially if it is large and has many different
Step 2: Developing Locally Relevant Indicators
Rationale
You have developed immediate and intermediate objectives and now        like in your program’s setting. For indicators to do their job of telling
you need to think about how you will know if you have achieved          you how well your program works, they need to be valid (it measures
them. Indicators are what you actually collect information on to see    what it is supposed to), reliable (the meaning is stable across people,
if the impact you have been aiming for has occurred. It is very         place and time), and feasible (collecting the data for the indicator
difficult to know if your program actually CAUSED the change you        will be possible and affordable). You will need at least one
see. But we use indicators to infer that the intervention (program)     indicator for each of your objectives.
was a part of what caused the change we measure with the indicator.
You will develop at least one indicator for each of your objectives.    Use Pre-existing Indicators When Possible
                                                                        Regardless of whether or not you have already developed some
Task 1: What do you see? Find out what your objective looks             indicators you can benefit from looking at the indicators that
like in your program’s setting.                                         others have developed. There is no need to re-invent the wheel.
Now it’s time to select, adapt or create your evaluation indicators.    And it is often preferable not to do so. Indicators that have been
The more specific, descriptive, and tied to your program your           developed by others will often have been tested for reliability
objectives are the easier it will be to develop and come to some        and validity. Ask colleagues at agencies similar to yours, and at
agreement with your team about what an appropriate indicator is.        universities, if they have already developed locally meaningful
SRHR programs that incorporate a rights-based social justice            indicators. However, you need to be sure that they are really
perspective address aspects of complex social concepts, such as         relevant to your program’s context or they will not provide a
gender equity and reproductive rights. These concepts can be            fair measure of your program’s impact. You will probably need
difficult to “see” and measure. Therefore, you need think about what    to adapt the indicators you have found to make sure they work
the components of these complex concepts are, and what they look        with your target population and in your program’s context.


                                                                                                     STEPS    Module 4: Your Indicators    Page 4
Please see Indicators and Instruments- Selected References for existing indicators which have been used in sexual and reproductive health and
rights programs around the world. In addition, the references in this module have links to some very useful pre-existing indicators:
         Tips: Examples: Indicators That Measure Components of Complex Social Concepts
         Tips: MEASURE Evaluation Publications, including Compendium of Indicators:
         http://www.cpc.unc.edu/measure/publications?searchterm=ms-02-06

Indicators for Complex Social Concepts
        Example: Women’s Empowerment
Women’s empowerment is a key aspect to programs that use a rights-based social justice perspective on their work. One aspect of women’s
empowerment is decision making in the home.

Here is a breakdown of how you might adapt and make more specific the indicator “decision making in the home.”


If your objectives focus
                              The content of your indicators might include
on changes in

Women’s Empowerment:

Decision-making in the
                                 Women report participating in decision making on certain items in a list of topics
home
                                 Women’s and men’s perceptions of who makes the decisions
                                 The process of how women make decisions even when their husbands are authoritarian
                                 How women view their role in the household and in decision making


It is useful to consider indicators that have been developed for, and agreed upon by, a broad international audience. For example, this list of five
dimensions of women’s decision making in the home has been found to be an internationally important and useful set of indicators. This list is
considered an index of indicators; together they are a good measure of women’s empowerment - Bertrand, Jane T. and Gabriela Escudero. 2002.




                                                                                                            STEPS    Module 4: Your Indicators   Page 5
Compendium of Indicators for Evaluating Reproductive Health Programs. Volume One: Overview, Indicators that Crosscut Programmatic Areas.
MEASURE Evaluation Manual Series, No. 6)

           Determining own health care
           Making large household purchases
           Making daily household purchases
           Visiting family or relatives
           Deciding what to prepare for daily meals

        Tips: Examples: Indicators That Measure Components of Complex Social Concepts

Task 2: What would success look like in your setting?
Now that you have thought about what your objective looks like in       Group Exercise: What would success look like?
your program’s setting, you need to decide what kind of change          Whether or not you do the below exercise as a group (ideally with
would be positive. In other words, if your program were successful      program staff and key stakeholders who are knowledgeable of the
and had the impact that you had intended (meets the objective that      community and the program), the exercise should help guide your
you had aimed for) what would that look like? How could an              development of your locally relevant indicators.
existing indicator be modified to reflect success in your setting?
                                                                        Consider involving:
For example, if women’s empowerment is an aspect of one                     Partner agencies that work in similar populations.
of your objectives, and your team has decided that women’s                  Beneficiaries and members of the priority
perceptions of their role in household decision making is an                   population, especially marginalized and vulnerable
indicator of empowerment, then your team needs to decide                       groups. If you involve community members who are
what success would be with regards to household decision                       not aware of these concepts, you may first have to
making in the context of participants’ lives. If you had                       teach them about the concept itself in order to get
reached your objective would that be indicated by a                            their perceptions of how it looks in their lives. Even
particular type of decision- like the ability to decide when to                though this may take a bit more time, it is often well
go to the doctor? Is that a realistic expectation in your                      worth the effort.
program setting? Or is that a change that could only occur                  Stakeholders and other knowledgeable colleagues.
after a much more intense program experience or length of                   Local researchers, particularly anthropologists,
time?                                                                          sociologists, or public health specialists.



                                                                                                    STEPS    Module 4: Your Indicators   Page 6
     Materials:
     -Worksheet A: Finding Out What Success Looks Like (half of worksheet also used in mod 3)
     -columns
     -Your Causal pathway

     How to proceed:
      Refer to your program objectives, the complex social concepts that are part of your objectives, and the indicators that you or
     others have developed (if you have some). If you filled out the objectives section of Worksheet A (columns 1, 2, 3) then you will
     want to refer to the relevant components that you listed in column 3.

      For each component of your objective, answer the following questions:
1.   How would you know when you see success? In other words, what would it look like in the community if the program’s objective
     were met?
2.   How would you know when you haven’t achieved your objective? In other words, what would it look like if the objective were
     not met?
3.   List these suggestions in Worksheet A, column 4. Circle the ones that are most important meaning you would not consider the
     program successful without changes in those specific items.
4.   What would the range of behaviors, observations or events be in the program’s setting?
5.   How much agreement is there among the participants on different indicators they have suggested and on what high vs. low levels
     would look like in the program’s setting?     Tips: Why Is It Important to Assess Agreement on Possible Indicators?
6.   List those indicators, with their high and low levels, on which the participants agreed. If you have held separate meetings with
     different kinds of stakeholders and colleagues, compare the results of those meetings and look for indicators on which there was
     agreement across the groups. Make sure the voices of more vulnerable and marginalized are heard and taken into account. List the
     “winners” in Column 5.       Tips: Why Is It Important to Assess Agreement on Possible Indicators?
7.   Make the appropriate adjustments if you are using pre-existing indicator




                                                                                                      STEPS    Module 4: Your Indicators   Page 7
This worksheet can help you pinpoint what success would look like in your program setting.

  Worksheet A: Objectives (Module 3)                                                       Worksheet A: Indicators (Module 4)
Measuring Complex Social Concepts                                                       Finding out What Success Looks Like

                          For use in Module 3 - Objectives                                         For use in Module 4 - Indicators

            1                            2                              3                            4                               5
Complex Social concepts      Which components do you      Which are relevant- and       What would success and        On which of these is there
                             and your Team think your     possible to target- to your   failure (high and low/good    good agreement among the
(i.e. women’s                program can impact?          local context and to your     and bad levels) on each       people you consulted?
empowerment)                                              program?                      component look like in your
                             (e.g. decision making)                                     local setting?
                                                          (e.g. decision making. But
                                                          not freedom of mobility.)




Task 3: Evaluation needs and your indicators:
Numerical, non-numerical indicators, or both.
You have thought about what success looks like in your program               think about how your indicators address the purpose of your
context- in other words what the indicators are. Now you need to             evaluation. You should think about your evaluation questions and



                                                                                                         STEPS   Module 4: Your Indicators   Page 8
the audiences for your evaluation, so that you can develop and             in the results- should also be considered. Some audiences may only
choose indicators that best address your evaluation needs. You will        be interested in how many participants learned something
need to think about the kind of information, or data, you will need to     (numerical), while others may be more interested in questions that
collect for each indicator. Some measures (or indicators) are              have to do with how a program works, or the words and stories of
expressed as numbers (numerical) and others as words (non-                 participants (non-numerical). It is likely to be the case that your
numerical). Both numerical and non-numerical indicators have their         evaluation questions will include both types of questions, and you
roles and purposes in program evaluation. Making the choice                will want to use both kinds of indicators.
between numerical and non-numerical indicators is shaped by your
evaluation questions and audiences.                                        The table below explains the kinds of questions each type of
                                                                           indicator can address and gives some examples of numerical and
Review your main evaluation questions, if they are more focused on         non-numerical indicators as they relate to sexual and reproductive
the scope, degree and quantity of change, you should use numerical         health and rights program work.
indicators. If they focus on program process (why and how) and                     Tips: Types of Indicators: Numerical and Non-
what things mean to the population, then use non-numerical                         Numerical
indicators. The audience for your evaluation- who will be interested


Type of Indicator             Example


                                                           NUMERICAL INDICATORS
When to use:
 can determine the quantity of each type of response
 tell you about the scope of your program (such as the number of people reached or the number of activities conducted) and the degree of
  change (such as the proportion of men who improved their attitudes towards women working outside the home).
 can be expressed as counts (number of…), rates (number of…per…), proportions (percentage of…), and averages (mean number of…).




                                                                                                       STEPS    Module 4: Your Indicators   Page 9
Counts                 Number of youth programs that promote comprehensive sexuality education
                       Number of new STIs among clinic patients
                       Number of radio advertisements aired
                       Number of congresspersons who support new SRHR-related legislation


Rate                   Number of abortions per 1000 patients per year
                       Number of women who have been beaten by partner per 100 pregnant clinic patients
                       Number of girls who graduate primary school per number who enter primary school
                       Number of ads with stigmatizing/demeaning content per number of ads per year


Percentage             Proportion of women in your program reporting that they participate in household decision making
Or                     Percent of married men who report having had or having sexual relations with men
Proportion             Percent of clients who rate staff attitudes as ‘excellent’
                       Proportion of men in your program who find out if their sexual partner is interested in having sex before they
                        have sex with him or her
                       Percent of young people who perceive that they can talk openly about sex with their parents


Average or Median      Median number of same-sex and other sex sexual partners among unmarried men
                       Average age at marriage among women
                       Average score on scale of empowerment among women in refugee camps




                                                                                                STEPS    Module 4: Your Indicators   Page 10
                                                    NON-NUMERICAL INDICATORS
When to use:
 especially well suited to complex social concepts
 can help you understand your numerical indicators
 tell you about the quality of people’s perceptions and experience
 can help you to understand the “how” and “why” things work as they do/program impact
 describe phenomena your team may not be familiar with
 help you understand the local meaning of things and provide locally relevant language.
 Particularly effective for telling you about stigmatized/unpopular attitudes/beliefs/practices
 help explain why the program is or is not working
 tell you about the meaning of the program to participants
 tell you about the content of people’s attitudes so you can understand how important attitudinal change has been
 perception of the changes they have gone through
 can be expressed as quotes, typologies, themes, paraphrases or summaries. (and can be transformed into numerical indicator)


Category                       Existence of laws against spousal abuse
                               Existence of youth programs that promote comprehensive sexuality education
                               Existence of trained counselors in the clinic
                               Existence of radio advertisement developed by young people which includes information about the youth
                                center’s location, hours and services
                               Counseling training developed
                               Creation of inter-agency reproductive health commission

Quotes and testimonies
                               Quotes from young people about their perceptions of the youth-friendliness of the program
                               Testimonies about the meaning of having been trained as a peer educator
                               Perceptions of young men about the meaning of masculine identity and their efforts to struggle against peer
                                pressure that encourages violence




                                                                                                      STEPS    Module 4: Your Indicators   Page 11
Themes                           Physicians see women who need emergency contraception as having been promiscuous and therefore do not
                                  want to encourage “bad” behavior by offering emergency contraception to everyone.
                                 Use of condoms seen as selfish since men see it as limiting their ability to nourish their wives through the power
                                  of their semen


Typologies                       Perception of loss of virginity by young girls grouped into those who see it as a gift to the boyfriend, as a failure
                                  or stigma, as entrance into womanhood and therefore an achievement, or as a sign of modernity and
                                  independence from tradition.
                                 Types of young boys in urban slum might be grouped into those who perceive positive nurturing figures in their
                                  lives, have greater respect for women/girls, show positive hopes and dreams about their future vs. boys who
                                  report no positive nurturing figure in their lives, see women as instruments for personal gain and expect
                                  achievements will come through violent, ex-legal means


Case study                       Personal history of women who participated in an income generation program that added a SRHR component
                                  vs. one without the SRHR component



        Tips: How Do Numerical and Non-Numerical Indicators Relate to Qualitative and Quantitative Data?

Developing your numerical indicators:                                                   facilities in the region that offer HIV testing). This is the
                                                                                        numerator.
Deciding What Numerators and Denominators to Use                                       the total number of possible events or phenomenon (such as
To calculate percentages, proportions, and rates for numerical                          the total number of women participating in your program
indicators and to know the size of the data set on which your non-                      who have one or more sexual partners or the total number of
numerical indicators are based, you have to determine:                                  healthcare facilities in the region). This is the denominator.
       the actual number of who/that exhibit a particular trait (such
                                                                                The denominator (or total group you are selecting from) you choose
        as the number of women who have one or more sexual
                                                                                should:
        partners who talked with their sexual partner(s) about
        condoms in the last month or the number of healthcare                       include only units (e.g. people, clinics, households) that
                                                                                        could be impacted by your program


                                                                                                            STEPS     Module 4: Your Indicators   Page 12
       be relevant to the event or phenomenon you are measuring                participating in your program talked about condoms with
       be meaningful in the program’s local context                            their sexual partner(s) in the last month (in other words, 37
       be relevant to your programming decisions                               out of 50 women discussed condoms, or 37/50 = .74;
                                                                                (.74x100 = 74%).
 basically the numerator and the denominator are two groups of                 If the total number of women participating in your program
people, events, or documents that you compare.                                  is 100, then only 37% or about one-third discussed condoms
                                                                                in the last month. The numerator remains the same (37) but
When you put the numerator over the denominator, you create a                   the denominator (50 or 100 in these cases) provides
fraction that enables you to determine percentages, proportions and             information on the scope of the phenomenon.
other rates.    Tips: Why are percents so useful?                               Different denominators can have dramatic effects on the
With small data sets, you will report how many people, events, or               results.
documents showed a certain finding (the numerator) “out of” the          Make sure:
total you had studied (the denominator). For example, 7 TV series              both the numerator and denominator are clear and specific
observed out of 10 showed women being beaten threatened or                     you use a relevant denominator to give meaning to the
exploited by their sexual partner.                                              numerator
                                                                              use denominators that reflect the reality in the local context
        Example: If you simply count the number of women who
                                                                               to only include the number of people who actually answered
        talked with their sexual partner(s) about condoms in the last
                                                                               the question in the denominator, if you are reporting a
        month, and find that the number is 37, it is difficult to know
                                                                               percent from an interview or survey in which all respondents
        if 37 is a significant achievement. However, if you know that
                                                                               were asked the same questions
        the total number of women participating in your program is
        50, then you know that 74% percent of the women




                                                                                                   STEPS    Module 4: Your Indicators   Page 13
    Worksheet: Selecting the Most Relevant Denominator- Program Examples
                                                                                                                       Program A –           Program B –
Program A and B both focus on increasing women’s awareness of and ability to negotiate for condom use                  small program in      larger program, in
with their male sexual partners.                                                                                       context in which it   context in which it
                                                                                                                       is rare for           is common for
Program A – small program in context in which it is rare for unmarried women to have sexual relations                  unmarried             unmarried
                                                                                                                       women to have         women to have
Program B – larger program, in context in which it is common for unmarried women to have sexual relations              sexual relations      sexual relations
Indicator:                                                                              Percent of women in the
                                                                                        program who talked to their
                                                                                        male sexual partner(s) about
                                                                                        condoms in the last month

Numerator:         Alone does not give any idea of the magnitude of the result          Number of women in the         37                    37
                                                                                        program who talked to their
                                                                                        male sexual partner(s) about
                                                                                        condoms in the last month


Broadest           Since you want to assess how well the women in your program          Total number of women in       37/50=74%             37/100=37%
Relevant           responded, logically the number of women in the program is the       the program
Denominator        largest relevant denominator. It would not be meaningful to
                   count all the women in the clinic or village.


Focused            But not all the women in the program will have a chance to talk      Total number of married        37/40=92.5%           37/40=92.5%
Denominator        with one or more partners about condoms. So the denominator          women in the program
                   could be tightened to focus on those who the program expects
                   will be having sexual relations. Given typical biases, the program
                   might assume that only married women are having sexual
                   relations.
More Focused       But in some contexts unmarried women might also have male            Total number of married        37/42=88%             37/70=52.8%
and Locally-       sexual partners and the program should include them in the           women and unmarried
relevant           denominator also.                                                    women in the program with
Denominator                                                                             male sexual partners




                                                                                                                   STEPS    Module 4: Your Indicators     Page 14
   Worksheet: Selecting the Most Relevant Denominator
                                                                                  Indicator components           Select the denominator that:
                                                                                                                  is most relevant to the
Indicator:            What percentage, proportion or rate do you want to                                             phenomenon being
                      measure?                                                                                       measured
                                                                                                                  is most realistic in the local
                                                                                                                     context
Numerator:            What is the absolute incidence that you want to capture?                                    most accurately represents
                                                                                                                     your program’s priority
                                                                                                                     population
                                                                                                                  excludes cases for which
                                                                                                                     you could not collect data
Broadest Relevant     What is the broadest relevant population exposed to your                                       for this indicator
Denominator           program and relevant to the numerator? e.g., total                                          includes cases that/who
                      program participants, people exposed to advocacy                                               are the focus of the
                      message, women who have been pregnant                                                          corresponding program
Focused               What sub-population would be more relevant to your                                             object
Denominator           program? e.g., those who attended certain activities,
                      people who have power to make decisions about abortion
                      legislation, women who have had abortions

More Focused and      Are there any specific subgroups you should add who are
Locally-relevant      not usually included but are very relevant? e.g. people
Denominator           who tried to use the services but were turned away,
                      people who oppose legal abortion but might change their
                      minds, women who were denied legal abortions




Developing non-numerical indicators:
Although you will not be calculating percentages, you will need to report how many cases (people, events, or documents) showed a certain finding
out of the total number that had been studied. This can give a sense of how common or typical the findings are and their implications for the



                                                                                                      STEPS    Module 4: Your Indicators   Page 15
program.
 You also need to decide in what form the indicator will be presented. They are:

       Direct quotes or paraphrase or summaries
       Themes identified from reading transcripts
       Typologies of cases
       Coding of pre-determined categories generated by: previous experience, the literature, and quantitative/numerical findings.


Task 4: Factors that Can Influence Participant Outcomes                               clinic unaccompanied, then it is important to consider the
It is important to keep in mind the factors that influence participant                demographic variables as well as the locally relevant factors
outcomes. Often these factors are demographic variables such as:                      that would impact this, such as woman’s age, her marital
gender, age, ethnic group/race, and socio-economic status. These                      status, and possibly her ethnic group/race. (Measuring
factors may also have to do with factors that are relevant to program                 Program Outcomes: A Practical Approach. 1996. United
participation and impact in some other way.                                           Way of America.)
         For example, the experience of the participant before they
got to your program.                                                          Types of information that you should consider:
                                                                                     a. demographic information
It is critical for you to think about which factors (or variables) you               b. participant’s level on issue the program targets (e.g.
think are likely to shape how your program impacts participants.                          experience with violence, or previous sexual experience)
That information might be important to include in our indicators.                    c. any other factors that you think are important to how the
           For example, if you are interested in your program’s impact                    program impacts the participant.
           on women’s empowerment, which in your program setting
           can be indicated in part by the ability to go to the health care




                                                                                                         STEPS    Module 4: Your Indicators   Page 16
Task 5: Write your Indicators
Write your indicators, once you have a clear idea of:
       a. the complex concept each objective addresses
       b. what it looks like in the program setting and what success for that indicator would look like
       c. what your main evaluation questions and audiences are and the type of indicator that would be best (numerical and/or non- numerical)
            to use
       d. other factors that can influence the program’s impact on participants


   Worksheet: Selecting the Most Relevant Indicator
Indicator Elements       Your Response                                                             Checklist for choosing indicators:
Objective: What
exactly will be                                                                                    Closely tied to program effect: Of all the possible
measured                                                                                           indicators under consideration to measure a
Immediate/Intermedi           Evaluation- Immediate result                                        particular objective, choose the indicators(s) that
ate: To what level of          _____________________________________________                       most accurately and unambiguously reflect the
objective does the            Evaluation- Intermediate result                                     correspondent program objective. Make sure
indicator correspond?          _____________________________________________                       changes in the indicator clearly reflect program
                                                                                                   success without being strongly influenced by
                              Monitoring indicator (directly linked to activities) (link to How
                                                                                                   other factors.
                               far to go/Monitoring in back)___________
Population: Exact                                                                                  Feasible: There are many different ways to
population among                                                                                   collect data on each indicator. Consider
whom data will be                                                                                  obstacles and skills that may be involved in
collected




                                                                                                          STEPS    Module 4: Your Indicators   Page 17
Numerical/Non-                Numerical:                                                      successfully collecting data for each indicator
numerical: Type of                o Count (number of…)                                         under consideration. Choose indicators that are
indicator and                     o Rate                                                       mostly likely to be successfully collected while
calculation                       o Percentage/proportion                                      giving you the information you need.
                                  o Average
                                                                                               Cost effective (money, staff time, and other
                              Non-numerical                                                   resources): Costs will always have to be
                                  o Category                                                   weighed against the convenience and depth of
                                  o Quotes                                                     meaning of the indicator. For example,
                                  o Themes                                                     indicators that require waiting a certain amount
                                  o Typologies                                                 of time after the program is completed may
                                  o Case Study                                                 entail greater costs to find the respondents but
                                                                                               give a more profound indication of the “real
Numerator/Denomin         Numerator =                                                          world” results of the program.
ator: Size of group
from which data are       Denominator =                                                           Rights-Based: Make sure all indicators are
collected                                                                                          sensitive to Rights-based social justice
External Factors:                                                                                  considerations. Tips: Key Rights-
By what categories will                                                                            based Considerations for Indicators
the indicator be
divided (i.e. gender,                                                                          Remember that indicators should not specify
age, marital status)                                                                           the direction of change being sought since that
                                                                                               information is included in your objectives. (pop-
                                                                                               up Elements up/link) (Elements E)


Step 3: Creating your Logical Framework
Logical Framework tables (log frames) are commonly used for program planning, monitoring and evaluation, especially in international program
contexts. A log frame explicitly ties your program plan to your monitoring and evaluation plan. We want to stress that the log frame should be
used flexibly so that it remains tied to the work of your program and is truly relevant to the work you do.

Another tool that is used for program planning is the causal pathway. Sometimes the causal pathway is also used to plan monitoring and



                                                                                                      STEPS    Module 4: Your Indicators   Page 18
evaluation. Please use the log frame and/or causal pathway tables as best fits your planning, monitoring and evaluation needs. The log frame is
covered in Module 4, and the causal pathway is covered in Module 3.

If you are working on the CD-ROM or have downloaded a copy of STEPS please save a copy of this to your computer and work on it from there.
If you are working on the Internet, and you would like for your work to be saved on the web version of the log frame, then you will need to enter
your email address on the site so that you will be able to save your work and access it later.

Task 1: Fill in Columns 1 and 2 of the Logical Framework
Group Exercise: Filling out Indicators on your Logical Framework

A.     For impact indicators, focus only on the objectives. (     Tips: What About the Goals?)
         1. Gather the tools you will need with your team:
                  o   The logical framework table
                  o   Worksheet: Your Turn - Developing M&E Indicators.
                  o   notes from all stakeholder meetings you have held including filled out versions of Finding out What Success Looks
                      Like
          2. Review your objectives and select at least one indicator for each objective based on all your preparatory work.
          3. Complete the Indicators Worksheet for each indicator. Refer to the checklist in the worksheet to ensure that the indicators you
             have identified meet the criteria for good indicators.
          4. Complete a master version of the Indicators Worksheet by listing all your indicators.

B.     For monitoring (process) indicators, focus solely on your program activities.

          1. Use straightforward measures that tell you if the activity was performed as planned
          2. Put these measures in Column 2. These are sometimes referred to as outputs.




                                                                                                        STEPS    Module 4: Your Indicators   Page 19
   Worksheet Logical Framework
Please add columns and rows as you need them.


     Causal Pathway                     Indicators                                                  Data Collection Plan

                                                                   Methods, Instruments,             Frequency of Data
                                                                                                                                        Person responsible
                                                                         Sources                         Collection

              1                               2                                3                               4                                  5

Your program plan.              How will you know and           Where and how will you get       How often you will get the
                                demonstrate what you have       the information?                 information?
                                accomplished?
                                                                (i.e. interviews with            (i.e. at the end of every
                                                                participants, surveys of         workshop, once a month, 1
                                                                patients, review of clinic       month before/after program)
                                                                records)

GOAL:                           Goals are almost always out of reach for any one intervention. Therefore, you do not need to include your goal in your evaluation
(What is the long-term          plan.
change you are working
towards?)

OBJECTIVES:                     INTERMEDIATE (IMPACT)
INTERMEDIATE                    INDICATORS:
(What intermediate impact       (Specific observable
could your program activities   accomplishment or change.
have - this may include a       Tells you whether or not your
target number.)                 objective has been achieved.)




                                                                                                                   STEPS      Module 4: Your Indicators   Page 20
OBJECTIVES:                     IMMEDIATE (IMPACT)
IMMEDIATE                       INDICATORS:
(What immediate impact          (Specific observable
could your program activities   accomplishment or change.
have- this may include a        Tells you whether or not your
target number.)                 objective has been achieved.)

                                MONITORING (PROCESS)
KEY ACTIVITIES                  INDICATORS
What can your program do to     (What kind, quality, and
create the change you hope      quantity of outputs will be
to achieve?                     produced by your activities?
                                i.e., # of participants, # of
                                workshops held, by what
                                date)

1                               1




2                               2




3                               3




                                                                STEPS   Module 4: Your Indicators   Page 21
Risks and Assumptions: External factors that can impact the delivery and impact of your intervention. These factors are outside of the purview of
your intervention (i.e. you continue to get the grant that pays for your building), and may also be external to your organization (i.e. a general
election). These factors are typically out of your control (i.e. the weather or political stability).




                                                                                                       STEPS    Module 4: Your Indicators   Page 22
Summary
By now you should have:

   A good understanding of the importance of using well                 Specified all the key elements of your monitoring and evaluation
    constructed indicators that are relevant to the program’s local       indicators
    setting                                                              Filled in the Logical Framework Worksheet
   A good understanding of what “success” will look like in the
    program’s local context
   Selected, adapted or developed appropriate indicators

    Tips

Choosing Among Good Indicators

Constructing Indicators and Instruments: Selected References *

Deciding How Far to Go

Elements Your Indicators Should Contain

How Do Numerical and Non-Numerical Indicators Relate to Qualitative and Quantitative Data?

Indicators That Measure Components of Complex Social Concepts

Key Rights-Based Social Justice Considerations for Indicators

MEASURE Evaluation Publications, including Compendium of Indicators:
http://www.cpc.unc.edu/measure/publications?searchterm=ms-02-06


                                                                                                STEPS    Module 4: Your Indicators   Page 23
What about the Goals?

Why Are Percents so Useful?

Why Is It Important to Assess Agreement on Possible Indicators?


Deciding How Far to Go
The level of resources you have for evaluation will determine what levels of evaluation indicators you will use:

You will be able to measure:

                        Monitoring Indicators                                               Evaluation Indicators

If your           Activities                       Immediate Results                  Intermediate Results              Final results
evaluation
resources are:


Low               XXX                              X or XX

Medium            XXX                              XX or XXX                          X

High              XXX                              XXX                                XXX                               XXX


Evaluation Indicators are directly linked to your objectives at all                      changes the program was expected to contribute to and
three levels: immediate, intermediate and final results.                                 unexpected changes that have occurred.
They provide data about:



                                                                                                             STEPS   Module 4: Your Indicators   Page 24
Keep in mind the distinction among the different levels of evaluation           usually do not have sufficient resources, expertise or
indicators:                                                                     timeframe needed to affect and measure change at the
     Indicators of immediate objectives reflect the changes that               goal/final results level which are quite complex and are the
        happen as a direct consequence of carrying out the program              product of many kinds of interventions over time.
        activities, such as: the number of adequately trained staff or
        people who are aware of the program or certain information       Elements Your Indicator Should Contain
        through outreach efforts; the number of people who learned          A. Provide clear and precise information on exactly what will
        certain information in the program’s course; or the number             be measured, avoiding the use of any jargon or terms that
        of people who report positive reactions to the program’s               might not be widely understood. For example, if condom
        campaign messages. (Some evaluation schemes call these                 use will be measured, you will need to include information
        process Indicators).                                                   about when condom use will be assessed: will it be measured
     Indicators of intermediate objectives reflect the changes                at last sex, or first sex, or the last three times sexual
        that take place one or more step removed from your                     intercourse occurred, or over the past month? You will even
        program. These are measures of what happened as a result               need to make sure you are specific about what is meant by
        of the immediate objectives being achieved. For example, if            “sex”. Does it only refer to vaginal intercourse, or also
        staff are trained by the program adequately, they will change          include anal and oral intercourse. And what is meant by
        their attitudes and behavior with clients (one step removed)           “condom use”? Does it mean every time there is sexual
        and clients will feel more satisfied with the services offered         contact in one intimate session or only once during a
        (two steps removed); if a TV program focused on involving              session? And is the usage correct or does any kind of usage
        more men in “mothering” their children is seen and well-               count?
        received, men might report a greater desire to engage in            B. Specify the exact population from which the sample will be
        child care (one step removed) and actually participate more            drawn, including the numerator and denominator for
        in child care after seeing the program (two steps removed)             percentages, proportions and other rates. In addition, the
     Indicators of final results or goals reflect the long-term               indicator should identify the specific population among
        changes to which your program will contribute, such as the             whom condom use will be assessed: will it be measured
        HIV/AIDS infection rate; percent of girls who are not                  among women who have been heterosexually active in the
        genitally cut, or percent of boys and girls who suffer sexual          past month, among heterosexually-active young people in a
        abuse. Remember that even though you identified what the               given school, or among truck drivers participating in your
        final results or goals your program is working towards, most           training program?
        community-based or small-scale SRHR programs will not               C. Identify the type of calculation to be conducted, such as a
        need to develop indicators at this level. Such programs                count, percentage, rate, existence, perception, quote,
                                                                               paraphrase, typology, etc. Imagine you are interested in


                                                                                                   STEPS    Module 4: Your Indicators   Page 25
       increasing condom use in your community. There are many                    that include information about each individual’s age and
       things you could measure: the percent of people who report                 sexual activity. Specifying this in the indicator reminds
       they always use condoms; frequency of certain kind of                      everyone that data collection forms need to include space to
       perceptions by men of women who carry or use condoms;                      record this information. For more on specifying the
       the number of free condom distribution sites in the                        population, see the section below on numerators and
       community; what “correct” condom use means for different                   denominators.
       groups of people; the meanings evoked in advertisements,                E. Do not specify the direction of change being sought. The
       etc. You’ll want to specify exactly what calculation you’ll do             measurement will tell you whether something increased or
       or how you will present the data you collected from                        decreased. In other words, your indicator will not say
       interviews, observations, and document review.                             increased condom use at last sex since you will need to
    D. Indicate if data should be separated into categories, such as              measure condom use to determine if it has increased,
       by sex, gender, age, type of respondent, location or site, etc.            decreased, or stayed the same. Thus, your measure will be
       You will often want to know whether or not the program is                  condom use at last sex.
       equally effective with different groups, whether one group is
       participating differently than another, or whether different        How Do Numerical and Non-Numerical Indicators Relate to
       groups of participants have different perceptions/attitudes         Qualitative and Quantitative Data?
       about the program. The indicator can tell you how you will              Qualitative data can be transformed into numerical indicators
       look at the data by specifying categories such as age groups,              by counting the number of people who report certain kinds
       sexual activity status, etc. For example, if you want to make              of perceptions, opinions, and meanings.
       sure that the young people who are being recruited to be peer           When it is more important to maintain the actual words of
       educators in your program include youth aged 10-14 as well                 the people, use non-numerical indicators such as direct
       as older youth, or sexually active and non-sexually active                 quotes or paraphrased text.
       young people, you might include an indicator such as:                   Quantitative data will always be reported as numerical
       number of peer educators participating in the training                     indicators.
       program, by age and/or by sexual activity. In this case, you
       know that you will need to develop data collection forms

For a graphic description    Tips: Description of the relationships between the kind of data and the kind of indicators you will use




                                                                                                     STEPS    Module 4: Your Indicators   Page 26
Indicators That Measure Components of Complex Social Concepts
If your objectives focus on   The content of your indicators might include (These examples could be described in numerical or
changes in:                   non-numerical indicators):

Empowerment:


Decision-making in the home      Women report participating in decision making on certain items in a list of topics
                                 Women’s and men’s perceptions of who makes the decisions
                                 The process of how women make decisions even when their husbands are authoritarian
                                 How women view their role in the household and in decision making




                                                                                              STEPS    Module 4: Your Indicators   Page 27
Control over economic             Women who report earning money and controlling its use
resources                         Couples who share economic resources earned by both
                                  Women’s and men’s perceptions of the control of economic resources in their household
                                  Stories by men and/or women about how economic resources have been utilized and why
                                  Observations of couples making decisions (at home or in a programmatic setting)


Freedom of mobility               Women who report that they do not have to ask permission to leave the house
                                  Places a woman can go to without having to ask permission of her husband
                                  Perception by women that they can go to all the places they want to go, and/or that there are places they
                                   are not allowed to go to, where, by whom, why, and how they feel about and deal with that (i.e. is there
                                   ever any resistance, bending the rules, etc.)
                                  Stories about when women have gone somewhere they weren’t supposed to and what happened
                                  What the rules there are concerning women’s mobility and what happens if someone breaks them


Gender Equity:


Couple communication              Topics couples discuss frequently
                                  Level of agreement/disagreement in couple discussions
                                  Wife’s and husband’s perceptions of the last time they had to resolve a disagreement and what happened

Tenderness and solidarity by      Level of emotional support provided by husband/father (as per a scale found/adapted/developed by Team)
males                             Boys/men who report rejecting violence as a norm
                                  Perception of what it would look like if women felt supported by the men in their lives
                                  Perceptions of how it felt when men in their lives did or did not support them, express solidarity and/or
                                   tenderness.
                                  How do they express tenderness and/or solidarity with their partner




                                                                                                    STEPS    Module 4: Your Indicators   Page 28
Strength and confidence by       Score on self-esteem scale
women                            Kinds of decisions women feel confident to make
                                 Reports or stories by women/girls who tell about situation where they felt they had the right to do what they
                                  wanted to do- e.g., see a friend, read a book, or ride the bus.
                                 Perceptions by men/boys of how they feel when women make certain kinds of decisions
                                 Situations in which women/girls did not like/feel comfortable with/agree with something and they did
                                  something about it

Youth Participation:

Ability to inform community      Inclusion of young people on new civic council
decision-making                  Youth-generated proposals that were included in council’s recommendations
                                 Perception of community leaders and/or youth of how and with what impact youth participated in recent
                                  community decisions

Advocacy skills                  Young people who had passing scores on post-test of skills-building workshops
                                 Young people who engage in advocacy activities during the 6 months after the end of the program
                                 Perceptions or stories from young people about something they think should be changed in their
                                  community and what, if anything, they have done about it, and why or why not




                                                                                                    STEPS     Module 4: Your Indicators   Page 29
Key Rights-based Social Justice Considerations for Indicators

Program
Component         Level of indicator      Working Definitions                                             Rights-based Considerations

Activities        Monitoring indicators   Measure implementation of the program; resources                   Address distribution of power and
                  (sometimes called       dedicated to the program; content and quality of the                resources among staff and projects
(Program          output or process       program; attendance; training of those who implement the           Quality of training and outreach
implementation)   indicators)             program; dissemination of messages by program; events               materials
                                          held                                                               Where applicable, need to capture
                                                                                                              holistic and comprehensive nature
                                          Ask: can the program control this?                                  of care and services provided



Immediate         Evaluation Indicators   Relate to the effects that are closely tied to the program’s       Make certain that messages were
objectives        of Immediate Results    implementation, but not totally controlled by the program           understood and skills were learned
                  (sometimes called       implementation itself:                                             Be certain marginalized or
(Directly         short-term outcome          1) knowledge, attitudes, intentions changed by                  underserved subgroups were not
influenced by     indicators)                     program as measured immediately through post-               stigmatized or left-out inadvertently
the program                                       tests                                                      Were the messages or program
activities)                                   2) number of people trained by the program who have             content received by the priority
                                                  acquired certain skills                                     populations as culturally appropriate
                                              3) number of messages received by target audience               and of high quality?

                                          Ask: how was the program content received,
                                          understood?




                                                                                                         STEPS   Module 4: Your Indicators   Page 30
Intermediate       Evaluation Indicators     Relate directly to whether the desired change actually           Focus on changes in social or
objectives         of Intermediate           occurred in the population served but in a way that is one        behavioral norms that uphold
                   Results (sometimes        or more steps away from the program, possibly through             inequities in the community
(Changes in the    called medium             time lapse or because the change builds on the immediate         Changes in attitudes, knowledge,
population that    outcome indicators)       results and goes one or more steps toward the goal                behaviors in real life that will
can lead to - or                                                                                               logically lead to the final result.
are needed to                                                                                                 Programmatic improvements and
achieve - the                                Ask: what else happened in the population because of              policy changes that can lead to the
final goal)                                  the program that will help produce the desired final              final results.
                                             result?

Goal               Evaluation Indicators     Show the impact of the program at the population level.          Changes in issue your program is
                   of Final Result or Goal                                                                     trying to change at the population
(Measures your                               Remember: you will not be expected to measure changes             level
long-term goal)                              at this level since they are too complex to be achieved by       Changes in status of women and
                                             one relatively small program.                                     other marginalized groups
                                                                                                              Elimination of legal, social and
                                             Ask: Is this the difference we want to make in the                cultural obstacles to SRHR and
                                             population’s health and well-being?                               gender equity


MEASURE Evaluation Publications, including Compendium of                      They do NOT tell you what effect or result the program had.
Indicators
http://www.cpc.unc.edu/measure/publications?searchterm=ms-02-06               Since they should be useful for strengthening your work, if you can’t
                                                                              measure all your activities, choose to measure:
Monitoring Indicators are directly linked to your activities.                      new or innovative activities
They provide data about:                                                           activities that have not been tested before
    what and how many activities were conducted                                   activities you are not sure are being carried out correctly
    who participated or received the program’s messages                           activities you want to highlight or share with others
    how well those activities were carried out                                    activities that are central to the program
    what your program is doing or did
    if the program did the activities as planned or better or worse          What about the Goals? Or, How can you know if changes are
                                                                              happening at the level of your goal?


                                                                                                          STEPS   Module 4: Your Indicators   Page 31
In most cases you will not be expected to collect indicators of the       view held more by women than men? Even though more women
broad program goals (also called final results) on your own.              than men agreed with this statement, it might be a misleading
However, other organizations such as universities, ministries of          conclusion depending upon how many women and how many men
health, the United Nations, or other large-scale agencies periodically    participated in the survey. The interpretation changes if 1000
collect and publish data at the local or regional level. Far more         women (47.6%) and 1000 men (34.1%) participated in the survey, or
complex and highly funded evaluations are needed to demonstrate           if 1562 women (30.5%) and 547 men (62.3%) participated in the
the achievement of final results or goals.                                survey.

                                                                          Why Is It Important to Assess Agreement on Possible
Why Are Percents so Useful?
                                                                          Indicators?
Percents are by far the most commonly used proportion. They
                                                                          This process of developing indicators separately and then agreeing
translate numbers onto a scale that is familiar and standardized, and
                                                                          on them actually helps you see which indicators are more valid and
thus help us to understand the magnitude of “How much?” or “How
                                                                          can be measured more reliably, i.e., people agree on them to a
many?”
                                                                          greater degree. Make sure you have included people in the discussion
                                                                          who belong to high priority groups, such as women; ethnic, racial,
For example, if 476 women and 341 men agreed with the statement,
                                                                          sexual minorities; and other vulnerable or marginalized people of
‘If a woman is disrespectful to her husband, it is important that he
                                                                          relevance to the program. And make sure their voices are heard
punish her to remind her who is charge’, what would it mean? Is
                                                                          during the discussions.
this a popular view, or does it characterize only a small group? Is the




                                                                                                    STEPS    Module 4: Your Indicators   Page 32
References

Bertrand, Jane T. and Gabriela Escudero. 2002. Compendium of       esenciales en el desarrollo de indicadores de salud sexual y
Indicators for Evaluating Reproductive Health Programs. Volume     reproductiva: una bibliografía con notas explicativas. Nueva York:
One: Overview, Indicators that crosscut Programmatic Areas.        Fundación Ford e IPPF/WRR; agosto, 2002.
MEASURE Evaluation Manual Series, No. 6. USAID. > Link to
MEASURE Evaluation Publications page:                              GrantsCraft. Making Measures Work for You: outcomes and
http://www.cpc.unc.edu/measure/publications                        evaluation. GrantsCraft, Practical Wisdom for Grantmakers.
                                                                   http://www.grantcraft.org/index.cfm?fuseaction=Page.viewPage&pa
Link to the Compendium of Indicators:                              geID=835 (Last accessed July 16, 2009)
http://www.cpc.unc.edu/measure/publications?searchterm=ms-02-06
                                                                   Kapoor, Ilan. Indicators for Programming in Human Rights and
Bertrand J T, Solis M. (2004) Evaluando Proyectos de Prevención    Democratic Development: A Preliminary Study. Political and Social
de VIH/SIDA: Un Manual con Enfoque en las Organizaciones No        Policies Division, Policy Branch, Canadian International
Gubernamentales. Guatemala: Proyecto de Evaluación MEASURE y       Development Agency (CIDA). Gatineau, Quebec, Canada: CIDA;
Asociación Guatemalteca para la Prevención y Control del SIDA;     July 1996. http://www.acdi-
2000.                                                              cida.gc.ca/INET/IMAGES.NSF/vLUImages/HRDG/$file/INDICEN
                                                                   G.pdf (Last accessed July 16, 2009)
Canadian International Development Agency (CIDA). Guide to
Gender-Sensitive Indicators. Hull, Quebec: CIDA; August 1997.
                                                                   Measure Evaluation and UNAIDS. National AIDS Programmes: a
                                                                   guide to monitoring and evaluation.
Dallabetta, G. and S. Hassig (edts). Indicators for Reproductive
                                                                   http://www.emro.who.int/gfatm/home.html (Last accessed July 16,
Health Program Evaluation. Final Report of the Subcommittee on
                                                                   2009)
Adolescent Reproductive Health Services. The Evaluation Project.
                                                                   Measuring Program Outcomes: A Practical Approach. 1996. United
Chapel Hill, NC: Carolina Population Center, University of North
                                                                   Way of America.
Carolina at Chapel Hill; 1995.
www.cpc.unc.edu/measure/publications/pdf/ms-95-02b.pdf (Last
accessed July 16, 2009).                                           Status of Women Canada. Gender equality indicators: public
                                                                   concerns and public policies: proceedings of a symposium held at
Fundación Ford y Federación Internacional de Planificación de la   Statistics Canada, March 26 and 27, 1998. Otawa: Health Canada,
Familia Región del Hemisferio Occidental (IPPF/WHR). Temas         Statistics Canada and Human Resources Development Canada;
                                                                   January 1999.


                                                                                             STEPS    Module 4: Your Indicators   Page 33
UNFPA. Identificación de los indicadores de programas. Primera parte:
Identificación de los indicadores de programas - Conceptos básicos .
Conjunto de herramientas de planificación, seguimiento y evaluación del
administrador de programas. Herramienta número 6. New York: UNFPA,
División de Servicios de Supervisión; Agosto 2004.
http://www.unfpa.org/monitoring/toolkit/spanish/tool6i-spanish.doc
(Last accessed July 16, 2009)

Enlace a todo el Conjunto de herramientas de planificación,
seguimiento y evaluación del administrador de programas. En
inglés, español, francés y árabe:
http://www.unfpa.org/monitoring/toolkit.htm (Last accessed July 16,
2009)

WHO. Monitoring Reproductive Health: Selecting a short list of
national and global indicators. WHO/RHT/HRP/97.26. Division of
Reproductive Health (Technical Support) and
UNDP/UNFPA/WHO/World Bank; 1997.
http://www.who.int/reproductive-
health/publications/hrp_97_26/monitoring_rh_national_global_indic
ators.pdf (Last accessed July 16, 2009)

Yinger, N., A. Peterson, M. Avni, J. Gay, R. Firestone, K. Hardee, E.
Murphy, B. Herstad, C. Johnson-Welch. A Framework to Identify
Gender Indicators for Reproductive Health and Nutrition
Programming. Wasington D.C: Interagency Gender Working Group;
October 2002.
http://www.prb.org/pdf/FramewkIdentGendrIndic.pdf(Last accessed
July 16, 2009).




                                                                          STEPS   Module 4: Your Indicators   Page 34

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:6
posted:12/19/2011
language:
pages:34