Newsletter 5 by wuyunyi



                         INTERNAL LEARNING SYSTEM: Participatory Evaluation
                         By Rebecca Stewart

                         Internal Learning System (ILS) is an evaluation
                         tool developed by Dr. Helzi Noponen. Dr.
                         Noponen, now teaching at the Hubert H.
                         Humphrey Institute at the University of
       The Quarterly     Minnesota, worked for a number of years
    Newsletter of the
                         with non-governmental organizations in India.
      Improve Group
                         The Ford Foundation asked her to develop a
            ISSUE 6      tool that could be used to do participatory
      OCTOBER 2006       evaluation with illiterate individuals. “Partici-
                         patory” means that, instead of the data being
                         gathered, analyzed and reported by an outside
                         evaluator, the people involved in the target
                         program share in gathering and making sense
                         of the data.

                         With ILS, program participants keep a diary
                         in which they document home, work and               Becky Stewart went to India to develop an ILS diary
                         community aspects of their lives. Pictures          for the Handloom Weavers Development Society
                         are used to depict scenes and situations so         (HLWDS). Here she is with some staff members of
                                                                             the HLWDS.
                         illiterate populations can “read” the diary.
                         Program participants draw lines, make tick          users. As women in one self-help group circled
                         marks or draw circles to indicate their present     the pictorial indicator, they began to support
                         situation and progress. Diary users then share      each other and intervene in cases of abuse.
                         and reflect on their diaries in small self-help     When one husband saw the abuse picture in
                         groups (often already formed in micro-credit        the ILS diary, he tore it up and beat his wife.
                         and other similar programs).                        When the wife told her self-help group what
                                                                             had happened, the group came to the house
                         ILS has been field tested with non-governmen-       and together confronted the husband, telling
                         tal organizations in the north and south of         him how the diary was a means of access-
                         India. It is currently being expanded for use       ing micro-loans, and that the group would be
                         other sites in India and Honduras. Those            watching him in the future. Shamed by the
                         using the diaries have ranged from people with      exposure, the husband apologized and the wife
                         lower levels of literacy to those who had to        reported he stopped beating her. The organiza-
                         learn how to hold a pencil to make tick marks       tion, working with these self-help groups, used
1 • Internal Learning    or draw a circle. So far, both the organiza-
    System:                                                                  data from ILS diaries to record a decrease in
    Participatory        tions and people using the diary have given         domestic abuse from over 60% to under 5% of
    Evaluation           positive feedback about its usefulness. The         its members experiencing it in a five year time
                         organizations can gather data using the diaries     span.
2 • Experiences in ILS   to provide program evaluation data to funders.
  • From the Interns
                         At the same time, participants use diaries to       ILS can be most useful in circumstances where
3 • Institutional        reflect together on aspects of their lives,         a population might have low or mixed literacy;
    Review Boards        support each other to take action to address        in addition, it is most powerful when diary
  • Company News         issues in their lives and record progress or        users meet in groups to reflect on their diaries
                         change.                                             and hold one another accountable. ILS could
    Girl Scouts          ILS IN ACTION                                       be useful in the U.S. when working with young
    Research Institute   One of the issues faced by program partici-         children, groups of mixed-language speakers,
                         pants was domestic violence, and ILS diaries        illiterate or low literacy populations, or those
6 • Grants & Requests                                                        who prefer to share information in a way other
    for Proposals        document domestic abuse with pictures to
                         inquire about the experiences of diary-             than in surveys or other traditional methods.

                                                helping organizations prov ide more effective ser v ices
                              Experiences in ILS                                                                                 2
                              By Rebecca Stewart
PROGRAM                       I had the opportunity to work with the
The Improve Group is          Handloom Weavers Development Society
now accepting applica-        (HLWDS) in Balaramapuram, India this
tions for our Corporate       summer to develop a new ILS diary for
Giving program. We will       weavers to use. With another graduate

                                                                                                                                 Photo Credit: Andrea Lindgren
provide grants between        student from the Humphrey Institute,
$500 and $1,000 to            I spent 2 months working with the
five organizations. We        organization to understand the pressing
will take applications        issues facing handloom weavers.
through October and
                              HLWDS is a membership organization formed
make a final decision
                              by weavers 15 years ago with a mission to
and distribute grant
                              rally the traditional livelihood of handloom
funds in November.
                              weaving, an important occupation for many
                              in this region of India, through micro-loans,
For more information
                              training and advocacy. The ultimate goal is
on our Corporate Giv-                                                            A woman at work weaving a dhoti in one of the
                              to develop handloom weaving as an eco-
ing program, visit our                                                           weaving villages of Kerala.
                              nomically viable occupation, particularly for
website at http://www.
                              the region’s women.                                test pages during field testing. But, when the
community.html or con-        It was an invaluable experience to put my          weavers told us that we had captured important
tact Deborah Mattila at       training in ILS into action. I found it chal-      aspects of their lives, I felt rewarded. As HLWDS
deborahm@theimproveg-         lenging to ask the right questions to tease        was a pilot site for an early version of the diary,                     out the many complications and aspects of          I was inspired by stories of how these weav-
                              a livelihood and society with which I was          ers had already put those early versions of the
                              unfamiliar. We struggled through frequent          diaries to powerful use – demanding due rights
                              power outages and slow-as-molasses inter-          from their government, overcoming domestic
                              net connections, which made me appreciate          violence and seeking new ways to market their
The conference room in                                                           products. It is an experience I will always re-
the Improve Group of-         the ease of work in the U.S. There were no
                              PowerPoint presentations as we huddled             member; the sights and sounds of village life in
fice is available for use                                                        India are still with me.
by non-profit organiza-       around a few dog-eared print outs of diary
tions for meetings free
of charge, on condition
of availability.             From the Interns                                     she designs cultural clothing and plans cultural
                             By James Douglass and Asha Omar                      weddings. She has won many awards including
Please contact Deborah                                                            Volunteer of the Month for the Somalia com-
                             As interns with The Improve Group, we have
Mattila at deborahm@                                                              munity.
                             an interesting answer to the question: “So ...
                             What did you do over the summer?” But before
for inquiries.                                                                    As the nature of an internship can vary across
                             we answer this, we should explain how we
                             arrived here.                                        and within companies, James’ experience dif-
NEW OFFICES                                                                       fered from Asha’s. Asha’s operational knowledge
On June 5 we moved           Though we have different backgrounds, we share       of computers and software is the one area
into our new offices in      our involvement in the job readiness program         in which her experience and knowledge in-
Bloomington. Our new         STEP-UP, coordinated by Achieve!Minneapolis, a       creased the most. Now, Asha feels that she is
space has a number           nonprofit that “galvanizes community resources       ready for most computer-related tasks that may
of advantages: a large       to support student success” (http://www.             arise in the future - important in most business
conference room, plenty                            settings. James developed his knowledge of
of space to grow into,                                                            Excel, particularly in the area of data analysis.
and location accessible      Reflecting the varied backgrounds of Improve         Most interesting and mentally challenging for
by LRT. We ended up          Group staff, Asha and James arrived with many        him was the analysis for the Performing Arts
with more space than         experiences and interests. James is very active      Workshop project on which he worked alongside
we planned (which is         in the Boy Scouts of America, and is almost fin-     Nathan Paine.
a long story in and of       ished with his Eagle Scout. He enjoys video edit-
                             ing, silversmithing, and other artistic ventures.    Beginning this fall, James begins undergraduate
itself – log onto our blog
                                                                                  studies at St. Lawrence University in Canton,
at http://www.theim-
                             Asha has been involved in the Somali commu-          NY. Asha is attending Minnesota Community
                             nity many years. She has volunteered at the          Technical College. We have enjoyed the op-
for more information).
                             Somalia Youth Camp, interpreting the Somalia         portunity to work alongside the Improve Group
                             language into English. Asha speaks fluent Somali     staff, and look forward to the future and what
                             and Arabic. When not volunteering or working,        it may bring.

      Institutional Review Boards
      By Deborah Mattila
     If you have ever conducted a study that                       tion projects. The IRB process varies; typically a review begins with the
     collected any data from or about people, and                  submission of materials by the researcher, including all data collection
     that was funded by the Federal government,                    instruments, informed consent forms and a full research plan. Over the
     you have probably had to acquire an Institu-                  course of a few weeks to a few months, the review board will review
     tional Review Board (IRB) approval. You may                   documents, request additional materials and finally approve or deny
     also have been required to obtain IRB approval                a research study. Sometimes, the review board may require modifica-
     on research projects that were not federally                  tions to the research design or that a researcher develop skills before
     funded. Even if you are never required to obtain              approval of their study.
     IRB approval on other research projects, it
                                                                   Some human subject research is exempt from IRB review. Studies con-
     can be worthwhile to have an IRB review your
                                                                   ducted in common educational settings involving normal educational
     research methods to be sure they are sound,
                                                                   practices that use educational tests, surveys, interviews or already ex-
     ethical and validated. The primary purpose of
                                                                   isting data are exempt if no identifying data (such as names or ID num-
     an IRB is to protect the safety, rights and wel-
                                                                   bers) are used. Federally-funded human subjects research that involves
     fare of people who participate in research, the
                                                                   surveys or interviews of minors is NEVER exempt from IRB review, but
     human subjects.
                                                                   may be eligible for expedited review. In an expedited review, one or
                                                                   a few of the IRB chairs review the proposed research activity instead
     An IRB is a group that reviews studies proposed               of the full board. It is beneficial to request an expedited review, if
     by researchers. Each IRB is composed of re-                   eligible, because you will receive a quicker response and the process is
     searchers and community members.                              less costly. The conditions for expedited review are related to the risk-
                                                                   level and involvement of the human subjects in the research process.
     IRB review occurs before research begins.
                                                                   For example, studies using surveys that ask for opinions about pop
     In a typical evaluation project, this is after
                                                                   culture would likely be eligible, while those asking for detailed personal
     instruments and data collection protocols have
                                                                   history would not.
     been established, but before they have been
     implemented. Often, a continuing IRB review                   Although researchers, evaluators and program administrators might
     must occur in each year of multi-year evalua-                 find the IRB process cumbersome and confusing, it evolved from the
                                                                   detailed ethical principles established in the Belmont Report, a sum-
                                                                   mary of ethical principles identified by the National Commission for the
                                                                   Protection of Human Subjects of Biomedical and Behavioral Research in
                                                                   1976. The Belmont Report was a national response to revelations about
                                                                   human research studies, such as the Tuskegee syphilis study, which
                                                                   raised concern about protecting individual rights in research. The basic
                                                                   ethical principles defined by the Belmont Report are: (1) Respect for
                                                                   Persons, or acknowledging the autonomy of each individual; (2) Benefi-
                                                                   cence, or maximizing benefits and minimizing harms; and (3) Justice, or
                                                                   a fairness in the distribution of benefits and risks.

                                                                   Company News
                                                                   By Leah Goldstein Moses
                                                                   It has been a very busy summer at the Improve Group. We’ve welcomed
                                                                   new faces and places and said goodbye to some familiar ones.

                                                                   SUMMER INTERNS
                                                                   This summer we welcomed two interns from the Achieve! Minneapolis
                                                                   Step Up! program ( James and Asha
                                                                   have written about their experience and what they’ve learned on page
                                                                   2. What they didn’t say is how much we’ve learned from them. Having
                                                                   them around all summer helped remind us how important support is as
                                                                   you make transitions in your life. They also did incredibly good work
                                                                   for us – we moved into our office just days before they started and they
                                                                   helped get us organized and settled. We’ll miss them and are looking
                                                                   forward to hearing about their adventures in the coming years.
                                                                   (Continued on pg. 4)

Printed on paper manufactured from 30% recycled postconsumer fiber and with 38% of electricity from emission-free wind power.
4                                                                                            (
                                                                                             was recently awarded a grant from
                                                                                             the U.S. Department of Education
(Company News, continued from pg. 3)                                                         for the ARISE project (Artist Residen-
                                                                                             cies in Special Education), receiving
NEW STAFF                                                                                    the twelfth highest score among over
In June, we hired two new research                                                           140 applicants. This four-year project
analysts. Elizabeth Radel (Liz) recently                                                     includes an extensive evaluation using
completed her Masters in Public Health                                                       a combination of quasi-experimental
from the University of Minnesota where                                                       and action-research methods. The ARISE
she specialized in epidemiology. Her                                                         project has an ethic of social justice
evaluation interests are in programs for                                                     and equity and the Workshop will be
children with special needs and those                                                        working closely with the San Francisco
that promote reproductive and sexual                                                         Unified School District to serve students
health. Liz joined us at the end of June                                                     who are disproportionately classified as
and after just a week in the office left                                                     having special needs.
for China; she returned only to imme-
diately head up to Itasca and Koochich-
ing Counties. She provides wonderful                                                         STAFF TRAVELS
perspective in her primary project of                                                        AND ADVENTURES
evaluating Minnesota’s Home and Com-           The Improve Group and DHS staff members       Congratulations to Deborah and Andy
                                               visit many Minnesota landmarks on their       Mattila on their marriage! After a
munity Based Services (HCBS) programs          travels for the Waiver Review Project.
for the Department of Human Services                                                         beautiful ceremony, they honeymooned
(DHS).                                                                                       in Belize (see our blog at http://www.
                                               NEW PROJECTS                         for more
                                               We have recently started three large          wedding & honeymoon pictures)
                                               evaluation projects. In collaboration
                                               with the Department of Human Services         Rebecca Stewart recently returned from
                                               (, we              India, where she spent two months
                                               are monitoring Home and Community             working with a local NGO in conducting
                                               Based Services in each of Minnesota’s 87      a program evaluation. See her articles
                                               counties and in several health plans and      about her experience on pages 1 and 2.
                                               tribal governments. For this project we
                                               are reviewing implementation by each          Brooke Ahlquist trained and completed
                                               agency to ensure compliance with both         the Twin Cities Breast Cancer 3-day
                                               state and federal laws. This project          walk through much of the Twin Cities,
                                               involves collection and analysis of data      starting at the Minnesota Zoo in Apple
The Improve Group staff enjoyed their annual   from staff, participant records, pro-         Valley and ending at the State Capitol.
retreat at Hyland Lake Park in August.         vider contracts and state system data.
                                               Starting in 2007, we will begin collect-      CONFERENCES
Eric Wong, also a recent graduate of the       ing data directly from participants in        The Improve Group will be presenting
University of Minnesota who completed          interviews or focus groups. This project      in two conferences this fall. Our work
his Master’s in Public Policy in June, is      is expected to last through 2011.             with Girl Scouts USA has been selected
particularly interested in economic pol-                                                     by the American Evaluation Association
                                               We have begun a new project with one
icy issues. Prior to joining the Improve                                                     ( for a presentation
                                               of our long-term clients the Girl Scouts
Group he conducted an analysis of                                                            on challenges and lessons in cluster and
                                               of the USA. In 2005, we completed
projected revenue losses due to policy                                                       multi-site program evaluation.
                                               an evaluation of their Girl Scouts in
changes for the Minnesota Department
                                               Detention Centers program and in June
of Revenue and researched the impact                                                         We will also be working with DHS to
                                               of this year we began an evaluation
of need-based financial aid on academic                                                      present our work on the statewide
                                               of the Project Anti-Violence Education
progress. His economic background                                                            HCBS evaluation at the 22nd National
                                               (P.A.V.E.) program, offered by 26 coun-
has proven valuable in thinking about                                                        Home and Community Based Services
                                               cils around the U.S. and in Europe. We
expenditures in the HCBS program. His                                                        Conference (
                                               are collecting data from each council,
parents own the Golden Inn restaurant                                                        conference). Because the HCBS evalu-
                                               using retrospective pre-test surveys
in Superior, WI, and he helps out there                                                      ation responds to a recently expanded
                                               with girls from 10 councils and success-
a few times each month. Between                                                              requirement for state accountability in
                                               case site visits to 6 councils. This evalu-
the HCBS evaluation and his travels to                                                       HCBS programs, we will provide details
                                               ation focuses on personal growth and
Superior, Eric definitely gets the most                                                      of the methods used and our findings so
                                               leadership development outcomes.
mileage of anyone in our office.                                                             far to a national audience.
                                               One of our long term clients, the
                                               Performing Arts Workshop,
WORKSHOP SERIES                                                                                                                      5
Much of our client base
comes from organizations
                                FEATURED CLIENT:
that have funds, either         Dr. James Riedel, Girl Scouts Research Institute
targeted or as part of a        By Marian Kimball Eichinger
program fund, specifically
for research, evaluation        James Riedel, Ph.D., is a Senior Researcher for      MKE: Why is this topic of research important
and planning. However,          the Girl Scouts Research Institute (GSRI). GSRI      to you?
we receive a number of          is a center for research and public policy infor-    JR: Girl Scouting has a 95-year history of doing
phone calls from smaller        mation on the healthy development of girls.          wonderful things for girls. I believe in all of the
organizations that want         James currently serves as a liaison between          initiatives of the organization. Anti-violence
to do research, evaluation      GSUSA and The Improve Group which is                 education is important because it cuts across
and planning but don’t          conducting the evaluation of GSUSA’s Project         every subgroup of humanity. It is a world issue.
have specific budgets to        Anti-Violence Education (P.A.V.E. the Way).
take on those projects.
                                                                                     MKE: Why is this research important for girls
                                MKE: How long have you been with GSUSA               and our society overall?
In response to this need,       and what do you do for them?
the Improve Group will be                                                            JR: The work of a girl-only organization is
                                JR: I have worked at GSUSA for eight years,          crucial, and our own research, The Ten Emerg-
offering workshops on basic
                                beginning as a research and evaluation analyst       ing Truths (2002), uncovered several reasons.
methods and tools. We are
                                in 1998. My primary work is program evaluation.      Girls need empowerment and a climate in which
working with Aurora Con-
                                I investigate the benefits to girls of participat-   they can connect with each other. This is sup-
sulting, Catalyst Consult-
                                ing in various programs. Some of the programs I      ported through a “by girls, for girls” approach.
ing, DeYoung Consulting,
                                have evaluated were designed to produce equity       Furthermore, certain topics are specific to girls
and the Word Applied to
                                in science and technology; develop leadership        and others are of a very sensitive nature, calling
develop a series for small
                                and communication skills; or prevent youth           for the need of a secure, safe and comfortable
non-profit and public agen-
                                violence.                                            place to learn and grow.
cies, and plan to roll the
series out in January 2007.
                                MKE: What is your background and how did it          The mission of Girl Scouting is “to build girls of
Workshop topics will cover:     bring you to the GSRI to perform research for        courage, confidence and character, who make
• Basic evaluation tools        girl-based programming?                              the world a better place.” It is important
  and planning                  JR: I have conducted evaluations for 14 years.       for girls that we measure the outcomes and the
                                While my concentration is evaluation, the GSRI       processes of their participation. Without dem-
• Logic model design for
                                does original research studies and literature        onstrating that the girls’ involvement produces
  programs and organiza-
                                reviews as well. Examples of recent research         positive results, there are only gut-feelings and
                                conducted by my colleagues include:                  hear-say as evidence that membership is worth-
• Business writing, report-                                                          while. Girls, volunteers, staff and funders all
  ing and presentations         • The New Normal? What Girls Say About               look for more when deciding to what organiza-
• Effective teams, meetings       Healthy Living (2006);                             tions they devote time and money.
  and work planning             • Voices of Volunteers 18-29 (2003); and
• Building staff, board and     • Feeling Safe: What Girls Say (2003).               For society, well, there are a lot of youth-devel-
  volunteer leaders                                                                  opment programs and extra-curricular activities
                                I earned my Ph.D. in Measurement, Statistics         that pre- and adolescent girls can get involved
The workshops will provide      and Evaluation at the University of Delaware.        in. It is important to show that the program ac-
both learning and practical     From 1996-1998, I evaluated government-funded        tivities of the largest girl-serving organization in
opportunities, and be ap-       initiatives in some of New York City’s economi-      the world yield the growth that it purports to.
plicable for staff and board    cally poorest elementary and middle schools,
members. Because these          working directly with children and the adults        For information about the Girl Scouts USA
workshops are in develop-       responsible for their care and enrichment. My        and studies by the Girl Scouts Research
ment, we will offer the         doctoral studies focused on the engagement of        Institute, visit
first series at a discount in   middle and high school children. I was always
exchange for detailed feed-     interested in education, be it through the for-
back from participants. If      mal K-12 system or the more non-formal
you are interested in either    Scouting opportunities.
the series or a single work-
shop, please let us know by
emailing Susan Murphy at

   Upcoming Grant and Requests for Proposals
   By Susan Murphy
   Honoring fall’s arrival and the departure of our summer interns to their respective colleges, the Improve Group
   features the following grant opportunities in education. If you need advice on applying for grants or have ques-
   tions about the notices listed above please contact Susan Murphy at

   OF AMERICA (EFA)                         Qwest for Education focuses on enrich-    Sponsored by Toyota Motor Sales,
   EFA makes grants to qualifying non-      ing the lives of children by supporting   U.S.A., Inc. and administered by the
   profit organizations that have tax-ex-   solid preK-12 educational programs.       National Science Teachers Associa-
   empt status and those that are not       They consider programs that:              tion this fund offers 50 grants of up
   private foundations. Projects must       • Effectively use technology to           to $10,000 each and a minimum of 20
   be located within the United States.       improve preK-12 public school           “mini-grants” of $2,500 each to K-12
   Areas of interest include, but are not     instruction                             teachers of science. To apply, qualified
   limited to, the environment, repro-                                                teachers must write a Toyota TAPES-
   ductive freedom, theatre, educa-         • Promote innovative models to            TRY proposal and submit it for receipt
   tion, medicine, drug policy reform,        strengthen preK-12 public school        at NSTA by January 18, 2007. Grants
   democracy, peace & national security       education                               are awarded in three categories: Envi-
   issues and human services. Important     • Improve the skills and leadership       ronmental Science Education; Physical
   characteristics considered by EFA are      of educators and parents                Science Applications; and Literacy and
   an organization’s record of achieve-     • Promote innovative early childhood      Science Education. Go to http://www.
   ment, intended broad impact, sound         education programs            
   financial practices, increasing inde-                                              htm for more information.
   pendence, and correspondence with        Requests are reviewed on an on-going
   EFA objectives. Grant guidelines can     basis and grants generally range from
   be found at         $500 and up. To learn more about this
   Inquiry%20Guidelines.htm.                opportunity visit http://www.qwest.

                                                                                                  877-IMPRVGP (467-7847)
                                                                                                  Phone and Fax toll-free at:

                                                                                            Bloomington, MN 55425-1805
                                                                                            2051 Killebrew Drive, Suite 620

                                                                                                  Rebecca Stewart
                                                                                                  Sarah Myott
                                                                                                  Deborah Mattila
                                                                                                  Marian Kimball Eichinger
                                                                                                  Research Managers

                                                                                                      Susan Murphy
                                                                                                      Executive Assistant

                                                                                                     Leah Goldstein Moses
                                                                                                     President & CEO


To top