INTERNAL LEARNING SYSTEM: Participatory Evaluation
By Rebecca Stewart
Internal Learning System (ILS) is an evaluation
tool developed by Dr. Helzi Noponen. Dr.
Noponen, now teaching at the Hubert H.
Humphrey Institute at the University of
The Quarterly Minnesota, worked for a number of years
Newsletter of the
with non-governmental organizations in India.
The Ford Foundation asked her to develop a
ISSUE 6 tool that could be used to do participatory
OCTOBER 2006 evaluation with illiterate individuals. “Partici-
patory” means that, instead of the data being
gathered, analyzed and reported by an outside
evaluator, the people involved in the target
program share in gathering and making sense
of the data.
With ILS, program participants keep a diary
in which they document home, work and Becky Stewart went to India to develop an ILS diary
community aspects of their lives. Pictures for the Handloom Weavers Development Society
are used to depict scenes and situations so (HLWDS). Here she is with some staff members of
illiterate populations can “read” the diary.
Program participants draw lines, make tick users. As women in one self-help group circled
marks or draw circles to indicate their present the pictorial indicator, they began to support
situation and progress. Diary users then share each other and intervene in cases of abuse.
and reflect on their diaries in small self-help When one husband saw the abuse picture in
groups (often already formed in micro-credit the ILS diary, he tore it up and beat his wife.
and other similar programs). When the wife told her self-help group what
had happened, the group came to the house
ILS has been field tested with non-governmen- and together confronted the husband, telling
tal organizations in the north and south of him how the diary was a means of access-
India. It is currently being expanded for use ing micro-loans, and that the group would be
other sites in India and Honduras. Those watching him in the future. Shamed by the
using the diaries have ranged from people with exposure, the husband apologized and the wife
lower levels of literacy to those who had to reported he stopped beating her. The organiza-
learn how to hold a pencil to make tick marks tion, working with these self-help groups, used
1 • Internal Learning or draw a circle. So far, both the organiza-
System: data from ILS diaries to record a decrease in
Participatory tions and people using the diary have given domestic abuse from over 60% to under 5% of
Evaluation positive feedback about its usefulness. The its members experiencing it in a five year time
organizations can gather data using the diaries span.
2 • Experiences in ILS to provide program evaluation data to funders.
• From the Interns
At the same time, participants use diaries to ILS can be most useful in circumstances where
3 • Institutional reflect together on aspects of their lives, a population might have low or mixed literacy;
Review Boards support each other to take action to address in addition, it is most powerful when diary
• Company News issues in their lives and record progress or users meet in groups to reflect on their diaries
change. and hold one another accountable. ILS could
5 • FEATURED CLIENT
Girl Scouts ILS IN ACTION be useful in the U.S. when working with young
Research Institute One of the issues faced by program partici- children, groups of mixed-language speakers,
pants was domestic violence, and ILS diaries illiterate or low literacy populations, or those
6 • Grants & Requests who prefer to share information in a way other
for Proposals document domestic abuse with pictures to
inquire about the experiences of diary- than in surveys or other traditional methods.
helping organizations prov ide more effective ser v ices
Experiences in ILS 2
By Rebecca Stewart
IMPROVE GROUP GRANTS
PROGRAM I had the opportunity to work with the
The Improve Group is Handloom Weavers Development Society
now accepting applica- (HLWDS) in Balaramapuram, India this
tions for our Corporate summer to develop a new ILS diary for
Giving program. We will weavers to use. With another graduate
Photo Credit: Andrea Lindgren
provide grants between student from the Humphrey Institute,
$500 and $1,000 to I spent 2 months working with the
five organizations. We organization to understand the pressing
will take applications issues facing handloom weavers.
through October and
HLWDS is a membership organization formed
make a final decision
by weavers 15 years ago with a mission to
and distribute grant
rally the traditional livelihood of handloom
funds in November.
weaving, an important occupation for many
in this region of India, through micro-loans,
For more information
training and advocacy. The ultimate goal is
on our Corporate Giv- A woman at work weaving a dhoti in one of the
to develop handloom weaving as an eco-
ing program, visit our weaving villages of Kerala.
nomically viable occupation, particularly for
website at http://www.
the region’s women. test pages during field testing. But, when the
community.html or con- It was an invaluable experience to put my weavers told us that we had captured important
tact Deborah Mattila at training in ILS into action. I found it chal- aspects of their lives, I felt rewarded. As HLWDS
deborahm@theimproveg- lenging to ask the right questions to tease was a pilot site for an early version of the diary,
roup.com. out the many complications and aspects of I was inspired by stories of how these weav-
a livelihood and society with which I was ers had already put those early versions of the
unfamiliar. We struggled through frequent diaries to powerful use – demanding due rights
power outages and slow-as-molasses inter- from their government, overcoming domestic
net connections, which made me appreciate violence and seeking new ways to market their
The conference room in products. It is an experience I will always re-
the Improve Group of- the ease of work in the U.S. There were no
PowerPoint presentations as we huddled member; the sights and sounds of village life in
fice is available for use India are still with me.
by non-profit organiza- around a few dog-eared print outs of diary
tions for meetings free
of charge, on condition
of availability. From the Interns she designs cultural clothing and plans cultural
By James Douglass and Asha Omar weddings. She has won many awards including
Please contact Deborah Volunteer of the Month for the Somalia com-
As interns with The Improve Group, we have
Mattila at deborahm@ munity.
an interesting answer to the question: “So ...
What did you do over the summer?” But before
for inquiries. As the nature of an internship can vary across
we answer this, we should explain how we
arrived here. and within companies, James’ experience dif-
NEW OFFICES fered from Asha’s. Asha’s operational knowledge
On June 5 we moved Though we have different backgrounds, we share of computers and software is the one area
into our new offices in our involvement in the job readiness program in which her experience and knowledge in-
Bloomington. Our new STEP-UP, coordinated by Achieve!Minneapolis, a creased the most. Now, Asha feels that she is
space has a number nonprofit that “galvanizes community resources ready for most computer-related tasks that may
of advantages: a large to support student success” (http://www. arise in the future - important in most business
conference room, plenty achieveminneapolis.org/). settings. James developed his knowledge of
of space to grow into, Excel, particularly in the area of data analysis.
and location accessible Reflecting the varied backgrounds of Improve Most interesting and mentally challenging for
by LRT. We ended up Group staff, Asha and James arrived with many him was the analysis for the Performing Arts
with more space than experiences and interests. James is very active Workshop project on which he worked alongside
we planned (which is in the Boy Scouts of America, and is almost fin- Nathan Paine.
a long story in and of ished with his Eagle Scout. He enjoys video edit-
ing, silversmithing, and other artistic ventures. Beginning this fall, James begins undergraduate
itself – log onto our blog
studies at St. Lawrence University in Canton,
Asha has been involved in the Somali commu- NY. Asha is attending Minnesota Community
nity many years. She has volunteered at the Technical College. We have enjoyed the op-
for more information).
Somalia Youth Camp, interpreting the Somalia portunity to work alongside the Improve Group
language into English. Asha speaks fluent Somali staff, and look forward to the future and what
and Arabic. When not volunteering or working, it may bring.
Institutional Review Boards
By Deborah Mattila
If you have ever conducted a study that tion projects. The IRB process varies; typically a review begins with the
collected any data from or about people, and submission of materials by the researcher, including all data collection
that was funded by the Federal government, instruments, informed consent forms and a full research plan. Over the
you have probably had to acquire an Institu- course of a few weeks to a few months, the review board will review
tional Review Board (IRB) approval. You may documents, request additional materials and finally approve or deny
also have been required to obtain IRB approval a research study. Sometimes, the review board may require modifica-
on research projects that were not federally tions to the research design or that a researcher develop skills before
funded. Even if you are never required to obtain approval of their study.
IRB approval on other research projects, it
Some human subject research is exempt from IRB review. Studies con-
can be worthwhile to have an IRB review your
ducted in common educational settings involving normal educational
research methods to be sure they are sound,
practices that use educational tests, surveys, interviews or already ex-
ethical and validated. The primary purpose of
isting data are exempt if no identifying data (such as names or ID num-
an IRB is to protect the safety, rights and wel-
bers) are used. Federally-funded human subjects research that involves
fare of people who participate in research, the
surveys or interviews of minors is NEVER exempt from IRB review, but
may be eligible for expedited review. In an expedited review, one or
a few of the IRB chairs review the proposed research activity instead
An IRB is a group that reviews studies proposed of the full board. It is beneficial to request an expedited review, if
by researchers. Each IRB is composed of re- eligible, because you will receive a quicker response and the process is
searchers and community members. less costly. The conditions for expedited review are related to the risk-
level and involvement of the human subjects in the research process.
IRB review occurs before research begins.
For example, studies using surveys that ask for opinions about pop
In a typical evaluation project, this is after
culture would likely be eligible, while those asking for detailed personal
instruments and data collection protocols have
history would not.
been established, but before they have been
implemented. Often, a continuing IRB review Although researchers, evaluators and program administrators might
must occur in each year of multi-year evalua- find the IRB process cumbersome and confusing, it evolved from the
detailed ethical principles established in the Belmont Report, a sum-
mary of ethical principles identified by the National Commission for the
Protection of Human Subjects of Biomedical and Behavioral Research in
1976. The Belmont Report was a national response to revelations about
human research studies, such as the Tuskegee syphilis study, which
raised concern about protecting individual rights in research. The basic
ethical principles defined by the Belmont Report are: (1) Respect for
Persons, or acknowledging the autonomy of each individual; (2) Benefi-
cence, or maximizing benefits and minimizing harms; and (3) Justice, or
a fairness in the distribution of benefits and risks.
By Leah Goldstein Moses
It has been a very busy summer at the Improve Group. We’ve welcomed
new faces and places and said goodbye to some familiar ones.
This summer we welcomed two interns from the Achieve! Minneapolis
Step Up! program (http://achieveminneapolis.org/). James and Asha
have written about their experience and what they’ve learned on page
2. What they didn’t say is how much we’ve learned from them. Having
them around all summer helped remind us how important support is as
you make transitions in your life. They also did incredibly good work
for us – we moved into our office just days before they started and they
helped get us organized and settled. We’ll miss them and are looking
forward to hearing about their adventures in the coming years.
(Continued on pg. 4)
Printed on paper manufactured from 30% recycled postconsumer fiber and with 38% of electricity from emission-free wind power.
was recently awarded a grant from
the U.S. Department of Education
(Company News, continued from pg. 3) for the ARISE project (Artist Residen-
cies in Special Education), receiving
NEW STAFF the twelfth highest score among over
In June, we hired two new research 140 applicants. This four-year project
analysts. Elizabeth Radel (Liz) recently includes an extensive evaluation using
completed her Masters in Public Health a combination of quasi-experimental
from the University of Minnesota where and action-research methods. The ARISE
she specialized in epidemiology. Her project has an ethic of social justice
evaluation interests are in programs for and equity and the Workshop will be
children with special needs and those working closely with the San Francisco
that promote reproductive and sexual Unified School District to serve students
health. Liz joined us at the end of June who are disproportionately classified as
and after just a week in the office left having special needs.
for China; she returned only to imme-
diately head up to Itasca and Koochich-
ing Counties. She provides wonderful STAFF TRAVELS
perspective in her primary project of AND ADVENTURES
evaluating Minnesota’s Home and Com- The Improve Group and DHS staff members Congratulations to Deborah and Andy
visit many Minnesota landmarks on their Mattila on their marriage! After a
munity Based Services (HCBS) programs travels for the Waiver Review Project.
for the Department of Human Services beautiful ceremony, they honeymooned
(DHS). in Belize (see our blog at http://www.
NEW PROJECTS theimprovegroup.com/weblog/ for more
We have recently started three large wedding & honeymoon pictures)
evaluation projects. In collaboration
with the Department of Human Services Rebecca Stewart recently returned from
(http://www.dhs.state.mn.us), we India, where she spent two months
are monitoring Home and Community working with a local NGO in conducting
Based Services in each of Minnesota’s 87 a program evaluation. See her articles
counties and in several health plans and about her experience on pages 1 and 2.
tribal governments. For this project we
are reviewing implementation by each Brooke Ahlquist trained and completed
agency to ensure compliance with both the Twin Cities Breast Cancer 3-day
state and federal laws. This project walk through much of the Twin Cities,
involves collection and analysis of data starting at the Minnesota Zoo in Apple
The Improve Group staff enjoyed their annual from staff, participant records, pro- Valley and ending at the State Capitol.
retreat at Hyland Lake Park in August. vider contracts and state system data.
Starting in 2007, we will begin collect- CONFERENCES
Eric Wong, also a recent graduate of the ing data directly from participants in The Improve Group will be presenting
University of Minnesota who completed interviews or focus groups. This project in two conferences this fall. Our work
his Master’s in Public Policy in June, is is expected to last through 2011. with Girl Scouts USA has been selected
particularly interested in economic pol- by the American Evaluation Association
We have begun a new project with one
icy issues. Prior to joining the Improve (http://eval.org/) for a presentation
of our long-term clients the Girl Scouts
Group he conducted an analysis of on challenges and lessons in cluster and
of the USA. In 2005, we completed
projected revenue losses due to policy multi-site program evaluation.
an evaluation of their Girl Scouts in
changes for the Minnesota Department
Detention Centers program and in June
of Revenue and researched the impact We will also be working with DHS to
of this year we began an evaluation
of need-based financial aid on academic present our work on the statewide
of the Project Anti-Violence Education
progress. His economic background HCBS evaluation at the 22nd National
(P.A.V.E.) program, offered by 26 coun-
has proven valuable in thinking about Home and Community Based Services
cils around the U.S. and in Europe. We
expenditures in the HCBS program. His Conference (www.nasua.org/waiver-
are collecting data from each council,
parents own the Golden Inn restaurant conference). Because the HCBS evalu-
using retrospective pre-test surveys
in Superior, WI, and he helps out there ation responds to a recently expanded
with girls from 10 councils and success-
a few times each month. Between requirement for state accountability in
case site visits to 6 councils. This evalu-
the HCBS evaluation and his travels to HCBS programs, we will provide details
ation focuses on personal growth and
Superior, Eric definitely gets the most of the methods used and our findings so
leadership development outcomes.
mileage of anyone in our office. far to a national audience.
One of our long term clients, the
Performing Arts Workshop,
WORKSHOP SERIES 5
Much of our client base
comes from organizations
that have funds, either Dr. James Riedel, Girl Scouts Research Institute
targeted or as part of a By Marian Kimball Eichinger
program fund, specifically
for research, evaluation James Riedel, Ph.D., is a Senior Researcher for MKE: Why is this topic of research important
and planning. However, the Girl Scouts Research Institute (GSRI). GSRI to you?
we receive a number of is a center for research and public policy infor- JR: Girl Scouting has a 95-year history of doing
phone calls from smaller mation on the healthy development of girls. wonderful things for girls. I believe in all of the
organizations that want James currently serves as a liaison between initiatives of the organization. Anti-violence
to do research, evaluation GSUSA and The Improve Group which is education is important because it cuts across
and planning but don’t conducting the evaluation of GSUSA’s Project every subgroup of humanity. It is a world issue.
have specific budgets to Anti-Violence Education (P.A.V.E. the Way).
take on those projects.
MKE: Why is this research important for girls
MKE: How long have you been with GSUSA and our society overall?
In response to this need, and what do you do for them?
the Improve Group will be JR: The work of a girl-only organization is
JR: I have worked at GSUSA for eight years, crucial, and our own research, The Ten Emerg-
offering workshops on basic
beginning as a research and evaluation analyst ing Truths (2002), uncovered several reasons.
methods and tools. We are
in 1998. My primary work is program evaluation. Girls need empowerment and a climate in which
working with Aurora Con-
I investigate the benefits to girls of participat- they can connect with each other. This is sup-
sulting, Catalyst Consult-
ing in various programs. Some of the programs I ported through a “by girls, for girls” approach.
ing, DeYoung Consulting,
have evaluated were designed to produce equity Furthermore, certain topics are specific to girls
and the Word Applied to
in science and technology; develop leadership and others are of a very sensitive nature, calling
develop a series for small
and communication skills; or prevent youth for the need of a secure, safe and comfortable
non-profit and public agen-
violence. place to learn and grow.
cies, and plan to roll the
series out in January 2007.
MKE: What is your background and how did it The mission of Girl Scouting is “to build girls of
Workshop topics will cover: bring you to the GSRI to perform research for courage, confidence and character, who make
• Basic evaluation tools girl-based programming? the world a better place.” It is important
and planning JR: I have conducted evaluations for 14 years. for girls that we measure the outcomes and the
While my concentration is evaluation, the GSRI processes of their participation. Without dem-
• Logic model design for
does original research studies and literature onstrating that the girls’ involvement produces
programs and organiza-
reviews as well. Examples of recent research positive results, there are only gut-feelings and
conducted by my colleagues include: hear-say as evidence that membership is worth-
• Business writing, report- while. Girls, volunteers, staff and funders all
ing and presentations • The New Normal? What Girls Say About look for more when deciding to what organiza-
• Effective teams, meetings Healthy Living (2006); tions they devote time and money.
and work planning • Voices of Volunteers 18-29 (2003); and
• Building staff, board and • Feeling Safe: What Girls Say (2003). For society, well, there are a lot of youth-devel-
volunteer leaders opment programs and extra-curricular activities
I earned my Ph.D. in Measurement, Statistics that pre- and adolescent girls can get involved
The workshops will provide and Evaluation at the University of Delaware. in. It is important to show that the program ac-
both learning and practical From 1996-1998, I evaluated government-funded tivities of the largest girl-serving organization in
opportunities, and be ap- initiatives in some of New York City’s economi- the world yield the growth that it purports to.
plicable for staff and board cally poorest elementary and middle schools,
members. Because these working directly with children and the adults For information about the Girl Scouts USA
workshops are in develop- responsible for their care and enrichment. My and studies by the Girl Scouts Research
ment, we will offer the doctoral studies focused on the engagement of Institute, visit http://www.girlscouts.org/.
first series at a discount in middle and high school children. I was always
exchange for detailed feed- interested in education, be it through the for-
back from participants. If mal K-12 system or the more non-formal
you are interested in either Scouting opportunities.
the series or a single work-
shop, please let us know by
emailing Susan Murphy at
Upcoming Grant and Requests for Proposals
By Susan Murphy
Honoring fall’s arrival and the departure of our summer interns to their respective colleges, the Improve Group
features the following grant opportunities in education. If you need advice on applying for grants or have ques-
tions about the notices listed above please contact Susan Murphy at email@example.com.
THE EDUCATION FOUNDATION QWEST FOUNDATION TOYOTA TAPESTRY GRANT PROGRAM
OF AMERICA (EFA) Qwest for Education focuses on enrich- Sponsored by Toyota Motor Sales,
EFA makes grants to qualifying non- ing the lives of children by supporting U.S.A., Inc. and administered by the
profit organizations that have tax-ex- solid preK-12 educational programs. National Science Teachers Associa-
empt status and those that are not They consider programs that: tion this fund offers 50 grants of up
private foundations. Projects must • Effectively use technology to to $10,000 each and a minimum of 20
be located within the United States. improve preK-12 public school “mini-grants” of $2,500 each to K-12
Areas of interest include, but are not instruction teachers of science. To apply, qualified
limited to, the environment, repro- teachers must write a Toyota TAPES-
ductive freedom, theatre, educa- • Promote innovative models to TRY proposal and submit it for receipt
tion, medicine, drug policy reform, strengthen preK-12 public school at NSTA by January 18, 2007. Grants
democracy, peace & national security education are awarded in three categories: Envi-
issues and human services. Important • Improve the skills and leadership ronmental Science Education; Physical
characteristics considered by EFA are of educators and parents Science Applications; and Literacy and
an organization’s record of achieve- • Promote innovative early childhood Science Education. Go to http://www.
ment, intended broad impact, sound education programs nsta.org/programs/tapestry/program.
financial practices, increasing inde- htm for more information.
pendence, and correspondence with Requests are reviewed on an on-going
EFA objectives. Grant guidelines can basis and grants generally range from
be found at http://www.efaw.org/ $500 and up. To learn more about this
Inquiry%20Guidelines.htm. opportunity visit http://www.qwest.
Phone and Fax toll-free at:
Bloomington, MN 55425-1805
2051 Killebrew Drive, Suite 620
Marian Kimball Eichinger
Leah Goldstein Moses
President & CEO
PERMIT NO. 4656