The ABCs of Measuring Program Impact: AC ase Study of TimeBank by P84oEH


									  The ABCs of Measuring Program Impact:
A Case Study of TimeBank Outcome Research
                     June 30, 2010

              Mashi Blech, Director
        VNSNY Community Connections TimeBank

         Alexis Stern, Research Analyst
      Center for Home Care Policy & Research, VNSNY

                             Community Connections TimeBank
          The TimeBank Model

•   Alternative community currency based
    on the hour (pay it forward model)
•   1 hour of service = 1 earned TimeBank
•   International model with 300+ banks in
    23+ countries (grass roots to nationwide)
•   Over 1,000,000 hours exchanged each
    Core Values of TimeBanking

1. Assets: Everyone has something of
   value to contribute
2. Redefine Work: Preserving community
   is real work
3. Reciprocity: Acknowledge and honor
   providers and receivers
4. Social Capital: Social networks require
   ongoing investment
        Goals of TimeBanking

   Mobilize available time and talent
   Match unmet needs with untapped resources
   Improve quality of life
   Strengthen communities
   Aging in place strategy
 Why Conduct Program Research?

 Sometimes data speaks louder
  than words
 Can we do better?
 Can we do more?
 Engaging the stakeholders
  Key Research Decisions

• For whom?
• About what?
• From whom?
• What is your minimum “burden of proof”?
   – Blinded, randomized studies
   – Control groups
   – Sample size and response rate
• Retrospective and prospective studies
• Process measures and outcome measures
• Collecting quantitative and qualitative data
• Data collection methods
   – Focus Groups
   – Case Studies
   – Surveys
            Survey Tool

• Use validated tools whenever
• Prioritize to keep it manageable
• Closed and open ended questions
• Don’t lose track of target
  audience and goals of research
    Use of Research Partners

• Pros and cons
• Potential partners
• Financial considerations
       Assessing Your Resources
• Time frame
• Availability of research personnel
   – Development of strategy/tools
   – Data collection: interviewers, focus group
     leaders etc.
   – Training/oversight of those collecting data
   – Data entry
   – Data analysis
• Software/Hardware
• Incentives
       The 2009 VNSNY Community
      Connections TimeBank Survey

• One year project funded by the Fan Fox
  & Leslie R. Samuels Foundation
• Telephone survey of 61 active TimeBank
  members aged 60 and older
• Two interviewers (one Spanish-speaking)
• Survey tool combined questions from a
  tested tool and our own additions
  The 2009 VNSNY Community
Connections TimeBank Survey Tool
  The 2009 VNSNY Community
Connections TimeBank Survey Tool
 Results of the 2009 VNSNY Community
    Connections TimeBank Survey

• High response rate
• Outcomes exceeded expectations for
  every measure
• Extremely valuable outcome results
• Powerful responses to open ended
              Data Highlights

• All members surveyed said that they benefit
  from being a Time Bank member
  – 69% said that they benefitted a great deal
  – Non-English speakers and those with lowest
    income reported the greatest benefit
• Members are engaged
  – 96% said that they provide services
  – 67% said that they receive services
           Data Highlights (cont’d)
Measure                            Responded “Increased”
                                   or “Increased Greatly”
Number of Friends                           87%
Feelings of Belonging to the                85%
Quality of Life                             82%
Participation Social Gatherings/            77%
Neighborhood Events
Mental Health and Well-Being                72%
Self-Confidence/Self-Esteem                 64%
Physical Health and Well-Being              48%
Data Highlights (cont’d)
Data Highlights (cont’d)
Data Highlights (cont’d)
         Data Highlights (cont’d)
• Confirmation that members underreport
  exchanges and hours. Of the 51 respondents
  who said that, outside of meetings, they talk to,
  meet with, or exchange services with friends
  they made through the TimeBank:
   – Only 10% said they always report the
  – 35% said they sometimes report the
  – 55% said they never report the exchange
      Qualitative Data Highlights
• “Now I don’t feel like an old woman without
  a future.”
• “If it wasn’t for my friends from the TimeBank
  who encouraged me to continue on, I would
  have died of pain and loneliness.”
• “The TimeBank has given me new energy to
  go forward every day.”
• “The TimeBank, for me, has been the best
  thing that can happen to someone to feel
        Dissemination of Results
• Executive Summary
• Internal – newsletters; presentations to senior
  management and at key department
  meetings; share results with participants and
  volunteer group as a whole
• External- potential funders and volunteers;
  publishing options include industry journals,
  research publications, books, press releases;
              Lessons Learned
• The investment and risk paid off
• Interviewer skills are key
• Addition of Spanish language survey tool was
  well worth the effort
• Getting internal and external buy-in was critical
  to our success
• Figuring out what’s important to measure is
• Deciding on how to slice the data is not always
          So What’s Next?

• Ongoing measurement including
  collection of baseline data?
• Periodic surveys?
• Specific issues   (e.g., underreporting)

To top