Executive Summary Family Engagement Program Indicators of Change

Document Sample
Executive Summary Family Engagement Program Indicators of Change Powered By Docstoc
					                                                          Executive Summary
                              Family Engagement Program: Indicators of Change
Evaluation Report • September 2005 – November 2006 • Fred J. De Jong, MAPA, MSW, Ph.D., ACSW; Rachel Venema, MSW; Stephanie Richards, BSW Program

Project Management
                                       Family Engagement Program
                                       Program Description

                                       FEP supports single mothers who struggle with recovery from a substance use dis-
Service Providers                      order. Since the program’s inception, 103 such families have received therapeutic
                                       support and intensive case management. The program seeks to not only support the
                                       mother’s recovery, but also ameliorate the effects of current or past active addiction on
                                       the children in these households. Most of all, the project hopes to protect these chil-
                                       dren as they mature from experiencing similar addictive behaviors, to keep children at
                                       their appropriate grade level in school and out of trouble with the law. In this manner,
                                       the program may break the cycle of cross-generational addiction, poverty, deviance
Local Evaluation                       and other social problems that are so often concomitant to substance abuse.

                                       Evaluation Background

                                       Demonstrating change in a family requires skill, perseverance, a willingness to change
                                       among both participants and staff, and agreement upon goals and how to measure them.
                                       The first 3 years of FEP struggled with these prerequisites through the resignation or
                                       termination of the first two evaluators. At the same time, therapeutic staff at the two
                                       contracted sites resisted heavy data collection requirements and had no agreed upon
                                       goals across participating households or even between the two providers. Beginning in
                                       spring, 2005, contracted provider staff and Network180 professionals brought in Calvin
                                       College as the third evaluator and formed a collaborative process involving provider
                                       therapists and case managers. All participants in this six-month process pain-stakingly
                                       developed measures of progress and success in each area of family functioning which
                                       they regard from experience as the most important and/or problematic for participating
                                       FEP families. After pre-testing these measures, developing data collection protocols and
                                       conducting training, FEP providers implemented these family outcome measures begin-
                                       ning October, 2005. Although a total of 103 families have received service under FEP 31 ,
                                       families have data using the agreed upon family outcome measures for at least 9 months
                                       at the time that analysis could begin for the first evaluation report.

Executive Summary Family Engagement Program: Indicators of Change

   Key Baseline Findings for Program Clients
   Who is being served by the Family Engagement Program?
   Summary of Household Characteristics at Intake

   At the time of Intake, usually before any FEP service begins, the mothers commonly show
   risk factors inhibiting their own recovery from substance abuse and for negative family
   functioning. These indicators of need at baseline or intake for FEP include:

          ➡ about ½ of mothers are involved in corrections
          ➡ the majority are not working outside of the home
          ➡ most mothers set goals, but therapists report only about ½ have the
            confidence/determination to meet their goals

          ➡ one-quarter do not live in a safe environment
          ➡ one-quarter do not have stable living arrangements
          ➡ about ½ of clients do not have sufficient income to meet their basic
          ➡ one in five households experience domestic violence

          ➡ two-thirds do not regularly spend quality time with their children
          ➡ very few receive child support
          ➡ therapists point out that almost no families entering the FEP
            regularly have supportive social networks

          ➡ two-thirds do not have a support person who contributes to family
            finances, about one third has a support person who abuses drugs or
            alcohol and about half have a significant other who abuses drugs or
          ➡ on average every family has at least one child not living them, either
            in publically funded foster care or in other family placements by court

   See the Full Report for detailed statistics: “Summary Report at Baseline” and
   “Frequencies at Baseline.”

Executive Summary Family Engagement Program: Indicators of Change

   Early Indicators of Effectiveness
   Does the FEP Work?
   Participating FEP families show 7 areas of statistically significant positive change.

                               Significant Positive Change
                                Children Living in the Household
                                   Shelter/Living Environment
                             Substance Use/Treatment Acceptance
                                  Relationships/Support System

   The greatest statistically significant change in this first analysis period is the dramatic
   drop in out of home placement of children among the participating families. In ad-
   dition, mothers show significant improvement in number of hours of paid work per
   week, reduction in substance use, greater treatment acceptance, better shelter/living
   environment, improved health, improved finances, increased social support, more
   quality time with their families, and more regularly provided age-appropriate and
   child-specific monitoring for their children.

Executive Summary Family Engagement Program: Indicators of Change

   After at least nine months in the Family Engagement Treatment Program…
      1. Children in Household: Significantly more children live in the family’s household
         (85.5% compared to 59.8% at Intake).

      2. Living Environment: Both mothers and therapists report significantly more stable,
         safe, and orderly home environments.

      3. Treatment: Mothers show increased perceptions of the benefits of treatment/therapy.
         They have developed more skills to set goals, keep appointments more regularly, and
         increasingly trust their workers.

      4. Support System: Mothers report a more regular and healthy support system.

      5. Health: Mother’s perceptions of their family’s health significantly improved. Mothers report
         increases in their own ability to care for their children’s and their own health needs.

      6. Finances: Mothers show positive change in their attitudes and behaviors regarding
         finances, demonstrating awareness that spending decisions affect the family’s financial
         situation and more often applying effective budgeting strategies in daily life.

      7. Employment: Mothers show a dramatic increase in the average number of hours of
         paid work per week (Baseline-10 hours to Follow-up-23 hours).

                                                              Percentage Change

                                          30                        28.11
                                          25                                                                                22.67
                                          10          8                                                    7.68


                                   Number of Households
                                                    nt              ors              em             lth             nt           eek
                                                 me             cat              yst            Hea              me          rs/W
                                        nv  iron         tI ndi             rt S        Health            an age         Hou
                                      E               en                 po
                                  ing             atm                Sup                            ial
                              Liv             Tre                ger                            anc
                       ro ved         iti ve             S tron                          ter
                                                                                             Fin Paid Hours/Week
                   Imp            Pos                                                Bet
                                       Stronger Support System
           Improved Living Treatment Indicators Financial Management
                  Positive Environment

                                                     Focus on the change from Intake to Follow-up within each topic area.
                                                     The topic areas should not be compared with each other since the
                                                     sumscore range varies depending on the scale/topic.

   See Full Report Section II. Using FEP Data for Program Feedback for detailed change
   analysis statistics.

Executive Summary Family Engagement Program: Indicators of Change

   Areas of Non-significant Change in at least 9 months
   Areas that demonstrated little change include the mother’s emotional
   health, parenting, family functioning, and receipt of public assistance.

                                   Non-Significant Change
                                      Emotional Health
                                      Family Functioning

   This initial follow-up report points out what could have been expected—change
   would occur at a different pace depending on the presenting issue and characteristics
   of each family. Some change, such as finding better housing, turns out to be an easier
   fix, than other changes, such as improving parenting skills. Family dynamics, such as
   children re-entering the household and entering their teen years (which can be try-
   ing even under the best of circumstances), impact self-report and observed parenting
   skills. Such behaviors, developed over the years and influenced by one’s own parents’
   practices, are not easily changed in the short run or in the long term. Expectations of
   change must be tempered, at a minimum, with what could be expected of families not
   under duress as compared to those families in FEP who do experience much hardship.

   In addition, many of the parenting questions in the evaluation focus on the children’s
   perception of parenting and mothers may rate themselves low due to increased expec-
   tations of themselves. A lack of child care and child support, as noted at baseline and
   follow-up, may also contribute little change in perceptions of parenting and family

   Early indicators of change in FEP families may also indicate a kind of Maslow’s hier-
   archy of needs. Mothers and families are showing significant improvement in many
   areas that represent a progress in meeting the family’s basic needs. Once these basic
   needs are met, as some of the significant findings show so far, families have the chance
   to focus on higher order needs, to start opening up in therapy and to address such
   next level challenges as emotional health issues, parenting, or family functioning.

   In conclusion, families generally become successful in stabilizing the household
   through concrete support during the first year in the FEP program; what remains are
   more complex and long-term issues.

Executive Summary Family Engagement Program: Indicators of Change

   Continue the current measures, data collection protocols and reporting
   With only 1 year of experience with the collaboratively developed measures, early
   results successfully show areas of possible success and also areas that need greater
   emphasis over the long term. It is difficult to develop measures that can differentiate
   progress and client change, as the current set does. In addition, the measures begin
   with a high face validity, that is, practitioners and mothers agreed upon them as indi-
   cators of success at Intake, so the measures are likely accurate for the setting. Finally,
   the protocol and training for data collection is in place, and heavily supported by eval-
   uation staff. Any disruption of the current level of data collection support, or upon the
   agreed upon measures, is likely to inhibit future long-term evaluation.

   Focus FEP intervention on more complex, hard-to change outcomes.
   Clearly the next stage of the evaluation, as currently structured, should provide technical
   assistance to the provider agencies not only to continue current support for basic family
   needs, but then add more intensive or long term strategies for change. These strategies
   should target the more complex aspects of the mother’s emotional health, various mea-
   sures of family functioning, and concentrated parenting skills training and support.

   Track children individually on grade-level and deviance.
   Current measures do not individually track each household’s children by name, grade
   level and other child specific statuses. As is, the Children’s Checklist applies to all the
   household’s dependent minors in the aggregate. Calvin evaluators initially proposed
   and utilized child specific tracking; however, the reporting and analysis burden caused
   both researchers and clinical staff to omit individual child tracking. This recommenda-
   tion is for evaluators and providers to work together to find a least burdensome man-
   ner to collect the minimum data required for each child (i.e. living in the household,
   juvenile justice status, grade level, etc.).

   Stronger evaluation design with matched comparison group
   Positive results from the first year of the evaluation provide a rationale for adding a
   comparison group of households to the evaluation analysis. If such a group receiving
   conventional support could be found and matched by several important FEP house-
   hold characteristics, outcome findings could have greater claim to causality. In other
   words, addition of a comparison group would allow researchers and fundors more
   confidence that the FEP intervention is actually the stimulus for the observed family
   changes rather than being due to other stimuli currently existent in the families’ envi-
   ronments. In more technical terms, the addition of the comparison group would move
   the evaluation design from a pre-experimental one-group pre-test post-test design to
   a quasi-experimental matched comparison group design, which has greater protection
   from threats to internal validity.