National Wraparound Initiative by ewghwehws

VIEWS: 2 PAGES: 87

									     Wraparound Fidelity Monitoring in
             the Massachusetts CBHI
                                  Summary of 2011 Results

                                                 August 12, 2011




Eric Bruns and April Sather
University of Washington
Wraparound Evaluation and Research Team
ebruns@uw.edu / sathea@uw.edu
With thanks to Andrea Gewirtz and Melissa King
 Purpose for this Meeting


1. Very quick review of wraparound
   fidelity assessment and research
2. Review data from Massachusetts
   fidelity monitoring
3. Discuss what has been found and what
   it means
Wraparound Competence:
What do we want to measure?
  Wraparound Principles

1. Family voice and choice
2. Team-based
3. Natural supports
4. Collaboration
5. Community-based
6. Culturally competent
7. Individualized
8. Strengths based
9. Persistence
10.Outcome-based
                      Walker, Bruns, Adams, Miles, Osher et al., 2004
    Wraparound Competence
  Implementing the practice model:
  The Four Phases of Wraparound
Phase
 1A         Engagement and Support

   Phase
    1B                 Team Preparation

        Phase
          2                  Initial Plan Development

            Phase
                                 Implementation
              3
                    Phase
                                       Transition
                      4


                                Time
Phase 1 A and B

      Phase 1 :   Engagement and Team Preparation


  •    Care Coordinator & Family Partner meets with the
       family to discuss the wraparound process and listen to
       the family’s story.
  •    Discuss concerns, needs, hopes, dreams, and
       strengths.
  •    Listen to the family’s vision for the future.
  •    Assess for safety and make a provisional safety plan if
       needed
  •    Identify people who care about the family as well as
       people the family have found helpful for each family
       member.
  •    Reach agreement about who will come to a meeting to
       develop a plan and where we should have that
       meeting.
Phase 2

    Phase 2: Initial Plan Development


  • Conduct first Care Planning Team (CPT) meeting with people
    who are providing services to the family as well as people who
    are connected to the family in a supportive role.
  • The team will:
     – Review the family vision
     – Develop a Mission Statement about what the team will
       be working on together
     – Review the family’s needs
     – Come up with several different ways to meet those
       needs that match up with the family’s strengths
  • Different team members will take on different tasks that have
    been agreed to.
Phase 3

          Phase 3: Plan Implementation


  • Based on the CPT meetings, the team has created a written
    plan of care.
  • Action steps have been created, team members are committed
    to do the work, and our team comes together regularly.
  • When the team meets, it:
     – Reviews Accomplishments (what has been done and
       what’s been going well);
     – Assesses whether the plan has been working to achieve
       the family’s goals;
     – Adjusts things that aren’t working within the plan;
     – Assigns new tasks to team members.
Phase 4

                 Phase 4: Transition

  • There is a point when the team will no longer need to
    meet regularly.
  • Transition out of Wraparound may involve a final
    meeting of the whole team, a small celebration, or
    simply the family deciding they are ready to move on.
  • The family we will get a record of what work was
    completed as well as list of what was accomplished.
  • The team will also make a plan for the future, including
    who the family can call on if they need help or if they
    need to re-convene their team.
  • Sometimes transition steps include the family and their
    supports practicing responses to crises or problems that
    may arise
Wraparound Fidelity Index, v.4

• Items on the principles and core activities, organized
  by the 4 phases of wraparound
   – Engagement: Did you select the people who would
     be on your youth and family team?
       • Principle = Team based
   – Planning: Does the plan include strategies for
     helping your child get involved with activities in the
     community?
       • Principle = Community based
   – Implementation: Does the team evaluate progress
     toward the goals of the plan at every team meeting?
       • Principle = Outcome based
   – Transition: Will some members of your team be
     there to support you when formal wraparound is
     complete?
       • Principle = Persistence
 Team Observation Measure

• Consists of 20 items, with two items dedicated
  to each of the 10 principles of wraparound.
• Each item consists of 3-5 indicators of high-
  quality wraparound practice as expressed
  during a care planning team meeting.
• Internal consistency very good
• Inter-rater reliability found to be adequate
  (Average 79% agreement for all indicators)
• Correlates well with WFI scores
   – In previous studies and evaluations…
Document Review Measure


• Mass version consists of 26 items
• Each wraparound principle linked to several items
• Scale = 0-3, with criteria for each point on the scale
• Source material = documentation (electronic or
  paper) related to youth’s wraparound process
   – Strengths, needs, culture discovery
     documentation
   – Wraparound plan of care
   – Safety plan
   – Transition plan
   – Progress notes
Measures of the WFAS seem to tap
into the same constructs
 Mean WFI-4 and TOM total scores by site in one state’s QI project
        100%

          90%

          80%

          70%

          60%

          50%
                   Site 1   Site 3   Site 5   Site 6   Site 7   Site 8   Site 9   Site 10
WFI Combined       84%      77%      79%      79%      69%      72%      72%       81%
Team Observation   93%      84%      83%      78%      67%      56%      63%       78%
 WFAS Measures seem to be related to
 what we would expect

• Families who experience better outcomes have
  staff who score higher on fidelity tools (Bruns,
  Rast et al., 2006; Effland, McIntyre, & Walton,
  2010)
• Wraparound initiatives with positive fidelity
  assessments demonstrate more positive
  outcomes (Bruns, Leverentz-Brady, & Suter,
  2008)
• Wraparound initiatives with better support to
  implementation have higher fidelity scores
    What does it take to get high
    fidelity scores?


• Training and coaching: Found to be
  associated with gains in fidelity and
  higher fidelity
• Community, Fiscal, and Service system
  support: Better developed supports =
  fidelity scores
• Sticking with it: Fidelity scores tend to
  increase over time and with persistence
  of effort
   – But… they can also backslide…
Caregiver WFI Fidelity over time in NV


                             100%
Average WFI Fidelity Score




                              90%
                              80%
                              70%
                              60%
                              50%
                              40%                                       86%
                                                       72%                               73%
                              30%       64%
                              20%
                              10%
                               0%
                                    Pre Training   Training Only    Training and   After going to
                                       (2001)         (2002)       Coaching (2004) scale (2007-09)

                                     Bruns, Rast, Walker, Peterson, & Bosworth (2006).
                                     American Journal of Community Psychology.
 What can we do with the data?


• Measurement as communication
  – What are we attempting to do for
    families?
  – How well are we stacking up against
    benchmarks? What is holding us back?
  – How well are we doing over time?
  – What is associated with positive
    outcomes in Massachusetts?
 Measurement for quality
 improvement

• Examine data at the program level to discuss
  what is and is not happening consistently
     • What can be reinforced in individual and group
       supervision?
     • What are the team, program, and resource
       barriers?
     • In what areas do we need additional training?

  – At the community or system level by
    collaborative groups, along with reports of
    team, program, and resource barriers
    reported up from CSAs or programs
 What kind of data get used?


• What kind of data?
  – Item scores – Ns who endorse each
    response option (Yes, No, Somewhat)
  – Members on teams
  – Specific plan of care elements
  – Qualitative feedback on open-ended
    items
Part 2: Results from Massachusetts

     Scores on the WFI, TOM, and
              Mass-DRM
Total scores on the WFAS instruments


                 100
                  80

     Percent of 60
    total fidelity 40

                  20
                   0
                        WFI   TOM   DRM
 Massachusetts 2010     78    83    61
 Massachusetts 2011     77    85    74
 National mean          74    77     -
Correlations between WFI, TOM, and
DRM across Massachusetts CSAs

      2   • WFI – TOM = .121
      0
      1
          • WFI – DRM = -.473 (p<.001)
      0   • DRM – TOM = .012

      2   • WFI – TOM = -.256
      0
          • WFI – DRM = .326 (p<.05)
      1
      1   • DRM – TOM =.168
      WFI-DRM association at a site level


                                  2010 –                                              DRM / WFI       -   2011
                              
                                                                    

      80.00                                                                
              

                                               


                                                                        
                                                               
      70.00
                                    
                                                           
                                                                             
                                                                                      
DRM




      60.00                                                                      
                                             
                                                            
                                               
                                                        
      50.00                             
                                                                                          
                                                                
                                                                                             


                                                                                      
      40.00
                                                                    
                                                                                


                  72.00            76.00                            80.00                     84.00

                                        WFI
WFI-DRM association at a site level


             2011 –
             DRM/TOM
Wraparound Fidelity Index

       Massachusetts Overall
     WFI-4 Total Fidelity Scores
     All CSAs

    2010 N = 637 / 2011 N = 587

            Total Phase                    Principle

            Mean    Eng   Pln   Imp   Tr   FVC   TB   NS   Col CB    CC   Ind SB    Pe   OB
            Over-                                                         iv        r
            all
Mass        78      86    82    79    64   88    84   53   89   74   95   70   83   83   63
2010
Mass        77      85    82    77    64   85    82   52   89   74   93   73   79   78   65
2011
Nat         74      75    74    79    66   82    71   61   84   69   90   64   82   79   64
Mean
(CG only)
WFI Fidelity by Phase
All CSAs 2010 N=637 / 2011 N = 587
         90%
         80%
         70%
         60%
         50%
         40%
         30%
         20%
         10%
          0%
                Total   Eng   Plan   Impl   Trans
     2010       78%     86%    82%   79%    64%
     2011       77%     85%    82%   77%    64%
     Nat Mean   74%     75%    74%   79%    66%
WFI Fidelity by Principle
All CSAs                  2010 N=637 / 2011 N = 587

     100%

         90%

         80%

         70%

         60%

         50%

         40%

         30%

         20%

         10%

         0%
               Total   FVC   TB    NS    COL   CB    CC    IND   SB    PER   OB

  2010         78%     88%   84%   53%   89%   74%   95%   70%   83%   83%   63%
  2011         77%     85%   82%   52%   89%   74%   93%   73%   79%   78%   65%
  Nat Mean     74%     82%   71%   61%   84%   69%   90%   64%   82%   79%   64%
  Item Scores: Engagement
   All CSAs


ITEMS                                                         2010   2011    NAT
                                                                            MEAN

CG1.1 - When you first met ICC, were you given time to talk
about strengths *and* Did this process help you appreciate?   1.78 1.74 1.65
CG1.2 - Before your 1st team meeting, did your ICC fully
explain the WA process and the choices you could make?        1.78 1.82 1.68
CG1.3 - At beginning of wrap process, did you have a
chance to tell WF what things have worked in the past?        1.83 1.78 1.75
CG1.4 - Did you select the people who would be on your WA
team?                                                         1.41 1.34 0.86
CG1.5 - Is it difficult to get team members to attend team
meetings when they are needed?                                1.66 1.65 1.57
CG1.6 - Before your 1st wrap team meeting, did you go
through a process of i.d.ing what leads to crises for child   1.81 1.90 1.52
and family?
    Item Scores: Planning
    All CSAs

ITEMS                                                                            NAT
                                                                  2010   2011   MEANS

CG2.1 - Did you and your team create a written plan that
describes how the team will meet your child's needs? *and* Do     1.83   1.87   1.64
you have a copy?
CG2.2 - Did the team develop any kind of written statement
about what it is working on with your child and family? *and*
Can you describe what your team mission says?
                                                                  1.76   1.76   1.56

CG2.3 - Does your wrap plan include mostly professional
services?                                                         .99    1.10   0.61
CG2.4 - Are the supports and services in your WA plan connected
to the strengths and abilities of your child and family?          1.80   1.72   1.74
CG2.5 - Does the wrap plan include strategies for helping your
child get involved w/ activities in his/her community?            1.31   1.26   1.24
CG2.6 - Are there members of your wrap team who do not have
a role in implementing your plan?                                 1.73   1.73   1.67
Item Scores: Planning
All CSAs


ITEMS                                                  2010   2011
                                                                      NAT
                                                                     MEAN


CG2.7 - Does your team brainstorm many strategies
to address your family's needs before selecting one?   1.79   1.79 1.73
CG2.8 - Is there a crisis plan? *and* does this plan
specify how to prevent crisis?                         1.48   1.57 1.43
CG2.9 - Do you feel confident that, in crisis your
team can keep your child in the community?             1.57   1.58 1.5
CG2.10 - Do you feel like other people on your team
have higher priority than you in designing your wrap   1.83   1.73 1.53
plan?
CG2.11 - During planning process, did team make
enough time to understand values? *and* Is your        1.89   1.84 1.73
wrap plan in tune w/ family's values?
Item Scores: Implementation
All CSAs



ITEMS                                                    2010   2011
                                                                        NAT
                                                                       MEAN
CG3.1 - Are important decisions made about your
child or family when you are not there?
                                                         1.90   1.86   1.64
CG3.2 - When your wrap team has a good idea for
support, can they find resources or make it happen?
                                                         1.58   1.54   1.7
CG3.3 - Does your wrap team get your child
involved w/ activities they like and do well?
                                                         1.05   0.95   1.2
CG3.4 - Does the team find ways to increase the
support you get from friends & family?
                                                         1.09   1.13   1.22
CG3.5 - Do the members of your team hold each
another responsible for doing their part?
                                                         1.73   1.73   1.7
CG3.6 - Is there a friend or advocate of your child or
family who actively participates in wrap team?
                                                         .68    .66    0.95
CG3.7 - Does your team come up w/ new ideas?
*and* Does your team come w/ ideas when                  1.75   1.70   1.74
something's not working?
Item Scores: Implementation
All CSAs



ITEMS                                                              2010   2011
                                                                                  NAT
                                                                                 MEANS

CG3.8 - Are the services and supports in your wrap difficult for
you family to access?                                              1.61 1.66 1.54
CG3.9 - Does the team assign specific tasks to all team
members at end of mtng? *and* Does team review team                1.67 1.63 1.59
member's follow-through at next mtng?
CG3.10 - Do members of your team always use language you
can understand?                                                    1.96 1.94 1.93
CG3.11 - Does your team create a positive atmosphere around
successes and accomplishments at each team meeting?                1.88 1.91 1.86
CG3.12 - Does your team go out of its way to make sure all
members present ideas and participate in decisions?                1.82 1.84 1.67
CG3.13 - Do you think your wrap process could be discontinued
before you're ready?                                               1.48 1.32 1.35
CG3.14 - Do all the members of your team demonstrate respect
for you and your family?                                           1.96 1.91 1.88
CG3.15 - Does your child have the opportunity to communicate
their own ideas when it comes to decisions?                        1.49 1.39 1.71
   Item Scores: Transition
    All CSAs

ITEMS                                                          2010   2011
                                                                               NAT
                                                                              MEAN

CG4.1 - Has your team discussed a plan for how wrap will
end *and* Does your team have a plan for when?                 .46    0.70   0.68
CG4.2 - Has the wrap process helped your child develop
friendships w/ other youth                                     .94    .92    1.2
CG4.3 - Has the wrap process helped your child to solve
their own problems?                                            1.09   1.02   1.3
CG4.4 - Has your team helped you and your child prepare
for major transitions?                                         1.50   1.64   1.35
CG4.5 - After formal wrap ends, do you think the process
will be able to be 're-started' if you need it?                1.82   1.80   1.61
CG4.6 - Has the wrap process helped your family to
develop or strengthen relationships that will support you      1.45   1.42   1.49
when WA is finished?
CG4.7 - Do you feel like you and your family will be able to
succeed on its own?                                            1.33   1.41   1.22
CG4.8 - Will some members of your team be there to
support you when formal wrap is finished?                      1.60   1.40   1.65
 WFI – Summary

• Overall: Significantly higher than national mean
• Overall, no change from 2010
• Relative strengths:
   – Engagement, Planning phases
   – Family voice and choice, team based, collaborative,
     Individualized
   – Initial engagement, family selecting team, crisis planning,
     vision/mission statements, balance of formal and informal
     supports
• Relative weaknesses:
   – Natural supports, involvement of friends/advocates,
     involvement in the community
   – Youth voice, youth friendships, youth involvement in
     community
 WFI – Summary of trends from
 2010

• Areas of improvement from 2010:
   – Crisis identification and plans
   – Transition planning
       • including prep for transitions from wrap and within wrap

• Areas of decline:
   – Family voice and choice – selecting team, perception of
     family influence
   – Youth friends, youth voice, youth problem solving
   – Perception that wrap could be discontinued before family
     ready
Team Observation Measure

        Mass CBHI overall
TOM Total Fidelity Scores
All CSAs                2010 N=285 / 2011 N=658




            Total     Principle
            Overall   FVC   TB   NS   Col   CB   CC   Ind   SB   Per   OB
            Mean

Mass CBHI       83    95    84   43   88    91   92   83    88   89    73
Mean 2010
N = 285
Mass CBHI       85    94    85 51 92 91 93 86 90                 92    78
Mean 2011
N = 658
National        77    86    72   58   81    84   82   72    79   83    68
Mean
TOM Fidelity by Principle
All CSAs                 2010 N=285 / 2011 N=658


        100           95 94                   92 91 91 92 93
                                           88                       90 89 92
                       86 84 85                               86 88
          90    83 85                       81
                                                  84    82 83           83
                 77                                               79            78
          80                72                               72              73
                                                                              68
          70
                                    58
          60
                                      51
          50                       43
          40
          30
          20
          10
            0
                Total F V &
                              TB    NS     Col   CB    CC   Indiv   SB   Per   OB
                Score   C
Mean 2010        83    95     84    43      88    91   92    83     88   89    73
National Mean    77    86     72    58      81    84   82    72     79   83    68
Mean 2011        85    94     85    51      92    91   93    86     90   92    78
     TOM Item & Indicator Scores
     All CSAs

                         TEAM BASED                                   2010   2011   Nat Mean

         Item 1: Team Membership and Attendance                       3.10   3.09    2.39
a. Parent/caregiver is a team member and present at the
meeting.                                                              .99    .99      .93
b. Youth (over age 9) is a team member and present at the
meeting.                                                              .61    .61      .75
c. Key school or other public stakeholder agency representatives
are present.                                                          .61    .60      .38
               Item 2: Effective Team Process                         3.61   3.71     3.4
a. Team meeting attendees are oriented to the wraparound
process and understand the purpose of the meeting.                    .88    .91      .86
b. The facilitator assists the team to review and prioritize family
and youth needs.                                                      .95    .95      .89
c. Tasks and strategies are explicitly linked to goals.*              .91    .95      .85
d. Potential barriers to the nominated strategy or option are
discussed and problem-solved.                                         .86    .91      .84
  TOM Item & Indicator Scores
  All CSAs

                                                                                       Nat
                      COLLABORATIVE                                     2010   2011   Mean

                 Item 3: Facilitator Preparation                        3.50   3.68   3.36
a. There is a clear agenda or outline for the meeting, which provides
an understanding of the overall purpose of the meeting and the
major sections of the meeting.                                          .86    .91    .84
b. The meeting follows an agenda or outline such that team
members know the purpose of their activities at a given time.           .84    .91    .84
c. The facilitator has prepared needed documents and materials
prior to the meeting.                                                   .90    .96    .86
d. A plan for the next meeting is presented, including time & date.     .89    .91    .87
               Item 4: Effective decision making                        3.50   3.69   3.14
a. Team members demonstrate consistent willingness to
compromise or explore further options when there is disagreement.       .96    .97    .91
b. Team members reach shared agreement after having solicited
information from several members or having generated several
ideas.                                                                  .87    .95    .81
c. The plan of care is agreed upon by all present at the meeting.       .94    .98    .83
d. The facilitator summarizes the content of the meeting at the end
of the meeting, including next steps and responsibilities.              .76    .82    .87
   TOM Item & Indicator Scores
   All CSAs


                                                                                         Nat
                      INDIVIDUALIZED                                      2010   2011   Mean

          Item 5: Creative Brainstorming and Options                      3.11   3.16   2.46
a. The team considers several different strategies for meeting each
need and achieving each goal that is discussed.                           .83    .87    .78

b. The team considers multiple options for tasks or action steps.         .81    .86    .73
c. The facilitator leads a robust brainstorming process to develop
multiple options to meet priority needs.                                  .67    .69    .40
                  Item 6: Individualized process                          3.53   3.70   3.3
a. Planning includes action steps or goals for other family members,
not just identified youth.                                                .85    .92    .79
b. Facilitator and team members draw from knowledge about the
community to generate strategies and action steps based on unique
community supports.                                                       .85    .91    .78
c. Team facilitates the creation of individualized supports or services
to meet the unique needs of child and/or family.*                         .89    .96    .83
d. Youth, caregiver, & family members give their opinions about
potential services, supports, or strategies; including describing what
has or has not worked in the past.                                        .93    .95    .91
  TOM Item & Indicator Scores
  All CSAs

                                                                                     Nat
                   NATURAL SUPPORTS                                   2010 2011     Mean

           Item 7: Natural and community supports                     1.54   1.61   2.38
a. Natural supports for the family are team members and are
present.                                                              .27    .27    .26
b. Team provides multiple opportunities for natural supports to
participate in significant areas of discussion.                       .75    .80    .66
c. Community team members and natural supports participate in
decision-making.                                                      .72    .79    .66
d. Community team members and natural supports have a clear
role on the team.*                                                    .72    .81    .61
                 Item 8: Natural support plans                        1.94   2.47   2.37
a. Brainstorming of options and strategies include strategies to be
implemented by natural and community supports.                        .70    .77    .66
b. The plan of care represents a balance between formal services
and informal supports.                                                .45    .58    .36
c. There are flexible resources available to the team to allow for
creative services, supports, and strategies.                          .21    .58    .91
TOM Item & Indicator Scores
All CSAs

                     UNCONDITIONAL/                                              Nat
                       PERSISTANCE                                2010   2011   Mean

               Item 9: Team mission and plans                     3.44   3.61   3.23
 a. The team discusses or has produced a mission/vision
 statement.                                                       .84    .90    .75
 b. The team creates or references a plan that guides its work.   .91    .96    .87
 c. The team has confirmed or is creating a crisis plan.          .78    .90    .77
 d. The team plan contains specific goals that are linked to
 strategies and action steps.                                     .92    .96    .87
               Item 10: Shared Responsibility
                                                                  3.66   3.72   3.42
 a. The team explicitly assigns responsibility for action steps
 that define who will do what, when, and how often.*              .87    .91    .81
 b. There is a clear understanding of who is responsible for
 action steps and follow up on strategies in the plan.            .92    .92    .86
 c. Providers and agency representatives at the meeting
 demonstrate that they are working for the family and not
 there to represent a different agenda or set of interests.       .96    .98    .94
TOM Item & Indicator Scores
All CSAs

                                                                                       Nat
                  CULTURAL COMPETENCE                                   2010   2011   Mean

                     Item 11: Facilitation skills                       3.55   3.62   2.97
 a. Facilitator is able to impart understanding about what the
 wraparound process is, how it will work for this family, and how
 individual team members will participate.                              .83    .88    .74
 b. Facilitator reflects, summarizes, and makes process-oriented
 comments.                                                              .89    .92    .77
 c. Facilitator is able to manage disagreement & conflict and elicit
 underlying interests, needs, and motivations of team members.          .92    .92    .73
 d. Talk is well distributed across team members and each team
 member makes an extended or important contribution.                    .93    .92    .87
           Item 12: Cultural and Linguistic Competence                  3.76   3.86   3.56
 a. The youth, caregiver, and family members are given time to talk
 about the family’s values, beliefs, and traditions.                    .87    .95    .86
 b. The team demonstrates a clear and strong sense of respect for the
 family’s values, beliefs, and traditions.                              .95    .97    .88
 c. Meetings and meeting materials are provided in the language the
 family is most comfortable with.                                       .97    .98    .98
 d. Members of the team use language the family can understand (i.e.
 no professional jargon/acronyms)                                       .97    .98    .96
    TOM Item & Indicator Scores
    All CSAs

                                                                                       Nat
                      OUTCOMES BASED                                    2010   2011   Mean

                Item 13: Outcomes Based Process                         2.88   3.06   2.83
a. The team uses objective measurement strategies.                      .67    .76    .68
b. The team assesses goals/strategies using measures of progress.       .72    .77    .62
c. The team revises the plan if progress toward goals is not evident.   .84    .88    .87

            Item 14: Evaluating Progress and Success                    2.99   3.15   2.62
a. The team conducts a systematic review of members’ progress on
assigned action steps.                                                  .78    .84    .71
b. The facilitator checks in with the team members about their
comfort and satisfaction with the team process.                         .74    .79    .73
c. Objective or verifiable data is used as evidence of success,
progress, or lack thereof.                                              .72    .78    .61
    TOM Item & Indicator Scores
    All CSAs

                                                                                     Nat
                     VOICE AND CHOICE                                 2010   2011   Mean

                Item 15: Youth and Family Voice                       3.89   3.86   3.53
a. The team provides extra opportunity for caregivers to speak and
offer opinions, especially during decision making.                    .98    .98    .95
b. The team provides extra opportunity for the youth to speak and
offer opinions, especially during decision making.                    .93    .93    .91
c. Caregivers, parents, and family members are afforded
opportunities to speak in an open-ended way about current and
past experiences and/or about hopes for the future.                   .98    .98    .95
d. The youth is invited to speak in an open-ended way about
current and past experiences and/or about hopes for the future.       .96    .93    .88
               Item 16: Youth and Family Choice                       3.72   3.69   3.38
a. The youth prioritizes life domains, goals, or needs on which he
or she would like the team to work.                                   .78    .79    .76
b. The caregiver or parent prioritizes life domains goals, or needs
on which he or she would like the team to work.                       .93    .96    .89
c. The family and youth have highest priority in decision making      .97    .95    .90
     TOM Item & Indicator Scores
     All CSAs


                                                                                       Nat
                       STRENGTH BASED                                   2010   2011   Mean

                    Item 17: Focus on Strengths                         3.31   3.47   2.89
a. Team members acknowledge or list caregiver/youth strengths.          .92    .95    .82
b. Team builds an understanding of how youth strengths contribute to
the success of team mission or goals.                                   .78    .85    .67
c. In designing strategies, team members consider and build on
strengths of the youth and family.                                      .82    .89    .72
d. Facilitator and team members analyze youth & family member
perspectives and stories to identify functional strengths.              .78    .87    .69
                  Item 18: Positive Team Culture                        3.7    3.69   3.44
a. The team focuses on improvements or accomplishments throughout
the meeting.                                                            .91    .92    .82
b. The facilitator directs a process that prevents blame or excessive
focus on or discussion of negative events.                              .97    .95    .89
c. The facilitator encourages team culture by celebrating successes
since the last meeting                                                  .88    .93    .78
d. There is a sense of openness and trust among team members.           .94    .93    .90
TOM Item & Indicator Scores
All CSAs

                                                                                          Nat
                     COMMUNITY BASED                                       2010   2011   Mean

                    Item 19: Community Focus                               3.41   3.45   3.16
a. The team is actively brainstorming and facilitating community
activities for the youth and family.*                                      .82    .85    .71

b. The team prioritizes services that are community-based.                 .82    .86    .79
c. The team prioritizes access to services that are easily accessible to
the youth and family.                                                      .93    .94    .90

             Item 20: Least Restrictive Environment                        3.92   3.86   3.59
a. The team’s mission and/or identified needs support the youth’s
integration into the least restrictive residential and educational
environments possible.*                                                    .99    .97    .98
b. When residential placements are discussed, team chooses
community placements for the child or youth rather than out-of-
community placements, wherever possible.                                   .87    .88    .94
c. Serious challenges are discussed in terms of finding solutions, not
placement in more restrictive residential or educational
environments.                                                              .95    .94    .94
 TOM – Summary

• Overall: Significantly higher than national mean
• Significant improvement from 2010
• Relative strengths:
   – Nearly all principles. Particular strengths in:
       • School/Agency reps present, effective teams and facilitation, facilitator
         prep, Creative/individualized services, services for all family members,
         crisis plans

• Relative weaknesses:
   – Youth involvement/presence at meetings
   – Natural supports present and involved
   – Flex resources available and used when needed
 TOM – Summary of trends from
 2010

• Areas of improvement from 2010:
   – Facilitator prep, team process
   – Flexible resources available
   – Balance of informal/formal services/supports
   – Crisis planning and plans
   – Use of objective measures and following up
   – Tying strengths to planning
• Areas of decline:
   – Youths being asked their priorities
   – Youths present
Document Review Measure

        Mass CBHI overall
Mass-DRM Principle Scores




Total Mass DRM improvement: 61% to 74%
 DRM Item scores: Relative strengths
                                                                                                  2010      2011


8    The family is present at all meetings.                                                     2.83     2.91
4    There is a clearly articulated long range vision of the youth and family.                  2.31      2.9
     There is evidence that the family and/or youth are making decisions about the direction
17   and methods of the team.                                                                   2.53     2.84
     There is evidence that all members on the CPT are in regular communication, to ensure
11   coordinated efforts on the youth and family's behalf.                                      2.71     2.74
     Services, supports and tasks are based on the needs of the youth and family, not
15   availability of services or support that are needed.                                       2.34     2.73
     The RMSP identifies triggers, signs or behaviors that indicate the crisis is beginning and
23   interventions to deescalate crisis situations.                                             2.62     2.68
2    The family and youth's needs or concerns are identified across life domains.               2.13     2.62
     The risk management/safety plan (RMSP) includes strengths of youth and family and
22   strategies for preventing potential crises.                                                2.51     2.59
     There is evidence that the wraparound plan, including the RMSP, is updated at each
12   team meeting or as needed.                                                                 2.25     2.56
18   There is documentation that progress toward goals and tasks has been monitored.            2.29     2.56
     Detailed examples of family and youth strengths, assets, resources, culture, values, and
5    beliefs are included to support planning for areas of priority need.                        2.07    2.52
     The RMSP identifies specific steps to be taken if the crisis occurs, people responsible for
25   each step, including the youth's local ESP/MCI.                                             2.43    2.39
16   There is evidence that the team brainstorms more than 6 options to address goals.           1.99    2.33
DRM item scores: Needs for
improvement
                                                                                             2010    2011

     There is evidence that natural supports are actively involved in the planning and
     implementation process or there are ongoing and persistent efforts to identify and
10   engage natural supports.                                                                0.8    0.97

     There is documentation that an attempt has been made to collaborate with the
     primary care physician (PCP) or primary care clinician (PCC).
13                                                                                          1.72    1.21
     The wraparound plan includes opportunities for the youth to engage in community
20   activities that he or she likes and does well.                                          1.05   1.34
     There is documentation of transition and sustainability planning that identifies needs,
     services and support that will continue after formal wraparound ends or when the
21   youth transitions to the adult service system.                                          1.16    1.4

26   The RMSP is proactively shared with the MCI team.                                      1.43    1.97
     The RMSP is updated as needed; always when a hub-dependent support joins the
24   youth and family team.                                                                 1.94    2.09
     There is evidence that a strengths, needs, and culture discovery process was
6    completed within 10 days of enrollment of the youth.                                   1.64    2.11
     The initial wraparound plan was developed within 35 days of enrollment with the
7    youth and family.                                                                      2.47    2.16
     The plan includes strengths-based goals that are measurable with specific
14   measurement strategies.                                                                1.75    2.17
 DRM Summary

• Greater correlation with WFI/TOM from 2010
• Large improvement from 2010, particularly:
   – Vision statements
   – Updated plans
   – Needs ID’d across domains
   – Examples of strengths, beliefs, culture
   – SNCD within 10 days
• Declines in:
   – Wrap plans w/in 35 days
   – Collaboration with PCPs
 DRM Psychometrics


• Total Alpha = .85
• Principal Components Analysis
  (w/Varimax Rotation) found
  Interpretable 8-Factor Solution
  explaining 60.6% of variance
• Correlation with WFI = .326 (p<.05)
  – Certain items negatively correlate with
    WFI and/or have negligible correlation
    with DRM total
6 “Bad” DRM items
Do not correlate with the WFI and have low
item-total correlations

There is evidence that a CANS and full assessment were completed with the
youth and family within 10 days of enrollment.
The family is present at all meetings.

There is evidence that natural supports are actively involved in the planning and
implementation process or there are ongoing and persistent efforts to identify
and engage natural supports.
There is evidence that all members of the CPT are in regular communication, to
ensure coordinated efforts on the youth and family's behalf.
There is documentation that an attempt has been made to collaborate with the
primary care physician (PCP) or primary care clinician (PCC).
The RMSP is proactively shared with the MCI team.
 11 “Good” DRM Items
 Significant correlation with WFI AND
 high item totals
Documentation is strengths based and identifies and prioritizes the needs for the
youth and family.
Detailed examples of family and youth strengths, assets, culture, values and
resources are included to support planning for areas of priority need.
The initial ICP was developed within 28 days of enrollment with the youth and family.
The ICP specifies the goals that will help the family reach their long-range vision.
There is evidence that the ICP is updated at each team meeting and as needed.
Services, supports and tasks are based on the needs of the youth and family and not
on the availability of services or support that are needed.
There is evidence that the CPT brainstorms more than six options to address goals.
There is documentation that progress toward goals and tasks has been monitored.
There is evidence that the youth and family are making progress toward their goals.
The RMSP identifies triggers, signs or behaviors that indicate the crisis is beginning
and notes interventions to deescalate crisis situations.
The RMSP identifies specific steps to be taken if the crisis occurs, people responsible
for each step, including the youth's local ESP/MCI.
 DRM Principal Components
 Analysis

• Goals and Strategies (5 items)
• Strengths and Needs (4 items)
• Progress monitoring and timely updates (4 items)
• Crisis and safety plans present and of quality (3 items)
• Natural supports and community based services evident
  (3 items)
• Families as primary decision makers (3 items)
• Meeting mandated timelines (2 items: CANS and ICP)
• RMSP is shared with MCI team (1 item)
           DRM Principal Components
           Analysis                                                                                                                                        Natural
                                                                                                                                  Progress                 supports
                                                                                                                                  monitoring               and       Family de               Meet
                                                                                                         Str and       Goals and and timely Crisis and     community cision    MCI           mandated
Item                                                                                                     Needs         Strategies updates    safety        based     making    sharing       timelines
There is evidence that a CANS and full assessment were completed with the youth and family
within 10 days of enrollment.                                                                                  0.098        -0.035     0.100      -0.075        0.055     0.040      0.069        0.830
The family and youth's needs or concerns are identified across life domains.                                   0.840         0.066     0.042      -0.011        0.086     0.072      0.030        0.128
Documentation is strengths based and identifies and prioritizes the needs for the youth and family.            0.745         0.125     0.135       0.156       -0.057     0.104     -0.080       -0.060
There is a clearly articulated long range vision of the youth and family.                                      0.245         0.338     0.067       0.183       -0.237     0.435     -0.187       -0.002
Detailed examples of family and youth strengths, assets, culture, values and resources are included
to support planning for areas of priority need.                                                                0.708        0.187      0.048      0.176         0.137     0.003     0.089        -0.051
There is evidence that a strengths, needs, and culture discovery process was completed before the
first CPT meeting.                                                                                             0.614        -0.130     0.137      -0.124        0.176      0.042     0.195        0.359
The initial ICP was developed within 28 days of enrollment with the youth and family.                          0.026         0.214     0.110       0.208       -0.048     -0.076    -0.073        0.677
The family is present at all meetings.                                                                         0.079         0.027     0.133       0.024       -0.004      0.776     0.145        0.022
The ICP specifies the goals that will help the family reach their long-range vision.                           0.169         0.675     0.386      -0.019       -0.051      0.098     0.006        0.020
There is evidence that natural supports are actively involved in the planning and implementation
process or there are ongoing and persistent efforts to identify and engage natural supports.                   0.125        -0.027     0.064      0.085         0.601     0.145     -0.271        0.155
There is evidence that all members of the CPT are in regular communication, to ensure
coordinated efforts on the youth and family's behalf.                                                          0.088        0.478      0.074      0.141         0.165     0.301     -0.078       -0.067
There is evidence that the ICP is updated at each team meeting and as needed.                                  0.091        0.209      0.680      0.044         0.064     0.232      0.058        0.161
There is documentation that an attempt has been made to collaborate with the primary care
physician (PCP) or primary care clinician (PCC).                                                               0.027        0.242      0.011      -0.155       -0.037      0.088    0.703         0.162
The ICP includes strengths-based goals that are measurable with specific measurement strategies.               0.049        0.676      0.020       0.050        0.101     -0.035    0.319         0.085
Services, supports and tasks are based on the needs of the youth and family and not on the
availability of services or support that are needed.                                                           0.253        0.600      0.403      0.018        -0.031     0.013     0.103         0.067
There is evidence that the CPT brainstorms more than six options to address goals.                            -0.066        0.566      0.037      0.083         0.244     0.233     0.046         0.095
There is evidence that the family and/or youth are making decisions about the direction and
methods of the team.                                                                                           0.040        0.226      0.128      0.108         0.172     0.754      0.063       -0.045
There is documentation that progress toward goals and tasks has been monitored.                                0.047        0.161      0.739      0.141         0.219     0.198     -0.003        0.058
There is evidence that the youth and family are making progress toward their goals.                            0.106        0.143      0.648      0.106         0.405     0.101      0.017        0.079
The ICP includes opportunities for the youth to engage in community activities that he or she likes
and does well.                                                                                                 0.080        0.183      0.174      -0.028        0.705     0.001     -0.051       -0.073
There is documentation of transition and sustainability planning that identifies needs, services and
supports that will continue after formal wraparound ends or when the youth transitions to the
adult service system.                                                                                          0.066        0.114      0.078      0.190         0.667     -0.014    0.390        -0.032
The risk management/safety plan (RMSP) includes strengths of youth and family and strategies for
preventing potential crises.                                                                                   0.145        -0.019     0.204      0.777        -0.008     -0.011    0.002         0.044
The RMSP identifies triggers, signs or behaviors that indicate the crisis is beginning and notes
interventions to deescalate crisis situations.                                                                 0.105        0.070      0.160      0.800         0.048      0.026    0.049         0.015
The RMSP is updated as needed, and reviewed at every CPT meeting.                                              0.119        0.029      0.538      0.222        -0.145     -0.213    0.316         0.029
The RMSP identifies specific steps to be taken if the crisis occurs, people responsible for each step,
including the youth's local ESP/MCI.                                                                          -0.051         0.169     -0.077     0.689         0.189     0.291     -0.018        0.038
The RMSP is proactively shared with the MCI team.                                                              0.077        -0.001      0.147     0.137        -0.014     0.091      0.745       -0.109
Summary of Findings
 Summary of findings

• Continued strong fidelity results, especially for a large
  statewide initiative
   – WFI 3 points higher than national mean
• WFI scores stay stable, TOM shows small improvement,
  DRM show large improvement
• Big improvements noted for:
   – Crisis planning
   – Transition planning
   – Team process / Facilitator prep
   – Availability of flexible resources (but is this an artifact
     of TOM scoring clarification?)
   – Documentation of strengths, needs, goals, crisis plans
 Summary of findings


• Areas of strength continue to be:
  – Managing the details of the wrap
    process BUT ALSO
  – Family voice and choice
  – Collaborative teaming
  – Individualized plans with balance of
    formal svcs and informal supports
    Looking deeper: Relative weaknesses
•   Certain bigger ideas of wraparound not yet happening consistently
     – Natural supports and natural support plans
     – Connection to community activities and informal supports
     – Full youth engagement
     – Youth and family self-reliance and self-efficacy
     – Youths present at meetings
•   Better on some things, but still room for more improvement:
     – Robust brainstorming
     – Objective goal setting and monitoring of progress
     – Transition planning
•   Structural barriers:
     – Flexible funds
     – Local community collaboratives
     – Time to develop more creative plans linked to community supports
       and natural supports
     – Consistent understanding of how FPs and ICCs work together
 Possible strategies: System level
• Fiscal supports
   – Use of case rates
   – Availability of flexible funds
   – Methods to ensure less emphasis on productivity vs fidelity
     and outcomes?
• Community supports, collaboration, transition planning:
   – Encouragement to taking a community-level system of care
     approach
   – Hire resource development specialist(s)
   – Training and support to community collaborative teams
   – Continued social marketing of CBHI and cross agency
     training
   – Collecting info on and reviewing barriers faced by teams
     and CSAs
• Youth engagement
   – Use of youth peer to peer support partners?
  Possible strategies: Organizational
  level

• Natural supports:
    – Specific strategies (family finding?), training, time,
      supervision emphasis
    – Role for family partner?
• Transition planning
    – Clear understanding of interactions between medical
      necessity determination and Unconditional Care
        • Ensure clear expectations are delivered to families about transition
• Outcomes based:
    – Greater attention to consistent tracking of goal attainment
      and meeting needs
        • Use a dashboard approach?
• Staff turnover/attrition:
    – Training and guidance on staff recruitment and selection;
      compensation, caseloads, expectations (e.g., taking
      vacation to attend trainings)
 Possible strategies: Practice level

• Invest in specific strategies for engaging youth
    – E.g., Achieve My Plan! (AMP) from Portland State University
• Individualized care
    – More attention to developing individualized, creative plans
      in supervision, QI
• Emphasis on training and supervision on:
    – Transition planning
    – Individualized care plans
    – Natural support action steps
    – Teaching self-efficacy
 Measurement issues

• WFI: Start to interview youths
   – As a way of modeling a new era of engagement of Youth
   – To find out what is going on from their perspective
• How to use the TOM?
   – Ceiling effect now evident
   – Doesn’t correlate with WFI in Mass
       • Bias among raters/supervisors

   – Recommend use of independent TOM raters for a
     subsample of selected teams
 Measurement issues


• Continue to use the DRM?
   – Now clearly demonstrating utility as a fidelity tool
       • Could be evidence for the benefit of its continued use
       • OR…. Could mean don’t need to use it – duplicative

   – Perhaps revise based on data and disseminate for use by
     supervisors / CSA QA?
    Measurement issues

•   Analytic questions: What influences fidelity?
     – Established vs. new sites
     – Caseload sizes
     – Urban vs. rural
     – Characteristics of families served
     – Time enrolled in wraparound
     – Agency culture
     – Turnover
     – Perception of Community supports
 WHAT ARE THE OUTCOMES???


• Residential
• Medicaid expenditures/Fiscal
• CANS/Functional
Part 4: Massachusetts Data by
CSA
    WFI CSA Results

CSA              N    Total Score
                                               Nat Mean              0.77

18 Lawrence      18   0.86                   28 S Central                   0.8

                                             5 Cape Ann                      0.81
29 Springfield   20   0.82                22 New Bedford                     0.81
24 Park          14   0.82
Street                                       23 N Central                    0.81



                                    CSA
20 Lynn          19   0.82                    2 Attleboro                    0.81

                                             12 Gandara                           0.82
12 Gandara       14   0.82
                                                 20 Lynn                          0.82
2 Attleboro      18   0.81                 24 Park Street                         0.82
23 N. Central    22   0.81                 29 Springfield                         0.82

22 New           21   0.81                  18 Lawrence                                  0.86
Bedford
5 Cape Ann       20                                     0.7   0.75     0.8         0.85     0.9
                      0.81
                                                              Percent Fidelity
28 S. Central    17   0.80
     WFI CSA Results

CSA              N    Total
                      Score
                                         Nat Mean               0.77
                                    11 Framingham          0.76
10 Fall River    19   0.80
                                    31 Worcester E         0.76
16 Holyoke       16   0.80               14 Harbor              0.77
6 Cape and Isl   19   0.79               8 Coastal              0.77
1 Arlington      18   0.79                                          0.78


                              CSA
                                    32 Worcester W


                 21                                                 0.78
                                         19 Lowell
25 Pittsfield         0.79
                                       25 Pittsfield                     0.79
19 Lowell        20   0.78
                                        1 Arlington                      0.79
32 Worcester W   20   0.78                6 C and I                      0.79
8 Coastal        21   0.77              16 Holyoke                           0.8

14 Harbor        18   0.77            10 Fall River                          0.8

31 Worcester E   18   0.76                        0.74   0.76     0.78     0.8     0.82
11 Framingham    20   0.76                               Percent Fidelity
    WFI CSA Results

CSA             N    Total
                     Score
                                      Nat Mean                              0.77
                                     30 Walden                     0.55
13 Greenfield   19   0.76
                                   17 Hyde Park                           0.69
27 RVW          20   0.76
                                   4 Cambridge                            0.7
9 Dimock        20   0.75           26 Plymouth                            0.72
21 Malden       22   0.75                                                  0.73


                             CSA
                                     3 Brockton

15 Haverhill    20   0.74                 7 CSR                            0.74
                                    15 Haverhill                           0.74
7 CSR           11   0.74
                                      21 Malden                            0.75
3 Brockton      18   0.73              9 Dimock                            0.75
26 Plymouth     20   0.72               27 RVW                              0.76

4 Cambridge     22   0.70          13 Greenfield                            0.76

17 Hyde Park    20   0.69                          0         0.5                   1
30 Walden       3    0.55                              Percent Fidelity
  TOM CSA Results

CSA             N    Total
                     Score
                                        Nat Mean        0.77

8 Coastal       14   0.93                 20 Lynn                  0.9

                                   22 New Bedford                   0.91
7 CSR           58   0.93
                                      26 Plymouth                   0.91
10 Fall River   32   0.93
                                   11 Framingham                    0.91
5 Cape Ann      38   0.92


                             CSA
                                       21 Malden                    0.91
30 Walden       4    0.92              30 Walden                     0.92
21 Malden       44   0.91             5 Cape Ann                     0.92
11              22   0.91            10 Fall River                       0.93
Framingham
                                           7 CSR                         0.93
26 Plymouth     18   0.91
                                        8 Coastal                        0.93
22 New          23   0.91
Bedford                                          0.7
20 Lynn         19   0.90                              Percent Fidelity
    TOM CSA Results

CSA              N    Total
                      Score
                                         Nat Mean             0.77
                                     13 Greenfield                    0.83
6 C and I        32   0.89
                                      15 Haverhill                       0.85
16 Holyoke       14   0.86
                                     4 Cambridge                         0.85
9 Dimock         13   0.86            23 N Central                       0.85
2 Attleboro      10   0.86                                               0.85


                              CSA
                                       1 Arlington

31 Worcester E   23   0.86           18 Lawrence                         0.85
                                    31 Worcester E                           0.86
18 Lawrence      10   0.85
                                       2 Attleboro                           0.86
1 Arlington      18   0.85               9 Dimock                            0.86
23 N Central     31   0.85             16 Holyoke                            0.86

4 Cambridge      16   0.85               6 C and I                                  0.89

15 Haverhill     21   0.85                       0.7   0.75     0.8   0.85      0.9
13 Greenfield    16   0.83                             Percent Fidelity
     TOM CSA Results

CSA              N    Total
                      Score
                                         Nat Mean                     0.77
                                       12 Gandara              0.73
28 S Central     14   0.83
                                     24 Park Street              0.75
17 Hyde Park     8    0.83           29 Springfield                   0.77
25 Pittsfield    25   0.82          32 Worcester W                    0.77
3 Brockton       11   0.81                                              0.79


                              CSA
                                         19 Lowell


                 18                                                     0.79
                                           27 RVW
14 Harbor             0.79
                                         14 Harbor                      0.79
27 RVW           25   0.79
                                        3 Brockton                            0.81
19 Lowell        16   0.79             25 Pittsfield                           0.82
32 Worcester W   13   0.77            17 Hyde Park                              0.83

29 Springfield   31   0.77             28 S Central                             0.83

24 Park Street   18   0.75                        0.65   0.7   0.75     0.8     0.85
12 Gandara       6    0.73                               Percent Fidelity
   DRM CSA Results

CSA              N    Total
                      Score
                                         MA Mean         0.74

                                       5 Cape Ann               0.78
28 S Central     10   0.82
                                       1 Arlington                0.79
6 Cape and Isl   10   0.82
                                      4 Cambridge                 0.79
11               10   0.82
                                    31 Worcester E                 0.8
Framingham

                              CSA
                                         14 Harbor                 0.8
20 Lynn          10   0.80
                                      18 Lawrence                  0.8
18 Lawrence      10   0.80
                                          20 Lynn                  0.8
14 Harbor        10   0.80
                                    11 Framingham                         0.82
31 Worcester     10   0.80
                                         6 C and I                        0.82
E
                                      28 S Central                        0.82
4 Cambridge      10   0.79

1 Arlington      10   0.79                       0.7    0.75     0.8        0.85

                 10   0.78                             Percent Fidelity
5 Cape Ann
     DRM CSA Results

CSA              N    Total
                      Score
                                         MA Mean                  0.74
                                             7 CSR             0.73
32 Worcester W   10   0.76
                                        2 Attleboro            0.73
8 Coastal        10   0.76
                                     29 Springfield            0.73
3 Brockton       10   0.76          22 New Bedford                0.74
17 Hyde Park     10   0.76                                        0.74


                              CSA
                                       26 Plymouth

23 N Central     10   0.76            10 Fall River                   0.75
                                       23 N Central                        0.76
10 Fall River    10   0.75
                                      17 Hyde Park                         0.76
26 Plymouth      10   0.74              3 Brockton                         0.76
22 New Bedford   10   0.74               8 Coastal                         0.76
29 Springfield   10   0.73          32 Worcester W                         0.76

2 Attleboro      10   0.73                        0.7   0.72   0.74      0.76     0.78
7 CSR            10   0.73                              Percent Fidelity
    DRM CSA Results

CSA              N    Total
                      Score
                                        MA Mean                                     0.74
                                      12 Gandara               0.64
13 Greenfield    12   0.71
                                        19 Lowell               0.65
21 Malden        10   0.71
                                        9 Dimock                0.65
16 Holyoke       10   0.71             30 Walden                  0.66
24 Park Street   10   0.70                                               0.69


                              CSA
                                          27 RVW

15 Haverhill     10                   25 Pittsfield                      0.69
                      0.69
                                      15 Haverhill                       0.69
25 Pittsfield    10   0.69
                                    24 Park Street                        0.7
27 RVW           10   0.69             16 Holyoke                            0.71
30 Walden        10   0.66             21 Malden                             0.71

9 Dimock         10   0.65          13 Greenfield                            0.71

19 Lowell        10   0.65                       0.55   0.6   0.65     0.7      0.75
12 Gandara       10   0.64                              Percent Fidelity
      Combined Scores
      by CSA
 1

0.9

0.8

0.7

0.6

0.5

0.4             18                            11
                        5 Cape                       10 Fall                28 S     22 New       1
      20 Lynn Lawrenc            6 C and I Framing             8 Coastal                               Nat Mean
                          Ann                         River                Central   Bedford Arlington
                 e                           ham
WFI    0.82    0.86      0.81      0.79     0.76      0.8        0.77        0.8      0.81      0.79     0.77
TOM     0.9    0.85      0.92      0.89     0.91      0.93       0.93       0.83      0.91      0.85     0.77
DRM     0.8     0.8      0.78      0.82     0.82      0.75       0.76       0.82      0.74      0.79     0.74
      Combined Scores
      by CSA
 1

0.9

0.8

0.7

0.6

0.5

0.4     31                2                 26                            4      29       32
               23 N                                21     16      14                               Nat
      Worcest         Attlebor   7 CSR   Plymout                       Cambrid Springfi Worcest
              Central                            Malden Holyoke Harbor                            Mean
       er E               o                 h                            ge      eld     er W
WFI    0.76     0.81    0.81     0.74     0.72    0.75     0.8    0.77    0.7    0.82    0.78     0.77
TOM    0.86     0.85    0.86     0.93     0.91    0.91    0.86    0.79    0.85   0.77    0.77     0.77
DRM     0.8     0.76    0.73     0.73     0.74    0.71    0.71    0.8     0.79   0.73    0.76     0.74
      Combined Scores
      by CSA
 1

0.9

0.8

0.7

0.6

0.5

0.4      13       3       25       15
                                          17 Hyde 24 Park    9            19     12      30      Nat
      Greenfie Brockto Pittsfiel Haverhil                        27 RVW
                                            Park   Street Dimock        Lowell Gandara Walden   mean
         ld       n        d        l
WFI    0.76    0.73    0.79    0.74    0.69    0.82     0.75    0.76    0.78    0.82    0.55    0.77
TOM    0.83    0.81    0.82    0.85    0.83    0.75     0.86    0.79    0.79    0.73    0.92    0.77
DRM    0.71    0.76    0.69    0.69    0.76     0.7     0.65    0.69    0.65    0.64    0.66    0.74
 Percent improvement by CSA by tool

• WFI top 3 +
  – 19 Lowell +.07
  – 18 Lawrence +.05
  – 25 Pittsfield and 22 New Bedford +.05
• WFI bottom 3 -
  – 4 Cambridge -.09
  – 17 Hyde Park -.14
  – 30 Walden -.24 *only N = 3 in 2010
 Percent improvement by CSA by tool

• TOM top 3 +
   – 9 Dimcock +.34
   – 8 Coastal + .12
   – 5 Cape Ann + .12
   – 25 Pittsfield +.12
   – 6 C and I +.09
• TOM bottom 3 -
   – 14 Harbor -.06
   – 29 Springfield -.10
   – 24 Park Street -.11
 Percent improvement by CSA by tool

• DRM top 3 +
  – 31 Worcester E +.43
  – 32 Worcester W + .40
  – 5 N. Cape Ann + .38
• DRM bottom 3 -
  – 27 RVW -.10
  – 19 Lowell -.13
  – 30 Walden -.16

								
To top