Status Report on Compliance with Annual Performance Outcome Standards by iuq51574

VIEWS: 8 PAGES: 22

									    Fiscal Year 2007-2008 Legislative Status Report


       Compliance with Annual Performance Outcome Standards
                Per Section 394.745, Florida Statutes


           Substance Abuse and Mental Health Programs
               Department of Children and Families

                         November 1, 2008




           




George H. Sheldon                                       Charlie Crist
Secretary                                               Governor
              Compliance with Annual Performance Outcome Standards
                  Substance Abuse and Mental Health Programs
                       Department of Children and Families
                  Fiscal Year 2007-2008 Legislative Status Report
                       Per Section 394.745, Florida Statutes
                                    Executive Summary
The purpose of this report is to demonstrate the compliance with Section 394.745, Florida
Statutes, which requires the Department of Children and Family Services to submit an annual
report describing the status of contracted substance abuse and mental health (SAMH)
providers’ compliance with the annual performance outcome standards established by the
Legislature. In Fiscal Year (FY) 2007-2008, there were 227 SAMH contracts documented in the
SAMH contract database as having performance data and minimum standards that permitted
the calculation of success indicators. Of these 227 contracts, 215 met the level of success
defined in the “Method” section, for a 94.7 % success rate as compared to 92.5% in FY 2006-
2007. This report describes the corrective actions for the 12 contracts that failed the
compliance test in
FY 2007-2008.

This report also summarizes the history of Performance Based Program Budgeting (PB)2 since
1994, and provides a brief review of (PB)2 implementation by the Substance Abuse and Mental
Health Programs, beginning in FY 1997-98. The report also describes some major data
improvement initiatives by the SAMH Program Office, including data improvement workgroup
meetings, technical assistance and support, compliance with state and federal data
requirements, collaboration with other agencies, and monitoring data quality.

This report provides a list of General Appropriation Act (GAA) performance outcome measures
and the number of SAMH contracts involved in each of these performance outcome measures
in FY 2007-2008. The report also provides, by district, the number of SAMH contracts according
to contracted amounts, as well as the number of clients served per target population group
(n = 260,385 for mental health and n = 163,656 for substance abuse).

The overall contract performance is calculated by obtaining a single score for each contract.
This score is then used to determine the contract’s success or failure in meeting performance
outcome expectations. To ensure fairness of the method used to evaluate contracted
providers, this single score is based on comparisons with other contracts serving the same
target populations and using similar performance measures.

The recommendation section examines district corrective actions for the 12 contracts that
failed to meet the GAA performance outcome standards. One of these corrective actions is a
new requirement for SAMH Program Office to produce a regular year-to-date monthly report,
which will provide SAMH contract managers and service providers timely information to
identify and monitor the SAMH contracts, which are having difficulty meeting the GAA
performance target standards.

October 23, 2008                                                                          Page ii
 Compliance with Annual Performance Outcome Standards - FY 2007-08 Legislative Status Report



                          Table of Contents


Section                                                                                          Page
Executive Summary                                                                                 ii

I.      Purpose                                                                                   1

II.     History of Performance Based Program Budgeting (PB)2                                      1

III.    Implementation of (PB)2 in Substance Abuse and
        Mental Health Programs (SAMH)                                                             1

IV.     Data Improvement Initiatives                                                              2

V.      Overview of FY 2007-2008 Performance Contracting                                          5

VI.     DCF Background On Performance Contracting                                                 7

VII.    Method for Determining “Successful” Contract Performance                                  11

VIII.   Performance Outcome Results                                                               15

IX.     District Corrective Actions Taken for Unsuccessful Contracts                              16

X.      Discussion and Recommendations                                                            19




                                                                                      Page iii
         Compliance with Annual Performance Outcome Standards
             Substance Abuse and Mental Health Programs
                  Department of Children and Families
             Fiscal Year 2007-2008 Legislative Status Report
                  Per Section 394.745, Florida Statutes
I. Purpose
The Department is submitting this report in compliance with Section 394.745, Florida Statutes,
which requires the Department of Children and Family Services to submit an annual report as
specified in the paragraph below:

      “By November 1 of each year, the Department of Children and Family Services
      shall submit a report to the President of the Senate and the Speaker of the House
      of Representatives which describes the compliance of providers that provide
      substance abuse treatment programs and mental health services under contract
      with the Department of Children and Family Services. The report must describe
      the status of compliance with the annual performance outcome standards
      established by the Legislature and must address the providers that meet or
      exceed performance standards, the providers that did not achieve performance
      standards for which corrective action measures were developed and the
      providers whose contracts were terminated due to failure to meet the
      requirements of the corrective plan”.

II. History of Performance Based Program Budgeting (PB)2
In 1994, the Florida Legislature enacted the General Appropriations Act (GAA), which required
the implementation of performance-based program budgeting (PB)2 in Florida's state agencies.
"Performance-based budgeting is a method of relating appropriations to program performance
and expected outcomes." 1 (PB)2 requires that state agencies, as part of their budget requests
for the fiscal year, establish performance outcome targets they intend to achieve on various
performance measures. The Department of Children and Families (DCF) began the (PB) 2
implementation in Substance Abuse and Mental Health (SAMH) programs in FY 1997-98.1

III. Implementation of (PB)2 in Substance Abuse and Mental Health Programs (SAMH)
As the first human services program to implement (PB)2, the SAMH Program Office met the
challenge of the new approach by building upon work already underway that integrated client,
service, provider, fiscal, and outcome data. This preliminary work was developed nationally as
part of the Mental Health Statistics Improvement Project (MHSIP) and funded by the U.S.
Department of Health and Human Services (DHHS), Division of Substance Abuse and Mental
Health Service Administration (SAMHSA), Center for Mental Health Services (CMHS). The
purpose of this MHSIP project was to develop minimum standards for implementing an
information system model that sought to answer the management question, "Who Receives
What from Whom, at What Cost with What Effect?" Prior to FY 1997-98, the SAMH Program
1
 A Report on Performance-Based Program Budgeting in Context: History and Comparison, Report 96-77A, Office
of Program Policy Analysis and Government Accountability, April 1997.

                                                                                                     Page 1
        Compliance with Annual Performance Outcome Standards - FY 2007-08 Legislative Status Report


Office was in the process of implementing and refining a new unit cost approach to contracting
with the private providers that serve SAMH clients, and a data system based on the MHSIP
model. This work eased the transition to (PB)2. By the end of FY 2007-2008, the SAMH
Program Office had completed their tenth year of (PB)2 implementation.

IV. Data Improvement Initiatives
In FY 2007-2008, the Substance Abuse and Mental Health (SAMH) Program Office initiated
and/or completed the following activities to support the (PB) 2 process and other statutory
requirements to improve the quality of the data collected and processed:

1. Data Improvement Workgroup (DIWG)
This is an interdisciplinary team of various SAMH stakeholders and advocacy groups, including
contracted SAMH providers, staff in central and circuit SAMH Program Offices, researchers from
the Florida Mental Health Institute (FMHI) at the University of South Florida, representatives of
the Florida Council for Community Mental Health (FCCMH) and Florida Alcohol and Drug Abuse
Association (FADAA). This group met in September 2007 in Orlando to discuss FY 2008-2009
data requirements including issues related to data quality assurance and improvement, data
analysis and reporting, performance measurement and target setting. The group met again in
May 2008 via Video Teleconference to review and finalize the strategies for implementing
FY 2008-2009 data policy requirements as summarized in a letter dated April 9, 2008, from the
SAMH Assistant Secretary to Contracted SAMH Service Providers.

2. Data Liaisons with DCF Regions and Circuits
The Region/Circuit Data Liaisons form a network of staff from each DCF region and circuit; they
serve as linkages between the SAMH Central Program Office in Tallahassee and SAMH
community provider agencies. In FY 2007-2008, these individuals met weekly through
conference calls with staff in the SAMH Central Program Office to discuss ongoing technical
support and training issues related to statewide implementation of data policies and
procedures, data collection and submission, and data analysis and reporting.

3. State Data Requirements
Chapter 2008-243, Laws of Florida, requires the Department of Children and Families (DCF) to
improve not only accountability for a local system of behavioral health care services to meet
performance outcomes and standards through the use of reliable and timely data, but also the
integration, accessibility, and dissemination of behavioral health data for planning and
monitoring purposes. Furthermore, Chapter 2003-279, Laws of Florida, requires the
Department of Children and Families (DCF) and the Agency for Health Care Administration
(AHCA) to establish or develop data management and reporting systems that: (1) promote
efficient use of data by the service delivery system; (2) address the management and clinical
care needs of the service providers and managing entities; and (3) provide information needed
by the Department for state and federal reporting requirements.




                                                                                             Page 2
        Compliance with Annual Performance Outcome Standards - FY 2007-08 Legislative Status Report


The following are some of the major projects initiated and/or completed in FY 2007-2008 to
continue the implementation of these state data requirements:

   Co-Occurring Capability: modified the Substance Abuse and Mental Health Information
    System (SAMHIS) to allow the tracking and reporting of service events and outcome data
    pertaining to persons with co-occurring mental health and substance abuse disorders.

   Substance Abuse Licensure: designed and implemented a web-based application for
    Substance Abuse Licensure Information System (SALIS), which allows substance abuse
    licensing staff in the DCF central and circuit offices to license substance abuse providers and
    to monitor their compliance with various licensing requirements.

   Recovery Oriented System Indicators (ROSI) Survey: completed the Recovery Oriented
    System Indicators (ROSI) survey and designed a web-based application that uses Crystal
    Reports to analyze the ROSI survey data. This database provides information regarding
    client satisfaction with various domains of the mental health system of care.

   Data Visibility: increased data visibility and accessibility by converting an existing Query
    Facility application from a standalone, window-based environment into a web-enabled
    environment that uses .Net as the web application and SQL Server 2005 as the relational
    database. The first phase of this web-enabled Query Facility will be completed by
    December 17, 2008 and the second phase will be completed by February 28, 2009. The new
    Query Facility provides stakeholders with 24/7 online capability for producing both standard
    and ad hoc reports, including the DCF dashboard data pertaining to the General
    Appropriation Act (GAA) performance measures.

4. Federal Data Requirements
The Substance Abuse and Mental Health Services Administration (SAMHSA) in the U.S.
Department of Health and Human Services (DHHS) requires the SAMH Program Offices to
collect and report various data as specified in various federal grants, including those pertaining
to the National Outcome Measures (NOMS) and Evidence-Based Practices (EBP), the State
Outcomes Measurement and Management (SOMMS), the Treatment Episode Data System
(TEDS), and the Uniform Reporting System (URS). The SAMH Program Office has already
developed the appropriate computer algorithms and stored procedures to retrieve, analyze and
report the majority of the data required by the SAMHSA Center for Community Mental Health
Services (CMHS) and the Center for Substance Abuse Treatment (CSAT).

5. Collaboration with Other Agencies
 DCF-AHCA Coordination Meetings - Section 394.9082, F.S., requires DCF and the Agency for
   Health Care Administration (AHCA) to develop service delivery strategies to improve the
   coordination, integration, and management of the delivery of mental health and substance
   abuse treatment services to persons with emotional, mental, or addictive disorders. One of
   the goals of this legislation is to improve compliance with annual performance outcome
   standards. In response to this legislation, high-level management staff members from both
   agencies, including the Assistant Secretary for SAMH Programs, have held regular


                                                                                             Page 3
        Compliance with Annual Performance Outcome Standards - FY 2007-08 Legislative Status Report


    coordination meetings in FY 2007-2008, which will continue in FY 2008-2009, to discuss
    various agenda topics, including data sharing and performance measurement issues.
    Traditionally, providers have been required to provide data to DCF for clients whose
    services are paid in full, or in part, by DCF contracts, Medicaid or local match. Because of
    Medicaid reforms and AHCA’s contracts with managed care organizations, e.g., Health
    Maintenance Organizations (HMO) and pre-paid behavioral health plans, data submission
    on clients served by these organizations has substantially been reduced. This has resulted
    in a reduction in DCF capacity to account for the performance of publicly-funded services,
    including persons served, services provided to these individuals, as well as the costs and
    outcomes of these services. DCF and AHCA representatives will continue the collaboration
    in FY 2008-2009 to maximize mutual data sharing between the two agencies.

   Department of Corrections (DC) Aftercare Planning Web-based Application - The SAMH
    Program Office staff collaborated with their counterparts in the Department of Corrections
    (DC) to develop a web-based application, which currently allows both agencies to track
    inmates released from DC institutions for aftercare planning and service provision in state-
    contracted community mental health provider agencies. The statewide implementation of
    this system began July 1, 2008. Aftercare coordinators in the DCF central and circuit offices
    currently use this system for online communication with DC staff and SAMH provider
    agencies regarding when individuals will be released from DC institutions, whether or not
    these individuals have any appointments scheduled with a provider agency, and whether or
    not these appointments have been kept.

   Incident Reporting and Assessment System (IRAS) - The DCF Office of Information Systems,
    in collaboration with various DCF program offices, has designed IRAS as a web-based
    application for collecting and reporting critical incidents involving consumers and staff
    members in state-contracted, state-licensed, or state-operated provider agencies. This
    system is being piloted and Beta-tested in FY 2008-2009 for statewide implementation in
    FY 2009-2010; it is designed to provide online information, including instant notifications
    and alerts, on various critical incidents (e.g., serious injuries or deaths related to seclusion
    and restraint in substance abuse and mental health facilities).

6. Technical Assistance and Support
The SAMH Program Office has continued to use the Performance and Resource Management
Teams (PaRTs) process to provide ongoing technical assistance and support needed by region,
circuit, and provider staff on the performance measures. This process typically includes the
discussion of the following topics: (a) definitions of the performance measures per target
population; (b) identification of the state standard targets required by the General
Appropriation Act (GAA) for each performance measure; (c) a discussion of the algorithms used
to calculate the actual performance on each measure; and (d) the reasons for any discrepancies
between standard targets and actual performance outcomes. Ongoing site visits to providers,
regions and circuits were also conducted throughout the year to address various data quality
and data improvement issues.




                                                                                              Page 4
        Compliance with Annual Performance Outcome Standards - FY 2007-08 Legislative Status Report


6. Monitoring Data Quality
Through collaboration with DCF Contract Oversight Unit (COU) staff, region/circuit data liaison
staff, and district contract management staff, the SAMH Program Office staff has developed
several exception reports that are used to monitor and improve data quality as part of the
contract monitoring process. COU staff use these reports as data validation tools to ensure the
conformity between information in the Department’s data system and information in the
corresponding provider client records, and to verify that all the required data on enrolled
clients is submitted to the Department’s data systems. These exception reports are available
and accessible online to authorized system users at the following DCF Intranet web site:
http://ewas1.dcf.state.fl.us/PDADM/SAMHRpts/Default.asp.

V. Overview of FY 2007-2008 Performance Contracting
Table 1 provides a list of twelve GAA performance outcome measures contracted in FY 2007-
2008, as documented by the contract managers in the SAMH contract database. This table also
provides the number of contracts involved in each of these performance outcome measures.
The focus of this report is limited to outcome rather than output measures, as specified in
s.394.745, F.S., to read, "The report must describe the status of compliance with the annual
performance outcome standards established by the Legislature." Only the contracted
performance outcome measures, therefore, are listed for each target population group. The
same legislation limits the scope of the report to "…service under contract with the Department
of Children and Family Services." Thus, no measures are listed that pertain to state-operated or
state-contracted mental health treatment facilities.

The GAA outcome measures are minimally required in each provider's contracts. However with
approval from the State Program Offices, the districts may include additional measures to
assess individual provider performance. Although the GAA specifies the level of performance
required at the state level for each measure, an individual contract may include one or more
targets for performance standards that are set above or below the state standard. These
targets are set based on available resources and the provider's past performance with the
intent to obtain improved performance over time. These additional non-GAA performance
measures are not included in this report.




                                                                                             Page 5
          Compliance with Annual Performance Outcome Standards - FY 2007-08 Legislative Status Report


 Table 1: Substance Abuse and Mental Health Contracted GAA Outcome Measures for FY 2007-08
 Dashboard                                                                      Number of
                                                        Performance Outcome
Performance            Target Population Group                                   Contracts
                                                         Measure Description
 Measure ID                                                                      Involved
                                                      Average annual number of
               AMH: SPMI: Adults with Severe and
   M0001                                              days spent in the             123
               Persistent Mental Illness (SPMI)
                                                      community
               AMH-SPMI: Adults with Severe and            Average annual days
  M0003                                                                                       109
               Persistent Mental Illness (SPMI)            worked for pay
                                                           Average annual number of
               AMH-Forensic: Adults with Forensic
  M0010                                                    days spent in the                   62
               Involvement
                                                           community
                                                           Median length of stay in
               AMH-Crisis: Adults in Mental Health
  M0376                                                    Crisis Stabilization Unit or        45
               Crisis
                                                           Inpatient services

               CMH-SED: Children with Serious
                                                           Annual number of days
  M0011        Emotional Disturbance (SED) - excluding                                        118
                                                           spent in the community
               those in juvenile justice facilities
               CMH-SED: Children with Serious
                                                           Percent of school days
  M0012        Emotional Disturbance (SED) - excluding                                         45
                                                           attended
               those in juvenile justice facilities
               CMH-SED: Children with Serious
                                                           Percent who improved
  M0378        Emotional Disturbance (SED) - excluding                                        120
                                                           their level of functioning
               those in juvenile justice facilities
               CMH-ED: Children with Emotional
                                                           Annual number of days
  M0025        Disturbance (ED) - excluding those in                                           99
                                                           spent in the community
               juvenile justice facilities
               CMH-ED: Children with Emotional
                                                           Percent who improved
  M0377        Disturbance (ED) - excluding those in                                          100
                                                           their level of functioning
               juvenile justice facilities

                                                           Percent employed upon
               ASA-SA: Adults with Substance Abuse
  M0058                                                    discharge from treatment            70
               Problems
                                                           services
                                                           Percent successfully
               ASA-SA: Adults with Substance Abuse
  M0062                                                    completed treatment                 75
               Problems


               CSA-SA: Children with Substance Abuse       Percent successfully
  M0045                                                                                        48
               Problems                                    completed treatment


                                                                                               Page 6
        Compliance with Annual Performance Outcome Standards - FY 2007-08 Legislative Status Report




VI. DCF Background on Performance Contracting
This section provides a context for interpreting the level of effort associated with the
measurement of GAA performance outcomes. It provides FY 2007-2008 district information on:
(a) SAMH contracts accountable for GAA performance outcomes (Column 8 in Table 2);
(b) SAMH contract allotment amounts (Column 7 in Table 2); (c) the number of clients served in
community mental health providers (Table 3); and (d) the number of clients served in
community substance abuse providers (Table 4).

1. Contracts Accountable for GAA Outcome Measures
    Column 9 of Table 2 shows a total of 412 SAMH contracts reported in the DCF Schedule of
    Allotment Balances for FY 2007-08. However, only contracts that meet the following
    conditions were considered for inclusion in further analysis of the performance compliance
    with GAA outcome measures in this report:

     The contract has Exhibit D in the SAMH contract database, which has at least one target
      population for which there is at least one GAA performance outcome rather than an
      output measure; and
     The contract has GAA performance outcome data reported in the Substance Abuse and
      Mental Health Information System (SAMHIS).

    As shown in Column 8 of Table 2, there were 227 SAMH contracts that met the above
    conditions and, therefore, were used in this report to analyze contract success during
    FY 2007-2008. The remaining 185 contracts did not meet the above conditions mainly
    because they provided non-client specific services (e.g., Prevention, Outreach, Information
    and Referral, Mental Health ClubHouse, and Drop-In/Self Help) and other services
    (e.g., Assessment-only Services, Clinical Supervision Services, and Research and Evaluation
    Services), which do not require the submission of GAA performance outcome measures.

2. Contracted Amounts
   Table 2 shows the total contract allotment amounts per district in FY 2007-2008 and the
   distribution of the 412 contracts by district, for each of the five contract amount groupings.
   As shown in this table, the majority of the contracts (253 of the 412 or 61.4%) have less
   than $500,001 contracted amount. These 253 contracts include those listed in columns 2
   and 3 of Table 2.




                                                                                             Page 7
                  Compliance with Annual Performance Outcome Standards - FY 2007-08 Legislative Status Report



            Table 2: Number of SAMH Contracts for FY 2007-08 and Allotment Amounts per District


  (1)        (2)           (3)          (4)           (5)            (6)             (7)            (8)         (9)
                                                                                                  Number
                                                                                                 reporting
              $0                      $500,001                     Greater      Total Contract                 Total
                        $100,001                   $1,000,001                                    outcome
District      to                         to                          than         Allotment                  Number of
                       to $500,000                to $3,000,000                                   Data in
           $100,000                  $1,000,000                   $3,000,000      Amounts                    Contracts
                                                                                                  SAMHIS

  01          3            7             1             3              3        $28,047,481.00       5           17
  02          5            4             0             1              4        $24,124,701.00       8           14
  03          8            6             2             2              1        $18,520,836.00       10          19
  04           7           11            3             6              3         $33,373,981.00      11          30
  07           8           14            2             5              6         $60,485,653.00      24          35
  08           6           10            1             5              4         $33,842,106.00      14          26
  09          10           16            2             4              3         $31,382,676.00      21          35
  10           7           23            5             9              3         $44,783,690.00      27          47
  11          16           17            5            13              6         $71,893,309.00      31          57
  12           6            4            2             4              1         $12,895,272.00      12          17
  13           2            4            2             2              3         $23,160,432.00       5          13
  14          10            6            1             6              2         $23,745,734.00      14          25
  15           2           12            2             3              6        $166,595,224.00      16          25
  23          18           11            3            10             10        $106,258,747.00      29          52
 Total       108          145           31            73             55        $679,109,842.00     227          412




     3. Clients Served
        Clients served are persons enrolled in the various target population groups and whose
        services are paid entirely or partially by SAMH contracts, Medicaid, or matching funds from
        local governments.

           Table 3 shows, by district and by target population group, the total number of people
           served in community mental health programs in FY 2007-2008. This table also shows the
           trend data from FY 2002-2003 to FY 2007-2008. Across the five year intervals shown in this
           table, the number of children served increased by 5.1 percent from 77,803 to 81,954, which
           accounts for an average increase of 830 children served yearly or a 1.0 percent annual
           average increase. During the same period, adults served increased by 26.3 percent from
           140,010 to 176,787. This accounts for an average increase of 7,355 adults served annually
           or a 5.26 percent annual average increase.




                                                                                                             Page 8
               Compliance with Annual Performance Outcome Standards - FY 2007-08 Legislative Status Report


    Table 3: Number of Clients Served in Community Mental Health Providers in FY 2007-08

                                                                                                                     Mental
                Children's Mental Health (CMH)                          Adult Mental Health (AMH)
                                                                                                                     Health
                             At-Risk of MH           Total                            Community            Total    Total (CMH +
District    SED      ED                    Unknown             SPMI       In Crisis             Unknown
                               Problems              CMH                               Forensic            AMH         AMH)
   1        2,462   2,904       1,419       na       6,785    9,688        6,590         197      na      16,475      23,260
   2        2,161    906         13         na       3,080    6,537         943          184      na      7,664       10,744
   3        2,327   1,576        260        na       4,163    6,485        1,472         222      na      8,179       12,342
   4        3,710   2,074        140        na       5,924    8,475        2,002         288      na      10,765      16,689
   7        6,558   2,535        448        na       9,541    19,454       4,028         131      na      23,613      33,154
   8        1,772   1,442        126        na       3,340    6,512        1,236          70      na      7,818       11,158
   9        1,592   1,406        60         na       3,058    3,844        2,444         106      na      6,394       9,452
  10        3,894   1,233        471        na       5,598    8,004        2,185         723      na      10,912      16,510
  11        8,466   4,640        269        na       13,375   20,310       2,307         411      na      23,028      36,403
  12        1,908    758         45         na       2,711    2,873        2,948         114      na      5,935       8,646
  13        2,358   1,340        17         na       3,715    8,007        2,169         239      na      10,415      14,130
  14        1,895   2,518        73         na       4,486    8,952        1,141         249      na      10,342      14,828
  15         812     334         295        na       1,441    4,313         650           69      na      5,032       6,473
  23        8,093   5,169       1,286       na       14,548   26,557       4,536         955      na      32,048      46,596
                                                        Trend Data
FY 07-08   48,101   28,981      4,872       na       81,954   138,913     34,181        3,693     na      176,787     258,741
FY 06-07   53,885   33,633      6,062       na       93,580   137,951     35,184        3,282     na      176,417     269,997
FY 05-06   58,237   32,370      5,285       na       95,892   128,705     31,711        3,819     na      164,235     260,127
FY 04-05   56,375   29,069      2,801       na       88,245   102,030     40,420        3,167     na      145,617     233,862
FY 03-04   49,935   28,752      4,451      3,468     86,606   78,319      48,101        2,507   11,822    140,749     227,355
FY 02-03   48,785   25,667      3,355       na       77,803   75,298      62,995        1,717     na      140,010     217,813


    Note:
           FY 2007-08 numbers only include data reported and posted on the DCF Dashboard by 9-15-08
           deadline; they do not include data reported in the SAMH database after this due date.
           District numbers do not add to the statewide totals because the latter have been unduplicated for
           clients served in more than one district
           As a result of the Medicaid reforms, the Agency for Health Care Administration (AHCA) no longer
           requires the managed care organizations to report client data in the Substance Abuse and Mental
           Health Information System (SAMHIS). This has reduced the number of children reported in the
           SAMHIS database, and to a lesser extent the number of adults reported. For example, the number
           of children served in the community mental health programs decreased by 12.4 percent from
           93,580 in FY 2006-2007 to 81,954 in FY 2007-2008. During the same period of time, the number of
           adults served in the community mental health programs increased slightly by 1.0 percent from
           176,417 to 176,787.
           From FY 2004-2005 to FY 2005-2006, the number of adults in mental health crisis decreased by
           21.5 percent from 40,420 to 31,711, and the number of adults with severe and persistent mental
           illness (SPMI) increased by 12.8 percent from 102,030 to 128,705, due to a change in the definition
           of those target population groups. That change divided the previously defined category of “Adults
           in mental health crisis” into two new categories: “Adults with serious and acute mental illness”,


                                                                                                                    Page 9
         Compliance with Annual Performance Outcome Standards - FY 2007-08 Legislative Status Report


     and “Adults with mental health problems.” As a result, some clients previously counted as “adults
     in mental health crisis” were transferred to the “SPMI” category.

Table 4 shows, by district, the total number of adults and children served in community
substance abuse programs during FY 2007-2008. This table also shows the trend data from
FY 2002-2003 to FY 2007-2008. Across the five year intervals shown in this table, the number of
children served decreased by 19.4 percent from 62,681 to 50,502. This accounts for an average
yearly decrease of 2,439 children served or a 3.9 percent annual average decrease. During the
same period, adults served increased by 10.6 percent from 101,661 to 111,366, which accounts
for an average increase of 1,941 adults served annually or a 1.9 percent annual average
increase.
 Table 4: Number of Clients Served in Community Substance Abuse Providers in FY 2007-08


          District              Adults                  Children              Total (Adults +
                                                                                Children)
            1                   5,552                  2,562                      8,114
            2                   5,667                  2,187                      7,854
            3                   3,526                  1,058                      4,584
            4                   9,200                  3,294                     12,494
            7                   13,654                 7,277                     20,931
            8                   5,695                  1,626                      7,321
            9                   9,586                  4,301                     13,887
            10                  10,883                 2,283                     13,166
            11                  12,649                 6,763                     19,412
            12                  6,189                  1,769                      7,958
            13                  4,028                  2,312                      6,340
            14                  4,021                  2,658                      6,679
            15                  3,245                  1,547                      4,792
            23                  18,510                11,614                     30,124
                                             Trend Data
     FY2007-08 Total           111,366                50,502                     161,868
     FY2006-07 Total           107,811                53,024                     160,835
     FY2005-06 Total           102,345                51,929                     154,274
     FY2004-05 Total           109,355                56,941                     166,296
     FY2003-04 Total            93,541                60,862                     154,403
     FY2002-03 Total           101,661                62,681                     164,342

 Note:
    FY 2007-08 numbers only include data reported and posted on the DCF Dashboard by 9-15- 08
     deadline; they do not include data reported in the SAMH database after this due date.
    The average annual decrease in the number of children served is mostly due to change in the
     algorithm for counting clients served rather than to decrease in the number of clients actually


                                                                                                Page 10
         Compliance with Annual Performance Outcome Standards - FY 2007-08 Legislative Status Report


      served. For example, prior to FY 2004-2005, the count of clients served in substance abuse
      programs was based on client admission records rather than on service events records. This count
      also included clients served in all licensed substance abuse programs regardless of whether or not
      they had a SAMH contract. Since the migration of the data system from Oracle database into the
      CACHE web-based application in FY 2003-2004, the count of clients served has included clients
      admitted in substance abuse programs, who also had at least one service event paid by the SAMH
      contract, Medicaid, or local match.
     The number of adults served (n = 102,345) in FY 2005-06 does not include clients served in two
      programs: Access to Recovery and the Brief Intervention and Treatment for the Elderly (BRITE)
      grant. If numbers served by those two programs are included, the total number of clients served
      would be 108,924.

VII. Method for Determining “Successful” Contract Performance
This section explains the method used to define and determine whether or not provider
contracts successfully met the GAA performance outcome standards.

Each contract which has GAA outcome measures in Exhibit D has a minimum performance
standard, i.e., target, associated with each Target Population and Performance Measure
Combination (PopMeasure). Therefore, a contract with ten PopMeasures would have ten
performance standards. In Florida’s mental health and substance abuse contracting systems
for FY 2007-08, the number of standards per contract varies widely. Since a provider can meet
one standard and not another on the same contract, it was necessary to find a way to express
the overall contract performance across all the PopMeasures in a single number. The following
are statistical procedures used to accomplish this task.
1. PopMeasure Performance Ratio (PPR) - For each PopMeasure in a contract, a PPR was
   calculated by subtracting the Performance Standard from the PopMeasure and dividing the
   result by the Performance Standard. This deviation ratio measure is zero if a contract
   performance is identical to the standard. Performance scores above the standard have
   positive values and performance scores below the standard have negative values. For one
   measure in Mental Health (M0376) desired scores are equal to or below the standard, i.e.,
   lower scores are better. In this case the described deviation ratio is multiplied by -1 to
   reverse the direction.

     For example, the SPMI (Severe and Persistently Mentally Ill) target population has a
     measure known as Days in the Community. If the minimum standard was 350 days and the
     actual performance was 360 days, this ratio would be 0.0286, as indicated in the examples
     shown in Figure 1.
     (PPR = [Contract Actual - Contract Standard] / Contract Standard)




                                                                                                 Page 11
        Compliance with Annual Performance Outcome Standards - FY 2007-08 Legislative Status Report




   Figure 1, below, provides example for calculation of PPR as defined above in Step 1.


     Contract XX402
       Target                                   Performance             Actual
     Population              Measure              Standard           Performance            PPR
     AMH: SPMI       Avg. Annual Work for Pay        40                   30              -0.2500
     AMH: SPMI          Days in Community            350                 360               0.0286
     AMH: Crisis        Median LOS in CSU             3                    6              -1.0000
     CMH: ED        Percent Improved Function        64                   64               0.0000
      CMH: SED          Days in Community            350                 365               0.0429

Figure 1: Sample Step 1 - Calculation of PopMeasure Performance Ratios (PPR)

2. State Performance Ratio (SPR) – For each PopMeasure, this ratio was calculated by dividing
   the sum of all the PPRs statewide, by the number of PPRs statewide. This is simply the
   average PPR for that PopMeasure across all contracts. For example, if 126 contracts had a
   PopMeasure (for the SPMI population) of "Days in the Community,” then the State
   Performance Ratio (SPR) for contracts serving that population would be the sum of their
   PPRs divided by 126. If the sum of 126 PPRs was 110, the SPR would be 0.87. Note that
   each PopMeasure may only have one State Performance Ratio.
   (SPR = Sum of PPRs / Number of PPRs)

3. The Statewide Standard Deviation (SSD) of the SPR was then calculated for each
   PopMeasure to measure the variance between PopMeasure Performance Ratio (PPR) and
   State Performance Ratio (SPR). Simply put, the SSD is the average difference between the
   PPRs and the SPR; in technical terms, it is the square root of the variance between the PPRs
   and the SPR. See Figure 2, for an example of steps 2 and 3.
   (SSD =Square root of [Sum of [PPR - SPR]2 /(Number of PPRs-1))]

   Figure 2, provides example for calculation of SPR and SSD as defined above in Steps 2 and 3.




                                                                                           Page 12
        Compliance with Annual Performance Outcome Standards - FY 2007-08 Legislative Status Report


     PopMeasure = SPMI / Days in Community


       Contract                  Performance                        Actual
       Number                      Standard                      Performance                      PPR
        XX402                         350                            360                          1.03
        ZZ201                         345                            365                          0.94
        AB690                         75                              60                          1.25
        CR304                         320                            200                          1.60
                                                                                        Total     4.82

                 Total of PPRs (4.82) divided by # of PPRs (4) = State Performance Ratio (SPR)    1.20
     Average difference between the PPRs and the SPR = Statewide Standard Deviation (SSD)         0.29

Figure 2: Example for Calculation of SPR and SSD

4. PopMeasure Z-Score (PZ) - Since performance measures have different standards based on
   different measurement scales (e.g., community days, median length of stay, and percent of
   people with improved level of functioning), the Z-scores are generally used to compare
   values from different measurement instruments or scales. In this case, the Z-score was
   calculated for each PopMeasure in each contract by subtracting the State Performance
   Ratio (SPR) from the PopMeasure Performance Ratio (PPR) and dividing the result by the
   State Standard Deviation (SSD). A Z-Score indicates the number of standard deviations the
   PPR is above or below the SPR. For example, a Z-score of +1.00 is one standard deviation
   above the SPR, while a Z-Score of –1.00 is one standard deviation below the SPR. With a
   normal distribution, about 68 percent of the PPR values will lie between –1.00 and +1.00.
   Furthermore, about 95 percent of the PPR values will lie between –2.00 and +2.00 or two
   standard deviations.
   (PZ = [PopMeasure Performance Ratio - State Performance Ratio] / Statewide Standard
   Deviation).

5. Overall Contract Performance (OCP) - Next, all the Z-scores for all the PopMeasures in the
   contract were averaged to determine the overall contract performance. The OCP provides
   the single number needed to represent the contract performance across all the different
   combinations of target populations and performance measures. The distribution of these
   OCPs across contracts was examined and determined to be approximately normal, with a
   standard deviation of 0.7615 and a mean (average) of 0.0558.
   (OCP = Sum of [PZ1 thru PZn] / Number of PopMeasures in contract).

6. OCPz - To better show the magnitude of differences among OCP values, OCP scores were
   converted to standard scores with a mean of 0 and a standard deviation of 1. The statistical
   data analysis determined that a contract with an OCPz value at or above one standard
   deviation below the mean (–1.00) was considered to have successful performance. This
   method of “grading to the curve” helps compensate for some of the variance due to
   unspecified variables that often unfairly influence outcome performance due to the



                                                                                                 Page 13
        Compliance with Annual Performance Outcome Standards - FY 2007-08 Legislative Status Report


   providers’ inability to control them.
   (OCPz = (OCP-Average OCP)/Standard Deviation of OCP’s).


Figure 3, below, provides an example for calculating OCP and OCPz as defined above in Steps 5
and 6. Because the OCPz of –0.608 in this sample data set is above -1.00 (using 1.00 as the
hypothetical standard deviation and 0 as the hypothetical mean for z-scores), contract XX402
passed the performance compliance test.


   Contract XX402




Performance          Actual
 Population         Measure               Standard        Performance            PPR               PZ
    SPMI       Days in Community             350              360               0.0286          -0.429
    SPMI    Avg. Annual Work for Pay         40                30              -0.2500          -0.140
     SED      % Improved Function            64                64               0.0000            -1.06
     SED       Days in Community             350              200              -0.4286          -0.959
                                                                                           Sum -1.600
                                                Overall Contract Performance (OCP) Average -0.400
                                                                                Mean of OCP = 0.0527
                                                                    Standard Deviation of OCP = 0.7446
                                      Final Scaled Overall Contract Performance (OCPz)          -0.608



Figure 3: Example of Steps 4 and 5 - Calculation of PZ and OCP for a Single Contract with a
           Sample Data Set


7. Summary of acronyms used in this method, the terms to which they apply, and how they
   relate:
   PopMeasure = Target Population and Performance Measure Combination (defined by the
         M-Code)
   PPR = PopMeasure Performance Ratio: calculated for each PopMeasure in a contract by
          dividing the Actual Performance by the Target
   SPR = State Performance Ratio: the average PPR for a specific PopMeasure across all
           contracts, statewide
   SSD = Statewide Standard Deviation: the average difference between the PPR and the SPR
           for each PopMeasure across all contracts
   PZ = PopMeasure Z-Score: calculated for each PopMeasure in a contract by subtracting the
          SPR from the PPR and dividing by the SSD


                                                                                             Page 14
              Compliance with Annual Performance Outcome Standards - FY 2007-08 Legislative Status Report


        OCP = Overall Contract Performance: a single score for each contract that is the average of
               the PZs in the contract
        OCPz = Overall Contract Performance: represented as a standardized score with a mean of
               zero and a standard deviation of one.

   VIII. Performance Outcome Results

   As shown in Table 5, Column 2, there were 227 SAMH contracts with performance data and
   minimum standards that met the conditions for calculating the success indicators in FY 07-08.
   As shown in Column 3 of this table, of these 227 contracts, 215 met the level of success defined
   above in the Method section. This column also indicates that the SAMH contract success rate
   has increased by 2.2 percent from 92.5 percent in FY 2006-2007 to 94.7 percent during FY 2007-
   2008.

                                Table 5: Contract Performance Results for FY 2007-08
                                                        (3)                                (4)
                        (2)
                                                  UNSUCCESSFUL                         SUCCESSFUL
            SAMH Contracts that Require
  (1)                                          Contracts with Overall     Contracts with Overall Performance
          Outcome Measures and Have GAA
                                         Performance that Was Below the –      that Was Above or at the –
            Standards and Outcome Data
                                              1.00 OCP Z-Score Cutoff            1.00 OCP Z-Score Cutoff
District FY04-05 FY05-06 FY06-07 FY07-08 FY04-05 FY05-06 FY06-07 FY07-08 FY04-05 FY05-06 FY0607 FY0708
   1        6        11       4        5    0       1         1       0      6          10       3        5
   2        14       19      18        8    1       2         1       0      13         17      17        8
   3        8         7       8       10    0       1         0       0      8          6        8       10
   4        15       41      31       11    0       3         3       1      15         38      28       10
   7        24       27      30       24    3       1         2       3      21         26      28       21
   8        5        19      13       14    0       1         1       0      5          18      12       14
   9        28       25      23       21    1       0         2       2      27         25      21       19
  10        33       33      31       27    2       3         5       1      31         30      26       26
  11        37       65      38       31    7       6         3       2      30         59      35       29
  12        13       14      12       12    0       0         0       0      13         14      12       12
  13        7         9       5        5    0       1         0       1      7          8        5        4
  14        10       19      14       14    1       3         0       0      9          16      14       14
  15        23       19      15       16    1       0         1       0      22         19      14       16
  SC        36       45      38       29    0       3         2       2      36         42      36       27
               2                3                                         243/259 328/353 259/280 215/227
 Total     259      353     280      227   16       25       21       12
                                                                           (94%)      (93%) (92.5% ) (94.7%)




   2
     These 259 contracts include 2 duplicate contracts reported in multiple districts. The unduplicated number of successful contracts is
   257
   3
     These 280 contracts include 2 duplicate contracts reported in multiple districts. The unduplicated number of contracts accountable
   for GAA performance standards is 278.


                                                                                                                               Page 15
        Compliance with Annual Performance Outcome Standards - FY 2007-08 Legislative Status Report



IX. District Corrective Actions Taken for Unsuccessful Contracts

This section provides information on district actions taken concerning each of the 12 contracts
that did not meet the cutoff for success in FY 2007-2008. Table 6 lists the 12 unsuccessful
contracts per district and shows the number of standard deviations below the mean for each
contract. The last column in this table provides district comments, including corrective actions
taken to improve FY 2008-2009 performance outcomes of these unsuccessful SAMH contracts.
Furthermore, for all unmet GAA outcome measure targets, the SAMH Program Office is working with
districts and local providers to perform detailed analyses of these outcome measures to determine the
root causes and drivers needed to implement the actions necessary to improve performance. This
corrective action will be implemented and monitored in FY 2008-2009 as part of the Performance
Management Teams (PMT) process.




                                                                                               Page 16
                                  Table 6: District Comments for 12 Unsuccessful Contracts (FY 2007-08)

                                                                                Number of
                         OCPZ - Standard    Contract Rank       Number of
                                                                                  Failed       District Comments Including Corrective Actions Being
District   Contract ID   Deviations below   (lowest rank is   PopMeasures in
                                                                               PopMeasures                            Taken
                            the Mean        worst)             the Contract
                                                                                (PZ<-1.00)
                                                                                             Issues pertaining to this measure are being discussed to
 D04         DH651           -3.8983               2                1               1
                                                                                             examine areas that need improvement in FY 08-09.
 D07         GDH05           -1.4555               5                1               1        Contract terminated as of June 30, 2008.
 D07         GHG26           -1.0095              12                5               3        Contract terminated as of May 31, 2008.
                                                                                             FY 2007-2008 standard targets for this performance
                                                                                             outcome measure were unrealistic for the FACT teams.
 D07         GHH42           -1.1220              10                3               1
                                                                                             These targets were adjusted for FY 08-09. With more
                                                                                             realistic targets, we will be able to reach them this year.
                                                                                             Agency Response: Only 75% of the goal was met. There
                                                                                             has been a slowdown in the economy affecting the
                                                                                             joblessness rate---this reality has affected our clients as
                                                                                             well. We have recently increased employability services
                                                                                             through our Family Resource Center to assist clients in job
 D09         ID087           -1.3275               8                3               2
                                                                                             readiness. We are currently negotiating with TED Center
                                                                                             to bring additional employability training to our clients.
                                                                                             We expect to complete this negotiation within the next
                                                                                             few weeks. We are also looking for other employment
                                                                                             and employability resources for our varied client base.
                                                                                             Agency Response: In regard to targets pertaining to
                                                                                             improvement in level of function and days spent in the
                                                                                             community, Boys Town has had numerous conversations
                                                                                             with the local SAMH office and the SAMH office in
                                                                                             Tallahassee about the types of children served under this
                                                                                             contract. Boys Town continues to serve children while
                                                                                             they are in State Inpatient Psychiatric Placements (SIPP
                                                                                             placements). As a result, this significantly affects the
 D09         IH558           -1.4122               6                4               1
                                                                                             number of days in the community and level of
                                                                                             functioning. In the last conversation with staff in
                                                                                             Tallahassee, a representative from that office
                                                                                             recommended that Boys Town's standards be modified to
                                                                                             reflect the severity of the population that Boys Town
                                                                                             serves. The Contract Manager will review this request
                                                                                             along with additional solutions that will be addressed at
                                                                                             October Adolescent Provider Meeting.
                                                                                                                                                    Page 17
                                                       Compliance with Annual Performance Outcome Standards - FY 2007-08 Legislative Status Report
                                  Table 6: District Comments for 12 Unsuccessful Contracts (FY 2007-08)

                                                                                Number of
                         OCPZ - Standard    Contract Rank       Number of
                                                                                  Failed       District Comments Including Corrective Actions Being
District   Contract ID   Deviations below   (lowest rank is   PopMeasures in
                                                                               PopMeasures                            Taken
                            the Mean        worst)             the Contract
                                                                                (PZ<-1.00)
                                                                                             Target rate may be higher than appropriate for specific
                                                                                             population. Department and provider will work together
                                                                                             to determine appropriate target including reviewing
 D10         JD225           -1.1148              11                3               2        impact of disability level of population. Provider claims
                                                                                             completion rate is higher today. Department data liaison
                                                                                             and the Provider staff will evaluate current data entry
                                                                                             methods and their impact on this performance outcome.
                                                                                             Provider Response: Since some FACT clients were
                                                                                             hospitalized or in jail during this period, the target for
                                                                                             days spent in community was not met. Closer monitoring
 D11         KH154           -1.2947               9                3               2        will be provided by the FACT team to improve this target.
                                                                                             Furthermore, data were not correctly reported for this
                                                                                             outcome; appropriate steps are being taken to reported
                                                                                             data correctly in FY 08-09.
                                                                                             Provider Response: Sandy Pines is a residential lockdown
                                                                                             facility and, therefore, our residents have a limited
                                                                                             number of days in the community while they are here.
 D11         KH190           -7.7525               1                1               1
                                                                                             That is why our performance value of 45.96 in this area is
                                                                                             much lower than the contract performance target of 360
                                                                                             days in the community.
                                                                                             District corrective action is pending the response from the
 D13         PDG06           -3.5888               3                1               1
                                                                                             provider.
                                                                                             Provider’s problems with data submission resulted in low
 D23         QH6E8           -2.5972               4                1               1        performance outcome. A corrective action plan (CAP) has
                                                                                             been requested to improve performance in FY 08-09.
                                                                                             The current corrective action plan (CAP) is to continue to
 D23         QH7A5           -1.3525               7                3               2        monitor and provide technical assistance with plan
                                                                                             revision as needed. (Both measures).




                                                                                                                                                   Page 18
X. Discussion and Recommendations
1. There were 412 SAMH contracts recorded in the DCF Schedule of Allotment Balances for
   FY 2007-2008; only 227 of these contracts met the conditions for contracts that are accountable for
   GAA performance outcome measures in FY 2007-2008. Of the 227 contracts accountable for GAA
   outcome measures in FY 2007-2008, 215 met the compliance test, for a 94.7% success rate.
2. As shown in Table 6, districts provided comments and recommendations regarding the corrective
   actions for the 12 contracts that failed to meet the GAA performance standards. The SAMH
   Program Office is working with districts and local providers to implement and monitor these
   corrective action plans as part of the Performance Management Teams (PMT) process. The PMT
   process includes regular conference call meetings with district data liaisons, DCF Contract Oversight
   Unit staff, and district contract managers to discuss pending performance compliance problems
   and pertinent corrective action plans.
3. A more rapid feedback approach is being implemented in FY 2008-2009 to provide timely
   information needed by contract managers and provider staff to identify and monitor the
   performance outcome status of each SAMH contract each month throughout the fiscal year. The
   SAMH Program Office will produce a year-to-date monthly report showing SAMH contracts that are
   below, meet or exceed the GAA standard target for each performance outcome measure. Based on
   this report, contracted providers performing below targets will be required to develop and
   implement a corrective action plans (CAP) needed to improve their performance outcomes.
4. The SAMH Program Office has already developed online Exception Reports, which are available to
   district contract managers to ensure that the SAMH contract database is updated regularly to
   include all the GAA outcome measures and targets for all the contracts that have these statutory
   data requirements. These reports are also available to providers to identify and track contracts in
   the SAMH contract database that are not submitting data in the SAMH data system. These online
   exception reports are accessible statewide via the DCF Intranet at
   http://ewqas1.dcf.state.fl.us/PDADM/SAMHRpts/frmMenuExceptRpts.asp.




                                                                                                 Page 19

								
To top