The Arizona Works Pilot Program A Three Year Assessment

The Arizona Works Pilot Program A Three-Year Assessment Executive Summary Cambridge, MA Lexington, MA Hadley, MA Bethesda, MD Washington, DC Chicago, IL Cairo, Egypt Johannesburg, South Africa February 2003 Prepared for Arizona Department of Economic Security 1789 West Jefferson, S/C 901A Phoenix, AZ 85007 Prepared by Bob Kornfeld, Project Director Diane Porcari, Abt Associates Inc. Abt Associates Inc. 55 Wheeler Street Cambridge, MA 02138 Laura R. Peck Arizona State University, School of Public Affairs Executive Summary This report is an evaluation of Arizona Works, a program that privatized TANF (Temporary Assistance for Needy Families), the cash assistance program for low-income families with children. Under Arizona Works, a private contractor, MAXIMUS, Inc., administered intake, eligibility, and employment and training programs. Beginning in April 1999, Arizona Works operated in Eastern Maricopa County, on the eastern edge of the city of Phoenix. The program featured a strong “work first” orientation and a commitment to help TANF recipients leave TANF through employment and self-sufficiency. As the Arizona Works contractor, MAXIMUS received a base payment for administering the program, and also received additional incentive payments for attaining specified performance standards. Many states have employed private contractors to administer training programs or support services, but Arizona Works and Wisconsin’s W-2 program are unusual in that these programs also privatized intake, eligibility, and benefit determination. The federal government denied the state of Arizona’s request for waivers to allow MAXIMUS, Inc. to administer eligibility and benefit determination for the Food Stamp Program and Medicaid. As a result, public sector employees from the Arizona Department of Economic Security (DES) remained in the local Arizona Works offices to administer these programs and non-TANF child care. MAXIMUS was responsible for operating the TANF program, the TANF employment program, child care for TANF families, Transitional Child Care, the state-funded General Assistance Program, and the Food Stamp Employment and Training program. In the rest of Maricopa County and in the rest of the state, TANF was and continues to be administered by the Arizona Department of Economic Security (DES) under a program known as EMPOWER Redesign. EMPOWER Redesign also features a strong commitment to work and selfsufficiency. From April 1999 to October 2002, Arizona Works and EMPOWER Redesign differed because only Arizona Works was privatized, because the two programs had some different program rules, and because only MAXIMUS was awarded incentive payments for attaining performance standards. Some of the performance standards compared full-time employment rates and other outcomes attained by those subject to Arizona Works and those subject to EMPOWER Redesign in the designated comparison area, which was the rest of Maricopa County. The evaluation of Arizona Works describes the implementation of the program and compares the performance of Arizona Works and EMPOWER Redesign. The study reviews events from the beginning of Arizona Works in April 1999 through March 2002. In June 2002, new legislation changed Arizona Works dramatically. The new law restored intake and benefit determination to the public sector, repealed program rules specific to Arizona Works, eliminated the Arizona Works Agency Procurement Board, and called for the privatization of the state’s Jobs Program statewide. Although this report examines Arizona Works as it existed before this legislation, many of its findings remain relevant for the state’s current plans for privatizing TANF services. The evaluation consists of several parts: • The process study describes the development, planning, organization, staffing, community partnerships, service delivery, and implementation experience of Arizona Works and EMPOWER Redesign. The impact study compares a wide range of outcome measures -- such as earned income, Executive Summary ix • Abt Associates Inc. levels of welfare benefits received, and participation in work-related activities -- achieved by the families subject to Arizona Works and the families in the comparison area subject to EMPOWER Redesign. This comparison takes into account the differences in program rules, the performance measures, and pre-existing differences in the characteristics of the caseloads and neighborhoods served by the two programs. • • The cost-benefit analysis assesses whether the Arizona Works Pilot Program made the welfare families and the Arizona state government and taxpayers “better off” or “worse off.” The client survey asks some adults in families served by Arizona Works and EMPOWER Redesign questions about client satisfaction with the programs and other issues. Main Findings The Process Study During the first three years of the Arizona Works Pilot Program, MAXIMUS, Inc. clearly met the requirements of its contract to administer Arizona Works. The organization and staffing plans of Arizona Works were clearly capable of meeting the goals of the program. Key roles were welldefined and procedures for monitoring and problem resolution were put in place. Arizona Works local office staff quickly established procedures for managing intake, eligibility, and benefit determination. EMPOWER Redesign also features a well-designed organization and staffing plan. During these years, these two work-focused programs had many similar procedures and used identical application forms, and the two programs offered a wide array of often similar support services and job search and readiness programs. Other conclusions from the process study are as follows: • Private companies such as MAXIMUS, Inc. can successfully implement large TANF programs that administer intake and benefit determination and welfare-to-work programs. In Arizona, the success of privatization depended, however, on cooperation and support from state government, which provided facilities, access to program data, monitoring and oversight, and staff. The privatization of TANF intake and benefit determination faces numerous d ifficulties because the federal government requires that public sector staff administer intake and benefit determination for food stamps and medical assistance. Because intake and benefit determination procedures of all of these programs have many tasks in common, it is difficult to divide these responsibilities efficiently across agencies. The implementation of a successful privatized TANF program takes several months for even the most experienced contractors as they hire new staff, implement new systems and procedures, and generally “learn by doing.” Ongoing changes to staffing arrangements and operating procedures are inevitable, as they are in public agencies. Community groups, advocacy groups, and local service providers expressed both concerns and favorable opinions about Arizona Works and EMPOWER Redesign. In general, private contractors may encounter opposition from some local groups, especially when some of these groups believe that the existing relationships with state and other public agencies are effective. Outreach efforts to address these concerns may be a key challenge for private contractors. Executive Summary Abt Associates Inc. • • • x • Complex programs such as Arizona Works require professional staff to monitor the performance of the contractor, negotiate contract disputes, and facilitate coordination between the contractor and the public sector. Professional staff could be acquired from a consulting firm or from the public sector. One of the major challenges of any privatization effort is the creation of an independent group to oversee the project when virtually all interested parties work for either the public or the private sectors. The costs of a privatization scheme include not only the value of the contract but also the labor costs of monitoring the program, calculating performance measures, holding public meetings, and conducting private meetings to coordinate the efforts of public and private employees. Although these costs are difficult to quantify, staff interviewed felt that these monitoring costs added significantly to the overall costs of the pilot. Privatization, like any major institutional change, should be implemented decisively as planned. In Mohave County, the delay and ultimate suspension of Arizona Works left local service providers and welfare recipients “in limbo” since it was unclear how TANF services would be administered. These uncertainties may have inhibited service provision. • • The pilot program as a “test:” A key question for this study is whether privatization led to differences in the organization of local offices or in the procedures for delivering public services. Despite the occasionally fierce debate about privatization, these differences were not obvious. The two organizations had staff playing many similar roles, and they provided many similar programs and support services. The absence of a large number of clear-cut procedural differences between the two programs may explain the results of the impact study, which generally found that the two programs were about equally successful in increasing the employment and earnings of participants. Some lessons from the study of Arizona Works as a “test” of privatization include the following: • Although Arizona Works and EMPOWER Redesign were different in that many TANFrelated services were privatized under Arizona Works, the pilot was less than a clear-cut test of privatization. Arizona Works relied on data systems and facilities from the public sector, and hired many former public sector employees. The public sector still operated food stamps and medical assistance. Similarly, the public program was not entirely public and subcontracted several services. The two programs also had some different rules regarding grant size, time limits, and other issues. Some tension between public and private staff members in the local offices was present during the pilot. This tension may be inevitable in any attempt to test the relative effects of public and private systems. This competitive atmosphere could have improved services but also could have discouraged cooperation and promoted less productive behavior. Privatization efforts may need an independent study to assess what, if anything, is truly different about the operations of public and private programs. One cannot simply assume the two programs are dramatically different. • • Abt Associates Inc. Executive Summary xi The Impact Study In this report, the “impact” of Arizona Works is defined as the program’s effect, relative to the effect of EMPOWER Redesign, the state program that operated in the comparison area and in the rest of the state. These impacts are estimated for the entire pilot group of adults and families that were on TANF in the Arizona Works pilot area during the first three program years, regardless of whether these adults or families eventually left TANF or received earned income over the observation period. During the first three program years, Arizona Works had no impact on earned income and had varying effects on reliance on TANF among these adults. The evidence indicates that: • Relative to EMPOWER Redesign, Arizona Works did not increase the percentage of adults who were employed and did not increase the average earnings of adults. This finding is based on employer-reported data, which are the best available data for assessing the impacts of Arizona Works because these data include employment obtained while families were on welfare and after they left welfare. Arizona Works increased many measures of total employment, full-time employment, and earnings that were based on data reported by local welfare office staff, although these same data indicate that Arizona Works did not increase full-time employment among adults in the second and third years of the program. The welfare office data were used to calculate the Arizona Works performance measures. The welfare office data provide an incomplete picture of employment, however, because they do not record jobs of some welfare recipients and because they do not record jobs received after persons left welfare. Arizona Works increased the proportion of adults who participated in unpaid community service or work experience, GED or ESL programs, job search assistance, and job readiness and training activities. Arizona Works reduced the period of time in which the average adult participated in TANF by about one to two weeks during the first three years of the program. Arizona Works increased the average monthly TANF grant by about $22. Arizona Works did not change the average period of time in which the average adult received food stamps during these three years. Arizona Works reduced average monthly food stamp benefits by about $9 per month. Arizona Works increased the period of time in which the average adult was enrolled in public medical assistance programs, including Medicaid. The average size of this impact was about two weeks. During January-March of 2002, the last calendar quarter of the second program year, Arizona Works did not increase total family income. The program increased the proportion of all adults who had left TANF with earned income by 1.5 percentage points, but did not reduce the proportion of all adults who had left TANF without earned income. Arizona Works had no effect on documented and substantiated instances of child abuse and neglect. Executive Summary Abt Associates Inc. • • • • • • • • xii Through the first three years of the Arizona Works Pilot Program, Arizona Works and EMPOWER Redesign were about equally successful in encouraging TANF recipients to leave TANF for paid employment and attain self-sufficiency. Under both programs, large numbers of adults found employment, left TANF, and participated in work-related activities. These estimated impacts are not the impacts of privatized TANF services alone but the combined impact of privatization, the Arizona Works performance incentives, and differences in the rules of the two programs. The separate impact of each of these factors is difficult to determine because one can only observe their combined impact. It seems likely that much of the increase in average TANF benefits is the result of differences in the program rules rather than the result of privatization alone. The separate impact of each of these factors on other outcomes is less clear. The discrepancies in estimated impacts on employment and earnings obtained from the two data sources exist mainly because the proportion of adults with employment recorded only employers, and not by welfare office staff, was larger among those subject to EMPOWER Redesign than among those subject to Arizona Works. One explanation for this finding is that the performance measures of Arizona Works gave local office staff a relatively greater incentive to report earnings. The Arizona Works program rules (under which earnings are not subtracted from TANF grants) also gave recipients a relatively greater incentive to conceal earnings under EMPOWER Redesign. The impact estimates and the performance measures: The performance measures and impact study reach some different conclusions about the effectiveness of Arizona Works. The impact study finds that Arizona Works had little impact on total employment and earnings but the performance measures suggest that the program increased some but not all measures of full-time employment. Unlike the performance measures, the preferred impact estimates use the more comprehensive employer-reported earnings and consider the role of caseload and neighborhood characteristics by examining outcomes in the pilot and comparison areas before Arizona Works. The impact estimates based on the employer-reported records and the performance measures of full-time employment cannot be compared directly because the employers record only total earnings and not hours or fulltime status. The weaknesses of the welfare office data and the conflicting impact estimates on total employment from the two data sources nevertheless cast doubt on whether the performance measures and the welfare office data should be seen as reliable indicators of the impact of Arizona Works. Competitive effects: The perceived competition between Arizona Works and EMPOWER Redesign could, in theory, have also led to improvements in the performance of EMPOWER Redesign in ways that are not directly measured by the impact study, but the evidence for such a “competitive effect” of Arizona Works is weak. During these years, average earnings and placements in work-related activities did increase under EMPOWER Redesign. It is possible that Arizona Works increased work placements under EMPOWER Redesign, or the proportion of already-existing jobs recorded by EMPOWER Redesign staff. However, many other factors – the strong economy, t e increasing h minimum work participation rates required under TANF, and ongoing efforts to improve the state TANF program –probably caused at least a substantial portion of these trends. Similar trends in employment, earnings, and work activities also occurred among TANF recipients across the nation during these years, when the vast majority of states experienced a strong economy and implemented publicly administered TANF programs but did not employ competing public and private programs. Accordingly, there is no evidence that welfare families are either better off or worse off under privatized TANF. Abt Associates Inc. Executive Summary xiii The Cost -Benefit Analysis The cost-benefit study estimates the cost of the Arizona Works Pilot Program, relative to the costs that would have been incurred in the absence of the program. The original Arizona Works contract was structured to cost less than the state’s projections of costs that would have been incurred, had EMPOWER Redesign operated all welfare programs in Maricopa County instead. Despite these contract provisions, the total costs of Arizona Works most likely exceeded these projected costs, from the perspective of the state government and state taxpayers. When the federal waivers were denied, the state had to administer food stamps and medical assistance, and the responsibilities and base contract payments for Arizona Works were scaled back. The size of the available pool of incentive payments remained unchanged, however. As a result, it was possible for total contract payments to exceed the projected operations costs of the subset of programs operated by Arizona Works. The pilot program’s total costs – including contract payments and several other categories of costs – probably exceeded the projected costs that would have been incurred under the state program. It is ultimately difficult to assess whether TANF-related programs were relatively more expensive to administer under Arizona Works or EMPOWER Redesign because it is difficult to measure the separate costs of administering TANF-related programs, food stamps and Medicaid under EMPOWER Redesign. TANF, food stamps and Medicaid have many administrative tasks in common and most TANF recipients are eligible for all three programs, so one cannot observe the separate administrative costs of each program. It should be recognized, however, that the total cost of Arizona Works also included the cost of professional staff to oversee the pilot, the cost of the independent evaluation, and the substantial increase in average monthly TANF benefits caused by the program rules of Arizona Works. The sum of all of these costs of Arizona Works most likely exceeded the costs that would have been incurred without the pilot. It is important to note that the main factors behind the increase in costs under Arizona Works – the features of the initial Arizona Works contract and the increase in average TANF payments caused by the Arizona Works program rules–no longer exist given recent legislative changes to the program. This cost analysis does not apply to future privatization of employment services alone. From the perspective of the welfare recipients, Arizona Works increased average monthly TANF payments, reduced average food stamp benefits, and did not increase average earnings. As a result, the program’s impact on total family income was statistically insignificant, and by this measure, the families subject to Arizona Works were generally neither better off nor worse off as a result of the program. The increase in TANF payments was probably caused by t e unique program rules of h Arizona Works, which were recently eliminated. The Client Survey The participant satisfaction survey, a telephone survey of over 400 current and recent TANF recipients subject to Arizona Works or EMPOWER Redesign in the comparison area, was designed to learn about the perceptions of local office operations expressed by TANF recipients. The potential sample of respondents included only recent, “work-ready” TANF recipients to minimize recall problems. This initial potential sample of respondents omitted adults who were exempt from work xiv Executive Summary Abt Associates Inc. requirements or who left TANF long ago. As a result, the final sample of survey respondents drawn from this initial sample are not a representative sample of the entire pilot group of TANF recipients subject to the two programs, and the survey responses cannot be used to compare the operations of the two programs. The responses to this survey instead provide a general sense of client perspectives. The findings of the survey include the following: • Most respondents served by both programs were “very” or “somewhat” satisfied with local office procedures for handling applications for assistance programs and for assessing the needs of families. Most respondents served by both programs found job search assistance, job readiness and career preparation programs, and support services either “very helpful” or “somewhat helpful.” Some of the responses were less positive. About half of respondents claimed that they were not told that they could continue to receive food stamps after they left TANF for paid work. Although many respondents said they were encouraged to find work, somewhat less than half of respondents reported that they were not strongly encouraged to find work. About one quarter of respondents felt that local office staff “were not there to help them get the services they needed.” • • These less positive responses are difficult to interpret. It is possible that the two programs need to continue to work to improve the effectiveness of local office operations. It is also possible that local office operations are effective but that some respondents did not recall these messages. Because so many TANF recipients lack basic skills, and many cannot effectively communicate or listen to the details of messages from local office staff, it seems likely that at least some respondents will not receive even the clearest messages from local office staff. Regardless of one’s interpretation of these less positive responses, it should be noted that most responses to the participant survey expressed positive views of local office operations. The Future As a result of recent legislation, intake and benefit determination was returned to the state employees, the unique program rules of Arizona Works were dropped, and the state began to prepare for the statewide privatization of welfare-to-work programs. These changes are in many ways consistent with the findings of this study. Although some of the findings of the evaluation no longer apply to the future because of these legislative changes, other findings remain relevant. Welfare-to-work programs can be operated by private companies, but there is no clear evidence that privatization itself either helps or harms the average TANF recipient. Differences in the operations of various private and public welfare-to-work programs will probably not be obvious without additional independent research. Whether statewide privatization is cost-effective and helpful to the state’s efforts to manage public services will depend in part on the state’s ability to assist and monitor contractors and contracts efficiently. A key finding of the evaluation is that performance measures based on information reported by local welfare office staff can provide a misleading sense of the effectiveness of welfare-to-work programs. These welfare office records miss some earnings of welfare recipients, do not measure outcomes in the post-welfare period, and do not control for numerous other local factors that could also affect earnings and other outcomes. As welfare-to-work programs are privatized statewide, it may not be Abt Associates Inc. Executive Summary xv possible to assess which public and private programs are relatively more effective because so many factors could influence differences in the earnings of welfare recipients in different areas of the state. The next sections provide a more detailed description of Arizona Works and the findings of the process and impact studies. Overview of Arizona Works The original legislation that created Arizona Works called for a private contractor to administer welfare programs, including intake, eligibility, and employment programs, in District I-E, or Eastern Maricopa County, excluding Guadalupe and the reservation areas. MAXIMUS, Inc., a firm with extensive experience in the administration of welfare programs, was selected as the Arizona Works contractor. The pilot program began on April 1, 1999. The federal government subsequently denied the state’s request for waivers to allow MAXIMUS, Inc. to administer eligibility and benefit determination for the Food Stamp Program and Medicaid. As a result, public sector employees from the Arizona Department of Economic Security (DES) remained in the local Arizona Works offices to administer these programs and non-TANF child care In the rest of Maricopa County and in the rest of the state, TANF was and is administered by DES under the program known as EMPOWER Redesign. Both programs feature a strong “work first” orientation. The evaluation compares the performance of Arizona Works and EMPOWER Redesign in a comparison area, which is the remainder of Maricopa County. The comparison is difficult because the two programs and areas differed in ways other than the extent of privatization. Program Rules From 1999-2002, Arizona Works and EMPOWER Redesign had some different program rules. These differences in program rules were stipulated by the original Arizona Works legislation. As the Arizona Works contractor, MAXIMUS was required to follow these rules. The two programs had somewhat different procedures for assigning clients to activities. Under Arizona Works, participants were placed in one of four “levels” of employment. The most workready adults were assigned to Level One, full-time, unsubsidized employment. The program originally intended to place some participants in Level Two, subsidized employment, but no one was ever assigned to Level Two. The most common placement was Level Three, an unsubsidized, unpaid trial job with a community organization, for those who needed to improve their work skills or who needed further experience or training. Level Four, community referrals, were placements in work activities with community-based service providers that tried to provide mentoring and improve a participant’s employability. Although EMPOWER Redesign did not explicitly use these placement levels, adults eligible for TANF under EMPOWER Redesign were also assigned to employment and training programs such as paid employment, unpaid work and work experience, and activities with community groups. Both programs could also assign participants to other activities such as literacy and GED programs. Policies for deferments were similar under the two programs. Some of the other major similarities and differences in the two programs were as follows: • xvi Under Arizona Works, the maximum monthly grant was $390 for Level Three families and $350 for Level Four families, regardless of family size. Under EMPOWER Redesign, grant Executive Summary Abt Associates Inc. levels increase with family size: the monthly maximum grant (assuming zero earnings) is typically $275 for a two-person family, $347 for a three-person family, $418 for a fourperson family, and $489 for a five-person family. For large families, the maximum grant was larger under EMPOWER Redesign than under Arizona Works. For smaller families, the maximum grant was larger under Arizona Works than under EMPOWER Redesign. • Both programs determined eligibility the same way, and used the same maximum income ceilings and disregarded the same income sources. A difference between the programs was that only EMPOWER Redesign used earnings levels and earnings disregards to determine TANF grant levels. Under EMPOWER Redesign, grant levels declined as countable income rises, as long as families remain eligible. Under Arizona Works, families received the same flat grant as their income rose, as long as they remained eligible. Only EMPOWER Redesign families were subject to the Family Benefit Cap, which prevents additional cash assistance to mothers who give birth to additional children while receiving benefits. Arizona Works did not impose the Family Benefit Cap, which was irrelevant because the Arizona Works flat grants did not rise with family size. The two programs had different time limit rules. Under EMPOWER Redesign, adults may receive cash assistance for no more than 24 months in any 60-month time period. When an adult meets this time limit, the TANF grant for his or her family is reduced by the adult portion of the grant. Under Arizona Works legislation, clients could participate in Levels 2, 3, or 4 for no more than 24 months. At the end of 24 months of activity (as a Level 4 client, for example), a client was moved to a different level, where the client could then continue to receive TANF for another two years. The time limit rules of the two programs were not the same because, under Arizona Works, clients could be moved to another level after 24 months and still receive the adult portion of the TANF grant. Because Arizona Works clients were not subject to the “24 of 60 month” time limit under the state TANF plan, Arizona Works recipients were subject to the federal lifetime limit of 60 months of receipt of assistance, a limit that applied to the entire assistance unit. Both programs imposed graduated sanctions, leading to full family sanctions, for failure to comply with work requirements and cooperation with child support collections. Under EMPOWER Redesign, families may also incur progressive sanctions for failure to comply with immunization and school attendance requirements. Arizona Works Level Three and Four placements may have incurred hourly sanctions, at a rate of $3.25 per hour, for each hour in which activities were not performed without good cause. Many other program rules were similar under the two programs. The two programs shared, in addition to a strong work focus, similar rules for determining eligibility and countable income, and similar earnings disregard schedules. The application forms and application procedures for the two programs were identical. • • • • Performance Incentives Another major difference between EMPOWER Redesign and Arizona Works was the presence of performance incentives for Arizona Works. These incentives were based on performance measures developed to assess the extent to which the Arizona Works program moved families to selfAbt Associates Inc. Executive Summary xvii sufficiency. These incentives were also developed to provide motivation for the Arizona Works vendor to run an effective program. The performance incentives could have had an effect on welfareto-work outcomes distinct from the effect of privatization. Some of the performance incentives required meeting an absolute level of performance, while others required meeting a level of performance that was higher than the level of performance achieved in the comparison area subject to EMPOWER Redesign. The performance incentives were based on: 1. placement rates of all TANF recipients into full-time employment, relative to the comparison group; 2. placement of a specified percentage of deferred or exempt participants; 3. placement rates of the non-deferred population into full-time employment, relative to the comparison group; 4. placement of a specified percentage of measure “3” placements into full-time jobs with hourly wages at least $3.00 higher than the minimum wage; 5. placement of a specified percentage of measure “3” placements into full-time jobs with employer-provided health insurance; 6. reduction of the long-term caseload by a rate higher than seen in the comparison area; 7. reduction of the TANF caseload due to earned income by a rate higher than in the comparison area; and 8. retention of employment for at least 90 days. In the first three years of the pilot program, MAXIMUS, Inc. received some but not all of the available Administrative Payment Incentives for its performance. In the first year of program operations, Arizona Works attained performance standards 1, 3, 4, and 5. For meeting these standards, MAXIMUS, Inc. received $1,342,403 of a possible $2,440,732 in Administrative Payment Incentives. In the second year of program operations, Arizona Works attained performance standards 2, 3, 4, 5, and 6, and received $1,067,061 of a possible $1,778,435 in Administrative Payment Incentives. During the third program year (April 2001 through March 2002), Arizona Works attained performance 3, 4, 5, and 6, and received a total of $1,095,978 of a possible $2,435,507 in Administrative Payment Incentives. These calculated performance measures reflect more than simply the effects of privatization. They measure the combined effects of privatization, the Arizona Works performance incentives, differences in program rules, and differences in the rate at which the two programs recorded existing participation in activities or paid work. The performance measures that compare outcomes in the pilot and comparison areas also reflect the effects of pre-existing differences in the characteristics of caseloads and neighborhoods in the areas served by Arizona Works and EMPOWER Redesign. Because the performance measures reflect the effect of so many factors, they are unreliable measures of the relative performance of the public and the private welfare systems. xviii Executive Summary Abt Associates Inc. The Process Study The Organization and Staffing of Arizona Works and EMPOWER Redesign The Arizona Works Agency Procurement Board, a group of up to nine voluntary members appointed by the Governor, administered the pilot program from its beginning through October 2002. The Board members were responsible for key decisions concerning the Arizona Works budget, performance measures, and the second pilot site. The Board relied on the assistance of paid professional staff from FOX Systems, Inc., a consulting firm based in Scottsdale. Later, when the state legislature did not renew the contract with FOX Systems, professional staff from DES assisted the Board. There were no specific complaints voiced by MAXIMUS staff about the actions or impartiality of DES when it replaced FOX Systems, although there was some general concern over the removal of an independent mediator. MAXIMUS, FOX, and DES staff reported that, throughout the program, negotiations concerning performance measures and contractual issues consumed a significant amount of time, although an exact estimate of this time commitment is not available. The organization and staffing of the Arizona Works Program itself was developed by staff from MAXIMUS through a coherent planning process. This organizational plan was clearly capable of meeting the goals of the Arizona Works Pilot Program and fulfilling the terms of the Arizona Works contract. Key roles were well-defined. Procedures for monitoring and problem resolution were quickly put in place. Similar organizational structures were in place in each local office. MAXIMUS initially established an organizational structure for the central office that was headed by the Program Manager and Deputy Program Managers who reported to the Program Manager and who were responsible for Operations, Workforce Development, and Community Outreach. Reductions in caseloads, reductions in the state’s budget for administering TANF, and other factors led to a subsequent reorganization of administrative and supervisory positions. Some administrative staff were relocated to local offices in an attempt to move toward site-based management. At the beginning of the third year, the management position that previously had encompassed staff training and community outreach was divided, a change necessitated by greater demands on both positions. Another change during this period saw the consolidation of workforce and case management activities under a single manager. At the end of the third year, staff that reported directly to the Project Manager included the Deputy Manager for Administration; managers for Operations, Human Resources, and Community Outreach; and the Staff Developer. MAXIMUS also established work groups that brought together staff from several areas in an effort to monitor and improve program performance and to improve communication. In the local offices, a Case Manager Supervisor (CMS) oversaw all program operations, and a Lead Case Manager assisted the CMS. Midway through the program, the number of CMS staff was reduced but this change proved unworkable and the original number of CMS staff was restored. The Case Managers provided the strongest link between the participant and the program, and were initially responsible for intake, assessment, assignment to work-related activities, monitoring, and many support services. A number of Customer Service Representatives were stationed at the front desk in each office to register applications, answer the telephone, and perform other vital functions. The MAXAcademy program provided important work preparation for adult TANF recipients. In the first and second years of operations in the Arizona Works local offices, one Case Manager typically handled all intake and ongoing case activities in a way that was consistent with the “one Abt Associates Inc. Executive Summary xix worker concept” of the program. The Case Manager could assess the family’s entire range of service needs, employment skills, and barriers to employment, and could then place persons in activities and provide appropriate assistance. Arizona Works staff asserted that the practice of linking a participant with a single Case Manager was a major strength of the program. EMPOWER Redesign provided a similar array of services as well, however, so it is difficult to assess the relative effectiveness of the two programs from descriptive information alone. In the second and third years, Arizona Works imposed some changes in its local offices. In two offices during the second year, MAXIMUS separated intake and ongoing functions on a trial basis and, by September 2001, rolled out a modified version of this model to all offices. Intake workers determined eligibility, while ongoing case managers developed the Employment Plan and discussed barriers to work and support services. Arizona Works also instituted another form of specialization -intensive case management -- to provide services and support to hard-to-serve participants and those in cris is situations whose participation is typically deferred. Arizona Works management believed that these changes improved service delivery and were consistent with the single worker concept at each phase of service delivery. Nevertheless, the designation of intake and ongoing workers in the third year of Arizona Works meant that most TANF recipients were required to see three workers, including state staff for food stamp benefits and medical assistance. This structure was somewhat similar to the structure in EMPOWER Redesign offices, where one staff member is responsible for intake and assessment, and another is responsible for employment and training. During these years, the EMPOWER Redesign central and local offices also featured a coherent organization and staffing plan. There were fewer changes in the EMPOWER Redesign offices during these months. Some of the job-related Resource Centers in the EMPOWER Redesign office were dismantled because of budget cuts. In general, these two groups of offices had many staff members with similar roles, a finding that is not surprising because the two programs had many similar goals. The practical significance of these organizational changes and any differences between the Arizona Works and EMPOWER Redesign offices is unclear. Most welfare-to-work programs make ongoing organizational and staffing changes to incorporate new knowledge or to respond to budgetary challenges or staff turnover. It is not obvious that either office format was inherently better than the other, and it is not clear that local Arizona Works and EMPOWER Redesign offices would differ significantly in the perception of most clients. Intake and Benefit Determination The Arizona Works local office staff quickly established and maintained procedures for managing intake, eligibility, and benefit determination. These tasks, which were an important part of the Arizona Works contract and an important goal of the program, were implemented in all of the local offices. A careful review of the “flow” of participants in Arizona Works and EMPOWER Redesign offices suggests that the two groups of offices had many similar procedures. The two programs used the same application form and the same automated systems, and both required an initial interview. Both programs provided applicants with an orientation that emphasized a “work first” message and required participants to develop a plan for obtaining work. Most of these procedures remained intact during the first two program years, but some changes were instituted in the third program year. In the Arizona Works local offices, the intake and eligibility determination interview and the employment and training needs assessment interview were separated, with different staff members assigned to each function. Arizona Works dropped its group orientation xx Executive Summary Abt Associates Inc. sessions. The information in the orientation session was provided in the employment and training needs assessment interviews, which were one-on-one meetings between client and caseworker. EMPOWER Redesign also dropped its combined orientation session in which several program elements were discussed, and instead provided its “Jobs Welcome” sessions and other needed information to clients in one-on-one meetings. These changes are not unusual. Welfare-to-work programs often modify their intake, assessment, and orientation procedures over time as staff learn about which procedures seem more effective than others. The denial of the federal waivers for the processing of food stamps and Medicaid under Arizona Works clearly created difficulties for the ongoing determination of benefit levels. In the first year of Arizona Works, MAXIMUS and DES staff perceived that staff from the other group misplaced files that needed to be shared. In the second year of the program, MAXIMUS and DES agreed to maintain entirely separate files. Many but not all staff members believed that the new filing system enhanced cooperation and reduced the loss of files. As of summer 2002, staff in both agencies reported that they had made progress in reducing problems associated with interagency coordination. A second problem arising from the denial of the waivers was that, when MAXIMUS or DES staff changed benefit levels or eligibility for one program, benefits from other programs were frequently deauthorized and needed to be reauthorized to prevent a gap in the participant’s receipt of benefits. This problem continued to create additional work for MAXIMUS and DES staff. Staff members from both MAXIMUS and DES worked to address this issue, and the local offices responded to this issue in different ways. Some felt that the situation had improved, but others did not. These and other issues produced tension in the Arizona Works local offices, although staff members from both programs reported that members of the other program made sincere efforts to address these problems. Employment, Training, and Support Services MAXIMUS developed procedures for providing Arizona Works participants with a wide array of activities and support services. MAXIMUS clearly fulfilled its obligation to provide key programs and services as the Arizona Works vendor. These activities were accessible to participants in each of the local offices. Arizona Works and EMPOWER Redesign offered many similar sets of services. The programs shared the same set of core objectives and used similar funding sources for support services. Linking participants to jobs was a primary goal of both programs. Each program appeared to convey to participants the message that cash assistance is not a long-term entitlement and the expectation that they actively engage in activities that lead to employment. The two programs appeared to use many similar procedures for preparing clients for work, linking clients to employment, monitoring progress, providing post placement services, and imposing sanctions. Both Arizona Works and EMPOWER Redesign attempted to maximize the use of available support service dollars to achieve their program goals. Both programs offered an extensive array of similar support services. Budget constraints and other factors led both programs to change their employment, training, and support services to some degree. In the second program year, MAXIMUS established a relationship with the East Valley Institute of Technology (EVIT). MAXIMUS held MAXAcademy classes at EVIT, which provided facilities and childcare services. In October 2001, this relationship was ended and MAXIMUS staff resumed providing MAXAcademy sessions in their own offices. Over time, the MAXAcademy curriculum was modified somewhat, with a heightened emphasis on immediate employment. In the third program year, EMPOWER Redesign also modified its work preparation Abt Associates Inc. Executive Summary xxi program to some extent, although the program remained work-focused. Arizona Works staff expressed a greater willingness to apply sanctions, which were given less emphasis in the first program year, when the program was just beginning and overcoming some initial start-up challenges. An analysis of administrative data confirms that over three-quarters of adults subject to Arizona Works and over three-quarters of adults subject to EMPOWER Redesign in the comparison area either had paid work or were engaged in work-related activities during these three years. These unpaid activities included community service or work experience activities, GED or ESL, job search, or job readiness and training. These participation rates were substantial, especially when one considers that roughly 30 percent of adults were initially deferred from activities. The Second Pilot Site In June 2000, the Arizona Works Agency Procurement Board voted to select Mohave County as the second site for the Arizona Works Pilot Program. For several months following the Board’s decision, both MAXIMUS and DES initiated activities that would lead to the transfer of responsibilities for TANF from DES to MAXIMUS on January 1, 2001, the start-up date for the second pilot site stipulated by the original contract. During the remaining months of calendar year 2000, the Board, MAXIMUS and DES were unable to resolve contract issues related to the Maricopa County program and were unable to approve a contract amendment to authorize operation of the second program. This stalemate eventually halted preparations that were well underway in Mohave County, after both agencies had committed substantial resources to this effort. In November 2001, the Board reversed its earlier decision, citing several concerns, and voted to operate a privatized program in Greenlee County, a much smaller county. Arizona Works began in Greenlee in April 2002 – 15 months after the planned start-up date – and lasted only six months. Because of its short duration, the Greenlee Pilot provides an extremely limited test of privatization of TANF services in a rural setting. Conclusions from the Process Study Based on this evidence, MAXIMUS, Inc. met the requirements of the Arizona Works Pilot Program, but the program was less than a clear-cut test of privatization because MAXIMUS was never truly independent of DES. MAXIMUS remained dependent on the state’s automated systems. MAXIMUS hired such a large number of former DES employees that its operations must have been influenced by previous training received by these employees at DES. The denial of the waivers to operate food stamps and Medicaid meant that MAXIMUS and DES needed to integrate the operations of two organizations in one location. The perceptions of DES staff suggest more subtle ways in which DES and MAXIMUS staff share day to day work burdens: each organization may have benefited from the hours spent by staff from the other organization to register applications, process files, manage the lobby, and handle numerous other day-to-day tasks. Arizona Works and EMPOWER Redesign were nevertheless different in an institutional sense. Many key functions of Arizona Works are run by MAXIMUS, a for-profit company. As long as the findings are interpreted carefully, the pilot program still has the potential to provide an interesting test of the effects of a major effort to privatize TANF and many related services. We attempted to find ways in which privatization led to clear differences in the procedures for delivering public services., but these differences were not obvious. The two organizations had staff playing many similar roles, and they provided many similar programs and support services. xxii Executive Summary Abt Associates Inc. The Impact Study The "Pilot Group" of Adults and Families Studied for This Report This report covers events from the initial implementation of Arizona Works on April 1, 1999 through the end of March 2002. The “pilot group” of adults and their families studied in this report includes all adults and their families that were on TANF in the Arizona Works area or in the comparison area from April 1, 1999 through March 31, 2002. Over this period, 11,693 adults participated in TANF under Arizona Works and 31,253 participated in TANF under EMPOWER Redesign in the comparison area. The impact study also examines the effect of Arizona Works on three large subgroups of the pilot group that were analyzed to calculate the annualized performance measures of the first, second, and third program years of Arizona Works. The “first year pilot group,” which was analyzed to produce the annualized performance measures for the first program year, includes those subject to Arizona Works or EMPOWER Redesign in the comparison area from April 1, 1999, through March 31, 2000. The second year pilot group, which was analyzed to produce the annualized performance measures for the second program year, i cludes those subject to Arizona Works, or EMPOWER Redesign in n the comparison area, from April 1, 2000, through March 31, 2001. The third year pilot group includes those in the pilot area during the final program year. These families were observed through the end of March 2002. The observation period available for each family depends on when the family first received TANF since April 1, 1999. For those receiving TANF on April 1, 1999, the start date of Arizona Works, an observation period of 36 months is available. The available observation period is 25-36 months for the first year pilot group, 13-24 months for the second year pilot group, and 1-12 months for the third year pilot group. Departures from TANF were especially rapid for these adults, in part because of the strong economy and because of recent policy changes that probably influenced the behavior of low-income persons. Of the adults who received TANF on April 1, 2000, only about one-quarter still received TANF one year later. These programs had only a few months to influence the behavior of most recipients. The caseloads and neighborhoods served by the two programs had many characteristics in common, but some differences were also evident. The adult TANF recipients in the comparison area served by EMPOWER Redesign had spent relatively more previous months on TANF (or AFDC) on average, were relatively more likely to have larger families, and were relatively less likely to have received a high school diploma or GED. Clearly, the economy in both parts of Maricopa County was very strong during this period. Nevertheless, the area served by Arizona Works was somewhat more prosperous than the rest of Maricopa County. The transition from welfare to work and the values of the performance measures could have been affected by these differences in caseload and neighborhood characteristics, although the exact effects of these caseload and neighborhood characteristics are difficult to assess. Methods for Estimating Program Impacts The strategy used to estimate the impact on Arizona Works, relative to EMPOWER Redesign, employs widely-used techniques to take into account these differences in caseload and neighborhood characteristics. The impact study uses regression analysis to control for the measurable differences in these pre-existing characteristics of the two areas. Regression analysis alone may not be enough to Abt Associates Inc. Executive Summary xxiii produce reliable impact estimates because the available measures of background factors do not capture all pre-existing characteristics that could have led to different outcomes for the two groups. An additional technique used in the impact study involves the construction of a “pre-pilot group” of welfare clients who received TANF in the Arizona Works and comparison areas of Maricopa County at some time during the three years before the introduction of Arizona Works. Outcomes for the prepilot group were observed through March 31, 1999, the day before the start of Arizona Works. The available observation period for both the pilot group and the pre-pilot group varied from 1-36 months. The impact of Arizona Works was estimated by comparing regression-adjusted differences in outcomes achieved by the pilot group and the pre-pilot group. During the pre-pilot period, TANF clients in both areas were subject to EMPOWER or EMPOWER Redesign, so regression-adjusted differences in outcomes achieved by welfare clients in the two areas reflect unmeasured differences in client and neighborhood characteristics at that time, but not the effect of Arizona Works. During the pilot period, regression-adjusted differences in outcomes achieved by TANF clients in the two areas reflect the effect of unmeasured differences in client and neighborhood characteristics, plus the effects of Arizona Works. By subtracting differences in outcomes in the two areas in the pre-pilot period from differences in outcomes in the pilot period, one can isolate the impact of Arizona Works. Intuitively, if differences in outcomes in the Arizona Works and comparison areas are the same before and after Arizona Works, then factors other than Arizona Works probably caused those differences. If, on the other hand, differences in outcomes in the two areas change significantly after the start of Arizona Works, then Arizona Works could have caused those differences. The Impact of Arizona Works on the Percentage of Adults in Unpaid Work-Related Activities: The Entire Pilot Group 60 Percentage of adults 50 40 30 20 10 0 Pilot Group Pre-Pilot Group Impact (19 percentage points) Arizona Works Area Comparison Area The impact of Arizona Works is estimated as the difference in outcomes (Arizona Works area minus comparison area) for the pilot group minus the difference in outcomes (Arizona Works area minus comparison area) for the pre-pilot group. xxiv Executive Summary Abt Associates Inc. Impacts on Work-Related Activities Arizona Works increased the proportion of all adults in the pilot group who were placed in unpaid activities by 19 percentage points during the first three program years. These unpaid activities are designed to prepare people for work. This finding is based on information recorded by local office staff. According to these data, Arizona Works increased the proportion of adults who participated in unpaid community service or work experie nce positions, high-school level education (GED/ESL), job search assistance, and job readiness and training activities. Although these estimated impacts on specific activities should be viewed with caution because a specific activity could reasonably be classified in several ways, the overall rate of participation clearly increased under Arizona Works. Sanctions Sanctioning for noncompliance with work requirements was initially more frequent under EMPOWER Redesign but later became more frequent under Ariz ona Works. The percentage of adults in the first year pilot group that were sanctioned in the first program year was about 28 percent under EMPOWER Redesign but only 13 percent under Arizona Works. EMPOWER Redesign also sanctioned a relatively higher proportion of recipients in the second program year. By the third program year, however, sanction rates were higher under Arizona Works: 10 percent of Arizona Works clients but only 5 percent of EMPOWER Redesign clients were sanctioned. Arizona Works staff reported that they were more willing to utilize sanctions over time. EMPOWER Redesign staff reported that they reduced sanction rates in the third program year in response to the Olea v. Blessing case, which limited the state’s ability to sanction adults who were not receiving TANF. Employment and earnings Data sources: The preferred source of information on employment and earnings comes from employer-reported records from the state’s unemployment insurance (UI) wage records. These records cover the vast majority of jobs in the state and provide earnings of all pilot group members, even those that left public assistance. Accordingly, these data are widely used in evaluations of programs for low-income persons. These records provide only earnings per cale ndar quarter, however, and cannot be used to calculate the performance measures of full-time employment. The other source of information on employment and earnings is the records collected and reported by staff in the local welfare offices. This information provides employment placements along with hours worked, earnings, and eligibility for employer-provided health care. These data have been used to calculate the Arizona Works performance measures. These welfare office-reported data are less reliable than the employer-reported data because the welfare office-reported data do not record earnings received after adults leave TANF, and because the welfare office-reported data reflect any differences in the consistency with which the two programs record earnings of TANF recipients. Impacts: According to the more reliable employer-reported records, Arizona Works had no impact on the employment and average quarterly earnings. Arizona Works did not increase employment and average earnings among all pilot group members, and did not increase these outcomes among all adults in the first, second, or third year pilot groups. The average adult under both programs earned about $1,400 in the last quarter of the observation period. Because the employer-reported data provide the more complete source of information on earnings, an important conclusion of the impact study is that Arizona Works did not increase average levels of earned income and employment, relative to levels that would have occurred under EMPOWER Redesign. Abt Associates Inc. Executive Summary xxv The Impact of Arizona Works on Employment, as Reported by Employers: The Entire Pilot Group 100 Percentage of adults employed, last qtr 80 60 40 20 0 Pilot Group Pre-Pilot Group Impact (not statistically significant) Arizona Works Area Comparison Area The Impact of Arizona Works on Average Quarterly Earnings, as Reported by Employers: The Entire Pilot Group $1,450 Avg Quarterly Earnings, last qtr $950 $450 -$50 Pilot Group Pre-Pilot Group Impact (not statistically significant) Arizona Works Area Comparison Area According to the welfare office data, Arizona Works caused statistically significant increases in total employment and several types of employment placements. These data indicate that Arizona Works increased rates of placements in full-time employment among all adults in the first year pilot group, but not among all adults in the second and third year pilot groups. The welfare office data also suggest that Arizona Works increased full-time employment among the “initial groups” of adults on xxvi Executive Summary Abt Associates Inc. TANF at the start of each program year. The information from the local office staff also show that Arizona Works tended to increase part-time employment, full-time work with hourly wage rates of at least $8.15 and with employer-provided health insurance, earned income, hours worked, and weeks worked. These findings are similar to the performance measures that are based on the same data. Because the welfare office-reported data do not include post-TANF earnings and do not record some earnings of TANF recipients, findings based on these data are less reliable than the findings based on employer-reported records. The discrepancies in estimated impacts obtained from the two data sources exist mainly because the proportion of adults with employment recorded only employers, and not by welfare office staff, was larger among those subject to EMPOWER Redesign than among those subject to Arizona Works. A relatively larger proportion of adults subject to EMPOWER Redesign had employment that was reported only by employers even in quarters in which adults received TANF. The performance measures of Arizona Works gave local office staff a greater incentive to report earnings under Arizona Works, and the Arizona Works program rules (under which earnings are not subtracted from TANF grants) gave recipients a relatively greater incentive to conceal earnings under EMPOWER Redesign. A perhaps less likely explanation is that Arizona Works could have slightly lengthened TANF spells or slightly accelerated job placements among small subgroups of working adults (even though the program’s overall effects were minimal), and these small behavioral changes among some recipients could have increased the chance that welfare office staff recorded jobs. Estimated Impacts of Arizona Works on the Average Percentage of Months on TANF: The Entire Pilot Group 50 Avg Pct of Months on TANF 30 10 Pilot Group -10 Pre-Pilot Group Comparison Area Impact (1.9 percentage pts***) Arizona Works Area TANF Receipt The impact of Arizona Works on TANF receipt varied over time. During the program’s first two years, Arizona Works slightly increased the average number of months in which adults participated in Abt Associates Inc. Executive Summary xxvii TANF by about one week. During the third year of the program, Arizona Works reduced TANF receipt. Over this entire period, Arizona Works reduced the average period of TANF receipt by about one or two weeks. It is unclear whether privatization of TANF services alone led to the initial increase in TANF participation. Several aspects of Arizona Works -- the limited sanctioning in the earliest months of the program, the relatively larger maximum grants for small families, the absence of EMPOWER’s 24-month time limit, and the flat grants which did not fall as earnings rise – all could have made TANF receipt relatively more attractive during the program’s first two years. It is not possible to separate the effects of these factors, the performance measures and privatization. Similarly, more than one explanation could account for the decrease in TANF receipt under Arizona Works in the third year. Arizona Works could have eventually succeeded in reducing welfare dependence, even if it did not increase earnings. Other changes could also explain these changing impacts. During the last year of the program, fewer adults under EMPOWER Redesign received time limited benefits because the initial five-year time-limit enforcement period ended, and sanction policy became more lenient under EMPOWER Redesign and perhaps stricter under Arizona Works; these changes could have made TANF receipt relatively less likely under Arizona Works. The estimated impacts of Arizona Works on TANF receipt are consistent with the Arizona Works performance measure that rewards reduction in the “long-term caseload,” defined as more than 36 months of prior welfare receipt The performance measures showed that this decline was larger under EMPOWER Redesign in the first program, and about 7 percentage points larger under Arizona Works in the third program year. Even after the impact analysis corrected for the role of factors other than Arizona Works, similar findings were obtained. The contributions of program rules, sanctions, and privatization to these findings are not clear. Estimated Impacts of Arizona Works on Average Monthly TANF Benefits: The Entire Pilot Group $200 Avg Monthly TANF Benefit $100 $0 Pilot Group Pre-Pilot Group Impact ($22, Arizona Works Area Comparison Area statistically significant) xxviii Executive Summary Abt Associates Inc. TANF Benefits Arizona Works increased average monthly TANF benefit levels by an average of $22 per month over the entire pilot period. This finding is based on all families in the pilot group, regardless of whether they continued to participate in TANF. Arizona Works increased average TANF benefits among all families in the first, second, and third year pilot groups. The small, initial increases in TANF participation under Arizona Works rates do not explain most of the increase in average benefits. Differences in program rules probably account for most if not all of the impact on average benefits. A portion of earned income is subtracted from TANF benefits under EMPOWER Redesign but not under Arizona Works. Only EMPOWER Redesign featured the 24-month time limit on the adult portion of the grant. The proportion of the caseload consisting of two- and three-person families -for whom maximum grants were relatively higher under Arizona Works -- exceeded the proportion of the caseload consisting of very large families -- for whom maximum grants were relatively higher under EMPOWER Redesign. Limitations on sanctions in the earliest months of Arizona Works could also have increased TANF benefits. Food Stamps and Medical Assistance Arizona Works had some effects on participation in other public assistance programs. The program increased the average number of months in which adults received food stamps during the first two program years, by about one week, but had no impact on food stamp receipt during all three program years. Arizona Works increased the average number of months in which adults were enrolled in public medical assistance programs, including Medicaid, by about two weeks. The program rules or privatization could have led to these modest impacts on receipt of public assistance. Arizona Works also reduced average monthly food stamp benefits -- estimated over all months, for entire Pilot group -- by $9 per month. The reduction in food stamp benefits could have been caused by the increase in average TANF benefits, which would translate into a partial reduction in food stamp benefits because TANF benefits are counted as income for the purpose of determining levels of food stamp benefits. Total Income Arizona Works did not increase average quarterly total income by statistically significant amounts among all families in the pilot group in the first quarter of 2002, the last quarter observed. Total income is defined as the sum of employer-reported earnings and TANF and food stamp benefits. In this calendar quarter, the average total family income was roughly $2,000. By this measure, families were neither worse off nor better off under Arizona Works, relative to the state program. Conclusions The impact estimates and the performance measures: The performance measures and the impact study reach somewhat different conclusions. The impact study finds that Arizona Works had little impact on total employer-reported employment and earnings but the performance measures suggest that the program increased some but not all measures of full-time employment. Unlike the performance measures, which are based on welfare office records, the preferred impact estimates use the more comprehensive employer-reported earnings and consider the role of caseload and neighborhood characteristics by examining outcomes in the pilot and comparison areas before Arizona Works. Abt Associates Inc. Executive Summary xxix A main shortcoming of the performance measures of employment appears to be that they miss some earnings that were received during and after TANF spells. The impact estimates based on the employer-reported records and the performance measures cannot be compared directly because the employers do not record full-time status. The conflicting estimates on total employment obtained from the two data sources and the weaknesses of the welfare office data nevertheless cast doubt on the performance measures as reliable indicators of the impact of Arizona Works. These findings underscore the potential limitations of commonly used performance measures as indicators of the success of welfare-to-work programs. The possible role of each component of Arizona Works: These impact estimates measure the combined impacts of privatization, the performance incentive payments, and differences in program rules. MAXIMUS, Inc. had no choice but to accept the performance incentives and program rules because they were stipulated by the original Arizona Works legislation, in part to attempt to simulate practices common among employers in the private sector. Although many consider these elements of Arizona Works as a “package,” it is of interest to assess their separate impacts. It is, however, difficult to assess the separate effects of each part of Arizona Works because we can only observe their combined impact. The differences in the program rules and sanction policies could have led to differences in several outcomes, even without privatization. These differences in rules and sanctioning clearly tended to make average TANF grants relatively larger under Arizona Works, but the size and direction of the effect of these program rules on most outcomes is unknown. The rules and sanction policies initially made TANF receipt more attractive under Arizona Works. The rules and approaches to sanctions could have produced relatively higher employment rates and earnings under either Arizona Works or EMPOWER Redesign. In the third program year, many time-limited families under EMPOWER Redesign began to receive full benefits again, EMPOWER Redesign staff grew less willing to impose and sanctions (because of the Olea case) and Arizona Works staff became more willing to impose sanctions. These changes by themselves tended to make TANF receipt relatively less attractive under Arizona Works. After these changes, the combined effects of the program rules and sanctions on TANF receipt and earnings under the two programs were unclear. The privatization of TANF-related services alone may or may not have increased employment and earnings. It is possible that differences in program rules alone resulted in relatively lower levels of employment and earnings under Arizona Works, and that the privatization of TANF services with performance incentives caused an offsetting increase in employment and earnings under Arizona Works. It is also possible that differences in program rules caused relatively higher levels of employment and earnings under Arizona Works, and that privatization of TANF services and performance incentives led to an offsetting decline in employment and earnings under Arizona Works. A third possibility is that none of these program components changed employment and earnings. All one can say is that, if privatization of TANF services alone led to increases in earnings, then this effect was not large enough to be seen in the first three years of Arizona Works. Possible reasons for observed impacts: Two large groups of TANF recipients will presumably not be affected by any attempt to improve the operations of welfare-to-work programs. One group consists of TANF recipients who are already able to work but who are experiencing temporary crises and who need only a short period of TANF receipt to address their problems and attain self sufficiency. These adults can help themselves and need not benefit from welfare-to-work programs. Even before the recent imposition of these welfare-to-work programs, many welfare recipients attained financial independence without any assistance from the welfare office. Another group of xxx Executive Summary Abt Associates Inc. welfare recipients, the “hard to serve,” have problems so severe that even the best programs will have little impact. Institutional changes such as privatization could possibly help the remaining fraction of families that have some problems but that could become self-sufficient if given appropriate services. Even if one makes the unverified assumption that privatization increases earnings for this remaining group, this impact could be hard to detect in a full sample of TANF recipients. Arizona Works obvious ly did not change the character of unskilled labor markets, which offer mostly low-wage jobs with few benefits. The characteristics of these jobs and the cost of child care both limit the prospective returns to work. Although Arizona Works placed many adults in jobs with higher wages and some benefits, and provided some child care assistance, enough good jobs and low cost child care centers may not have been available for all TANF families in Maricopa County. As the process study found, Arizona Works was only partially privatized and relied on the state for data processing, administration of food stamps and Medicaid, and other activities. A more completely privatized system may or may not have had different impacts. Many services and procedures of the public and private programs were very similar. Even if the two programs featured some different approaches, these differences may not have been large enough to produce differences in the earnings of TANF recipients. The performance measures could have increased the perceived incentive of Arizona Works local office staff to help TANF recipients find work, but even this conclusion is uncertain. The employment information recorded by local office staff was not necessarily informative as to the ultimate post-welfare status of these adults. TANF recipients, especially those about to leave the program, often fail to communicate their circumstances to local office staff. Accordingly, local office staff of both programs had, at best, a vague sense of whether and how their actions were helping TANF recipients find and keep work. Performance incentives may be expected to have weak effects on caseworkers who had such a limited day-to-day sense of their effectiveness. Possible “competitive effects” of Arizona Works: In theory, it is possible that the Arizona Works Pilot Program improved the performance of EMPOWER Redesign as a result of the competition between the public and private programs. This potential effect can be called the “competitive effect” of Arizona Works, to distinguish it from the impact estimates, which measure only the differential effectiveness of the two programs. Many efforts to privatize the provision of public services are set up to produce these competitive effects. It is hoped that private and public service providers – or multiple private service providers – will compete with one another by improving the quality of services or by cutting the cost of services. It is not easy to measure this competitive effect because there is no comparison group of TANF recipients that is subject to EMPOWER Redesign in a city similar to Phoenix, but without the perceived competition with Arizona Works. The available information does not, however, provide strong evidence for a competitive effect of Arizona Works. In the comparison area served by EMPOWER Redesign, earnings and placements in work-related activities did increase from the pre-pilot period to the pilot period. Although some of these trends could have been caused by a competitive effect of Arizona Works, many other factors – the strong economy, the increasing work participation requirements of TANF, and ongoing attempts to improve the state’s TANF program – could also have caused much or all of these trends. Abt Associates Inc. Executive Summary xxxi Average Annual Employer-Reported Earnings of Adults Who Received TANF in the Comparison Area $5,000 $4,000 Avg Annual Income $3,000 $2,000 $1,000 $0 April 1998-March 1999 (pre-pilot period) April 1999-March 2000 (first year pilot group) April 2000-March 2001 (second year pilot group) April 2001-March 2002 (third year pilot group) Program Year Average Annual Earnings of Adults Who Received TANF in the United States $7,000 $6,000 Avg Annual Income (CPS) $5,000 $4,000 $3,000 $2,000 $1,000 $0 1998 1999 2000 2001 Calendar Year xxxii Executive Summary Abt Associates Inc. Nationally representative surveys, such as the Current Population Survey (CPS) show that earnings and employment of current and former TANF recipients have increased across all states -- the vast majority of which had a strong economy and a publicly run TANF program without a competing private program such as Arizona Works. As the accompanying figures show 1 , earnings trends among welfare recipients in the comparison area (based on employer-reported records) and earnings trends among welfare recipients in the United States (based on the CPS) look very similar. National TANF work participation data also show that work participation rates increased after 1997. The similarity of trends in outcomes in the comparison area and across the nation suggests that factors other than the competition between public and private sectors could explain much if not all of these trends. In sum, there is no strong evidence that privatized TANF has helped or harmed the average TANF recipient. 1 The annual earnings in the comparison area were calculated by estimating the average quarterly earnings of adults that received TANF in each program year (or in the year just before Arizona Works) and multiplying by 4. The annual earnings figures for welfare recipients in the U.S. were calculated using the March CPS surveys. The survey reports annual income from wages and salaries in the previous year and also receipt of public assistance in the previous year. To obtain these average, the responses were topcoded at $50,000. Executive Summary xxxiii Abt Associates Inc.