Docstoc

Retirement Calculator Done in Visual Basic Tutorials

Document Sample
Retirement Calculator Done in Visual Basic Tutorials Powered By Docstoc
					                Social Security
                Administration
                $1 for $2 Benefit
                Offset Demonstration


                Quantitative Evaluation
                Approaches & Methods

Cambridge, MA
Lexington, MA   Contract #SS00-04-60110
Hadley, MA
Bethesda, MD
Chicago, IL

                DRAFT

                August 11, 2006


                Prepared for
                Mark Green
                Social Security Administration
                3520 Annex
                6401 Security Blvd.
                Baltimore, MD 21235


                Prepared by
                David A. Long
                Larry Orr
                David C. Stapleton
                Judith D. Feins
                Howard Rolston


                Abt Associates Inc.
                55 Wheeler Street
                Cambridge, MA 02138
Table of Contents

Chapter 1: Introduction ................................................................................................................... 1
     1.1 Background and Objectives of the Demonstration ............................................................ 1
     1.2 Demonstration Design ....................................................................................................... 2
           Eligibility for the Demonstration ....................................................................................... 2
           Treatments to be Tested in the Demonstration .................................................................. 3
     1.3 Evaluation of the Demonstration ....................................................................................... 4
     1.4 Overview of this Report .................................................................................................... 5

Chapter 2: Sample Design ................................................................................................................ 7
     2.1 Demonstration Site Selection ............................................................................................ 7
           Site Selection Plan ............................................................................................................. 7
           Site Selection Issues .......................................................................................................... 8
     2.2 Selection of Sample Members ......................................................................................... 12
           Selection and Recruitment of Current SSDI-Only Beneficiaries .................................... 12
           Selection and Recruitment of SSDI Applicants............................................................... 16
           Outreach and Engagement ............................................................................................... 18

Chapter 3: Analytic Methods ......................................................................................................... 20
     3.1 Framework for the Analysis ............................................................................................ 20
           Estimates of Interest ........................................................................................................ 20
           Samples and Subgroups ................................................................................................... 21
     3.2 Impact Analysis ............................................................................................................... 21
           Estimation of Impacts, Sample Sizes, and Precision of Estimates – Existing
           Beneficiaries .................................................................................................................... 22
           Estimation of Impacts, Sample Sizes, and Precision of Estimates – Applicants ............. 27
           Summary of Sample Sizes and Precision ........................................................................ 30
           Estimation Methods ......................................................................................................... 32
           Generalization of the Impact Estimates ........................................................................... 33
     3.3 Participation Analysis ...................................................................................................... 33
           Treatment Group Outcomes ............................................................................................ 35
           Treatment-Control Differences........................................................................................ 36
           Connecting the Impact and Process Analyses ................................................................. 37
     3.4 Benefit-Cost Analysis ...................................................................................................... 38
           Estimation of Benefits ..................................................................................................... 40
           Estimation of Costs.......................................................................................................... 41
           Analysis of Benefits and Costs ........................................................................................ 41
     3.5 Forecasting Long-term Results and Costs ....................................................................... 42

Chapter 4:         Data Sources for the Quantitative Analyses .............................................................. 45
     4.1          Uses of Administrative Data............................................................................................ 45
     4.2          Uses of Survey Data ........................................................................................................ 46
     4.3          Uses of Demonstration Site Operations Data .................................................................. 46
     4.4          Uses of Demonstration Site Characteristics Data ............................................................ 47



Abt Associates Inc.                                                                              Table of Contents                              i
Chapter 5: Presentation of Analytic Results................................................................................. 61
     5.1 Results of the Impact Analysis ........................................................................................ 61
           Reporting the Impact Analysis Results ........................................................................... 61
           Presenting the Impact Analysis Results........................................................................... 63
     5.2 Results of the Benefit-Cost Analysis ............................................................................... 68

Chapter 6: Timeline for the Treatments and the Evaluation...................................................... 69
     6.1 Expected Time Path of the Treatments............................................................................ 69
           Implementation Timeline ................................................................................................ 70
           BOND Treatment Timeline ............................................................................................. 73
     6.2 Expected Time Path of the Impacts ................................................................................. 76
     6.3 Expected Time Path of the Evaluation ............................................................................ 77
           Key Elements of Timeline ............................................................................................... 77
           Overview of Timeline for the First-Stage Random Assignment Experiment ................. 79
           Overview of Timeline for the Second-Stage Random Assignment Experiment ............. 83
           Overview of Timeline for EI Experiment ........................................................................ 83
           Primary Areas in Which SSA Actions/Decisions Can Enhance the Timeliness of Results
           ......................................................................................................................................... 83

References ........................................................................................................................................... 85




Abt Associates Inc.                                                                                   Table of Contents                                ii
Chapter 1:               Introduction

This report describes the evaluation approaches and methods we will apply to the quantitative
analysis of the Benefit Offset National Demonstration. It lays out the plan for the impact evaluation,
detailing the evaluation approaches, statistical methods, and models that will be employed in
analyzing the impact of the demonstration treatments.

This deliverable is submitted under Task 4.1.2 of the demonstration design contract Scope of Work,
which requires that we:

           Develop and/or specify all appropriate statistical or econometric models for use in the
            evaluation and to forecast long-term project results and costs; and
           Prepare a written report describing all analytic approaches, statistical methods and model
            specifications that will be utilized in the evaluation of the demonstration project.


In addition, this draft is intended to complete Tasks 4.1.3 and 4.1.4, which require the report to:

           Detail how data from the various sources (SSA administrative records, project office
            data, survey data and process activities) will be structured in support of the analysis;
           Specify table shells, graphs and charts that will convey project results to policymakers;
            and
           Develop an evaluation timeline that identifies important milestones and when they will
            occur.

We have combined the separate reports under these tasks into a single draft for SSA’s review, in
order to provide a clear, comprehensive picture of the evaluation design and analysis.1



1.1     Background and Objectives of the Demonstration

Abt Associates and its subcontractors and consultants are working closely with the SSA in designing
this demonstration project for the SSDI program. Congress mandated the $1 for $2 benefit offset
demonstration, as part of the Ticket to Work and Work Incentives Improvement Act of 1999, in an
effort to encourage SSDI beneficiaries to work more and increasingly support themselves. Originally,
the demonstration project was focused on existing SSDI beneficiaries. However, as the project
progressed, the target population was expanded to include an early intervention component for
applicants who meet criteria that indicate they are likely to qualify for SSDI benefits.



1
    We have not included the measurement of induced demand in this document and suggest that it be
    addressed separately after further discussions with SSA.




Abt Associates Inc.                                              Chapter 1: Introduction                 1
The benefit offset national demonstration will rigorously test several versions of a new program for
SSDI beneficiaries and provisionally eligible applicants against status quo policies and against each
other. Currently, beneficiaries lose their entire cash benefit if their earnings exceed the Substantial
Gainful Activity (SGA) threshold for a sustained period of time. The offset component of the
demonstration will allow earnings above this level, reducing benefits by $1 or less for each additional
$2 earned—and thereby eliminating the ―cliff‖ currently in effect. The employment supports
component will provide guidance and resources to assist beneficiaries in obtaining and maintaining
employment that enables them to take advantage of the offset. Finally, the demonstration’s health
benefits package will provide health coverage to beneficiaries from the time they join the
demonstration. Currently, beneficiaries purchase coverage (from a private company or from a state
Medicaid buy-in program if they are eligible), or go uncovered, until they become eligible for
Medicare nearly two and a half years after qualifying for the SSDI program.



1.2     Demonstration Design

The treatment groups in the demonstration will receive different versions of the same core benefit.
This core benefit has the following features:

           A monthly benefit structure, similar in concept to an Earned Income Tax Credit, which
            helps a SSDI beneficiary reach a target income level. The beneficiary who volunteers to
            join the demonstration and is randomly assigned to a treatment group shifts from the
            regular SSDI benefit schedule to a new schedule designed for workers. The benefit offset
            is built into this new structure.
           An independence account, a subsidized savings account that (a) increases the
            beneficiary’s incentive to work, (b) provides funding for employment supports, (c)
            includes savings incentives, (d) helps the participant maintain income during periods
            when earnings decline, thereby avoiding a return to the current program, and (e)
            ultimately can increase retirement income.
           A variety of employment supports, which will be partially funded through up-front
            payments and partially funded with deferred incentive payments.
           A special health benefit package, which the beneficiary begins receiving when he or she
            enrolls in the program and shifts to the new benefit structure (whether or not the
            beneficiary is employed).
One treatment group will receive only the monthly benefit—namely, the $1 for $2 benefit that the
Congress required to be tested—while the other groups will receive all components of the core benefit
(with variations).

Eligibility for the Demonstration

Both current SSDI-only beneficiaries and applicants who are likely to be found eligible for SSDI
benefits will be eligible to participate in the benefit offset demonstration.




Abt Associates Inc.                                            Chapter 1: Introduction                2
Current Beneficiaries
Almost all adult SSDI-only beneficiaries living in the demonstration sites, including new
beneficiaries and disabled adult children, will be eligible for recruitment. The only exceptions are the
following three groups:

        1. Beneficiaries who have previously assigned their Ticket, unless the ticket has been de-
           assigned and no payments have been made or are due;
        2. Beneficiaries receiving private disability insurance (PDI) or Workers Compensation
           (WC) indemnity payments; and
        3. Beneficiaries who will reach the normal retirement age by the end of the demonstration.


We will use SSA administrative data to exclude those who have used their tickets, those who are
receiving WC payments, and those who will age out of SSDI during the demonstration period. Those
receiving PDI benefits will have to be identified and screened out during the intake process.

Applicants
Applicants include all individuals who have applied for SSDI benefits but have not yet completed the
eligibility determination process. In the demonstration, a provisional eligibility determination will
allow some of these individuals to begin receiving SSDI cash benefits—as well as offset program
benefits—before their SSDI application is approved.

SSDI-only applicants will be eligible to participate if they meet the inclusion criteria for beneficiaries
(see above) and it is determined that they are likely to receive SSDI benefits because they would
satisfy SSA’s medical eligibility criteria if they did not have earnings above SGA (i.e., because they
would ―meet or equal the listings‖2). We use the term ―provisional eligibility‖ to describe the status
of applicants who meet demonstration criteria for ―likely to meet or equal the listings.‖ The
demonstration will need to establish an ad hoc screening process to determine provisional eligibility,
as well as a process to verify the original screen.

Although we have integrated the treatment of applicants and beneficiaries to a considerable degree,
maintenance of some differential treatment of applicants and existing beneficiaries is necessary, due
to the possibility that the generous provisions of the demonstration would induce many individuals to
apply for SSDI who otherwise would not have done so. This danger is reduced by two means: by
limiting eligibility to those applicants most likely to meet SSA’s medical eligibility criteria if they
were not working; and by providing a less generous treatment until medical eligibility for SSDI has
been established.

Treatments to be Tested in the Demonstration

We have recommended that five versions of the proposed offset program be implemented for SSDI
beneficiaries in ten sites around the country. These five versions of the program will differ in terms

2
    The listings are the medical regulations known as the Listing of Impairments. Applicants whose
    impairments ―meet or equal‖ the listings are automatically considered disabled and eligible for SSDI. (See
    SSA 2003, p. 4).



Abt Associates Inc.                                                Chapter 1: Introduction                   3
of selected aspects of the financial benefits and employment supports. All other aspects of the
program, including the health insurance package, will be the same. Three combinations of these
program components also will be implemented for SSDI applicants. The effectiveness of all of these
program packages will be evaluated in separate random-assignment tests conducted for beneficiaries
and for applicants.

The demonstration will have two beneficiary ―current law‖ control groups: a group representing all
beneficiaries and a group representing volunteers for employment supports. Similarly, the
demonstration will have two randomly assigned treatment groups that receive a ―pure‖ 50 percent
offset: a group representing all beneficiaries and a group representing those who volunteer for
employment supports. Together, these two treatment groups provide the test of the $1 for $2 offset
that Congress required to be evaluated in the Ticket to Work and Work Improvement Act of 1999.
The latter group is necessary to determine if adding employment supports produces larger effects than
the more generous treatment of earnings alone.

Beneficiaries assigned to the 50 percent offset only will experience a change in benefit schedule but
no other change in their SSDI benefits.3 For these beneficiaries, it will be important to provide
explanatory materials showing that—no matter how much they earn—their incomes under the new
schedule will be at least as high as under the current benefit. They will also learn that the new
schedule offers substantial opportunities to increase income by increasing earnings, opportunities they
do not have under the current benefit. (It will also be important to notify potential volunteers that
some other non-SSA benefits may be reduced as the individual’s income increases.) The other
treatment groups will receive more generous monthly benefits, independence accounts, and
employment supports as well as the health benefit package. The exact features of the core benefit to
be provided to the different treatment groups in the offset demonstration will be determined shortly.4



1.3     Evaluation of the Demonstration

The demonstration’s evaluation has four components. The impact analysis will address the question:
What is the effect of different versions of the benefit offset—alone or in combination with various
employment supports and health insurance coverage—on employment, disability benefits, health
status, and other outcomes? An experimental research design, with eligible, consenting individuals
randomly assigned to treatment and control groups, will be used to measure these effects in the

3
    In particular, they will remain eligible for TTW, with provisions to prevent financial harm to providers, due
    to the beneficiary’s offset schedule.

4
    As defined on p. 2, the core benefit includes the ES and health insurance that were presented initially in the
    ―Comprehensive Benefit Design‖ and ―Employment Supports Design‖ documents submitted to SSA in
    December 2005 (see Stapleton et al. and Long, et al., respectively). Further details have been worked out
    in the period since those reports were submitted. Products delivered to SSA in January and March 2006
    provided responses to specific SSA concerns and revised design features. See the Design Discussions
    memorandum of January 27, 2006, the PowerPoint presentation on the benefit design of March 9, 2006,
    and the Design Issues memo of March 24, 2006. The materials in the present draft report reflect the
    cumulative refinements in the design to date. Final parameters and design decisions should occur in
    August 2006.



Abt Associates Inc.                                                   Chapter 1: Introduction                    4
demonstration’s impact study. The random selection of sites, combined with random assignment
within sites, will ensure that the estimated impacts of these offset/employment supports/ health
coverage packages will be applicable to the entire nation.

Another component of the evaluation, the participation analysis, will examine the proportion of
applicants and beneficiaries who take up the offer of the benefit offset and/or employment supports,
the timing and intensity of their participation, the characteristics of participants and non-participants,
and the reasons for participation and nonparticipation.

The process study will examine various aspects of demonstration implementation and operations to
assist SSA in understanding and interpreting project results and identifying ways that the
interventions might be modified to improve quantitative outcomes. The study will document
implementation and operations, identifying important differences across sites and over time, to define
clearly the interventions that were tested and to draw lessons to guide future replication of the
demonstration in a national program. It also will provide qualitative information that will be helpful
in interpreting the results of the impact analysis. The process analysis will allow us to describe the
program experience of the beneficiaries and applicants who participate in the demonstration and also
understand the program treatment that is generating the impacts we measure in the evaluation.

Finally, the project’s cost analysis will estimate the net budgetary effects of the offset, employment
supports, and health insurance provided in each treatment package. These budgetary effects will be
separately estimated for SSA, the federal government, and state and local governments. In addition,
the cost analysis will examine the net benefits to participants and society. The analysis will draw on
the impact and process analyses in estimating the net budgetary effects of the program treatments for
the various levels of government.

Data will be collected consistently for all individuals—both treatment and control group members—
regardless of whether they are applicants or beneficiaries. Data will be gathered from multiple
sources, including participant surveys at baseline and 12 and 36 months for samples of those who
volunteer to participate in the demonstration; administrative systems; demonstration program
operations systems; published sources; health insurance records; and interviews with program staff
and employers.

In order to ensure that the treatments evaluated in the demonstration well represent the underlying
models that SSA wishes to test and that there is adequate follow-up, the entire project will require 8
years and 4 months to complete. Although this is a long time, much usable, reliable information will
be reported to SSA throughout the project in the form of brief, informal reports on findings, interim
reports and final reports on elements of the project which can be completed earlier.



1.4     Overview of this Report

The remainder of this report is organized as follows. Chapter 2 presents the sample design. The
chapter first describes demonstration site selection and then explains the selection of sample members
from among SSDI beneficiaries and applicants.




Abt Associates Inc.                                              Chapter 1: Introduction                     5
Chapter 3 begins by describing the framework for the quantitative analysis of benefit offset
demonstration impacts. Then it discusses the impact analysis (including methods for estimating
experimental impacts, required sample sizes, and precision of the estimates), the cost-effectiveness
analysis (including estimation of direct and indirect costs to SSA, other agencies and levels of
government, participants and society), and models for forecasting long-term project results and costs.

In Chapter 4, we address how the different analytic approaches described in Chapter 3 will be applied
to the various data sources planned for the demonstration. These data sources include a variety of
administrative records from SSA, CMS, and other agencies; baseline and follow-up surveys; and
offset demonstration program data.

Chapter 5 discusses how we plan to present the results from the quantitative analyses. The discussion
includes consideration of the types of tables, graphs, and charts we think will be useful for presenting
the findings to policymakers.

The last chapter of this report describes the timeline for the demonstration implementation and
treatments and how we expect this to affect the timeline for completing the quantitative aspects of the
evaluation.




Abt Associates Inc.                                             Chapter 1: Introduction                6
Chapter 2:                Sample Design

This chapter describes two aspects of the sample design for the benefit offset national demonstration:
the selection of demonstration sites and the selection of sample members.5



2.1     Demonstration Site Selection

In this section, we first lay out our site selection plan, and then discuss several issues that arise in
selecting sites for the offset demonstration.

Site Selection Plan

We have defined sites for this demonstration as SSA Area Offices. The advantages of using area
offices as sites are:

        1. Each eligible beneficiary and applicant is assigned to a specific area office; and
        2. The area offices are large enough to provide the required number of current beneficiaries
           and applicants.


The national list of area offices (excluding the areas noted above) constitutes the sampling frame for
the selection of sites.

From this frame, we will select a stratified random sample of ten sites. This number of sites is
required to capture adequately the variation in outcomes across area offices, as well as to provide the
required demonstration sample size within a reasonable intake period. In this section, we describe the
proposed procedure for the selection of the sample (final site selection will not occur until we are
ready to implement the demonstration).

The basic design for selection is as follows. We first exclude those area offices that are involved in
the four state pilot offset demonstrations, to avoid contamination of either demonstration by the other.
We also exclude the area that includes Puerto Rico. We then stratify the remaining population of area
offices by Census region. The sample of ten demonstration sites will be allocated to each Census
region in proportion to the total number of beneficiaries in the region.

In addition to geographic dispersion, as we argued in our December report, it is important that the
demonstration sample be representative of the proportion of beneficiaries participating in Medicaid

5
    These topics have been addressed in previous reports, most recently the report on Outcomes Data
    Requirements, and Sample and Survey Design (Orr et al., October 2005) and the Benefit Design and
    Parameter Recommendations report (Stapleton et al., December 2005). In addition, this chapter draws on
    the Design Issues memorandum of March 24, 2006 and the memo on ―Updated Estimates of Precision and
    Sample Sizes‖ (revised) of July 17, 2006.




Abt Associates Inc.                                             Chapter 2: Sample Design                     7
buy-in programs. Availability of the Medicaid buy-in to controls may substantially offset an
important element of the demonstration treatment, the health insurance benefit to be provided as part
of the demonstration benefit package. Therefore, it is important that the distribution of the
demonstration sample reflect the distribution of the national SSDI population with respect to the buy-
in. Moreover, because many area offices are concentrated within a single state, the proportion of
beneficiaries in each office who live in Medicaid buy-in states tends to be either very high or very
low. In a demonstration with a small number of sites, this can lead to serious non-representativeness
in this dimension.

In each Census region, we have divided the potential sites into ―high‖ and ―low‖ groups on this
variable6 and will select equal numbers of sites from each of those two strata. The resulting
allocation of the demonstration sites is shown in the last column of Exhibit 2-1. For each selection
stratum, the exhibit shows the number of area offices in each region, the estimated number of
beneficiaries in 2004, and the number of demonstration sites to be selected.

As can be seen from the exhibit, stratification on Census region and proportion of beneficiaries living
in Medicaid buy-in states exhausts the potential for stratification in a sample of only 10 sites—each
stratum contains only one or two sites. We have chosen these stratifying variables as those most
likely to capture important site-level variation in factors likely to affect the impacts of the
demonstration and, therefore, dimensions on which it is important to ensure that the sites are
representative of the nation.7

Site Selection Issues

Two site selection issues that have the potential to challenge the external validity of results from the
benefit offset demonstration. The first concerns the competition among SSA demonstrations for sites
across the country. The second concerns the difficulty posed for this demonstration by Area Office
jurisdictions that cross state boundaries.

Selecting Sites for Four Simultaneous SSA Demonstrations
Developments regarding the other SSA Office of Disability Insurance and Income Security
demonstrations appear to pose substantial problems for obtaining a nationally representative sample
of sites for the benefit offset demonstration. In the December design report, we removed from
consideration four area offices affected by the pilot offset demonstrations in Vermont, Connecticut,
Wisconsin, and Utah. This left 46 area offices, as shown in Exhibit 2-1.




6
    As noted in the December report, this variable is simply the proportion of beneficiaries who live in states
    with Medicaid buy-in programs, not the proportion that participate in such programs. The latter data are
    unavailable. We believe, however, that the proportion living in states with a buy-in program is a
    reasonable proxy for the proportion who participate in the program.
7
    In the December report, working with 46 area offices, we produced an illustrative sample of 10 sites for
    this demonstration.




Abt Associates Inc.                                               Chapter 2: Sample Design                        8
                                       Exhibit 2-1
     Allocation of the Sample of Area Offices by Census Region and Proportion of
                     Beneficiaries Living in Medicaid Buy-in States
                          Proportion of                Number of        Estimated           Number of
                      Beneficiaries living in          SSA Area         Number of         Demonstration
 Census Region        Medicaid Buy-in States            Offices        Beneficiaries          Sites
                               Low                         5              425,150               1
 Northeast
                               High                        6              549,860               1
                               Low                         3              308,370               1
 Midwest
                               High                        7              637,540               1
                               Low                         5              741,560               2
 South
                               High                       11            1,153,560               2
                                Low                        1              163,970               1
 West
                                High                       8              622,190               1
 Total                                                    46            4,602,200              10
Source: Ticket Research File, 10% sample, 2004 data.

SSA has recently set in motion three other major demonstrations involving SSDI beneficiaries, two of
which are also mandated to generate nationally representative results. Here is the current status of
these other projects:

            Mental Health Treatment Study—This demonstration is directed at SSDI beneficiaries
             with severe mental illness (schizophrenia or affective disorders) and is meant to provide
             health benefits administration and supports in a set of 21 nationally representative sites
             across the country. We understand a set of sites has already been selected for this study.
            Accelerated Benefits Demonstration—This demonstration will provide an immediate
             medical benefits package and employment supports to certain beneficiaries upon
             approval of their Social Security Disability Insurance (SSDI) claims. The target
             population will be new SSDI beneficiary awardees who have impairments most likely to
             improve and who, with medical and employment supports, would most likely return to
             work. This project, like the offset demonstration, must be conducted in a manner that will
             allow SSA to evaluate the appropriateness of implementing such a program on a national
             scale. Site selection will occur fairly early in this study, which has already begun.
            California HIV/AI Demonstration—This demonstration is designed to provide
             immediate access to healthcare services and treatment options required for a successful
             return to work, to SSDI beneficiaries with HIV or autoimmune disease. The treatment
             combines a health plan with employment support service coordination. It will be
             conducted in four large, urban counties in California (Alameda, Los Angeles, San Diego,
             and San Francisco), and arrangements for waivers and administrative coordination have
             already been made with these counties. The demonstration thus overlaps the design of the
             national benefit offset demonstration in at least two dimensions: the target population
             and the treatment features.




Abt Associates Inc.                                            Chapter 2: Sample Design                   9
These other demonstrations pose issues for the national benefit offset demonstration insofar as the
projects must share sites. First, site overlap will generate confusion for beneficiaries and/or
applicants as well as service providers. Second, the demonstrations could be competing for a
common pool of potential participants.

For two demonstrations—the Mental Health Treatment Study (MHTS) and the California HIV/AI
Demonstration—we believe that the problems from confusion and lost sample will be manageable.
While persons with mental illness constitute a substantial proportion of SSDI-only beneficiaries, the
MHTS sites are geographically limited. Thus, so long as these sites do not expand, beneficiaries with
pertinent diagnoses can be excluded from offset random assignment within designated areas;
applicants and beneficiaries with other diagnoses will not be affected.

The California project will cover larger areas—Los Angeles, San Diego, San Francisco and Alameda
Counties—but is confined to a smaller diagnostic subgroup of the eligible SSDI population. Again,
therefore, we plan to exclude beneficiaries with pertinent diagnoses from random assignment in the
designated areas, but include both applicants and beneficiaries with other diagnoses. While we do not
know what proportion of the potential offset participants in the four California counties have HIV or
auto-immune disabilities, we do know that over 40 percent of the count of beneficiaries shown in
Exhibit 2-1 above for the area offices in the West High stratum are in the four area office jurisdictions
overlapping the counties in the HIV/AI study.8

The Accelerated Benefits (AB) Demonstration is a more serious challenge. The newly approved
beneficiaries targeted by AB are also a key population for the offset demonstration, for two reasons.
First, most of them lack health insurance until (after two years and five months from onset) they gain
Medicare coverage. Thus, the health care package is probably the most significant inducement for
this group, and both demonstrations would be offering it. Second, new beneficiaries are closer to the
work place and are considered more likely to return to work; this is the group potentially siphoned off
by the accelerated benefits demonstration.

Excluding these beneficiaries from the samples for shared sites would seriously harm the offset
evaluation. Thus, we recommend that the offset and AB projects identify mutually exclusive sites for
the two studies.

We will select BOND sites without regard to the location of other demonstration sites. We will then
determine where overlaps with other demonstrations—in terms of geographic coverage and
beneficiary type—occur, and tailor our demonstration outreach to avoid those beneficiaries in the
overlap. This will result in a sample that is generalizable to the national SSDI-only population less
those beneficiaries who have diagnoses that make them eligible for the other demonstrations and who
live in the geographic areas where the those demonstrations overlap with BOND. It is important to
recognize that we will still be able to represent those diagnoses with the BOND sample in the non-
overlapping sites.

8
    It appears that the following California area office jurisdictions contain part or all of the four HIV/AI
    demonstration counties: H86 (Oakland, CA), H88 (Carlsbad, CA), H89 (Sherman Oaks, CA), and H92
    (Los Angeles, CA). Absent maps showing the actual area office boundaries, we cannot tell whether
    additional offices are affected.




Abt Associates Inc.                                               Chapter 2: Sample Design                      10
For example, in any BOND site that includes an MHTS site, we will not conduct targeted recruitment
to existing beneficiaries or enroll applicants with a mental health diagnosis covered by MHTS in the
areas of geographic overlap with MHTS. Because we are sampling roughly 20 percent of all SSA
Area Offices, we would expect to overlap with about one-fifth of the MHTS sites, or about four sites.
It is our understanding, from the description of these sites provided by SSA, that the MHTS sites
comprise the catchment areas of 22 mental health treatment facilities; thus, the MHTS sites are much
smaller than the proposed BOND sites. If this is correct, we would expect only a very small fraction
of the overall BOND target population to be excluded by this criterion. If necessary, to ensure an
adequate sample of such beneficiaries in those sites, we will over-sample beneficiaries and applicants
with those mental health diagnoses in the non-overlapping portion of the site.

The same strategy seems feasible with respect to the HIV/AI demonstration. Because these diagnoses
are likely to constitute only a small proportion of the overall beneficiary population, simply excluding
them from outreach in those areas where BOND and the HIV/AI demonstrations overlap should have
little effect on the overall BOND sample. Again, if necessary, we can over-sample such cases in the
non-overlapping portion of these sites.

We have less information about site selection in the Accelerated Benefits demonstration. We would
expect to pursue the same basic approach with respect to this demonstration, but will want to
coordinate with the evaluation contractor for the demonstration, as we get ready for site selection.
One possibility that we would like to explore with SSA and the Accelerated Benefits evaluation
contractor is simultaneous selection of sites for the two demonstrations. This would allow both to
obtain a nationally representative set of sites with no overlap between the two demonstrations.

Area Offices that Cross State Lines
We initially recommended use of the SSA Area Office as the unit for site selection for two reasons:
because each beneficiary is uniquely associated with one office; and because these offices serve
sufficient numbers of beneficiaries and applicants to provide the sample needed for the
demonstration. An additional advantage is that in many cases the area offices cover major
metropolitan-area labor markets.

In the draft employment supports design (Long et al., December 2005), we recommended that, for
each demonstration site, responsibility for case counseling and management be given to a consortium
of organizations spanning the range of services needed to support RTW efforts in the demonstration
population. These consortiums would be known as Work Support Organizations (WSOs). Each
consortium would have a lead agency—which would have overall responsibility for program
operations—and several partner organizations. Key local organizations, such as centers for
independent livings, could submit a bid to be the lead organization, but they also could be partners in
bids led by other organizations.

We also noted that sites covering more than one state would present special issues, as illustrated by
the following examples:

        1. Multiple State Vocational Rehabilitation Agencies (SVRAs) in a site could have different
           Orders of Selection and other salient differences in how they serve the SSDI population;



Abt Associates Inc.                                          Chapter 2: Sample Design                   11
        2. Medicaid buy-in programs differ substantially from state to state in their coverage
           features and their eligibility standards, while some states do not offer buy-ins at all; and
        3. Jurisdictions of other agencies likely to be part of the WSO partnerships (such as BPAOs
           and P&A organizations) might also be bounded by state lines.


In the past (e.g., Stapleton et al., December 2005) we have urged SSA to consider excluding isolated
field offices located in different states from the bulk of the offices within any SSA Area that is
selected for the Demonstration. However, other steps can be taken as part of the WSO solicitation
process to address the issues such offices raise. First, proposal requirements and rating criteria can be
used to encourage bidders to develop partnerships that adequately address the needs of isolated
offices. Second, the solicitation can make WSO service expectations clear, specifying that
employment services be offered everywhere in the site, including isolated offices. Third, we can
issue separate solicitations for each state within a site. Organizations could then easily bid on work
covering all states within a site, but they would not be forced to do so.



2.2     Selection of Sample Members

In Section 1.2, we described the basic eligibility requirements for the demonstration. Here, we detail
how demonstration sample members will be selected and recruited from the eligible beneficiaries and
applicants. Within each demonstration site, current beneficiaries and new applicants will be recruited
into the sample over a two-year period and randomly assigned to several different treatment
conditions, including current law. This description has been revised based on recent changes to the
proposed treatments (see the memo ―Updated Estimates of Precision and Sample Sizes for the Benefit
Offset National Demonstration‖ of July 17, 2006).

Selection and Recruitment of Current SSDI-Only Beneficiaries

Selection of Current Beneficiaries
Exhibit 2.2 shows the flow of current beneficiaries into the various demonstration groups as revised.
In this design, all eligible existing beneficiaries in each demonstration site are randomly assigned to
three experimental conditions in the first stage: a current law control group (C0), a treatment group
that receives only the 50 percent offset (T0), and a group that is to be solicited for participation in the
employment support program (the employment support target group, ES).9 This last group does not
receive any treatment at this stage.

This first-stage random assignment is done from administrative records; it involves no direct contact
with beneficiaries. However, those assigned to the 50 percent offset will receive materials from SSA
explaining the new, more generous treatment of earnings should they go to work and assuring them

9
    In practice, there will probably be some exclusions from random assignment. For example, we will
    probably want to exclude beneficiaries who will reach retirement age before the end of the demonstration.
    For this reason, and to distinguish those randomly assigned at this stage from those who subsequently
    volunteer for the employment supports component of the demonstration, we refer to this first group as
    ―eligible beneficiaries‖—i.e., beneficiaries who are eligible to participate in the demonstration.




Abt Associates Inc.                                             Chapter 2: Sample Design                    12
that this does not affect their SSDI beneficiary status in any other way, including entitlement to full
benefits during the Trial Work Period and three-month grace period, use of Ticket-to-Work, use of
benefits planning and assistance services, and entitlement to Medicare. If they receive assistance from
other programs, such as their state’s medical assistance program or housing subsidies, they will be
cautioned that such assistance might be reduced or eliminated if they increase their earnings
sufficiently, and they will advised on how to determine whether such assistance would be affected.
Materials explaining this new treatment, along with other features of the demonstration, will also be
prepared and distributed to SSA staff and to the broader set of interested actors and agencies.

Beneficiaries in the ES target group will be sent letters and/or contacted in person inviting them to
participate in the employment supports component of the demonstration.10 Those who volunteer for
the ES component will be randomly assigned to one of several treatments, described below.

Within each site, we will also randomly select Workforce Investment Boards to receive special
funding, which they will use to augment the employer-linked services their One-Stop Centers offer
for demonstration participants. Thus, each site will be divided into two distinct geographic areas: the
catchment areas of the enhanced One-Stops and the catchment areas of the regular One-Stops.
Within each of these areas, those in the ES target group who volunteer for the employment support
program will be randomly assigned to one of four experimental treatments:

             A current law control group;
             A group that receives the 50 percent offset ($1-for-$2) only;
             A group that receives the 50 percent offset plus the BOND core benefit; and
             A group that receives the 25 percent offset plus the BOND core benefit.


(For ease of description, we refer to the combination of employment supports, Independence




10
     In practice, there may be a lag between first-stage random assignment and solicitation of the ES target group,
     because we do not want to delay the implementation of the 50 percent offset for the eligible beneficiary
     population while we develop the employment support systems in the sites. To ensure that the group solicited for
     the employment supports component of the demonstration is representative of the overall eligible beneficiary
     population, it will be necessary to solicit not only those assigned to the ES target group in the first stage of
     random assignment, but also a proportionate share of eligible beneficiaries who have come onto the rolls
     between the time of first-stage random assignment and the time of solicitation. Thus, for example, if half of all
     existing eligible beneficiaries are assigned to the ES target group in the first-stage random assignment, we would
     solicit half of the eligible beneficiaries who came onto the rolls in the interval between first-stage random
     assignment and solicitation.




Abt Associates Inc.                                                   Chapter 2: Sample Design                       13
                                     Exhibit 2-2
               Recruitment and Random Assignment, Current Beneficiaries


                                          Existing SSDI-only
                                            Beneficiaries
     C0: Current Law
      Control Group
         (Eligible
      beneficiaries)                          Random
                                             Assignment

     T0: 50% Offset
        (Eligible
                                        ES: Employment
     beneficiaries)
                                      Support Target Group




                         Enhanced One-                     Regular One-
                           Stop Areas                       Stop Areas




                      Outreach                                             Outreach



                Volunteers                                                 Volunteers




                 Random                                                     Random
                Assignment                                                 Assignment



              CCE: Current Law Controls                                   CCR: Current Law
                                                                          Controls
                       C50E: 50% Offset
                                                                              C50R: 50% Offset

               T50E: 50% Offset + Health
              Insurance + Empl. Supports                                    T50R: 50% Offset +
                 +Enhanced One-Stop                                       Health Insurance + Empl.
                                                                                  Supports

               T25E: 25% Offset + Health
              Insurance + Empl. Supports                                     T25R: 25% Offset
                 +Enhanced One-Stop                                         +Health Insurance +
                                                                              Empl. Supports




Abt Associates Inc.                                            Chapter 2: Sample Design              14
Accounts, and the health benefit as the BOND core benefit.) In addition, within the catchment areas
of the enhanced One-Stops, the latter two groups will also receive the enhanced One-Stop services.

Recruitment of Beneficiary Volunteers
Beneficiaries assigned to the Employment Support Target Group at the first stage will receive an offer
to volunteer for the full set of demonstration benefits (offset plus core benefit). Those in the ES group
who do not volunteer for this option (the ―non-volunteers‖ in Exhibit 2-2) do not receive any
demonstration treatment at all, and data about them only come from administrative records to
determine how they differ from those who volunteer.

The eligible beneficiaries assigned to the ES target group in Exhibit 2-2 will initially be sent a letter
inviting them to consider participation in the study, with the possibility that they will be selected to
receive an experimental benefit that will allow them to maintain SSDI eligibility and still receive
partial SSDI benefits if they return to work. Some will also be selected to receive significant
additional support for working that is not currently available to other beneficiaries.

The recruitment letter will ask them to call a toll-free number to arrange an initial phone interview, or
to return a postcard indicating that they are not interested in being considered further for the study.
Those who take neither action within two weeks will be called and recruited by phone.

At the beginning of the telephone interview with those who express interest in the demonstration,
each interviewee will be asked to verify that he/she meets demonstration eligibility criteria, including
non-receipt of private disability insurance or workers compensation indemnity payments. The
interview will be terminated if the individual does not pass this screen. If the interviewee passes the
screen, the interviewer will provide additional information about the study, including:

            A summary description of the benefits that the individual might be offered if he/she
             enters the study; 11
            A statement that the beneficiary might be included in the study but not receive any
             additional benefits; and
            A statement that, to be included, the beneficiary would have to agree to cooperate with
             the study’s data collection activities, responding to a series of surveys and granting the
             evaluators permission to access his/her administrative data for evaluation purposes only.


To conclude, the interviewer will invite the beneficiary to attend an in-person enrollment session, to
provide him/her with further information about current and demonstration benefits, confirm
demonstration eligibility, and (if still interested) enroll him/her in the demonstration.

Those attending the in-person enrollment session will be provided with information on the work
incentive and employment support features of the current program and what would happen to their
benefits (under current rules) if they were to increase their earnings to amounts of interest to them.

11
     We will inform individuals that they may receive either a 25 or 50 percent offset, and may also receive
     additional supports to help them find and maintain employment. We will, of course, also inform them that
     they may be assigned to the control group.




Abt Associates Inc.                                              Chapter 2: Sample Design                   15
They will also receive an initial description of the demonstration benefits that may be offered to them
if they decide to participate.

However, the potential volunteers at the enrollment session will not receive any individual benefits
counseling at this time, because of the second-stage random assignment to treatment or control status
still to come. It is vital to strike a balance here. The information and explanation provided at intake
are critically important for genuine informed consent and for enrolling an appropriate population in
the demonstration. The less information provided at intake, the more likely it is that the
demonstration will enroll people who will later refuse (or not take advantage of) the offered
treatments. A high take-up rate on treatments is one key to generating measurable impacts in this
demonstration. On the other hand, the enrollment session should not provide so much information to
control group members (those later assigned to CCE or CCR) that it changes their behavior after
random assignment. If that happens, the experimental design is undermined, because the control
groups no longer accurately represent the volunteers among existing SSDI beneficiaries.

A few subjects may be screened out at this stage. The enrollment counselor12 will also explain the
informed consent process in more detail. All subjects who complete the process to this point will
then be offered the opportunity to complete the informed consent document and the baseline survey.
All those who complete these steps (i.e., who ―volunteer‖) will be randomly assigned to treatment or
control status.13

Selection and Recruitment of SSDI Applicants

Exhibit 2-3 shows the intake process for the Early Intervention (EI) component of the demonstration.
As described earlier (see Section 1.2), SSDI-only applicants will be eligible to participate in the
demonstration if they meet the inclusion criteria for beneficiaries and are also likely to be found
eligible for SSDI. We have used the term ―provisional eligibility‖ to describe the status of applicants
who meet demonstration criteria for ―likely to meet or equal the listings.‖

SSDI-only applicants will be screened for likely SSDI eligibility very shortly after their application is first
filed, using the IBM Quick Decision (QD) tool. Those with high probability of eligibility will be offered
participation in the EI component of the demonstration, but they will need to provide some medical
documentation to verify their potential eligibility.14 They will be invited by mail to call an 800 number
if they are interested in participating, or (if not) to return a postcard requesting that they not be
considered further for the study. Those who take neither action within two weeks will be called and

12
     The intake counselor may be from the Work Support Organization (the lead agency for the demonstration
     in a particular site), as long as separate staff who do not provide case guidance to treatment group members
     are utilized. Alternatively, the demonstration operator and evaluator could handle recruitment and random
     assignment for the demonstration.
13
     Although we have described a group session format for enrollment, we should anticipate and plan for in-
     person, one-on-one sessions in people’s homes or other locations (as needed).
14
     We understand that the QD tool outputs a predicted probability for each case. Based on that understanding,
     we plan to use these predicted probabilities to target a broad range of applicants—perhaps nearly the entire
     upper half of the distribution. But it also makes sense to require medical evidence of everyone who has not
     already provided ti to SSA. Those who cannot provide such evidence will screen themselves out.




Abt Associates Inc.                                               Chapter 2: Sample Design                     16
                                       Exhibit 2-3
                  Sample Selection and Random Assignment, Applicants




Abt Associates Inc.                              Chapter 2: Sample Design   17
recruited by phone. For applicants who have been approved for expedited determination,15 the phone
interview and the remainder of the intake process will be the same as for beneficiaries (see above),
except that during the process they will learn about benefit provisions that apply to applicants.

Applicants who do not qualify for expedited review—if interested and otherwise eligible—will be
screened further by phone. The additional screen will be a series of questions about the applicant’s
medical conditions, designed to determine whether the applicant can bring medical evidence to the in-
person interview that will be considered equivalent to the minimum evidence required for expedited
review. Applicants who pass this screen will be invited to the interview and instructed to bring the
evidence identified. A qualified intake counselor or health worker will examine this evidence at the
beginning of the interview and determine whether the individual is provisionally eligible.16 We also
recommend that SSA offer such individuals an expedited determination—the same process offered to
those determined provisionally eligible via the IBM screening tool.

All those determined to be provisionally eligible will be provided with benefits counseling on SSDI
to the same extent they currently are and invited to complete the informed consent and enrollment
documents. They will then complete the baseline survey.

From that group, those who volunteer will be randomly assigned to one of three groups: a group that
receives the core benefit with a low ES incentive, one that receives the core benefit with a high ES
incentive, or a current law control group. 17 Since each site will contain Workforce Investment Boards
(WIBs) that receive special funding to provide enhanced One-Stop services and other WIBs that do not
receive the special funding, this creates six distinct treatments. (Note that—for simplicity—the split
between regular and enhanced One-Stop services is not shown in the exhibit.)

Outreach and Engagement

There is a critical point in demonstration intake when beneficiaries and applicants must choose
whether to participate in the demonstration. For beneficiaries, this point comes when those assigned
to the ES target group are contacted and invited to volunteer. For applicants, the point comes when
those who meet the medical screen requirement are invited to continue through the demonstration
enrollment process.

An outreach and engagement effort—with some counseling component— must be targeted to both
sets of potential volunteers. All potential participants should be able to make a genuinely informed
decision about demonstration participation. For current beneficiaries, vigorous outreach with a clear

15
     This is SSA’s new process. The criteria for expedited determination still need to be established.
16
     The intake function may be carried out by staff of the local Work Support Organization or by the
     demonstration operator (to separate the recruitment function from case guidance provided to treatment
     group members).
17
     As with beneficiaries, it will simplify the discussion with potential volunteers if, prior to the enrollment
     session, we randomly assign each potential volunteer to the treatment they would receive if assigned to the
     treatment group and discuss only that treatment (and the possibility of being assigned to the control group)
     with the applicant. Once they have volunteered, signed the informed consent, and completed the baseline,
     they would then be randomly assigned to either that treatment or the control group.




Abt Associates Inc.                                                Chapter 2: Sample Design                    18
message and strong supporting materials will be needed to ensure that those assigned to the
demonstration treatments understand the opportunities available and are likely to take advantage of
the demonstration’s benefits. SSDI applicants will probably need to receive an explanation of the
basics of SSDI under current rules, before the intake worker goes on to describe the benefits the
demonstration offers.

However, these individuals actually commit little, and face little risk, by agreeing to be part of the
demonstration. While random assignment to a treatment group means an automatic shift to the new
cash benefit regime, the monthly cash benefit that the individual receives will not be reduced unless
the beneficiary earns enough for the offset to take effect, in which case every additional dollar
reduces benefits by a fraction of a dollar (50 cents or 25 cents, depending on the treatment group).18
The outreach effort should emphasize this fact.

Once volunteers are assigned to treatment plans that include employment supports, convincing them
(in a manner consistent with how it would be done in a national, non-demonstration program) to make
a real effort to take advantage of those supports will be both challenging and important to the ultimate
success of the demonstration. The job of working with enrollees to understand and use the benefits
will fall to the WSO’s Work Support Specialists. At a minimum, this engagement effort must entail
(1) discussing the program with as many treatment group members as possible (offering counseling
and pertinent information), and (2) encouraging each individual to take the first step in program
participation. The first step may involve benefits planning, initial contact with the health benefits
provider, discussing an increase in employment hours with a current employer, beginning to prepare
for new employment (through training or other activities), or other actions that address the enrollee’s
own individual situation.

The point being made here is that outreach and engagement efforts—although they begin with the
recruitment and intake of demonstration participants—will need to continue in various forms
throughout the demonstration period. We think the outreach plans and materials, as well as the
performance of the intake workers and Work Support Specialists, will prove to be crucial to the
success of the demonstration.




18
     During the first year following program enrollment, the individual’s cash benefit is not reduced. After the
     first year, the benefit deposited to the independence account by SSA is reduced when the treatment group
     member is unemployed or has very low earnings. However, this reduced cash deposit is supplemented by
     funds already deposited to the person’s independence account. Thus, while the treatment group member
     has an incentive to become employed—in order to maintain the value of his or her independence account as
     well as increase the size of the primary payments—he or she can continue to withdraw the full cash benefit
     allowed under current rules.



Abt Associates Inc.                                               Chapter 2: Sample Design                   19
Chapter 3:               Analytic Methods

In this chapter, we discuss the range of quantitative analytic methods that will be brought to bear on
the data generated for and by the benefit offset national demonstration (BOND). After a brief
summary of the analytic framework, in Section 3.2 we discuss the impact analysis—our general
approach to estimating experimental impacts, how we plan to analyze multiple treatment groups, and
the precision of the estimates from the impact analysis; impact estimation methods; and
generalization of the impact estimates.

Section 3.3 then turns to the cost-effectiveness analysis, describing the estimation of direct and
indirect costs and showing how these analyses will be included in a larger cost-benefit assessment. In
Section 3.4, we detail two different approaches we will use for forecasting long-term offset results
and costs. One involves use of the benefit offset simulation model (BOSIM), while the other is an
alternative forecasting model.



3.1     Framework for the Analysis

In this section, we describe the overall analytic approach, including the estimates of interest and the
samples and subgroups to be analyzed. In Section 3.2, we describe the impact analysis in more detail
and present required sample sizes and estimates of the precision they will afford.

Estimates of Interest

As described in Chapter 2, BOND will provide three different treatments to existing beneficiaries and
two to applicants, in local areas with and without enhanced One-Stop services. The same treatments
will be tested in each site with, to the extent possible, the same sample allocation among treatments in
each site. In addition, the experimental design includes current law control groups of all eligible
beneficiaries and all volunteers for the employment support component.

We assume that SSA has an interest in estimates of the impacts and cost-effectiveness of each of
these discrete treatments, relative to both current law and to the 50 percent offset alone. Estimates
relative to current law show the effect of replacing the current system with the treatment in question.
Estimates relative to the 50 percent offset only measure the incremental impact of features other than
the 50 percent offset—e.g., the degree to which treatments with employment supports add to the
effectiveness of the financial incentive embodied in the offset, taken by itself.

We also assume that SSA is interested in the effect of the 50 percent offset, taken by itself, on both
eligible beneficiaries and on volunteers for the employment support component. If an offset were to
be enacted without accompanying employment supports, there would be no need for beneficiaries to
volunteer and enroll in a program in order to take advantage of it. SSA would just change the way it
treats earnings in computing SSDI benefits and inform beneficiaries of that change. Since some
beneficiaries who would not volunteer for an employment support program would return to work and
take advantage of the offset, estimates based solely on volunteers would underestimate the aggregate



Abt Associates Inc.                                       Chapter 3: Analytic Methods                 20
effect of the offset. Therefore, it is important to estimate the impact of the offset alone on eligible
beneficiaries, as well as on volunteers.

Samples and Subgroups

The demonstration sample will comprise two major program groups: existing beneficiaries and
applicants. Existing beneficiaries are those receiving SSDI at the time the sample is drawn.
Applicants are individuals who apply for, and are judged likely to be eligible for, SSDI benefits
during the demonstration intake period. Within each of these groups, we will solicit volunteers for
the employment support treatments. Separate impact analyses will be conducted for existing
beneficiary volunteers and applicant volunteers. In addition, for the reasons discussed above, we will
conduct a separate analysis of the effects of the 50 percent offset alone on all eligible beneficiaries
(including those who do not volunteer for the employment support program) using SSA
administrative data.

Within each of these major groups, to the extent that sample sizes permit, we will also analyze the
impact of the demonstration treatments on subgroups that might be important for policy, either for
targeting the program for maximum impact or for assessing the equity of the program. Subgroups to
be analyzed will be defined by such baseline characteristics as:

           Type of disability;
           Age;
           Education;
           Gender;
           Length of benefit spell; and/or,
           Prior earnings level.

It is important to recognize that, because of their smaller sample sizes, subgroup impacts will be
estimated less precisely than those on the overall beneficiary or applicant groups, for which precision
estimates are presented in the next section. In general, a subgroup that comprises half the beneficiary
or applicant population will have a minimum detectable effect (MDE) approximately 1.4 times as
large as the MDE for the overall population; a subgroup that comprises a quarter of the population
will have an MDE that is twice as large as that for the overall population.



3.2     Impact Analysis

In this section, we describe the methods to be used to estimate the impacts of the demonstration
treatments, the samples sizes required for the impact analysis and their allocation across the
demonstration treatments, and the precision of the estimates attainable. We discuss estimation of
impacts on existing beneficiaries and applicants in turn. (See Chapter 4 for further discussion of
outcome measures.)




Abt Associates Inc.                                         Chapter 3: Analytic Methods                   21
Estimation of Impacts, Sample Sizes, and Precision of Estimates – Existing Beneficiaries

As explained in Chapter 2, those in the ES target group who volunteer for the employment support
program will be randomly assigned to one of eight experimental cells (see Exhibit 2-2), defined by
the combination of the four following treatments with a geographic division between regular and
enhanced One-Stop services in each site:

             A current law control group (CCR and CCE);
             A group that receives the 50 percent offset ($1-for-$2) only (C50R and C50E);
             A group that receives the 50 percent offset plus the BOND core benefit (T50R and
              T50E);
             A group that receives the 25 percent offset plus the BOND core benefit (T25R and
              T25E).


Direct comparison of these various treatment and control groups allows estimation of the following
net and differential impacts:1920

             Impact of 50 percent offset alone on eligible beneficiaries, relative to current law (T0 vs.
              C0);
             Impact of 50 percent offset alone on volunteers, relative to current law (C50E + C50R vs.
              CCE + CCR);
             Impact on volunteers of 50 percent offset plus health insurance and employment
              supports, relative to either current law or the 50 percent offset alone, in areas with regular
              One-Stop services (T50R vs. C50R or T50R vs. CCR);
             Impact on volunteers of 50 percent offset plus the BOND core benefit and enhanced One-
              Stop services, relative to either current law or the 50 percent offset alone (T50E vs. C50E or
              T50E vs. CCE);
             Impact on volunteers of 25 percent offset plus the BOND core benefit, relative to current
              law, in areas with regular One-Stop services (T25R vs. CCR); and
             Impact on volunteers of 25 percent offset plus the BOND core benefit and enhanced
              One-Stop services, relative to current law (T25E vs. CCE).




19
     Other comparisons are possible, but do not seem informative for policy. For example, comparison of the
     25% offset plus health insurance and employment supports with the 50% offset-only treatment confounds
     two different policy decisions: the level of the offset rate and the addition of health insurance and
     employment supports to the purely financial incentive provided by the offset rate.
20
     In addition to these comparisons, which provide impact estimates, comparison of outcomes between
     eligible beneficiaries subject to the 50% offset with volunteers under the 50% offset only treatment will
     provide a measure of self-selection into the employment support component of the demonstration.




Abt Associates Inc.                                             Chapter 3: Analytic Methods                      22
It is possible to infer other impacts and (as we will see) achieve greater power by estimating impacts
in a pooled regression that takes advantage of the fact that multiple treatment groups share common
values of the policy parameters. If, for example, we pool all volunteers and estimate an impact
equation with separate indicator variables for the 50 percent offset, the 25 percent offset, the core
benefit in regular One-Shop areas, and the core benefit in enhanced One-Stop areas, we can estimate
the following impacts:

             The effect of the 25 percent offset rate, relative to current law;21
             The effect of the 25 percent offset relative to the 50 percent offset;
             The incremental effect of the BOND core benefits in the regular One-Stop areas, relative
              to the 50 percent offset alone;
             The incremental effect of the BOND core benefits in the enhanced One-Stop areas,
              relative to the 50 percent offset alone; and
             The effect of enhanced One-Stop services, relative to the BOND core benefits under a 50
              percent offset rate.
The reason that a pooled regression gives greater power and smaller MDEs than direct comparisons is
that, while there are eight distinct volunteer groups, we can get all the information we need from only
five parameters. Thus, we do not have to estimate the mean of the dependent variable in eight
separate sub-samples. The ―savings‖ come from the following three places:

     (1) CCE and CCR are the identical treatment, implemented in different (randomly selected)
         catchment areas. Therefore the mean of the outcome measure is the same for these two
         groups, in expected value, and we don’t have to estimate them separately.

     (2) Likewise, C50E and C50R are the same treatment, implemented in different (randomly
         selected) catchment areas. Neither one includes employment services, health insurance, and
         so on. So again the mean of the outcome measure is the same for these two groups in
         expected value, and we don’t have to estimate them separately.

     (3) Finally, we are willing to assume that the interaction between the offset rate and the
         enhancement of services is zero. That is, the difference in outcomes between T50E and T50R
         is in expected value the same as the difference in outcomes between T25E and T25R. This
         eliminates yet another parameter to be estimated.

Exhibit 3-1 shows minimum detectable effects for these impacts, under the following assumptions
about sample sizes:

             The demonstration is implemented in 10 SSA area offices.
             There are 80,000 existing beneficiaries in each area office.

21
     The MDEs shown for this and the following estimate assume no interaction between the impact of the
     offset rate and the effects of employment supports. This assumption can be tested empirically, and the size
     of the interaction estimated; any significant interaction will increase the MDE for the combined effect.




Abt Associates Inc.                                             Chapter 3: Analytic Methods                   23
                              Exhibit 3-1. Minimum Detectable Effectsa and Sample Sizes – Beneficiaries
Abt Associates Inc.




                                                                                                                                                     Annual Earnings, Total SSDI Benefits,
                              Treatment Group/Effect                                                     N        Relative to:               N         Months 1-24        Months 1-24


                              Estimates based on comparison of individual cells

                              Eligible beneficiaries: 50% Offset only (T0)                            200,000     Current Law (C0)         200,000         $242               $67

                              Volunteers: All areas
                               50% Offset only (C50R+C50E)                                             5,000      Current law (CCR+CCE)     5,000          $467               $83


                              Volunteers: Regular One-Stop areas
                                   25% Offset plus core benefits (T25R)                                2,500      Current law (CCR)         2,500          $661               $118
                                   50% Offset plus core benefits (T25R)                                2,500      Current law (CCR)         2,500          $661               $118
                                   50% Offset plus core benefits (T50R)                                2,500      50% Offset only (C50R)    2,500          $661               $118


                              Volunteers: Enhanced One-Stop areas
                                   25% Offset plus core benefits (T25E)                                2,500      Current law (CCE)         2,500          $661               $118
                                   50% Offset plus core benefits (T50E)                                2,500      Current law (CCE)         2,500          $661               $118
Chapter 3: Analytic Methods




                                   50% Offset plus core benefits (T50E)                                2,500      50% Offset only (C50E)    2,500          $661               $118


                              Estimates based on pooled impact regression

                              Effect of 25% offset                                                       (b)      Current law                (b)           $661               $118
                              Effect of 50% offset                                                       (b)      Current law                (b)           $495               $96
                              Effect of 50% offset                                                       (b)      Effect of 25% offset       (b)           $495               $96
                              Incremental effect of core benefits, regular One-Stop areas                (b)      50% Offset only            (b)           $273               $70
                              Incremental effect of core benefits, enhanced One-Stop areas               (b)      50% Offset only            (b)           $273               $70
                              Incremental effect of enhanced One-Stop services                           (b)      Regular One-Stop           (b)           $792               $136
                                                                                                                  services
                              a
                                  Minimum detectable effects based on 80% power, .05 significance level (2-tail test).
                              b
                                  Based on pooled regression.
24
             In the first stage of random assignment, 50 percent of existing beneficiaries are randomly
              assigned, in equal numbers, to T0 and C0; the remaining 50 percent are solicited for the
              employment supports component of the demonstration.
             Of those invited to participate in the employment supports component, 5 percent
              volunteer.22
             Volunteers are assigned in equal numbers to the eight treatment and control groups
              involved in the second stage of random assignment.


These assumptions imply the following total sample sizes, across all sites:

             200,000 eligible beneficiaries are assigned to the 50 percent offset in the first stage of
              random assignment;
             200,000 eligible beneficiaries are assigned to the current law control group in the first
              stage of random assignment; and
             400,000 eligible beneficiaries are solicited for participation in the employment supports
              component.
             As a result, 20,000 eligible beneficiaries (5 percent) volunteer for the employment
              supports component; and
             2,500 volunteers are assigned to each of eight treatment and control groups in the second
              stage of random assignment.


Because two of the eight treatment conditions are current law control groups (CCR and CCE), this
implies that 15,000 volunteers (6 x 2,500) will receive some demonstration treatment. Of these,
5,000 will receive the 50 percent offset only (C50R and C50E); the remaining 10,000 will receive
health insurance and employment supports, as well as a 25 percent or 50 percent offset. Of those
receiving health insurance and employment supports, 5,000 (or 500 per site), will be in areas with
enhanced One-Stops.

The MDE estimates shown in Exhibit 3-1 are for national estimates, based on 80 percent power and a
two-tailed test of statistical significance at the .05 level. They take into account the effect of the
likely cross-site variation in impacts on the precision of the national estimates. We show MDEs for
the two outcomes that are most central to the demonstration’s objectives—earnings and SSDI
benefits. The specific measures analyzed here are average annual earnings over the first two years
after random assignment and total SSDI benefits over the same period. In the demonstration, it is
likely that impacts will be estimated over shorter and/or longer periods. We chose two years as a
mid-range follow-up period. Estimates for longer periods will be slightly more precise, as




22
     If the actual volunteer rate turns out to be different from 5 percent, we will adjust either the number of
     beneficiaries solicited or the length of the intake period to obtain the required sample size. The number of
     beneficiaries solicited can be increased or decreased by changing the fraction of the beneficiary population
     assigned to T0 or C0 at stage 1.

Abt Associates Inc.                                             Chapter 3: Analytic Methods                    25
percentages of the control mean or, equivalently, for impacts on average monthly earnings or benefits.
The corresponding estimates for shorter periods will be somewhat less precise.23

As can be seen, for the very large sample of eligible beneficiaries who receive the 50 percent offset only,
we will be able to detect impacts on annual earnings as small as $242, or about 9 percent of the control
mean, and impacts on total 24-month SSDI benefits of $67, or less than 1 percent of the control mean.
This level of precision, of course, does not apply to a typical individual affected by the treatment, but
rather to the whole group. We expect that the vast majority of the treatment group will be unaffected and
continue to have zero earnings or earnings below SGA, but a relatively small proportion of individuals will
increase their earnings to a level that is high enough to take advantage of the benefit offset (i.e., above
SGA). This implies that the proportion gaining must be maximized by the concerted efforts of the site
agencies, and that those gaining must make sizeable gains, for the effects to be detectable.24

Among the volunteers for the employment supports component, we will have the greatest precision
for the 50 percent offset, taken by itself. This treatment, which does not involve One-Stop services,
can be pooled across the regular and enhanced One-Stop catchment areas and compared to the current
law control group, also pooled across those areas. This will allow us to be confident of detecting
impacts of $467 on annual earnings (17 percent of the control mean) and $83 on 24-month SSDI
benefits (less than 1 percent of the control mean).

For each of the individual treatment groups within either the regular or enhanced One-Stop areas, we
will be able to detect with confidence impacts of $661 on annual earnings, or about 25 percent of the
control mean. We will have much better precision for estimating impacts on SSDI benefits—we will
be confident of detecting impacts of $118 on total 24-month benefits, or about 1 percent of the control
mean. The greater precision for impacts on benefits reflects the fact that the variance of benefits,
controlling for baseline characteristics, is much lower than the corresponding variance of earnings,
and its mean is higher.25


23
     This reflects the fact that the variances of earnings and benefits tend to rise less as the follow-up period increases
     than the corresponding mean. For example, in the Project NetWork data, the average ratio of the standard error
     of estimate of impacts on annual benefits to mean annual benefits, across sites in the first two years after random
     assignment, was .0353, whereas the ratio of the average standard error of estimate of impacts on total benefits
     over two years to mean benefits, across the same sites in the same two years, was .0336.
24
      Although they are not an identifiable group, we can still estimate the impact for those who experience increases
      in earnings above SGA and determine the statistical significance of the increases. The procedure is simple
      (attributed to Bloom and valid if the impact on everybody else—treatment group members who do not use the
      benefit offset—is zero). The mean impact on the entire treatment group is divided by the proportion in the
      treatment group that used benefit offset (i.e., benefits were partially reduced due to earnings above SGA in at
      least one month) to get the mean impact on the latter. The standard error is obtained by dividing the standard
      error for the whole group by the same proportion.
25
     The estimated within-site variance of the outcome and cross-site variance of the impacts used here are
     derived from Project NetWork data. Appendices A and B to this report describes the derivation of the
     cross-site variances and MDEs based on the corresponding variances of the Project NetWork impact
     estimates. For both outcomes, baseline values of the outcome, as well as other baseline characteristics of the
     beneficiary, are included as covariates in the impact regression. This set of covariates explains 75-80 percent of the
     variance of SSDI benefits, largely because most sample members’ benefits change very little over time. Baseline
     earnings have much less predictive power, because earnings are much less stable over time. Mean annual earnings
     and total 24-month SSDI benefits for this sample, in 2006 dollars, were $2,675 and $21,063, respectively.

Abt Associates Inc.                                                   Chapter 3: Analytic Methods                        26
For most impact estimates, the precision of the estimated impacts will be better in the pooled model,
in which we effectively combine treatment groups containing the same value of the parameter of
interest for purposes of estimating the impact of that parameter value. For those estimates, MDEs for
annual earnings range from $273 to $792, or 10 to 30 percent of the control mean. Of particular
interest is the MDE for the effect of the 50 percent offset, relative to either current law or the 25
percent offset; on these measures, we will be confident of detecting impacts of $495, or 19 percent of
the control mean. In the pooled estimates, the MDEs for impacts on 24-month benefits range from
$70 to $136; all are less than 1 percent of the control mean.

For assessing the adequacy of these levels of precision, the absolute dollar amount of the MDE may
be a better standard than the MDE as a percent of the control mean. Mean earnings are quite low in
this population; thus, even modest impacts might be large relative to mean earnings. Moreover,
relatively large impacts on earnings may be required for the treatments to be cost-effective. If that is
the case, there may be little policy interest in smaller impacts.26 When the bounded cost estimates for
the demonstration have been completed, we plan to compute the impacts that would be required for
the demonstration treatments to be cost-effective.

It is also important to note that the MDEs presented here are for the estimated impact of the ―intent to
treat‖—i.e., comparisons of outcomes for all individuals assigned to a particular treatment or policy
parameter with those of the entire control group, whether or not all those assigned to treatment
actually received it. To the extent that there is nonparticipation in the demonstration treatments, the
impact of the treatment on those who do participate must be proportionally larger for these ―intent to
treat‖ effects to be realized. This caveat applies only to the volunteers in treatment groups (i.e, those
assigned to treatment in Stage 2); all research subjects assigned to the offset-only treatment in Stage 1
will automatically receive that treatment. As noted in Chapter 2, this heightens the importance of a
continuing effort to engage those assigned to the employment support treatments to ensure that they
actually understand the potential value of those supports and then decide to use them.

Finally, it should be noted that in deriving these MDEs we have used two-tailed statistical tests. For
some estimates, one-tailed tests would be appropriate and would lead to greater power. For example,
if a one-tailed test were used, the MDE for the incremental effect of enhanced One-Stop services
would be reduced from $792 to $684.

Estimation of Impacts, Sample Sizes, and Precision of Estimates – Applicants

As described in Chapter 2, SSDI-only applicants with high probability of eligibility will be offered
participation in the EI component of the demonstration. From that group, those who volunteer will be
randomly assigned to one of three groups, within geographic areas with either regular or enhanced One-
Stop services: a group that receives the core benefit with a relatively low employment support (ES)
incentive (ALR and ALE), one that receives the core benefit with a relatively high employment support
incentive (AHR and AHE), or a current law control group (CCR and CCE).




26
     As in most return-to-work programs, we expect earnings to be the main social benefit of the Benefit Offset
     National Demonstration treatments. Although SSA quite reasonably has a strong interest in the impact of the
     demonstration on SSDI benefits, from a social perspective the impact on benefits is a transfer between
     beneficiaries and taxpayers that does not affect the net social benefit of the program.

Abt Associates Inc.                                              Chapter 3: Analytic Methods                       27
Direct comparison of these six experimental treatment and control groups allows estimation of the
following net and differential impacts:

             The effect of the core benefit with a low ES incentive, relative to current law, in areas
              with regular One-Stop services (ALR vs. ACCR);
             The effect of the core benefit with a high ES incentive, relative to current law, in areas
              with regular One-Stop service (AHR vs. ACCR);
             The effect of the core benefit with a low ES incentive, relative to current law, in areas
              with enhanced One-Stop services (ALE vs. ACCE);
             The effect of the core benefit with a high ES incentive, relative to current law, in areas
              with enhanced One-Stop services (AHE vs. CCE);
             The effect of the core benefit with a low ES incentive, relative to current law, across all
              areas (ALE+ALR vs. ACCE+ACCR);
             The effect of the core benefit with a high ES incentive, relative to current law, across all
              areas (AHE+AHR vs. ACCE+ACCR);
             The average effect of the core benefit, in areas with regular One-Stop services
              (AHR+ALR vs. ACCR); and
             The average effect of the core benefit, in areas with enhanced One-Stop services
              (AHE+ALE vs. ACCE).


Exhibit 3-2 shows minimum detectable effects for these impacts, under the following assumptions
about sample sizes:

             The demonstration is implemented in 10 SSA area offices.
             There are 17,500 SSDI-only applicants per year in each area office.
             Demonstration intake of applicants runs for two years in each site.
             50 percent of all applicants are judged to have a high probability of eligibility (using the
              IBM QD tool);27 and
             Of those invited to participate in the demonstration, 10 percent volunteer.28
             40 percent of the volunteers are randomly assigned to the current law control group, and
              the remaining 60 percent are split equally between the demonstration treatment plans
              with high and low ES incentives.29


27
     This means that, for purposes of BOND eligibility, the threshold on the probability levels that are the QD
     tool’s outputs would be set between 50 and 75%.
28
     If either the eligibility rate or the volunteer rate turn out to be different from those assumed here, we will
     adjust the length of the intake period to achieve the required sample size.
29
     This allocation was chosen because it minimizes the MDEs for the individual treatment groups. Where
     multiple treatment groups are compared to the same control group, it is optimal to allocate a larger share of
     the sample to the control group than to any individual treatment.

Abt Associates Inc.                                                Chapter 3: Analytic Methods                        28
                              Exhibit 3-2. Minimum Detectable Effects and Sample Sizes – Applicants
Abt Associates Inc.




                                                                                                                            Annual       Total SSDI
                                                                                                                           Earnings,      Benefits,
                              Treatment group/effect                             N            Relative to:         N      Months 1-24   Months 1-24

                              Estimates based on comparison of individual cells

                              Volunteers: Regular One-Stop areas
                               Low ES incentive (ALR)                           2,625      Current law (ACCR)     3,500      $610          $111
                               High ES incentive (AHR)                          2,625      Current law (ACCR)     3,500      $610          $111

                              Volunteers: Enhanced One-Stop areas
                               Low ES incentive (ALE)                           2,625      Current law (ACCE)     3,500      $610          $111
                               High ES incentive (AHE)                          2,625      Current law (ACCE)     3,500      $610          $111

                              Estimates based on pooling across cells

                              Effect of low ES incentive, all areas (ALR+ALE)   5,250   Current law (ACCR+ACCE)   7,000      $462          $91
Chapter 3: Analytic Methods




                              Effect of high ES incentive, all areas            5,250   Current law (ACCR+ACCE)   7,000      $462          $91
                              (AHR+AHE)
                              Average effect of core benefit, areas with        5,250      Current law (ACCE)     3,500      $531          $100
                              regular One-Stops (ALR+AHR)
                              Average effect of core benefit, areas with        5,250      Current law (ACCR)     3,500      $531          $100
                              enhanced One-Stops (ALE+AHE)
29
These assumptions imply the following total sample sizes, across all sites:

             There will be a total flow of 350,000 SSDI-only applicants over two years of intake in the
              demonstration sites.
             Of these, 175,000 will be predicted to have a high probability of eligibility and will be
              invited to participate in the early intervention component of the demonstration.
             17,500 of those invited to participate will volunteer.
             7,000 volunteers will be assigned to the current law control group.
             5,250 volunteers will be assigned to the core benefit with low ES incentive, split evenly
              between areas with enhanced and regular One-Stop services.
             5,250 volunteers are assigned to the core benefit with high ES incentive, split evenly
              between areas with enhanced and regular One-Stop services.


As with the MDEs shown earlier for beneficiaries, the MDE estimates in Exhibit 3-2 are for national
estimates, based on 80 percent power and a two-tailed test of statistical significance at the .05 level. They
take into account the effect of the likely cross-site variation in impacts on the precision of the estimates.30

As shown in the exhibit, we will have about the same precision for the impacts of the individual
treatments for applicants as for beneficiaries—MDEs for annual earnings of $610 and MDEs for 24-
month SSDI benefits of $111. Pooling across cells improves the precision of the estimates
somewhat—pooling either the low or high reimbursement rate treatments across One-Stop areas
reduces these MDEs to $462 and $91, respectively, and pooling low and high reimbursement
treatments within regular or enhanced One-Stop areas yields MDEs of $531 and $100.

Summary of Sample Sizes and Precision

Exhibit 3-3 summarizes the definition and sample sizes for all of the random assignment groups. We
believe that the sample sizes assumed here for both beneficiaries and applicants, though large, are
realistic, both in terms of the likely number of volunteers in the demonstration sites and in terms of
the ability of the demonstration to provide services. We can readily adjust for any deviation from the
beneficiary volunteer rate assumed here by adjusting either the first-stage assignment rate or the
length of the intake period. Moderate deviations from the applicant volunteer rate assumed here can
be accommodated by changing the length of the intake period. In each site, the demonstration will be
required to provide employment supports to 1,000 beneficiaries and 1,050 applicants, with half of
each of these groups located in areas with enhanced One-Stop services.

The estimated MDEs for impacts on SSDI benefits are quite small, reflecting the low variability of
benefits. The MDEs for annual earnings are larger, especially relative to the control mean, even in
the pooled estimates. It may be, however, that demonstration costs are a better standard against
which to measure the MDEs for earnings than the control mean. Once the bounded cost estimates

30
     The estimated within-site variance of the outcome and cross-site variance of the impacts used here are derived
     from Project NetWork data. For earnings, we use the upper bound of the 90 percent confidence interval around
     the estimated cross-site variance; for benefits, we use the upper bound of the 95 percent confidence interval.



Abt Associates Inc.                                               Chapter 3: Analytic Methods                     30
Exhibit 3-3. Definition and Size of Randomly Assigned Groups In the Benefit Offset
National Demonstration
                                                                 RA         # Assigned          Total #
  Group                    Treatment (base)
                                                                Stage         per Site         Assigned
 SSDI-Only Beneficiaries
            Current law control group (eligible
 C0                                                                1          20,000             200,000
            beneficiaries)
 T0         50% offset only (eligible beneficiaries)               1          20,000             200,000
            Current law control group from enhanced
 CCE                                                               2            250               2,500
            One-Stop areas (volunteers)
            50% offset only from enhanced One-Stop
 C50E                                                              2            250               2,500
            areas (volunteers)
            Core benefit with 50% offset, from enhanced
 T50E                                                              2            250               2,500
            One-Stop areas (volunteers)
            Core benefit with 25% offset, from enhanced
 T25E                                                              2            250               2,500
            One-Stop areas (volunteers)
            Current law control group from regular One-
 CCR                                                               2            250               2,500
            Stop areas (volunteers)
            50% offset only from regular One-Stop
 C50R                                                              2            250               2,500
            areas (volunteers)
            Core benefit with 50% offset, from regular
 T50R                                                              2            250               2,500
            One-Stop areas (volunteers)
            Core benefit with 25% offset, from regular
 T25R                                                              2            250               2,500
            One-Stop areas (volunteers)
            Beneficiaries (volunteers), all groups                 2           2,000             20,000
 SSDI-Only Applicants (Early Intervention)
            Current law control group from enhanced
 ACCE                                                             N/A           350               3,500
            One-Stop areas (likely eligible applicants)
            Core benefit with low ES incentive, from
 ALE        enhanced One-Stop areas (likely eligible              N/A           262               2,625
            applicants)
            Core benefit with high ES incentive, from
 AHE        enhanced One-Stop areas (likely eligible              N/A           263               2,625
            applicants)
            Current law control group from regular One-
 ACCR                                                             N/A           350               3,500
            Stop areas (likely eligible applicants)
            Core benefit with low ES incentive, from
 ALR        regular One-Stop areas (likely eligible               N/A           262               2,625
            applicants)
            Core benefit with high ES incentive, from
 AHR        regular One-Stop areas (likely eligible               N/A           263               2,625
            applicants)
            Applicants (Early Intervention), all groups                        1,750             17,500
NOTES:
Volunteers are those members of the employment support target group that give their informed consent to join
the demonstration.
The core benefit refers to the combination of: Independence Accounts; employment supports (benefits
counseling and employment services); and the BOND health benefit.




Abt Associates Inc.                                           Chapter 3: Analytic Methods                  31
have been completed, we plan to compare the MDEs for annual earnings with the impacts that would
be required to make the demonstration cost-effective.

Estimation Methods

With properly designed and implemented random assignment, treatment-control comparisons of raw
means provide unbiased estimates of impact. Use of regression analysis to control for baseline
characteristics that affect the outcome improves the precision of the estimates while preserving their
unbiased character. The estimates of precision presented above assume such regression adjustments,
with precision gains based on those obtained in the Project NetWork evaluation.

While ordinary least squares regression will yield unbiased impact estimates, for a clustered sample
such as this one it will not give correct standard errors because it does not properly account for
variation in impacts across sites. To obtain the correct standard errors of estimate, we plan to
estimate the impacts with hierarchical linear models, treating site and individual sample members as
two different levels of analysis. The equations for the model will be of the form:

        yij = α + ΣβjkTijk + Σγm Xijm + δj + εjj
        δj = η1 + θ1 Zj + ν1j
        βjk = η2 + θ2k Zj + ν2jk, k=1 to K

        where yij is the outcome for participant i in site j,
              Tijk is an indicator that participant i in site j is receiving treatment k,
              Xijm is a measure of baseline characteristic m for individual i in site j, and
              Zj is a measure of a characteristic for site j.

The first of these equations explains outcomes for individual participants; the second and third
explain variations across sites. With only 10 sites, we assume that we can incorporate no more than
one site characteristic, e.g. the local unemployment rate. These equations therefore imply that:

            Individual outcomes can be decomposed into an overall mean and components
             representing the effects of the site, individual baseline characteristics, and the treatment;
            Effects of individual characteristics on the outcome do not vary across sites;
            The effects of each treatment (k=1 to K) do vary across sites, depending on the
             unemployment rate and also random variation; and
            Even absent any treatment, expected outcomes differ across sites depending on the
             unemployment rate and random variation.


Alternatively, if there is reason to believe that variations in effects across site are associated with (for
example) the local unemployment rate, to obtain the correct standard errors of estimate we plan to
estimate a mixed model with random site effects and random site-treatment interactions.

Finally, some outcomes of interest, such as benefit receipt in a particular month, are dichotomous
rather than continuous variables. For such outcomes, linear regression —even with hierarchical linear
modeling —gives inefficient estimates and may predict impossible values of the outcome (i.e., values



Abt Associates Inc.                                          Chapter 3: Analytic Methods                  32
less than zero or greater than one). For such outcomes, we will use methods designed to handle
dichotomous variables, such as logistic regression or probit.

Generalization of the Impact Estimates

The BOND sample is designed as a nationally representative, self-weighting sample. If the sample is
drawn as described here, both the impact estimates derived from this sample and their standard errors
will apply directly to the national eligible SSDI-only population at the time of random assignment and
to the population of likely eligible SSDI-only applicants during the demonstration intake period. But
some minor departures from this straightforward applicability of the results are likely, for several
operational reasons.

First, as noted in Chapter 2, it will probably be necessary to exclude some eligible beneficiaries and
applicants from demonstration outreach in those areas that are served by other demonstration (i.e., the
Accelerated Benefits, California HIV/AI Study, and the Mental Health Treatment Study). This will
mean that the BOND results are generalizable to the national SSDI population less the specific groups
(e.g., those with mental health diagnoses) in the specific areas that were excluded. We expect this to
have a negligible effect on the generalizability of the estimates.

Second, to avoid some of the operational difficulties of integrating the BOND treatments with private
benefits, we propose to exclude those beneficiaries who receiving private disability insurance or
workers’ compensation. This limits somewhat the population to which our estimates can be
generalized.31

Third, if volunteer rates among existing beneficiaries or applicants depart from those assumed here, it
may be necessary to adjust the random assignment rates for the relevant group(s). If this occurs after
sample build-up begins, it will be necessary to assign weights to individual sample members
randomly assigned under different ratios, to balance treatment and control groups. This is a common
procedure in experimental analysis that has little effect on the resulting estimates or their precision.32



3.3      Participation Analysis

This component of the benefit offset national demonstration evaluation will quantitatively assess the
intensity and participation patterns of the various demonstration treatments. The analysis will focus
on the comprehensive benefit and health insurance outcomes shown in Exhibit 3-4.




31
     The demonstration will, however, provide information that will be useful for the design of an offset applied
     to combined private and SSDI benefits.
32
     The main reason to avoid such ―midstream‖ changes in the random assignment ratio is that it entails some
     loss of precision. However, for small differences in random assignment ratios or differences that affect
     only a small proportion of the sample, this loss is minimal. For a detailed discussion of this procedure, see
     Orr et al. (2003), Appendix B, sections 3.4 and 6.1.




Abt Associates Inc.                                              Chapter 3: Analytic Methods                    33
                                             Exhibit 3-4
    Participation Analysis Outcomes for the Benefit Offset National Demonstration

                                              Measurable for

                                             Treatment   Control
                                                     a         b                  c
 Outcome                                      Groups     Group     Data Sources

 Ever received BOND cash payments                         NA      BOND operations data, interim and
                                                                     follow-up surveys

 Average length of BOND cash payment                      NA      BOND operations data, interim and
    spells                                                           follow-up surveys

 Average amount of monthly BOND cash                      NA      BOND operations data, interim and
    payments                                                         follow-up surveys

 Ever used BOND independence account                      NA      BOND operations data, interim and
                                                                     follow-up surveys

 Ever used BOND independence account for                  NA      BOND operations data, interim and
    employment supports                                              follow-up surveys

 Ever used BOND independence account to                   NA      BOND operations data, interim and
    supplement cash payments while                                   follow-up surveys
    unemployed

 Ever received BOND medical benefit                       NA      BOND operations data, interim and
    payments                                                         follow-up surveys

 Average amount of annual BOND medical                    NA      BOND operations data, interim and
    payments                                                         follow-up surveys

 Ever received BOND medical benefit                       NA      BOND operations data, CMS data, interim
    payments while covered by Medicare or                            and follow-up surveys
    Medicaid (supplementary coverage)

 Ever received monthly earnings that                             SSA data, UI data, interim and follow-up
    qualified for BOND cash payments                                 surveys

 Ever received monthly earnings high                             SSA data, UI data, interim and follow-up
    enough to receive zero BOND cash                                 surveys
    payments

 Ever received vocational training                               BOND data, RSA data, WIA data, interim
                                                                     and follow-up surveys

 Ever took community college classes                             BOND data, RSA data, WIA data, interim
                                                                     and follow-up surveys

 Ever took high school or GED-preparation                        BOND data, RSA data, WIA data, interim
    classes                                                          and follow-up surveys

 Ever received job search or job placement                       BOND data, RSA data, WIA data, interim
    assistance                                                       and follow-up surveys

 Ever received benefits counseling                               BOND data, RSA data, WIA data, interim
                                                                     and follow-up surveys

 Ever received career counseling                                 BOND data, RSA data, WIA data, interim
                                                                     and follow-up surveys

 Ever received pre-employment supports                           BOND data, RSA data, WIA data,




Abt Associates Inc.                                        Chapter 3: Analytic Methods                  34
                                              Measurable for

                                             Treatment   Control
                                                     a         b                   c
 Outcome                                      Groups     Group      Data Sources
     (internship, on-the-job training,                                 interim and follow-up surveys
     supported work)

 Covered by public health insurance (BOND,                        BOND program data, CMS data, interim
    Medicare, Medicaid)                                               and follow-up surveys

 Average amount of annual BOND, Medicare                          BOND program data, CMS data, interim
    and Medicaid payments                                             and follow-up surveys
 a
   Excludes the first-stage 50 percent offset only treatment group.
 b
   Excludes the first-stage current law group.
 c
   UI=Unemployment Insurance; RSA=Rehabiliation Services Administration; WIA=Workforce Investment Act.



Treatment Group Outcomes

Some of these outcomes, such as those related to receipt of the BOND cash benefit and use of the
independence account, can only be measured for treatment group members. Although the
information will not be available for controls, it will be available for all the treatment groups (except
the first-stage offset-only group), and thus provide a differential treatment analysis. This will be very
important information in understanding the nature of the treatments as compared to each other. Part of
the analysis of these outcomes will be descriptive helping to document the treatments received by
members of our evaluation sample. This descriptive analysis of treatment-group-only outcomes will
focus on:

             Participation and utilization rates. Each of these rates represents the percentage of
              treatment group members who achieve the given outcome, e.g. the percentage of sample
              members in a given treatment group who ever receive a BOND cash benefit or use an
              employment support funded by independence accounts.
             Participation and utilization patterns. Patterns involve the achievement of one or more
              outcomes during specified periods of time. Examples include the number of months in
              which sample members receive BOND cash benefits during the entire observation period,
              the receipt of more than one support during the first year following random assignment,
              and utilization of both cash benefits and independence-account-funded supports.


This is an essential aspect of the evaluation. Indeed, the way in which the various BOND treatments
will ultimately be characterized will depend on measured participation rates and patterns. For
example, if a small fraction of sample members assigned to a particular treatment group ever receives
a cash benefit or takes advantage of employment supports, but all utilize their BOND health benefit
package, the treatment might be viewed primarily as a health insurance intervention. It also is likely
that the treatment’s work incentives will be judged to have been inadequate to generate a substantial
employment-related response from sample members.

The analysis of treatment-group-only outcomes also will involve systematic comparisons across sites,
cohorts of treatment group members, and subgroups. We will estimate separate equations for



Abt Associates Inc.                                        Chapter 3: Analytic Methods                   35
subgroups. This answers the question, how did the program work for a particular group of people
(e.g., mentally disabled beneficiaries) with all their other typical characteristics? That is usually what
policy makers want to know.

Substantial differences in outcomes between sites or groups could be important to the evaluation’s
conclusions. For example, if the overall rates of cash benefit receipt and employment support use are
low, but the rates are much higher at one or two sites, this could be related to site differences in
program implementation (such as differences in marketing) or environment (such as in the availability
of a Medicaid Buy-In program or the strength of the job market).

Similarly, if the rates are low during most of the observation period, but rise during the period’s last
few months, this could raise questions about the time required by disabled individuals to respond to
the opportunities provided by the BOND treatments.

The examination of treatment-group-only outcomes will involve more sophisticated statistical
analysis. The specific analyses will be dictated by questions raised by the participation results. For
example, it is likely that we will assess the factors that determine key participation outcomes such as
receipt of BOND cash payments.

For applicants, it will be especially important to examine how impacts vary with the applicant’s score
on the screener used to determine the probability of allowance, as well as on other baseline
characteristics that might potentially be used for screening purposes in a national program. We have
recommended that SSA set a fairly low cutoff for the probability of allowance. As a result, a
substantial share of applicant participants might ultimately be medically ineligible for SSDI.
Although such applicants will likely benefit from the program, their benefits might be low relative to
program costs, and their participation is likely to increase program costs. Hence, policymakers might
be interested in more stringent medical eligibility criteria for applicants. The analysis of applicant
data can examine how participation, benefits and costs would change under more stringent eligibility
criteria.

Treatment-Control Differences

Although these analyses of treatment-group-only outcomes will provide valuable information about
the BOND treatments, they will not tell us the extent to which cash payments, supports, and medical
benefits received by the treatment groups differ from those received by the control group. The
impacts on these outcomes tell us about BOND’s net treatment. It is the net differences in payments,
supports, and benefits that will determine whether the ultimate objectives of the BOND
interventions—medical improvement, return to work, and increased economic self-sufficiency—are
achieved. We can only estimate treatment differences (impacts on cash assistance, health insurance
payments, and service utilization) for outcome measures that are observed for both treatment and
control group members. Exhibit 3-4 indicates outcomes that will be measured for both groups. Some
of these outcomes will be measured using on BOND program data, while others will be captured
using SSA, UI, RSA and CMS automated records data. Additional measures will be provided by the
follow-up surveys.




Abt Associates Inc.                                         Chapter 3: Analytic Methods                    36
As indicated earlier, BOND program data will permit us to measure the extent of use of the programs’
health benefit package. By combining these data with CMS data on Medicaid and Medicare, we will
be able to determine the treatments’ net effect on public health insurance coverage. Similarly, we can
examine the net treatment for cash payments (looking at regular DI payments as well as BOND
payments) and employment supports (using RSA data).

This is an important issue for many reasons, e.g. because it is possible that demonstration benefits
will, to some extent, substitute for services that would otherwise have been provided by Medicaid.
Another possible result of the demonstration is that treatment group members will obtain medical
treatment or employment services earlier than control group members.

The approach we will use to estimate the impacts of the treatments on cash benefits, health insurance,
and employment support outcomes will be the same as the approach we will use to estimate impacts
on other outcomes. The analysis will focus on the mean effect of the BOND treatments on these
outcomes—that is, the impact of ―intent to treat,‖ which is estimated by comparing outcomes for the
treatment group to those of the control group.

Connecting the Impact and Process Analyses

The results of the participation analysis will inform the interpretation of the impact findings. This is
often done subjectively. However, there is one further and more formal way to examine the
relationship between the participation analysis and the generation of impacts on medical
improvement, economic self-sufficiency, and other key outcomes: testing of causal linkages posited
to be important if changes in beneficiaries’ employment outcomes are to be affected. Sometimes
called ―theories of change‖ or ―pathway analysis‖, this methodology uses logic models to map the
anticipated linkages between demonstration intervention activities and intended changes in participant
outcomes. Once the ―map‖ is laid down—ahead of looking at the data—impact analysis data are used
to measure and test whether the hypothesized ―linking events‖ between intervention intentions,
service delivery, intermediate outcomes, and final outcomes actually take place.

The value of this form of causal attribution of impacts to the source mechanisms relies heavily on
theory in the application of the classical scientific method: state a hypothesis, explain one’s basis for
thinking the hypothesis is true, define an empirical exercise capable of falsifying or confirming the
hypothesis, then run that test. The crucial steps are the first ones—in this application, specifying in
advance exactly why one thinks the interventions undertaken would change individuals’ behaviors
and resulting labor market and benefit receipt outcomes. One version of this approach lays out these
advance hypotheses using ―logic models,‖ which are essentially path diagrams of the connections
hypothesized to stretch from what SSA does to prompt, fund, and support the BOND interventions to
the final, ―furthest down-stream‖ beneficiary outcomes sought, such as improved health status and
higher sustained earnings. Other external factors besides the demonstration intervention are also
recognized in the structure that gets mapped out, to ensure their influence is not overlooked in
subsequent analysis.

―Change theory‖ models then consider which of the links in this pathway of steps to successful
impacts are supported by existing health science, labor economics, and benefits policy research. In
effect, they ask: ―Are there flaws in the logic model, given what we know about how the target




Abt Associates Inc.                                        Chapter 3: Analytic Methods                 37
populations behave or local employment agencies and physicians cope with newly-defined missions
and assignments?‖ The past empirical foundation for the intervention’s motivating logic is called into
question to ensure its robustness and—depending on which domains of beneficiary outcomes are in
fact found to experience an impact—add a basis for interpreting why the intervention, say, helped
sample members with physical conditions obtain jobs faster than those with mental health disorders.
Because the logic model connects factors other than the intervention to the various outcomes of
interest, possible mediating influences get equal play in discussing what may be going on ―behind the
scenes.‖

The final, and statistically most formal, part of this technique is to check for measured impacts—
treatment group events that are different, on average, from control group events—at all the
hypothesized link points in the logic model that can be observed once the intervention is
implemented. Differences in participation outcomes shown in Exhibit 3-1 will be critical to this
process. For example, higher employment support utilization rates by sample members with physical
injuries could help explain why they experienced higher impacts on long-term outcomes.



3.4     Benefit-Cost Analysis

The benefit-cost analysis will compare the dollar value of each BOND treatment’s impacts to its
costs. This comparison will be made from the perspectives of SSA Trust Funds, the rest of the
federal government, state and local government, and eligible SSDI applicants and beneficiaries, as
well as society as a whole.

Exhibit 3-5 summarizes the analytical framework for the cost-effectiveness study. The exhibit
indicates the components we plan to include in the analysis (the rows) and the perspectives from
which they will be assessed (the columns) for each BOND intervention. The pluses and minuses in
the table indicate whether each item is expected to be a benefit or a cost (the actual results of the
evaluation, of course, could be different).

As with the impact study, the proposed benefit-cost analysis will estimate net benefits and costs—that
is, the benefits and costs relative to what would have been without the BOND interventions. The
estimation methods will mirror the regression techniques described for continuous outcomes in the
impact analysis. Once measured, particular impacts or treatment-control expenditure differences will
constitute benefits or costs, or transfers, depending on the analytical perspective:

           The recipients’ perspective will identify gains and losses to members of the various
            treatment groups. This perspective will indicate how recipients’ income changes as a
            result of BOND cash benefits and the impacts of the interventions on earnings, taxes,
            SSDI and SSI payments, UI and TANF payments, and work-related expenses. There are
            also intangible benefits associated with faster rehabilitation and increased self-
            sufficiency, as well as costs of reduced time for leisure or volunteer work.
           The perspective of the SSA Trust Funds captures all budgetary gains and losses pertinent
            to the funds. Of critical importance to the results for this point of view is whether the
            intervention encourages beneficiaries to earn more, thereby reducing SSDI program




Abt Associates Inc.                                       Chapter 3: Analytic Methods                38
                                                Exhibit 3-5
               Expected Effects for Components of the Benefit-Cost Analysis,
                                By Accounting Perspective
                                                                        Federal
                                                                      Government
                                      Expected       Applicants
                                     Direction of       and         SSA Trust           State/Local
Component of Analysis                  Effects      Beneficiaries    Funds      Other   Government     Society
SSDI benefits                           Lower             -            +          0          0           0
SSI benefits                            Lower             -            +          0          0           0
SSDI and SSI administrative costs       Lower            0             +          0          0           +
Earnings and fringe benefits           Higher            +             0          -         0               +
Payroll taxes                          Higher            -             +          +         0               0
Income and sales taxes                 Higher            -             0          +         +               0
Medicare/Medicaid payments             Lower             +             0          -         -               0
Medicaid buy-in premiums               Higher            +             0          -         -               0
Medicare and Medicaid                  Lower             0             0          -         -               -
   administrative costs
VR program costs (not funded by        Higher            0              0         -          -              -
   SSA)
UI payments                            Higher            -              0         +         +               0
UI administrative costs                Higher            0              0         +         +               +
TANF payments                          Lower             -              0         +         +               0
TANF administrative costs              Lower             0              0         +         +               +
Food stamps                            Lower             -              0         +         0               0
Food stamps administrative costs       Lower             0              0         +         +               +
BOND cash payments                     Higher            +              -         0         0               0
BOND independence accounts             Higher            +              -         0         0               0
BOND health plan                       Higher            0              -         -         -               -
SSA-funded employment support
   costs                               Higher            0              -         -         -               -
BOND administrative costs              Higher            0              -         -         -               -
Value of faster rehabilitation and     Higher            +              0                   0               +
   improved self-sufficiency not
   captured elsewhere



             expenditures and increasing payroll tax revenue. BOND expenditures on medical
             benefits and independence accounts, as well as Social Security payroll taxes, also must
             be taken into account. (Note: The induced demand for SSDI benefits resulting from
             BOND will be considered as part of the sensitivity tests described below.)


            The other-federal-government perspective tallies benefits and costs to federal agencies
             outside SSA. This assessment will require valuing these other agencies’ share of the
             program investment in the BOND interventions—adding the program expenses incurred
             by the Department of Education Rehabilitation Services Administration (RSA), for
             vocational rehabilitation services; the Department of Health and Human Services (CMS),
             for Medicare and the federal-share of Medicaid; the Department of Labor, for federal
             workforce development expenditures and the federal share of unemployment insurance;
             and potentially programs supported by other federal agencies (e.g., Food Stamps, and




Abt Associates Inc.                                           Chapter 3: Analytic Methods              39
              housing subsidies) on behalf of treatment group members, and then subtracting the other-
              federal costs associated with any services or allowances provided to the control group.
              This perspective will also take into account program effects on outcomes such as federal
              tax payments.
             The state and local government perspective tallies the benefits and costs of BOND to
              state and local government treasuries. This will take account of the state-funded costs of
              state VR agencies (not paid for by RSA or by SSA), the state and local share of the costs
              of Medicaid (not paid for by CMS), and other programs along with program effects on
              state and local taxes and other pertinent outcomes.
             The social perspective combines the other perspectives. For example, a program-
              generated increase in federal income taxes would be a cost to recipients and a benefit to
              the federal government. Consequently, this transfer of money within society would be
              neither a benefit nor a cost to society as a whole. From this perspective, the primary
              benefit is the increased (pre-tax) earnings of beneficiaries and the primary costs are the
              additional resources used to provide additional medical and employment services, and to
              administer the program.


The various monetary effects of the BOND intervention—the benefits and costs that can be valued in
dollars—will be added together to determine the net dollar value of the program from each of these
perspectives. The effects that cannot be measured in dollars will be taken into account in the
sensitivity analysis.

Estimation of Benefits

Some of the expected monetary effects of the programs that are shown in Exhibit 3-5, such as the
effects on earnings and SSDI payments, will be estimated as part of the impact analysis described
earlier. In several other cases, a measured impact will be multiplied by an estimated dollar value in
order to determine the benefit. For example, the program’s effect on Unemployment Insurance (UI)
administrative costs will be estimated by multiplying the impact on months receiving UI by the
estimated monthly administrative cost per case. In still other cases, impacts will be estimated
specifically for this benefit-cost assessment, using estimation models similar to those described for
the impact analysis. An example is the program impact on income above specified thresholds, which
will be estimated in order to compute the effects of the BOND intervention on federal and state
income taxes.

All such calculations will be based on program impacts measured during the follow-up period, which
will vary for eligible beneficiaries, volunteer beneficiaries and EI volunteers as well as data source.
This period will be 36 months for the outcomes measured using follow-up surveys and as much as 72
months for those measured with program records data.33 In order to be useful, however, the benefit-
cost analysis must (1) take into account the fact that the impacts can be expected to last well beyond


33
     The follow-up period will be longest for the first cohort of enrollees at sites that begin the random assignment
     process early in the evaluation. The length of the period will also depend on the type of administrative records
     (i.e., the delay in obtaining complete individual-level records from SSA, CMS, UI, and other agencies.




Abt Associates Inc.                                                Chapter 3: Analytic Methods                          40
this period, and (2) measure benefits and costs over a consistent time period (the length of this period
should not vary by data source). Impacts will consequently be extrapolated beyond what is actually
observed (see discussion in next section).

Several program effects cannot be fully monetized. For example, the full value of employment, and
the more independent living it makes possible, may not be captured by impacts on earnings. Other
benefits, such as improved emotional wellbeing or increased job satisfaction, cannot be assessed in
solely monetary terms. However, as discussed earlier in this proposal, we will assess impacts on
outcomes like these. To the extent that such impacts cannot be monetized, the estimate of total
benefits will represent a lower bound on actual benefits. However, assumptions regarding their dollar
value can be made in the sensitivity analysis.

Estimation of Costs

Cost estimation will be challenging, especially to the extent that BOND sites are organized differently
and use different accounting systems. However, the objectives of this effort will be clear:

           Identifying the resources used by the BOND programs from SSA and other federal, state,
            and local sources;
           Collecting expenditure data corresponding to the use of these resources during the time
            recipients are enrolled in the BOND programs;
           Identifying the costs of services and allowances provided to sample members (controls as
            well as treatment group members) by programs other than BOND; and
           Estimating total program costs per treatment group member and per control group
            member.


It is quite possible that individual sites will handle certain expenditure issues differently than one
another. For example, different procedures may be used to allocate joint expenditures among several
programs, one of which could be BOND. There also may be different practices for addressing
allowance costs, the documentation of individual expenses, and accrued program costs. However, the
proposed evaluation team is experienced in addressing inconsistencies in expenditure data and
making appropriate adjustments as necessary.

Analysis of Benefits and Costs

Once they are estimated, the benefits and costs listed in Exhibit 3-5 will be aggregated to determine
the net present value of the BOND interventions from each of the accounting perspectives—that is,
the difference in the present discounted values of all pertinent benefits and costs for each perspective.
These net present value estimates are the traditional ―bottom line‖ of benefit-cost analyses.

Additional types of analysis will also be undertaken. First, given that some important program effects
will be measured and projected with uncertainty and others will not be measured in dollars, a
sensitivity analysis will be undertaken. All major assumptions used in the analysis will be described.
Then a series of sensitivity tests will be performed in which the net present value of the interventions,
from each of the perspectives, is re-estimated given a change in one or more assumptions. For



Abt Associates Inc.                                        Chapter 3: Analytic Methods                 41
example, one set of tests will involve lengthening the time horizon from ten to 20 years and
shortening it to cover only the follow-up period. Another will include changes in the assumptions we
use (to reflect more optimistic and pessimistic scenarios) regarding the future earnings for participants
whose education or training has not been completed by the end of the study’s observation period.
Finally, dollar values can be assigned somewhat arbitrarily to significant program effects that cannot
be monetized. These sensitivity analyses will help identify the net present value of the BOND
intervention under various projections and provide a sense of the robustness of the benefit-cost
results.

One set of critically important sensitivity tests will involve the results of the non-experimental
analysis of induced entry. These tests will be consistent with the staged approach to studying induced
entry that we have recommended to SSA (Stapleton et al., December 2005). In the first stage, we will
first use non-experimental modeling to measure the effects of the major changes in the SGA level in
the 1990s, which posed incentives similar to those that would be created by the offset. Another non-
experimental estimation effort could use data from the offset experiment. During the life of the
demonstration research as currently contemplated, sensitivity tests of benefit-cost results will utilize
these non-experimental findings.

In the second stage, we will work with SSA to assess the preliminary results from the offset
demonstration. If the offset treatments appear to have sufficient impacts on employment and earnings
to be of policy interest, we would then design a separate experimental test to measure induced entry
more rigorously. This separate experiment—while it would extend the offset demonstration’s
timeline—would also permit us to measure directly any effects on program entry and consequently
allow us to strengthen this potentially crucial aspect of the benefit-cost analysis.

Depending on the net present value findings and sensitivity test results, several additional analyses
can be helpful. For example, if the net present value of BOND for society as a whole was found to be
negative after ten years—that is, the program produced a net cost to society—the non-monetized
benefits of the programs could be systematically compared to this net cost. Alternatively, suppose the
net present value to the SSA Trust Funds is found to be positive after ten years, but the result depends
on projecting measured effects beyond the period covered by the follow-up survey. In this case we
could estimate how long measured effects would have to last beyond the 24th month in order for the
net value to be positive.



3.5     Forecasting Long-term Results and Costs

The results of the impact and cost-benefit analyses will be limited to the evaluation’s five-year
observation period. However, both the impacts and net costs of BOND can be expected to extend
well beyond the end of this period. Hence, we will forecast future impacts on key annual outcomes,
including: earnings, benefits, benefit deposits, after-tax household income, tax revenues (federal
income taxes, state and local income taxes, and payroll taxes), end-of-year balances in the benefit and
employment support sub-accounts of the independence account (if relevant), and program
expenditures for employment and health services. These projections are only intended to estimate
impacts on demonstration participants. They will not take into consideration potential impacts on the




Abt Associates Inc.                                        Chapter 3: Analytic Methods                42
number of SSDI applicants and awardees (i.e., induced demand). That important issue will be
addressed outside of the demonstration’s evaluation.34

The projections will be based on models developed from data generated in the five-year BOND
observation period. We will project impacts over the 10-year period following the observation
period; thus, counting the observation period, impacts will be estimated over a 15-year period. All
monetary values during the observation period will be adjusted for inflation, using the same consumer
price index that SSA uses to make cost-of-living adjustments, and the projections for the post-
observation period will be for inflation-adjusted values. Present values of impacts for the full 15
years will be computed, using the discount rate that SSA’s actuaries use to make Trust Fund
projections. As the reliability of projected impacts declines with the length of the projection period,
we will also compute present values of impacts for the first five years (i.e., the observation period),
and the first 10 years. The impact estimation for the observation period will produce confidence
intervals for the 5-year estimates.

The methodology for the projections will draw on our experience in modeling and projecting key
outcomes under the current program for the Benefit Offset Simulation Model (BOSIM). It will have
two parts. The first part will project which members of the study population have not been terminated
from SSDI due to retirement or death.35 The second part will project the mean values of the key
outcome variables in each projection year for those in each treatment and control group who are not
terminated from SSDI due to retirement or death. Differences between treatment and control
projections for these means will be used to estimate mean impacts for those who remain eligible, and
will be multiplied by the projected number not terminated due to retirement or death to obtain total
impacts.

Part 1: Projection of Terminations Due to Age or Mortality. Termination due to retirement will be
based on whether or not the participant has reached the normal age of retirement. To project
mortality, we will apply a binomial (e.g., probit or logistic regression) model for the probability of
mortality. The mortality model will be estimated from the observation period data. We plan to
estimate three models: one for the eligible beneficiary treatment and control groups, one for
beneficiary volunteers, and one for applicant volunteers. The key independent variables in each model
will be age, sex, primary impairment and, in the beneficiary models, years since first SSDI
entitlement. For each participant, we will use the relevant model to produce a predicted probability,
then take a draw from a binomial distribution with that probability to determine whether the
individual will be withdrawn from the sample for projections in that year and future years.

Part 2: Projection of Key Outcomes for Those Remaining in the Sample. For each year, we will
project outcomes for each individual based on past year outcomes, using simple models developed
from the observation period data. As the dynamics for each outcome are likely to be different for each

34
     As discussed in Stapleton et al. (December 2005), estimates of induced demand will be made using non-
     experimental methods.
35
     The vast majority of SSDI terminations are due to retirement or death. Terminations due to other factors,
     including medical recovery and return to work, will not be modeled because they may well be sensitive to
     the intervention. It is also possible that mortality will be sensitive to the intervention. We will test for that,
     but our expectation is that any impact will be too small to be measurable.




Abt Associates Inc.                                                 Chapter 3: Analytic Methods                       43
treatment and control group, due to the nature of the treatments, we will estimate these models
separately for each group. The means of the projected values for individuals will be used to complete
the impact projections.

For example, to project earnings, we expect to estimate a two-equation model for each group. The
first equation will be a binomial model to predict whether or not the individual has positive earnings.
This will be applied in a fashion analogous to the use of the mortality model to project which
participants have earnings in the year. Predictors will include age, sex, impairment, an indicator for
earnings in the previous year, years since SSDI entitlement (for beneficiary groups only) and possibly
other variables. The second equation, applied only to those with positive earnings, will be a
regression equation for the logarithm of earnings, using the logarithm of earnings in the most recent
year with positive earnings (the previous year in most cases) as the key explanatory variable.36 To
project the logarithm of earnings for those predicted to have positive earnings, we will compute the
predicted value from the earnings equation and then add a random draw from a normal distribution
with a mean of zero and a standard deviation equal to the standard error of the earnings regression
model.




36
     These specifications reflect the specifications that we have found to work well for beneficiaries who have
     entered the EPE. We will experiment with alternatives using the observation period data.




Abt Associates Inc.                                             Chapter 3: Analytic Methods                       44
Chapter 4:                 Data Sources for the Quantitative
                           Analyses
In this chapter, we detail how the various kinds of data that will be collected for the Benefit Offset
National Demonstration will be used in the quantitative analyses. This chapter first describes the uses
of administrative data, then of survey data, and finally of site data (both operations data and
contextual data).



4.1      Uses of Administrative Data

Administrative data from SSA and other public agencies will play a key role in the quantitative
analyses for the BOND evaluation. They will be the source of several important outcome measures
for the impact analysis, as well as providing baseline covariates and some cost data for the study.
Indeed, the entire first-stage impact analysis will be carried out entirely with administrative data.

Exhibit 4-1 shows the uses of a wide range of administrative data elements. In groups by topic, the
table indicates their sources, purposes, and analytic uses. Some of the administrative data will be used
to examine sample composition and test the comparability of the different treatment and control
groups. For instance, we will compare the demographic characteristics, disability status, and
employment characteristics (work history and current status) of treatment group members in T0 to
control group members in C0, to identify any slight differences that might have occurred by chance
between these groups.37

Data collected prior to random assignment will be used to adjust for known and pre-existing
differences among sample members in the impact analysis. For example, including indexed monthly
earnings as a covariate in a regression testing for impacts on current earnings has the effect of
adjusting for earnings differences related to employment before disability onset.

We have continued to include multiple possible sources of earnings data in the listing of
administrative data. Discussions are ongoing with SSA regarding whether the several concurrent
demonstrations will need to collect earnings information from state Unemployment Insurance (UI)
programs, which involves negotiation of data use agreements on a state-by-state basis. There are two
other possibilities that would be far more efficient, if the arrangements can be made. The first would
be to gain researcher access to the National Directory of New Hires (NDNH), which already collects
UI data from all states for purposes of child support enforcement. The second would be to depend on
the annual earnings data the IRS provides to SSA (although quarterly data would be preferable); we
have laid out the evaluation timeline in Chapter 6 based on use of these annual earnings data.

Exhibit 4-1 also shows that we will use TTW and Rehabilitation Services Administration (RSA) data
on return-to-work activities. For the offset-only and the second-stage RA control groups, these are
the key data to which BOND ES levels will be compared.

37
     We continue to use the group labels from Chapter 3 (see Exhibit 3-3).




Abt Associates Inc.                         Chapter 4: From Data to Quantitative Analyses               45
4.2      Uses of Survey Data

Survey data will be collected from beneficiary and applicant volunteers for BOND at several different
points in time: at baseline (during the enrollment process and prior to random assignment), at 12
months after RA, and at 36 months after RA. Exhibit 4-2 shows the extensive set of data elements
that will be gathered and which of the three instruments will contribute to the various groups of
elements.

As with the pre-program administrative data, some baseline survey information will be used to
examine sample composition and test the comparability of the different treatment and control groups.
Whereas the administrative data sources provide a fairly minimal set of personal characteristics on
current beneficiaries—and even less on applicants—the baseline survey will provide more detailed
and complete information (e.g., on race and ethnicity) as well as a wider range of variables (such as
education and marital status). The baseline data will be used extensively as covariates in the impact
analyses involving the beneficiary second-stage RA groups and the applicant RA groups. (Refer to
Exhibit 3-3 for the complete list of these groups).

The interim and follow-up surveys will be significant sources of outcome data for the impact
analyses, as well as providing input to the process study. Measurement of major outcomes in the areas
of household income, health status and functioning, barriers to employment, and return-to-work
activities will depend on these data.

Combinations of administrative and survey data will also be useful to the quantitative analyses. For
example, administrative data on participant earnings may be combined with survey data on the
earnings of other household members.



4.3      Uses of Demonstration Site Operations Data

The database environment being designed for the benefit offset national demonstration will serve a
number of different purposes and a wide variety of users. The critical set of variables on experimental
status for the entire demonstration and analysis will reside in the BOND data system from the start.
We expect to draw the participant sample from SSA administrative data—on the universe of current
beneficiaries and applicants—resident in the data warehouse, using Abt Associates’ random
assignment software running in the database environment. This software will generate a unique case
identifier and a permanent record of eligibility and beneficiary/applicant status against the full set of
cases in each site, at the start of random assignment. As RA proceeds, the software will generate
permanent variables recording the random assignment date(s) and group assignment(s) of each case.38
This set of variables will guide the demonstration treatment.




38
     Volunteers from the ES target group of beneficiaries will have two RA dates and two group assignments,
     due to the staged intake process. The second-stage date and assignment will govern BOND treatment. The
     applicant volunteers will only have one RA date and one group assignment.




Abt Associates Inc.                        Chapter 4: From Data to Quantitative Analyses                  46
Another major function (and set of system requirements) for the BOND database environment is
directed to assisting the on-site staff that will work with participants assigned to many of the
treatment groups. These staff members may be Work Support Specialists, other staff of the WSO
lead agency, staff of the One-Stop Career Centers that will deliver enhanced employer–linked
services to some treatment groups, or staff of other employment service providers working with a
large number of BOND participants.39 The data system will help these workers to:

          Access information about individuals;
          Record the results of interactions with them (such as benefit planning sessions, development
           of employment plans, and service referrals); and
          Record a variety of outcomes related to the demonstration treatment components (monthly
           cash benefit, independence accounts, employment supports, enhanced employer-linked
           services, and health-related services).

An important by-product of staff data system usage is the collection of data about the delivery of
supports and services to BOND participants. Most of the site operations data elements listed in
Exhibit 4-3 derive from the anticipated interactions of these staff members with their clients. In some
categories (e.g., disability status), the basic information will have already been gathered from
administrative data and the baseline survey, but the BOND data system will capture additional detail
and also reflect the participant’s own view of his/her current strengths and limitations. In other
categories (e.g., employment outcomes), the system can capture information not otherwise available,
such as data from employers (via the Fiscal Agent managing the independence accounts) on work
hours, FUTA payments, and tax withholding.



4.4        Uses of Demonstration Site Characteristics Data

The last major type of data likely to play a role in the quantitative analyses is information on the
characteristics of the ten demonstration sites. Although the data elements listed in Exhibit 4-4 are
largely directed toward the process analysis, it is quite likely that some of them will be useful in the
impact analysis, as controls for salient differences among the sites.

Some examples should clarify how the contextual data gathered for the process study might be
applied to the quantitative analysis. Regarding site organizational environment (the first group of data
elements in Exhibit 4-4), we might expect that the availability of the benefit offset could stimulate
demand for employment supports among current beneficiaries. Thus, we might test for any
relationship between measures of Ticket to Work activity level—such as the number or proportion of
beneficiaries with assigned tickets in the site—and employment support use by the first-stage
beneficiary RA groups. Regarding SSA office characteristics for the sites, we might expect that in
areas with greater availability of work-incentive staff (AWICs and WILs) in SSA area and field
offices, the T0 group would be more likely to understand the benefit offset and be better able to use it.



39
     No determination has been made yet of the extent to which other agencies serving BOND participants will be
      linked into the BOND database environment.




Abt Associates Inc.                          Chapter 4: From Data to Quantitative Analyses                   47
Thus, we might use measures of SSA staffing in equations testing for impacts on offset use (e.g.,
number of months with earnings above SGA).

Differences in demonstration implementation between sites could also affect outcomes for the various
treatment groups. For example, sites with more employer involvement might see more participants
with jobs and/or more participants with jobs with benefits. Sites with a certain type of WSO lead
agency (e.g., a mental health agency) might be more likely to have significant impacts for different
subgroups among the BOND population. We would create dummy variables to represent these
different characteristics of the implementation, based on careful analysis and documentation in the
process study. They would then be carried over into the quantitative analysis of impacts.




Abt Associates Inc.                     Chapter 4: From Data to Quantitative Analyses               48
Abt Associates Inc.




                                                Exhibit 4.1

                                                Uses of Administrative Data for the Quantitative Analyses

                                                Administrative Data Elements                                     Source                     Purpose                      Study Use
                                                Personal characteristics
                                                Identifiers: Participant name, date of birth, and SSN            SSA MBR                    Identifier data serve as     Impact analyses
                                                                                                                                            link to all analysis
                                                                                                                                            variables
                                                Demographics (subset): Race; ethnicity; sex                      SSSA MBR                   Baseline covariate data      Same

                                                Experimental status
Chapter 4: From Data to Quantitative Analyses




                                                Beneficiary or applicant at entry, random assignment                                        Output data from random
                                                group, whether first- or second-stage random assignment,                                    assignment software,
                                                random assignment date.                                          BOND operations data       distinguishing and guiding   Same
                                                                                                                                            treatment for all demo
                                                                                                                                            participants

                                                Disability status
                                                Type of primary disability; secondary disability, if any; date   SSDI data (only for        Background information       Same
                                                of onset; functional limitations; years of SSDI receipt          beneficiaries)             on disability
                                                Current receipt of SSDI                                          SSA MBR                    Current benefit receipt      Impact and cost
                                                                                                                                                                         analyses
                                                Indexed monthly earnings; type of claim; date of initial         SSDI administrative data   Amount of benefits           Impact and cost
                                                entitlement; date of current entitlement; beneficiary status     (beneficiaries only)                                    analyses
                                                (monthly); benefit amount (monthly)


                                                Income, earnings, and other benefit receipt
                                                Earnings data from SSA (SER) or UI; benefits data and            SSA, UI                    Outcomes: income and         Impact and cost
                                                primary insurance amount (PIA) from SSA                                                     earnings                     analyses
                                                Medicare receipt: Whether beneficiary, coverage start            SSA, CMS                   Outcomes: coverage,          Same
49




                                                date; which Parts; amount of claims paid; amount of                                         amount paid, amount
                                                premiums paid in.                                                                           claimed
Abt Associates Inc.




                                                Exhibit 4.1

                                                Uses of Administrative Data for the Quantitative Analyses

                                                Administrative Data Elements                                    Source                          Purpose                      Study Use
                                                Medicaid receipt: Whether beneficiary, coverage start date;     Same                            Outcomes: coverage,          Same
                                                amount of claims paid; amount of premiums paid in.                                              amount paid, amount
                                                                                                                                                claimed
                                                Ticket-to-Work Program: Status of Ticket at demo entry,         SSA                             Eligibility. Outcomes:       Same
                                                amount of provider payments made to date, pending                                               employment support
                                                                   40
                                                provider payments.                                                                              usage and costs


                                                Employment outcomes
Chapter 4: From Data to Quantitative Analyses




                                                Work history: Earnings for 12 months prior to random            UI, SSA (SER)                   Covariate: prior earnings.   Impact and cost
                                                assignment; earnings during demonstration period                                                Outcome: earnings            analyses
                                                Current employment status, if employed: Current monthly         UI, SSA (SER). National         Outcomes: employment         Same
                                                earnings, quarterly UI earnings; annual earnings                Directory of New Hires          status, earnings, benefits
                                                                                                                (NDNH) is possible
                                                                                                                alternative source.


                                                Return-to-work activities
                                                Employment services: Service providers; dates of service        SSA (for TTW); RSA (for         Outcomes: service            Impact, process and
                                                receipt; types of services received; completion of service;     state VR agencies); other       provision and variations     cost analyses
                                                elapsed time between random assignment and service              state and local agency          across models
                                                completion; cost of each service.                               administrative data
                                                Education and training: Participant use of job training or      Same                            Outcomes: education and      Impact and process
                                                education services to help obtain work; types and duration                                      training                     analyses
                                                of trainings




                                                40
                                                     If current beneficiaries who have assigned their tickets in the past are excluded from demonstration eligibility, as iskikely, these elements will
50




                                                     be used for eligibility determination. During the demonstration, control group members and those in the offset-only treatment groups will still be
                                                     eligible for TTW, and admin data will be gathered on their Ticket use.
Abt Associates Inc.




                                                Exhibit 4.1

                                                Uses of Administrative Data for the Quantitative Analyses

                                                Administrative Data Elements                                Source                     Purpose                Study Use
                                                APPLICANTS ONLY
                                                Predicted probability of eligibility for SSDI               Output of IBM Solutions
                                                                                                                                       Eligibility            Impact analysis
                                                                                                            Quick Decision tool
                                                SSDI application decisions and decision dates: Approvals    SSDI administrative data   SSDI eligibility and   Impact and cost
                                                and rejections; appeals; appeal reviews                                                benefit receipt        analyses
Chapter 4: From Data to Quantitative Analyses
51
Abt Associates Inc.




                                                 Exhibit 4.2

                                                 Uses of Survey Data for the Quantitative Analyses

                                                                                                                                     41
                                                 Survey Data Elements                                              Survey Sources                Purpose                    Study Use
                                                Personal characteristics
                                                Identifiers: Participant name and date of birth                    Baseline, interim, and        Serve as link among all    All analyses
                                                                                                                   follow-up surveys             data sources and link to
                                                                                                                                                 all analysis variables
                                                Demographics: Marital status, highest grade of education           Baseline survey, updated in   Baseline covariate data,   Impact analysis
                                                completed; race; ethnicity; sex; current living situation;         follow-up survey              outcome data (highest
                                                number of persons in household; name, DOB/age                                                    grade of education)
Chapter 4: From Data to Quantitative Analyses




                                                Disability status
                                                Functional limitations; description of disability (parts of body   Baseline survey, functional   Background information     Same
                                                affected, limitations of disability)                               limitations updated in        on disability; baseline
                                                                                                                   follow-up survey              covariate data; outcome:
                                                                                                                                                 change in functional
                                                                                                                                                 limitations over time
                                                Current receipt of SSDI                                            Baselineand follow-up         Outcome: knowledge of      Impact and cost
                                                                                                                   surveys                       current benefit receipt    analyses
                                                Employed at disability onset? If yes, type of job; monthly         Baseline survey               Covariates: employment     Impact analysis
                                                earnings; availability of benefits through employer; hours                                       history
                                                worked per week; did that job require the use of a
                                                computer?

                                                Income, earnings, and other benefit receipt
                                                Income and earnings: amounts and sources                           Baseline, interim, and        Outcomes: income and       Impact and cost
                                                                                                                   follow-up surveys             earnings                   analyses


                                                41
                                                     Survey sources specific to BOND are the baseline survey (administered prior to random assignment), the interim survey (administered at 12
52




                                                     months after RA), and follow-up survey (administered at 36 months after RA). All three surveys collect data only from beneficiaries who
                                                     undergo second-stage random assignment and from applicants who volunteer for BOND and undergo random assignment.
Abt Associates Inc.




                                                 Exhibit 4.2

                                                 Uses of Survey Data for the Quantitative Analyses

                                                                                                                               41
                                                 Survey Data Elements                                        Survey Sources                Purpose                  Study Use
                                                Medicare receipt                                             Same                          Outcome: coverage        Same
                                                Medicaid receipt                                             Same                          Same                     Same
                                                TANF and food stamps receipt                                 Baseline and follow-up        Outcomes: income and     Impact analysis
                                                                                                             surveys                       earnings
                                                Receipt of other government assistance: housing,             Same                          Same                     Same
                                                childcare, energy assistance, etc.
                                                Income and benefit receipt of household members              Same                          Same                     Same
                                                Applicants: Receipt of monthly cash benefit through BOND     Follow-up surveys             Same                     Same
Chapter 4: From Data to Quantitative Analyses




                                                (Early Intervention)

                                                Employment outcomes
                                                Work history: 12 months prior to random assignment; type     Baseline survey               Covariates: employment   Impact and cost
                                                of work done; earnings; hours worked per week; tenure                                      history                  analyses
                                                Applicants: When they last worked; receipt of paid           Baseline survey               Same                     Same
                                                earnings since disability onset; earnings since disability
                                                onset
                                                Work history for 36-months post-RA: Type(s) of               Follow-up survey              Outcomes: employment     Same
                                                employer(s); wages earned; start and end dates of job(s);                                  history
                                                average earnings
                                                Access to transportation: Whether participant has a valid
                                                                                                             Baseline and follow-up        Contextual variables     Impact and process
                                                drivers license, access to a car, access to public
                                                                                                             surveys                                                analyses
                                                transportation
                                                Current employment status: Whether employed, type of job,    Baseline, interim (current    Covariates (baseline)    Impact and cost
                                                current monthly earnings, availability of benefits through   employment status and         Outcomes: employment     analyses
                                                employer, hours worked per week; looking for work if not     earnings only), and follow-   status
                                                currently employed; satisfaction with current job;           up surveys
                                                accommodations at current job
53
Abt Associates Inc.




                                                 Exhibit 4.2

                                                 Uses of Survey Data for the Quantitative Analyses

                                                                                                                                 41
                                                 Survey Data Elements                                           Survey Sources           Purpose                    Study Use
                                                Return-to-work activities
                                                Employment services: Receipt of services and supports,          Interim survey           Service provision and      Impact and cost
                                                help from WSOs in obtaining services, reasons for using                                  variations across models   analyses
                                                services, services needed but not received, problems
                                                obtaining or using services
                                                Employment services: Receipt of One-Stop services,              Interim survey           Same
                                                receipt of enhanced employer-linked services from One-
                                                Stops
Chapter 4: From Data to Quantitative Analyses




                                                Education and training: Participant engagement in job           Baseline and follow-up   Employment and training    Impact and process
                                                training or education services to help obtain work; types       surveys                  outcomes                   analyses
                                                and duration of trainings; currently enrolled in school?
                                                Working toward a degree or certificate? In what area?
                                                Success in reaching employment goals                            Interim and follow-up    SSDI and demonstration     Impact and process
                                                                                                                surveys                  program benefits           analyses
                                                Barriers to employment
                                                Psychological impact of disability on ability to work: Does     Baseline and follow-up   Covariates (baseline)      Impact and process
                                                participant believe he/she can work; does he/she want to        surveys                  Outcomes: employment       analyses
                                                work; does participant have any job goals? Does he/she                                   barriers
                                                discuss goals with family or friends?
                                                Physical impact of delayed access to care: Participant’s        Same                     Same                       Same
                                                perception of health since onset; participant’s perception of
                                                physical limitations
                                                Lack of access to training; premature loss of benefits; loss    Same                     Same                       Same
                                                of ongoing employment supports; disability income cash
                                                cliff; job loss and difficulty of reinstatement; work-related
                                                overpayments
                                                Return-to-work job: Occupation; date job began; is              Follow-up survey         Outcomes: employment       Same
                                                employer involved in employment supports?
54
Abt Associates Inc.




                                                 Exhibit 4.2

                                                 Uses of Survey Data for the Quantitative Analyses

                                                                                                                                      41
                                                 Survey Data Elements                                                Survey Sources           Purpose                  Study Use
                                                Health status and functioning
                                                General health status; change in health since random                 Baseline and follow-up   Outcomes: health         Impact analysis
                                                assignment; use of mobility aids; difficulty seeing, hearing,        surveys                  outcomes
                                                or communicating; treatment for health condition
                                                Activities of daily living: Difficulty lifting objects; difficulty   Baseline and follow-up   Same                     Same
                                                walking up stairs or for long periods of time                        surveys
                                                Does participant experience difficulty getting around inside         Baseline and follow-up   Covariates: health       Impact analysis
                                                or outside house? Getting in and out of bed or a chair?              surveys                  Outcomes: health
Chapter 4: From Data to Quantitative Analyses




                                                Doing daily tasks? Have any of these activities been worse
                                                over the past 4 weeks?
                                                Mental health: Feeling nervous? Feeling hopeless?                    Baseline and follow-up   Same                     Same
                                                Feeling peaceful and calm? Feeling sad? Difficulty                   surveys
                                                sleeping? Did participant seek medical help for these
                                                feelings?
                                                 Health care coverage received through demonstration:                Interim and follow-up    Outcomes: SSDI and       Impact and process
                                                 Use of BOND insurance, prescription drug coverage, other            surveys                  BOND program benefits    analyses
                                                 features
                                                Private health care coverage                                         Baseline, interim, and   Covariates: health       Same
                                                                                                                     follow-up surveys        Outcomes: health
                                                Response to Other Features of BOND treatments
                                                Knowledge of benefit offset: whether has used offset                 Interim survey           SSDI and demonstration   Impact and process
                                                                                                                                              program benefits         analyses
                                                Knowledge and use of independence accounts                           Interim survey           Same                     Same
                                                Satisfaction with BON demonstration (all features)                   Interim survey           Same                     Same
                                                What factors influenced decision to apply for SSDI? Did              Baseline and follow-up   Same                     Same
                                                anyone help them in their decision? Was the application              surveys
                                                process difficult? Were the rules clear? What aspects
55




                                                were particularly challenging?
Abt Associates Inc.




                                                Exhibit 4.3

                                                Uses of Demonstration Site Operations Data for the Quantitative Analyses

                                                 Operations Data Element                                         Source                         Purpose                      Study Use
                                                Personal characteristics
                                                                                                                                        42
                                                Identifiers: Participant name, date of birth, and SSN;           BOND operations data           Serve as link among all      All analyses
                                                unique BOND identifier                                           (random assignment and         data sources and link to
                                                                                                                 participant intake)            all analysis variables
                                                Demographics: Marital status, highest grade of education
                                                completed; current living situation; number of persons in        BOND operations data           Baseline covariate data      Same
                                                household                                                        (participant intake)
Chapter 4: From Data to Quantitative Analyses




                                                Experimental status
                                                Beneficiary or applicant at entry, random assignment             BOND operations data           Output data from random
                                                group, whether first- or second-stage random assignment,         (random assignment and         assignment software,
                                                random assignment date.                                          participant tracking)          distinguishing and guiding   Same
                                                                                                                                                treatment for all BOND
                                                                                                                                                participants

                                                Disability status
                                                Type of primary disability; secondary disability, if any; date   BOND operations data (fill     Covariates: background       Same
                                                of onset; functional limitations; description of disability      in missing fields/update and   information on disability
                                                (parts of body affected, limitations of disability; changes in   enhance SSA admin data)
                                                functional limitations over time, etc.)
                                                Beneficiary status (monthly); benefit amount (monthly)           BOND operations data (IA       Outcomes                     Impact and cost
                                                                                                                 reporting)                                                  analyses




                                                42
                                                     BOND operations data applying to the entire demonstration population include the identifiers and experimental status variables. The remaining
56




                                                     BOND operations data elements apply only to members of treatment groups interacting with the Work Support Organizations and/or the One-
                                                     Stop Career Centers in the demonstration sites.
Abt Associates Inc.




                                                Exhibit 4.3

                                                Uses of Demonstration Site Operations Data for the Quantitative Analyses

                                                 Operations Data Element                                      Source                     Purpose                    Study Use
                                                Income, earnings, and other benefit receipt
                                                Earnings data from employment during demo period              BOND operations data (IA   Outcomes: income and       Impact and cost
                                                                                                              reporting)                 earnings                   analyses
                                                Medicare receipt: Whether beneficiary in given month,         Same                       Outcomes: coverage,        Same
                                                amount of premiums paid.                                                                 amount of benefits
                                                Medicaid receipt: Whether beneficiary in given month,         Same                       Same                       Same
                                                amount of premiums paid.
Chapter 4: From Data to Quantitative Analyses




                                                Employment outcomes
                                                Work history during BOND: whether working in given            BOND operations data (IA   Outcomes: employment       Impact and cost
                                                month; earnings; hours worked per week or month               reporting)                 history                    analyses
                                                Current employment status, if employed: current monthly       Same                       Same                       Same
                                                earnings, hours worked per week, FUTA and tax
                                                withholdings


                                                Return-to-work activities
                                                Employment services: Service providers; dates of service      BOND operations data;      Outcomes: service          Impact, process, and
                                                receipt; types of services received; completion of service;   state and local program    provision and variations   cost analyses
                                                elapsed time between random assignment and service            data                       across models
                                                completion; cost of each service; who paid; decision to use
                                                employment services; attitude toward services provided
                                                Education and training: Participant engagement in job         BOND operations data;      Outcomes: education and    Impact and process
                                                training or education services to help obtain work; types     state and local program    training                   analyses
                                                and duration of trainings; currently enrolled in school?      data
                                                Working toward a degree or certificate? In what area?
57
Abt Associates Inc.




                                                Exhibit 4.3

                                                Uses of Demonstration Site Operations Data for the Quantitative Analyses

                                                 Operations Data Element                                         Source                         Purpose                      Study Use
                                                Demonstration services delivery
                                                WSO client service utilization: number of services per           BOND operations data           Outcomes: participant        Participation analysis
                                                client, average hours of service, average duration                                              service receipt
                                                WSO clients with service plans completed: number,                Same                           Same                         Same
                                                characteristics, employment and SSDI status at completion
                                                WSO clients with service plans interrupted: number,              Same                           Same                         Same
                                                characteristics, reasons, employment and SSDI status at
                                                time of interruption
Chapter 4: From Data to Quantitative Analyses




                                                Service delivery patterns: frequency of most common              Same                           Same                         Same
                                                combinations of services
                                                One-Stop Career Centers: client contact, number and types        Same                           Outcomes: treatment          Impact and participation
                                                of services per client, average hours of service, average                                       contrasts                    analyses
                                                duration
                                                Health services provided through BOND43                          Same                           Same                         Same

                                                Demonstration services outcomes
                                                Total demonstration participants served (beneficiaries vs.       BOND operations data           Analysis of participants’    Participation analysis
                                                applicants), by provider                                                                        service receipt
                                                Percentage of participants with completed employment             Same                           Same                         Same
                                                plans (beneficiaries vs. applicants), by provider
                                                Service utilization rates, by type of service, by provider       Same                           Same                         Same
                                                Average length of service, by type of service, by provider       Same                           Same                         Same
58




                                                43
                                                     The nature of these services has not yet been determined. Specification of data elements will occur during the health benefits design task.
Abt Associates Inc.




                                                Exhibit 4-4

                                                Site-Level Contextual Variables

                                                Data Element                                                         Sources                  Purpose                   Study Use
                                                Site organizational environment
                                                VR agencies: caseloads, service availability               Published data             Description and analysis   Process and impact
                                                                                                                                      of program treatments      analyses
                                                ENs: number and range of organizations, level of Ticket    Ticket program manager     Same                       Same
                                                activity
                                                One-stops: overall number in site area, number with        Published data, staff      Same                       Same
                                                navigators and other disability-focused capacity           interviews
                                                Independent living centers: overall number in site area,   Published data, staff      Same                       Same
Chapter 4: From Data to Quantitative Analyses




                                                number with employment capacity                            interviews
                                                Number and types of BPAOs; use of other types of benefit   SSA, published data        Same                       Same
                                                counseling
                                                Number and types of P&A organizations in site area         Same                       Same                       Same
                                                Number and types of employer organizations in site area    Published data             Same                       Same

                                                SSA field and area office characteristics for site
                                                Number of claims representatives (CRs), average caseload   SSA administrative data    Description and analysis   Process and impact
                                                                                                                                      of program treatments      analyses
                                                Characteristics of CRs (tenure with SSA, education,        SSA administrative data,   Same                       Same
                                                training, previous experience)                             staff interviews
                                                Level of AWIC and WIL activity                             SSA administrative data    Same                       Same
                                                Staff dedicated to demonstration (number, levels,          Staff interviews           Same                       Same
                                                responsibilities, caseloads)

                                                Demonstration implementation in site
                                                Service providers involved in ES                           Interviews with project    Understand the site-       Process and impact
                                                                                                           managers, case managers,   specific implementation    analyses
                                                                                                           employment service         issues
59




                                                                                                           providers, SSA staff
Abt Associates Inc.




                                                Exhibit 4-4

                                                Site-Level Contextual Variables

                                                Data Element                                                     Sources          Purpose          Study Use
                                                Employers involved in ES                                  Same             Same             Same
                                                One-Stop Centers involved in ES                           Same             Same             Same
                                                ILCs involved in ES                                       Same             Same             Same
                                                BPAOs involved in demonstration                           Same             Same             Same
                                                P&A organizations involved in demonstration               Same             Same             Same
                                                Type of organization serving as the WSO lead agency       Same             Same             Same
                                                Number of One-Stops delivering enhanced employer-linked   Same             Same             Same
Chapter 4: From Data to Quantitative Analyses




                                                services
                                                Lead agency for early intervention outreach and intake    Same             Same             Same
                                                Health benefits package characteristics and delivery      Same             Same             Same
60
Chapter 5:               Presentation of Analytic Results

This chapter describes how we will present the analytic results obtained by applying the quantitative
research methods in the manner detailed in earlier chapters.



5.1     Results of the Impact Analysis
Reporting the Impact Analysis Results

For the purpose of reporting results, BOND will be regarded as three separate but integrated
experiments based on three different sample populations: (1) beneficiaries assigned at the first stage
of random assignment to receive either the 50 percent offset or current law benefits; (2) beneficiaries
who volunteer to receive the core benefit (including employment supports and comprehensive health
insurance) and who are randomly assigned at the second stage to receive either one of several
treatments or current law benefits; and (3) applicants randomly assigned either to one of several EI
treatments or to current law benefits. Because of the time necessary for ES program start-up and the
flow rate of individuals applying for SSDI benefits, the timing for completion of sample enrollment
for each of the three groups will be different, with the samples being completed in the order listed
above. By reporting on each sample separately, SSA will get results much more quickly than if
integrated reports were delivered (with the timing based on the sample that takes the longest to
complete). In addition, reports based on separate samples will be less diffuse and easier to assimilate.
However, because the three experiments are not truly independent and important lessons will be
learned across them, a final synthesis report will be produced that summarizes and integrates the
findings. Chapter 6 discusses the sequence and timing of these reports in more detail.

The results of the impact analysis will be presented in an ongoing series of letter reports (primarily
tables with sufficient narrative to explain the content of the tables and the general nature of the
findings) containing impact findings, in interim and final comprehensive reports for each of the three
populations sampled, and in a final synthesis report. The letter reports will take advantage of
administrative data that becomes available periodically between the formal reports. The interim
reports will include a description of the demonstration and the data and research methods we use; an
assessment of BOND implementation (a qualitative analysis not addressed in this report); a
participation analysis; and a presentation of initial findings from the impact study. These will be the
demonstration’s first comprehensive reports—that is, the first documents presenting impact results in
addition to implementation and participation findings.

The presentation of impact findings in the interim reports will be based on the interim survey data and
administrative records data. These reports will provide extensive tables of impact results, some of
which will be included in appendices, and an explanatory narrative. The presentation of results will
focus first on treatment-control differences in outcomes for the overall beneficiary, beneficiary
volunteer and EI applicant samples, as well as pertinent subgroups.




Abt Associates Inc.                         Chapter 5: Presentation of Analytic Results               61
With respect to the latter two samples, the presentations also will provide differential impact results—
that is, differences in outcomes between pairs of demonstration treatments. These results will
indicate, for example, the marginal effect of reducing the benefit reduction (―tax‖) by a specified
amount in calculating the cash benefits provided to eligible beneficiaries and applicants.

In addition, within the three overall samples, differences in impacts between groups defined by
diagnostic categories are expected to be noteworthy. We hypothesize that certain other subgroup
differences, such as age and time since disability onset, also will be important. In addition to findings
for key subgroups that will be included in the man body of the report, impact results for all
noteworthy subgroups will be presented in appendices.

Hypotheses regarding demonstration sites, based on demonstration program implementation
differences and/or variations in site policy and economic environments, are expected to lead to
analyses of differential impacts across groups of sites. However, substantial implementation
differences will be contingent on the way in which demonstration sites are selected and the
approaches the sites choose to follow. It the site selection process produces different types of
organizational partnerships and program approaches, the analysis of site differences will be more
important than it would otherwise be. While comparisons of impact results across individual sites are
often made in program evaluations, it is usually more instructive to compare results of groups of sites
defined by objective differences in treatment approach. Results for individual sites will not be
highlighted in the text of the report but will be presented in a report appendix.

The interim reports will also include a thorough discussion of important analytical issues. Thus, the
reports will bring a range of new information and issues to light. Feedback on these results and issues
will help inform the analysis and preparation of the final reports.

The final reports, which will present impact results in a similar fashion to the interim reports, will be
based on both the interim and follow-up surveys as well as additional administrative records data.
Because of the increased follow-up period available on sample members, the final report will be
better able to address the development of impacts over time. Past research suggests that this will be
an important issue in this evaluation. Also, the final reports will be in a better position to draw policy
conclusions from the results.

The reports will provide primary results, differential treatment results, subgroup results, and site
results, and also cover a wide range of outcomes. Consequently, we will present succinct summaries
of some of these many topics, and produce separate reports that provide the additional details and
discussion these topics deserve. The demonstration will plan to produce several topical reports (and
resources to prepare them will be budgeted), but the specific topics for these reports will be finalized
later based on discussion with SSA. The importance of some topics can be expected to rise to
prominence, based on economic and political developments during the course of the demonstration.
Also, the findings of other SSA demonstrations may underscore the importance of certain topics. For
example, evidence from the Mental Health Treatment demonstration may make it especially desirable
to devote a report to subgroup findings for sample members with mental health conditions.

Since there will be periodic administrative data on employment and earnings available between and
after the development of interim and final reports, letter reports will be provided to SSA frequently.



Abt Associates Inc.                          Chapter 5: Presentation of Analytic Results                 62
These reports will include impact tables along with simple narrative describing the most recent results
as well as comparisons to earlier impacts. These will not be polished products ready for formal
publication, but they will provide SSA with the ongoing, current findings that it can use for internal
analysis or for sharing with external parties, including Congressional staff, the media or the public
more generally.

Finally, there will be a synthesis report that draws together the findings from across the three sample
populations and extracts key information from the earlier reports and subsequent findings that have
emerged from additional administrative records. This document would be designed as the central
repository for the information that SSA will use to extract policy and program recommendations.

Presenting the Impact Analysis Results

As we have described elsewhere (Orr et al., March 2005), the demonstration impacts will be
estimated by comparing the mean outcomes of all sample members assigned to a given treatment (or
combination of treatments) with the mean outcomes of all sample members assigned to the
appropriate current law control group(s). This approach yields an estimate of the ―intent to treat‖
(ITT) impact—the impact on the entire group randomly assigned to treatment, whether they availed
themselves of the treatment or not. To obtain the impact on those who actually make use of the
employment supports or work above SGA and benefit from the offset (the groups of most interest for
policy), one would have to adjust the ITT estimate, using the adjustment developed by Bloom (1984).

In each of the deliverables on demonstration impacts, we will make use of standardized table shells
for presenting the impact results. Samples of these shells are found in Exhibits 5-1 to 5-4. The overall
format was developed by Abt Associates in prior impact evaluations and has proven effective for
displaying the contrasts of interest in various policy studies.

Exhibit 5-1 shows the simplest contrast, for the first-stage random assignment of eligible
beneficiaries. Comparing the T0 group (which receives a 50 percent offset only) with the C0 group, it
presents the impact estimates for three key outcomes: current monthly SSDI benefit, cumulative
benefits since RA, and proportion of the group currently on the SSDI roles. For each of the outcomes,
it shows estimated impacts for all members of the groups and also for subgroups defined on the basis
of whether or not the sample member’s primary disability is a mental condition. The first data column
shows the mean values for the controls. The next column gives the intent-to-treat effect estimate,
which is the difference in mean value for T0 in contrast to C0. Standard errors and statistical
significance level markings are provided to make it clear whether an impact has been detected. The
last column gives the treatment-on-treated effect estimates.




Abt Associates Inc.                         Chapter 5: Presentation of Analytic Results               63
                                     EXHIBIT 5-1
               KEY BOND SSDI STATUS OUTCOMES FROM ADMINISTRATIVE DATA
                                                              ELIGIBLE BENEFICIARIES
                                                                          50% Offset Only
                                                    Control                  vs. Control
                                                                           a                 a
                                                     Mean              ITT               TOT
         Current Monthly Benefit (n=          )
          All                                         $nnn
                                                    $nnn                                     $nnn
                                                      (nnn)                                  (nnn)
          Primary disabling condition is  $nnn        $nnn                                   $nnn
          mental                                      (nnn)                                  (nnn)
          Primary disabling condition is  $nnn        $nnn                                   $nnn
          not mental                                  (nnn)                                  (nnn)
         Cumulative Benefit Payments Since Random Assignment (n=                       )
          All                             $nnn        $nnn                                   $nnn
                                                      (nnn)                                  (nnn)
          Primary disabling condition is  $nnn        $nnn                                   $nnn
          mental                                      (nnn)                                  (nnn)
          Primary disabling condition is  $nnn        $nnn                                   $nnn
          not mental                                  (nnn)                                  (nnn)
         Current Beneficiary Status (n= )
                                          0.nn         0.nn                                   0.nn
          All
                                                      (nnn)                                  (nnn)
          Primary disabling condition is  0.nn         0.nn                                   0.nn
          mental                                      (nnn)                                  (nnn)
          Primary disabling condition is  0.nn         0.nn                                   0.nn
          not mental                                  (nnn)                                  (nnn)
        * = p<.05 on t-test. Robust standard errors are shown in parentheses.
        Sources:
        Sample:
        Notes: a) ITT = Intent-to-Treat; TOT = Treatment-on-Treated. See Appendix X for details.
                b) Control means and impact estimates are regression-adjusted with robust standard errors.



As the demonstration continues and second-stage random assignment is conducted, data will begin to
accumulate on the beneficiary volunteers and how they differ in characteristics from the eligible
beneficiaries in groups T0 and C0. (The beneficiary volunteers include all the second-stage RA
groups of beneficiaries shown in Exhibit 3-3.) Later on, we will be able to contrast their outcomes, as
shown in Exhibit 5-2. The exhibit shows impact estimates for the same three outcome measures, but it
is different from the prior shell in two ways. First, there are different control groups for the eligible
beneficiary sample and the beneficiary volunteer sample, so separate control means are provided (in
data columns 1 and 4). Second, subgroup results are presented based on whether or not the sample
member was employed at baseline, because we anticipate that employed beneficiaries will be over-
represented among the volunteers and therefore the baseline employment rate for volunteers will
differ from that of the overall eligible population.




Abt Associates Inc.                               Chapter 5: Presentation of Analytic Results                64
                                      EXHIBIT 5-2
                KEY BOND SSDI STATUS OUTCOMES FROM ADMINISTRATIVE DATA
                              ELIGIBLE BENEFICIARIES           BENEFICIARY VOLUNTEERS
                                         50% Offset Only                    50% Offset Only
                            Control         vs. Control        Control        vs. Control
                                              a           a                    a          a
                             Mean         ITT        TOT        Mean       ITT       TOT
  Current Monthly Benefit (n=Y eligible beneficiaries, Z beneficiary volunteers)
   All                       $nnn        $nnn         $nnn      $nnn       $nnn      $nnn
                                         (nnn)       (nnn)                (nnn)      (nnn)
   Employed at baseline      $nnn        $nnn         $nnn      $nnn      -$nnn      $nnn
                                         (nnn)       (nnn)                (nnn)      (nnn)
   Not employed at           $nnn        $nnn         $nnn      $nnn       $nnn      $nnn
   baseline                              (nnn)       (nnn)                (nnn)      (nnn)
  Cumulative Benefit Payments Since Random Assignment
                            (n=Y eligible beneficiaries, Z beneficiary volunteers)
    All                             $nnn  $nnn        $nnn       $nnn       $nnn                       $nnn
                                          (nnn)       (nnn)                (nnn)                       (nnn)
   Employed at baseline       $nnn        $nnn        $nnn       $nnn      -$nnn                       $nnn
                                          (nnn)       (nnn)                (nnn)                       (nnn)
   Not employed at            $nnn        $nnn        $nnn       $nnn       $nnn                       $nnn
   baseline                               (nnn)       (nnn)                (nnn)                       (nnn)
  Current Beneficiary Status (n= Y eligible beneficiaries, Z beneficiary volunteers)
                              0.nn         0.nn        0.nn      0.nn       0.nn                        0.nn
   All
                                          (nnn)       (nnn)                (nnn)                       (nnn)
                              0.nn         0.nn        0.nn      0.nn       0.nn                        0.nn
   Employed at baseline
                                          (nnn)       (nnn)                (nnn)                       (nnn)
   Not employed at            0.nn         0.nn        0.nn      0.nn       0.nn                        0.nn
   baseline                               (nnn)       (nnn)                (nnn)                       (nnn)
  * = p<.05 on t-test. Robust standard errors are shown in parentheses.
  Sources:
  Sample:
  Notes: a) ITT = Intent-to-Treat; TOT = Treatment-on-Treated. See Appendix X for details.
          b) Control means and impact estimates are regression-adjusted with robust standard errors.


As the demonstration continues and data are collected in the interim and follow-up surveys, other
outcome measures can be tested for impacts. Exhibit 5-3 illustrates the presentation of impact
estimates for some key outcomes measured by survey data. The results are for different treatments
among the second-stage beneficiary groups, both compared to the same control group. They would be
generated from pooled impact regressions (see Exhibit 3-1). Again, we show the control mean in the
first data column and then the ITT and TOT impact estimates for the different treatments. But in this
table we also show the explicit double differences: how the impacts estimated for these treatment
groups—which differ in offset rate but not in other treatment features—compare to each other. The
differences between estimated impacts, and the significance levels of those differences, are shown in
the right-most two data columns.




Abt Associates Inc.                             Chapter 5: Presentation of Analytic Results                    65
                                       EXHIBIT 5-3
                      KEY BOND EMPLOYMENT OUTCOMES FROM SURVEY DATA
                                                  BENEFICIARY VOLUNTEERS
                                       50% Offset +                25% Offset +                 50% Offset +
                                          Core                        Core                        Core vs.
                                        Benefit vs.                 Benefit vs.                 25% Offset +
                                         Control                     Control                       Core
                       Control
                                          a             a               a           a                a         a
                       Mean    ITT                TOT             ITT         TOT              ITT       TOT
  Total Earnings Last Month (n= )
                    $nnn      $nnn                $nnn            $nnn        $nnn             $nnn      $nnn
   All
                              (nnn)               (nnn)          (nnn)        (nnn)           (nnn)      (nnn)
   Employed at      $nnn      $nnn                $nnn           -$nnn        $nnn            -$nnn      $nnn
   baseline                   (nnn)               (nnn)          (nnn)        (nnn)           (nnn)      (nnn)
   Not employed     $nnn      $nnn                $nnn            $nnn        $nnn             $nnn      $nnn
   at baseline                (nnn)               (nnn)          (nnn)        (nnn)           (nnn)      (nnn)
  Total Income Last Month (n= )
                    $nnn      $nnn                $nnn           $nnn         $nnn            $nnn       $nnn
   All
                              (nnn)               (nnn)          (nnn)        (nnn)           (nnn)      (nnn)
   Employed at      $nnn      $nnn                $nnn           $nnn         $nnn            $nnn       $nnn
   baseline                   (nnn)               (nnn)          (nnn)        (nnn)           (nnn)      (nnn)
   Not employed     $nnn      $nnn                $nnn           $nnn         $nnn            $nnn       $nnn
   at baseline                (nnn)               (nnn)          (nnn)        (nnn)           (nnn)      (nnn)
  Current Employment Status (n= )
                     0.nn      0.nn                0.nn           0.nn         0.nn            0.nn       0.nn
   All
                              (nnn)               (nnn)          (nnn)        (nnn)           (nnn)      (nnn)
   Employed at       0.nn      0.nn                0.nn           0.nn         0.nn            0.nn       0.nn
   baseline                   (nnn)               (nnn)          (nnn)        (nnn)           (nnn)      (nnn)
   Not employed      0.nn      0.nn                0.nn           0.nn         0.nn            0.nn       0.nn
   at baseline                (nnn)               (nnn)          (nnn)        (nnn)           (nnn)      (nnn)
  * = p<.05 on t-test. Robust standard errors are shown in parentheses.
  Sources:
  Sample:
  Notes: a) ITT = Intent-to-Treat; TOT = Treatment-on-Treated. See Appendix X for details.
          b) Control means and impact estimates are regression-adjusted with robust standard errors.


The last of the three experiments to have full data ready for analysis will be the early intervention
study of SSDI applicants recruited into the demonstration. As discussed further in Chapter 6,
applicant recruitment will begin at the same time as second-stage RA for beneficiaries, but it is likely
to take longer, extending over a two-year period (to accumulate sufficient sample). The interim and
follow-up surveys are timed relative to RA, so the end of the 36-month follow-up survey data
collection on applicants will certainly take place more than five years after recruitment begins.

However, the presentation of results for the applicant sample will be fairly similar to that for the
beneficiary volunteers. To present impact estimates for the EI study, we will use tables similar to the




Abt Associates Inc.                              Chapter 5: Presentation of Analytic Results                       66
one shown in Exhibit 5-4, which is based on survey data and addresses impacts separately for
applicants in the regular vs. enhanced One-Stop areas.

                                       EXHIBIT 5-4
                      KEY BOND EMPLOYMENT OUTCOMES FROM SURVEY DATA
                             APPLICANT VOLUNTEERS,                           APPLICANT VOLUNTEERS,
                              Regular One-Stop Areas                         Enhanced One-Stop Areas
                                      High ES Incentive                             High ES Incentive
                            Control       vs. Control                       Control     vs. Control
                                            a           a                                a          a
  Outcome                    Mean       ITT       TOT                        Mean    ITT       TOT
  Total Earnings Last Month (n= )
   All                       $nnn      $nnn        $nnn                      $nnn         $nnn    $nnn
                                       (nnn)       (nnn)                                 (nnn)    (nnn)
   Employed at baseline      $nnn      $nnn        $nnn                      $nnn        -$nnn    $nnn
                                       (nnn)       (nnn)                                 (nnn)    (nnn)
   Not employed at           $nnn      $nnn        $nnn                      $nnn         $nnn    $nnn
   baseline                            (nnn)       (nnn)                                 (nnn)    (nnn)
  Total Income Last Month (n= )
   All                       $nnn      $nnn        $nnn                      $nnn         $nnn    $nnn
                                       (nnn)       (nnn)                                 (nnn)    (nnn)
   Employed at baseline      $nnn      $nnn        $nnn                      $nnn        -$nnn    $nnn
                                       (nnn)       (nnn)                                 (nnn)    (nnn)
   Not employed at           $nnn      $nnn        $nnn                      $nnn         $nnn    $nnn
   baseline                            (nnn)       (nnn)                                 (nnn)    (nnn)
  Current Employment Status (n= )
                             0.nn       0.nn        0.nn                      0.nn        0.nn     0.nn
   All
                                       (nnn)       (nnn)                                 (nnn)    (nnn)
                             0.nn       0.nn        0.nn                      0.nn        0.nn     0.nn
   Employed at baseline
                                       (nnn)       (nnn)                                 (nnn)    (nnn)
   Not employed at           0.nn       0.nn        0.nn                      0.nn        0.nn     0.nn
   baseline                            (nnn)       (nnn)                                 (nnn)    (nnn)
  * = p<.05 on t-test. Robust standard errors are shown in parentheses.
  Sources:
  Sample:
  Notes: a) ITT = Intent-to-Treat; TOT = Treatment-on-Treated. See Appendix X for details.
     b) Control means and impact estimates are regression-adjusted with robust standard errors.




Each panel of Exhibit 5-4 is for one of those contrasts; within each panel, the first data column gives
the control mean and the next two give the ITT and TOT impact estimates (with standard errors and
significance markings). The control means are for the respective—separate—control groups; see
Exhibit 3-2 for reference. The impacts are for the same treatment (high ES incentives) but for
different geographical areas. Similar tables will be used to present the contrasts for different
treatments (high vs. low ES incentives) in the same geographical areas.

We will probably also use other means of presenting the impact findings in our evaluation reports to
SSA, such as simplified impact summaries (covering a large number of contrasts but just showing
statistical significance) and graphics. However, the details of all impact findings will be provided
through table shells like the ones just discussed, whether in text or in report appendices.



Abt Associates Inc.                             Chapter 5: Presentation of Analytic Results               67
5.2     Results of the Benefit-Cost Analysis

The results of the cost-effectiveness analysis will be presented alongside the findings from the impact
and participation analyses. In the comprehensive interim report, the results will be limited to short-
term intervention costs and benefits, because it will be too early to extrapolate estimated impacts
beyond what has been observed. Later, however, the final report will present a full cost-effectiveness
assessment of the interventions.

Several objectives will be pursued in presenting the cost-effectiveness results in both of these reports.
First, the links between these results and the findings of the impact and participation analyses, as well
key assumptions and estimates, will be entirely transparent.

Second, a carefully planned series of sensitivity tests will play a central role in the presentation of the
cost-effectiveness results. Many of these tests will identify the implications of analytical
uncertainties, such as error in our impact estimates and in our projections of future effects or the lack
of certainty regarding the dollar values we assign to certain intervention effects. We will simplify the
findings of such tests by computing upper- and lower-bound estimates of the interventions’ net
present value, based on the most optimistic and pessimistic sets of alternative assumptions.

In the evaluation’s final report, the sensitivity tests also will address specific policy questions raised
by SSA. For example, some questions would ask how the benefit-cost results would be expected to
change if interventions were modified in specified ways. Others might ask how the results would
change if the policy environment in which the interventions were implemented changed, e.g. if SSDI
benefits or income tax rates were raised substantially. Such questions might reflect policy
development during the course of the demonstration or important policy proposals made around the
time of the evaluation’s conclusion.

Third, the analysis results, which will be voluminous, will be organized so that evaluation users can
easily find what they need. The text of each comprehensive report will briefly explain the research
methodology, while providing additional discussion and technical details in a specified appendix.
The text also will discuss the results for the costs, effects, and net present values of the various
interventions, as well as the sensitivity tests. Separate appendices will provide additional details on
intervention costs and the estimated dollar value of intervention impacts.




Abt Associates Inc.                          Chapter 5: Presentation of Analytic Results                  68
Chapter 6:                Timeline for the Treatments and the
                          Evaluation

In this chapter, we describe two related processes: (1) the overall timeline for the treatments including
the key milestones in their development, implementation, and phase-down, as well as how the
impacts are likely to emerge; and (2) the overall timeline for the evaluation. Clearly these two
processes are interrelated and assumptions regarding the former have important implications for the
latter. The goal of setting out these potential timelines and describing their key features is to allow
SSA to make important decisions regarding the best approach to implementing BOND, including the
appropriate level of resources and their allocation within the project.



6.1      Expected Time Path of the Treatments

In this section we describe the treatment time path, with reference to the main tasks and milestones of
the demonstration implementation. Detailed implementation planning is beyond the scope of the
present design contract—yet, for the treatments to take effect, the full design needs to be
implemented.

We have therefore described an implementation timeline, although the details remain to be worked
out (and of course the definition of tasks is up to SSA). To do so, we have had to make two key
assumptions: that SSA will award the implementation contract by March 1, 2007; and that it will be
possible to begin detailed implementation planning and preparation earlier, in October 2006. The
necessity of doing so will become apparent as the timeline unfolds.

We have laid out the time path of the treatments with several objectives in mind:

            To begin first-stage random assignment (RA) and the $1-for-$2-only treatment before the
             end of calendar 2007;
            To phase in site operations—so that the fastest-developing sites can go on-line and begin
             second-stage RA by March 2008, and so that all sites are running by July 2008;
            To provide all sites with a start-up period, during which we will monitor the fidelity and
             levels of services until they reach predetermined benchmarks of ―full service level;‖44 and
            To support carrying out data collection and analysis on three separate tracks (first-stage
             RA, second-stage beneficiary assignment, and all applicant RA), in order to get results to
             SSA as early as possible.



44
     This is the point at which the WSO and One-Stops have sufficient capacity to handle the full flow of
     beneficiaries and applicants from second-stage random assignment, with timely and high-quality service
     provision.




Abt Associates Inc.            Chapter 6: Timeline for the Treatments and the Evaluation                      69
This section first describes the implementation timeline and then summarizes the treatment time path.

Implementation Timeline

Components of the preparatory work for BOND implementation include the following:

        1. Operationalizing the program design;
        2. Developing the BOND database environment/data warehouse;
        3. Developing the BOND benefit calculator;
        4. Public information and outreach; and
        5. Demonstration site preparation.


The first four components can and should occur simultaneously. They will need to begin before
March 1, 2007 (the assumed date of implementation contract award). The last of these components,
site preparation, follows later, after recruitment of the WIBs and WSOs. We describe this sequence
in further detail at the end of the section.

Operationalizing the Program Design
As shown in Exhibit 6-1, the initial steps in detailed implementation planning involve developing the
specifications for the roles to be played in BOND by the Workforce Investment Boards and One-Stop
Career Centers. This is the system through which the enhanced employer-linked services will be
offered to demonstration participants in certain treatment groups. Within the framework of an
agreement between SSA and DOL, and in consultation with both agencies, we will develop a
description of services, performance benchmarks, and estimates of caseload and schedule. We will
then develop recruitment materials for the WIBs and for them to use in attracting One-Stops to this
project.

We assume that the 10 BOND sites (defined as SSA Area Office jurisdictions) will have been
selected (and approved by SSA) before the end of September 2006. The Employment Support (ES)
design calls for randomly selected WIBs within each demonstration site to deliver the enhanced
services. This random draw of WIBs will be done in December 2006, with contacting and recruitment
during the following two months. The goal is to have participating WIBs identified when WSO
solicitation begins.

While the WIB/One-Stop specifications are being developed, a parallel process will take place for the
WSOs. Recruitment materials and site visits will be used to interest potential WSO lead agencies and
partners in the demonstration; an RFP or RFQ process will be used to select or establish the WSOs in
the sites. By June 2007, the WIBs and WSOs should be known, local health sources should be
identified, and operating procedures and manuals should be drafted.

The third parallel procurement process is for the Fiscal Agent that will house the Independence
Accounts and manage flows of funds between SSA and the demonstration participants in the
appropriate treatment groups. We have set the procurement date at September 30, 2007, with the key
milestone being the start-up of the first site activities in March 2008.



Abt Associates Inc.         Chapter 6: Timeline for the Treatments and the Evaluation              70
Exhibit 6-1. Timeline: Operationalizing the Program Design
Start       End                                                     Tasks
                                                                                    a
10/2006     11/2006          Develop detailed WIB/One-Stop service specifications
11/2006     12/2006          Prepare WIB recruitment materials
            12/2006          Make random draw of WIBs in BOND sites
                                                                                                              b
1/2007      2/2007           Contact/visit/recruit sampled WIBs; replace WIBs unable/unwilling to participate
10/2006     11/2006          Develop detailed WSO specifications
            12/2006          Prepare WSO RFP/RFQ (includes identifying participating WIBs)
11/2006     12/2006          Prepare WSO recruitment materials, visit sites
            12/2006          Release advance announcement of WSO procurement with recruitment materials
1/2007      2/2007           Issue RFP/RFQ and communicate with potential bidders
1/2007      3/2007           Develop detailed Fiscal Agent specifications and identify potential offerors
4/2007      5/2007           Prepare Fiscal Agent RFP/RFQ
            4/30/3007        Receive WSO proposals (due date)
5/2007      6/2007           Select winning WSO bidders
6/2007      7/2007           Issue Fiscal Agent RFP/RFQ and communicate with potential offerors
1/2007      6/2007           Develop WSO operating procedures and manuals
1/2007      6/2007           Identify local health benefits sources
            7/31/2007        Receive Fiscal Agent proposals (due date)
8/2007      9/2007           Select winning Fiscal Agent offeror
a)   Assumes that a Memorandum of Understanding has been put in place between SSA and DOL to govern
     funding WIBs to carry out certain functions in the demonstration, with details to be specified through the
     implementation planning process.
b)   Replacements will be selected randomly from remaining Workforce Investment Act areas in the sites.



Developing the BOND Database Environment and Data Warehouse
The steps involved in developing the full-scale version of the BOND database environment and data
warehouse are shown in Exhibit 6-2. This system will offer online operational capabilities for the
different types of local agencies that will be part of the demonstration. In addition, it will handle the
SSA administrative data from which first-stage random assignment will take place, as well as storing
and providing access to all the data gathered during the demonstration.

From the point where system requirements are fully known, it should take about 10 months to build
the production system, including tutorials and Help facilities. Both project staff and on-site personnel
will need to be trained and supported in system use. The key milestone for this development process
is to be able to carry out first-stage random assignment starting in September 2007. (We assume that
project rather than site staff. will conduct this process.)




Abt Associates Inc.              Chapter 6: Timeline for the Treatments and the Evaluation                        71
Exhibit 6-2. Timeline: Develop the BOND Data Warehouse
 Start      End                                            Tasks
 10/2006    11/2006    Complete design of full database environment/data warehouse
 11/2006    8/2007     Build and test production database environment/data warehouse
 11/2006    8/2007     Develop training materials, tutorials, Help facilities
 8/2007     9/2007     Deliver training to project staff
 10/2007    2/2008     Deliver training to BOND on-site personnel



Developing the BOND Benefit Calculator
The BOND benefit calculator is the tool that the Work Support Specialists will use with
demonstration participants to explain the treatment features and help the participants make choices for
themselves based on BOND’s incentives and features. We also anticipate that a simpler version of
the calculator will be provided to the BPAOs and other organizations that may be called upon by
beneficiaries randomly assigned to the $1-for-$2-only treatment in the first stage.

Exhibit 6-3 shows the tasks involved in turning the prototype benefit calculator built in the design
phase into a fully operational software application, with appropriate tutorials and Help facilities. We
show this process on the same timeline as the BOND data warehouse. But for the calculator to be
useful for helping beneficiaries in the first-stage RA, the key milestone is the start of that random
assignment in September 2007.


Exhibit 6-3. Timeline: Develop the BOND Benefit Calculator
 Start      End                                 Tasks
 10/2006    11/2006    Complete design of benefit calculator
 11/2006    8/2008     Build and test production version of calculator
 11/2006    8/2008     Develop training materials, tutorials, Help facilities
 8/2007     9/2007     Deliver training to project staff and BPAOs
 10/2007    2/2008     Deliver training to BOND on-site personnel



Public Information and Outreach
There are a number of different parts of the demonstration preparation that involve planning and
materials preparation for public information and outreach. We see at least three different audiences to
address:

        1. The general public and the broad set of individuals and organizations interacting with the
           SSDI program and population;
        2. The beneficiaries assigned to treatment in first-stage RA and the individuals and
           organizations interacting with them (notably the SSA field offices and BPAOs); and
        3. The beneficiaries and applicants who make up the ES target population—those who need
           to be attracted to the demonstration and volunteer for it.




Abt Associates Inc.          Chapter 6: Timeline for the Treatments and the Evaluation                72
Exhibit 6-4 shows the tasks involved and a likely timeline for them. All the materials to explain the
demonstration to SSA staff, beneficiaries, and BPAOs need to be ready and distributed before first-
stage RA begins in September 2007. The outreach campaign for the beneficiary ES target group and
the likely eligible applicants must be ready before the first site begins second-stage RA, in March
2008.

Exhibit 6-4. Timeline: Public Information and Outreach Preparation
 Start      End                                        Tasks
 10/2006    1/2007     Plan public information campaign
 2/2007     4/2007     Develop public information materials
 10/2006    12/2006    Plan information campaign for beneficiaries assigned in first stage
 1/2007     3/2007     Develop written materials for beneficiaries assigned in first stage
 1/2007     5/2007     Develop written materials and training for SSA staff
 1/2007     5/2007     Develop written materials and training for BPAO staff
 1/2007     6/2007     Plan outreach campaign for potential BOND volunteers (B & A)
 7/2007     12/2007    Develop outreach materials for potential BOND volunteers (B & A)



Demonstration Site Preparation
The final group of preparatory tasks for the BOND implementation is site preparation, which follows
after the tasks to operationalize the program design (shown previously in Exhibit 6-1). The site prep
tasks encompass refinement of operations procedures and manuals—with input from the
organizations that will implement the demonstration—as well as monitoring and support as the WSOs
and One-Stops hire staff and otherwise get ready for start-up. We see initial site staff training as
occurring between January and June 2008 (depending on site readiness); it should also be anticipated
that they will need further training as a result of operational problems or staff turnover.

Exhibit 6-5 lists these tasks and their time frames, including ongoing technical assistance and training
over the full length of the demonstration period. What determines the length of that period is a
combination of treatment and evaluation factors, which are described in Section 6.3 below.

Exhibit 6-5. Timeline: BOND Site Preparation
 Start      End                                         Tasks
                       Refine/complete WSO & One-Stop operations manuals (input from
 7/2007     9/2007        WSOs & One-Stops)
 8/2007     6/2008     Monitor and support WSO and One-Stop hiring and other preparation
 1/2008     6/2008     Train WSO staff
 1/2008     6/2008     Train One-Stop staff
 1/2008     6/2008     Work with local health providers, as needed
 1/2008     1/2015     Provide ongoing TA to WSOs and One-Stops


BOND Treatment Timeline

We can now summarize the timeline for development of the experiment’s treatments based on the
implementation timeline described above. Exhibit 6-6 shows the full implementation timeline for
BOND and the time path for the treatments.


Abt Associates Inc.          Chapter 6: Timeline for the Treatments and the Evaluation                73
                                        Exhibit 6-6
           Implementation Timeline for the Benefit Offset National Demonstration
                                            a
                      Tasks and Events                                     Start Month               End Month
   Assumed award date for implementation contract                      3/1/2007 (Month 1)
                                                                                 b
Operationalize the program design                                             -6                           6
Issue WSO RFP/RFQ                                                               -2
WSO proposals due                                                               +1
Select WSO winners                                                              +3
Issue Fiscal Agent RFP/RFQ                                                      +3
Select Fiscal Agent winner                                                      +6
Develop the BOND data warehouse                                                 -6                        12
Develop the BOND benefit calculator                                              -6                       12
Develop the demonstration sites                                                  -6                       29
Site selection confirmed                                                         -6
Demonstration site preparation                                                   5                        17
Site staff training                                                             10                        17
ES activities begin in fastest developing sites                                 13
ES monitoring and measurement                                                   13                        29
ES activities underway in all sites                                             17
First site reaches full service level                                           19
Last site reaches full service level                                            29
Regular demonstration monitoring & TA                                           13                        89
Public information campaign about BOND                                          -5                        17
Public info and outreach planning                                                -5                       2
Public information campaign                                                      3                        17
Outreach campaign for potential BOND volunteers                                  -2                       53
Plan info & outreach campaign                                                    -2                       10
Conduct outreach for ES volunteers                                              11                        53
BOND sample enrollment and build-up                                             7                         53
First-stage random assignment (beneficiaries)                                   7                         10
Baseline surveys and second-stage RA (beneficiaries)                            13                        36
Baseline surveys and RA of applicants (EI)                                      13                        53
Employment support activities at full service level                             29                        89
Running in all sites for at least 1 year                                        41
Running in all sites for at least 2 years                                       53
Running in all sites for at least 3 years                                       65
Running in all sites for at least 4 years                                       77
Running in all sites for at least 5 years                                       89
Site Transition and Shutdown                                                    79                        95
Transition & shut-down activities begin in earliest site(s)                     79
Transition & shut-down activities begin in latest site(s)                       89
All transitions completed, all site activities shut down                                             95 (2/2015)
a)   Includes only selected events from Exhibits 6-1 through 6-5.
b)   Negative month values refer to the period before March 1, 2007; these values all fall between September 2006 and
     February 2007.




Abt Associates Inc.                Chapter 6: Timeline for the Treatments and the Evaluation                            74
First-Stage Random Assignment
If the assumptions hold true regarding a March 1, 2007 start date for the implementation contract and
regarding our ability to start preparatory tasks in October 2006, it should be possible to begin the
first-stage random assignment process in the 10 demonstration sites in September 2007. Because of
the large numbers involved (200,000 treatment group members nationwide), we suggest phasing the
implementation of the assignments across a four-month period. If the required notice period for a
change in benefit schedule is 30 days, implementation of the $1-for-$2-only treatment could begin in
October 2007.45 Prior to this time, SSA field office personnel and BPAO staff will have been
informed about the demonstration and will have received training to assist beneficiaries who inquire
about the benefit schedule change. Thus, based on this sequence of implementation activities, the
demonstration will actually complete first-stage random assignment and the $-1-for-$2-only treatment
before the end of calendar 2007.

Second-Stage Random Assignment
The second-stage RA for BOND cannot begin until both the site agencies and the Fiscal Agent are
ready to provide services to participants. That readiness depends on:

             Phasing the preparation of written and visual materials and training for WSO and One-
              Stop staff and then for potential BOND volunteers (both beneficiaries and applicants);
             Selecting the WSOs in mid-2007 and assisting them to prepare for start-up;
             Providing training to project and site staff on procedures and operations, on the BOND
              benefit calculator, and on the BOND data system; and
             Having the Fiscal Agent in place with the infrastructure established for the Independence
              Accounts.


Realistically, SSA should expect to phase in site operations, so that the fastest-developing sites can go
on-line (and begin second-stage RA) by March 2008 and so that all sites are running by month 16
(July 2008). As mentioned above, we think it is critical to allow for a start-up or pilot period, during
which the sites will work with the first parts of the second-stage treatment groups. The timeline shows
a six-month period for monitoring the fidelity and levels of services in each site—and assisting with
improvements—before they demonstrate sufficient capacity and quality to handle the full flow of
beneficiaries and applicants from second-stage random assignment with timely service provision.
Based on this scenario, the last site will reach the full-service level by month 28 (July 2009). Section
6.3 below discusses the handling of participant data from the pilot period. Including the pilot months,
the random assignment of volunteers from among the current beneficiaries will span a 12-month
period and should be complete by no later than month 34.

Although all beneficiaries in the second-stage treatment groups should be enrolled and receiving
services by month 35 or 36, the recruitment and intake for applicants will continue for a longer
period. Indeed, we have anticipated from the beginning of the design process that it will take two
years (post-pilot period) to accumulate sufficient sample for the Early Intervention portion of this

45
     If the required notice period is longer, it may be possible to perform the assignment and begin mailing
     notices in August 2007.




Abt Associates Inc.             Chapter 6: Timeline for the Treatments and the Evaluation                      75
study. Outreach to applicants will begin in January 2008 and will continue even as the initial random
assignments occur two months later. From the point of full service in each site, it will take 24 months
for the applicant sample to accumulate. Thus, we anticipate recruitment and random assignment of
applicants will not be complete until month 42.

In sum, the timeline shown in Exhibit 6-6 provides the implementation framework for the treatments
and lays out their time paths. In the next sections of this chapter, we address the time path of possible
impacts and then discuss how to carry out data collection and analysis on three separate tracks (first-
stage RA, beneficiary assignment, and all applicant RA), in order to get results to SSA as early as
possible.



6.2     Expected Time Path of the Impacts

Since there is not a substantial body of research that provides rigorous information on expected
impacts for applicants for and beneficiaries of SSDI, assumptions about patterns of BOND impacts
and the timeframe over which they will occur can be based on indirect evidence only. The largest and
most systematically synthesized research on the impacts of employment and training programs is
based on welfare populations. This research strongly suggests that on average impacts emerge shortly
after enrollment in the program, peak at about three years, and decline to zero by about six years.

However, there are important differences between the SSDI population and the welfare population,
including, for example, those based on age, education, work experience, degree and nature of
employment barriers, and likelihood of caring for minor children. These are reflected in prior
employment experience for SSDI recipients being much more substantial, and post-random
assignment employment experience for welfare recipients being much higher. Thus, a great deal of
control group ―catch-up‖ occurs for welfare recipients, and this does not seem to emerge for SSDI
applicants and beneficiaries, which could lead to more persistent impacts in BOND. In addition,
there are important program differences between BOND and the welfare-to-work programs, including
that the former is voluntary and the latter are typically mandatory (including all those covered in the
Greenberg synthesis (Greenberg et al., 2003).

Although the differences described above are important in creating caution about directly applying
welfare findings to BOND, there is the important similarity that both populations are adults. Thus,
for an adolescent population where the primary programmatic intervention is education or training,
one would not expect to see employment and earnings impacts to emerge until the youth have reached
young adulthood and have completed their education. Furthermore, we expect that the BOND
financial incentives, employment and medical supports will serve to connect applicants and
beneficiaries to employment by overcoming financial and other barriers, rather than providing longer-
term training.

Taking these considerations all together, we expect that BOND employment and earnings effects will
begin to emerge within the first several quarters after random assignment and will be substantial by
the three-year mark. Thus, we do not expect to find ―sleeper effects,‖ in which an extended period of
no impacts is observed followed by a period of significant impacts. We do not have clear




Abt Associates Inc.           Chapter 6: Timeline for the Treatments and the Evaluation                76
expectations about how long impacts will persist, and thus we have laid the groundwork for long-term
follow-up.



6.3     Expected Time Path of the Evaluation

In this section, we describe the overall timeframe for the evaluation as well as key milestones that
will be achieved in the course of the demonstration. Of course, there are numerous assumptions on
which this timeline depends, and not all these assumptions will be proven accurate in the course of
implementing the project. Furthermore, although we have set forth a concrete example rather than
simply a large number of hypothetical scenarios, we have done so in order to illustrate how the
various activities of BOND interact based on a plausible set of assumptions. Our purpose is not to
provide a recommended set of assumptions but to clarify the consequences of decisions and to enter
into a discussion with SSA, in order to establish the most plausible assumptions and their implications
for the availability of findings to SSA. These discussions will also help to inform decisions about
how best to allocate project resources as well as their necessary overall level.

Key Elements of Timeline

There are five critical elements that will determine the overall length of the BOND:

        1. Program Development: Although a great deal of attention will need to be given to
           operationalizing the core benefit including the offset itself, establishing the two
           Employment Support (ES) interventions and piloting them to ensure their robustness
           prior to second stage random assignment will surely take longer. The WSOs will be new
           organizations (based primarily on existing service providers and brokers), and recruiting
           and organizing them, developing contracts for them, and training local staff will certainly
           be challenging. In addition, the One-Stop Career Centers in the areas selected to provide
           the enhanced employer-linked services will need training to provide this intensive service
           at a scale beyond what has so far been achieved under the Customized Employment
           Program (CEP). Sufficient time for program development is vital to ensure that the ES
           programs are quality implementations of the models that they represent and can provide a
           very substantial increment above the level of services that control group members will
           receive.

        2. Random Assignment: The length of time required to enroll the research sample via the
           random assignment process is the second key element. As described earlier, the timeline
           for BOND is best captured by thinking of it as three separate but interrelated
           experiments:

               An impact analysis that compares a treatment group of beneficiaries receiving the 50
                percent -offset ($1 for $2) -only that will be compared to a control group whose
                earnings will be treated under current law, based on SSA administrative records (1st
                stage random assignment);




Abt Associates Inc.          Chapter 6: Timeline for the Treatments and the Evaluation              77
               An impact analysis that will compare, to each other and to current law control
                group(s), treatment groups of volunteer beneficiaries who will receive combinations
                of ES treatments and benefit offsets, based on administrative records and surveys (2nd
                stage random assignment which excludes beneficiaries assigned to the two 1st stage
                groups described above); and

               An impact analysis that compares a treatment group of qualified applicant volunteers
                who will receive Early Intervention (EI) to a current law control group, based on
                administrative records and surveys.

            Each of these experiments will have its own random assignment process (although first-
            and second-stage RA will not be fully independent), and the lengthiest one will be the
            applicant (EI) component of the demonstration. The current assumption is that the flow
            of applicants and the assumed proportion of those likely to be found eligible for SSDI
            who will volunteer for BOND will require two years to complete the sample. There are
            sufficient beneficiaries to conduct first-stage random assignment at one time, although we
            have assumed a period of four months in order to mitigate the impact on SSA operations.
            The second-stage for existing beneficiaries can be completed as quickly as volunteers can
            be identified, enrolled and randomly assigned. That is, the capacity of WSOs to provide
            services immediately after random assignment is the main determinant of how fast the
            second-stage beneficiary research sample can be completed. Clearly, the shorter the
            period, the quicker the evaluation results can be available. The time gap between first-
            and second-stage random assignment may be considerable, and there is thus a great
            advantage—in timeliness of findings—to conducting the first stage as early as possible.

        3. Length of Follow-Up: The length of the follow-up period should be sufficient to
           determine if there are effects on main outcomes—employment, earnings and benefit
           receipt—and to determine the persistence of these effects, including either shrinkage or
           growth. As described above in 6.2, past history suggests that the second issue is more
           important than the first, in that there is very consistent evidence from previous
           employment-focused evaluations that if there are impacts, they emerge at some level
           quickly. Conversely, there is very little empirical support that sleeper effects will arise
           after several years. Three years after random assignment seems like a reasonable period
           for the first in-depth follow-up survey focused on outcomes (with an option for a later
           round of follow-up survey should findings justify it).

            Although it does not affect the length of the demonstration, an interim survey focused
            primarily on the amount and nature of services received, fielded approximately 12
            months after random assignment, will also be important for determining the
            treatment/control service differential as well as the service differential between
            treatments. This could be based on a sub-sample, since service receipt differentials will
            hopefully be large (at least between the ES treatment groups and the control groups) and
            the measures of service use would not require the precision of estimates of employment
            and earnings. However, this interim survey will also include a modest amount of
            employment (or other outcome) information for early impact analysis.




Abt Associates Inc.          Chapter 6: Timeline for the Treatments and the Evaluation               78
        4. Availability of Data for Analysis: For survey-based outcomes, data should be available
           with a very short delay after participant data collection. The intention is for data to be
           transmitted from the field and cleaned on a flow basis. Although undoubtedly there will
           have to be multiple attempts to interview some sample members, hopefully a high
           response rate can be achieved with no more than two additional months. For
           administrative records, particularly SSA earnings records, the picture is different. We
           understand that these data are about 95 percent complete by mid-August for the previous
           calendar year. If this schedule cannot be met, the draft evaluation timeline we present
           here will need to be adjusted, as well as consideration being given to obtaining state UI
           records. However, opting for the latter will most likely be time-consuming, difficult, and
           expensive.

        5. Analysis of Data and Drafting Reports: Major reports that are prepared for publication
           would be submitted in draft to SSA six months after the necessary data are available.
           This time is needed to ensure that the data are clean and consistent, to conduct analyses
           and explore interpretations, and to draft the report. We have also included ―letter reports‖
           to SSA in the timeline. These are to be based on administrative data and will consist
           primarily of impact tables, with a descriptive narrative. They are not intended to be
           published as reports. By narrowing and simplifying their contents, they can be submitted
           three months after the required data are available.

Exhibit 6-7 shows a summary of the three separate but interrelated experiments against the main
milestones of the implementation timeline (which was shown in more detail in Exhibit 6-6). Exhibit
6-8 gives the schedule of analytic deliverables SSA can expect from the full set of evaluation efforts.

Overview of Timeline for the First-Stage Random Assignment Experiment

There is a very large timeliness advantage to conducting the first-stage random assignment before the
end of CY2007. Doing so will mean that a full year of follow-up data (CY2008) will be available by
the summer of 2009 from SSA earnings records. The timeline assumes that random assignment—as
well as notification to treatment group members of their eligibility for the 1 for 2 disregard—will
begin in October 2007 and continue through January 2008 (producing a very slight loss of power by
this four-month spread).

Note that the timeline requires that SSA and Abt undertake a number of activities while the
procurement is underway. If a mechanism to accomplish this is not available, then although the first
year findings can be prepared on the same schedule, there will be a significant loss of power, since
the greater the proportion of the year that is prior to random assignment the harder it will be to detect
impacts which can, by definition, only occur after assignment.




Abt Associates Inc.           Chapter 6: Timeline for the Treatments and the Evaluation                 79
Abt Associates Inc.




                                                            Exhibit 6-7
                                                                                                                       2007                          2008                              2009                             2010                      2011
                                                                                 Task               Months 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52
                                                             1. D em o Program Im plementation

                                                               Award date for impl ementati on contr act                      March 1, 2007
                                                               Impl ementati on pl anning , si te
                                                               r ecr ui tment and prep (beg un ear lier )
                                                               ES activities start-up in sites

                                                               ES activities reach full ser vi ce level
Chapter 6: Timeline for the Treatments and the Evaluation




                                                               Full- scale si te oper ations
                                                               Site tr ansition and shutdown activities

                                                             2. First -St age Random Assignment
                                                             ( Benef iciaries)
                                                                1st-stag e RA occur s, offset is
                                                                impl emented for $1- for -$2-only
                                                                tr eatment g roup                                                                                                                                          Second Year
                                                                                                                                                                                            First Year
                                                                Impact analysi s using SSA benefit and
                                                                ear ning s records
                                                             3. Second-St age Random Assignment
                                                             ( Benef iciaries)
                                                                Baseli ne Sur vey and 2nd-stag e RA for
                                                                benefi ci ar y vol unteers
                                                               Inter im Sur vey on ES receipt and
                                                               employment
                                                               Anal ysis of implementati on, ES
                                                               differences, and ear ly i mpacts

                                                               36- month Follow- up Sur vey
                                                               Impact analysi s using SSA benefit and
                                                               ear ning s records
                                                             4. Applicant s (Early Int erv ention)
                                                               Baseli ne Sur vey and 2nd-stag e RA for
                                                               appl icants
                                                               Inter im Sur vey on ES receipt and
                                                               employment
                                                               Anal ysis of implementati on, ES
                                                               differentials, and earl y impacts

                                                               36- month Follow- up Sur vey

                                                               Fir st to thir d year impacts
                                                             5. Synthesis Report



                                                                                                             Letter Report
   80




                                                                                                             Draft Full Report
                                                                                                             Fi nal Ful l Report
Abt Associates Inc.




                                                                                                                     2011                      2012                                2013                                2014                       2015
                                                                                Task Months                    53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99100
                                                            1. D em o Program Im plement ation

                                                              Award date for impl ementati on contr act
                                                              Impl ementati on planning , site
                                                              r ecr ui tment and pr ep (beg un ear lier )
                                                              ES activities start- up in sites

                                                              ES activities reach full ser vi ce l evel

                                                              Full- scale si te oper ations
                                                              Site tr ansition and shutdown activities
Chapter 6: Timeline for the Treatments and the Evaluation




                                                            2. First -St age Random Assignment
                                                            ( Benef iciaries)
                                                               1st-stag e RA occur s, offset is
                                                               impl emented for $1- for -$2-only
                                                               tr eatment g roup                                                                  Fourth Y ear                         Fif th Year
                                                                                                                    Third Year                                                                                                   Final
                                                               Impact analysis using SSA benefit and
                                                               ear ning s recor ds
                                                            3. Second-Stage Random Assignment
                                                            ( Benef iciaries)
                                                               Baseli ne Sur vey and 2nd-stag e RA for
                                                               benefi ci ar y volunteers
                                                              Inter im Sur vey on ES receipt and
                                                              employment
                                                              Anal ysis of implementation, ES
                                                              differences, and ear ly i mpacts

                                                              36- month Follow-up Survey
                                                              Impact analysis using SSA benefit and               Second Year                     Third Year                               Fourth Y ear                       Fif th Year
                                                              ear ning s recor ds
                                                            4. Applicant s (Early Int erv ent ion)
                                                              Baseli ne Sur vey and 2nd-stag e RA for
                                                              appl icants
                                                              Inter im Sur vey on ES receipt and
                                                              employment
                                                              Anal ysis of implementation, ES
                                                              differentials, and earl y impacts

                                                              36- month Follow-up Survey

                                                              Fir st to thir d year impacts
                                                            5. Synthesis Report



                                                                                                            Letter Report
                                                                                                            Draft Full Report
   81




                                                                                                            Fi nal Ful l Report
                                         Exhibit 6-8
               Timeline for Analytic Deliverables From the BOND Evaluation
         Deliverable                       Sample                Data Available                   Due
                                      st
 Letter report on one-year           1 stage             SSA earnings and benefit
                                                                                              Month 32
 impacts                             beneficiaries       records
                                      st
 Letter report on two-year           1 stage             SSA earnings and benefit
                                                                                              Month 44
 impacts                             beneficiaries       records
                                                         E/C service differentials and
                                      nd                                                      Month 56
 Draft implementation report         2 stage             early employment impacts from
                                                                                              (Final Month
 and two-year impacts                beneficiaries       Interim Survey and SSA
                                                                                              59)
                                                         earnings and benefit records
                                      st                                                      Month 59
 Draft three-year earnings           1 stage             SSA earnings and benefit
                                                                                              (Final Month
 and benefit impact report           beneficiaries       records
                                                                                              62)
                                      st
 Later report on four-year           1 stage             SSA earnings and benefit
                                                                                              Month 68
 impacts                             beneficiaries       records
                                      nd
 Letter report on three-year         2 stage             SSA earnings and benefit
                                                                                              Month 68
 impacts                             beneficiaries       records
                                                         E/C service differentials and
                                                                                              Month 73
 Draft EI implementation                                 early employment impacts from
                                     Applicants                                               (Final Month
 report                                                  Interim Survey and SSA
                                                                                              76)
                                                         earnings and benefit records
                                      st
                                     1 stage             5-year SSA earnings and
                                     beneficiaries       benefit records                      Month 79
 Draft final report on
                                      nd                 Follow-up Survey plus 3-4 year       (Final Month
 volunteer beneficiaries             2 stage
                                                         SSA earnings and benefit             82)
                                     beneficiaries
                                                         records
 Letter report on one- to                                1- to 3-year SSA earnings and
                                     Applicants                                               Month 79
 three-year impacts                                      benefit records
                                      nd
 Letter report on four-year          2 stage             SSA earnings and benefit
                                                                                              Month 92
 impacts                             beneficiaries       records
                                                         Follow-up Survey plus 3-year         Month 97
 Draft EI final report               Applicants          SSA earnings and benefit             (Final Month
                                                         records                              100)
                                                                                              Month 97
 Draft synthesis report for all
                                     All                 All                                  (Final Month
 three groups
                                                                                              100)




On this assumption, SSA would receive a letter report on first year impacts just over 2.5 years after
contract award. Having this element of BOND, which is the main part of the study that is statutorily
required, underway in 2007 would show that SSA is meeting the legal requirement for the
demonstration and would produce reasonably timely results. A second letter report on the first-stage
sample would follow in 2010, and a final report on this sample would be submitted to SSA 59 months
after contract award. There will be ongoing tracking of administrative records after that time, and
updated findings will be included in annual letter reports to SSA as well as the final synthesis report.




Abt Associates Inc.               Chapter 6: Timeline for the Treatments and the Evaluation                  82
Overview of Timeline for the Second-Stage Random Assignment Experiment

Assuming that a means can be found for SSA to start the Abt team on detailed implementation
planning steps (to lay the groundwork for effective employment supports) before the award of the
implementation contract, the timeline provides that a year after contract award the WSOs will be
organized and the appropriate One-Stop Career Centers will begin offering enhanced employer-linked
services in the most advanced sites. We assume that by two months later, this will be the case in all
sites. By month 16, second-stage random assignment will begin in all sites and careful monitoring
will be underway to determine the proportion of treatment group members receiving service and the
fidelity of those services to the program model. To the extent that it is determined that the ES
services do not meet the required level of quality in a particular site prior to the date SSA determines
the site is ―up to speed,‖ those participants served under those circumstances (the early-assigned
cohort) will be tracked by administrative records only and will not be in the samples for the interim or
follow-up surveys.

By month 29, it is assumed that SSA will have compelling evidence that ES services are much more
intensive and purposeful than the background level and that the survey sample cohort can be put
underway in all sites. In the event not all sites are at this level, the survey sample will be initiated
nonetheless in order to preserve the nationally representativeness of the demonstration. By no later
than month 35, the current beneficiary research sample will be complete. (Because of the gap
between 1st and 2nd stage random assignment, there will have to be adjustments to the beneficiary
sample to account for those who have left and come on the roles in the period between the two
stages.) The interim survey on service receipt will begin in all sites no later than month 41 and be
complete six months later. An initial report on implementation and early employment and earnings
impacts (including 1-2 years of impacts based on administrative records) will be submitted to SSA in
month 56. The much more extensive three-year survey will be underway in all sites in month 65, and
a draft final report will be submitted to SSA in month 79.

Overview of Timeline for EI Experiment

The EI experiment will generally follow the same timeline as for the second-stage volunteer
beneficiaries, with random assignment for both beginning at the same time. However, because it will
take 18 months longer to fill out the applicant research sample (limited by the flow of volunteer
applicants), each of the surveys will take an additional 18 months to complete. Thus, the EI draft
implementation and early impact report will be submitted in month 73, and the final EI report will be
submitted in month 97.

In addition, in the same month, a draft synthesis report covering six years of administrative follow-up
for the first-stage sample, five years for the second-stage beneficiary sample, and three years for all
applicants will be submitted to SSA.

Primary Areas in Which SSA Actions/Decisions Can Enhance the Timeliness of Results

Employment Supports
The speed with which robust WSOs and Enhanced One Stops can be implemented to the point where
they provide timely services to a significant proportion of applicant and beneficiary volunteers that



Abt Associates Inc.          Chapter 6: Timeline for the Treatments and the Evaluation                83
are a major increment beyond control group services is can have a major effect on the timeliness of
findings. To that end it would be highly desirable for SSA and Abt to begin that process with a sense
of urgency now.

Administrative Employment and Earnings Data
Depending on the timeliness of SSA earnings records becoming available for analysis, the timeliness
of BOND findings might be improved by using state UI earnings records which are compiled
quarterly and are typically available within six months of the close of the quarter. However, UI
records are less comprehensive, and are becoming harder and more costly to access, especially given
the large number of states that would be involved in BOND. Child Support’s National Directory of
New Hires contains all state UI data and has the authority to provide access to these data for research
purposes. However, there are substantial restrictions and to this point, this access has never been
provided. Clarifying these options is important for SSA to reach a decision on the best approach.




Abt Associates Inc.          Chapter 6: Timeline for the Treatments and the Evaluation               84
References
Bloom, Howard. 1984. ―Accounting for No-shows in Experimental Evaluation Designs.‖
       Evaluation Review 8 (April): 225-46.

Greenberg, David. H., Karl Ashworth, Andreas Cebulla, and Robert Walker. February 2003. "Do
       Welfare-to-Work Programmes Work for Long?" Unpublished manuscript.

Long, David, Alan Werner, Bonnie O’Day, Judith D. Feins, and David C. Stapleton. December
       2005a. Social Security Administration $1 for $2 Benefit Offset Demonstration: Employment
       Supports Design. Report prepared for the U.S. Social Security Administration. Cambridge,
       MA: Abt Associates Inc.

Orr, Larry L., Judith D. Feins, Robin Jacob, Erik Beecroft, Lisa Sanbonmatsu, Lawrence F.Katz,
        Jeffrey B. Liebman, and Jeffrey R. Kling. 2003. Moving to Opportunity Interim Impacts
        Evaluation Final Report. Washington, D.C.: U.S. Department of Housing and Urban
        Development.

Orr, Larry L., David Long, Judith D. Feins, David C. Stapleton, Alan Werner, Raymond Glazier,
        David Vandergoot, Mark Hill, and Scott Szymendera. March 2005. Social Security
        Administration $1 for $2 Benefit Offset Demonstration: Treatment and Research Design
        Options. Report prepared for the U.S. Social Security Administration. Cambridge, MA: Abt
        Associates Inc.

Orr, Larry, Judith D. Feins, David Long, Debi McInnis, and David Stapleton. October 2005. $1 for
        $2 Benefit Offset Demonstration: Outcomes, Data Requirements, and Sample and Survey
        Design. Prepared for the U.S. Social Security Administration. Cambridge, MA: Abt
        Associates Inc.

Social Security Administration. 2003. Annual Statistical Report on the Social Security Disability.
        Washington, D.C.

Stapleton, David, Larry Orr, David Long, and Judith D. Feins. December 2005. $1 for $2 Benefit
        Offset Demonstration: Benefit Design and Parameter Recommendations. Prepared for the
        U.S. Social Security Administration. Cambridge, MA: Abt Associates Inc.




Abt Associates Inc.                                                       References                 85

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:9
posted:7/22/2011
language:English
pages:88
Description: Retirement Calculator Done in Visual Basic Tutorials document sample