Significant, but Not Decisive Unabridged PDF

							          http://www.educationnext.org/unabridged
                   /20012/goldhaber.pdf



What Can We Infer From Recent Experiments with Educational
                       Vouchers?


                     Dan Goldhaber
                   The Urban Institute
         A great deal of research on education has yielded relatively few
definitive findings to guide policymakers on what educational interventions
might be used to truly enhance student achievement. The studies reviewed
here, concerning the effectiveness of privately financed educational
vouchers programs in New York City, NY, Dayton, OH, Washington, DC
(Howell et al., 2001), and Charlotte, NC (Greene, 2001), by contrast, suggest
private schools lead to dramatic improvements in achievement for at least
some students. This proposition is potentially quite important given the
ongoing debate over the appropriateness and advisability of public provision
of subsidies (or “vouchers”) to parents to be used at their discretion at either
public or private schools.
         Each of the voucher programs studied shared common programmatic
features. The programs are all relatively new, with the Washington
program, originally established in 1993, as the longest running. 1 Although
eligibility varies slightly, all of the programs are targeted toward low-income
students, primarily those who receive free- or reduced-price lunch, and offer
modest subsidies. The most generous of the programs offered a maximum
elementary school voucher of $1700 per year. 2 Perhaps most importantly
from a research perspective, it was necessary for each program to establish a
lottery system as the demand for the programs exceeded the available supply
of vouchers.3 This offered a relatively unique opportunity to evaluate the
programs in an experimental context, potentially mitigating many of the
statistically concerns inherent in evaluating the effect of private schooling in
non-experimental settings.
1
  Although this program was first established in 1993, it was fairly small until it was expanded in 1997 and
again in 1999.
2
  In most of the cities there was the expectation that families would supplement the voucher amount.




                                                       1
         In addition to the occurrence of common program features across the
studies, the researchers utilized similar methodologies in assessing program
effects. Researchers administered short questionnaires to students and
surveyed students’ families. In general, voucher recipients were found to be
more satisfied with a variety of aspects of their chosen schools (e.g. safety,
teacher quality, etc.). This finding is consistent with the literature on
satisfaction with schooling, but results based on perceptions have a tendency
to be misleading. They may reflect parents’ reports of genuine differences
in school quality. Alternatively it may reflect a “buy-in effect,” in which
students and parents who are given choices report greater satisfaction simply
by virtue of having the choice itself. Certainly, greater school satisfaction
should be regarded as a positive effect, however, we simply cannot know
whether these findings reflect genuine differences in school quality or
simply the impact that granting options has on satisfaction. 4 Given this
ambiguity, in this review I will focus primarily on reported student
achievement results.
         While the effects of private schooling varied across studies, grades,
and racial/ethnic groups, positive effects were consistently found for African
American students who participated in the programs for at least two years.
The average estimated (positive) effect (for African American students in
the Dayton, New York City, and Washington DC programs) of attending a
private school rather than a public school is over 6 percentile points in both
mathematics and reading. In the Charlotte program, where effects are not

3
  The lotteries discussed here were held amongst all applicants to the programs, not the general population.
4
  One way researchers might test this is by regressing parental satisfaction with schooling characteristics on
both a dichotomous variable identifying school sector and actual school data, such as class size, safety, etc.
The statistical significance and magnitudes of the coefficients of these variables could be used to determine
if, all else equal, individuals appeared to be more satisfied if they, for instance, were in smaller classes or in
safer environments.



                                                        2
broken out by race, the private school effect was also estimated to be an
increase of approximately 6 percentile points after 1 year.
      The size of the estimated private school effect on student achievement
is quite large by social science standards. For instance, in Washington, DC
where the largest effects were found, after two years the private school effect
for African American students in grades 2-5 was estimated to be 10
percentile points in mathematics and 8.6 percentile points in reading (after
two years in the program). This represents a gain of about 0.33 standard
deviations relative to African American counterparts in the control group.
For some perspective on what this finding means, it is useful to compare the
effects to those found in an evaluation of the well-known STAR class size
reduction program in Tennessee (Finn and Achilles, 1999). This study
found that a reduction in class size of about 8 students from a class of 22-26
students to one with 13-17 students (a relatively expensive intervention)
(Brewer et al., 1999), is estimated to increase minority students’ second-
grade reading achievement by about the same amount as the Washington,
DC voucher program – 0.33 standard deviations. Thus, the estimates from
the studies reviewed here suggest that even modestly sized vouchers, which
may be considerably less costly than the marginal cost of educating students
in the public sector, could provide substantial benefits for some students.
However, there are several cautions regarding this interpretation of the
findings.


Cautions Regarding Findings
      In assessments of program effects in non-experimental settings one
statistical concern for researchers is the potential that important individual
characteristics are omitted from analyses. Many individual characteristics,

                                        3
such as motivation, are difficult to quantify in data (henceforth these will be
referred to as “unobservables”). It is referred to as “sample selection” or
“selection effects” if these unobservables are important in determining
outcomes systematically related to the intervention being analyzed. 5 In the
case of private schooling, there are multiple reasons why one might think
that students and parents in each school sector might differ along both
observable and unobservable dimensions. For instance, the fact that parents
utilize a voucher (and in the case of private schooling, spend additional
money) could indicate that they provide a home environment conducive to
academic achievement. If selection effects come into play, the impact of
unobservables may be wrongly identified as school sector effects.
         The strength of the studies reviewed here is that they take advantage
of the fact that the programs are oversubscribed to compare the outcomes of
students that were randomly assigned either to public (the control group) or
private (the treatment group) schools. This use of random assignment has
intuitive appeal, is easily understood, and is used in other contexts, such as
medical research. Under ideal conditions, when using this methodology
there should be little concern about selection effects, and therefore no need
to control for the background characteristics, observable or unobservable, of
students either in the control or treatment groups.6 Of course, ideal
conditions rarely, if ever, exist. Small deviations from the ideal are unlikely
to result in biased findings, however, larger deviations can bias the result of
even a carefully designed social experiment, possibly resulting in misleading

5
  There are ways to statistically account for these potential unobservable differences but the techniques
require many assumptions, the validity of which is difficult to test. For a review of methodologies used to
account for differences in students’ backgrounds, see Goldhaber and Eide (2000).
6
  Both studies also control for students’ baseline test scores and the Greene study also included student and
family background variables in the student achievement models.



                                                     4
conclusions. Though it was not explicitly designed as a social experiment,
as I describe below, the results from a “natural experiment” with vouchers in
Milwaukee were highly dependent on the methodology used to deal with
attrition out of the experimental groups.
       There are several findings of the studies that are somewhat troubling
in that they raise concerns about the quality of the control and treatment
groups. First, the magnitude of the private school effect is substantially
larger than what is typically found in non-experimental studies on
differences between public and private schools, even those that focus on
minority students (McEwan, 2001). Second, one of the findings common to
all the programs that include separate analyses for students in different
racial/ethnic categories is that there are no statistically significant differences
between public (control) and private (treatment) students in test score
performance of non-African Americans students. Finally, in some cases
there were rather large swings in the estimated effects from year 1 to year 2.
In Washington, DC the estimated private school impact on African
American students in reading went from a statistically significant negative
effect of 9 percentile points in year 1 to a statistically significant positive
effect of 8 percentile points in year 2, a net positive swing of 17 percentile
points from one year to the next. This means that an additional year of
private schooling is estimated to produce a staggering gain of 0.6 standard
deviations.
       What might account for the above anomalies? As the authors note, it
is not clear why one would observe these rather inconsistent findings. Of
course, the quality of the results depends crucially on the integrity of the




                                          5
experiments.7 It is important to prevent or account for contamination of the
experiment, such as might occur if there were differences in response
between the treatment or control groups or non-random placement into
either group.
        The authors present evidence on the demographic characteristics of
students who received vouchers and those who did not, and, in general, these
data suggest that there is little difference between lottery winners and losers
(although there were some statistically significant differences in Charlotte).
However, many students who “won” vouchers in the random lottery, chose
not to use them. In Washington, DC, 47 percent of recipients did not use
their voucher, in Dayton this figure is 46 percent, and in New York it is 24
percent (Zernike, 2000). Even if voucher users and non-users (and control
group study participants) appear to be similar along observable lines, they
may not be similar in terms of unobservable characteristics. In fact, it is
quite plausible that those parents who select not to use a voucher do so at
least in part because they perceive that sending their child to private school
will not necessarily be beneficial based on the specific public and private
school options available in their locality. 8
        The studies use a statistical technique (“instrumental
variables”) that explicitly accounts for potential differences,
observable or unobservable, between voucher-users and non-
users. As a result, the results are not likely to suffer from this

7
  For a comprehensive review of the issues surrounding the design of (voucher) experiments, see Doolittle
(1999).
8
  Greene (2001) also employs an instrumental variables approach to account for the potential that students
who use a voucher may be different from those who do not in unobserved ways. However, this approach is
also potentially problematic. If the probability of using the voucher is correlated with the likelihood of
remaining in the sample, the estimated findings using this approach may be biased.



                                                    6
potential source of bias. However, this technique cannot account
for the potential bias associated with “response attrition” — that is,
the potential that there are systematic differences between those
who participate in the testing and survey sessions and those that do
not. In some cities, such as Charlotte, there were fairly low participation
rates in the testing sessions. In Charlotte, the overall response rate was 40
percent, but it was only 20 percent for those students who applied for and
received a voucher but elected not to use it (Greene, 2001). 9
         There exist similar concerns about attrition out of the sample over
time. If attrition from the treatment sample (those who won the lottery) is
different from attrition from the control sample (those who applied but did
not win the lottery) in terms of the composition (e.g. ability) of students who
leave, then the results of the experiment may be biased. For instance, if the
students who left the treatment group tended to be of lower ability than those
who left the control sample, the experiment would overstate the effect of
private schooling. Given that there were sizable drops in the sample sizes in
all the cities where the voucher programs have been studied for two years,
non-random attrition may be an issue. The attrition rates are generally
similar between the treatment and control groups (and the
researchers (Howell et al., 2001) use weights to adjust for
differences in observed characteristics) and the two groups remain

9
  Greene (2001) suggests that there is little evidence of “creaming” by private schools. He reports that
there were few differences between public and private schools in parental reports of physical handicaps or
that English was not the primary language spoken in the home. However, the parental reports are
somewhat suspect given that only 3 percent of choice students reported physical handicaps and 2 percent of
public school students, figures much lower than what is estimated in national data. Similarly, parents who
did not remain in private schools did not report that their children were expelled. But again, these reports
may be suspect if one believes that parents are less than honest in reporting this particular student outcome.



                                                      7
similar along observable dimensions, however, there is still a
concern that over time attrition results in unobservable differences
between the two groups.
         A review of findings from the (similarly structured publicly financed)
voucher program in Milwaukee is instructive in showing the potential
impact of contamination of treatment and control groups. Researchers
(Greene et al., 1998) studying this program who employed similar
methodologies to those used in the studies reviewed here estimate relatively
large private school effects for students who attended private schools for
several years: 6 percentile points in reading and 7 percentile points in
mathematics in year three of the program. By contrast, Rouse (1998) uses a
methodology that accounts for the possibility that individual unobserved
exist that are correlated with school sector; for instance, students and parents
who remained in private schools might be more motivated. This might have
resulted from the high levels of attrition out of both the treatment and control
groups. Rouse estimates similar private school effects in math (1 to
2 percentage points effect per year) but no statistically significant
positive private school effect in reading.10
         It is important to make clear that there is no direct evidence that the
results reported by Howell et al. and Greene are in fact biased. If data
collection on the programs in these cities continues it will be possible to
employ statistical techniques to account for this potential. Nevertheless, if
these findings are accurate assessments of private school effects they
certainly lend credence to arguments in support of vouchers. Given this

10
  Rouse uses a “fixed-effects” estimation approach where individual-specific controls are included in the
regression specification.



                                                     8
possibility it is instructive to consider what we might infer from larger scale
implementation of such programs. This subject is explored in the next
section.


Generalizability of the Voucher Experiments
      There are several reasons why the findings from relatively small
experimental programs may not be generalizable, a point acknowledged by
the authors. The evaluations did not include controls for the demographics
or ability of the other (non-voucher) students in the private schools. It is
possible that what is perceived to be a private school effect is actually a peer
effect. To the degree that peers influence educational outcomes and the
students in private schools were higher achievers, we might expect to find
private school effects, in either experimental or non-experimental settings,
even if there really are not significant contributions made by the school, per
se. However, in a larger scale implementation of a voucher-program, any
peer effect would diminish if private schools began to more closely resemble
public schools in terms of demographics.
      In the case of these voucher experiments, the randomization to
determine who gets treated was only among those students and parents who
expressed an interest in attending private schools. The estimated impact on
this group that desires to attend private schools is not necessarily the effect
on the general population (Angrist, Imbens, and Rubin, 1996; Heckman and
Smith, 1993).11 A similar point applies in terms of the schools that are
implicitly included in the experiment. There is likely considerable variation
in the quality of both public and private schools, but in these experiments




                                        9
researchers really do not know whether they are comparing the average
public schools to the average private schools, or schools that fall elsewhere
on the quality distribution. 12 For instance, one might assume that those
students who are enrolled in ineffective public schools are more likely to
apply to be a part of a program. On the other hand, one might argue that
only the least effective private schools would have the capacity to take on
additional students, and were likely to be included in the experiment. The
bottom line is that it is not at all clear whether similar findings would be
expected from a larger scale program.


Other Issues To Consider
         I believe the authors are right to speculate on the role that teachers
play in explaining any differences in effectiveness between public and
private sectors, should they exist. There exists significant variance in
teacher effectiveness (Wright, Horn, and Sanders, 1997; Rivkin, Hanushek,
and Kain, 1998; Goldhaber, Brewer and Anderson, 1999) as well as
institutional differences in the type of teachers who are hired and the way
they are compensated (Ballou and Podgursky, 1997, 1998). 13 However,
because the focus of these studies is on the impact on those receiving
vouchers, they only capture a slice of what we might expect from a large-
scale voucher policy.
         There are currently too few private school slots to accommodate a
large shift of students into the private sector. We might expect that the

11
   The authors distinguish between the direct voucher effect and the offer effect, but the offer effect also
only applies to those with an interest in receiving the treatment.
12
   For a more comprehensive explanation of this point, see Goldhaber (2001).
13
   For a review of the potential reasons why private schools might be more effective than public and why
greater school choice could lead to better educational outcomes, see Goldhaber and Eide (2000).



                                                      10
increase in demand associated with a voucher policy would elicit a supply
response from the private sector, but we do not know what type of private
schools would form to meet the increased demand. Economic theory
suggests that those suppliers who are most efficient are likely already in the
marketplace implying the quality of future private school entrants would be
lower.14 Though there is at least some evidence to suggest that competition
from private schools leads to increased public school efficiency (Couch et.
al.,1993; Dee, 1998; Hoxby, 1994), we really know little about how public
schools would respond to the competitive threat of losing students under a
voucher system. 15 The programs studied by Howell et al. and Greene do not
provide sufficient evidence on these important issues to be considered in the
policy debate over vouchers. However, the evidence presented in these
studies is encouraging with regards to students who use vouchers,
particularly minority students, and much more may be learned as additional
data becomes available.




14
   Greene argues that the “infusion of cash that a publicly funded voucher would provide to private schools
might make them even more effective”, and that newly formed private schools “might be better or worse
than existing private schools.” (p. 7). As I argue above, from a theoretical economic prospective, future
entrants into the market are unlikely to be as effective (at a given price level) as those already in the market.
15
    Though Levin and Driver (1997) and Levin (1998) estimate that the public costs of a voucher plan in a
representative US context could raise public educational costs by 25 percent or more, we really know little



                                                       11
References
Brewer, Dominic J., Cathy Krop, Brian Gill, and Robert Reichardt (1999),
“Estimating the Cost of National Class Size Reductions Under Different
Policy Alternatives.” Educational Evaluation and Policy Analysis,
21(2):179-192.

Doolittle, Fred (1999), “Designing Education Voucher Experiments:
Recommendations for Researchers, Funders, and Users.“ Paper prepared for
the Setting the Agenda Conference, National Center for the Study of
Privatization in Education, Teachers College, Columbia University.

Couch, Jim F., William F. Shughart, and Al L. Williams (1993), “Private
School Enrollment and Public School Performance.” Public Choice, 76:
301-312.

Finn, J.D., and C.M. Achilles (1999), “Tennessee’s Class Size Study:
Findings, Implications, and Misconceptions.” Educational Evaluation and
Policy Analysis, 21(2): 97-109.

Goldhaber, Dan D., Dominic J. Brewer, and Deborah J. Anderson (1999),
“A Three-Way Error Components Analysis of Educational Productivity.”
Education Economics, 7(3): 199-208.

Goldhaber, Dan D.(2001), “The Interface Between Public and Private
Schooling: Market Pressure and the Impact on Performance.” Paper
prepared for the Improving Educational Productivity Conference: Lessons
from Economics. Laboratory for Student Success, Tempe University Center
for Research on Human Development and Education.

Goldhaber, Dan D., and Eric R. Eide (2000), “Methodological Thoughts on
Measuring the Impact of Competition in the Educational Marketplace.”
Unpublished Working Paper.

Greene, J. P., Peterson, P. E., & Du, J. (1998). School choice in Milwaukee:
A randomized experiment. In P. E. Peterson & B. C. Hassel (Eds.), Learning
from school choice (pp. 335–356). Washington, DC: The Brookings
Institution.




                                     12
Hanuskek, Eric A., John F. Kain, and Steven G. Rivkin (1998), “Do Higher
Salaries Buy Better Teachers?” Paper presented at the 1999 AEA meetings
in New York, NY, January.

Hoxby, Caroline M. (1996), “The Effects of Private School Vouchers on
Schools and Students.” In Helen F. Ladd (editor), Holding Schools
Accountable: Performance-Based Reform in Education, Washington, D.C.:
The Brookings Institution.

McEwan, Patrick J. (2000), “The Potential Impact of Large-Scale Voucher
Programs.” Forthcoming in Review of Education Research.

Rouse, Cecilia E. (1998). Private school vouchers and student achievement:
An evaluation of the Milwaukee parental choice program. Quarterly Journal
of Economics, 113, 553–602.

Wright, Paul S., Sandra P. Horn, and William L. Sanders (1997), “Teacher
and Classroom Context Effects on Student Achievement: Implications for
Teacher Evaluation.” Journal oI Personnel Evaluation in Education, 11.




                                    13