VIEWS: 4 PAGES: 44 POSTED ON: 10/1/2012
Vouchers, Public School Response and the Role of Incentives: Evidence from Florida Rajashri Chakrabarti Federal Reserve Bank of New York Motivation Concern over public school performance in U.S. Results from international tests show that the U.S has been lagging behind several other countries, both developed and developing In NAEP, between 1971 and 1999 (Hoxby) Productivity in math for 9 year olds fell by 54.9% Productivity in reading for 12 year olds fell by 73.4% Between 1960 and 1995 (Hanushek) Pupil-teacher ratio fell by a third Real spending per pupil tripled Student outcomes for 17 year olds remained the same Public school reform and vouchers This paper Motivated by the need to understand the behavior of public schools facing vouchers Focuses on the Florida voucher program Motivation (continued) The literature looking at the effect of vouchers on public schools typically focuses on student and mean school scores. This paper Tries to go inside the black box to investigate some of the ways in which schools facing the threat of vouchers in Florida behaved. Florida voucher program or “Opportunity Scholarship” Program Implemented in June 1999 All students of a school became eligible for vouchers if the school received 2 “F” grades in a period of four years Thus a “threat of voucher” program The threatened schools have strong incentives to try to avoid the second “F” and thereby avoid vouchers Motivation (Continued) How did the threatened public schools behave? Did incentives matter? Exploits institutional details to analyze the incentives built in to the system Investigates whether threatened public schools responded to incentives Examines three alternative forms of behavior that the program incentives might have induced, and investigates whether the Florida data support such behavior Motivation (continued) Three forms of Responses Certain % of students had to score above pre-designated cutoffs in the score scale to escape the second “F”. Did this induce the threatened schools to focus more on students expected to score below these cutoffs rather than equally on all? Schools needed to pass the minimum criteria in only one of the three subjects to escape an “F” grade. Did this induce the threatened schools to concentrate more on one subject, rather than equally on all? If so, which subject is it? Scores of students in several special education categories (ESE categories) were not included in the computation of grade. Therefore, did the threatened schools tend to reclassify their low performing students in to these “exempt” categories so as to remove them from the relevant test taking pool and artificially boost scores? Policy Implications Relationship to the literature Related to two strands of literature Gaming behavior of schools facing accountability programs Cullen and Reback (2002), Figlio and Getzler (2002), Jacob (2005) Jacob and Levitt (2003), Figlio (2006), Figlio and Winicki (2005) This study: Do public schools exhibit similar behavior when facing vouchers? Difference in empirical strategies: Regression Discontinuity in this study in addition to difference-in-differences, unlike in above literature Effect of vouchers on public schools Greene (2001) Greene and Winters (2003), Chakrabarti (2005), Figlio and Rouse (2006), West and Peterson (2006) Goldhaber and Hannaway (2004) Hoxby (2003a, 2003b), Chakrabarti (2005) Most of the studies above investigate effect of vouchers on students and school scores in public schools. This study examines some of the alternative ways in which public schools facing the threat of vouchers behaved. Institutional Details of the Program Florida Program Embeds a voucher program within an accountability system Testing System FCAT Reading and Writing grades 4, 8, 10 FCAT Math grades 5, 8, 10 FCAT grade 4 Reading and grade 5 Math: Score Range 100-500, 5 levels 1-5 FCAT grade 4 Writing: Score Range 1-6, 5 levels 1-5 Grading Criteria School grades based on student scores in reading, math and writing Grade F → if fails minimum criteria in reading, math and writing Grade D → if fails minimum criteria in one or two subjects Minimum Criteria: Read and Math → at least 60% students need to score at or above level 2 Writing → at least 50% students need to score above 3 in writing Scores of regular students and students in only a few ESE and LEP categories count towards grade formation Incentives Created by the Program and Alternative Avenues of Public School Response Focusing on students below the minimum criteria cutoffs Level 1 in Reading and Math Levels 1 and 2, and especially level 2 in writing Choosing between subject areas Did the threatened schools focus on a single subject area rather than equally on all? Focus on subject area closest to the cutoff ? Role of different extents of difficulties of subjects Reclassifying low performing students into “Exempt” ESE categories Costs and the McKay Scholarship program Data Disaggregated school level data obtained from the Florida Department of Education FCAT grade 4 Reading, grade 5 Math and grade 4 Writing Data on % of students scoring in the different levels 1-5 in reading and math, 1998- 2002 Data on % of students scoring at 1, 1.5,..,6 in writing, 1994-96 and 1998-2002 ESE data Data on % of students in each of the ESE categories in Florida 15 categories for 1993-97, 21 categories for 1998-2002 Data on School Accountability in Florida Data on Socio-economic characteristics of schools Grade distribution of students, race, sex, % of students eligible for free or reduced price lunches, real per pupil expenditure Empirical Strategy Forming Treatment and Control groups 1999 F schools treatment group F schools from now on 1999 D schools control group D schools from now on Focus on three years after program Difference-in-differences estimation strategy and Regression discontinuity strategy Graphical Representation: Did threatened schools focus on students expected to score below cutoffs? Distribution of percentage of students in level 1 Math and level 2 Writing, 1999 and 2000 (F and D Schools) F schools D schools F schools D schools .04 .04 .03 .03 .02 .02 .01 .01 0 25 50 75 100 0 25 50 75 100 % in level 1, 1999, Math % in level 1, 2000, Math Panel A Panel B F schools D schools F schools D schools .08 .06 .06 .04 .02 .02 0 25 50 75 100 0 25 50 75 100 % in level 2, 1999, Writing % in level 2, 2000, Writing Panel C Panel D Graphical Representation (Continued) Distribution of Math Scores by Treatment Status, 1999 and 2000 level1 level2 level3 level1 level2 level3 level4 level5 level4 level5 Percentage of Students in Different Levels Percentage of Students in Different Levels 80 80 60 60 40 40 20 20 0 F Schools D Schools 0 F Schools D Schools Distn. of Math Scores in 1999, by Treatment Status Distn. of Math Scores in 2000, by Treatment Status Graphical Representation (Continued) The case of Reading and Math, 1999-2002 F D F D 70 70 60 % in level 1 % in level 1 60 50 50 40 40 30 1999 2000 2001 2002 1999 2000 2001 2002 % in level 1, FCAT Reading % in level 1, FCAT Math Graphical Representation (Continued) The case of Writing, 1999-2002 F D F D 30 45 % in level 2 % in level 1 20 35 10 25 0 15 1999 2000 2001 2002 1999 2000 2001 2002 % in level 1, FCAT Writing % in level 2, FCAT Writing Sensitivity Checks: Existence of Pre-program trends Sensitivity Checks (Continued): Compositional Changes of Schools and Student Sorting Sensitivity Checks (Continued) Is Mean Reversion a Concern? Mean Reversion Exploiting pre-program data and institutional details of program Idea is to measure the extent of declines (if any) in level 1 (in reading and math) and level 2 (in writing) in schools that received an F grade in 1998 relative to schools that received a D grade in 1998 during 1998-99 Since 1998 is pre-program period, this gain can be taken as mean reversion effect mean reversion corrected effects Assigning letter grades to schools: 98F and 98D schools Is Mean Reversion a concern? Regression Discontinuity Analysis An alternative way to get rid of mean reversion Quasi-experimental research design Highly non-linear and discontinuous relationship between the % of students scoring above a threshold and probability that the school’s students become eligible for vouchers in the near future Regression discontinuity strategy exploits the design of the Florida program and state grade assignment formula Consider F and D schools that failed minimum criteria in reading and math in 1999 F schools also failed in writing, while D schools passed Discontinuous relationship between probability of treatment and % of students scoring above 3 in writing Sharp discontinuity at 50% Discontinuity sample 1 ( 7 ) and Discontinuity Sample 2 ( 5) Also consider other corresponding discontinuity samples both F and D schools fail minimum criteria reading and writing (math and writing) Regression Discontinuity Analysis: Assignment to treatment (Sample of F and D Schools that failed minimum criteria in reading and math in 1999) 1 1 Treatment Status Treatment Status 0 0 15 25 35 45 55 65 75 85 95 15 25 35 45 55 65 75 85 95 % at or above 3 in writing, 1999 % at or above 3 in writing, 1999 Regression Discontinuity Analysis: Effect of the program on % of students scoring in level 1 Reading and level 1 Math 65 55 % in level 1 Reading, 2000 % in level 1 Math, 2000 60 50 55 45 50 40 40 50 60 40 50 60 % at or above 3 in writing, 1999 % at or above 3 in writing, 1999 Panel A Panel B Regression Discontinuity Analysis: Effect of the program on % of students scoring in levels 1 and 2 in Writing 15 30 % in level 2 Writing, 2000 % in level 1 Writing, 2000 10 25 5 20 0 15 40 50 60 40 50 60 % at or above 3 in writing, 1999 % at or above 3 in writing, 1999 Panel A Panel B The problem of Underestimation: Are D schools Untreated? D schools do not directly face the threat of vouchers They are close to getting an “F” likely to face an indirect threat Effects above are underestimates Problem likely to be more prominent in regression discontinuity analysis F versus C analysis Rescale the above treatment effects by difference in the probabilities of treatment of F and D schools (Wald estimator) Use pre-program data to calculate these probabilities Calculate these scaling factors separately for the full sample and the discontinuity samples Using this strategy, correcting for underestimation amounts to scaling up the diff- in-diff estimates by 1.15 and the RD estimates by 1.27 Is Stigma Effect of Getting the Lowest Performing Grade driving the Results? F lowest performing grade in the Florida grade scale If stigma associated with getting the lowest performing grade Response may be due to stigma effect rather than the effect of threat of vouchers Alternative strategies to address this issue: Exploit institutional details of the pre-program accountability system and pre- program data Pre-1999 accountability system 1-4 (1-low, 4-high) Exploit post-program grades None of the 1999 F schools received another F grade in either 2000 of 2001 Stigma effect would not be applicable in 2001 and 2002, while the threat of voucher effect would still be applicable Choosing between subjects with different extents of difficulties versus focus on subjects closer to the cutoff Rank subject areas in each F school in terms of distances from cutoff “low”, “mid” and “high” Standardize reading, math and writing scores (% of students below cutoff) by grade, subject and year to have means of 0 and standard deviations of 1 Are year specific shocks in writing driving the results? Doing a Regression Discontinuity Analysis Anecdotal Evidence Phone Interviews: Widespread beliefs among Florida school administrators that writing scores are much easier to improve than reading and math scores Team approach in writing writing across the curriculum School-wide projects related to writing Longer time blocks in writing Anecdotal Evidence Phone Interviews with School Administrators: Focused on lower performing students in various ways enlisted the help of the corresponding parents Introduced guided reading for weaker students Introduced “cooperative learning group” combine children of different abilities in a single group, so that the lower performing students also learnt from the higher performing ones Reclassifying low performing students in to “Exempt” ESE categories Shifts in % ESE (total ESE classification as % of total enrollment) Shifts in classification in excluded categories relative to included categories Classification in mutable excluded versus included categories Strategy: Difference-in-Differences Analysis Check for Existence of Pre-program trends Regression Discontinuity Analysis Graphical Representation: Classification in to ESE categories F D 25 20 15 10 1998 1999 2000 2001 2002 YEAR Total ESE Classification F D 15 10 5 0 1998 1999 2000 2001 2002 YEAR Classification into Excluded Relative to Included ESE Categories Figure 5. Classification into ESE Categories, F and D schools Not much Evidence of Reclassification: Is this consistent with incentives? Absence of reclassification does not imply that schools did not respond to incentives Costs of reclassification Approved by parents, group of experts (physicians, psychologists) Too much classification may lead to audits by Florida DOE Costs of Services McKay Scholarship program for disabled students Conclusions This paper studies the behavior of public schools facing the threat of vouchers in Florida. The literature investigating the effect of voucher programs on public schools typically focuses on overall scores. In contrast, this paper tries to go inside the black box to investigate some of the ways in which public schools facing the threat of vouchers in Florida behaved Analyzing the incentives built in to the system, it examines whether the threatened public schools responded according to the incentives built into the system. It finds robust evidence that they did respond to incentives. First, relative to the D schools, the threatened public schools concentrated more on students expected to score below the high stakes cutoffs. Second, they concentrated relatively more on writing, which is regarded as the easiest subject area. There is not much evidence in favor of reclassification of students in to excluded categories. This is consistent with the existence of considerable costs associated with such reclassification. Conclusions (Continued) Policy Implications Public Schools responded to incentives Implies that policy can be targeted to induce public schools to behave in desirable ways The 2002 changes Conclusions and Policy Implications This paper studies the behavior of public schools facing the threat of vouchers in Florida. Analyzing the incentives built in to the system, it examines whether the threatened public schools behaved strategically to respond to these incentives. It finds strong evidence that they did respond to incentives strategically. First, relative to the D schools, the threatened public schools concentrated more on students expected to score below the high stakes cutoffs. Second, they concentrated relatively more on writing, which is regarded as the easiest subject area. There is not much evidence in favor of strategic reclassification of students in to excluded categories. Conclusions and Policy Implications (Continued) Contributions: The literature investigating the effect of voucher programs on public schools typically focuses on overall scores In contrast, this paper tries to go inside the black box to investigate some of the ways in which public schools facing the threat of vouchers in Florida behaved First paper to investigate these kinds of strategic behaviors of schools facing vouchers First paper to use a quasi-experimental strategy--regression discontinuity design-- to study strategic behaviors of schools Important Policy Implications Public Schools responded to incentives Policy can be targeted to carve public school behavior in desirable ways If more attention on reading and math warranted calls for a change in grading rules to decrease weight of writing and increase those of reading and math If more attention on higher performing students desired calls for their inclusion in the computation of F and D grades Policy changes in 2002 suggest interaction of policy and public school response and that policy has been a response to public school response Is there a Stigma Effect?
"Do Public Schools Facing Vouchers Behave Strategically Evidence"