Do Public Schools Facing Vouchers Behave Strategically Evidence

					Vouchers, Public School Response and the
Role of Incentives: Evidence from Florida


            Rajashri Chakrabarti
     Federal Reserve Bank of New York
Motivation
   Concern over public school performance in U.S.
        Results from international tests show that the U.S has been lagging behind several other
         countries, both developed and developing

   In NAEP, between 1971 and 1999 (Hoxby)
        Productivity in math for 9 year olds fell by 54.9%
        Productivity in reading for 12 year olds fell by 73.4%

   Between 1960 and 1995 (Hanushek)
        Pupil-teacher ratio fell by a third
        Real spending per pupil tripled
        Student outcomes for 17 year olds remained the same

   Public school reform and vouchers

   This paper
        Motivated by the need to understand the behavior of public schools facing vouchers
        Focuses on the Florida voucher program
Motivation (continued)
   The literature looking at the effect of vouchers on public schools
    typically focuses on student and mean school scores.

   This paper

        Tries to go inside the black box to investigate some of the ways in which
         schools facing the threat of vouchers in Florida behaved.

   Florida voucher program or “Opportunity Scholarship” Program

        Implemented in June 1999
        All students of a school became eligible for vouchers if the school
         received 2 “F” grades in a period of four years
        Thus a “threat of voucher” program
        The threatened schools have strong incentives to try to avoid the second
         “F” and thereby avoid vouchers
Motivation (Continued)
   How did the threatened public schools behave? Did
    incentives matter?

       Exploits institutional details to analyze the incentives built in to
        the system

       Investigates whether threatened public schools responded to
        incentives

       Examines three alternative forms of behavior that the program
        incentives might have induced, and investigates whether the
        Florida data support such behavior
Motivation (continued)
   Three forms of Responses

        Certain % of students had to score above pre-designated cutoffs in the score scale
         to escape the second “F”. Did this induce the threatened schools to focus more on
         students expected to score below these cutoffs rather than equally on all?

        Schools needed to pass the minimum criteria in only one of the three subjects to
         escape an “F” grade. Did this induce the threatened schools to concentrate more on
         one subject, rather than equally on all? If so, which subject is it?

        Scores of students in several special education categories (ESE categories) were not
         included in the computation of grade. Therefore, did the threatened schools tend to
         reclassify their low performing students in to these “exempt” categories so as to
         remove them from the relevant test taking pool and artificially boost scores?

   Policy Implications
Relationship to the literature
   Related to two strands of literature

        Gaming behavior of schools facing accountability programs
              Cullen and Reback (2002), Figlio and Getzler (2002), Jacob (2005)
              Jacob and Levitt (2003), Figlio (2006), Figlio and Winicki (2005)

              This study: Do public schools exhibit similar behavior when facing vouchers?
              Difference in empirical strategies: Regression Discontinuity in this study in
               addition to difference-in-differences, unlike in above literature

        Effect of vouchers on public schools
              Greene (2001) Greene and Winters (2003), Chakrabarti (2005), Figlio and Rouse (2006),
               West and Peterson (2006)
              Goldhaber and Hannaway (2004)
              Hoxby (2003a, 2003b), Chakrabarti (2005)

              Most of the studies above investigate effect of vouchers on students and
               school scores in public schools. This study examines some of the alternative
               ways in which public schools facing the threat of vouchers behaved.
Institutional Details of the Program
   Florida Program Embeds a voucher program within an accountability system

   Testing System
        FCAT Reading and Writing  grades 4, 8, 10
        FCAT Math  grades 5, 8, 10
        FCAT grade 4 Reading and grade 5 Math: Score Range  100-500, 5 levels 1-5
        FCAT grade 4 Writing: Score Range  1-6, 5 levels 1-5

   Grading Criteria
        School grades based on student scores in reading, math and writing
        Grade F → if fails minimum criteria in reading, math and writing
        Grade D → if fails minimum criteria in one or two subjects

   Minimum Criteria:
        Read and Math → at least 60% students need to score at or above level 2
        Writing → at least 50% students need to score above 3 in writing

   Scores of regular students and students in only a few ESE and LEP categories count
    towards grade formation
Incentives Created by the Program and
Alternative Avenues of Public School
Response
   Focusing on students below the minimum criteria cutoffs
       Level 1 in Reading and Math
       Levels 1 and 2, and especially level 2 in writing

   Choosing between subject areas
       Did the threatened schools focus on a single subject area rather
        than equally on all?
       Focus on subject area closest to the cutoff ?
       Role of different extents of difficulties of subjects

   Reclassifying low performing students into “Exempt” ESE
    categories
       Costs and the McKay Scholarship program
Data
   Disaggregated school level data obtained from the Florida Department of
    Education

   FCAT grade 4 Reading, grade 5 Math and grade 4 Writing
        Data on % of students scoring in the different levels 1-5 in reading and math, 1998-
         2002
        Data on % of students scoring at 1, 1.5,..,6 in writing, 1994-96 and 1998-2002

   ESE data
        Data on % of students in each of the ESE categories in Florida
              15 categories for 1993-97, 21 categories for 1998-2002

   Data on School Accountability in Florida

   Data on Socio-economic characteristics of schools
        Grade distribution of students, race, sex, % of students eligible for free or reduced
         price lunches, real per pupil expenditure
Empirical Strategy


   Forming Treatment and Control groups
       1999 F schools  treatment group
            F schools from now on
       1999 D schools  control group
            D schools from now on


   Focus on three years after program

   Difference-in-differences estimation strategy and
    Regression discontinuity strategy
      Graphical Representation: Did threatened schools focus on
      students expected to score below cutoffs?
      Distribution of percentage of students in level 1 Math and
      level 2 Writing, 1999 and 2000 (F and D Schools)
          F schools                                      D schools                        F schools                                      D schools

.04                                                                             .04




.03                                                                             .03




.02                                                                             .02




.01                                                                             .01




      0               25                  50                         75   100         0               25                  50                         75   100
                            % in level 1, 1999, Math                                                        % in level 1, 2000, Math
                              Panel A                                                                         Panel B
          F schools                                      D schools                        F schools                                      D schools

.08                                                                             .06




.06




.04


                                                                                .02
.02




      0               25                  50                         75   100         0               25                  50                         75   100
                           % in level 2, 1999, Writing                                                     % in level 2, 2000, Writing
                              Panel C                                                                         Panel D
                                                      Graphical Representation (Continued)
                                                      Distribution of Math Scores by Treatment Status, 1999 and
                                                      2000

                                                  level1               level2   level3                                                          level1               level2   level3
                                                  level4               level5                                                                   level4               level5




                                                                                              Percentage of Students in Different Levels
Percentage of Students in Different Levels




                                             80                                                                                            80




                                             60                                                                                            60




                                             40                                                                                            40




                                             20                                                                                            20




                                             0             F Schools            D Schools                                                  0             F Schools            D Schools
                                             Distn. of Math Scores in 1999, by Treatment Status                                            Distn. of Math Scores in 2000, by Treatment Status
                    Graphical Representation (Continued)
                    The case of Reading and Math, 1999-2002
                            F                   D                                         F                  D
               70                                                             70



                                                                              60




                                                               % in level 1
% in level 1




               60

                                                                              50

               50
                                                                              40



               40                                                             30
                    1999           2000        2001     2002                       1999         2000         2001   2002

                           % in level 1, FCAT Reading                                     % in level 1, FCAT Math
                    Graphical Representation (Continued)
                    The case of Writing, 1999-2002

                           F                  D                                           F                  D
               30                                                             45




                                                               % in level 2
% in level 1




               20                                                             35




               10                                                             25




               0                                                              15
                    1999          2000        2001      2002                       1999          2000        2001      2002
                           % in level 1, FCAT Writing                                     % in level 2, FCAT Writing
Sensitivity Checks:
Existence of Pre-program trends
Sensitivity Checks (Continued):
Compositional Changes of Schools and Student Sorting
Sensitivity Checks (Continued)
Is Mean Reversion a Concern?

   Mean Reversion

       Exploiting pre-program data and institutional details of program

       Idea is to measure the extent of declines (if any) in level 1 (in
        reading and math) and level 2 (in writing) in schools that
        received an F grade in 1998 relative to schools that received a D
        grade in 1998 during 1998-99

       Since 1998 is pre-program period, this gain can be taken as mean
        reversion effect mean reversion corrected effects

       Assigning letter grades to schools: 98F and 98D schools
Is Mean Reversion a concern?
Regression Discontinuity Analysis
   An alternative way to get rid of mean reversion
   Quasi-experimental research design

   Highly non-linear and discontinuous relationship between the % of students scoring
    above a threshold and probability that the school’s students become eligible for
    vouchers in the near future

   Regression discontinuity strategy exploits the design of the Florida program and state
    grade assignment formula

        Consider F and D schools that failed minimum criteria in reading and math in 1999
        F schools also failed in writing, while D schools passed
        Discontinuous relationship between probability of treatment and % of students scoring above
         3 in writing
        Sharp discontinuity at 50%
        Discontinuity sample 1 ( 7 ) and Discontinuity Sample 2 ( 5)
        Also consider other corresponding discontinuity samples  both F and D schools fail
         minimum criteria reading and writing (math and writing)
                            Regression Discontinuity Analysis: Assignment to treatment
                            (Sample of F and D Schools that failed minimum criteria in reading and
                            math in 1999)



                   1                                                                    1




                                                                     Treatment Status
Treatment Status




                   0                                                                    0
                       15    25   35   45   55   65   75   85   95                          15   25   35   45   55   65   75   85   95

                             % at or above 3 in writing, 1999                                    % at or above 3 in writing, 1999
                                  Regression Discontinuity Analysis:
                                  Effect of the program on % of students
                                  scoring in level 1 Reading and level 1 Math


                             65                                                                          55
% in level 1 Reading, 2000




                                                                               % in level 1 Math, 2000
                             60                                                                          50




                             55                                                                          45




                             50                                                                          40
                                  40                50                    60                                  40               50                     60
                                       % at or above 3 in writing, 1999                                            % at or above 3 in writing, 1999
                                              Panel A                                                                    Panel B
                                  Regression Discontinuity Analysis:
                                  Effect of the program on % of students
                                  scoring in levels 1 and 2 in Writing

                             15                                                                             30




                                                                               % in level 2 Writing, 2000
% in level 1 Writing, 2000




                             10                                                                             25




                             5                                                                              20




                             0                                                                              15
                                  40                 50                   60                                     40                 50                   60
                                       % at or above 3 in writing, 1999                                               % at or above 3 in writing, 1999
                                              Panel A                                                                          Panel B
The problem of Underestimation:
Are D schools Untreated?
   D schools do not directly face the threat of vouchers
        They are close to getting an “F” likely to face an indirect threat
        Effects above are underestimates
        Problem likely to be more prominent in regression discontinuity analysis

   F versus C analysis

   Rescale the above treatment effects by difference in the probabilities of
    treatment of F and D schools (Wald estimator)

        Use pre-program data to calculate these probabilities
        Calculate these scaling factors separately for the full sample and the discontinuity
         samples
        Using this strategy, correcting for underestimation amounts to scaling up the diff-
         in-diff estimates by 1.15 and the RD estimates by 1.27
Is Stigma Effect of Getting the Lowest
Performing Grade driving the Results?
   F  lowest performing grade in the Florida grade scale
   If stigma associated with getting the lowest performing grade
        Response may be due to stigma effect rather than the effect of threat of vouchers

   Alternative strategies to address this issue:

   Exploit institutional details of the pre-program accountability system and pre-
    program data
        Pre-1999 accountability system  1-4 (1-low, 4-high)

   Exploit post-program grades
        None of the 1999 F schools received another F grade in either 2000 of 2001
        Stigma effect would not be applicable in 2001 and 2002, while the threat of
         voucher effect would still be applicable
Choosing between subjects with different
extents of difficulties versus focus on subjects
closer to the cutoff


   Rank subject areas in each F school in terms of distances
    from cutoff
       “low”, “mid” and “high”


   Standardize reading, math and writing scores (% of
    students below cutoff) by grade, subject and year to have
    means of 0 and standard deviations of 1
Are year specific shocks in writing driving the
results? Doing a Regression Discontinuity Analysis
Anecdotal Evidence

   Phone Interviews:

       Widespread beliefs among Florida school administrators that
        writing scores are much easier to improve than reading and math
        scores

       Team approach in writing writing across the curriculum

       School-wide projects related to writing

       Longer time blocks in writing
Anecdotal Evidence

   Phone Interviews with School Administrators:

       Focused on lower performing students in various ways


            enlisted the help of the corresponding parents

            Introduced guided reading for weaker students

            Introduced “cooperative learning group” combine children of
             different abilities in a single group, so that the lower performing
             students also learnt from the higher performing ones
Reclassifying low performing students in to
“Exempt” ESE categories
   Shifts in % ESE (total ESE classification as % of total enrollment)

   Shifts in classification in excluded categories relative to included
    categories


   Classification in mutable excluded versus included categories

   Strategy:
        Difference-in-Differences Analysis
             Check for Existence of Pre-program trends
        Regression Discontinuity Analysis
          Graphical Representation: Classification in to
          ESE categories
                              F                        D
                  25




                  20




                  15




                  10
                       1998        1999        2000          2001   2002
                                               YEAR
                                  Total ESE Classification


                              F                        D
                  15




                  10




                  5




                  0
                       1998        1999        2000          2001   2002
                                               YEAR
           Classification into Excluded Relative to Included ESE Categories

Figure 5. Classification into ESE Categories, F and D schools
Not much Evidence of Reclassification: Is this
consistent with incentives?
   Absence of reclassification does not imply that schools did not
    respond to incentives

   Costs of reclassification

        Approved by parents, group of experts (physicians, psychologists)

        Too much classification may lead to audits by Florida DOE

        Costs of Services

        McKay Scholarship program for disabled students
Conclusions
   This paper studies the behavior of public schools facing the threat of vouchers in
    Florida.

   The literature investigating the effect of voucher programs on public schools typically
    focuses on overall scores. In contrast, this paper tries to go inside the black box to
    investigate some of the ways in which public schools facing the threat of vouchers in
    Florida behaved

   Analyzing the incentives built in to the system, it examines whether the threatened
    public schools responded according to the incentives built into the system.

   It finds robust evidence that they did respond to incentives.
        First, relative to the D schools, the threatened public schools concentrated more on students
         expected to score below the high stakes cutoffs.
        Second, they concentrated relatively more on writing, which is regarded as the easiest subject
         area.
        There is not much evidence in favor of reclassification of students in to excluded categories.
         This is consistent with the existence of considerable costs associated with such
         reclassification.
Conclusions (Continued)
   Policy Implications


       Public Schools responded to incentives  Implies that policy can
        be targeted to induce public schools to behave in desirable ways

       The 2002 changes
Conclusions and Policy Implications
   This paper studies the behavior of public schools facing the threat of
    vouchers in Florida.

   Analyzing the incentives built in to the system, it examines whether
    the threatened public schools behaved strategically to respond to these
    incentives.

   It finds strong evidence that they did respond to incentives
    strategically.
        First, relative to the D schools, the threatened public schools concentrated
         more on students expected to score below the high stakes cutoffs.
        Second, they concentrated relatively more on writing, which is regarded as
         the easiest subject area.
        There is not much evidence in favor of strategic reclassification of
         students in to excluded categories.
Conclusions and Policy Implications
(Continued)
   Contributions:
        The literature investigating the effect of voucher programs on public schools
         typically focuses on overall scores
              In contrast, this paper tries to go inside the black box to investigate some of the ways in
               which public schools facing the threat of vouchers in Florida behaved

        First paper to investigate these kinds of strategic behaviors of schools facing
         vouchers
        First paper to use a quasi-experimental strategy--regression discontinuity design--
         to study strategic behaviors of schools

   Important Policy Implications
        Public Schools responded to incentives  Policy can be targeted to carve public
         school behavior in desirable ways
        If more attention on reading and math warranted calls for a change in grading
         rules to decrease weight of writing and increase those of reading and math
        If more attention on higher performing students desired calls for their inclusion
         in the computation of F and D grades

   Policy changes in 2002 suggest interaction of policy and public school response and
    that policy has been a response to public school response
Is there a Stigma Effect?

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:4
posted:10/1/2012
language:English
pages:44