Student Mobility and AYP by Z5HN4T6E

VIEWS: 7 PAGES: 21

									Student Mobility and AYP
                          Valid Score Study
                                    June, 2006



      Wendell Callahan & Claudette Inge, SDCOE
                              Ken Taylor, KCSS
                               Ted Price, OCDE
                           Vicki Barber, EDCOE


                                             1
The Problem
   County Offices of Education as LEAs
    experience student mobility of a magnitude
    much greater than school districts because of
    the nature of the COE-operated schools
   Higher student mobility makes accurate and
    statistically defensible determination of
    Adequate Yearly Progress (AYP) problematic



                                                    2
Percent Valid Method
 Data Source: Corrected 2005 California
  Department of Education Accountability
  Progress Report (APR) Data File
 Percent Valid Scores-
       Calculation:
                  Number of Valid Scores*
        (                                                          ) X 100
            Enrollment on First Day of Testing
    *students enrolled in the LEA or school continuously from CBEDS day to
     the first day of testing                                                3
Percent Valid Study Sample
   10,290 AYP records were processed
       School records without AYP data were not included
        in the analysis (n=156)
   AYP records included:
       1 state aggregate
       58 county offices of education
       14 COE direct-funded charter schools
       977 school districts
       273 school district direct funded charter schools


                                                            4
Percent Valid Results
                Figure 1. Who made AYP?
          100
                84.5%

           75
                                        62.9%
                                                Failed AYP
Percent    50
                                37.1%           Made AYP

           25           15.5%

            0
                  COEs           Districts


                                                        5
Percent Valid Results
       Figure 2. Distribution of Percent Valid
                    ELA Scores
                                                         97.9
             100

              75
Percent of                                                      COE
 Schools      50                     36.2
                            27.6                                District
              25                              17.2
                   10.3                                8.6
                      0.3      0       0.3       1.5
               0
                   0-19%    20-39%   40-59%   60-79% 80-100%
                       Valid Score Percentage Range


                                                                      6
Percent Valid Results
       Figure 3. Distribution of Percent Valid
          100
                    ELA Scores            97.9

                                             COE ASAM School
             75                              District ASAM School

Percent of                                   District School
             50   43.4
 Schools
                    26.7       29.6   29.5
             25                              19.7 21.6
                                                              12.9         9.2
                                                          2.6 1.5    4.6
                         0.3           0           0.3
             0
                   0-19%         20-39%        40-59%     60-79%     80-100%
                           Valid Score Percentage Range


                                                                                 7
Percent Valid Simulation
 Apply 70% Valid Score threshold to COE
  and School District AYP data
 Less than 70% Valid Scores poses a
  potential risk to the statistical integrity of
  the valid score sample for AYP
  determination



                                                   8
Simulation Results
        Figure 4. Simulation of Proposed 70% Valid
              ELA Score Threshold for LEAs
                              99.1
             100                                89.1
              80

Percent of    60                                                     COE
  LEAs                                                               District
              40

              20       10.9
                                                        0.9
               0
                   70% or more Valid ELA   Less than 70% Valid ELA
                          Scores                   Scores



                                                                           9
Valid Score Study
Purpose & Rationale
    Investigate the effect of low Percent Valid
    Scores on the generalizability of the COE Valid
    Score Sample
   If the valid scores do not represent a generalizable
    sample of student performance, then the validity of an
    AYP determination made on the basis of these scores is
    questionable
   Based on a possibly faulty AYP determination, County
    Offices of Education will be identified for Program
    Improvement (PI) and be subject to a range of potential
    interventions, including possible reorganization

                                                        10
Low Percent Valid Scores in San Diego
Resulted in A Biased Valid Score Sample

Table 1. Valid 10th Grade Scores by Referral Type
                               Valid scores        Annual Enrollment              Percentage
 Referral Type (WIC Section)
                                                                                  Differential
                               %          n        %               n
             300                   58         29       10.8                 407           47.2
             601                   34         17       18.4                 692           15.6
             602                    8          4       68.3                2565          -60.3


Table 2. Valid 10th Grade Scores by JCCS Program
                               Valid scores        Annual Enrollment              Percentage
       JCCS Program
                                                                                  Differential
                               %          n        %                   n
Residential Foster Care            48         24        1.2                  45           46.8
Delinquency Programs                2          1       63.8                2396          -61.8
Community Schools                  32         16       29.5                1110            2.5
Community Day Schools              16          8        5.2                 194           10.8


Source: SDCOE 2004-05 enrollment data                                                            11
Valid Score Study Method
   An explanation of the study, data collection
    instructions and an Excel spreadsheet were e-
    mailed to 34 County Offices of Education
   Data from COE Special Education programs
    were not included in the analysis
   11 COEs with fewer than 50 valid scores were
    not included in this phase of the study



                                                12
            County Offices of Education with
            Enrollment Less than 50
         Table 3. COEs with Enrollment Less than 50 (n=11)
                                                           Percent Valid Scores              Enrollment First Day of Testing
               County       Met All AYP Criteria?
                                                          ELA               Math                  ELA                Math
         Alpine            Yes                                100.0%           100.0%                       2                   2
         Inyo              No                                  47.4%            47.4%                      38                  38
         Lassen            No                                  22.2%            22.2%                       9                   9
         Mariposa          Yes                                 95.5%            95.5%                      22                  22
         Modoc             No                                  67.4%            61.2%                      49                  49
         Nevada            No                                  40.0%            50.0%                      10                  10
         Plumas            Yes                                100.0%           100.0%                       1                   1
         Sierra            No                                  71.4%            71.4%                       7                   7
         Siskiyou          Yes                                 11.8%              6.3%                     17                  16
         Trinity           No                                  70.3%            70.3%                      37                  37
         Tuolumne          Yes                                 13.3%            20.0%                      15                  15
         Note: LEAs with enrollment less than 50 are not subject to 95% Participation Rate criteria




                •A sample size of 50 or less makes accurate
                statistical decision-making difficult because of the
                presence of error
                                                                                                                                    13
Note: An enlargement of this table is included in the handout
Dealing with Less than 100 Valid
Scores: The Confidence Interval
   Currently, the CDE applies a confidence interval
    to the Percent Proficient when there are less
    than 100 Valid Scores
   The confidence interval is based on the
    Standard Error of the Proportion
   The Standard Error of the Proportion tells us
    how much error is associated with an observed
    proportion
       i.e., the margin of error associated with the percent
        proficient calculated from a small number of test
        scores
                                                                14
               County Offices of Education with
               Less than 100 Valid Scores
               Table 4. COEs with Fewer than 100 Valid Scores (n=38)
                                                                Percent Valid Scores           Enrollment First Day of Testing      Number of Valid Scores
                     County       Met All AYP Criteria?
                                                                ELA              Math               ELA                Math            ELA          Math
               Alameda           No                                 16.8%            16.3%                 202                 203           34           33
               Alpine            Yes                               100.0%           100.0%                    2                  2            2            2
               Amador            No                                 90.9%            82.7%                 110                 110          100           91
               Colusa            Yes                                67.1%            68.5%                   73                 73           49           50
               Contra Costa      No                                 29.8%            29.7%                 168                 172           50           51
               Fresno            No                                 12.1%            12.1%                 398                 396           48           48
               Glenn             No                                 59.2%            59.6%                 157                 156           93           93
               Humboldt          No                                 47.4%            29.8%                 114                 114           54           34
               Imperial          No                                 47.7%            46.7%                 195                 195           93           91
               Inyo              No                                 47.4%            47.4%                   38                 38           18           18
               Kings             No                                 23.7%            21.2%                 118                 118           28           25
               Lake              No                                 32.8%            30.3%                   67                 66           22           20
               Lassen            No                                 22.2%            22.2%                    9                  9            2            2
               Madera            No                                 23.8%            26.9%                 160                 160           38           43
               Marin             No                                 48.1%            48.1%                 206                 206           99           99
               Mariposa          Yes                                95.5%            95.5%                   22                 22           21           21
               Mendocino         No                                 42.4%            41.0%                 165                 166           70           68
               Modoc             No                                 67.3%            61.2%                   49                 49           33           30
               Mono              Yes                                94.5%            94.5%                   73                 73           69           69
               Monterey          No                                 47.0%            47.9%                 164                 165           77           79
               Napa              No                                 63.4%            65.1%                   82                 83           52           54
               Nevada            No                                 40.0%            50.0%                   10                 10            4            5
               Plumas            Yes                               100.0%           100.0%                    1                  1            1            1
               Sacramento        No                                 14.4%            13.5%                 327                 326           47           44
               San Benito        No                                 42.9%            41.8%                   98                 98           42           41
               San Luis Obispo   No                                 40.8%            39.1%                 179                 179           73           70
               Santa Barbara     No                                 32.1%            32.1%                 246                 246           79           79
               Santa Cruz        No                                 39.1%            41.0%                 156                 156           61           64
               Sierra            No                                 71.4%            71.4%                    7                  7            5            5
               Siskiyou          Yes                                11.8%             6.3%                   17                 16            2            1
               Solano            No                                 19.2%            20.8%                 120                 120           23           25
               Sonoma            No                                 39.3%            38.1%                 224                 223           88           85
               Sutter            No                                 34.5%            34.5%                   87                 87           30           30
               Tehama            Yes                                47.9%            46.6%                   73                 73           35           34
               Trinity           No                                 70.3%            70.3%                   37                 37           26           26
               Tuolumne          Yes                                13.3%            20.0%                   15                 15            2            3
               Ventura           No                                 26.8%            26.5%                 190                 189           51           50
               Yolo              No                                 70.0%            68.6%                   70                 70           49           48
               Note: LEAs with less than 100 Valid Scores have adjusted Annual Measureable Objectives (AMOs) based on a Confidence Interval (CI)
                                                                                                                                                               15
Note: An enlargement of this table is included in the handout
Valid Score Study Method
   After calculating the Standard Error of the Proportion,
    the valid score sample was deemed unrepresentative of
    the COE’s annual enrollment if:
      The Standard Error exceeded .05 (e.g., more than 5
        percent margin of error) for at least one school or
        program type
   In this case, the Standard Error of the Proportion (i.e.,
    margin of error) tells us how much confidence we
    should place in the sample for further analysis and
    decision-making.

                                                           16
Valid Score Study
Preliminary Results
    100% of COEs had unrepresentative
    valid score samples
   22 (out of 22) had unrepresentative valid score
    samples based on school or program type
   18 (out of 20) had unrepresentative valid score
    samples based on grade span
       2 COEs did not report complete grade span data



                                                         17
 Valid Score Study
 Preliminary Results Summary Table
Table 5. COE Sample Representativeness Based on Percent Valid Scores, School Type and Grade Span (n=22)

                                     Met all     # AYP      # AYP      Percent  < 70%      School Type    Grade Span
        County             Class      AYP       Criteria   Criteria   Valid ELA  Valid    Representative Representative
                                     Criteria     Met      Possible    Scores   Scores?      Sample         Sample
Los Angeles                  I         No             18         30          44   Yes          No             Yes
Alameda                      II        No              6         14          17   Yes          No             No
Kern                         II        No              9         22          40   Yes          No             No
Orange                       II        No             11         30          48   Yes          No             Yes
Riverside                    II        No              8         20          39   Yes          No             No
San Bernardino               II        No             10         24          26   Yes          No             No
San Diego                    II        No              6         20          24   Yes          No             No
Sacramento                   II        No              5         16          14   Yes          No             No
San Joaquin                 III        No              4         21          42   Yes          No             No
San Mateo                   III        No             12         24          48   Yes          No             No
Ventura                     III        No             10         12          27   Yes          No             No
Stanislaus                  III        No             14         20          36   Yes          No             No
Santa Barbara               III        No              5         12          32   Yes          No             No
Butte                       IV         No             14         20          61   Yes          No             No
Imperial                    IV         No              1         12          48   Yes          No             No
Merced                      IV         No              2         18          41   Yes          No             No
Placer                      IV         No             14         18          65   Yes          No             No
Santa Cruz                  IV         No              8         12          39   Yes          No             No
San Francisco               IV         No              3         22        69.8   Yes          No              *
Shasta                       V         No              8         16          42   Yes          No             No
Mendocino                   VI         No              8         12          42   Yes          No              *
Del Norte                   VII        No              5         18          65   Yes          No             No
*enrollment data incomplete or under review

Note: An enlargement of this table is included in the handout                                                        18
Tentative Conclusions
   High student mobility resulting in low Percent
    Valid Scores appears to influence the
    composition of the valid score sample
   The resulting valid score sample is typically not
    proportionally representative of the students
    enrolled in the school, program and/or grade
    span of the COE on an annual basis
   The disproportionality of the valid score sample
    potentially introduces significant amounts of
    error into the AYP determination
                                                    19
Discussion
   Propose expansion of current use of confidence
    interval method or County/District averages to
    LEAs and schools based on 70% Valid Score
    threshold
   Propose & pilot alternative AYP method based
    on more appropriate and accurate
    measurement criteria
       i.e., ASAM Plus
       Include longitudinal (historical), value-added and
        transition measures

                                                             20
   Appendix 1: Preliminary Sample Error
   Estimates for County-Operated Programs
                  Figure 5. Average Margins of Error of COE Sample
                          Proportions for Court, Community
                                and 10th Grade Span
           0.50
                         0.39
           0.40

Standard
           0.30
                                           0.16
  Error
           0.20
                                                             0.10
           0.10
           0.00
                   Court Schools   Community Schools   10th Grade Span
                      (n=2 2)           (n=2 2)             (n=2 0)

                                                                         21

								
To top