Chicago Public Schools - CPS

Document Sample
Chicago Public Schools - CPS Powered By Docstoc
					Understanding Value-Added
   Lesson 2: How Value-Added Works




 Performance Management        CPS
What is the Value-Added Metric?
     Value-Added is the District’s measure of elementary school growth.
     Value-Added is a nationally recognized way of measuring growth.




                     Academic Growth = Student Learning
           2009                                                2010



           Emphasizes continual student improvement

           Provides information to understand what drives continual
            improvement


2
Measuring Growth, Not Attainment
                                        In this school, the percent meeting state
                                        standards is 25% in both Year 1 and Year 2.

                                        Attainment is unchanged – but are students
                                        learning?



                                                                    Analyzing growth provides this
                                                                    information




                                            Meets State Standards
                                                                                            (Year 2)

        200   210   220   230   240   250                           260    270      280     290      300


ISAT
Scale                                                                                       (Year 1)
Score

        200   210   220   230   240   250                           260    270      280     290      300



3
Accounting for Student Populations
     Student academic growth varies by grade, prior performance, and
      demographics.

     The goal of the Value-Added metric is to measure the school’s impact on
      student learning independent of student demographic factors.

     Value-Added accounts for the following student factors:

                 Prior ISAT Reading Score       Low-Income Status
                   Prior ISAT Math Score            ELL Status
                        Grade Level                 IEP Status
                          Gender                  Homelessness
                       Race/Ethnicity                 Mobility


     Controlling for the factors above gives proper credit for growth to low
      attainment schools and schools that serve unique populations.

4
How it Works
     Value-Added is not a comparison to similar schools.
            We do not look for a comparison group of schools that match each
             other on all 9 student factors…such a group might not exist.


     Rather, Value-Added compares growth of students in each school to
      growth of students across the District, controlling for the list of student
      factors.


     To do this, we utilize a regression methodology, developed in
      collaboration between CPS and academic experts from the University of
      Wisconsin.



5
Regression Lines
Regression shows how growth relates to another variable—in this case prior
performance on the ISAT.
     Scale Score Gain, 2009 to 2010




                                      25           All ISAT Math Scores for the District

                                      20

                                      15

                                      10

                                      5

                                      0
                                           125   175          225        275               325   375
                                                               2009 ISAT Score
6
Regression Lines
Regression shows how growth relates to another variable—in this case prior
performance on the ISAT.
     Scale Score Gain, 2009 to 2010




                                      25         3rd to 4th Grade ISAT Test Scores Only

                                      20

                                      15

                                      10

                                      5

                                      0
                                           125   175          225        275              325   375
                                                               2009 ISAT Score
7
Regression Lines
Regression shows how growth relates to another variable—in this case prior
performance on the ISAT.
        This line shows the average gain on ISAT math between 2009 and 2010 for 4th graders.
    It is downward-sloping because at higher levels of prior performance, average growth is smaller.
         Scale Score Gain, 2009 to 2010




                                          25

                                          20

                                          15

                                          10

                                          5

                                          0
                                               125   175   225        275     325          375
                                                            2009 ISAT Score
8
Regression Lines
Regression shows how growth relates to another variable—in this case prior
performance on the ISAT.
     Scale Score Gain, 2009 to 2010




                                      25         Repeat the process for each grade level.

                                      20

                                      15

                                      10

                                      5

                                      0
                                           125   175           225        275               325   375
                                                                2009 ISAT Score
9
Controlling for One Variable
                                  Gain of 4th to 5th Grade students at a single school controlling for prior
                                               performance compared to the District average

                                                        This student grew faster than other 5th grade
                                                        students with the same prior ISAT score.
 Scale Score Gain, 2009 to 2010




                                  25

                                  20

                                  15

                                  10

                                  5
                                             This student grew slower.
                                  0
                                       150        175             200            225             250
                                                            2009 ISAT Score
10
Controlling for Multiple Variables
Regression allows us to control for multiple factors at one time – in this case
prior performance and ELL status.

                                   This line shows the average gain for all students from 4th to 5th grade.
 Scale Score Gain, 2009 to 2010




                                  25

                                  20

                                  15

                                  10

                                  5

                                  0
                                       150       175           200          225           250
                                                         2009 ISAT Score
11
Controlling for Multiple Variables
Regression allows us to control for multiple factors at one time – in this case
prior performance and ELL status.

                                         Now we identify which students are English Language Learners
 Scale Score Gain, 2009 to 2010




                                  25

                                  20

                                  15

                                  10

                                  5

                                  0
                                       150        175          200          225         250
                                                          2009 ISAT Score
Controlling for Multiple Variables
Regression allows us to control for multiple factors at one time – in this case
prior performance and ELL status.

                    The blue line shows the average gain for ELL students between 4th and 5th grade.
 Scale Score Gain, 2009 to 2010




                                  25

                                  20

                                  15

                                  10

                                  5

                                  0
                                       150   175       200           225     250
                                                   2009 ISAT Score
13
Controlling for Multiple Variables
Regression allows us to control for multiple factors at one time – in this case
prior performance and ELL status.

The orange line shows the average gain of non-ELL students between 4th and 5th grade.
 Scale Score Gain, 2009 to 2010




                                  25

                                  20

                                  15

                                  10

                                  5

                                  0
                                       150   175       200           225   250
                                                   2009 ISAT Score
14
Controlling for Multiple Variables
Regression allows us to control for multiple factors at one time – in this case
prior performance and ELL status.
 Scale Score Gain, 2009 to 2010




                                  25

                                  20

                                  15

                                  10

                                  5

                                  0
                                       150   175       200           225   250
                                                   2009 ISAT Score
15
Controlling for Multiple Variables
Regression allows us to control for multiple factors at one time – in this case
prior performance and ELL status.
                                                     Although this student grew slower than other 5th graders with the
                                                   same pretest score, she grew faster than other ELL students with the
                                                                           same pretest score.
 Scale Score Gain, 2009 to 2010




                                  25

                                  20

                                  15

                                  10

                                  5

                                  0
                                       150   175             200              225              250
                                                      2009 ISAT Score
16
Controlling for Many Variables at Once
                                  Now, we can control for other factors besides prior performance for Student A.
 Scale Score Gain, 2009 to 2010




                                                         2009 ISAT Score

17
Controlling for Many Variables at Once
                                  Based on Student A’s demographics, adjustments are made
 Scale Score Gain, 2009 to 2010




                                               2009 ISAT Score

18
Controlling for Many Variables at Once

                                  Compared to similar students district-wide, Student A has above average gain
 Scale Score Gain, 2009 to 2010




                                                        2009 ISAT Score

19
Summary of Regression
 By measuring the impact of each student factor, the
  regression model isolates the impact of the school on student
  growth.


 In other words, some growth is explained by external factors.
  We can measure the average impact of these external factors
  on growth at the District level and subtract that impact from
  the school’s absolute growth.

 The growth that is left over after removing the impact of these
  factors is attributed to the school. This is the value added by
  the school.


20
Oak Tree Analogy

 For an illustrative example of regression, view the
  ―Oak Tree Analogy‖ presentation at:

     http://research.cps.k12.il.us/cps/accountweb/Research/ValueAdded/


 The Oak Tree presentation illustrates the Value-
  Added model by using an analogy of two gardeners
  tending to oak trees.




21
Some Things to Know
 Tested Students
      All students making normal grade progression who took ISAT in both the
       previous year and current year are included in analysis.

 Mobile Students
      Mobile students count towards the Value-Added score in each school they
       attended, but are weighted in the analysis by the amount of time they were in
       the school during the year.

 English Language Learners
      ELL students in Program Years 0 through 5 are excluded from the analysis.
      This includes students who were in PY0-5 during the pretest year, even if they
       have since exited the ELL program or moved to PY6.

 Students with Disabilities
      IEP status is differentiated by type of IEP.
      For example, the impact of a severe and profound disability is considered
       separately from the impact of a speech and language disability.


22
Value-Added Scores
Value-Added measures the difference between the growth of
students at a school and the growth of similar students across
the District.

     A positive score indicates a school or grade whose
     students are growing at a faster pace than similar
     students.


           Zero (0) is the District average. A score near zero
           indicates a school or grade whose students are growing
           at about the same pace as similar students.


                 A negative score indicates a school or grade whose
                 students are growing at a slower pace than similar
                 students.



23
Standardization of Scores
 Growth on the ISAT is measured in ISAT scale score
  points

                                      Student A “grew” by 35 ISAT scale
                                      score points
        200   210   220   230   240




 However, one ISAT scale score point of growth is
  more difficult to obtain in some grade levels than
  others.

 As a result, standardization is used to ensure that all
  Value-Added scores are on the same scale.

24
Standardization of Scores
 Standardization is a common statistical process. In
  this case, it is used to convert ISAT scale score
  points to a standard scale.

 The unit of measure is the ―standard deviation‖ which
  is a measure of distance from the mean.
        i.e., how much does School A’s score deviate from the mean?


 This places all scores on the same scale, allowing
  for more precise comparisons between scores at
  different grade levels.


25
The Standard Scale
      Features of the Standard Scale
       The scale ranges from approximately -6 to 6.
       Zero (0) is the District average.
       About 68% of scores fall between -1 and 1.
       About 95% of scores fall between -2 and 2.
       About 99% of scores fall between -3 and 3.
       Only about 1% of scores are less than -3 or more than 3.




                             34%     34%

           2.5%     13.5%                             2.5%
                                            13.5%


26
Reading the Value-Added Reports
                                                           Percentile: This is the
                      Value-Added Score                  percent of scores that fall
                                                        below this score. Percentiles
                                                           range from 0th to 99th
                                                                                        Performance Category:
                                                                                          This is based on the
                                                                                               percentile.




                                          Confidence Interval: This is
  Number of Students in
                                           explained in the next set of
  the calculation: This is
                                                     slides.
 weighted by the amount of
 time students were in the
    school between the
    pretest and posttest.


27
Confidence Intervals
 The Value-Added model controls for factors that CPS can
  measure, but there are some factors that cannot be
  measured, such as:

                    Motivation to learn
                    Family circumstances
                    Health

 In addition, the Value-Added model is a statistical estimation
  of the school’s impact on student learning and therefore
  contains a certain amount of random error.

 For these reasons, the Value-Added model includes
  confidence intervals.

28
Real World Example: Political Polling
 A Political Polling company surveys a representative random sample
 of 1,000 community households about for whom they are going to
 vote on Election Day. The question they pose is:

     If the election were held today, for whom would you cast
     your ballot?

 The percentages of responses breakdown as follows:
     Candidate Jones would receive 54% of the vote
     Candidate Smith would receive 46% of the vote
     There is a +/- 3% margin of error




29
Confidence Intervals in Political Polling
With the margin of error of +/- 3%, the range of the percentage of people who plan on
voting for each candidates is as follows:


                       Candidate Jones would receive between 51% and 57% of the vote.




     43%   44%   45%   46%   47%   48%   49%   50%   51%   52%   53%   54%   55%   56%   57%




     Candidate Smith would receive between 43% and 49% of the vote.



The confidence intervals do not overlap. Therefore the race is NOT ―too close to call.‖ We can
predict with a high degree of confidence that Candidate Jones will win the race.


30
Confidence Intervals in Value-Added
 A confidence interval is a range of scores around the Value-
  Added estimate.

                                             The   Value-Added estimate is 1.0.
 Example:                  1.0
              0.7                     1.3    The   confidence interval is ± 0.3.
                                             The   confidence interval range is from 0.7 to 1.3.



 We are 95% confident that the true Value-Added score falls
  within the confidence interval range.

 The confidence interval is ―n‖ dependent, meaning larger
  samples yield smaller confidence intervals.
        This is because in larger samples, a score that is different from the
         average is less likely to be due to random error alone.


31
Statistical Significance
 If the confidence interval does not include zero, we say that the score is
  statistically significant, meaning we are 95% confident that the score is
  different from zero.

 A color is associated with each score based on the statistical significance:
How Confidence Intervals are Reported
This is how Value-Added scores are displayed in the reports.

                                                             This school has a Value-Added
                                                                 score of -0.5 in reading
                                                          (the score is ½ of a standard deviation
                                                                     below the mean)




                      The confidence interval ranges from -1.9 to 0.8

            Because the confidence interval includes zero, we say that this school
              is not statistically different from zero at the 95% confidence level.

                            For that reason, the bubble is yellow.


33
Using Value-Added Information
 Performance Management
      As an assessment of school performance
      To identify areas needing additional support or professional
       development
      To identify best practice strategies for improving student growth


 School Accountability (i.e., Performance Policy)

 Additional Compensation Plans (i.e. Chicago TAP)


         In all of these applications, Value-Added is used as just one
             additional piece of information, along with other data.

34
For More Information

 More lessons and other resources for understanding
  Value-Added are available at:

     http://research.cps.k12.il.us/cps/accountweb/Research/ValueAdded/


         Lesson 2 (Part 2): Oak Tree Analogy

         Lesson 3: Technical Specifications of the Value-Added Regression
          Model




35

				
DOCUMENT INFO