2008 STAR Interpreting and Using Results by qao20272

VIEWS: 18 PAGES: 70

									      2008 STAR
Interpreting and Using
        Results
   August/September 2008
                        Objectives

Workshop participants will be able to:
• Describe the purposes of STAR reports
• Interpret STAR results
• Explain key statistics
• Compare and contrast types of reports
• Identify proper uses of reports




August/September 2008     Post-Test Workshop   2
                        Agenda

•   Quiz
•   Post Test Guide
•   What’s New?
•   Results and Statistical Analysis
•   Using Results
•   Summary and Internet Reports
•   Individual Student Reports
•   Teacher Reports



August/September 2008   Post-Test Workshop   3
                   Post-Test Guide Layout                     Post-Test Guide
                                                               Pages (Pp) i–iii

• Part I: General Info                • Part III: Appendices
      – New in 2008                             – Reporting clusters
      – Interpreting
        Reports                                 – Scale score ranges
      – Comparing Results                       – Statistical tables
                                                – Test Score
• Part II: Report                                 Verification
  Descriptions                                    procedures
      –   Summary
                                                – Demographic Data
      –   Individual
                                                  Correction
      –   Teacher
                                                  procedures
      –   Internet
                                                – California reading
                                                  list
                                                – Glossary
August/September 2008      Post-Test Workshop                       4
                        What’s New in 2008
                                                   Post-Test Guide
                                                               P2
• CMA reports
      – Percent correct
      – Grades 3–5
• STS reports
      – Percent correct
      – Grades 5–7 added
• CAPA Science percent correct
• Ethnicity for Economic Status subgroup
  summary (CSTs only)
• STAR Student Reports sorted for district
      – All without addresses
      – All with addresses
August/September 2008         Post-Test Workshop        5
                            Results:             Post-Test Guide
                    Purposes of STAR Reports                 P3


• Report progress of students toward
  proficiency on the state’s academic
  content standards
• Inform where improvement needed
      – To help students’ achievement
      – To improve educational programs
• Provide data for state and federal
  accountability programs




August/September 2008       Post-Test Workshop        6
           Results: Performance Levels
                                             Post-Test Guide
                                                         P7
• State goal: PROFICIENT or higher
   – Scale score of 350 or higher for CSTs
   – Scale score of 35 or higher for CAPA
• Other performance levels:
   – Advanced
   – Basic (300 CSTs, 30 CAPA)
   – Below Basic
   – Far Below Basic
   – CST and CAPA cut points vary for
     Advanced, Below Basic, and Far
     Below Basic by
      • Subject
      • Grade
August/September 2008   Post-Test Workshop        7
                        Results: Scale Scores       Post-Test Guide
                                                              P 7–8


• Scale scores allow same score to mean
  same thing across test forms within
  grade and content area and across time
• Scale score ranges by program:
      – CSTs: 150–600 for each grade and
        subject
      – CAT/6 Survey: 1–999 for each grade and
        subject
      – CAPA: 15–60 for each grade and
        subject
      – CMA, STS, CAPA Science: Not available
        until 2009
August/September 2008          Post-Test Workshop        8
                        Results: Equating
                                                  Post-Test Guide
                                                           Page 7
• Psychometric procedure
• Adjusts for test difficulty
• Additional information in CST Technical Report
  on CDE Web site
• For 2008
   – CSTs and CAPA (except science) are equated
   – CMA (3–5), CAPA (science), and STS (2–4) and
     will be equated starting in 2009
• CAT/6 nationally norm-referenced; not equated
  for STAR




August/September 2008        Post-Test Workshop        9
              Results: Reporting Clusters Post-Test Guide
                         (Content Areas)                     P 36
                                                  Appendices A & B

• Four to six clusters for each content area (subject)




• May be useful as indicators of individual or group
  strengths and weaknesses
• But… Reporting clusters
  should be interpreted with
  caution

 August/September 2008       Post-Test Workshop         10
              Results: Reporting Cluster      Post-Test Guide
                       Cautions                       P 8–10
                                             Appendices A & B

• Based on small numbers
  of items; therefore may not
  be reliable or generalizable
• NOT equated from year to
  year
• Should not compare
  reporting cluster percent
  correct from year to year



August/September 2008   Post-Test Workshop         11
          Interpreting Reporting Clusters or
                                           Post-Test Guide
                    Content Areas               P 8–10; 84

                  in the Same Year        Appendices A & B


• CSTs—Performance on the clusters for
  students statewide scoring from
      – Lowest score for proficient to
      – Lowest score for advanced




August/September 2008    Post-Test Workshop        12
        CSTs 2008 Reporting Clusters — Number of
         Questions and Average Percent CorrectPost-Test Guide
                                                       Page 138
                                                    (Appendix A)




August/September 2008     Post-Test Workshop          13
                                   Examples—Interpreting Reporting
                                    Clusters for CST for Geometry Post-Test Guide
                                                                                                                                                                               P 9
            My Class 2007     Just Proficient Students 2007   Just Advanced Students 2007

                100%
                 90%
                 80%
Avg % Correct




                 70%
                 60%
                 50%
                 40%
                 30%
                 20%
                 10%
                  0%
                                                                                                        My Class 2008   Just Proficient Students 2008   Just Advanced Students 2008
                         Logic/Geom    Volume & Area         Angle         Trigonometry
                           Proofs                         Relationships                                     100%
                                                                                                             90%
                                                                                                             80%


                                                                                            Avg % Correct
                                                                                                             70%
                                                                                                             60%
                                                                                                             50%
                                                                                                             40%
                                                                                                             30%
                                                                                                             20%
                                                                                                             10%
                                                                                                              0%
                                                                                                                   Logic/Geom    Volume & Area          Angle            Trigonometry
                                                                                                                     Proofs                         Relationships




                       August/September 2008                                      Post-Test Workshop                                                                14
                        Using Results           Post-Test Guide
                                                            P3


• For instructional decisions with other
  data: CSTs, CMA, CAPA, STS
• Used in Academic Performance Index
  (API) calculations: CSTs, CAT/6 Survey,
    CAPA
• Used in adequate yearly progress (AYP)
  calculations for grades 2–8 in ELA and
  math for CSTs and 2–8 and 10 in ELA
  and math for CAPA



August/September 2008      Post-Test Workshop        15
               Year-to-Year Comparisons            Post-Test Guide

        Do Compare Same Grade and Same
                                                         Pp 11–14


                 Content Area

• Mean scale score
  – Same content and grade, varying years
• Percent in each performance level
  – Same content by grade across years
             • e.g., 2007 ELA grade 10 with 2008 ELA
               grade 10




August/September 2008       Post-Test Workshop          16
               Year-to-Year Comparisons
          Do Compare Same Grade or Same
                  Content Area        Post-Test Guide
                                            Pp 11–14

• Percentage of students scoring at
  PROFICIENT and above
      – For a given grade and subject, e.g., Percent
        proficient and above for grade 3 math in 2007
        & 2008
      – For a given subject and aggregated grades,
        e.g., % proficient and above for grades 2 to 6
        math in 2007 & 2008
      – Across grades and a subject, e.g., % proficient
        and above in all courses and all grades




August/September 2008   Post-Test Workshop          17
               Year-to-Year Comparisons         Post-Test Guide
                    DON’T Compare                     Pp 11–14


• Individual scale scores or statistics based on
  scale scores for different grades or content
  areas
      – Subjects by grade are independently scaled
      – Different content standards are measured in
        different grades
• Cohorts across grades
• CMA, STS scores because not
  scaled




August/September 2008   Post-Test Workshop            18
       Example—Using CST Results to                           Post-Test Guide
                                                                         P 12
Compare Grade Results from Year to Year

                        2007 CST             2008 CST
                         for ELA              for ELA
Grade                   % Prof or            % Prof or     Difference
                         Above                Above
Grade 2                   31%                        35%      4%
Grade 3                   33%                        33%      0%
Grade 4                   29%                        31%      2%
Grade 5                   34%                        32%     -2%
Grade 6                   31%                        32%      1%
All Grades                32%                        33%      1%

August/September 2008           Post-Test Workshop                 19
               Aggregate (Summary) Reports     Post-Test Guide
                                                     Pp 17–21


• What are they?
      –   Student Master List Summary
      –   Student Master List Summary EOC
      –   Subgroup Summary
      –   Group Summary: CAT/6 Survey
• Report Emphasis: California Standards Tests
  (CSTs)
      – Criterion-referenced tests
      – Progress is measured in percent of students
        scoring proficient and advanced




August/September 2008    Post-Test Workshop         20
         Student Master List Summary
                                                Post-Test Guide
                                                  Pp 17; 26–37
•   By grade
•   CSTs, CMA, CAPA, and STS
•   Lists subjects
•   % and # at each Performance Level
      – Not for CMA, CAPA Science, STS
• Mean Scale Score
      – Not for CMA, CAPA Science, STS
• Reporting Cluster: Mean Percent Correct
      – Not for CMA, CAPA, and STS grades 5–7




August/September 2008   Post-Test Workshop           21
               Student Master List Summary    Post-Test Guide
                                                         P 27




August/September 2008    Post-Test Workshop        22
               Student Master List Summary    Post-Test Guide
                                                         P 27




August/September 2008    Post-Test Workshop        23
                        Who Counts?             Post-Test Guide
                                                           P 28



• Number Enrolled
      – All CST, CMA, CAPA, and STS multiple-
        choice answer documents submitted as
        scorable
      – Minus
         • Documents marked as ―Student
           enrolled after the first day of testing
           and was given this test‖




August/September 2008      Post-Test Workshop        24
                        Who Counts?                Post-Test Guide
                                                              P 28



• Number, Percent Tested
      – All CST, CMA, CAPA, STS answer documents
        with one or more answers
      – Plus
         • Z = Tested but marked no answers
      – Not included
         • P = Parent/guardian exemptions
         • A = Students absent
         • T = Enrolled first day, not tested, tested at
           previous school
         • E = Not tested due to significant medical
           emergency
         • Students with inconsistent grades
August/September 2008      Post-Test Workshop           25
                           Who Counts?             Post-Test Guide
                                                              P 27


• Number Valid Scores
  – For the subject, number of students tested
    at grade level who received a score for the
    test (that is, a scale score or percent
    correct).
  – Not included:
     • Incomplete tests
     • Modified tests
     • Non-English learners who took the STS
     • Unknown EOC math or science tests
     • Inconsistent grades
  Why does STS not have 100% valid scores?
   August/September 2008      Post-Test Workshop        26
                        Who Counts?             Post-Test Guide
                                                           P 28


• Number Tested with Scores
  – All tests taken, including those taken
    with modifications, that receive a score
  – All STS and CMA tests taken
  – Not included
     • Incomplete tests
     • Non-English learners who took the
       STS
     • Unknown EOC math or science tests
     • Inconsistent grades


August/September 2008      Post-Test Workshop        27
                        Who Counts?             Post-Test Guide
                                                           P 29



• Performance Level Counts for CSTs
      – Advanced, Proficient, Basic, Below
        Basic
         • All valid scores falling in the
           performance level
      – Far Below Basic
         • All valid scores falling in the
           performance level
         • Modified tests (in aggregate reporting
           and accountability only)

August/September 2008      Post-Test Workshop        28
               Student Master List Summary    Post-Test Guide
                   Performance Levels                    P 30




August/September 2008    Post-Test Workshop        29
               Student Master List Summary     Post-Test Guide
                    Reporting Clusters                    P 30




Compare to:
Average percent correct range for students
statewide who scored proficient on the total
test
August/September 2008    Post-Test Workshop         30
          Student Master List Summary:
                    Writing         Post-Test Guide
                                               P 30




B     =    Blank                        R = Refusal
C     =    Copied Prompt                T = Off Topic
I     =    Illegible                    W = Wrong prompt (Prompt
L     =    Language other than              from an earlier
           English                          administration)




August/September 2008        Post-Test Workshop              31
                   Student Master List            Post-Test Guide
                 Summary: End-of-Course                 Pp 32–37


• By Subject
• Lists each grade eligible to take test
• Math (Grades 7–11)
      – General Math, Algebra I, Geometry, etc.
• Science (Grades 9–11)
      – Earth Science, Biology, Chemistry, etc.
• History–Social Science (Grades 9–11)
      – World History
• Same statistics as grade-level Student Master
  List Summary




August/September 2008    Post-Test Workshop            32
             Subgroup Summary:    Post-Test Guide
         CSTs, CMA, CAPA, and STS       Pp 38–64


       • Disability Status
              – Based on ―Disability Code‖
       • Economic Status
              – Based on ―NSLP‖ & ―Parent Ed Level‖
       •    Gender
       •    English Proficiency
       •    Primary Ethnicity
       •    Ethnicity for Economic Status: CSTs
            and CAPA (ELA and Math) only


August/September 2008     Post-Test Workshop   33
              Subgroup Summary:       Post-Test Guide
                                            Pp 60–64
          Ethnicity for Economic Status




August/September 2008   Post-Test Workshop     34
                    Subgroup Summary:
               Ethnicity for Economic Status   Post-Test Guide
                                                     Pp 60–64




August/September 2008     Post-Test Workshop        35
            Subgroup Summary:
Ethnicity for Economically Disadvantaged
                                             Post-Test Guide
                                                Pages 60–64




August/September 2008   Post-Test Workshop        36
                        Internet Reports          Post-Test Guide
                                                      Pp 123–134


• Summaries based on same data as
  paper reports: CSTs, CMA, CAPA, STS
• Available to public online
• More subgroups than paper reports
      – English-language fluency
      – Parent education
      – Special Program Participation
• Access from http://star.cde.ca.gov



August/September 2008        Post-Test Workshop        37
                        Internet Reports:                 Post-Test Guide
                        Available Reports                     Pp 123-134

• CSTs                                  • CAPA
   – Mean scale scores                     – Mean scale scores
   – Percents by                           – Percents by
     performance levels                      performance levels
• CST Summary                              – Average percent
   – Percent proficient                      correct for science
     and advanced                       • CAT/6 Survey
• CMA                                      – Mean scale scores
   – Average percent                       – NPR for ―Avg.
     correct                                 Student‖
   – Grades 3–5 only                       – Percent above
• STS                                        percentile cuts
   – Average percent                       – Grades 3 and 7
     correct                                 only
   – Grades 2–7 only
August/September 2008        Post-Test Workshop                38
                        Internet Reports          Post-Test Guide
                         Access Reports                   Pp 124




August/September 2008        Post-Test Workshop        39
                        Internet Reports:         Post-Test Guide
                           CST Sample                       P 127




August/September 2008        Post-Test Workshop        40
                          Internet Reports:        Post-Test Guide
                        CST Summary Sample                   P 128




August/September 2008         Post-Test Workshop        41
                        Internet Reports:         Post-Test Guide
                              CMA                           P 129

                            % Correct




August/September 2008        Post-Test Workshop        42
                        Internet Reports:            Post-Test Guide

                              STS                            Pp 133


                            % Correct




August/September 2008           Post-Test Workshop        43
                        Exercise

• Work with a partner or in small groups
• Plan what data you’ll present to leaders
  in your district
• Remember the Do’s and Don’ts and
  Cautions




August/September 2008    Post-Test Workshop   44
Break — 15 minutes
                        Individual Reports         Post-Test Guide
                                                        Pp 70–112
• STAR Student Record Label
  – Adhesive label to affix to student’s
    permanent school record
• STAR Student Master List
  – Alphabetical list of students and their
    scores
  – Tests listed in order within grade
            •   CSTs-CAT/6 Survey
            •   CMA-CAT/6 Survey
            •   CAPA
            •   STS
• STAR Student Report: individual’s scores
      – 2 two-sided color copies
      – For parents/guardians
      – For each test
August/September 2008         Post-Test Workshop        46
                Student Record Label:      Post-Test Guide
            Student Name and Identification      Pp 70–73




August/September 2008     Post-Test Workshop        47
                 Student Record Label:        Post-Test Guide
                CST for Grade 10 Results            Pp 70–73




August/September 2008    Post-Test Workshop        48
             Student Master List:    Post-Test Guide
      CSTs/CMA-CAT/6 for Grade 3 Sample Pp 74–79




August/September 2008   Post-Test Workshop    49
             Student Master List:    Post-Test Guide
      CSTs/CMA-CAT/6 for Grade 3 Sample Pp 74–79




August/September 2008   Post-Test Workshop    50
                            Student Report:          Post-Test Guide
                        CST for Grade 11 Sample            Pp 81–90




August/September 2008           Post-Test Workshop        51
                            Student Report:          Post-Test Guide
                        CST for Grade 11 Sample            Pp 81–90




August/September 2008           Post-Test Workshop        52
                        Student Report:          Post-Test Guide
                    CSTs for Grade 10 Sample           Pp 81–90

                                                      Student
                                                      Name
                                                      On
                                                      Back




  ♦ =Percent correct obtained by the student on the
     reporting cluster/content area
▬▬▬ =Range of percent correct scores on the
     reporting cluster for students who scored
     proficient on the total test
August/September 2008       Post-Test Workshop        53
                  Reporting Cluster Range
              for Each Content Area by Grade

                              Reporting cluster average
                              percent correct from all
                              minimally advanced scores
                              minus one (e.g., 401, grade 4
                              math)


Reporting cluster average
percent correct from all
minimally proficient scores
(350)


 August/September 2008   Post-Test Workshop            54
            Explain to Parents
  Scale Score  Average % correct cluster
                score x 600
• Reporting Clusters not comparable
      – Different difficulty
      – Varying number of questions
      – Average % correct of clusters  % correct of
        total scale score
• Scale scores
      – Use conversion tables and other statistical
        techniques
      – Equating allows scores to have similar
        meaning
        (e.g., 350 = lowest score for CSTs proficient)
August/September 2008    Post-Test Workshop              55
                 Student Report:       Post-Test Guide
        CST for Grade 7 with CAT/6 Sample         P 85




August/September 2008   Post-Test Workshop      56
                  Student Report:       Post-Test Guide
         CST for Grade 11 with EAP Sample Pp 81–90




August/September 2008   Post-Test Workshop       57
                    Other Student Reports
• CMA (Pp 91–98)
   – Percent correct, no performance levels
   – Back, ―About the CMA,‖ P 93
   – No cluster reporting
• CAPA (Pp 99–104)
   – Back, performance level descriptors for ELA,
     math P 104
• STS (Pp 105–112)
   – In Spanish
   – Percent correct, no performance levels
   – Reporting clusters percent correct, grades 2–4
   – Back, how to use report P 108

August/September 2008      Post-Test Workshop   58
                        Exercise

• Work with a partner or in small groups
• Use a Student Report sample and
  explain content reporting diamonds
  and bars on back




August/September 2008    Post-Test Workshop   59
                        Unmatched Report

• Grade 3
      – CST but no CAT/6 Survey
      – CMA but no CAT/6 Survey
      – CAT/6 Survey but no CSTs or CMA
• Grades 4 and 7
      – Unmatched Multiple-Choice Report
         • CST Multiple-choice but no writing test
      – Unmatched Writing Report
         • Writing but no CSTs multiple-choice test




August/September 2008        Post-Test Workshop       60
                        Unmatched Report:




August/September 2008        Post-Test Workshop   61
                        Reports for Teachers        Post-Test Guide
                                                        Pp 113–122


• Summary of results for previous year’s class(es)
• Two types of summary reports
   – By grade level (e.g., grade 11 ELA) or course
     (e.g., Algebra I)
   – By teacher name or group name (e.g., GATE)
• Separate reports for:
   – CST ELA
   – CST Math
• Not available for:
   – CST science or history–social science
      – CMA
      – CAPA
      – STS


August/September 2008          Post-Test Workshop        62
                        Reports for Teachers         Post-Test Guide
                                                       Pp 113 – 122


• Individualized reports (teacher name or group
  name)
      – if name on SGID or Pre-ID
      – SGID priority over Pre-ID
• Group reports (grade level or course): all ELA
  and math teachers should receive
• EOC Math
      – Grades 8–11 by subject only
      – Not grade: e.g., All Algebra I in a school




August/September 2008          Post-Test Workshop         63
                        Report for Teachers:        Post-Test Guide
                              Sample                    Pp 114–122




August/September 2008          Post-Test Workshop        64
                        Report for Teachers:        Post-Test Guide

                         2008 Performance                   Pp 114


                        By Reporting Clusters




     Reporting clusters defined
     under this section.
August/September 2008          Post-Test Workshop       65
                        Report for Teachers:
                         Cluster Reporting
• For each reporting cluster, the average
  percent correct achieved by:
      –   Statewide students
      –   District students
      –   Your students
      –   Proficient range
• Definition of symbols
       = average % correct for teacher’s students
      ▬▬▬ = average % correct range for students
       statewide who scored proficient on total test



August/September 2008          Post-Test Workshop   66
                        Report for Teachers
                                                   Post-Test
                               Back                Guide P 117


• School % at each performance level by
  subgroups
• % Proficient and above
      – 2007 vs. 2008
      – School; district; state
      – By subgroup
• Resources




August/September 2008         Post-Test Workshop         67
     Early Assessment Program (EAP)
• Reporting Results
  – Student Statuses are on the STAR Report
  – Ready for CSU college-level English/mathematics
    courses
  – Ready for CSU college-level mathematics courses –
    Conditional
         • This status in conjunction with a grade of “C” or better in
           an approved math course or a supervised e-learning
           program can be exempt from the ELM
         • A list of courses and e-learning programs can be found at
           www.csumathsuccess.org
  – Not yet demonstrating readiness for CSU college-level
    English/mathematics courses

 August/September 2008        Post-Test Workshop               68
                        Summary
• Do’s and Don’ts
      – Do compare mean scale score, percent at
        performance levels within same grade, same
        content area
      – Don’t compare across grades, content area
•   Summary reports
•   Individual reports
•   Teacher reports
•   Quiz answers
•   Evaluations




August/September 2008    Post-Test Workshop      69
                        Quiz Answers

•   1.      B
•   2.      D
•   3.      A
•   4.      C
•   5.      D




August/September 2008      Post-Test Workshop   70

								
To top