2009 STAR
Interpreting and Using
Results
August 2009
Webcast
Objectives
Handout (HO) 1
Workshop participants will be able to:
• Describe the purposes of STAR reports
• Interpret STAR results
• Explain key statistics
• Compare and contrast types of reports
• Identify proper uses of reports
2 August 2009 Post-Test Workshop
Agenda HO 1
• What’s New?
• Results and Statistical Analysis
• Using Results
• Summary and Internet Reports
• Data CDs
• Individual Student Reports
• Teacher Reports
• Early Assessment Program
3 August 2009 Post-Test Workshop
What’s New in 2009 HO 2–3
• CMA reports
– Scale score, performance levels for
grades 3–5
– Percent correct for grades 6–8
– Writing scores for grades 4 and 7
• CAPA new cut scores for all including
scale scores for science
• STS reports
– Scale score, performance levels for
grades 2–4
– Percent correct for grades 5–11
• Data CD: two files only
4 August 2009 Post-Test Workshop
Results:
Purposes of STAR Reports HO 4
• Report progress of students toward
proficiency on the state’s academic
content standards
• Notify where improvement needed
– To help students’ achievement
– To improve educational programs
• Provide data for state and federal
accountability programs
5 August 2009 Post-Test Workshop
Results:
Performance Levels
HO 4
• State Goal: All students score at
proficient or higher
• CAPA Proficient: 35 or higher scale
score
• 350 or higher scale score
– All CST
– STS, grades 2–4
– CMA, grades 3–5
6 August 2009 Post-Test Workshop
Results:
Other Performance Levels
HO 4
• Advanced
• Basic Cut Score
– CAPA: 30
– CST: 300
– CMA, grades 3–5: 300
– STS, grades 2–4: 300
• Below basic
• Far below basic
• Cut points vary for
advanced and below basic by
– Subject
– Grade
7 August 2009 Post-Test Workshop
Results: Scale Scores HO 5
• Scale scores allow same score to mean same
thing across test versions within grade and
content area
• Scale score ranges by program:
– CST, CMA (grades 3–5), STS (grades 2–4):
150–600 for each grade and subject
– CAPA: 15–60 for each level and subject
8 August 2009 Post-Test Workshop
Results: Equating HO 5
• Psychometric procedure
• Adjusts for test difficulty
• Additional information in the CST
Technical Report on the CDE Web site
9 August 2009 Post-Test Workshop
Results: Reporting Clusters
HO 6–7
(Content Areas)
• Three to six clusters for each subject
• May be useful as indicators of individual or group
strengths and weaknesses
• But… Reporting clusters
should be interpreted with
caution
10 August 2009 Post-Test Workshop
Results: Cluster Percent Correct
Available for 2009
• CST: all
• CMA:
– Grades 3–5
– Grade 7: Writing only
• STS
– Grades 2–4
– Grades 5–7, but no comparison with
proficient students statewide
11 August 2009 Post-Test Workshop
Results: Reporting Cluster
HO 6–7
Cautions
• Based on small numbers
of items; therefore, may not
be reliable or generalizable
• NOT equated from year to
year
• Should not compare
reporting cluster percent
correct from year to year
12 August 2009 Post-Test Workshop
Interpreting Reporting Clusters or
Content Areas HO 6–7
in the Same Year
• Compare to percent correct range of
proficient students statewide
13 August 2009 Post-Test Workshop
CSTs 2009 Reporting Clusters:
HO 8
Number of Questions and
Average Percent Correct
To be finalized with complete data in
2009 post-test guide.
14 August 2009 Post-Test Workshop
Examples—Interpreting Reporting
Clusters for CST for Geometry
15 August 2009 Post-Test Workshop
Using Results
• For instructional decisions with other data
• Used in Academic Performance Index (API)
calculations: CSTs, CAPA, CMA (grades 3–5
only)
• Used in adequate yearly progress (AYP)
calculations, ELA and mathematics:
– Grades 2–8 CSTs
– Grades 3–5 CMA
– Grades 2–8 and 10 CAPA
16 August 2009 Post-Test Workshop
Year-to-Year Comparisons HO 9
Do Compare CSTs:
Same Grade and Same Content Area
• Mean scale score
– Same content and grade, varying years
• Percent in each performance level
– Same content by grade across years
• e.g., 2008 ELA grade 10 with 2009 ELA
grade 10
17 August 2009 Post-Test Workshop
Year-to-Year Comparisons
Do Compare CSTs: HO 9
Percent Proficient and Advanced
• Percentage of students scoring at PROFICIENT
and above
– For a given grade and subject, e.g., Percent
proficient and above for grade 3 math in 2008
and 2009
– For a given subject and aggregated grades,
e.g., Percent proficient and above for grades
2– 6 mathematics in 2008 and 2009
– Across grades and a subject, e.g., Percent
proficient and above in all courses and all
grades
18 August 2009 Post-Test Workshop
Year-to-Year Comparisons
DON’T Compare
HO 9
• Individual scale scores or statistics based on
scale scores for different grades or content
areas
– Subjects by grade are independently scaled
– Different content standards are measured in
different grades
• Cohorts across grades
• Across tests
• CAPA, CMA, STS to previous years
19 August 2009 Post-Test Workshop
Example—Using CST Results to
Compare Grade Results from Year to Year
2008 CST 2009 CST
for ELA for ELA
Grade % Prof or % Prof or Difference
Above Above
Grade 2 31% 35% 4%
Grade 3 33% 33% 0%
Grade 4 29% 31% 2%
Grade 5 34% 32% -2%
Grade 6 31% 32% 1%
All Grades 32% 33% 1%
20 August 2009 Post-Test Workshop
Aggregate (Summary) Reports
• What are they?
– Student Master List Summary
– Student Master List Summary EOC
– Subgroup Summary
• Report Emphasis: California Standards Tests
(CSTs)
– Criterion-referenced tests
– Progress is measured in percent of students
scoring proficient and advanced
21 August 2009 Post-Test Workshop
Student Master List Summary HO 10
• By grade
• CSTs, CMA, CAPA, and STS
• Lists subjects
• % and # at each performance level
• Mean scale score (CST, CAPA, CMA
grades 3–5, STS grades 2–4)
• Reporting cluster: mean percent
correct (CST, CMA grades 3–5, STS
grades 2–7)
22 August 2009 Post-Test Workshop
Student Master List Summary
HO 10
Grade 7 Sample
23 August 2009 Post-Test Workshop
Student Master List Summary HO 10
Basic Statistics
24 August 2009 Post-Test Workshop
Who Counts?
Number Enrolled HO 10
• All CST, CMA, CAPA, and STS multiple-
choice answer documents submitted as
scorable
• Minus
- Documents marked as “Student
enrolled after the first day of testing and
was given this test”
25 August 2009 Post-Test Workshop
Who Counts? Number Tested HO 10
• All CST, CMA, CAPA, STS answer documents
with one or more answers
• Plus
– Z = Tested but marked no answers
• Not included
– A = Students absent
– E = Not tested due to significant medical
emergency
– P = Parent/guardian exemptions
– T = Enrolled first day, not tested, tested at
previous school
– Students with inconsistent grades
– Non English learners who took STS
26 August 2009 Post-Test Workshop
Who Counts? Number and
HO 10
Percent Valid Scores
• Number Valid Scores
– For the subject, number of students tested at
grade level who received a score for the test
(that is, a scale score or percent correct).
• Percent Valid Scores
– For the subject, number of valid scores
divided by the number of students tested.
– Not included:
Incomplete tests
Modified tests
Non-English learners who took the STS
Unknown EOC mathematics (except grade 7
math) or science tests
Inconsistent grades
27 August 2009 Post-Test Workshop
Who Counts?
Number Tested with Scores HO 10
• All tests taken, including those taken
with modifications, that receive a
score
• Not included:
– Incomplete tests
– Non-English learners who took the
STS
– Unknown EOC mathematics (except
grade 7) or science tests
– Inconsistent grades
28 August 2009 Post-Test Workshop
Who Counts? HO 10
Performance Levels
• All CSTs, CAPA
• CMA Grades 3–5
• STS Grades 2–4
• Advanced, proficient, basic, below basic
– All valid scores falling in the performance level
• Far below basic
– All valid scores falling in the performance level
– CSTs taken with modifications (in aggregate
reporting and accountability only)
29 August 2009 Post-Test Workshop
Student Master List Summary HO 10
Performance Levels
30 August 2009 Post-Test Workshop
Student Master List Summary HO 10
Reporting Clusters
Compare to:
Average percent correct range for students
statewide who scored proficient on the total
test
31 August 2009 Post-Test Workshop
Student Master List Summary:
HO 10
Writing
B = Blank R = Refusal
C = Copied prompt T = Off Topic
I = Illegible W = Wrong prompt
L = Language other (Prompt from an
than English earlier administration)
32 August 2009 Post-Test Workshop
Student Master List Summary: HO 11
End-of-Course (EOC) CST and STS
• By subject
• Lists each grade eligible to take test
• Math (Grades 7–11)
– CST General Math, Algebra I, Geometry, etc.
– STS Algebra I and Geometry
• CST Science (Grades 9–11)
– Earth Science, Biology, Chemistry, etc.
• CST History–Social Science (Grades 9–11)
– World History
• Same statistics as grade-level Student Master
List Summary
33 August 2009 Post-Test Workshop
Student Master List Summary
HO 11
End-of-Course, Biology
34 August 2009 Post-Test Workshop
Subgroup Summary:
CSTs, CMA, CAPA, and STS
• Disability status
– Based on Disability Status for CST, CMA, STS
– CAPA: each disability code
• If missing, correct with demographic data
corrections
• Economic Status
– Based on NSLP eligibility and
parent education level
• Gender
• English proficiency
• Primary ethnicity
• Ethnicity for Economic Status (only for CSTs,
CMA grades 3–5, and CAPA)
35 August 2009 Post-Test Workshop
Subgroup Summary: HO 12
Ethnicity for Economic Status
36 August 2009 Post-Test Workshop
Subgroup Summary:
HO 12
Ethnicity for Economic Status
Example: Economically
disadvantaged for each ethnicity
37 August 2009 Post-Test Workshop
Subgroup Summary: HO 12
Ethnicity for Economically Disadvantaged
38 August 2009 Post-Test Workshop
Break — 10 minutes
Internet Reports
• Summaries based on same data as
paper reports: CSTs, CMA, CAPA, STS
• Available to the public online for
school, district, county, and state
• More subgroups than paper reports
– Parent education
– Special program participation
• Access from http://star.cde.ca.gov
40 August 2009 Post-Test Workshop
Internet Reports:
• CSTs Available Reports
– Mean scale scores • CAPA
– Percents by – Mean scale scores
performance levels – Percents by
• CST Summary performance levels
– Percent proficient • STS
and advanced – Grades 2–4: mean
scale scores,
• CMA percents by
– Grades 3–5: mean performance levels
scale scores, – Grades 5-11:
percents by average percent
correct
performance levels
– Grades 6-8:
average percent
correct
41 August 2009 Post-Test Workshop
Internet Reports
HO 13
Access Reports
42 August 2009 Post-Test Workshop
Internet Reports:
CST Sample HO 13
43 August 2009 Post-Test Workshop
Internet Reports:
CST Summary Sample
HO 14
44 August 2009 Post-Test Workshop
Internet Reports: CMA HO 14
45 August 2009 Post-Test Workshop
Internet Reports: STS
HO 15
CAPA
on HO
16
46 August 2009 Post-Test Workshop
Data CDs HO 18
• What are they?
– Lists of information from answer documents
and scores of every student in district
– In .txt format: wraps
• What are they used for?
– Searching for specific data
– Creating unique reports
– Verifying paper reports
• What else is needed?
– Text editor
– or Desktop application
– or Student Information System
47 August 2009 Post-Test Workshop
View of Data HO 18
• As .txt, word wrap on
• With text editor, word wrap off
48 August 2009 Post-Test Workshop
Organization of Data HO 18
•Two files:
– Demographics, special conditions, and test
scores
– Accommodations, modifications, English
Learners, and irregularities
•Data Layout = guide to location of data
on files
– Position
– Number of characters
– Whether numeric or alpha
49 August 2009 Post-Test Workshop
Data Layout Sample HO 18
50 August 2009 Post-Test Workshop
Individual Reports
• STAR Student Record Label
– Adhesive label to affix to student’s permanent
school record
• STAR Student Master List
– Alphabetical list of students and their scores
– Tests listed in order within grade
• CSTs
• CMA
• CAPA
• STS
• STAR Student Report: individual’s scores
– 2 two-sided color copies for each test
– For parents/guardians
51 August 2009 Post-Test Workshop
Student Record Label Grade 10 Sample:
Student Name and Identification HO 19
52 August 2009 Post-Test Workshop
Student Record Label:
CST for Grade 10 Results
HO 19
53 August 2009 Post-Test Workshop
Student Master List:
CSTs/CMA for Grade 3 Sample HO 20
54 August 2009 Post-Test Workshop
Student Master List:
CSTs/CMA for Grade 3 Sample
HO 20
55 August 2009 Post-Test Workshop
Student Report:
CST for Grade 10 Sample HO 21–22
56 August 2009 Post-Test Workshop
Student Report:
CST for Grade 10 Sample HO 21
57 August 2009 Post-Test Workshop
Student Report:
HO 22
CSTs for Grade 10 Sample
Student name
on back
♦ = Percent correct obtained by the student
on the reporting cluster/content area
▬▬▬ = Range of percent correct scores on the
reporting cluster for students statewide
58 August 2009
who scored proficient on that test
Post-Test Workshop
Student Report:
CST for Grade 11 HO 23
with EAP Results Sample
Label the location of the EAP report on HO 23.
59 August 2009 Post-Test Workshop
CMA Grade 7 Report Back HO 24
Label the location of the CMA Writing
Application results on HO 24.
60 August 2009 Post-Test Workshop
Explain to Parents
Scale Score Average % correct cluster
score x 600
• Reporting Clusters not comparable
– Different difficulty
– Varying number of questions
– Average % correct of clusters % correct of
total scale score
• Scale scores
– Use conversion tables and other statistical
techniques
– Equating allows scores to have similar
meaning
(e.g., 350 = lowest score for CSTs proficient)
61 August 2009 Post-Test Workshop
Other Student Reports HO 24–26
• CMA
– Performance levels grades 3–5
– Percent correct grades 6–8
– Back, “About the CMA,”
– Back, cluster reporting grades 3–5 and grade 7
writing
• CAPA
– Back, “About the CAPA”
• STS
– In Spanish
– Performance levels and scale scores reported
for grades 2–4
– Percent correct for entire test for grades 5–11
– Back, cluster percent correct for grades 2–7
– Back, how to use report
62 August 2009 Post-Test Workshop
Unmatched Report HO 28
• Grades 4 and 7
– Unmatched Multiple-Choice Report
• CST multiple-choice score but no writing
score
• CMA Multiple-choice score but no writing
score
– Unmatched Writing Report
• Writing score but no CST multiple-choice
score
• Writing score but no CMA multiple-choice
score
– Students receive 2 reports if writing score
not matched to multiple-choice score
63 August 2009 Post-Test Workshop
Unmatched Report: HO 28
64 August 2009 Post-Test Workshop
California Reports for Teachers
• Summary of results for previous year’s class(es)
• Two types of summary reports:
– By grade level (e.g., grade 11 ELA) or course
(e.g., Algebra I) for all schools
– By teacher name or group name (e.g., GATE) if
on Pre-ID or SGID (latter takes precedence)
• Separate reports for:
– CST ELA
– CST Mathematics
• Not available for:
– CST science or history–social science
– CMA
– CAPA
– STS
65 August 2009 Post-Test Workshop
California Reports for Teachers
• Individualized reports (teacher name or group
name)
– if name on SGID or Pre-ID
– SGID takes priority over Pre-ID
• Group reports (grade level or course): all ELA
and mathematics teachers should receive
• EOC Math
– Grades 7–11 by subject only
– Not grade-specific:
e.g., All Algebra I in a school
66 August 2009 Post-Test Workshop
Report for Teachers:
HO 29–30
Grade 4 Sample
67 August 2009 Post-Test Workshop
Report for Teachers:
HO 29
2009 Performance
By Reporting Clusters
Reporting clusters defined and focus
suggested under this section.
68 August 2009 Post-Test Workshop
California Report for Teachers:
Cluster Reporting
HO 29
• For each reporting cluster, the average
percent correct achieved by:
– Statewide students
– District students
– Your students
– Proficient range
• Definition of symbols
= Average % correct for teacher’s students
▬▬▬ = Average % correct range for students
statewide who scored proficient on that test
69 August 2009 Post-Test Workshop
California Report for Teachers:
HO 30
Back
• School % at each performance level by
subgroups
• % Proficient and above
– 2008 vs. 2009
– School, district, state
– By subgroup
• Resources
70 August 2009 Post-Test Workshop
Summary
• Do’s and Don’ts
– Do compare mean scale score, percent at
performance levels within same grade, same
content area
– Don’t compare mean scale scores across
grades, content area
• Summary reports
• Data CDs
• Individual reports
• Teacher reports
• Quiz answers
• Evaluations
71 August 2009 Post-Test Workshop
For more information see:
• http://www.cde.ca.gov/ta/tg/sr/
• http://www.star.cde.ca.gov
• http://www.startest.org/
• STAR Technical Assistance Center
800-955-2954
• Data CD Webcast:
http://webcast.sdcoe.net/ets052709a/index.htm
• CDE Accountability
– aau@cde.ca.gov
– 916-319-0863
72 August 2009 Post-Test Workshop
To contact CDE
• CDE STAR office: 916-445-8765
– English–Language Arts tests, including
writing: Jamie Contreras, jcontrer@cde.ca.gov,
916-319-0353
– Mathematics tests: Jane Liang,
jliang@cde.ca.gov, 916-322-1854
– History–Social Science tests: Nicole
Jespersen, njespersen@cde.ca.gov, 916-319-
0364
– CAPA, CMA: Don Killmer,
dkillmer@cde.ca.gov, 916-319-0350
– STS: Dianna Gutierrez, dgutierrez@cde.ca.gov,
916-319-0337
73 August 2009 Post-Test Workshop