Why Assessment Results Are Hard to Use
(and what to do about it)
Hi…I’m the Assessment Director
The Theory
What Could Go Wrong?
Look familiar?
Assessment Design
Choosing an Achievement Scale
Pick a scale that corresponds to the educational progress we expect to see. Example: 0 = Remedial work 1 = Fresh / Soph level work 2 = Jr / Sr level work 3 = What we expect of our graduates
Subjective measurements
For complex goals like general education:
Use authentic data that already exists Sacrifices reliability for validity Create subjective ratings in context Sacrifices reliability for validity Assess what you observe, not just what you teach! Recovers some reliability
www.coker.edu/assessment
Survey Attitudes and Behaviors
CIRP, BCSSE, Student satisfaction inventory
Link student ID numbers to other indicators
These can be easily and productively
outsourced
Analysis and Reporting
Some Tools
ANOVA
Logistic Regression and ROC curves
http://statpages.org/logistic.html
Pivot tables
Longitudinal Analysis
Use data you already have
Grades
FAFSA, SAT, ACT items
Demographic data ePortfolio statistics or work
Library circulation statistics
Analysis: Example
Writing by Class and Library Use
1.8 1.6 1.4
Avg. Writing Score
1.2 1 0.8 0.6 0.4 0.2 0 Fall 05 Fall 04 Start Term Fall 03 Fall 02 > Library < Library
You don’t have to average
Measurement requires:
Units Ability to aggregate
What we actually do is estimation Without measurement we shouldn’t average
Averages
Proportions
Min / max
Major Code Art Class Student ID Last (4) Senior xxxxxxx Tatiana Min Remedial Remedial Fr/So Remedial Max Jr/Sr Fr/So Jr/Sr Jr/Sr Avg 1.00 0.80 1.17 1.00 First Tolstoy Observations 7
Analytical Thinking Creative Thinking Writing Effectiveness Speaking Effectiveness
Confidence Intervals
Longitudinal Writing Effectiveness in Day
2.50
2.00
Avg. Score
1.50
AF F AF M WF
1.00
WM
0.50
0.00 2003-4 2004-5 2005-6 2006-7 Academic Year
Portfolio analysis: epic FAIL!
Grammar Scores
5.00 4.50 4.00
Average Score
3.50 3.00 2.50 2.00 1.50 1.00 0.50 0.00 07 06 Entering Year 05 04 Day Evening
Analysis: Example
Proportions
Day Analytical Min Rating
100% 80% 60% 40% 20% 0% 2003-4 2004-5 2005-6 2006-7 Jr-Sr Fr-Soph Remedial
Conclusions
Design it yourself
Use a sensible longitudinal scale
Combine with other data Avoid averages
Last Requests?
highered.blogspot.com