The Relationship between Measures of Teacher Quality and Student Achievement at Vaughn Next Century Leaning Center H. Alix Gallagher SRI International November 21, 2002 Research Context • Standards-based reform • Need to improve teacher quality • Traditional teacher evaluation inadequate – Typical principal evaluations of teachers have little to no correlation with student outcomes. – Traditional teacher evaluation is generic and based on a small sample of teacher work and so has limited usefulness for improving teacher quality. Research Questions • Are there differences in learning between classrooms? • What is the relationship between teacher evaluation scores (TES) and classroom effects? This study compares subject area teacher evaluation scores and student subject area standardized test scores for reading, math, and language arts. Caveats • There is more to student learning than what is measured by the Stanford-9. • There are many causes of differences in classrooms besides teachers. • While we do not believe it is appropriate to evaluate teachers using student test scores, the relationship between evaluation scores and test scores is one measure of the validity of the evaluation system. Method and Sample • Pre-test/Post-test model measuring value-added achievement on Stanford-9 • 34 second-fifth grade teachers • 584 students were tested in reading and math in 2000 and 2001; 532 students in language arts • Approximately 10% of students lost to mobility Subject Area TES Variable Mean Standard Minimum Maximum Deviation Average 3.24 0.39 2.30 3.85 Literacy TES Average 3.13 0.42 2.25 4.00 Math TES Average 3.24 0.42 2.30 3.95 Language Develop- ment TES Literacy Correlation between TES and classroom effect .545** (p<.01) 20 10 0 -10 -20 2.2 2.4 2.6 2.8 3.0 3.2 3.4 3.6 3.8 4.0 Teachers' Average Literacy Evaluation Score Mathematics Correlation between TES and classroom effect .239 20 10 0 -1 0 -2 0 -3 0 2.0 2.2 2.4 2.6 2.8 3.0 3.2 3.4 3.6 3.8 4.0 4.2 Teachers' Average Math Evaluation Score Language Arts Correlation between TES and classroom effect .175 30 20 10 0 -10 -20 -30 2.0 2.2 2.4 2.6 2.8 3.0 3.2 3.4 3.6 3.8 4.0 Teachers' Average Language Arts Evaluation Score Discussion • Because there are other causes of differences in learning between classrooms, we would not expect a perfect correlation • Additionally, the lack of alignment between measures of student learning and teacher practice reduce the correlation (e.g. Vaughn’s language development teacher evaluation rubric and the SAT-9 language arts test) What is the practical significance? • An improvement of one point in a teacher’s evaluation score in reading would be predicted to lead to an increase in growth of 12.94 scale points—or approximately 1/3 of an average year gain. • In math the predicted increase in growth would be around 7 points, while in language arts the impact would be around 5 points. Why is the relationship stronger in literacy? • Vaughn has invested in substantial professional development in literacy as opposed to mathematics. • Both evaluators and teachers appear more knowledgeable about literacy instruction than mathematics instruction. Discussion Findings: • Teachers make a difference! There are significant classroom effects at Vaughn. • The evaluation scores are correlated with classroom effects, especially in literacy. • Evaluators’ pedagogical content knowledge appears to have a significant impact on the validity of teacher evaluation Limitations • Small sample size. • One year of data. • Stanford-9 is only indicator of student learning. • Stanford-9 is poorly aligned with the Vaughn teacher evaluation system in language arts.