Early Learning in Mathematics (ELM)
The Efficacy of a Kindergarten Curriculum
Implemented in Whole Classroom Settings
Scott K. Baker, PhD
Pacific Institutes for Research / University of Oregon
Ben Clarke, PhD
Pacific Institutes for Research
Hank Fien, PhD
University of Oregon
Keith Smolkowski, PhD
Oregon Research Institute
Chris Doabler, PhD
Pacific Institutes for Research
David Chard, PhD
Southern Methodist University
IES Conference June 2010
Acknowledgements
Institute of Education Sciences
The research reported here was supported by the Institute of
Education Sciences, U.S. Department of Education, through
Goal 2 development grant, #R305K040081, and a Goal 3
efficacy grant, #R305A080114, to Pacific Institutes for
Research. The opinions expressed are those of the authors
and do not represent views of the Institute or the U.S.
Department of Education.
Additional Oregon Project Staff
Kathy Jungjohann / Karen Davis: Curriculum development
Mari Strand Cary / Rhonda Griffiths: Coordination and research
Chris Doabler: Observation measurement and research
Early Learning in Mathematics
(ELM)
• 4-year randomized efficacy control trial
• Measuring the efficacy of a kindergarten
mathematics curriculum in Oregon and Texas.
• Research Design
• Randomized Controlled Block Design
• Classrooms within school matched on full / half
day and randomly assigned to treatment (ELM)
or control conditions 3
Purpose of the 4-year project
• Study 1: Efficacy trial of the whole group
curriculum (ELM) on kindergarten students’
mathematics achievement
• Study 2: Efficacy trial of small group ELM
component – Roots – on the achievement of
at-risk students
• Examine potential mediation variables, dose-
response variables, and moderation factors
Structure of the Curriculum
• Daily Calendar Lessons / Activities
– 15 minutes daily, whole class “circle” time
– Monthly booklets with objectives and
application activities
• 120 Core Lessons divided into 4 quarters
– 30 minutes whole class instruction
– 15 minutes teacher directed written work
– End of quarter assessment of progress
ELM Instructional Content
• National Math Advisory Panel (2008)
recommends a focused, coherent progression
of mathematics learning with emphasis on
proficiency with key topics
• ELM focuses on key strands rather than a
broad array of mathematical content
– Numbers and Operations
– Geometry NCTM Curriculum Focal
Points for K (2006)
– Measurement
– Vocabulary (NCTM Process Standard, 2000)
ELM Conceptual Framework
Development of Mathematics-related
Mathematical Vocabulary and
Concepts / Models Discourse
Procedural Fluency
Research questions
• What is the immediate impact of ELM taught in
general education kindergarten classrooms on
mathematics achievement compared to standard
district practice?
• Is impact moderated by student level of risk for
mathematics difficulties?
• Does rate of teacher models or student practice
opportunities mediate a condition effect?
• Is there evidence of an interaction between
condition and student practice on student
outcomes?
Study Sample
• Assignment at the classroom level blocked on
school
– Districts = 3
– Schools = 24
• Intervention classrooms = 34
• Control classrooms = 30
• Students nested in classrooms
– Whole class instruction
– N = 1,349
Student Demographics
• 56.3% eligible for free or reduced lunch
• 38.4% English Learners
• 8.4% special education
• 49.5% White
• 36.4% Latino
• 4.8% Asian / Pacific Islander
• 2.3% African American
Hypothesized Model of ELM Impact
Intervention Mediators Proximal Distal
Outcomes Outcomes
Models of Math
Representations
Instruction
ELM Math Procedural Student Math
Quality and Math
Instruction Fluency Achievement
Content
Math Vocabulary
Descriptive data on implementation
• Classroom fidelity observations – treatment
and control
• Classroom observations focusing on
instructional interactions – treatment and
control
• Teacher logs addressing content coverage –
treatment and control classrooms
Implementation Fidelity
• General ratings (8 items)
– Models skills/concepts appropriately and with
ease
– Engages students in learning throughout the
lesson
• Uses ELM / completes all lesson activities
(dichotomous)
• For each ELM activity (range 1-7 per lesson):
Full (2) / Partial (1) / Not Taught (0)
Implementation Fidelity Data:
ELM Lesson Activities
• 81 ELM (fidelity) observations during the year
• Fall : mean = 1.71 (SD = .19)
• Winter: mean = 1.65 (SD = .33)
• Spring: mean = 1.62 (SD = .43)
• Overall: mean = 1.65 (SD = .36) (83% of Full)
• 2 of 81 lessons had a mean below Partial (1)
level of implementation
Student measures of impact
• Test of Early Mathematics Ability (TEMA)
• Early Numeracy – CBM
– Oral Counting
– Number Identification
– Quantity Discrimination
– Missing Number
Method and Analysis Framework
• Competing curricula
– All students received instruction
– Time balanced across conditions
• Sample
– At risk (some or high risk)
• < 40th percentile
• 66% of student sample
– No risk
• ≥ 40th percentile
• 34% of student sample
Nested Time × Condition Analysis
• Outcome: net differences from pre to post
• Nested students within classrooms
– Control for nonindependence (e.g., ICCs)
– Controls for teacher effects
• Maximum likelihood (restricted)
– Includes all cases with data at either T1 or T2
– Reduces bias from missing data
• Moderation added Time × Risk × Condition
interaction
• Effect sizes: Hedges’ g
Sample Means, SDs, and Ns
T No Risk Some Risk
ELM Control ELM Control
TEMA Raw T1 29.0 (6.8) 28.8 (7.3) 14.1 (6.0) 14.6 (6.2)
T2 39.6 (6.3) 39.1 (7.9) 28.6 (8.4) 26.9 (8.0)
TEMA Percentile T1 66.5 (16.4) 65.1 (17.4) 14.5 (11.4) 15.4 (11.7)
T2 70.4 (18.3) 68.1 (20.8) 35.5 (23.6) 31.0 (22.4)
CBM Total T1 120.3 (44.0) 116.7 (43.5) 45.0 (32.6) 45.5 (35.5)
T2 193.1 (35.8) 187.9 (38.1) 138.4 (50.3) 126.6 (50.5)
Sample Size T1 203 181 397 343
T2 190 174 341 312
Note. Standard deviations (SDs) presented in parentheses. For students with some risk at T1, we collected
TEMAs from 53 fewer students in ELM classrooms and 48 fewer students in control classrooms.
TEMA Percentile Scores
Gains by Condition
• Gains
– Control: 10.94
– ELM: 14.73
– Difference: 3.79
• Test of Condition
– t = 2.10
– df = 61
– p = .0396
– ES = +0.14
• T1 differences were not
statistically significant (t = 0.57)
CBM Total Scores
Gains by Condition
• Gains
– Control: 77.20
– ELM: 84.87
– Difference: 7.67
• Test of Condition
– t = 1.99
– df = 61
– p = .0509
– ES = +0.14
• T1 differences were not
statistically significant (t = 0.60)
TEMA Raw Scores
Condition by Risk Status
• Main Effects
– Difference in gains: 1.32
– t = 2.41, df = 61, p = .0190
• Condition by Risk Status
– t = 2.47, df = 61, p = .0162
– No Risk ≥ 40th %tile
• Difference in gains: 0.04
• t = 0.51, df = 61, p = .9586
– Risk < 40th %tile
• Difference in gains: 1.98
• t = 3.29, df = 61, p = .0017
CBM Total Scores
Condition by Risk Status
• Main Effects
– Difference in gains: 7.67
– t = 1.99, df = 61, p = .0509
• Condition by Risk Status
– t = 2.24, df = 61, p = .0289
– No Risk ≥ 40th %tile
• Difference in gains: -0.27
• t = -0.05, df = 61, p = .9570
– Risk < 40th %tile
• Difference in gains: 10.81
• t = 2.54, df = 61, p = .0138
Effect Sizes (Hedges’ g)
Measure Not At Risk At Risk
Tema Raw Score +0.006 +0.242**
EN-CBM Total +0.014 +0.215*
*p < .05; **p < .01
Summary
• ELM classrooms outperformed controls
– TEMA raw and percentile scores
– EN-CBM Total
• Students at risk
– Improve on all measures more than no-risk students
– Control students at risk catching up on no-risk students
• TEMA: 14.0 percentile gain on no-risk students
• EN-CBM: 9.6 point gain on no-risk students
– ELM students at risk catching no-risk students faster
• TEMA: 18.6 percentile gain on no-risk students
• EN-CBM: 20.63 point gain on no-risk students
• No condition effects for students with no risk
Preliminary Analysis of Association between
Observation Data and Student Outcomes
Intervention Mediators Proximal Distal
Outcomes Outcomes
Coding of Academic Teacher-Student
Interactions (CATS) Observation Instrument
CATS uses a frequency count approach to measure
teacher-student instructional interactions
Observers code behavior occurrences in a continual, serial fashion.
Utilizes a strict coding structure
STUDENT BEHAVIORS TEACHER BEHAVIORS
Individual responses Teacher models
Group responses Academic feedback
Covert responses
Mistakes
CATS based on evidence of effective instruction in early literacy
and beginning mathematics, and adapted from the STICO
observation instrument (Smolkowski & Gunn, 2010)
26
Hypothetical Case of a
Instructional Interaction
The Role of Teacher Modeling and
Student Practice in Student
Outcomes
Preliminary Mediation Analysis
• Does rate of teacher models or student
practice opportunities mediate condition
effect?
• Rates of (a) teacher models, (b) student group
practice opportunity and (c) individual student
practice opportunities entered as mediators to
determine if they decreased condition effect
– Condition effect was still significant
– No evidence to support this mediation hypothesis
Secondary Analysis
• If student practice overall is not mediating
impact, perhaps the value (quality) of practice
differs by classroom and is related to condition
Interaction between Rate of
Practice and Condition
• By condition do rates of practice opportunities
show the same pattern of impact on student
outcomes?
• Are treatment – control differences on student
outcomes similar in classrooms with high rates
of practice vs. low rates of practice?
Interaction between Rate of
Practice and Condition
• Number of Classrooms Control ELM Total
Above 6 26 32
by Treatment Condition Median
and Median Rate of Below 24 8 32
Individual and Group Median
Total 30 34 64
Practice Opportunities
• Practice Opportunity Control ELM Total
Quartiles in Rate per Minimum 0.4 1.0 0.4
Minute 25th %ile 0.8 1.9 1.2
Median 1.3 2.3 1.9
75th %ile 1.7 3.0 2.4
Maximum 4.1 4.1 4.1
TEMA Scores by Rate of Practice
• Within ELM condition
– High-practice classrooms outperform low-practice classrooms
– Difference = 2.59, t = 2.58, df = 29, p = 0.0151
• Within control condition
– No difference between high- and low-practice classrooms
– Difference = 0.39, t = 0.31, df = 29, p = .7575
• Within classrooms with an above-median practice rate
– ELM classrooms (might) outperform control classrooms
– Difference = 2.33, t = 1.85, df = 29, p = .0747
• Within classrooms with a below-median practice rate
– No difference between ELM and control classrooms
– Difference = 0.14, t = 0.14, df = 29, p = .8935
EN-CBM Scores by Rate of Practice
• Within ELM condition
– No difference between high- and low-practice classrooms
– Difference = 9.83, t = 1.46, df = 29, p = 0.1538
• Within control condition
– No difference between high- and low-practice classrooms
– Difference = -7.59, t = -0.92, df = 29, p = .3635
• Within classrooms with an above-median practice rate
– ELM classrooms outperform control classrooms
– Difference = 19.37, t = 2.35, df = 29, p = .0257
• Within classrooms with a below-median practice rate
– No difference between ELM and control classrooms
– Difference = 1.95, t = 0.29, df = 29, p = .7731
Next steps
• Have just completed implementation of Study
1 in Dallas, Texas
• Have just completed implementation of Study
2 in Oregon
• Will implement Study 2 in Dallas, Texas in
2010-11
• Ongoing analysis to investigate impact of
condition and mediation and moderation
variables associated with impact