Get documents about "Analysis of 2005 MAP Results for eMINTS Students
Shared by: warrent
-
Stats
- views:
- 4
- posted:
- 11/5/2008
- language:
- English
- pages:
- 43
Document Sample
January 10, 2005
Analysis of 2004 MAP Results for eMINTS Students
Adam Bickford
The analysis of student MAP scores in the FY03 cohort of eMINTS schools
shows significant differences by eMINTS enrollment status on the MAP
Communication Arts test and smaller, but positive, differences on the MAP
Mathematics test. Analyses of instructional practices suggest that participation
in the eMINTS program helps all teachers more effectively support higher levels
of student achievement. Finally, analyses of MAP scores for Black students,
special education students and students receiving Title I services suggest that
eMINTS enrollment helps reduce the achievement gaps between these groups
and other students. These results support previous analyses of eMINTS cohorts.
This report is one in a series of annual analyses of Missouri Assessment Program (MAP)
test results for students enrolled in schools participating in the eMINTS program. This
analysis focuses on the 40 schools participating in the FY03 eMINTS cohort. The
eMINTS teachers in these schools began the eMINTS professional-development program
in the autumn of 2002 and completed the program in the spring of 2004. This cohort is
unique because of the substantial changes in the funding structure and eligibility
requirements for the eMINTS program. The schools in the FY03 cohort were the first to
be funded by the federal competitive Title IID Enhancing Education Through
Technology (EETT) grant program.
Several programmatic changes have affected the MAP-score analyses presented in this
report. The use of Title IID funds to support the eMINTS program changed the
population of schools eligible to participate in the FY03 cohort and allowed participating
schools to add additional eMINTS classrooms in the second school year of the grant (the
2003-2004 school year). FY03 was also the first-year that the eMINTS program worked
with principals to inform them about the goals of the eMINTS professional-development
program and encouraged principals to assist in the integration of eMINTS instructional
practices into the overall instructional programs of their schools. As the following
analyses will demonstrate, these changes tended to lessen the performance differences
between students enrolled in eMINTS classrooms and other students.
The analyses that follow are based on the 2004 MAP results released in September 2004.
This analysis was completed before the final review and revision of MAP records by
DESE and Missouri school districts for final annual year progress (AYP) assessments.
The AYP review is not scheduled to be completed before March 2005. Consequently, the
results presented here may not necessarily match the final building-level or district-level
MAP results, especially for AYP subgroups.
This report is one product of the eMINTS evaluation project. Other reports and their overall evaluation plan
are available at http://www.emints.org/evaluation.
The eMINTS Evaluation focuses on student impacts, teacher impacts, changes in learning environments
and outcomes of project services.
Table 1
Distribution of eMINTS Classrooms by Grade Level
eMINTS Classroom All Classrooms
Grade Level No Yes
2 4 1 5
2 and 3 1 0 1
2 to 5 0 1 1
3 55 46 101
3 and 4 1 1 2
4 58 58 116
5 7 15 22
6 12 6 18
All Teachers 138 128 266
MAP grades in Bold.
Table 2
Distribution of Teachers and Students by eMINTS Year and MAP Test:
Grades Three and Four
Number of Number of Number of Number of
Teachers Students Teachers Students
Communication Arts Science
Non-eMINTS Classroom 55 1144 27 564
First-Year eMINTS Classroom 6 123 4 84
Second-Year eMINTS Classroom 36 713 27 508
Total 97 1980 58 1156
Mathematics Social Studies
Non-eMINTS Classroom 58 1258 28 623
First-Year eMINTS Classroom 11 195 9 161
Second-Year eMINTS Classroom 46 899 32 631
Total 115 2352 69 1415
These analyses focus primarily on the MAP Communication Arts and Mathematics tests.
These two tests are the only standardized tests required for elementary-school students.
The science and social studies tests became optional in 2001. FY03 is the first year
substantially fewer schools administered these optional tests (see Table 2). Consequently,
the science and social studies results do not represent all students in the FY03 eMINTS
cohort. This report will not present the results from these two tests in the same detail as
the results from the Communication Arts and Mathematics tests.
Policy Brief #5: 2004 MAP Results 2
The FY03 eMINTS Cohort
The FY03 eMINTS cohort consists of 40 schools in 40 districts. The eMINTS program
funded at least two classrooms in each school and required that participating schools
install at least one eMINTS classroom in the third or fourth grade. Other classrooms
could be installed any other grades in the school. Six schools installed eMINTS
classrooms in third grade only, 12 schools installed eMINTS classrooms in fourth grade
only and 22 schools installed eMINTS classrooms in both the third and fourth grades.
Table 1 presents the grade-level distribution of eMINTS and non-eMINTS classrooms.
Table 2 presents the numbers, by MAP test, of third- and fourth-grade students in
eMINTS classrooms. The total number of students in grades three and four in the FY03
cohort schools is 4332 and about 45 % of these students were enrolled in eMINTS
classrooms in the 2003-2004 school year. Table 2 also shows that roughly half of the
students in the FY03 cohort took the science and social studies tests.
The range of grades represented in Table 1 highlights the complexity of the eMINTS
program beginning with the 2002-2003 school year. This complexity is due to the overall
grant structure of the Title IID program. This program enforced different school-
eligibility criteria and allowed for schools to add additional eMINTS classrooms and
teachers in the second year of the grant.
Table 2 illuminates the impact of these policies; it lists the numbers of teachers and
students in the two MAP grades (third and fourth grade) by their eMINTS enrollment
status and MAP-test subject area. For each test, a substantial number of students were
enrolled in classrooms where the teachers were at the midpoint of the eMINTS
professional-development program. For example, in Communication Arts, 15% of the
students in eMINTS classrooms were assigned to teachers who had not completed the
eMINTS professional-development program. Previous evaluation of the eMINTS
program has shown that teacher instructional practices change between the first and
second years of the program, as many teachers use the summer of the first year to revise
their lesson plans and instructional strategies (see eMINTS Evaluation Team, 2001a and
2002). At the time of the 2004 MAP test, students in first-year eMINTS classrooms had
been working with the full suite of eMINTS equipment for approximately three months,
compared to students in second-year eMINTS classrooms who had worked in eMINTS
environments from the beginning of the school year.
This situation has the effect of confounding the relationship between student enrollment
in an eMINTS classroom and MAP test scores. Consequently, it is necessary to control
for this programmatic difference by reporting results by the number of years a teacher has
been enrolled in the eMINTS professional-development program.
Plan of the Document
This report focuses on MAP Communication Arts and Mathematics test scores for
students enrolled in eMINTS classrooms. The first set of analyses presents basic MAP-
Policy Brief #5: 2004 MAP Results 3
performance variables by eMINTS enrollment status for the complete series of four MAP
tests: Communication Arts, Science, Mathematics and Social Studies. The second set of
analyses presents results for Communication Arts and Mathematics by the observed
characteristics of lessons using the eMINTS Lesson Typology (eMINTS Evaluation
Project, 2001b). The last set of analyses examines MAP-score differences by student
race, student income status, student special education status and student receipt of Title I
services.
Comparison of Schools
The FY03 cohort is the first eMINTS cohort to use federal Title IID competitive funds as
their primary funding source. Using this funding source substantially changed the
eligibility criteria for participation in the eMINTS program. Prior to 2002, staffing
constraints occasionally limited the geographic areas of applications, but, for the most
part, all Missouri elementary schools were eligible to apply. Beginning in 2002, the pool
of eligible schools was limited by federal poverty criteria. This change has rendered the
comparison of MAP results from eMINTS schools to statewide MAP results
inappropriate. As the population of eligible schools is no longer all Missouri elementary
schools, this report will not compare results from eMINTS schools to the statewide totals.
Instead, this report will compare MAP scores among the schools that applied to
participate in the FY03 cohort.
A total of 66 schools applied to participate in the FY03 eMINTS cohort. From this pool,
40 schools were accepted into the program.
Figures 1 and 2 present the distribution of MAP performance levels on the
Communication Arts test for the schools that applied to the eMINTS FY03 cohort.
Figure 1 compares the MAP performance levels for students in schools that applied to the
FY03 cohort to the performance levels of all students statewide. Figure 2 disaggregates
these results by whether or not a school was accepted into the eMINTS program.
The results from Figure 1 suggest that third-grade students in the set of schools that
applied to be in the eMINTS FY03 cohort scored slightly lower than third-grade students
statewide. Figure 2 shows that the eMINTS schools performed significantly better than
the other schools in the FY03 applicant pool, although this difference is also relatively
small. The largest percentage difference, 4.2 percentage points, is visible in the Proficient
category.
Figures 3 and 4 present similar results for the Mathematics test. As with Communication
Arts test, fourth-grade students in the FY03 applicant schools scored slightly lower than
all fourth-graders statewide. Figure 4 shows that the students in eMINTS schools scored
slightly higher than students in other schools, particularly students in the Advanced
category. However, these differences are not statistically significant.
Policy Brief #5: 2004 MAP Results 4
Figure 1
MAP Communication Arts Performance Level by FY03 School Status: Students in
FY03 eMINTS Applicant Schools versus All Students Statewide
2004 MAP Communication Arts
50.0
40.0
Percent
30.0
20.0
10.0
0.0
Step 1 Progressing Nearing Proficient Advanced
Proficiency
Performance Level
FY03 eMINTS Applicant Schools All Students, Statewide
FY03 eMINTS All Students,
Applicant Schools Statewide
Step 1 5.2 6.5
Progressing 18.6 19.1
Nearing Proficiency 43.1 39.8
Proficient 32.1 33.3
Advanced 0.9 1.4
Total 100.0 100.0
Number of Students 2682 64503
These figures suggest that comparing results from the FY03 eMINTS schools to
statewide results from all elementary schools is inappropriate. The criteria used to select
the FY03 eMINTS schools limited the population of applicant schools. On the whole,
students in these schools scored slightly lower that the population of students statewide.
A more appropriate measure of schoolwide results compares the schools accepted into the
eMINTS program and those schools not accepted into the eMINTS program for the FY03
cohort. The analysis of MAP performance levels in the next section uses this comparison.
Policy Brief #5: 2004 MAP Results 5
Figure 2
MAP Communication Arts Performance Level by FY03 School Status: Students in
FY03 eMINTS Applicant Schools
2004 MAP Communication Arts
50.0
40.0
Percent
30.0
20.0
10.0
0.0
Step 1 Progressing Nearing Proficient Advanced
Proficiency
Performance Level
eMINTS Schools non-eMINTS Schools FY03 eMINTS Applicant Schools
eMINTS non-eMINTS FY03 eMINTS
Schools Schools Applicant Schools
Step 1 5.1 5.5 5.2
Progressing 19.5 16.1 18.6
Nearing Proficiency 41.5 48.0 43.1
Proficient 33.0 29.5 32.1
Advanced 0.9 0.9 0.9
Total 100.0 100.0 100.0
Number of Students 2005 677 2682
P-Value 0.0393
MAP Performance-Level Results
This section examines the distribution of the MAP performance levels for all four MAP
tests. The following figures present the differences by eMINTS enrollment status for
students in the eMINTS schools and compare these results to the overall distribution of
performance levels for the set of non-eMINTS schools. The overall number of students in
the set of eMINTS schools is slightly smaller than the number of students in Figures 1
through 4. This discrepancy is due to the procedures used to match student MAP records
to teachers and classrooms. For example, for the Communication Arts test, the number of
students in eMINTS schools in Figure 2 is 2005 while the number of students in eMINTS
schools in Figure 5 is 1980, which represents an overall match rate of 98.7%. In total,
96% of student MAP records in eMINTS schools were successfully matched to a teacher
and classroom. On the Mathematics test, the match rate was 99.0%.
Policy Brief #5: 2004 MAP Results 6
Figure 3
MAP Mathematics Performance Level by FY03 School Status: Students in FY03
eMINTS Applicant Schools versus All Students Statewide
2004 MAP Mathematics
50.0
40.0
Percent
30.0
20.0
10.0
0.0
Step 1 Progressing Nearing Proficient Advanced
Proficiency
Performance Level
FY03 eMINTS Applicant Schools All Students, Statewide
FY03 eMINTS All Students,
Applicant Schools Statewide
Step 1 1.4 1.9
Progressing 16.7 15.7
Nearing Proficiency 43.4 42.1
Proficient 32.2 32.4
Advanced 6.4 8.0
Total 100.0 100.0
Number of Students 3020 66096
Figures 5 through 8 also disaggregate classrooms in the eMINTS schools by the teacher’s
year in the professional development program. As shown in Table 2, FY03 marks the first
year that a substantial number of first-year eMINTS teachers were included in analysis.
This change is due the grant regulations used in FY03, which allowed eMINTS schools
to add classrooms after the first year of the eMINTS grant. “Second-year eMINTS
Classrooms” indicate classrooms where the teachers had completed the full eMINTS
professional-development program by the end of the 2003-2004 school year. “First-year
eMINTS Classrooms” indicate classrooms where the teachers had completed the first half
of the program by the end of the 2003-2004 school year.
Policy Brief #5: 2004 MAP Results 7
Figure 4
MAP Communication Arts Performance Level by FY03 School Status: Students in
FY03 eMINTS Applicant Schools
2004 MAP Mathematics
50.0
40.0
Percent
30.0
20.0
10.0
0.0
Step 1 Progressing Nearing Proficient Advanced
Proficiency
Performance Level
eMINTS Schools non-eMINTS Schools FY03 eMINTS Applicant Schools
eMINTS Non-eMINTS FY03 eMINTS
Schools Schools Applicant Schools
Step 1 1.2 2.0 1.4
Progressing 16.8 16.3 16.7
Nearing Proficiency 43.5 42.8 43.4
Proficient 31.9 33.0 32.2
Advanced 6.5 5.9 6.4
Total 100.0 100.0 100.0
Number of Students 2375 645 3020
P-Value 0.5640
Results for Communication Arts
Figure 5 presents results for the Communication Arts test. These results show significant
differences between the eMINTS and non-eMINTS schools and significant differences by
the number of years a teacher has been in the eMINTS professional-development
program. Overall, a higher percentage of students in eMINTS schools scored in the
Proficient category compared to students in non-eMINTS schools. Within the eMINTS
schools, the percentage of students in the Proficient and Advanced categories is highest in
the second-year eMINTS classrooms, suggesting that a teacher’s completion of the
eMINTS professional-development program does have a positive impact on the test
performance of third-grade students. No similar difference appears to be associated with
enrollment in a first-year eMINTS classroom.
Policy Brief #5: 2004 MAP Results 8
Figure 5
MAP Performance Level by eMINTS Classroom Status: Communication Arts
MAP Performance Level
Communication Arts
60.0
50.0
40.0
30.0
20.0
10.0
0.0
Step 1 Progressing Nearing Proficient Advanced
Proficiency
All Third Grade Students in eMINTS Schools All Third Grade Students, non-eMINTS Schools
50.0
40.0
30.0
20.0
10.0
0.0
Step 1 Progressing Nearing Proficient Advanced
Proficiency
Non-eMINTS Classroom First Year eMINTS Classroom Second-Year eMINTS Classroom
Non- First-Year Second-Year All Third-Grade All Third-Grade
eMINTS eMINTS eMINTS Students in Students, non-
Classroom Classroom Classroom eMINTS Schools eMINTS Schools
Step 1 5.2 6.5 4.1 4.9 5.5
Progressing 21.7 20.3 15.7 19.4 16.1
Nearing Proficiency 41.1 43.1 43.3 42.0 48.0
Proficient 31.4 30.1 35.3 32.7 29.5
Advanced 0.6 0.0 1.5 0.9 0.9
Total 100.0 100.0 100.0 100.0 100.0
Number of Students 1144 123 713 1980 677
P-Values
eMINTS vs. non-eMINTS Schools 0.0393
Years in the eMINTS Program 0.0186
Policy Brief #5: 2004 MAP Results 9
Figure 6
MAP Performance Level by eMINTS Classroom Status: Science
MAP Performance Level
Science
50.0
40.0
30.0
20.0
10.0
0.0
Step 1 Progressing Nearing Proficient Advanced
Proficiency
All Third Grade Students in eMINTS Schools All Third Grade Students, non-eMINTS Schools
50.0
40.0
30.0
20.0
10.0
0.0
Step 1 Progressing Nearing Proficient Advanced
Proficiency
Non-eMINTS Classroom First Year eMINTS Classroom Second-Year eMINTS Classroom
First-Year Second-Year All Third-Grade All Third-Grade
Non-eMINTS eMINTS eMINTS Students in eMINTS Students, non-
Classroom Classroom Classroom Schools eMINTS Schools
Step 1 1.1 1.2 1.2 1.1 2.2
Progressing 8.5 16.7 7.9 8.8 11.2
Nearing Proficiency 41.3 36.9 38.2 39.6 39.8
Proficient 40.1 40.5 41.9 40.9 31.8
Advanced 9.0 4.8 10.8 9.5 15.0
Total 100.0 100.0 100.0 100.0 100.0
Number of Students 564 84 508 1156 507
P-Values
eMINTS vs. non-
eMINTS Schools 0.0002
Years in the
eMINTS Program 0.2269
Policy Brief #5: 2004 MAP Results 10
Results for Science
Figure 6 presents the results for the 2004 MAP Science test. Science is one of the
optional MAP tests and as seen in Table 2, only 58% of the students who took the
Communication Arts test also took the science test. Consequently, these results should be
treated with caution due to the school-level selection bias inherent in a school district’s
choice to administer this test.
The results for science show a significant difference between eMINTS and non-eMINTS
schools, with a higher percentage of students in non-eMINTS schools scoring in the
Advanced category and a higher percentage of students in eMINTS schools scoring in the
Proficient category. When these two categories are combined, 50.4% of the students in
eMINTS schools score in the upper two categories, compared to 46.8% of students in
non-eMINTS schools.
Some differences exist within eMINTS schools, but these differences are not large
enough to be statistically significant. The size of these differences is less than 2
percentage points, but they do show slightly higher scores for students in second-year
eMINTS classrooms.
Results for Mathematics
Figure 7 presents results for the Mathematics test. No statistically significant differences
exist between eMINTS and non-eMINTS schools or within eMINTS schools. A
difference of less than 2 percentage points exists between the performance of students in
eMINTS and non-eMINTS schools.
Within eMINTS schools, a higher percentage of students in eMINTS classrooms scored
in the Advanced category. There appears to be a progression in the percentage of students
scoring in the Advanced category; the difference between students in non-eMINTS
classrooms and first-year eMINTS classrooms is 0.9 percentage points and the difference
between students in first-year and second-year eMINTS classrooms is 1.3 percentage
points. While this pattern is interesting, the size of these differences is not large enough
to be statistically significant.
Results for Social Studies
Figure 8 presents results for the social studies test, the other optional MAP test. There are
virtually no differences between eMINTS and non-eMINTS schools and no notable
differences within eMINTS schools.
Policy Brief #5: 2004 MAP Results 11
Figure 7
MAP Performance Level by eMINTS Classroom Status: Mathematics
MAP Performance Level
Mathematics
50.0
40.0
30.0
20.0
10.0
0.0
Step 1 Progressing Nearing Proficient Advanced
Proficiency
All Fourth Grade Students in eMINTS Schools All Fourth Grade Students, non-eMINTS Schools
50.0
40.0
30.0
20.0
10.0
0.0
Step 1 Progressing Nearing Proficient Advanced
Proficiency
Non-eMINTS Classroom First Year eMINTS Classroom Second-Year eMINTS Classroom
Non- First-Year Second-Year All Fourth-Grade All Fourth-Grade
eMINTS eMINTS eMINTS Students in Students, non-
Classroom Classroom Classroom eMINTS Schools eMINTS Schools
Step 1 1.3 1.5 1.0 1.2 2.0
Progressing 15.8 16.9 17.0 16.4 16.3
Nearing Proficiency 44.3 42.6 44.5 44.2 42.8
Proficient 32.8 32.3 29.5 31.5 33.0
Advanced 5.8 6.7 8.0 6.7 5.9
Total 100.0 100.0 100.0 100.0 100.0
Number of Students 1258 195 899 2352 645
P-Values
eMINTS vs. non-eMINTS Schools 0.5640
Years in the eMINTS Program 0.5485
Policy Brief #5: 2004 MAP Results 12
Figure 8
MAP Performance Level by eMINTS Classroom Status: Social Studies
MAP Performance Level
Social Studies
50.0
40.0
30.0
20.0
10.0
0.0
Step 1 Progressing Nearing Proficient Advanced
Proficiency
All Fourth Grade Students in eMINTS Schools All Fourth Grade Students, non-eMINTS Schools
50.0
40.0
30.0
20.0
10.0
0.0
Step 1 Progressing Nearing Proficient Advanced
Proficiency
Non-eMINTS Classroom First Year eMINTS Classroom Second-Year eMINTS Classroom
Non- First-Year Second-Year All Fourth-Grade All Fourth-Grade
eMINTS eMINTS eMINTS Students in Students, non-
Classroom Classroom Classroom eMINTS Schools eMINTS Schools
Step 1 4.7 4.3 5.4 4.9 4.4
Progressing 18.6 18.0 22.8 20.4 16.4
Nearing Proficiency 27.8 30.4 28.2 28.3 32.6
Proficient 26.0 24.2 20.6 23.4 25.4
Advanced 23.0 23.0 23.0 23.0 21.2
Total 100.0 100.0 100.0 100.0 100.0
Number of Students 623 161 631 1415 457
P-Values
eMINTS vs. non-eMINTS Schools 0.1763
Years in the eMINTS Program 0.4349
Policy Brief #5: 2004 MAP Results 13
Summary
Figures 5 through 8 present results for the MAP performance levels on each of the four
MAP tests. The results for the MAP Communication Arts test show statistically
significant differences, differences on the other tests are not significant. Results for
Mathematics show an interesting progression in the percentage of students scoring in the
Advanced category by teacher status in the eMINTS professional-development program,
but these differences are not large enough to be statistically significant. On the science
test there is some indication that students in classrooms taught by second-year eMINTS
teachers scored slightly higher than other students, but this difference is not statistically
significant either. No differences were apparent in social studies.
These results are based on the categorization of student total MAP scores and reveal
gross differences in test performance across different groups of classrooms. The next set
of analyses looks at aggregate differences in the MAP scores themselves.
Total MAP Score by eMINTS Enrollment: Communication Arts and Mathematics
The previous section identified statistically significant differences on the MAP
Communication Arts test among different groups of classrooms. This section of the
report analyzes mean differences in total MAP score by eMINTS enrollment status for
the two mandatory tests, Communication Arts in the third grade and Mathematics in the
fourth grade. When analyzing the mean levels of student performance, it is necessary to
account for the hierarchal character of student test scores. Students are enrolled in
classrooms and these classrooms are embedded in schools. In particular, individual
schools tend to have a clustering effect on student performance, if only because
elementary schools draw primarily students from the community surrounding the school
building. These students are likely to come from a locally homogenous population,
although in a cohort of schools, sets of students may differ substantially between schools.
In the case of the FY03 eMINTS cohort, participating schools were drawn from a variety
of urban and rural districts, large and small districts, districts with a high percentage of
children meeting federal poverty criteria and districts with a large number of students
meeting federal poverty criteria. In order to fairly assess the impact of a classroom-level
intervention like the eMINTS program, the analysis needs to account for these and other,
unmeasured, school-level differences.
The schools in the FY03 cohort differ considerably from one another due to the selection
criteria employed by the Title IID program. Eligible schools are determined by a
combination of the number and percentage of children living in poverty for each eligible
school district. As might be expected, these criteria have produced a cohort with
considerable variation in test scores. As seen in Table 3, the range in the school-level
MAP scores on the Communication Arts test is nearly 48 points, while on the
Mathematics test the range is nearly 65 points.
Policy Brief #5: 2004 MAP Results 14
Table 3
Range in MAP Scores, FY03 Cohort
Communication Arts Mathematics
MAP Score MAP Score
Number of Standard Number of Standard
Students Mean Deviation Students Mean Deviation
Minimum 11 627.13 13.44 11 612.49 22.04
Maximum 265 675.09 34.36 279 677.36 42.50
Range 254 47.97 20.92 268 64.87 20.46
Number of Schools 28 34
To account for this variation between schools, it is necessary to analyze student-level
MAP scores using a randomized block design. This design accounts for variation
between schools by defining a school as a block. Schools are treated as randomized
blocks because the schools in the cohort were doubly selected from a larger population of
eligible elementary schools, i.e., they were selected by the Title IID eligibility criteria and
by the eMINTS application process.
The following analysis tests whether or not the effect of eMINTS enrollment is nested
within the school, that is, whether or not student enrollment in an eMINTS classroom
affects student performance after controlling for the variance between schools. This
analysis strategy is more complex than the approach used in earlier reports. However,
FY03 is the first year that such strict selection criteria applied to eMINTS schools.
Results for Communication Arts
Figure 9 and Table 4 present results for the Communication Arts test broken down by the
teacher’s year in the eMINTS professional-development program1. These results show a
significant difference for students enrolled in second-year eMINTS classrooms. The
overall effect of enrollment in an eMINTS classroom is statistically significant.
In addition, the results in Table 4 show that students enrolled in second-year eMINTS
classrooms scored an average of 5.50 points higher than students enrolled in non-
eMINTS classrooms and an average of 6.57 points higher than students enrolled in first-
year eMINTS classrooms. The difference between students enrolled in second-year
eMINTS classrooms and all other students is statistically significant. The difference
between students enrolled in non-eMINTS classrooms and those enrolled in first-year
eMINTS classrooms is not significant.
1
These plots, called “bi-plots” show the entire distribution of MAP scores for each group. The mean value
is represented by the large dot, the width of the 95% confidence interval is represented by the straight line
and extreme values are represented by the small dots beyond the lines (see Smith, 1997). These plots
represent mean differences and confidence intervals as recommended by the American Psychological
Association (see Wilkinson, 1999).
Policy Brief #5: 2004 MAP Results 15
Figure 9
Total MAP Score by eMINTS Enrollment: Communication Arts
Least Squares Means Estimate by eMINTS Enrollment: Communication Arts
660.00
650.00
-
-
-
-
640.00
-
-
630.00
Non-eMINTS Classroom First-Year eMINTS Classroom Second-Year eMINTS Classroom
Policy Brief #5: 2004 MAP Results 16
Table 4
ANOVA Results: MAP Communication Arts Score by eMINTS Enrollment
Controlling for School
Lower
95% Upper 95%
Mean Standard Confidence Confidence
Estimate Error Interval Interval
Non-eMINTS Classroom 639.72 1.81 636.18 643.26
First-Year eMINTS Classroom 638.65 3.17 632.43 644.86
Second-Year eMINTS Classroom 645.22 1.7294 641.83 648.61
Differences in Estimates
Non- First-year
eMINTS eMINTS
Classroom classroom
Non-eMINTS Classroom
First-Year eMINTS Classroom -1.07
Second-Year eMINTS Classroom 5.50 6.57
Variance Components
Percentage
Standard of
Estimate Error Z-Value P-Value Variance
School 52.34 21.88 2.39 0.0084 6.2
Residual 794.75 25.48 31.19 <0.0001
Total 847.09
Test for Fixed Effects
Df Df
Numerator Denominator F-Value P-Value
Student Enrollment in eMINTS
Classroom 1 1951 8.59 0.0034
These results suggest that, controlling for the variance between schools, students enrolled
in fully established eMINTS classrooms scored higher than other students in
Communication Arts.
Results for Mathematics
Figure 10 and Table 5 present similar results for the Mathematics test. These results
suggest that the score difference attributable to eMINTS is nearly significant. In addition,
there are some important differences to consider. First, the differences between first- and
second-year eMINTS classrooms are minimal. The estimated difference in means
between students in first and second-year eMINTS classrooms is 0.10 points.
Consequently, Figure 10 and Table 5 present only results from the basic comparison by
eMINTS enrollment.
Policy Brief #5: 2004 MAP Results 17
Figure 10
Total MAP Score by eMINTS Enrollment: Mathematics
Least Squares Means Estimate by eMINTS Enrollment: Mathematics
652.00
651.00
650.00
649.00
-
648.00
647.00
646.00
-
645.00
644.00
643.00
642.00
641.00
640.00 -
639.00
638.00
637.00
636.00 -
635.00
634.00
Non-eMINTS Classroom eMINTS Classroom
Policy Brief #5: 2004 MAP Results 18
Table 5
ANOVA Results: MAP Mathematics Score by eMINTS Enrollment
Controlling for School
Lower
95% Upper 95%
Mean Standard Confidence Confidence
Estimate Error Interval Interval
Non-eMINTS Classroom 640.69 2.48 635.83 645.55
eMINTS Classroom 644.24 2.21 639.9 648.58
Difference 3.55
Variance Components
Standard Percentage of
Estimate Error Z-Value P-Value Variance
School 128.38 38.56 3.33 0.0004 10.9
Residual 1047.11 31.23 33.53 <0.0001
Total 1175.49
Test for Fixed Effects
Df Df
Numerator Denominator F-Value P-Value
Student Enrollment in eMINTS
Classroom 1 2249 3.66 0.0558
The estimated Mathematics score difference between eMINTS students and non-eMINTS
students is 3.55 points. This difference is marginally significant; the p-value for the fixed
effect of eMINTS enrollment is 0.0558. These results show some general evidence for
the effect of student work in eMINTS classrooms. The analysis in the next section will
consider the effect of different types of instructional practice.
Summary
The analysis of the total MAP scores shows positive differences associated with eMINTS
enrollment. Students enrolled in eMINTS classrooms scored higher than non-eMINTS
students on both the Communication Arts and Mathematics tests. The differences on the
Communication Arts test were clearly significant and showed that students enrolled in
second-year eMINTS classrooms outperformed all other students. The differences on the
Mathematics test were more subtle. First, no differences existed between students
enrolled in first and second-year eMINTS classrooms. The difference between students in
eMINTS classrooms and other students was much smaller than the difference on the
Communication Arts test. The statistical test for the fixed effect (enrollment in an
eMINTS classroom) was only marginally significant. Nevertheless, the results suggest
that students enrolled in eMINTS classrooms did perform better on the Mathematics test
than other students.
Policy Brief #5: 2004 MAP Results 19
eMINTS and Instructional Practice
The eMINTS professional-development model aims to provide teachers with classroom
strategies that facilitate inquiry-based instruction using technology. This type of
instruction focuses on the creation of multidisciplinary long-term products that typically
go beyond the scope of traditional text-based learning. Conducting inquiry-based lessons
requires teachers to change the ways a classroom operates. For example, the completion
of a large project often requires that students work cooperatively in pairs or teams. The
successful completion of a cooperative-learning project requires classroom norms and
practices that support and encourage sharing, discussion and interdependent work. These
norms and practices differ considerably from classrooms where, for example, students are
expected to work independently and are prohibited from consulting with other students.
The eMINTS Lesson Typology rubric was designed to identify the classroom norms and
practices that facilitate the kinds of instruction taught by the eMINTS professional-
development program (eMINTS Evaluation Project, 2001b). This observational rubric
classifies classroom lessons by the ways that teachers structure their lessons, the ways
that teachers and students interact and the ways that students interact with one another.
This rubric has been part of the eMINTS evaluation-team classroom observations since
the team’s initial observations of classroom practices in 1999. In the observation of the
FY03 cohort, eMINTS evaluation-team members collected this information from 189 of
the 219 third and fourth grade classrooms in the participating schools .
The eMINTS Lesson Typology distinguishes among four broad types of lessons: teacher-
centered lessons, hybrid lessons, student-centered facilitated lessons and student-centered
unfacilitated lessons2. Of these types of lessons, student-centered facilitated lessons most
closely reflect the goals of the eMINTS professional-development program.
Past analyses (for example, the eMINTS Evaluation Project, 2002) analyzed differences
in MAP scores among eMINTS teachers in terms of this typology. Additionally, in past
analyses, classroom observations were limited to eMINTS classrooms. The following
analysis offers the first application of this observational rubric to all classrooms in the
eMINTS grades in an eMINTS cohort. This greater scope of observation permits a more
complete accounting of classroom norms and practices in participating schools. It also
allows for the investigation of the ways in which classroom practices and participation in
the eMINTS program interact to support student MAP scores.
The following analysis examines, for the classrooms in the FY03 cohort, the distribution
of observed lesson types by participation in the eMINTS program. The second part of the
analysis examines, for the MAP Communication Arts and Mathematics tests, the
relationships among total MAP score, classroom participation in the eMINTS program
and observed lesson type.
2
The central characteristics of these lesson types are described in Appendix A and in the original
evaluation-team report.
Policy Brief #5: 2004 MAP Results 20
Table 6
Distribution of Observed Lessons by eMINTS Lesson-Typology Classification
and eMINTS Classroom Status
First-Year Second-Year
Non-eMINTS eMINTS eMINTS
Classrooms Classrooms Classrooms Total
Third Grade
Counts
Teacher-Centered 24 2 7 33
Hybrid 10 2 12 24
Student-Centered Facilitated 4 1 15 20
Student-Centered Unfacilitated 2 1 2 5
Total 40 6 36 82
Percentages
Teacher-Centered 60.0 33.3 19.4 40.2
Hybrid 25.0 33.3 33.3 29.3
Student-Centered Facilitated 10.0 16.7 41.7 24.4
Student-Centered Unfacilitated 5.0 16.7 5.6 6.1
Total 100.0 100.0 100.0 100.0
Fourth Grade
Counts
Teacher-Centered 26 2 5 33
Hybrid 15 4 18 37
Student-Centered Facilitated 8 4 22 34
Student-Centered Unfacilitated 3 0 0 3
Total 52 10 45 107
Percentages
Teacher-Centered 50.0 20.0 11.1 30.8
Hybrid 28.8 40.0 40.0 34.6
Student-Centered Facilitated 15.4 40.0 48.9 31.8
Student-Centered Unfacilitated 5.8 0.0 0.0 2.8
Total 100.0 100.0 100.0 100.0
Distribution of Lesson Type by eMINTS Participation
Table 6 shows the distribution of observed lessons by lesson type and a classroom’s year
in the eMINTS program. The proportion of student-centered facilitated lessons is highest
in second-year eMINTS classrooms. In the third grade, 41.7% of all lessons observed in
second-year eMINTS classrooms classified as student-centered facilitated lessons. In the
fourth grade, nearly half of all lessons observed in second-year eMINTS classrooms
showed features endorsed by the eMINTS professional-development program3.
3
One problematic feature seen in this table is the number and distribution of student-centered unfacilitated
lessons. Among the 189 lessons observed, 8 were placed in this category. In the third grade, two of the five
unfacilitated lessons were observed in second-year eMINTS classrooms. In the fourth grade, all three
unfacilitated lessons were observed in non-eMINTS classrooms. Because of the low number of these
lessons, they have been removed from the MAP analysis that follows.
Policy Brief #5: 2004 MAP Results 21
Overall, these results suggest a wide adoption among teachers in second-year eMINTS
classrooms of the inquiry-based instructional practices supported by the eMINTS
program. The eMINTS evaluation team observed relatively fewer teachers in first-year
eMINTS classrooms employing these types of lessons. This finding is to be expected, as
at the time of the observation, teachers in first-year eMINTS classrooms had just begun
to teach in fully operational eMINTS environments and had not yet reached the midpoint
of their eMINTS professional development,. Previous reports have suggested that
eMINTS teachers typically change their instructional practices after the completion of the
first year of the eMINTS professional-development program. These results suggest that
the eMINTS professional-development program does support teachers in conducting
inquiry-based lessons.
MAP Scores by Lesson Typology
The relatively high proportion of student-centered facilitated lessons would suggest that
students in eMINTS schools score higher on their MAP tests. According to past analyses
of eMINTS classrooms, students in classrooms where the eMINTS evaluation team
observed student-centered facilitated lessons scored significantly higher on their MAP
tests than students in other eMINTS classrooms. One persistent question is how much of
this difference is due to the eMINTS program and how much is due to observed
instructional practices.
The following results show few significant differences on the Communication Arts test
for the fixed effects: eMINTS enrollment, the lesson typology classification or the
interaction of the two. One statistically significant difference exists on the Mathematics
test: eMINTS enrollment. One reason for this general lack of significance is the overall
variance within classrooms. For example, after controlling for school differences, 94% of
the variance in student scores on the Communication Arts test is unexplained.
Nevertheless, there are identifiable differences in the classroom-level means for the
combination of eMINTS enrollment and observed lesson type. These differences show
that within the categories of the eMINTS Lesson Typology, students in second-year
eMINTS classrooms scored higher than other students.
Table 7 and Figure 11 present results for the MAP Communication Arts test. Table 7
shows the main effects of the model, while Figure 11 shows the means for the interaction
between eMINTS enrollment and eMINTS Lesson Typology classification. The main
effects show positive differences for both eMINTS enrollment and eMINTS Lesson
Typology classification. On average, students in second-year eMINTS classrooms scored
3.4 points higher than students in non-eMINTS classrooms. In addition, students in
classrooms with observed student-centered facilitated lessons scored 3.3 points higher
than students in classrooms with observed teacher-centered lessons.
Policy Brief #5: 2004 MAP Results 22
Table 7
ANOVA Results: MAP Communication Arts Score by eMINTS Enrollment and
Lesson Typology, Controlling for School
Lower 95% Upper 95%
Mean Standard Confidence Confidence
Estimate Error Interval Interval
eMINTS Enrollment
Non-eMINTS Classroom 640.91 2.10 636.79 645.02
First-Year eMINTS Classroom 639.83 3.39 633.19 646.47
Second-Year eMINTS Classroom 644.32 1.78 640.82 647.81
Lesson Typology
Teacher-Centered 639.99 2.47 635.14 644.84
Hybrid 641.75 2.36 637.13 646.38
Student-Centered Facilitated 643.31 2.82 637.77 648.85
Standard Percentage
Variance Component Estimate Error Z-Value P-Value of Variance
School 48.50 21.62 2.24 0.0124 6.1
Residual 740.89 26.64 27.81 <0.0001
Total 789.39
Df Df
Test for Fixed Effects Numerator Denominator F-Value P-Value
eMINTS Enrollment 2 1551 1.87 0.1539
Lesson Typology 2 1551 0.59 0.5547
Interaction 4 1551 0.16 0.9589
The interactions in Figure 11 show that within each lesson-typology classification
students in second-year eMINTS classrooms had the highest mean value. The confidence
intervals around the means overlap, suggesting that not all students in these classrooms
experienced the same benefits of either eMINTS enrollment or lesson type. However,
these results suggest that eMINTS supports mean student performance across the range of
instructional practices.
Table 8 and Figure 12 present similar results for the MAP Mathematics test. One
important difference for Mathematics is that when controlling for the effect of the lesson
typology, a significant difference exists for eMINTS enrollment. As seen in Table 5, this
difference is between non-eMINTS classrooms and eMINTS classrooms. Students in
non-eMINTS classrooms scored an average of 5 points lower than students in eMINTS
classrooms.
Policy Brief #5: 2004 MAP Results 23
Figure 11
Total MAP Score by eMINTS Enrollment and Lesson Typology Classification:
Communication Arts
Least Squares Means Estimate by eMINTS Status and Lesson Typology: Communication Arts
670.00
660.00
-
650.00 - - -
-
- - -
-
-
640.00 -
-
-
-
-
630.00 - -
-
620.00
Non- First- Second- Non- First- Second- Non- First- Second-
eMINTS Year Year eMINTS Year Year eMINTS Year Year
eMINTS eMINTS eMINTS eMINTS eMINTS eMINTS
Classroom Classroom Classroom
Teacher-Centered Lessons Hybrid Lessons Student-Centered Facilitated Lessons
Policy Brief #5: 2004 MAP Results 24
Table 8
ANOVA Results: MAP Mathematics Score by eMINTS Enrollment and Lesson
Typology, Controlling for School
Lower 95% Upper 95%
Mean Standard Confidence Confidence
Estimate Error Interval Interval
eMINTS Enrollment
Non-eMINTS Classroom 639.68 2.73 634.32 645.04
First-Year eMINTS Classroom 644.69 3.71 637.41 651.97
Second-Year eMINTS Classroom 645.52 2.53 640.56 650.49
Lesson Typology
Teacher-Centered 643.00 2.72 637.66 648.33
Hybrid 642.42 2.88 636.77 648.06
Student-Centered Facilitated 644.49 3.38 637.86 651.11
Variance Components
Standard Percentage
Estimate Error Z-Value P-Value of Variance
School 143.52 42.27 3.40 0.0003 12.1
Residual 1037.90 32.26 32.18 <0.0001
Total 1181.42
Test for Fixed Effects
Df Df
Numerator Denominator F-Value P-Value
eMINTS Enrollment 2 2070 3.50 0.0305
Lesson Typology 2 2070 0.19 0.8307
Interaction 4 2070 2.01 0.0912
The interactions in Figure 12 show a pattern similar to the Communication Arts means.
Students in second-year eMINTS classrooms scored higher than other students in two of
the three lesson-typology categories. The one exception is for students whose teachers
were observed conducting student-centered facilitated lessons. Here, students in first-year
eMINTS classrooms scored highest.
Summary
The analyses of MAP scores by eMINTS classroom status and observed instructional
practice per the eMINTS Lesson Typology suggest that the eMINTS program supports
higher student performance across all types of lessons. On both MAP tests, students in
eMINTS classrooms scored higher than other students. This relationship, while not
statistically significant in all cases, supports the hypothesis that eMINTS can help
students in all types of classrooms and learning environments score higher on the MAP
tests.
Policy Brief #5: 2004 MAP Results 25
Figure 12
Total MAP Score by eMINTS Enrollment and Lesson-Typology Classification:
Mathematics
Least Squares Means Estimate by eMINTS Status and Lesson Typology : Mathematics
680.00
670.00
-
660.00
- -
-
650.00 - -
- -
- - - -
640.00
-
- -
630.00 -
-
-
620.00
Non- First- Second- Non- First- Second- Non- First- Second-
eMINTS Year Year eMINTS Year Year eMINTS Year Year
eMINTS eMINTS eMINTS eMINTS eMINTS eMINTS
Classroom Classroom Classroom
Teacher-Centered Lessons Hybrid Lessons Student-Centered Facilitated Lessons
However, in the FY03 cohort, these differences are small, between 3 and 5 points. In
addition, a considerable amount of variation appears within classrooms and among
schools in the cohort.
MAP Performance by Student Characteristics
The foregoing analysis of lesson type suggests a small positive effect of the eMINTS
program across different types of instruction. However, these models do not account for
performance differences among students; they assume all students respond to being
enrolled in an eMINTS classroom in the same way. What follows is a more descriptive
analysis of differences among groups of students, considering four characteristics :
student race, student income status, student special education status and student receipt of
Title I services.
Policy Brief #5: 2004 MAP Results 26
Looking at student differences requires a change in the previous modeling strategy. The
models documented earlier in this report account for school differences by treating
schools as randomized blocks and treating differences by eMINTS enrollment and lesson
type as crossed factors within these blocks. This strategy was appropriate because both
eMINTS enrollment and lesson type are classroom-level factors. However, in the analysis
of student characteristics, the model has to account for the nesting of students in types of
classrooms. So, instead of treating student characteristics as if they were crossed with
eMINTS enrollment (for example, as indicated in an interaction term), the following
models treat student characteristics as nested within the levels of eMINTS enrollment.
This structure has implications for the results since the overall differences by eMINTS
enrollment are relatively small. The difference between second-year eMINTS classrooms
and non-eMINTS classrooms is 5.50 points on the Communication Arts test (see Table 4)
and the difference between eMINTS and non-eMINTS classrooms is 3.55 points (see
Table 5) on the same exam. Consequently, disaggregating these classroom-level means
by student characteristics is unlikely to reveal large differences.
The focus of these analyses is on the difference between students enrolled in second-year
eMINTS classrooms and students enrolled in non-eMINTS classrooms. Of particular
interest is whether or not the MAP-score difference by eMINTS status for students in a
specified category is larger or smaller than the difference for most students. Finding a
larger MAP-score difference by eMINTS status for a specified category of students
constitutes evidence that eMINTS enrollment contributes to closing the achievement gap
for that category of students. For example, consider students receiving Title I services. If
the mean difference between “Title I” students enrolled in a second-year eMINTS
classroom and students enrolled in a non-eMINTS classroom is larger than the eMINTS-
non-eMINTS difference among “non-Title I students,” then one may conclude that
eMINTS enrollment helps lessen the performance gap between Title I and non-Title I
students. In most cases these differences are too small to be statistically significant, but
they suggest that eMINTS enrollment does have some positive effects for specified
categories of students.
Considering these differences is important for at least two reasons. The first reason
addresses federal policy concerns. Each of these student characteristics is relevant for
school-level AYP classification4. The second reason is consistency with past eMINTS
reports. Previous MAP analyses have suggested that low-income students and special
education students have benefited from enrollment in eMINTS classrooms. For example,
in the FY02 cohort, enrollment in an eMINTS classroom reduced the MAP-score deficit
for low-income students on the Mathematics test by about half (eMINTS Evaluation
Project, 2004).
4
Although annual yearly progress (AYP) is measured in terms of the percentage of students scoring at the
Proficient level or higher on the MAP test, this analysis focuses on the total MAP score. The choice to use
the continuous MAP score over the categorical MAP performance level is purely a statistical one.
Estimating categorical models that simultaneously control for school-level differences, classroom-level
differences and student differences is beyond the scope of this report. These results should not be
interpreted as predicting whether a school, or cohort of schools, has met the annual yearly progress goals
for the 2003-2004 school year.
Policy Brief #5: 2004 MAP Results 27
Table 9
Student Race by eMINTS Enrollment
Non- First-Year Second-Year Number
eMINTS eMINTS eMINTS All of
Classroom Classroom Classroom Students Students
Communication Arts
White 91.1 97.2 90.9 91.5 1437
Black 4.7 0.0 5.3 4.6 73
Other 4.1 2.8 3.8 3.9 61
All Students 100.0 100.0 100.0 100.0
Number of Students 801 108 662 1571
P-Value 0.1639
Mathematics
White 91.3 95.8 95.2 93.3 1956
Black 5.5 2.4 2.1 3.8 18
Other 3.2 1.8 2.8 2.9 24
All Students 100.0 100.0 100.0 100.0
Number of Students 1059 168 870 2097
P-Value 0.0015
The analysis in this section considers four types of student characteristics: student race;
student income status (as indicated by student eligibility for the free and reduced lunch
program); student special education status (as indicated by the student having a recorded
IEP); and student receipt of Title I services. Two of these characteristics, student race and
special education status, come directly from the MAP student-information records
accompanying student test scores. The other two come from information collected from
individual student records provided by the FY03 schools.
The results of the following models show that the nested factor (for example, identifying
the difference for low-income students within levels of eMINTS enrollment) is
statistically significant, indicating that student scores vary by a given characteristic after
accounting for classroom and school differences. Examination of the mean estimates
indicates benefits for eMINTS enrollment on at least one MAP test among Black
students, special education students and students receiving Title I services.
The tables below present the overall distribution of students by student and classroom
characteristics and the ANOVA tables for each characteristic separately. Results are
organized by student characteristic, with results for Communication Arts and
Mathematics reported separately.
Policy Brief #5: 2004 MAP Results 28
Table 10
Total MAP Score by Student Race and eMINTS Enrollment, Controlling for
School: Communication Arts
Lower 95% Upper 95%
Mean Standard Confidence Confidence
Mean Estimates Estimate Error Interval Interval
eMINTS Enrollment Student Race
Non-eMINTS Classroom White 640.84 1.89 637.14 644.54
Black 623.05 4.89 613.45 632.65
Other 637.90 5.10 627.90 647.91
First-Year eMINTS Classroom White 640.03 3.27 633.61 646.46
Other 655.55 15.87 624.42 686.68
Second-Year eMINTS Classroom White 645.72 1.70 642.38 649.06
Black 634.18 4.92 624.53 643.83
Other 632.14 5.70 620.96 643.32
All Students White 643.67 1.55 640.62 646.71
Black 628.90 3.72 621.60 636.20
Other 636.91 3.85 629.35 644.47
Standard Percentage
Variance Components Estimate Error Z-Value P-Value of Variance
School 42.12 18.82 2.24 0.0126 5.43
Residual 733.04 26.47 27.69 <0.0001
Total 775.16
Df Df
Test for Fixed Effects Numerator Denominator F-Value P-Value
Student Race within eMINTS
Enrollment 7 1537 4.73 <0.0001
Student Race
Distribution of Students
Table 9 presents the distribution of student race by eMINTS enrollment for each test. The
student-race field contains three categories: one for White students, one for Black
students and one for Other students. The last category includes Asian, Hispanic and
American Indian students.
Each panel of Table 9 shows that over 90% of the students are White. The panels also
show that the cell sizes for non-White students are very small, as few as 18 Black
students according to the MAP Mathematics database. The distribution of students differs
in the MAP Mathematics database; most of the Black and other-race students were
enrolled in non-eMINTS classrooms. No such differences exist for Communication Arts.
Policy Brief #5: 2004 MAP Results 29
Table 11
Total MAP Score by Student Race and eMINTS Enrollment, Controlling for
School: Mathematics
Lower 95% Upper 95%
Mean Standard Confidence Confidence
Mean Estimates Estimate Error Interval Interval
eMINTS Enrollment Student Race
Non-eMINTS Classroom White 642.40 2.62 637.27 647.53
Black 613.72 5.20 603.53 623.91
Other 654.89 6.03 643.07 666.71
First-Year eMINTS Classroom White 646.92 3.53 640.00 653.85
Black 607.33 16.31 575.34 639.31
Other 620.00 18.69 583.34 656.66
Second-Year eMINTS Classroom White 645.38 2.39 640.69 650.07
Black 613.34 7.98 597.69 629.00
Other 647.84 6.90 634.31 661.36
All Students White 644.61 2.21 640.27 648.94
Black 614.14 4.41 605.50 622.78
Other 651.18 4.66 642.04 660.32
Standard Percentage
Variance Components Estimate Error Z-Value P-Value of Variance
School 142.60 42.55 3.35 0.0004 12.36
Residual 1011.17 31.55 32.05 <0.0001
Total 1153.77
Df Df
Test for Fixed Effects Numerator Denominator F-Value P-Value
Student Race within eMINTS
Enrollment 8 2055 8.59 <0.0001
Total MAP Score
Tables 10 and 11 present the ANOVA models controlling for student race, eMINTS
enrollment and school. The results for the Communication Arts test in Table 10 show that
Black students enrolled in second-year eMINTS classrooms scored an estimated average
of 11.13 points higher than Black students enrolled in non-eMINTS classrooms. This
difference is larger than the 4.88 point difference for White students.
Policy Brief #5: 2004 MAP Results 30
Table 12
Student Income status by eMINTS Enrollment
First-Year Second-Year Number
Non-eMINTS eMINTS eMINTS All of
Classroom Classroom Classroom Students Students
Communication Arts
No 56.7 42.6 54.1 54.6 857
Yes 43.3 57.4 45.9 45.4 712
All Students 100.0 100.0 100.0 100.0
Number of Students 797 108 664 1569
P-Value 0.0203
Mathematics
No 57.6 47.4 47.1 52.4 1107
Yes 42.4 52.6 52.9 47.6 1005
All Students 100.0 100.0 100.0 100.0
Number of Students 1064 173 875 2112
P-Value <0.0001
The results for Mathematics in Table 11 show that Black students in second-year
eMINTS classrooms scored at approximately the same level as Black students enrolled in
non-eMINTS classrooms. In contrast, White students enrolled in second-year eMINTS
classrooms scored 2.98 points higher than White students enrolled in non-eMINTS
classrooms. The largest difference was for students in the other-race category. Other-race
students enrolled in second-year eMINTS classrooms scored 7.05 points lower than
other-race students enrolled in non-eMINTS classrooms. Some of this difference is
attributable to the very small number of non-White students in eMINTS classrooms.
Fully 95% of the fourth-grade students enrolled in eMINTS classrooms were White.
Student Income status
Distribution of Students
Table 12 presents the distribution of student income status, as measured by student
eligibility for the free and reduced lunch program. Significant differences exist for both
the Communication Arts and Mathematics tests. For the Communication Arts test, the
difference is attributable to a higher percentage of low-income students enrolled in first-
year eMINTS classrooms. On the Mathematics test the percentage of low-income
students enrolled in different types of eMINTS classrooms is approximately the same.
The percentage of low-income students enrolled in non-eMINTS classrooms is smaller,
42.4% versus the approximately 53% of students enrolled in eMINTS classrooms.
Policy Brief #5: 2004 MAP Results 31
Table 13
Total MAP Score by Student Income status and eMINTS Enrollment, Controlling
for School: Communication Arts
Lower 95% Upper 95%
Mean Standard Confidence Confidence
Mean Estimates Estimate Error Interval Interval
eMINTS Enrollment Income status
Non-eMINTS Classroom No 645.61 2.11 641.47 649.75
Yes 634.68 2.20 630.36 638.99
First-Year eMINTS Classroom No 643.15 4.40 634.51 651.78
Yes 637.69 3.97 629.91 645.48
Second-Year eMINTS Classroom No 650.58 2.01 646.63 654.53
Yes 638.51 2.11 634.38 642.64
All Students No 648.44 1.79 644.93 651.95
Yes 637.25 1.80 633.73 640.78
Standard Percentage
Variance Components Estimate Error Z-Value P-Value of Variance
School 48.71 21.94 2.22 0.0132 6.39
Residual 714.14 25.82 27.66 <0.0001
Total 762.85
Df Df
Test for Fixed Effects Numerator Denominator F-Value P-Value
Income status within eMINTS
Enrollment 5 1537 13.82 <0.0001
Total MAP Score
Tables 13 and 14 present differences in total MAP score for low-income students,
controlling for eMINTS enrollment and school. Low-income students in second-year
eMINTS classrooms scored an average of 3.83 points higher on the Communication Arts
test than low-income students enrolled in non-eMINTS classrooms, compared to 4.97
points for other students. The difference for Mathematics is much smaller; low-income
students in second-year eMINTS classrooms scored an average of 1.81 points higher than
low-income students enrolled in non-eMINTS classrooms. The difference for other
students is 3.75 points.
Both of these findings need to be considered in the context of the overall score difference
by eMINTS enrollment. For Communication Arts, students enrolled in second-year
eMINTS classrooms scored an average of 5.50 point higher than students enrolled in non-
eMINTS classrooms. The difference for low-income students is 69% of the overall
difference.
Policy Brief #5: 2004 MAP Results 32
Table 14
Total MAP Score by Student Income status and eMINTS Enrollment, Controlling
for School: Mathematics
Lower 95% Upper 95%
Mean Standard Confidence Confidence
Mean Estimates Estimate Error Interval Interval
eMINTS Enrollment Income status
Non-eMINTS Classroom No 648.11 2.74 642.73 653.49
Yes 635.66 2.78 630.20 641.11
First-Year eMINTS Classroom No 652.03 4.30 643.61 660.46
Yes 637.56 4.16 629.41 645.71
Second-Year eMINTS Classroom No 651.86 2.63 646.71 657.01
Yes 637.47 2.57 632.43 642.51
All Students No 650.46 2.32 645.90 655.01
Yes 637.01 2.30 632.50 641.51
Standard Percentage
Variance Components Estimate Error Z-Value P-Value of Variance
School 138.80 41.10 3.38 0.0004 12.16
Residual 1002.86 31.15 32.19 <0.0001
Total 1141.66
Df Df
Test for Fixed Effects Numerator Denominator F-Value P-Value
Income status within eMINTS
Enrollment 5 2073 17.5 <0.0001
For Mathematics, students enrolled in eMINTS classrooms scored 3.55 points higher than
students enrolled in non-eMINTS classrooms. The difference for low-income students is
51% of the overall difference.
Policy Brief #5: 2004 MAP Results 33
Table 15
Student Special Education Status by eMINTS Enrollment
Non- First-Year Second-Year Number
eMINTS eMINTS eMINTS All of
Classroom Classroom Classroom Students Students
Communication Arts
No 86.4 91.7 87.0 87.0 1380
Yes 13.6 8.3 13.0 13.0 206
All Students 100.0 100.0 100.0 100.0
Number of Students 808 108 670 1586
P-Value 0.3087
Mathematics
No 84.5 87.3 88.7 86.5 1826
Yes 15.5 12.7 11.3 13.5 286
All Students 100.0 100.0 100.0 100.0
Number of Students 1064 173 875 2112
P-Value 0.0257
Student Special Education Status
Distribution of Students
Table 15 presents the distribution of student special education status. Significant
differences exist on the Mathematics test. The percentage of special education students
enrolled in non-eMINTS classrooms is higher, 15.5%, compared to special education
students enrolled in first- and second-year eMINTS classrooms,12.7% and 11.3%
respectively.
Total MAP Score
Tables 16 and 17 present differences in total MAP score for special education students,
controlling for eMINTS enrollment and school. Although the differences are small, this
analysis is one instance where the difference in total MAP score for a specified group of
students is larger than the difference for other students. On the Communication Arts test,
special education students enrolled in second-year eMINTS classrooms scored an average
of 5.82 points higher than special education students enrolled in non-eMINTS
classrooms. For other students the difference was 4.96 points higher. On the Mathematics
test, the difference for special education students was 3.97 points while the difference for
other students was 1.64 points.
Policy Brief #5: 2004 MAP Results 34
Table 16
Total MAP Score by Student Special Education Status and eMINTS Enrollment,
Controlling for School: Communication Arts
Lower 95% Upper 95%
Mean Standard Confidence Confidence
Mean Estimates Estimate Error Interval Interval
eMINTS Enrollment Special Education Student
Non-eMINTS Classroom No 641.24 1.97 637.38 645.10
Yes 629.23 3.10 623.16 635.31
First-Year eMINTS Classroom No 640.02 3.39 633.36 646.67
Yes 639.25 9.20 621.21 657.30
Second-Year eMINTS Classroom No 646.20 1.81 642.65 649.74
Yes 635.05 3.25 628.68 641.42
All Students No 644.18 1.68 640.88 647.49
Yes 633.01 2.44 628.23 637.79
Standard Percentage of
Variance Components Estimate Error Z-Value P-Value Variance
School 51.21 21.39 2.39 0.0083 6.58
Residual 726.92 26.11 27.85 <0.0001
Total 778.13
Df Df
Test for Fixed Effects Numerator Denominator F-Value P-Value
Special Education Student within
eMINTS Enrollment 5 1554 8.16 <0.0001
Policy Brief #5: 2004 MAP Results 35
Table 17
Total MAP Score by Student Special Education Status and eMINTS Enrollment,
Controlling for School: Mathematics
Lower 95% Upper 95%
Mean Standard Confidence Confidence
Mean Estimates Estimate Error Interval Interval
eMINTS Enrollment Special Education Student
Non-eMINTS Classroom No 644.95 2.54 639.96 649.93
Yes 623.43 3.39 616.78 630.08
First-Year eMINTS Classroom No 649.28 3.49 642.43 656.13
Yes 614.35 7.20 600.23 628.48
Second-Year eMINTS Classroom No 646.59 2.33 642.03 651.16
Yes 627.40 3.79 619.97 634.84
All Students No 646.35 2.14 642.16 650.54
Yes 624.51 2.75 619.12 629.89
Standard Percentage of
Variance Components Estimate Error Z-Value P-Value Variance
School 131.21 38.87 3.38 0.0004 11.72
Residual 988.01 30.69 32.20 <0.0001
Total 1119.22
Df Df
Test for Fixed Effects Numerator Denominator F-Value P-Value
Special Education Student within
eMINTS Enrollment 5 2073 24.32 <0.0001
Policy Brief #5: 2004 MAP Results 36
Table 18
Student Title I Status by eMINTS Enrollment
Non- First-Year Second-Year Number
eMINTS eMINTS eMINTS All of
Classroom Classroom Classroom Students Students
Communication Arts
No 75.4 63.9 72.8 73.5 1166
Yes 24.6 36.1 27.2 26.5 420
All Students 100.0 100.0 100.0 100.0
Number of Students 808 108 670 1586
P-Value 0.0346
Mathematics
No 80.1 71.1 64.2 72.8 1537
Yes 19.9 28.9 35.8 27.2 575
All Students 100.0 100.0 100.0 100.0
Number of Students 1064 173 875 2112
P-Value <0.0001
Student Title I Status
Distribution of Students
Table 18 presents the distribution of student Title I status. This table presents the
percentage of students in the FY03 cohort that received Title I assistance in the 2003-
2004 school year. This information differs from Title I school status. As part of their data
submission, FY03 schools were asked to list the individual students who received
assistance from the school’s Title I teachers. DESE’s Office of Federal Programs
provided the funds for this data collection.
Table 18 shows significant differences in the distribution of Title I students on both tests.
For both tests, a higher percentage of Title I students were enrolled in eMINTS
classrooms than non-eMINTS classrooms.
Policy Brief #5: 2004 MAP Results 37
Table 19
Total MAP Score by Student Title I Status and eMINTS Enrollment, Controlling
for School: Communication Arts
Lower 95% Upper 95%
Mean Standard Confidence Confidence
Mean Estimates Estimate Error Interval Interval
eMINTS Enrollment Title I Student
Non-eMINTS Classroom No 648.31 2.22 643.96 652.66
Yes 619.01 2.69 613.74 624.28
First-Year eMINTS Classroom No 647.92 3.74 640.59 655.25
Yes 621.99 4.80 612.57 631.42
Second-Year eMINTS Classroom No 652.45 2.08 648.38 656.53
Yes 627.10 2.60 622.01 632.19
All Students No 651.12 1.99 647.21 655.03
Yes 623.73 2.22 619.37 628.10
Standard Percentage of
Variance Components Estimate Error Z-Value P-Value Variance
School 78.04 28.60 2.73 0.0032 11.24
Residual 616.11 22.12 27.85 <0.0001
Total 694.15
Df Df
Test for Fixed Effects Numerator Denominator F-Value P-Value
Title I Student within eMINTS
Enrollment 5 1554 64.23 <0.0001
Total MAP Score
Tables 19 and 20 present results for the total MAP score. On the Communication Arts
test, Title I students enrolled in second-year eMINTS classrooms scored an average of
8.09 points higher than Title I students enrolled in non-eMINTS classrooms. The
difference for other students was 4.14 points.
The differences on the Mathematics test were much smaller: 1.39 points for Title I
students and 3.04 points for other students.
Policy Brief #5: 2004 MAP Results 38
Table 20
Total MAP Score by Student Title I Status and eMINTS Enrollment, Controlling
for School: Mathematics
Lower 95% Upper 95%
Mean Standard Confidence Confidence
Mean Estimates Estimate Error Interval Interval
eMINTS Enrollment Title I Student
Non-eMINTS Classroom No 649.56 2.99 643.69 655.43
Yes 624.99 3.50 618.12 631.86
First-Year eMINTS Classroom No 651.95 4.02 644.07 659.83
Yes 623.58 5.33 613.14 634.03
Second-Year eMINTS Classroom No 652.60 2.85 647.01 658.20
Yes 626.38 3.24 620.03 632.73
All Students No 651.44 2.66 646.21 656.66
Yes 625.75 2.90 620.07 631.43
Standard Percentage of
Variance Components Estimate Error Z-Value P-Value Variance
School 204.41 59.25 3.45 0.0003 17.68
Residual 952.04 29.59 32.17 <0.0001
Total 1156.45
Df Df
Test for Fixed Effects Numerator Denominator F-Value P-Value
Title I Student within eMINTS
Enrollment 5 2073 38.84 <0.0001
Summary
These results suggest that eMINTS enrollment helps to lessen the achievement gaps of
Black students, special education students and Title I students. The difference for special
education students is larger than the difference for non-eMINTS students on both the
Communication Arts and Mathematics tests. For students receiving Title I services, the
difference on the Communication Arts test is approximately twice that for other students.
On the other hand, disaggregating MAP scores by student enrollment in the free and
reduced lunch program does not show any differential benefit for low-income students.
These differences are largely consistent with differences seen in earlier eMINTS cohorts.
Special education students, in particular, seem to benefit from both enrollment in an
inquiry-based, constructivist classroom environment and from the opportunity to work
with the technology prescribed by the eMINTS program. This finding has been
confirmed with each eMINTS cohort since the analysis of the 2001 MAP scores. In
Policy Brief #5: 2004 MAP Results 39
addition, many eMINTS teachers have described how working with technology and being
able to produce graphically sophisticated classroom products seems to help special
education students overcome the limitations identified by their individual education plans
(IEPs).
On the Communication Arts test, the difference for Black students by eMINTS
enrollment is almost three times the difference for White students. On the other hand,
there are virtually no differences by student race on the Mathematics test. This may be
due to the relatively small numbers of non-White students in fourth-grade eMINTS
classrooms.
These results also suggest a positive benefit for students receiving Title I services. This
relationship has been previously seen at the school level. For example, in the FY02
cohort eMINTS students enrolled in Schoolwide Title I schools scored higher than
eMINTS students in targeted-assistance schools (eMINTS Evaluation Project, 2004). The
results for the FY03 cohort are the first to identify Title I students within participating
schools. On the Communication Arts test, the difference for Title I students is twice the
difference for other students. These results further support the contention that eMINTS
enrollment can make Title I services more effective.
Conclusions
The analysis of MAP scores for students enrolled in the FY03 eMINTS schools reflects
the evolving complexity of the eMINTS program. The FY03 schools were the first to be
funded by federal competitive Title IID grant funds, a change that affected program
eligibility and the deployment of eMINTS classrooms. The analysis has accommodated
these changes by applying more complex statistical models to assess the MAP-score
differences associated with eMINTS enrollment. In particular, the analysis had to control
for variance between schools as well as variance within classrooms.
The results of these analyses suggest several important features of the eMINTS program.
First, eMINTS enrollment does support higher MAP scores. This finding is particularly
true for students enrolled in second-year eMINTS classrooms, those classrooms where
teachers and students began the school year in fully-functioning eMINTS environments.
The second major set of findings suggests that eMINTS helps teachers support student
performance regardless of the types of lessons they conduct. The analysis of the eMINTS
Lesson Typology demonstrates this program strength. On both MAP tests, students
enrolled in second-year eMINTS classrooms scored higher than students enrolled in non-
eMINTS classrooms across all lesson types.
Finally, evidence exists that eMINTS enrollment helps narrow the achievement gap
between special education students and other students persists. eMINTS enrollment also
reduces the achievement gaps for Black students and for students receiving Title I
services.
Policy Brief #5: 2004 MAP Results 40
In sum, these results continue to support the effectiveness of the eMINTS program as a
way to raise MAP scores. These differences may not be as pronounced as they have been
in previous years, but they do exist. They continue to show that the combination of
inquiry-based instructional practices and a carefully defined suite of multimedia
technology support higher student performance.
Policy Brief #5: 2004 MAP Results 41
Appendix A
Key Characteristics of the eMINTS Lesson Typology
Teacher-Centered Lessons
Teachers control the pace and content of work, they control the available time for work
and through their control over their classrooms, they exercise control over the knowledge
available for their students. In teacher-centered lessons, instruction is organized around
discrete instructional tasks, (for example, completing worksheets) and a conventional
single disciplinary structure dictates the sequence of these tasks. Students in these lessons
are largely passive receivers of knowledge. A lesson focuses substantively on the mastery
and memorization of basic facts.
Hybrid Lessons
Teachers attempt inquiry-based, student-centered lessons while maintaining key
characteristics of a teacher-centered lesson.
Student-Centered Facilitated Lessons
Teachers vary their teaching activities and methodologies according to each lesson's
instructional goal. Teachers lecture, conduct group discussions, facilitate student
brainstorming and so on, as appropriate to the academic goal of a lesson. Teachers
structure each lesson as a series of tasks and subtasks. Each task is broken down into
grade-appropriate activities, that is, individual activities constructed to meet both the
abilities of the students and the appropriate curriculum standards. These activities have an
explicit sequence to them and each activity has stated criteria for completion. In all cases,
student-performance criteria are stated and reinforced at multiple points throughout the
lesson. Teachers construct lesson goals by combining district curriculum goals from a
variety of subject areas. Lessons are less focused on issues of basic mastery and stress
higher order comprehension and logic. Finally, opportunities for consultation and
communication among students and with the teacher are structured into the execution of a
lesson.
Student-Centered Unfacilitated Lessons
Teachers typically focus on enabling the technical activities of their students (for
example, helping with searches, instructing students in use of software and so forth), or
they remove themselves from most classroom activity. Both roles have the effect of
distancing a teacher from the academic goals of a given lesson and from the learning
environment in which the students work.
Policy Brief #5: 2004 MAP Results 42
References
eMINTS Evaluation Project (2001a). A General Typology of eMINTS Lessons. Columbia,
Missouri: Office of Social and Economic Data Analysis.
http://www.emints.org/evaluation/reports/lesson-typology.pdf
eMINTS Evaluation Project (2001b). Rooms of change and achievements: teachers'
comments on working in an eMINTS classroom. Columbia, Missouri: Office of
Social and Economic Data Analysis.
http://www.emints.org/evaluation/reports/teacher-change.shtml
eMINTS Evaluation Project (2002). Analysis of 2001 MAP Results for eMINTS Students.
Columbia, Missouri: Office of Social and Economic Data Analysis.
http://www.emints.org/evaluation/reports/map2001-emints.pdf
eMINTS Evaluation Project (2004). Analysis of 2003 MAP Results for eMINTS Students.
Columbia, Missouri: Office of Social and Economic Data Analysis.
http://www.emints.org/evaluation/reports/map2003.pdf
Smith, R.W. (1997). Visual Hypothesis Testing with Confidence Intervals. Proceedings of
the Twenty-Second Annual SAS® Users Group International Conference.
Wilkinson, L. (1999). Statistical Methods in Psychology Journals: Guidelines and
Explanations. American Psychologist. August 1999, Vol. 54, No. 8, 594–604.
Policy Brief #5: 2004 MAP Results 43
Related docs
