U.S. Department of Education
Strategies for Effective
The contents of this presentation were produced
by the Coalition for Evidence-Based Policy, in
partnership with the National Opinion Research
Center (NORC) at the University of Chicago,
under a contract with the Institute of Education
Find a researcher with expertise in
conducting rigorous impact evaluations to
include on the study team.
Contact authors of previous, well-
designed impact evaluations.
Ask the evaluator to identify a plan
that describes in non-technical language,
how they would conduct the study.
Decide what research question(s) the
study seeks to answer.
Evaluate a specific, well-defined MSP
Measure the effect of the program on teachers’
Measure the impact on student knowledge.
If resources permit, ask questions about the
long-term effect of the MSP projects.
Decide on the study design.
Decide on overall design: preferably a
randomized controlled trial, or if not possible, a
well-matched comparison-group study.
Decide the sample size to measure teacher
content knowledge – at least 90 teachers needed.
Decide the sample size to measure student
achievement – at least 60 teachers needed.
Step 3 continued:
Decide how to recruit and allocate teachers.
The simplest way to allocate teachers in a RCT is
to apply random assignment to the entire sample.
However, if there are big differences across the
group (high-achieving, low-achieving) then
grouping in blocks and then random assignment.
In a comparison group study the teachers must be
very closely matched: First, teachers must be closely
matched by achievement levels; and second, by
Step 3 continued:
Decide how to measure MSP project outcomes
for student achievement. There are 3 conditions:
--must obtain scores for individual
--must obtain scores before they enter the
teachers class, and at the end of the study;
--must be able to convert the scores so that
they will enable comparisons across grade
Decide how to measure teacher content
Step 3 Example:
Gain the cooperation of teachers and
Identify one or more senior-level advocates for the
Gain teachers’ cooperation by explaining the
benefits of the research design.
Satisfy privacy (FERPA) concerns in your design.
Special consideration for matched comparison
group of teachers from another school district.
Allocate teachers to the program and control
(or matched comparison) groups.
The evaluator should conduct random
assignment of teachers rather than project
Ask participating teachers not to share their
Ensure that students are assigned to classes
using the normal procedures.
Collect the data needed to measure the
MSP project’s effectiveness.
Make process as short and streamlined as possible.
Provide a unique personal identifier for each
At the time of assignment, obtain pre-program
Make every effort to collect data from 80 percent of
the original sample of teachers.
Analyze and report the study’s results.
Obtain regression-adjusted estimates of
the MSP project’s effect on student
Use the students’ pre-program test score
as a covariate.
Use data from all of the original
Mission is to provide federal, state, and local education officials, researchers, program
providers, and educators with practical, easy-to-use tools to (i) advance rigorous
evaluations of educational interventions (i.e., programs, products, practices, and
policies), and (ii) identify and implement evidence-based interventions.
Phone: 1-866-WWC-9799 (8am-8pm ET Mon-Fri)