A publication of the Magnet Schools Assistance Program Technical Assistance Center
VoluMe 1 • Issue 3 • aprIl 2011
Fidelity of Program Scalability
Implementation It is important to maintain high FOI
when scaling up successful, innovative
An important element of establishing and managing a successful magnet programs. Scalability is the
magnet school program is fidelity of implementation (FOI). process of introducing a high-performing
Measuring FOI is critical to determining how well a magnet program is executed magnet program into other classrooms
during its life cycle and understanding why it is or is not achieving projected and schools with the goal of producing
outcomes. similar positive outcomes.
It is important for magnet school Implementing a magnet program
personnel and evaluators to understand across classrooms and schools requires
FOI concepts and processes and to precision; as such, it is important to
recognize that FOI assessment must be maintain FOI to critical program
part of their program’s strategic plan. components. This includes determining
whether the magnet curriculum is being
Often referred to as “adherence” or implemented as planned, the appropriate
“integrity,” the FOI framework extends classroom materials are being used, and
beyond these concepts to include magnet staff have been trained.
exposure, quality, and participant
responsiveness. The more inclusive To ensure sufficient conditions for
definition of FOI measures the extent implementation, it is equally important
to which the enacted magnet program is consistent with the program design. to maintain FOI to the organizational
While there is no single method for measuring FOI, evaluators agree that assessing systems. Explicit descriptions of system
a program’s critical components helps to identify program deviations or dilutions. variables—such as funding, personnel
Given that critical components influence expected program outcomes, tracking allocations, teaming practices, and data
FOI data will help magnet schools enhance or modify their programs and explain collection—can provide appropriate
both intended and unintended program results. settings for operationalizing critical
magnet program components.
This issue of The MagneT CoMpass highlights FOI principles and assessment strategies,
tips for tracking magnet program implementation, and additional FOI resources.
INSIDE THIS ISSUE Promoting diversity,
Principles of Fidelity of Implementing a High-Quality academic excellence,
Implementation . . . . . . . . . . . . . . 3 Magnet Curriculum . . . . . . . . . . . 8
and equity through
Implementation Matters . . . . . . . . 4 REGULAR FEATURES
Strategies for Assessing Fidelity of The Needle Point . . . . . . . . . . . . . 2
Implementation . . . . . . . . . . . . . . 5 Magnet Moments . . . . . . . . . . . . . 7
Fidelity of Implementation
Checklist . . . . . . . . . . . . . . . . . . . 6
A Message From the
This publication is produced by the
Magnet Schools Assistance Program
Technical Assistance Center (MSAP
Magnet Schools Assistance Program Center), a technical assistance resource
for MSAP grantees and the general
magnet schools community. The MSAP
current environment of funding cuts, Center provides grantees and magnet
coupled with a critical need to improve our schools with technical support by
offering tools, information, and strategies
nation’s schools, magnet programs need not
to assist in planning, implementing, and
only to raise student academic achievement
sustaining programs. The ultimate goal
and prepare students for the workforce and of the MSAP Center is to help magnet
Left to right: Anna Hinton, Director of Parental Options and
Information; Rosie Kelley, MSAP Team Lead, Education college but also to understand which of schools provide communities with
Program Specialist; and Brittany Beth, MSAP Program Officer,
Management and Program Analyst
their components are successful and why. If educational opportunities that promote
programs are not implemented as designed, diversity, academic excellence, and
The U.S. Department of Education it is impossible to evaluate their effectiveness equity.
is committed to being an “engine of accurately or replicate them. As the
innovation” for our public education system. Department looks to invest in programs that www .msapcenter .com
The Department has defined innovations demonstrate innovation and effectiveness, it
as “strategies, products, or approaches that
is increasingly important for magnet schools THE
to implement their programs according to
the original designs and to maintain high
program fidelity as they continue to operate.
significantly improve upon the status quo DIRECTOR
and that are scalable to address persistent
education challenges.” Although innovations schools,
Creating magnet schools,
occur daily in schools and classrooms, they
often do not extend to the school down the
getting them up and CONTRIBUTING WRITERS
street or the classroom down the hall. Thus, running, and serving the Seewan Eng
the Magnet Schools Assistance Program
(MSAP) strives to move the innovation right student mix takes a lot GRAPHIC DESIGN
agenda forward by helping to identify and of upfront time and effort.
then scale up effective innovations. EDITORIAL SERVICES
But the enduring challenge Synergy Enterprises, Inc .
Data show that several funded magnet
schools have raised student achievement, is making them work—
The views expressed in the newsletter
increased diversity, and improved student keeping the organizational do not necessarily reflect the position or policy
academic options. In designing and of the U.S. Department of Education (ED) and no
official endorsement by ED should be inferred.
implementing successful programs, eye
eye fixed squarely on the
magnet schools must be able to identify prize, being relentless about
and demonstrate how innovative practices
contribute to the schools’ successes. staying
staying the course, sustaining
Beforehand, however, they must ensure
that their magnet programs are being
momentum, and keeping
implemented with fidelity. aliv
commitment alive. This newsletter was produced in
whole or in part with funds from the
This issue of The MagneT CoMpass focuses U.S. Department of Education under
on FOI, without which magnet programs contract number: ED-OII-10-C-0079.
cannot adequately measure contributing
factors of successes or setbacks. In the
aprIl 2011 • page 2
aprIl 2011 • page
Principles of Fidelity of Quality is a qualitative aspect of program delivery
that must be defined and scaled prior to program
Implementation implementation. Addressing the skill and manner
in which activities are conducted informs magnet
Implementing a well-designed program with fidelity schools on how well their teachers know the program, buy
into the program, and demonstrate their capacity. Measuring
is a requisite for success. Current literature points quality requires determining the steps that staff must take to
to five concepts that must be examined when improve their magnet program delivery and instruction.
1 5 Participant Responsiveness is the extent of
Differentiation is the process of identifying the student participation or engagement in the magnet
critical components that distinguish your magnet program activities and content. Measuring
program from others as well as approaches and participant responsiveness recognizes that some
activities that influence outcomes. Determining critical components essential for implementation reside with
your magnet program’s critical components allows you to the students; it includes observing and documenting student
assess their integrity. interactions with the magnet program content, the teacher,
and one another during magnet activities.
2 Adherence/Integrity is the
determination of whether the
magnet program is being delivered
as designed. Comparing the design
to the systems implemented or services
delivered enables magnet staff to measure
whether the program is being implemented as
planned. Measuring adherence involves
examining whether the target student
population is being served, magnet school
staff is being trained properly, selected
protocols are being implemented, and
classroom materials are being used.
3 Exposure/Dosage is the
amount of magnet program content
received. Assessing the extent to
which specific magnet program
activities are implemented gives staff an idea
of whether the magnet program has been
integrated fully in the school—too little
exposure can weaken a magnet program.
Measuring exposure includes documenting
the amounts of theme-based instruction
students receive, activities families engage in,
and professional development that staff
aprIl 2011 • page 3
The Magnet Compass will present a
series of articles featuring exceptional,
innovative researchers. This article
features David Kikoler, principal officer at
American Education Solutions, Inc.(AES),
and an expert in program implementation.
AES provides proposal development,
program planning, and evaluation services
to school districts throughout the United States.
“Fidelity is important; implementing a When working with grantees on implementing their
model as written is important because magnet programs, Kikoler recommends using a set of
it was developed with a lot of thought practices within an implementation framework that includes
and is supported by research,” explains the magnet program model; the implementation plan;
David Kikoler, a longtime proponent the performance measures; and a system of assessment,
and supporter of magnet schools. feedback, and support. While this framework is important
Having managed project evaluations for all magnet programs, it is of particular importance for
David Kikoler for grant recipients of MSAP, Voluntary homegrown magnet programs in which the school and/or
Solutions, Inc . Public School Choice, and Smaller district designs the model, often based on practices supported
Learning Communities, Kikoler by research. Validated programs, such as International
works in collaboration with practitioners—former magnet Baccalaureate and Montessori, have spent years developing
teachers, principals, and project directors—and researchers and fine-tuning their models. In both cases, programs must
and universities to bring the practitioner’s and researcher’s be particularly dedicated to fully understanding their models
perspectives to every evaluation. and their components and being faithful to them.
This emphasis on FOI is a natural result of Kikoler’s Kikoler discusses how grantees sometimes leave out core
evaluation work through the years. Noting the influence components when implementing magnet programs, often
of fidelity on grant programs, AES partnered with Dr. attempting to adapt models to specific conditions or
David Silver of the National Center for Research on unforeseen circumstances: “When we look at fidelity of
Evaluation, Standards, and Student Testing to measure the implementation in a magnet program, we expect to see
relationship between the 2004 MSAP grantees’ degrees of the activities that are described in the proposal because, in
program implementation and their corresponding academic essence, that’s what the model is.” He continues, “You always
achievement and reduction of minority group isolation start with your proposal and you adapt only when you feel
(MGI). The 34 schools that participated in the study were you have to. If a district has spent time properly planning
sorted into groups according to their levels of program its homegrown [magnet program], there may be some reason
implementation and school performance status. The results for modifying it, but there shouldn’t be a need for major
showed that most schools with medium or high levels of adjustments. If the proposal wasn’t well thought through,
program implementation either improved or maintained their I could see more of a need for adapting. But my view is
status as high-performing schools and reduced MGI. “The that you have to be very careful not to change the scope of
literature is filled with the importance of maintaining fidelity work. Modifications of validated programs must be done in
of implementation; we were just able to drive the point home consultation with the developer.”
in a more concrete way for the districts that we work with,”
Regularly assessing FOI increases a
magnet program’s chances for success.
While magnet schools must remain
relevant to their environments, they
should also adhere to the theoretical
concepts and critical components of
their programs. The following strategies
will help magnet schools focus on critical
program components and ultimately
help achieve program goals.
1 Clearly define your
Having a complete understanding
of the magnet program design, its
critical components, and implementation
strategies will provide a foundation for success.
Clearly defining these program features as well
as regularly communicating them to magnet
staff will facilitate high FOI at various stages
of program delivery. Magnet staff must also
know expectations for delivering all program aspects with
quality; therefore, it is important to show them what high-
quality programming looks like. Checklists, manuals, and
handbooks are useful for operationalizing your program and Provide ongoing professional
helping staff execute high-quality magnet programming.
development and support.
Create FOI processes. Giving staff the knowledge, skills, and motivation
Establishing processes for tracking and measuring to implement magnet program activities increases
FOI will keep magnet programs moving towards the likelihood of high FOI. Successful magnet schools
set goals and will ensure transparency for all aspects provide ongoing, well-conducted professional development
of your program. The FOI processes should include clear and support. Well-conducted professional development
protocols that define when, where, and how data should be also helps to decrease resistance to change and to generate
collected, analyzed, and used to enhance programming. enthusiasm and commitment among magnet program
FOI protocols must measure fidelity both directly (e.g., staff. Professional development should be job-embedded,
observation checklists) and indirectly (e.g., samples of accounting for individual staff feedback and additional
student work, videos of teacher instruction) and be routinely support to staff that need it.
monitored either internally or by an evaluator to ensure that
magnet programs maintain high FOI.
aprIl 2011 • page 5
Fidelity of Implementation Checklist
Magnet staff at all levels play important roles in ensuring FOI. This table outlines specific activities and
responsibilities for magnet school coordinators, principals, and teachers.
MAGNET COORDINATOR PRINCIPAL TEACHER
• Monitor magnet theme • Create infrastructure that supports a • Collect data that help
integration, instructional collaborative implementation process . corroborate FOI .
delivery, and curriculum
development . • Provide routine access to professional • Review existing
development and other resources that checklists and
• Provide professional elevate program delivery . curriculum manuals
development and for guidance on
support . • Conduct regular walkthroughs, implementation .
observations, and discussions with staff
• Coordinate regular to assess FOI . • Self-assess instructional
teacher observations . practices and make
• Determine whether classroom changes as necessary .
• Provide meaningful and performance requires intervention, and
specific feedback to ensure a plan is in place to address the • Review program
teachers . situation . fidelity data with
• Respond to teacher • Ensure that the magnet program has the colleagues .
requests for assistance necessary materials and equipment for
or information . implementation .
aprIl 2011 • page 6
Fidelity: An Ongoing Process
“Stay true to your plan of operation. King/Robinson’s commitment to maintaining high FOI
Stay close to it; follow it. It’s not has been replicated in the district’s other magnet programs.
going to be easy; you’re going to Canelli explains, “At the beginning of the first year of
have bumps along the road. But, if the 2007 MSAP cycle, it was very difficult to implement
you stick to whatever you wrote in anything. You have a theme, but unless you have professional
that grant, you’re going to have great development and training for teachers to figure out how
success,” comments Robert Canelli, to implement this theme, how do you do that? So, at the
supervisor of Magnet Schools beginning, there was a lot of planning, training, and
in New Haven Public Schools professional development around implementation. You
(NHPS) in Connecticut. Having have that plan of
Supervisor of Magnet
worked in NHPS for 34 years, with operation guiding
many of those years as a magnet you, and as each
school teacher or principal, Canelli year passed,
is ecstatic to report that all of the 2007 MSAP grantee more teachers
schools made Adequate Yearly Progress (AYP) this past year. bought into the
According to Canelli, maintaining fidelity to the plan of program. There
operation, or the implementation plan, is fundamental for were teachers that
becoming a successful magnet school. didn’t want to be
in magnet; there
were teachers that
thought it was
just extra work
who said, ‘How
am I going to fit
all of this into
Canelli recalls that before becoming a magnet school, King/
Robinson International Baccalaureate School was a “cement
I’m supposed to
block” that looked more like a prison than a school. In
be doing?’ And my message to everybody was that being a
addition to an unwelcoming look and tough neighborhood
magnet school is not something extra. You need to figure out
location, it was also one of the lowest performing schools
how to implement your theme so you’re teaching what you’re
in NHPS. After receiving MSAP funding in 2004, King/
supposed to be teaching, but in the context of the theme.
Robinson built a new state-of-the-art building, brought in a
And once they get the hang of it, they write units, and they
new principal, hired tutors to work with lower performing
just go crazy. It’s just amazing where they started and where
students, and implemented the only K-8 International
they are now. So, you can’t do this overnight. When you talk
Baccalaureate (IB) program model in Connecticut. “What I
about fidelity, it’s got to be an ongoing process, year after
know about IB,” Canelli remarks, “is that it’s very rigorous;
year. And that’s important.”
it’s very scripted. They stuck to it.” Following these changes,
King/Robinson made AYP in 2009 through the Safe Harbor
http://msapcenter com/resource aspx
For more about FOI, visit http://msapcenter .com/resource .aspx .
aprIl 2011 • page 7
CEMSE. (2008). Critical Components: Definitions and
Explanations. Chicago: University of Chicago.
Century, J., Freeman, C., and Rudnick, M. (2008, March). A
Framework for Measuring and Accumulating Knowledge About
Fidelity of Implementation (FOI) of Science Instructional
Materials. Paper presented at the annual meeting of the
Curriculum National Association for Research in Science Teaching.
Formica, S., and Maika, M.M. (2010, January). Measuring
The following steps can facilitate high-quality and Tracking Fidelity [PowerPoint slides]. Retrieved April
4, 2011, from http://www.humaninteract.org/images/
instruction in the classroom: FindingBalance2.pdf.
Horner, R.H., and Sugai, G. (2006, February). Policy Brief: Scaling
1 Develop specific criteria to ascertain which practices up Effective Educational Innovations. Retrieved April 11, 2011,
adhere to or lack fidelity. Johnson, E., Mellard, D.F., Fuchs, D., and McKnight, M.A.
2 Train instructional mentors and school coaches in (2006). Responsiveness to Intervention (RTI): How to Do
It. Lawrence, KS: National Research Center on Learning
processes that measure fidelity.
Mihalic, S. (n.d.). Implementation Fidelity. Unpublished
3 Assign trained, qualified staff to observe instructional manuscript.
methods. U.S. Department of Education, Office of Innovation and
Improvement, Innovations in Education: Creating Successful
4 Schedule regular classroom observations of specific,
Magnet Schools Programs, Washington, D.C., 2004.
U.S. Department of Health and Human Services, Substance
Abuse and Mental Health Services Administration, Center
5 Provide more frequent observations for teachers with less
for Substance Abuse Prevention. Finding the Balance: Program
Fidelity and Adaptation in Substance Abuse Prevention:
magnet program experience and who request or need Executive Summary of a State-of-the-Art Review. Washington,
additional support. DC: Author. Retrieved April 4, 2011, from http://www.
6 Provide direct feedback to staff in
written observation assessments;
improvement plans; and
the demonstration of lessons.
aprIl 2011 • page 8