Document Sample
					Magnet CoMpass

                             A publication of the Magnet Schools Assistance Program Technical Assistance Center
                                                                                                                                    VoluMe 1 • Issue 3 • aprIl 2011

                                                                                                                                   Fidelity of
                                                                                                                                   Implementation for
Fidelity of Program                                                                                                                Scalability
Implementation                                                                                                                     It is important to maintain high FOI
                                                                                                                                   when scaling up successful, innovative
An important element of establishing and managing a successful                                                                     magnet programs. Scalability is the
magnet school program is fidelity of implementation (FOI).                                                                         process of introducing a high-performing
Measuring FOI is critical to determining how well a magnet program is executed                                                     magnet program into other classrooms
during its life cycle and understanding why it is or is not achieving projected                                                    and schools with the goal of producing
outcomes.                                                                                                                          similar positive outcomes.

                                                                       It is important for magnet school                           Implementing a magnet program
                                                                       personnel and evaluators to understand                      across classrooms and schools requires
                                                                       FOI concepts and processes and to                           precision; as such, it is important to
                                                                       recognize that FOI assessment must be                       maintain FOI to critical program
                                                                       part of their program’s strategic plan.                     components. This includes determining
                                                                                                                                   whether the magnet curriculum is being
                                           Often referred to as “adherence” or                                                     implemented as planned, the appropriate
                                           “integrity,” the FOI framework extends                                                  classroom materials are being used, and
                                           beyond these concepts to include                                                        magnet staff have been trained.
                                           exposure, quality, and participant
                                           responsiveness. The more inclusive                                                      To ensure sufficient conditions for
                                           definition of FOI measures the extent                                                   implementation, it is equally important
to which the enacted magnet program is consistent with the program design.                                                         to maintain FOI to the organizational
While there is no single method for measuring FOI, evaluators agree that assessing                                                 systems. Explicit descriptions of system
a program’s critical components helps to identify program deviations or dilutions.                                                 variables—such as funding, personnel
Given that critical components influence expected program outcomes, tracking                                                       allocations, teaming practices, and data
FOI data will help magnet schools enhance or modify their programs and explain                                                     collection—can provide appropriate
both intended and unintended program results.                                                                                      settings for operationalizing critical
                                                                                                                                   magnet program components.
This issue of The MagneT CoMpass highlights FOI principles and assessment strategies,
tips for tracking magnet program implementation, and additional FOI resources.

INSIDE THIS ISSUE                                                                                                                   Promoting diversity,
Principles of Fidelity of                                              Implementing a High-Quality                                  academic excellence,
Implementation  .  .  .  .  .  .  .  .  .  .  .  .  .  . 3             Magnet Curriculum  .  .  .  .  .  .  .  .  .  .  . 8
                                                                                                                                    and equity through
Implementation Matters  .  .  .  .  .  .  .  . 4                       REGULAR FEATURES
                                                                                                                                    magnet schools
Strategies for Assessing Fidelity of                                   The Needle Point  .  .  .  .  .  .  .  .  .  .  .  .  . 2
Implementation  .  .  .  .  .  .  .  .  .  .  .  .  .  . 5             Magnet Moments  .  .  .  .  .  .  .  .  .  .  .  .  . 7
Fidelity of Implementation
Checklist  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  . 6
Magnet C


A Message From the
                                                                                                                  This publication is produced by the
                                                                                                                  Magnet Schools Assistance Program
                                                                                                                  Technical Assistance Center (MSAP

Magnet Schools Assistance Program                                                                                 Center), a technical assistance resource
                                                                                                                  for MSAP grantees and the general
                                                                                                                  magnet schools community. The MSAP
                                                               current environment of funding cuts,               Center provides grantees and magnet
                                                               coupled with a critical need to improve our        schools with technical support by
                                                                                                                  offering tools, information, and strategies
                                                               nation’s schools, magnet programs need not
                                                                                                                  to assist in planning, implementing, and
                                                               only to raise student academic achievement
                                                                                                                  sustaining programs. The ultimate goal
                                                               and prepare students for the workforce and         of the MSAP Center is to help magnet
Left to right: Anna Hinton, Director of Parental Options and
Information; Rosie Kelley, MSAP Team Lead, Education           college but also to understand which of            schools provide communities with
Program Specialist; and Brittany Beth, MSAP Program Officer,
Management and Program Analyst
                                                               their components are successful and why. If        educational opportunities that promote
                                                               programs are not implemented as designed,          diversity, academic excellence, and
The U.S. Department of Education                               it is impossible to evaluate their effectiveness   equity.
is committed to being an “engine of                            accurately or replicate them. As the
innovation” for our public education system.                   Department looks to invest in programs that             www .msapcenter .com
The Department has defined innovations                         demonstrate innovation and effectiveness, it
as “strategies, products, or approaches that

                                                               is increasingly important for magnet schools                      THE
                                                               to implement their programs according to

                                                                                                                     C Mpass
                                                               the original designs and to maintain high
                                                               program fidelity as they continue to operate.

significantly improve upon the status quo                                                                                            DIRECTOR
and that are scalable to address persistent
                                                                                                                                Manya Walton
education challenges.” Although innovations                                            schools,
                                                                     Creating magnet schools,
occur daily in schools and classrooms, they
often do not extend to the school down the
                                                                     getting them up and                                    CONTRIBUTING WRITERS
                                                                                                                              Katherine Moone
street or the classroom down the hall. Thus,                         running, and serving the                                   Seewan Eng
the Magnet Schools Assistance Program
(MSAP) strives to move the innovation                                right student mix takes a lot                              GRAPHIC DESIGN
                                                                                                                              Manuel Valencia
agenda forward by helping to identify and                            of upfront time and effort.
then scale up effective innovations.                                                                                           EDITORIAL SERVICES
                                                                     But the enduring challenge                          Synergy Enterprises, Inc .
Data show that several funded magnet
schools have raised student achievement,                             is making them work—
                                                                                                                         The views expressed in the newsletter
increased diversity, and improved student                            keeping the organizational                     do not necessarily reflect the position or policy
academic options. In designing and                                                                                of the U.S. Department of Education (ED) and no
                                                                                                                    official endorsement by ED should be inferred.
implementing successful programs,                                    eye
                                                                     eye fixed squarely on the
magnet schools must be able to identify                              prize, being relentless about
and demonstrate how innovative practices
contribute to the schools’ successes.                                staying
                                                                     staying the course, sustaining
Beforehand, however, they must ensure
that their magnet programs are being
                                                                     momentum, and keeping
implemented with fidelity.                                                         aliv
                                                                     commitment alive.                                   This newsletter was produced in
                                                                                                                        whole or in part with funds from the
This issue of The MagneT CoMpass focuses                                                                               U.S. Department of Education under
on FOI, without which magnet programs                                                                                  contract number: ED-OII-10-C-0079.

cannot adequately measure contributing
factors of successes or setbacks. In the
                                                                                                                                 aprIl 2011 • page 2
                                                                                                                                   aprIl 2011 • page
Magnet C


Principles of Fidelity of                                               Quality is a qualitative aspect of program delivery
                                                                        that must be defined and scaled prior to program
Implementation                                                          implementation. Addressing the skill and manner
                                                                        in which activities are conducted informs magnet
Implementing a well-designed program with fidelity            schools on how well their teachers know the program, buy
                                                              into the program, and demonstrate their capacity. Measuring
is a requisite for success. Current literature points         quality requires determining the steps that staff must take to
to five concepts that must be examined when                   improve their magnet program delivery and instruction.
measuring FOI.

1                                                             5          Participant Responsiveness is the extent of
          Differentiation is the process of identifying the              student participation or engagement in the magnet
          critical components that distinguish your magnet               program activities and content. Measuring
          program from others as well as approaches and                  participant responsiveness recognizes that some
          activities that influence outcomes. Determining     critical components essential for implementation reside with
your magnet program’s critical components allows you to       the students; it includes observing and documenting student
assess their integrity.                                       interactions with the magnet program content, the teacher,
                                                              and one another during magnet activities.

2          Adherence/Integrity is the
           determination of whether the
           magnet program is being delivered
           as designed. Comparing the design
to the systems implemented or services
delivered enables magnet staff to measure
whether the program is being implemented as
planned. Measuring adherence involves
examining whether the target student
population is being served, magnet school
staff is being trained properly, selected
protocols are being implemented, and
classroom materials are being used.

3          Exposure/Dosage is the
           amount of magnet program content
           received. Assessing the extent to
           which specific magnet program
activities are implemented gives staff an idea
of whether the magnet program has been
integrated fully in the school—too little
exposure can weaken a magnet program.
Measuring exposure includes documenting
the amounts of theme-based instruction
students receive, activities families engage in,
and professional development that staff

                                                                                                              aprIl 2011 • page 3
Magnet C


The Magnet Compass will present a 

series of articles featuring exceptional, 

innovative researchers. This article 

features David Kikoler, principal officer at 

American Education Solutions, Inc.(AES), 

and an expert in program implementation. 

AES provides proposal development, 

program planning, and evaluation services
to school districts throughout the United States.

                      “Fidelity is important; implementing a       When working with grantees on implementing their
                      model as written is important because        magnet programs, Kikoler recommends using a set of
                      it was developed with a lot of thought       practices within an implementation framework that includes
                      and is supported by research,” explains      the magnet program model; the implementation plan;
                      David Kikoler, a longtime proponent          the performance measures; and a system of assessment,
                      and supporter of magnet schools.             feedback, and support. While this framework is important
                      Having managed project evaluations           for all magnet programs, it is of particular importance for
    David Kikoler     for grant recipients of MSAP, Voluntary      homegrown magnet programs in which the school and/or
 American Education
   Solutions, Inc .   Public School Choice, and Smaller            district designs the model, often based on practices supported
                      Learning Communities, Kikoler                by research. Validated programs, such as International
works in collaboration with practitioners—former magnet            Baccalaureate and Montessori, have spent years developing
teachers, principals, and project directors—and researchers        and fine-tuning their models. In both cases, programs must
and universities to bring the practitioner’s and researcher’s      be particularly dedicated to fully understanding their models
perspectives to every evaluation.                                  and their components and being faithful to them.
This emphasis on FOI is a natural result of Kikoler’s              Kikoler discusses how grantees sometimes leave out core
evaluation work through the years. Noting the influence            components when implementing magnet programs, often
of fidelity on grant programs, AES partnered with Dr.              attempting to adapt models to specific conditions or
David Silver of the National Center for Research on                unforeseen circumstances: “When we look at fidelity of
Evaluation, Standards, and Student Testing to measure the          implementation in a magnet program, we expect to see
relationship between the 2004 MSAP grantees’ degrees of            the activities that are described in the proposal because, in
program implementation and their corresponding academic            essence, that’s what the model is.” He continues, “You always
achievement and reduction of minority group isolation              start with your proposal and you adapt only when you feel
(MGI). The 34 schools that participated in the study were          you have to. If a district has spent time properly planning
sorted into groups according to their levels of program            its homegrown [magnet program], there may be some reason
implementation and school performance status. The results          for modifying it, but there shouldn’t be a need for major
showed that most schools with medium or high levels of             adjustments. If the proposal wasn’t well thought through,
program implementation either improved or maintained their         I could see more of a need for adapting. But my view is
status as high-performing schools and reduced MGI. “The            that you have to be very careful not to change the scope of
literature is filled with the importance of maintaining fidelity   work. Modifications of validated programs must be done in
of implementation; we were just able to drive the point home       consultation with the developer.”
in a more concrete way for the districts that we work with,”
explains Kikoler.
Magnet C


Strategies for
Assessing Fidelity
of Implementation
Regularly assessing FOI increases a
magnet program’s chances for success.
While magnet schools must remain
relevant to their environments, they
should also adhere to the theoretical
concepts and critical components of
their programs. The following strategies
will help magnet schools focus on critical
program components and ultimately
help achieve program goals.

1         Clearly define your
           Having a complete understanding
           of the magnet program design, its
critical components, and implementation
strategies will provide a foundation for success.
Clearly defining these program features as well
as regularly communicating them to magnet
staff will facilitate high FOI at various stages
of program delivery. Magnet staff must also
know expectations for delivering all program aspects with
quality; therefore, it is important to show them what high-

quality programming looks like. Checklists, manuals, and
handbooks are useful for operationalizing your program and                  Provide ongoing professional
helping staff execute high-quality magnet programming.

                                                                            development and support.
          Create FOI processes.                                               Giving staff the knowledge, skills, and motivation
          Establishing processes for tracking and measuring                   to implement magnet program activities increases
          FOI will keep magnet programs moving towards             the likelihood of high FOI. Successful magnet schools
          set goals and will ensure transparency for all aspects   provide ongoing, well-conducted professional development
of your program. The FOI processes should include clear            and support. Well-conducted professional development
protocols that define when, where, and how data should be          also helps to decrease resistance to change and to generate
collected, analyzed, and used to enhance programming.              enthusiasm and commitment among magnet program
FOI protocols must measure fidelity both directly (e.g.,           staff. Professional development should be job-embedded,
observation checklists) and indirectly (e.g., samples of           accounting for individual staff feedback and additional
student work, videos of teacher instruction) and be routinely      support to staff that need it.
monitored either internally or by an evaluator to ensure that
magnet programs maintain high FOI.

                                                                                                                   aprIl 2011 • page 5
Magnet C


Fidelity of Implementation Checklist
Magnet staff at all levels play important roles in ensuring FOI. This table outlines specific activities and
responsibilities for magnet school coordinators, principals, and teachers.

   MAGNET COORDINATOR                              PRINCIPAL                                TEACHER

  • Monitor magnet theme          • Create infrastructure that supports a           • Collect data that help
    integration, instructional      collaborative implementation process .            corroborate FOI .
    delivery, and curriculum
    development .                 • Provide routine access to professional          • Review existing
                                    development and other resources that              checklists and
  • Provide professional            elevate program delivery .                        curriculum manuals
    development and                                                                   for guidance on
    support .                     • Conduct regular walkthroughs,                     implementation .
                                    observations, and discussions with staff
  • Coordinate regular              to assess FOI .                                 • Self-assess instructional
    teacher observations .                                                            practices and make
                                  • Determine whether classroom                       changes as necessary .
  • Provide meaningful and          performance requires intervention, and
    specific feedback to            ensure a plan is in place to address the        • Review program
    teachers .                      situation .                                       fidelity data with
                                                                                      supervisors and
  • Respond to teacher            • Ensure that the magnet program has the            colleagues .
    requests for assistance         necessary materials and equipment for
    or information .                implementation .

                                                                                                     aprIl 2011 • page 6
Magnet C


Fidelity: An Ongoing Process
                            “Stay true to your plan of operation.    King/Robinson’s commitment to maintaining high FOI
                            Stay close to it; follow it. It’s not    has been replicated in the district’s other magnet programs.
                            going to be easy; you’re going to        Canelli explains, “At the beginning of the first year of
                            have bumps along the road. But, if       the 2007 MSAP cycle, it was very difficult to implement
                            you stick to whatever you wrote in       anything. You have a theme, but unless you have professional
                            that grant, you’re going to have great   development and training for teachers to figure out how
                            success,” comments Robert Canelli,       to implement this theme, how do you do that? So, at the
                            supervisor of Magnet Schools             beginning, there was a lot of planning, training, and
                            in New Haven Public Schools              professional development around implementation. You
       Robert Canelli
                            (NHPS) in Connecticut. Having                                                         have that plan of
    Supervisor of Magnet 
  worked in NHPS for 34 years, with                                                     operation guiding
                            many of those years as a magnet                                                       you, and as each
                            school teacher or principal, Canelli                                                  year passed,
is ecstatic to report that all of the 2007 MSAP grantee                                                           more teachers
schools made Adequate Yearly Progress (AYP) this past year.                                                       bought into the
According to Canelli, maintaining fidelity to the plan of                                                         program. There
operation, or the implementation plan, is fundamental for                                                         were teachers that
becoming a successful magnet school.                                                                              didn’t want to be
                                                                                                                  in magnet; there
                                                                                                                  were teachers that
                                                                                                                  thought it was
                                                                                                                  just extra work
                                                                                                                  who said, ‘How
                                                                                                                  am I going to fit
                                                                                                                  all of this into
Canelli recalls that before becoming a magnet school, King/
                                                                                                                  everything that
Robinson International Baccalaureate School was a “cement
                                                                                                                  I’m supposed to
block” that looked more like a prison than a school. In
                                                                     be doing?’ And my message to everybody was that being a
addition to an unwelcoming look and tough neighborhood
                                                                     magnet school is not something extra. You need to figure out
location, it was also one of the lowest performing schools
                                                                     how to implement your theme so you’re teaching what you’re
in NHPS. After receiving MSAP funding in 2004, King/
                                                                     supposed to be teaching, but in the context of the theme.
Robinson built a new state-of-the-art building, brought in a
                                                                     And once they get the hang of it, they write units, and they
new principal, hired tutors to work with lower performing
                                                                     just go crazy. It’s just amazing where they started and where
students, and implemented the only K-8 International
                                                                     they are now. So, you can’t do this overnight. When you talk
Baccalaureate (IB) program model in Connecticut. “What I
                                                                     about fidelity, it’s got to be an ongoing process, year after
know about IB,” Canelli remarks, “is that it’s very rigorous;
                                                                     year. And that’s important.”
it’s very scripted. They stuck to it.” Following these changes,
King/Robinson made AYP in 2009 through the Safe Harbor

More Information
                          http://msapcenter com/resource aspx
For more about FOI, visit http://msapcenter .com/resource .aspx .

                                                                                                                      aprIl 2011 • page 7
Magnet C


                                                               CEMSE. (2008). Critical Components: Definitions and

Implementing a
                                                                  Explanations. Chicago: University of Chicago.
                                                               Century, J., Freeman, C., and Rudnick, M. (2008, March). A

High-Quality Magnet
                                                                   Framework for Measuring and Accumulating Knowledge About
                                                                   Fidelity of Implementation (FOI) of Science Instructional
                                                                   Materials. Paper presented at the annual meeting of the
Curriculum                                                         National Association for Research in Science Teaching.
                                                               Formica, S., and Maika, M.M. (2010, January). Measuring
The following steps can facilitate high-quality                    and Tracking Fidelity [PowerPoint slides]. Retrieved April
                                                                   4, 2011, from
instruction in the classroom:                                      FindingBalance2.pdf.
                                                               Horner, R.H., and Sugai, G. (2006, February). Policy Brief: Scaling

1 Develop specific criteria to ascertain which practices           up Effective Educational Innovations. Retrieved April 11, 2011,
adhere to or lack fidelity.                                    Johnson, E., Mellard, D.F., Fuchs, D., and McKnight, M.A.

2 Train instructional mentors and school coaches in                (2006). Responsiveness to Intervention (RTI): How to Do
                                                                   It. Lawrence, KS: National Research Center on Learning
processes that measure fidelity.
                                                               Mihalic, S. (n.d.). Implementation Fidelity. Unpublished
3 Assign trained, qualified staff to observe instructional        manuscript.
methods.                                                       U.S. Department of Education, Office of Innovation and
                                                                    Improvement, Innovations in Education: Creating Successful
4 Schedule regular classroom observations of specific,
                                                                    Magnet Schools Programs, Washington, D.C., 2004.
                                                               U.S. Department of Health and Human Services, Substance
magnet-related practices.
                                                                    Abuse and Mental Health Services Administration, Center
5  Provide more frequent observations for teachers with less
                                                                    for Substance Abuse Prevention. Finding the Balance: Program
                                                                    Fidelity and Adaptation in Substance Abuse Prevention:
magnet program experience and who request or need                   Executive Summary of a State-of-the-Art Review. Washington,
additional support.                                                 DC: Author. Retrieved April 4, 2011, from http://www.
6 Provide direct feedback to staff in
     individual conferences;
     written observation assessments;
     improvement plans; and
     the demonstration of lessons.

                                                                                     For technical
                                                                                      contact the
                                                                                     MSAP Center.
                                                                                   Call toll-free:

                                                                                                                   aprIl 2011 • page 8

Shared By: