Docstoc

Protocol on Recording Training Evaluation Records

Document Sample
Protocol on Recording Training Evaluation Records Powered By Docstoc
					S3TAIR Project Effective Practice Validation Site Visit Protocol                                                                   Page 1 of 17

                                      S3TAIR Project Effective Practice Validation Site Visit Protocol


                                                                   Directions to Site Visitors:

       Information already entered below was obtained through an online survey and follow up phone interview. Please complete the portions
        relevant to your visit.

       Each visitor will individually complete a site visit protocol recording his or her impressions. Please provide a score for each area;
        scoring options are presented as a continuum across the elements of the rubric. Points are to be totaled and entered as a Total Score
        at the bottom of the first page. A total of 100 points is possible for all areas together. Note that the first protocol category is answered
        as a narrative response to interview questions; that category and the final category (Other Considerations) are not scored.

       Comments should support or explain the score and provide information that will assist in consensus building among the site visitors.

       The “ Look Fors” suggest but do not limit what to observe. Use the last category to capture and provide a rationale for other
        observations. Visitors are requested to fully describe any other factors taken into consideration in rating the nominated practice.

       Each team member completes one form.

       Each site visitor should complete one form. Upon completion of the visit, site visitors should come to consensus, with consensus
        statements and recommendation recorded on the sheet marked “Visit Summary”. All visitors’ forms and the summary will be submitted
        to VESID upon completion.




                S3TAIR (Supporting Successful Strategies To Achieve Improved Results) is a New York State Education Department project
                                          supported in part by an IDEA State Personnel Development Grant
S3TAIR Project Effective Practice Validation Site Visit Protocol                                             Page 2 of 17



                                                                   District Information

District/building:                                                     Practice:

Visit date(s):                                                         Visitor:
Persons interviewed:                          Name                                 Position/Title   Grade/Location




Staff observed:




Other information
sources accessed by
visitor:
S3TAIR Project Effective Practice Validation Site Visit Protocol                                                         Page 3 of 17


                                                                                                              Possible         Awarded
                                         Individual Visitor Point Summary


Exploring the School’s Process for Selection of the Nominated Practice - No Point Assignment

Data Collection for Analysis to Inform Practice (10 points)                                              10
Use of Data to Inform Decisions (10 Points)
                                                                                                         10
Fidelity of Implementation (10 Points)
                                                                                                         10
Context of Practice (10 Points)
                                                                                                         10
Evidence of Impact (20 Points)
                                                                                                         20
Evidence of Systemic Support for the Practice (15 Points)
                                                                                                         15
Generalization of Practice within Building or District (10 Points)
                                                                                                         10
Evidence of Sustainability (15 Points)
                                                                                                         15
Comment on impressions:



                                                                            Validation Recommendation:
Total Score Awarded by Visitor:
                                                                                        Yes                   No
S3TAIR Project Effective Practice Validation Site Visit Protocol                                                              Page 4 of 17



                  Exploring the School’s Process for Selection of the Nominated Practice - No Point Assignment
Questions to those responsible for initial implementation: Please tell us why you chose this particular practice.

Probe:    Was the practice recommended to you by someone, and if so, who?
Probe:    Did you explore education literature to determine the evidence supporting this practice before you adopted it? If so, can you give
          us some examples of studies you explored?
Probe:    Did you visit schools that had implemented the practice successfully? If so, did the school share data to document the
          effectiveness of the practice?
Probe:    Is there anything else you would like to tell us about your decision and the evidence that supported it?

Directions to Visitors: Document the answers to this question and probes in this space.
S3TAIR Project Effective Practice Validation Site Visit Protocol                                                           Page 5 of 17


                       Data COLLECTION for Analysis to Inform Practice (10 points)       - The district/building has:
o   Developed and implemented rigorous           o Developed and                o        Evidence of a data           o   No developed
    practice for data collection, including a      implemented practice for              collection process but no        practice for data
    centralized data manager or team.              data collection, including a          clear centralized data           collection related
o   Collected data collection from multiple        centralized data manger or            collection manager or            to the nominated
    sources, including but not limited to State,   team.                                 team.                            practice.
    commercially available, internal/external    o Collected data from two      o        Collected data from a
    formative and summative assessments,           sources.                              single data source.
    diagnostics and progress monitoring.         o Provided minimal training    o        Not provided training
o   Provided systematic and consistent training    on how to collect and                 related to data collection
    on how to collect and organize data.           organize data.                        and or organization.
o   Conducted periodic review of data collection o Periodic review to ensure    o        Periodic review to ensure
    process to ensure accuracy and                 accuracy and consistency              accuracy and consistency
    consistency.                                   of data collection not                of data collection not
                                                   conducted.                            conducted.

                                                                       Points
   10                9              8            7             6   5            4          3            2          1            0
Look for:
o   Centralized Data manager                                           o   Evidence that collected data are organized and reported or made
o   Evidence (documentation) of systematic and comprehensive               available to all stakeholders in a comprehensible format
    collection and reporting of data:                                  o   Evidence that PD on data collection processes is provided to all
     Pre-assessment (baseline), interim, and summative data               associated with data collection
     Evidence of and rationale for multiple sources of                o   Evidence of support for classroom-based data collection for
        measurement of student outcomes built into data sets for           progress monitoring and/or action research
        analysis                                                       o   Evidence of regular evaluation of data collection procedures and
     Data collection schedule or calendar                                 practices with adjustment for improvements as needed
     Charts and other graphic representations of data                 o   Adjustment to instruments, database, reporting procedures
     Evidence of data collection longitudinally                       o   Evidence of a practice of periodic reporting on collected data to
     Evidence of data collection for student cohorts                      stakeholders
S3TAIR Project Effective Practice Validation Site Visit Protocol                                         Page 6 of 17



                                           Data Collection for Analysis to Inform Practice (continued)
Total Points Assigned:              out of 10
Rationale for Scoring:
S3TAIR Project Effective Practice Validation Site Visit Protocol                                                                            Page 7 of 17


                                   ANALYSIS of Data to Inform Decisions (10 Points ) – The district/building:

o Provides on-going staff development of             o       Provides initial staff development of   o Provides little or no staff          Does not have a
  skills for use of data analysis in decision-               skills for use of data analysis in        development of skills for use        consistent practice
  making                                                     decision but has provided for             of data analysis in decision-        for using data to
o Makes data driven decision a priority by                   ongoing PD and TA                         making                               make decisions
  providing time and support                         o       Has not scheduled specific times        o Provides no scheduled or
o Analyzes data at the building & classroom                  for data analysis to inform               supported opportunity for
  level including individual and                             instruction                               staff to collaborate regarding
  disaggregated group performance on                 o       Analyzes data only at the building        data use and decision-
  group assessments and day to day                           level to disaggregate group               making
  classroom performance data                                 performance on group assessments        o Uses little or no
o Ensures building wide and individual use           o       Makes some data-based                     disaggregation of data of
  of analyzed data to make decisions about                   instructional decisions at                individual or group
  instruction at classroom and system level                  classroom/system level                    performance on
o Ensures that administrators and                    o       Administrators consider summative         assessments
  instructional staff collaboratively consider               data to make decisions about need       o Does not use data to make
  summative data annually for decisions                      for programmatic changes but do           decisions regarding school
  about programmatic changes                                 not engage other staff                    wide instructional practices
                                                                                                       or programmatic changes
                                                                              Points
       10                 9              8               7             6            5            4            3          2              1                  0
Look for:
o Database/reporting is comprehensive and accessible to all                        o    Evidence of data analysis and resulting decisions that led to
   stakeholders                                                                         changes in practice and improved student outcomes
o Acceptable turn around time for internal formative assessments &                 o    Written description of process used in decision-making (e.g.,
   progress monitoring                                                                  specific data sources analyzed, strategies for identifying
o PD (internal/external/district wide) on data analysis and decision-                   hypotheses and interventions from data, structures to support
   making - for example:                                                                implementation, analysis of impact, and adjustment to
      Understanding what data represent                                                interventions as needed)
      Using data to differentiate or form intervention groups                     o    Evidence that analysis of data has resulted in changes in core
      Strategies for progress monitoring.                                              program or supplements, addition of service providers (TA’s,
o Evidence of scheduled times dedicated to data analysis for                            Literacy Specialist, AIS), etc.
   decision-making by instructional team, grade level, content area,               o    Evidence of monitoring of decision-making process, with
   etc.                                                                                 identification of areas in need of improvement and provision of
                                                                                        PD/TA in the areas of need
S3TAIR Project Effective Practice Validation Site Visit Protocol                                      Page 8 of 17




                                                   Analysis of Data to Inform Decisions (continued)
Total Points Assigned:              out of 10
Rationale for Scoring:
S3TAIR Project Effective Practice Validation Site Visit Protocol                                                                  Page 9 of 17



                                         Fidelity of Implementation (10 Points)      - The district/building:
o   Has established a protocol for         o Has a protocol for monitoring       o    Has an informal practice of    o   Does not monitor
    monitoring implementation of the           implementation of the practice         monitoring implementation          implementation to
    practice to ensure fidelity of             to determine fidelity to the           when reports of drift from         determine fidelity of
    implementation                             model                                  the model are received             implementation
o   Has a defined strategy for             o Does not have documentation o            Has an informal practice for
    professional development/technical         of functional use of protocol          provision of technical
    assistance for correcting drift from   o Does not have a defined or               assistance/ professional
    fidelity                                   consistent technical                   development to correct drift
o   Has documentation of fidelity              assistance/ professional               from fidelity
    monitoring and adjustment protocol         development strategy for
    in action                                  correcting drift when fidelity is
                                               not found


                                                                     Points
   10               9            8            7          6          5            4           3             2          1             0
Look for:
o Evidence that administrators/instructional staff can accurately describe the critical elements of the practice as validated through research,
   and discuss how they should be and are implemented in the school
o Documented program goals and steps in implementation of the practice (i.e. manual describing implementation, classroom charts of
   practice steps, checklist of steps in implementation)
o Documented structures for checking fidelity of implementation (e.g., written guidelines of what to look for, how often to look, how to
   evaluate what is observed to determine fidelity of implementation)
o Evidence/documentation of validation procedures used consistently across all implementers using program guidelines
o Evidence results of fidelity observation are used to monitor and adjust practice implementation to recover from drift away from fidelity
Total Points Assigned:            out of 10
 Rationale for Scoring:
S3TAIR Project Effective Practice Validation Site Visit Protocol                                                                            Page 10 of 17


                                                         Context of Practice (10 Points) - There is evidence that:
The practice is delivered to          The practice is delivered to students with         The practice is delivered solely to      The practice is delivered solely to
students with disabilities in a       disabilities in a general education setting with   students with a disability but in a      students with disabilities in a
general education setting with        general education peers some or much of the        setting providing opportunities for      segregated setting
general education peers most or at    time.                                              students with disabilities to practice
all times.                                                                               skills with typical peers (e.g., co-
                                                                                         located in a general education
                                                                                         school)
                                                                              Points
10            9          8            7            6               5          4             3             2              1                  0
 Look fors represent some of the elements likely to be in place in each of the three described types of contexts. Most practice settings will not fall cleanly into
one or another of these contexts. Note that the rubric applies to the nominated practice only.
Students with disabilities impacted Students with disabilities impacted by the             Students with disabilities impacted    Students with disabilities
by the practice:                     practice:                                             by the practice:                       impacted by the practice:
o Share all academic and             o Receive most of their instruction in general       o Receive instruction or                 o Receive all instruction from
    special area classes and non-       education and share all special classes and           implementation of the practice in      special education teachers in
    academic activities with their      other activities with their typical peers             a special class setting or in a        special classes in settings
    typical peers                    o May be on separate behavior management                 reverse mainstreami setting            with little opportunity for
o Are addressed by classroom            programs rather than participating fully in       o Have opportunities to practice           interaction with typical peers
    and building behavior               class school-wide behavior management or              the skills learned through the       o May not share special classes
    management or group                 group contingency plans                               practice with their typical peers      and some other activities with
    contingency plans as their       o Have equal opportunities to participate in             in typical settings                    typical peers
    typical peers are                   activities in school (clubs, prep rallies) and    o Have opportunities to participate      o Have little contact with
o Have equal opportunities to           outside school (field trips, community                in school activities with some         general education teachers
    participate in activities in        service, sport teams).                                accommodations or restrictions         except for special classes and
    school (clubs, prep rallies) and Teachers involved in implementing the practice: Teachers involved in                            some school activities
    outside school (field trips,     o Reserve most responsibility for typical             implementation of the practice:           (assemblies, etc.)
    community service, sport             student’s learning and parent                     o Are primarily special education      Teachers involved in
    teams)                               communication for the general education                teachers                          implementation of the practice:
Teachers involved in                     teacher                                           o Conduct parallel instructional       o Are special education
implementation of the practice:      o Reserve most responsibility for students                 planning but typically do not co-    teachers
o Share responsibility between           with disabilities’ learning, implementation of         teach or collaborate on           o Have sole responsibility for
    general education and special        FBA/BIPs, and parent communication for                 evaluation of student progress       student’s BIPs, and
    education teachers for all           the special ed teacher                            o Tend not to share                       communicating with parents
    students, co-plan, share         o Reserve primary lesson planning and                      communication with parents           of students with disabilities
    responsibilities for teaching        instruction to general education teacher          o Rarely share responsibility for      o Have sole responsibility for
    and implementation of FBAs           and instructional accommodation and                    implementation of FBA/BIP,           adapting general education
    and BIPs and collaborate on          intensive intervention for students with               and maintain separate behavior       curricula often without input
    communication with parents           disabilities to special education teacher              or group contingency plans for       from general education
                                                                                                students with disabilities           teachers
S3TAIR Project Effective Practice Validation Site Visit Protocol                            Page 11 of 17



                                                          Context of Practice (continued)
Total Points Assigned:              out of 10
Rationale for Scoring:
S3TAIR Project Effective Practice Validation Site Visit Protocol                                                            Page 12 of 17


                       Evidence of Impact (20 Points) - The district/building’s data for students with disabilities show:
o Evidence from norm referenced and/or curriculum- o Evidence from norm referenced and/or                       o Little or no improvement in
   based measures that indicate students with                 curriculum-based measures that indicate             student outcomes
   disabilities are gaining skills at an accelerated          students with disabilities are gaining skills at  o Minimal success in attaining
   pace that will help close the gap with typical peers       or near a typical pace (e.g., a year for a          appropriate and relevant IEP
o Progress toward IEP goals is measured regularly             year)                                               goals
   and shows successful attainment of relevant and         o Progress toward appropriate and relevant           o No organized
   appropriate IEP goals                                      IEP goals is measured regularly and shows           documentation to provide
o Work samples that are evaluated to document                 that goals are achieved a majority of the time      evidence of students with
   growth in skill and adjust interventions                o Work samples are maintained but not                  disabilities’ achievement or
o Clear documentation of results from progress                analyzed in a structured way                        progression
   monitoring and intervention adjustment to achieve       o Documentation of progress monitoring
   successful outcome                                         demonstrates recognition of differing
o Soft data that support the effectiveness of the             response to interventions but is not complete
   practice, such as student and parent reports of            in documenting process of modifications to
   success in using skills or behaviors in out of             interventions
   school settings                                         o The school has not collected or has not kept
                                                              records of soft data on the effectiveness of
                                                              the practice
                                                                     Points
20     19       18     17     16       15    14      13  12     11     10      9      8     7     6       5    4     3       2      1        0
Look for:
o Evidence from statewide assessment of success in meeting or making progress toward SPP targets/AYP
o Interim measures (intermediate assessment data both summative and formative) reflecting positive trajectory of improvement
o Curriculum-based measurements/progress monitoring data documenting successful outcomes/trajectory of improvement
o Comparative student work samples demonstrating proficiency or progression in skill/content area
o Impact of practice on student referral rates to special education

Total Points Assigned:              out of 20
Rationale for Scoring:
S3TAIR Project Effective Practice Validation Site Visit Protocol                                                                      Page 13 of 17

                                 Evidence of Systemic Support for the Practice (15 points): The district/building:


o   Has developed a constituency for the practice             o    Provides on-going PD for         o   Presents evidence of      o Has not explicitly
    among families and community by actively                       veteran staff embedded within        on-going PD for             considered
    advertising and communicating about the                        the system and may include:          veteran staff               strategies to ensure
    practice and its impacts and by providing                      technology, text resources,      o   Has a practice in place     systemic support for
    training to ensure families can support practice               conference attendance                to ensure that new          the practice
    as partners in their children’s education                 o    Has a practice in place to           hires are brought up to
o   Presents evidence that on-going professional                   ensure that new hires are            speed
    development is provided to all veteran staff                   brought up to speed
    involved in project to consistently up-grade skill        o    Includes the practice in
    set.                                                           resource allocation decisions at
o   Has a practice in place for new hire PD to                     the building and district level
    ensure continuity
o   Includes the practice in resource allocation
    decisions at the building and district level
                                                                            Points
  15        14         13        12       11     10        9          8                  7       5        4       3        2      1          0
Look for:
Resource Allocation:                                                                         Family/Community support:
 o project budgets                           On-going PD:                                    o handbooks & home training modules & materials
 o BOE notes                                 o sign-in/evaluations of PD
                                                                                             o parent teacher conference notes
 o PDP or Strategic plan                     o conference day agendas
                                                                                             o PD Plans that involve families/community members
New hire PD:                                 o PD materials
                                                                                             o project meeting notes reflecting actions
 o Orientation/training materials/modules    o conferences attended
                                                                                             o newsletter articles
 o pre/post assessments
Total Points Assigned:              out of 15
Rationale for Scoring:
S3TAIR Project Effective Practice Validation Site Visit Protocol                                                            Page 14 of 17


                             Generalization of Practice within Building or District (10 points) - The district has:
o   Implemented and is monitoring      o Specific plans, resources and o Initiated discussion about         o The district or building does not
    the elements of the plan to           a timeline for generalizing the   generalizing the practice but      anticipate attempting to
    generalize the practice to more       practice (e.g., replicate it in   has not yet put effort or          generalize the practice.
    than one setting (within a            other classrooms, grades, or      resources into supporting
    school, across                        buildings).                       generalization.
    classes/grades/buildings and
    or within the district.
o   Documentation of the effective
    generalization of the practice

                                                                       Point
       10                  9             8            7            6       5         4          3            2          1            0
Look For:

    o   Documentation of discussions about generalization, including determining necessary resources to support generalization
    o   Specific documentation of strategies for generalization of the practice
    o   Documentation of areas/settings that have adopted program (classrooms, grade levels, school/district wide.)
    o   Documentation of partnership with another institution (i.e. university/school partnership) in support of generalization


Total Points Assigned:              out of 10
Rationale for Scoring:
S3TAIR Project Effective Practice Validation Site Visit Protocol                                                                 Page 15 of 17


                                       Evidence of Sustainability of Practice (15 points ) – The practice:
 o  Has 4+ years of historic evidence of             o Has 3-4 years of                o Has been in place 2-3               o Has been in place
    implementation                                       implementation                     years                              a year or less
 o Has survived changes in key personnel             o Has survived minimal            o May not have experienced
    (e.g., Superintendent, Principal, initiating         changes in key personnel           significant turnover with
    teachers) and/or practice is proactively         o Is referenced within district        key personnel with minimal
    reflected within system planning documents           planning documents                 evidence of practice
    (mission/vision statements, strategic plan)                                             embedded permanently
 o Is integrated with other established practices                                           within the system
    within district planning documents                                                 o Does not appear in district
 o Has a contingency plan for significant threat                                            planning documents
    to the practice (fiscal issues that might affect
    allocation of resources to support the
    practice, the inevitable eventual departure of
    staff who have been deeply committed to the
    development and implementation of the
    practice)
                                                                          Points
 15         14       13       12        11       10     9        8      7        6         5         4        3       2          1        0
 Look for:

 o   Archival records of the practice over time including records of implementation, monitoring for fidelity, and references to funding
 o   Longitudinal data reflecting the impact of the practice
 o   District or school planning documents recording support for the practice (PDP, Strategic Plan...)
 o   Interviews with key administrators and educators about the history of the practice
 o   Evidence of contingency planning relative to loss of key personnel, potential reduction in available resources, etc.

 Total Points Assigned:              out of 15
 Rationale for Scoring:
S3TAIR Project Effective Practice Validation Site Visit Protocol                                                Page 16 of 17

                                                            Other Considerations
                             Please Provide Additional Information Not Represented Above (No Points Assigned)

 General comment:




                                 “Look Fors”                                              Comments/Evidence
S3TAIR Project Effective Practice Validation Site Visit Protocol                                                   Page 17 of 17

                                                                    Site Visit Summary




 District/Building                                       Practice                        Visit Date(s):
                                      District-wide

 Practice is:
 (Check one)
                                      Grade or Level Specific (describe)



                                      Specific to One or More Buildings (describe)




                                      Specific to One or More Departments (describe)



 Total Score (average of all visitor’s totals):          Did the Team Reach Consensus?

                                                         No                                        Validation Recommendation:

                                                         Yes                                       Yes_____   No_____

				
DOCUMENT INFO
Description: Protocol on Recording Training Evaluation Records document sample