Universal Screening and Progress Monitoring

Document Sample
Universal Screening and Progress Monitoring Powered By Docstoc
					Generalization of RtI
Procedures to Written
      Merilee McCurdy, Ph.D.
  University of Nebraska - Lincoln
      Description From Program
As schools begin to implement RtI procedures, it is
  important to realize that RtI procedures can be
  used with all academic areas, including writing
  and mathematics. This presentation will review
  writing assessment issues such as norming
  writing skills, universal screening procedures,
  and progress monitoring techniques. In addition,
  writing interventions that can be used within RtI
  frameworks will be identified for each writing
  skills area.
                Research in Reading
To date, much of the literature and research in RtI has been done in the
   area of reading.

 Research has shown that those children who become adults with low
  levels of literacy are at a disadvantage in a society that has high demands
  for effective literacy skills in the work place.
        Torgeson, J.K. (2000)

   There has not been a significant change in the reading performance of
    students from 1992 to 2005 (National Center for Educational Statistics,

   A 1997 study documented that at least 20% of the population in the
    United States had reading difficulties (Lyon & Moats, 1997).
               RtI and Reading
   Research has focused efforts on understanding
    reading deficits and identifying effective reading
    programs or interventions.

   This concentration of RtI procedures in the area of
    reading has been appropriate as the field seeks to
    understand and develop initial RtI procedures.

   Unfortunately, this intense focus has not been
    similar in other academic areas, especially for
    written language.
         The Significance of Writing
   Writing has become the neglected element of school
    reform (National Commission on Writing, 2004)

   The Commission asserts that students must “struggle
    with details, wrestle with facts, and reword raw
    information and dimly understood concepts into
    language they can communicate to someone else.”

   In short, students must write if they are to learn.
                   The National Commission on Writing. (2003). The
                   neglected “R”: The need for a writing revolution.
       The Significance of Writing
   The National Commission on Writing for America’s
    Families, Schools, and Colleges (2004) described writing
    as a threshold skill for employment and promotion and
    indicated that people who cannot write well are less
    likely to be hired, retained, and/or promoted.
         National Commission on Writing for America’s Families, Schools, and
         Colleges. (2004). Writing: A ticket to work…or a ticket out. NY: The
                                                                 College Board

   Writing skills are often needed for demonstrating
    learning (e.g., responding to exam items) and
    communicating across a variety of employment settings.
             The Significance of Writing:
Percentage of students, by writing achievement level, grades 4, 8, and 12: 1998
                                   and 2002

                                                               At or
                     Below     At         At          At      above    At or above
                     Basic    Basic   Proficient   Advanced   Basic    Proficient
  Grade 4

              1998     16       61       22           1         84         23

              2002     14      58        26           2        86          28
  Grade 8

              1998     16       58       25           1        84          27

              2002     15      54        29           2        85          31
  Grade 12

              1998     22       57       21           1         78         22

              2002     26      51        22           2        74          24
Nebraska Statewide Writing Assessment
              All Students
              2005 - 2006

                4      8     11
Nebraska Statewide Writing Assessment
    Students in Special Education
             2005 - 2006

              4     8      11
                  RtI and Writing
Schools appear to be having difficulty preparing
  students to meet society’s demands in the area of
The RtI process can assist with:
   Identifying students who are at-risk in the area of writing
   Providing or developing intensive interventions in the area of
   Monitoring weekly progress of student progress
   Using data to make decisions regarding students’
         needs following a predetermined length of
                 RtI and Writing
   In many ways, RtI procedures developed for the area of
    reading can be easily generalized to writing

   However, due to the lack of research on written
    language, many questions remain.
Questions regarding RtI and writing assessment:
1)   How much writing time is needed for a valid evaluation
     of writing?
2)   Is writing performance stable across assessments?
3)   How can story organization and story development be
     evaluated using CBA techniques?
4)   Which variables should be used for initial decision
5)   Which variables should be used for ongoing decision
6)   Are some assessment variables useful independently or
     in a combination with others?
7)   What type of writing prompts should be used?
Questions regarding RtI and writing interventions:
1)   How are Tier 1 services evaluated?
2)   What are empirically based interventions for writing?
3)   Can interventions be delivered in small groups?
4)   How much intervention time is required to observe
5)   Which variables are more reliable and valid for
     progress monitoring?
6)   What gains should be expected?
7)   How can students be motivated to write?
    RtI Writing Team: Lincoln Public
   What we have achieved so far….
     Established    an RtI Writing Team (2004)

     Collected   normative data
        Grades K-2 (2005-2006)
        Grades 3-5 (2006-2007)

     Initiated   small pilot studies
     RtI Writing Team: Lincoln Public
   Future Plans….
       2007-08
          Pilot K-5 writing procedures
          Small pilots at high school and middle schools to determine
           normative data collection procedures
          Evaluate research based interventions
          Train others to score writing probes (reliably)

       2008-2009
            Implement RTI writing district-wide at elementary schools
            Begin norming at middle schools
                  others will score if meet criteria for training

       2009-2010
            ?????
     Core Components for Quality RtI
   School-wide buy-in and implementation plan
   Team Leadership
   Integration of Services
   Implementation Infrastructure
   Parent Involvement
   Universal Screening and Assessment **
   Individual Progress Monitoring **
   Planned Service Delivery Decision Rules
   Scientifically Supported Instruction **
   Intervention Delivery
   SLD Verification
   Curriculum-Based Measurement (CBM)
     Content of the assessments are based on the
      instructional curriculum.
     Measures are presented in a standardized format.
     Material for assessment is controlled for difficulty by
      grade levels.
     Measures are generally brief.
               Shapiro (2004)

   Development of Norms
   Universal Screening
   Progress Monitoring
    General Assessment Procedures for
   The student is given a “story starter” in the form
    of a starting sentence or partial sentence
       Provides the student with an idea to write about

   The student is asked to think about their story
    for one minute and to write for three minutes

   At the end of the writing period, the examiner
    collects the student’s writing
  Differences between Reading and
            Writing CBM
Reading                   Writing

     Takes one minute         Takes four minutes

     Administered             Can be administered in
      individually              groups

     Must be scored           Permanent products
      concurrently with         allow for later scoring
          Other data to consider
   Classroom writing products

   Classroom writing assessments/rubrics

   Statewide writing assessments

   Reading performance

   Motivation!
          Development of Norms
   LPS Normative Procedures
     N=250       per grade; randomly selected by
     One story starter administered at fall, winter,
      spring (different starter each period)
     Same starter used with each grade (1st/2nd)
     Kindergarten
        Letter writing (fall and winter)
        Story starter in spring
     Used   a “scoring team” to increase reliability
                             TLW          TWW            CWS           CMIWS            %CWS

K – Winter                    X
K – Spring                    X             X              X              X                 X
1 – Fall                                    X              X              X                 X
1 – Winter                                  X              X              X                 X
1 – Spring                                  X              X              X                 X
2 – Fall                                    X              X              X                 X
2 – Winter                                  X              X              X                 X
2 – Spring                                  X              X              X                 X

TLW          Total Letters Written        TWW        Total Words Written
CWS          Correct Writing Sequences    CMIWS Correct minus Incorrect Writing Sequences
                                  %CWS   Percent Correct Writing Sequences
                Universal Screening
The practice of assessing all students to identify
  those who are not making academic or behavioral
  progress at expected rates.
      National Association of State Directors of Special Education, Inc. (2005)

A classroom-wide, school-wide, or district-wide assessment
  used to identify students who are at risk for academic
  failure or behavioral difficulties and could potentially
  benefit from specific instruction or intervention.

A critical prerequisite to providing early school-based
  prevention and intervention services for students at risk
  for or with academic, behavioral, or emotional difficulties.
                 Glover & Albers (2007)
             Universal Screening
“Assessments are in place that provide teachers with
  information about the progress of all students to
  determine which students need closer monitoring,
  additional intervention, and/or adjusted learning

   Norms are necessary to (1) identify students who need
    additional supports, (2) develop growth expectations,
    and (3) set entrance and exit criteria.

   Develop norming procedures based on school resources
                Universal Screening
   Scoring: It doesn’t take that long!
       Requires between 22 and 31 seconds to score TWW
       Requires between 25 and 37 seconds to score WSC
       Requires between 57 and 151 seconds to score CWS

     If  all three scores are used, it takes between
        1 ½ minutes and 2 ½ minutes to score one
        writing sample.

         Gansle, Noell, VanDerHeyden, Naquin, & Slider, 2002
         Malecki & Jewell, 2003
Difficulties regarding RtI Writing and
          Universal Screening
   Several components of writing to consider:
       Spelling
       Grammatical usage
       Mechanics of writing
   Numerous variables to consider!!

   It has not yet been determined which variables:
      are most sensitive to change over time
      are most valid and reliable for a given grade
                       Writing Variables
   Production-dependent                   Production independent
       Correctly Written Words                % Correct Writing Sequences
       Total Words Written                    % Correctly Spelled Words
       Correct Writing Sequences              Mean Length of Correct Writing
       Mature Words
       Letter Sequences Correct
       Total Words Spelled Correctly

A measure of both fluency and accuracy = Correct Minus Incorrect
Writing Sequences (CMIWS), an accurate-production measure
How to set growth expectations?
   Low and variable levels of growth on many variables.
       From AIMSweb norms:

TWW                        Fall to Winter              Winter to Spring
2nd grade                            +5                      +7
3rd grade                            +6                      +2

2nd grade                            +7                      +5
3rd grade                            +7                      +5
Data represent growth across entire normative sample
              Progress Monitoring
The practice of assessing students to determine if
  academic or behavioral interventions are producing
  desired effects.
       National Association of State Directors of Special Education, Inc. (2005)

Provides critical information about student progress that is
  used to ensure the use of effective educational practices
  and to verify that students are progressing at an
  adequate rate.

“Assessments are in place that provide teachers with
  information about which students are benefiting from
  intervention and which need additional supports.”
  Progress Monitoring Procedures
1. Based upon the norms you have decided to use and
   each student’s screening results, set a goal for each
      This goal should reflect an average gain per week as determined
       by the normative data.

2. Once the student’s intervention has begun, monitor the
   student’s progress at least once per week. For writing,
   may want to include more than 1 writing sample.

3. Graph the student’s scores on a chart.
    Progress Monitoring Procedures
4. Review the chart weekly to determine whether progress
   is being made.

5. After the student has been in an intervention for a
   specified amount of time:
       Look at the level and the rate of progress
       Determine whether the goal was attained
       Plan for next steps

   These are general RtI procedures and should be the
    same for regardless of academic area!!
                        The Complexity of Writing:
                          Sample Student Data
                            TWW                                                               CWS

        25                                                           10        Baseline
                                  Intervention                                                      Intervention
        20   Baseline                                                 8


        10                                                            4
         5                                                            2

         0                                                            0
             1      2   3    4        5      6      7    8                 1       2      3     4       5      6      7    8
                             Session                                                            Session

                            WSC                                                               CMIWS

                                          Intervention               3.5                                    Intervention
        10                                                                 Baseline
             Baseline                                                2.5

         6                                                           1.5
         4                                                             1
         2                                                           0.5
         0                                                             0
             1      2   3    4        5      6      7    8                 1        2     3     4       5      6      7    8
                             Session                                                            Session
Further Complexity of Writing



        -2   1   2   3     4     5   6   7   8
     Advantages of Writing Variables
   Although numerous variables add to the
    complexity of assessment, they aid in identifying
    specific difficulties.

   Awareness of specific difficulties further assists
    in developing/identifying interventions for the
    next phase of intervention (if necessary)
 Scientifically Supported Instruction

“Core instruction and interventions have
  been validated through scientific research”

Must demonstrate that districts have a
 process for identifying interventions with
 scientific evidence of effectiveness.
          Do Not Forget Tier 1
Tier 1 must be evaluated to examine effectiveness of
  curriculum for most students. This can be
  accomplished through universal screening.

“When teachers use a scientifically supported
  curriculum and appropriate instructional practices,
  they are preventing many problems from
               Nebraska Department of Education Technical
                         Assistance Document (2006)
            Interventions in RtI
   Basic Message:

Intervention delivery for writing is very similar to
  reading procedures.


        What makes a good Writer?
   Writing extends beyond mastering grammar, punctuation
    and the ability to identify parts of speech.

   Writing is best understood as a complex intellectual activity.
       It requires students to “stretch their minds, sharpen their analytical
        capabilities, and make accurate and valid distinctions.” (National
        Commission on Writing)

   Writing is a way for students to demonstrate what they
    know AND a way to help them understand what they know.
                Areas for Intervention
   Three primary writing stages:
       Prewriting
            Planning
            Story Development
            Idea Development

       Writing
            Story Organization
            Grammar
            Spelling
            Fluency

       Post Writing
            Editing/Revising
Challenges to improving students’
   Time for writing in and out of the classroom

   Assessment or measurement of progress and

   Integrating technology into the teaching and
    learning of writing

   Support for teaching and other classroom issues

             The National Commission on Writing. (2003). The Neglected
             “R”: The need for a writing revolution.
                              Characteristics of Strong Writing
                           Writing Strategies - involves teaching students strategies for planning,
                            revising, and editing their compositions
                           Summarization - involves explicitly and systematically teaching students how
                            to summarize texts.
Effect Size above .50

                           Collaborative Writing - uses instructional arrangements in which adolescents
                            work together to plan, draft, revise, and edit their compositions.
                           Specific Product Goals - assigns students specific, reachable goals for the
                            writing they are to complete.
                           Word Processing - uses computers and word processors as instructional
                            supports for writing assignments.
                           Sentence Combining - involves teaching students to construct more complex,
                            sophisticated sentences.
                           Prewriting - engages students in activities designed to help them generate or
                            organize ideas for their composition.
                           Inquiry Activities - engages students in analyzing immediate, concrete data to
                            help them develop ideas and content for a particular writing task.
                           Process Writing Approach - interweaves a number of writing instructional
                            activities in a workshop environment that stresses extended writing
                            opportunities, writing for authentic audiences, personalized instruction, and
                            cycles of writing.
    Tier 2 and 3 Instruction: Identifying
                Target Skills
   Generate hypotheses based on accuracy or
    fluency concern:

   Questions to ask when identifying target skill:
      Does the student have letter identification and letter
       formation skills?
      Can the student produce legible text?
      Did the student use the planning time?
      Does the student produce an adequate amount of
      Does the student produce a grammatically correct
      Does the student understand story organization?
      Could motivation be impacting performance?
     Classwide Fluency Strategies
   Classwide and individual performance
    feedback strategies are very successful at
    increasing student writing production.

   Students must practice writing to improve
    writing skills!
    Building Effective Interventions
   Writing Program composed of:
       Direct Instruction
       Goal Setting
       Writing Practice

       Feedback
            Individual Feedback
            Charting
       Rewards

    McCurdy, M., Skinner, C. H., Watson, T. S., & Shriver, M. D. (accepted).
      Examining the effects of group contingencies on the writing
      performance of middle school students with learning disabilities in
      writing. School Psychology Quarterly.
    Evaluating Published Programs
   Examine for:
     Technical Adequacy
     Research Support
     Grade appropriateness/match to need
     Relationship to 6 Traits of Writing
     Price
Grade Name         Description (from website)              Writing Trait   Other
K12   Teaching     From the basic sentence to the formal   Ideas           Publisher:
345   Competence   essay, Teaching Competence in                           Pro-ed Inc.
      in           Written Language is the program to      Organization
      Written      choose to teach writing at any level.                   www.proedinc
      Language-    This program works because it           Voice           .com
      Second       teaches both basic and advanced
      Edition      writing skills in an easy to learn,     Word Choice     Price: $62.00
                   step-by-step format. Each lesson also
                   includes social (pragmatic) uses of     Convention
                   writing and proofreading and error
                   This program is different from other
                   programs in the field because it can
                   be used either:
                   As a step-by-step, structured
                   hierarchy of writing skills, from the
                   most basic to advanced, or
                   As a resource to plug into any level
                   where an individual or group needs
                   focused work. From the original 44
                   lessons, the program has been
                   modified and expanded to a total of
                   80 lessons for individuals or groups
                   at different levels of ability.
            Intervention Resources
   Strategic Instruction Model (SIM)
       Center for Research on Learning

   National Center on Accelerating Student Learning

   Mnemonics for Planning and Editing
       Cognitive Strategy Instruction; Dr. Bob Reid – UNL
        Intervention Delivery: Resources!
   Interventions can be delivered in small groups
     Behavior management procedures will be necessary
     Motivation and engagement is a concern

Consider and evaluate “comprehensive” reading
       These programs may also impact writing performance
       May serve as the first supplementary intervention
       Must monitor for improvement or lack of
    LPS Goals of Pilot Year (2007-2008)
   Identify any modifications to district RtI procedures
   Identify appropriate assessment variables for progress
    monitoring and decision making
   Incorporate district assessment data into decision
    making structure
   Establish structured decision making criteria for
    measuring adequate progress or referral to special
   Locally validate multiple interventions for use with
    writing concerns
   Prepare for the future
        Norming at upper levels (3, 5, or 10 minute writing times?)
        Procedures at upper levels
        Interventions for writing skills in the upper grades
Daly, E., Glover, T., McCurdy, M. (2006). Response to intervention: Technical
   assistance document. Lincoln, Nebraska: Nebraska Department of

Gansle, K. A., Noell, G. H., VanDerHeyden, A. M., Naquin, G. M., & Slider, N.
  J. (2002). Moving beyond total words written: The reliability, criterion
  validity, and time cost of alternative measures for curriculum-based
  measurement in writing. School Psychology Review, 31, 477- 497.

Glover, T. A., & Albers, C. A. (2007). Considerations for evaluating universal
   screening assessments. Journal of School Psychology, 45, 117-135.

Graham, S., & Perin, D. (2007). Writing next: Effective strategies to improve
   writing of adolescents in middle and high schools - A report to Carnegie
   Corporation of New York. Washington, DC: Alliance for Excellent Education.

Lyon, G.R. & Moats, L.C. (1997). Critical conceptual and methodological
   considerations in reading intervention research. Journal of Learning
   Disabilities, 6, 578-588.
Malecki, C. K., & Jewell, J. (2003). Developmental, gender, and practical
  considerations in scoring curriculum-based measurement writing probes.
  Psychology in the Schools, 40, 379-390.

McCurdy, M., Skinner, C. H., Watson, T. S., & Shriver, M. D. (accepted).
  Examining the effects of group contingencies on the writing performance of
  middle school students with learning disabilities in writing. School
   Psychology Quarterly.

National Association of State Directors of Special Education, Inc. (2005).
  Response to intervention: Policy considerations and implementation. NASDE.

National Center for Educational Statistics. (2003). NAEP 2002 writing report
  card for the nation and the states (Report NCES 1999-462). Washington,
  DC: Office of Educational Research and Improvement.

National Commission on Writing for America’s Families, Schools, and Colleges.
  (2004). Writing: A ticket to work…or a ticket out. NY: The College Entrance
  Examination Board
National Commission on Writing in America’s Schools and Colleges.
  (2003). The neglected “R:” The need for a writing revolution. NY:
  The College Entrance Examination Board.

Shapiro, E. S. (2004). Academic skills problems: Direct assessment and
  intervention (3rd ed.). NY: The Guilford Press.

Torgesen, J.K. (2000). Individual differences in response to early
   interventions in reading: The lingering problem of treatment
   resisters. Learning Disabilities Research and Practice, 15, 55-64.
     Contact Information

         Merilee McCurdy, Ph.D.
Assistant Professor of School Psychology
          Licensed Psychologist

    University of Nebraska - Lincoln
      234 Teachers College Hall
        Lincoln, NE 68588-0345