Decision Making with Progress Monitoring Data by rwi74592

VIEWS: 18 PAGES: 12

									  Decision Making with
Progress Monitoring Data:                                                      Our Working RTI Model
                                                                             RTI relies on a multi-tier prevention system to
   Considerations in                                                         identify students with academic skill weaknesses
                                                                             and LDs:
Determining Instructional                                                     – Primary prevention level (Tier 1)

     Effectiveness
                                                                              – Secondary prevention level (Tier 2)
                                                                              – Tertiary prevention level (Tier 3)
                                                                             Each tier in the model is focused on a set of
                                                                             interrelated tasks:
            John M. Hintze, Ph.D.                                             – Instructional Tasks
          University of Massachusetts                                         – Assessment Tasks
National Center for Student Progress Monitoring                               – Administrative/Managerial Tasks
                                                                             Responsiveness is determined via a dual
                hintze@educ.umass.edu                                        discrepancy analysis
                                                                              – Student performance relative to benchmark standards
                www.studentprogress.org
                                                                              – Student performance relative to rate-of-improvement or
                                                                                growth standards




Continuum of Schoolwide
Support                                                                     Basics of RTI
                                                  Tertiary Prevention:       Primary Prevention (Tier 1):
                                                  Further intensified and
                                                      individualized
                                                       intervention
                                                                              – All student receive a research supported core
                                    ~5%
                                                                                curriculum
                                    ~15%
                                                       Secondary
                                                      Prevention:
                                                                              – All students screened seasonally (Fall, Winter,
                                                  Intensified, validated
                                                  I t    ifi d   lid t d
                                                       intervention
                                                                                & Spring) to determine which students are
      Primary Prevention:                                                       suspected to be at risk.
    Schoolwide and classwide
           instruction                                                        – Students suspected to be at risk remain in
                                                                                primary prevention and their progress is
                                                                                monitored for 4 to 8 weeks.
                                                                              – Progress monitoring in Tier 1 is used too:
                               ~80% of students                                  • Disconfirm risk. These responsive students remain
                                                                                   in primary prevention.
                                                                                 • Confirm risk. These unresponsive students move to
                                                                                   secondary prevention.




                                                                                                                                         1
Basics of RTI                                                 Basics of RTI
 Secondary Prevention (Tier 2):                                 Tertiary Prevention (Tier 3):
  – Supplemental manualized intervention is                      – Intensified individualized intervention (may or
    provided in flexible groups for anywhere from                  may not involve special education services)
    10-15 weeks                                                  – Student progress is monitored weekly
  – Student progress is monitored weekly                         – Progress monitoring is used to:
                                                                    • Set Individualized education program (IEP) goals.
  – Student responsiveness is assessed                              • Design individualized instructional programs.
    continually :                                                   • Monitor student response.
       • Students who are responsive to Tier 2 intervention            – When progress monitoring indicates that a
         return to Tier 1                                                 student has met benchmark performance
       • Students who are under-responsive to Tier 2                      standards, student exits Tier 3 (or special
         intervention move to tertiary Tier 3 intervention                education), and returns to primary (Tier 1) or
                                                                          secondary (Tier 2) prevention, with ongoing
                                                                          progress monitoring.




Three Tiers of RTI                                                                Summary
  TIER 1: Primary Prevention                                  1. Screen all students to identify suspected
  - General education setting
  - Research-supported instruction
  - Screening to identify students
                                                                 at-risk students.
  suspected to be at risk
  - PM to (dis)confirm risk status                            2. Monitor progress of students suspected
                     AT RISK                                                                   risk
                                                                 to be at risk to (dis)confirm risk.
           TIER 2: Secondary
           Prevention
                                              RESPONSIVE
                                                              3. Provide supplemental standardized
           - Supplemental researched-
           based intervention
           - PM to assess responsiveness
                                              UNRESPONSIVE
                                                                 intervention to at-risk students, while
                                                                 progress is monitored to assess
           TIER 3: Tertiary Prevention
           - Intensified/Individualized
           intervention (special education)
                                              RESPONSIVE         response.
           - PM to set IEP goals
           - PM to formulate individualized   UNRESPONSIVE
           programs
           - PM to assess responsiveness




                                                                                                                           2
                                                So, RTI Is Embedded Within a
                                                Multi-Tier Prevention System:
               Summary                          Analogy to Health Care
                                                 High blood pressure (HBP) can lead to heart attacks or strokes
4. Students who are under-responsive to          (like academic failure can produce serious long-term negative
   secondary preventative intervention are       consequences).
                                                 At the annual check-up (primary prevention), HBP screening (like
   moved to tertiary intervention where          annual fall screening for low reading or math scores).
   intensified/individualized intervention is    If screening suggests HBP, then monitoring over 6-8 weeks
                                                 occurs to verify HBP (like PM to ([dis]confirm risk).
         id d (may or may not b under th
   provided (                 t be d the         If HBP is verified, second prevention occurs with relatively
   auspices of special education).               inexpensive diuretics, which are effective for vast majority, and
                                                 monitoring continues (like small-group Tier 2 intervention, using a
                                                 supplemental standard treatment protocol, with PM to index
5. Progress is formatively monitored to          response).
   tertiary intervention to determine            For patients who fail to respond to secondary prevention
                                                 (diuretics), then tertiary prevention occurs—experimentation with
   effective programs and IEP goals are set      more expensive medications (e.g., ACE inhibitors, beta blockers),
   (for students in classified for special       with ongoing monitoring, to determine which drug or combination
                                                 of drugs is effective (like individualized instructional programs
   educaiton), and determine exit                inductively formulate with progress monitoring).
   decisions.




                                                Tier 1—Primary Prevention
                                                 All students screened using CBM
   Setting Goals &                               Students scoring below benchmark are
  Making Decisions                               suspected at risk for reading or math

within the Three Tiers
                                                 difficulties
                                                 Suspected at-risk students monitored for 4
        of RTI                                   to 8 weeks during primary prevention
                                                 using CBM and their risk-status is
                                                 confirmed or disconfirmed




                                                                                                                       3
                                                                                                       Tier 1—Primary Prevention:
    Tier 1—Primary Prevention:                                                                         Confirming Risk Status With
Screening for Possible Reading Risk                                                                        Progress Monitoring
                                                                                                       At the end of 4–8 weeks, student risk
                                                                                                       status is confirmed or disconfirmed via
                               R-CBM                                                                   progress monitoring




  Tier 1—Primary Prevention:                                                                             Tier 1—Primary Prevention:
Confirming Risk Status With PM                                                                         Screening for Possible Math Risk


                                                                                                        M-CBM
                              25
Digits Correct in 3 Minutes




                              20       Anne’s Slope = 1.00

                              15
             t




                              10
                                                                                                                               X
                              5
                                                                                                                         Keep an eye on
                              0
                                                                                                                        Anne to see if she
                                   1   2   3   4   5     6    7   8    9      10   11   12   13   14
                                                                                                                          “catches up”
                                                       Weeks of Instruction




                                                                                                                                                 4
  Tier 1—Primary Prevention:                                                                              Tier 1—Primary Prevention:
Confirming Risk Status With PM                                                                          Screening for Possible Math Risk


                                                                                                        M-CBM
                             25
 Digits Corre in 2 Minutes




                             20
                                          Arthur’s slope= 0.0
                             15
            ect




                             10

                             5

                             0
                                  1   2     3   4   5     6    7   8    9      10   11   12   13   14
                                                                                                                          Arthur is not responding
                                                        Weeks of Instruction
                                                                                                                          to the core curriculum and
                                                                                                                          should move to Tier 2




Tier 1—Primary Prevention:                                                                                     Tier 2—Secondary
         Review                                                                                                    Prevention
 All classroom students screened to                                                                     Use the same goal setting and decision making
 identify suspected at-risk students.                                                                   standards as in Tier 1
                                                                                                        In addition to the core curriculum, students in Tier 2
 Suspected at-risk students remain in                                                                   receive supplemental manualized intervention for
 primary prevention and are monitored                                                                   10 to 15 weeks
 using CBM for 4–8 weeks:                                                                               At the end of Tier 2 intervention, student benchmark
     – Students with adequate slopes remain in                                                          and growth status as evaluated
       primary prevention.                                                                              – Students at or above benchmark return to Tier 1
     – Students with inadequate slopes move to                                                          – Students below benchmark, but making adequate (or
       Tier 2 (secondary prevention).                                                                     exceeding) growth progress may be maintained in Tier 2
                                                                                                        – Students below benchmark and continuing to demonstrate
                                                                                                          poor growth progress (i.e., under-responding) are moved
                                                                                                          to Tier 3




                                                                                                                                                                    5
                                                          Tier 3—Tertiary Prevention:
Tier 3—Tertiary Prevention
                                                                 Goal Setting
 Intensified/individualized programs are                   Three options for setting IEP goals:
 formulated for individual or small groups                  – Aggregated benchmark estimates
 (2-3) of students                                          – Aggregated rate of improvement (growth)
                                                              estimates
                                 (i.e.,
 For students who are classified (i e
 receiving special educational services)                    – Intra-individual framework
 IEP goals are established.




Tier 3—PM in Tertiary Prevention:                         Tier 3—PM in Tertiary Prevention:
 Setting Goals With End-of-Year                            Setting Goals With End-of-Year
         Benchmarking
                                                                   Benchmarking
 Setting IEP goals
  – End-of-year benchmarking                                M-CBM
     • Identify appropriate grade-level benchmark
     • Mark benchmark on student graph with an X
     • Draw goal-line from the baseline CBM scores to X




                                                                                                        6
                  Tier 3—Tertiary Prevention:                                                                   Tier 3—Tertiary Prevention:
                 Setting Goals With End-of-Year                                                                 Setting Goals With Rate of
                         Benchmarking
                                                                                                                  Improvement Standards

                                                                                                               M-CBM
                 50
                 45
                 40                                           end-of-year
                                                              benchmark
            ct




                 35
Digits Correc




                 30                                                                 X
                              goal-line
                 25
                 20
                 15
                 10
                          X
                  5                baseline
                  0
                      1   2   3    4      5     6    7   8    9      10   11   12   13   14
                                              Weeks of Instruction




                      Tier 3—Tertiary Prevention:                                                               Tier 3—Tertiary Prevention:
                      Setting Goals With Rate of                                                                Setting Goals With Rate of
                        Improvement Standards                                                                     Improvement Standards
                 Setting IEP goals:                                                                            50
                  – Rate of Improvement Standards                                                              45
                                                                                                               40
                       • First three scores average (baseline) = 14
                                                                                                      orrect




                                                                                                               35
                       • Norm for fourth-grade computation = 0.50                                              30                 g
                                                                                                                                  goal-line
                                                                                              Digits Co




                                                                                                               25
                       • Multiply norm by number of weeks left in year
                                                                                                               20
                                                                                                                                                                             x
                           – 16 × 0.50 = 8                                                                     15
                       • Add to baseline average                                                               10    x
                                                                                                                5
                           – 8 + 14 = 22
                                                                                                                0
                       • Student’s end-of-year goal is 22                                                           1 2   3 4 5   6 7 8       9 10 11 12 13 14 15 16 17 18 19 20
                                                                                                                                       Weeks of Instruction




                                                                                                                                                                                   7
   Tier 3—Tertiary Prevention:                                                      Tier 3—Tertiary Prevention:
        Setting Goals With                                                               Setting Goals With
   Intra-Individual Framework                                                       Intra-Individual Framework

 Setting IEP goals:                                                            50
                                                                               45
  – Intra-individual framework
                                                                               40
     • Identify weekly rate of improvement (slope) using at




                                                                        rect
                                                                               35           Slope = 0.625
       least eight data points




                                                              Digits Corr
                                                                               30
     • Multiply slope by 1.5                                                   25
                                                                               20
     • Multiply by number of weeks until end of year
                                                                               15
     • Add to student’s baseline score                                         10
     • This is the end-of-year goal                                             5
                                                                                0
                                                                                    1   2 3 4   5 6 7    8 9 10 11 12 13 14 15 16 17 18 19 20
                                                                                                        Weeks of Instruction




Tier 3—PM in Tertiary Prevention:                                              Tier 3—Tertiary Prevention:
        Setting Goals With                                                          Setting Goals With
   Intra-Individual Framework                                                  Intra-Individual Framework
 Setting IEP goals
  – Intra-individual framework                                                 50
                                                                               45
     • Identify weekly rate of improvement using at least                      40
       eight data points
                                                                        rect




                                                                               35
                                      0.625
         – First eight scores slope = 0 625
                                                              Digits Corr




                                                                               30
     • Multiply slope by 1.5                                                   25                                                         x
         – 0.625 × 1.5 = 0.9375                                                20
     • Multiply by number of weeks until end of year                           15
                                                                               10
         – 0.9375 × 12 = 11.25
                                                                                5
     • Add to student’s baseline score                                          0
         – 11.25 + 12.00 = 23.25                                                    1   2 3 4   5 6 7    8 9 10 11 12 13 14 15 16 17 18 19 20
     • 23.25 (or 23) is student’s end-of-year goal                                                      Weeks of Instruction




                                                                                                                                                8
                                Tier 3—Tertiary Prevention:
                              Decision Making with Formative                                                   Analysis of Trend
                                 Progress Monitoring Data
                               Interpretative Approaches                                                 Decision rules for progress monitoring
                                   – Visual Analysis                                                     data:
                                        • Useful in evaluating practice when changes (either
                                          level or slope) are obvious                                    – Based on four most recent consecutive scores
                                        • Based on baseline logic which can be an                                   student s trend-line
                                                                                                         – Based on student’s trend line
                                          impediment
                                   – Analysis of Trend
                                        • With reliable and sufficient data can make adequate
                                          formative decisions
                                        • More data means better decisions
                                        • Errors in decision making are affected by variability
                                          in the data
                                   – Tests of Statistical Significance
                                        • Hypothesis Test for Slope




                                    Tier 3—Tertiary Decision
                                   Making: Four-Point Method
                                                                                                               Analysis of Trend
                              30                                                                         Based on four most recent consecutive
Problems Corre in 7 Minutes




                              25                 most recent 4 points                                    scores
                              20                                                                    X    – If the four most recent consecutive scores are
                                                                                                                         g         ,   p
                                                                                                           all above the goal-line, keep the current
             ect




                              15

                              10
                                                                                      goal-line            intervention and increase the goal

                               5

                               0
                                    1    2   3   4   5    6    7     8    9     10   11   12   13   14
                                                         Weeks of Instruction




                                                                                                                                                            9
                                      Tier 3—Tertiary Decision
                                     Making: Four-Point Method
                                                                                                                                       Analysis of Trend
                                30                                                                                             Based on four most recent consecutive
                                                                                                                               scores
Problems Corre in 7 Minutes




                                25

                                20                                                                                        X    – If the four most recent consecutive scores are
                                                                                                                                 all above the goal-line, keep the current
                                                                                                                                               g         ,   p
             ect




                                15

                                                                                                         goal-line
                                                                                                                                 intervention and increase the goal
                                10
                                                                                                                               – If the four most recent consecutive scores are
                                 5
                                                        most recent 4 points                                                     all below the goal-line, keep the current goal
                                 0                                                                                               and modify the instruction
                                     1    2   3   4     5        6       7       8       9        10    11    12    13    14
                                                             Weeks of Instruction                                              – When the four most recent consecutive scores
                                                                                                                                 are neither above or below the goal-line,
                                                                                                                                 maintain the current goal and instruction and
                                                                                                                                 continue to progress monitor




                                         Tier 3—Tertiary Decision
                                         Making Based on Trend
                                                                                                                                       Analysis of Trend
                                30                                                                                             When the trend-line is steeper (i.e., accelerating)
                                                                                                                               relative to the goal-line, keep the current
            rect in 7 Minutes




                                25                    trend-line
                                                                                                                     X
                                                                                                                               intervention and increase the goal
                                20

                                15
                                                                                                                               When trend-line is lower (i.e., decelerating)
Problems Corr




                                                                                                   goal-line                   relative to the goal-line, keep the current goal and
                                10       X
                                                                                                                               modify the instruction
                                 5
                                                                                                                               When the trend-line is equal (i.e., parallel) to the
                                 0
                                     1    2   3   4    5     6       7       8       9       10    11    12    13    14
                                                                                                                               goal-line, maintain current goal and instruction
                                                            Weeks of Instruction                                               and continue to progress monitor




                                                                                                                                                                                      10
                                          Tier 3—Tertiary Decision
                                          Making Based on Trend
                                                                                                                         Analysis of Trend
                                 30                                                                              When the trend-line is steeper (i.e., accelerating)
Problems Cor rect in 7 Minutes




                                 25                                                                              relative to the goal-line, keep the current
                                 20                                                                         X    intervention and increase the goal
                                 15
                                                                                                                 When trend-line is lower (i.e., decelerating)
                                 10
                                                                                                goal-line        relative to the goal-line, keep the current goal and
                                          X
                                                                                                                 modify the instruction
                                  5
                                                   trend-line
                                  0
                                                                                                                 When the trend-line is equal (i.e., parallel) to the
                                      1    2   3     4    5      6    7    8   9      10   11   12   13     14   goal-line, maintain current goal and instruction
                                                               Weeks of Instruction
                                                                                                                 and continue to progress monitor




                                          Tier 3—Tertiary Decision
                                          Making Based on Trend
                                                                                                                         Analysis of Trend
                                 30                                                                              When the trend-line is steeper (i.e., accelerating)
Problems Cor rect in 7 Minutes




                                 25
                                                                                                                 relative to the goal-line, keep the current
                                 20
                                                                                                            X    intervention and increase the goal
                                 15
                                                                                                                 When trend-line is lower (i.e., decelerating)
                                 10
                                                                                                goal-line        relative to the goal-line, keep the current goal and
                                          X
                                                                                                                 modify the instruction
                                  5
                                                              trend-line
                                                                                                                 When the trend-line is equal (i.e., parallel) to the
                                  0
                                      1    2   3     4    5      6    7    8   9      10   11   12   13     14   goal-line, maintain current goal and instruction
                                                               Weeks of Instruction                              and continue to progress monitor




                                                                                                                                                                        11
 Some of our recent findings …..                                                                                     We’ve also found …..
                                                               Four-Point Rule                  Five-Point Rule
                                                                                                                      In order to get “reliable” data, 16 to 20
                                                               False           True             False      True
                                                              Positive        Positive         Positive   Positive    data points are probably needed
                                            2                  0.19             1.00            0.07       1.00       Up to about six weeks, a pre- post-test
                                            4                  0.27             1.00            0.12       1.00
                                                                                                                      analysis of difference scores is more
                                            6                  0.31             1.00            0.14       0.98
                                            8                  0.32             0.98            0.14       0.91
                                                                                                                      reliable than analysis of trend
                               SEb          10                 0.33             0.93            0.15       0.82       BUT ……
                                            12                 0.35             0.89            0.16       0.73
                                            14                 0.34             0.84            0.17       0.65
                                                                                                                      Decision making can be improved when
                                            16                 0.34             0.79            0.17       0.59       reliability of the measurement system is
                                            18                 0.35             0.75            0.17       0.54       improved




                               Hypothesis Test for Slope                                                                          Questions?
                              30
Problems Corre in 7 Minutes




                              25                                         t=b1/SEb1

                              20
                                       Slope = 1.71                      = 1.71/.35
             ect




                              15                                         = 4.89
                              10
                                                                         t with 6 df = 1.94
                               5                                         Significant > 0.005
                                       Standard Error = 0.35
                               0
                                   1    2   3    4   5    6     7    8    9     10   11   12    13   14
                                                         Weeks of Instruction




                                                                                                                                                                  12

								
To top