Docstoc

Data-Based Decision-Making

Document Sample
Data-Based Decision-Making Powered By Docstoc
					  Data-Based
Decision-Making
  SYSTEMS            PRACTICES             DATA
 Leadership Team     Identify Skills     Central Data
                                           System
Coaching Capacity     Instruction
                                       Consistent Data
                     Reinforcement         Entry
  Professional
  Development
                                       Efficient Referral
                       Instructive
 Communication                               Form
                       Discipline

                     Continuum of       On-going Data
Resources - Time,
                     Consequences         Analysis
 Money, People

                        Routines        Evaluation Plan
Policy Development
         Evaluation Plan
 Implementation Assessment – Are
  we doing what we said we would
  do?
  – SET - School-wide Evaluation Tool
  – EBS Self-Assessment Survey
  – TIC - Team Implementation Checklist
 Outcome Assessments
  School-wide Evaluation
       Tool (SET)
 Designed as a research instrument
  for determining the extent to which
  a school is implementing school-
  wide positive behavior support
 An outside person comes to the
  school and (a) interviews students
  and faculty, (b) observes school
  environments, (c) reviews school
  documents, and (d) interviews
  school administrator(s)
       SET - Logistics
 Visit is completed by a SET Coach
 Initial baseline SET prior to
  implementation AND once per year
  after implementation
 A school has documented
  implementation of school-wide
  behavior support when
  – 80% total overall SET score AND 80%
    on “Expectations Taught”
        Data from the SET
 The SET produces a summary score (%)
  and a score for each of seven features of
  school-wide behavior support
  – Behavioral expectations defined
  – Behavioral expectations taught
  – Behavioral expectation rewarded
  – Systematic responses to rule violations
  – Information gathered to monitor student
    behavior
  – Local management support for school-
    wide procedures
  – District level support for school-wide
    procedures
    Results from the SET
 Report will include
  – Graph of the percent scores
  – List of strengths
  – List of priority recommendations
 Remember, a school has
  documented implementation of the
  Core Cycle when
  – 80% total overall SET score AND 80%
    on “Expectations Taught”
                                                            Perc ent
Ex
     pe
       ct
               at
                  io
                         n
                                                                                 100




                                                      20
                                                           40
                                                                       60
                                                                            80




                                                  0
                             D
                                 ef
                                    in
Ex                                       ed
  pe
    ct
              at
                 io
                      ns
                             Ta
                                u      gh
                                         t
      R
           ew
                    ar
                      ds
                             Sy
                                   st
                                     em
     Vi
          ol
             a   tio
                    ns
                             Sy
                                  st
                                       em


                         M
                             on
                               ito
                                         rin
                                            g

                    M
                     an
                       ag
                                  em
                                         en
                                              t
          D
              ist
                    ric
                       tS
                             up
                               po
                                         rt


                     M
                         ea
                           n
                                   Sc
                                     or
                                              e
 Pre
                                                                Perc ent
 Ex
         pe
           ct
                   at
                      io
                             n
                                                                                     100




                                                          20
                                                               40
                                                                           60
                                                                                80




                                                      0
                                 D
                                     ef
                                        in
Ex                                           ed
         pe
           ct
                  at
                     io
                          ns
                                 Ta
                                    u      gh
                                             t
          R
               ew
                        ar
                          ds
                                 Sy
                                       st
                                         em
         Vi
              ol
                 a   tio
                        ns
                                 Sy
                                      st
                                           em


                             M
                                 on
                                   ito
                                             rin
                                                g

                        M
                         an
                           ag
                                      em
                                             en
              D                                   t
                  ist
                        ric
                           tS
                                 up
                                   po
                                             rt


                         M
                             ea
                               n
Pre




                                       Sc
                                         or
                                                  e
Post 1
                                                                Perc ent
    Ex
         pe
           ct
                   at
                      io
                             n
                                                                                     100




                                                          20
                                                               40
                                                                           60
                                                                                80




                                                      0
                                 D
                                     ef
                                        in
  Ex                                         ed
         pe
           ct
                  at
                     io
                          ns
                                 Ta
                                    u      gh
                                             t
          R
               ew
                        ar
                          ds
                                 Sy
                                       st
                                         em
         Vi
              ol
                 a   tio
                        ns
                                 Sy
                                      st
                                           em


                             M
                                 on
                                   ito
                                             rin
                                                g

                        M
                         an
                           ag
                                      em
                                             en
              D                                   t
                  ist
                        ric
                           tS
Pre


                                 up
                                   po
                                             rt


                         M
                             ea
                               n
Post 1




                                       Sc
                                         or
                                                  e
Post 2
                                                                Perc ent
    Ex
         pe
           ct
                   at
                      io
                             n
                                                                                     100




                                                          20
                                                               40
                                                                           60
                                                                                80




                                                      0
                                 D
                                     ef
                                        in
  Ex                                         ed
         pe
           ct
                  at
                     io
                          ns
                                 Ta
                                    u      gh
                                             t
          R
               ew
                        ar
                          ds
                                 Sy
                                       st
                                         em
         Vi
              ol
                 a   tio
                        ns
                                 Sy
                                      st
                                           em


                             M
                                 on
                                   ito
                                             rin
                                                g

                        M
                         an
                           ag
                                      em
                                             en
                                                  t
              D
                  ist
                        ric
Pre

                           tS
                                 up
                                   po
                                             rt


                         M
Post 1




                             ea
                               n
                                       Sc
                                         or
                                                  e
Post 2
     One of Your Tasks
         Today…
 Schedule a baseline SET visit prior
  to implementing PBS practices in
  your building
 Effective Behavior Support
        Survey (EBS)
 Self-assessment survey to assess the
  extent to which PBS practices and
  systems are in place within a school
  –   School-wide
  –   Non-classroom
  –   Classroom
  –   Individual Student
 Used to measure staff perception of
  priority for change (typically sometime
  after PBS overview)
        EBS - Logistics
 Online survey is completed by staff
  members
  – For each item, rate whether the
    feature is in place AND rate the
    priority level for improvement
 Initial baseline prior to
  implementation AND once per year
  after implementation
 Goal is to have ~ 75% of staff
  indicating that a system is “in
      Data from the EBS
 For each of the four “systems”
  – Current Status
     Percent “in place”
     Percent “partially in place”
     Percent “not in place”
  – Priority for Improvement
     High
     Medium
     Low
          EBS Survey: Current Status
                                           Example
% 100
   90
o
   80
f
   70
r 60
e
s 50
p   40
o
    30
n
s   20
e   10
s
     0
         In plac e partial   not   inplace partial   not   In plac e partial    not    In plac e partial   not


             School-wide                      Non-classroom                    Classroom
         Individual
                 EBS Survey: Priority for
                       Improvement Example
  100
%
   90
o 80
f
   70
r 60
e
s 50
p 40
o
n 30
s 20
e
s 10
    0
          high   med   low   high   med   low   high   med    low        high   med   low


            School-wide             Non-classroom            Classroom
        Individual
      One of Your Tasks
          Today…
 Schedule a time for all staff to
  complete the online EBS Self-
  Assessment Survey
      Team Implementation
        Checklist (TIC)
 Self-assessment tool for monitoring the
  implementation of school-wide PBS
 Team self-assessment in six core areas
  –   Establish commitment
  –   Establish and maintain team
  –   Self-assessment
  –   Establish school-wide expectations
  –   Establish information system
  –   Build capacity for function-based support
        TIC - Logistics
 One online survey is completed
  collectively by the PBS Leadership
  Team
 Initially completed at least three
  times per year
 Goal is to have a minimum of 80%
  of total items marked as
  “Achieved”
      Data from the TIC
 Overall self-rating of whether the
  components for each core area are
  – Achieved
  – In Progress
  – Not Started
                           Team Implementation Checklist Data: Cohort 1
                            % Achieved (Solid); % In Progress (Stripes)

          100
          90
          80
          70
Percent




          60
          50
          40
          30
          20
          10
           0
                May Oct Jan May Oct Jan May Oct Jan May Oct Jan May Oct Jan May Oct Jan
                05 05 06 05 05 06 05 05 06 05 05 06 05 05 06 05 05 06
                    1            2            3             4             5      6
                                                   Team
        Team Activity
 Complete the baseline TIC for your
  school as a team
How Do You Use the Results?
 Review the results of the
  implementation assessments in
  light of each other
 Identify strengths or progress and
  celebrate!
 Identify areas of needs and next
  steps; document on your Action
  Plan
         Evaluation Plan
 Implementation Assessment
  – SET - School-wide Evaluation Tool
  – TIC - Team Implementation Checklist
  – EBS Self-Assessment Survey
 Outcome Assessment
          Key Features
 Data are accurate
 Data are easy to collect
 Data are used for decision-making
 Data are available when decisions
  are made
 Data fits the context of the decision
 People who collect the data know
  the purpose and see the result
Outcome Assessment - Data
         Sources
 Office and classroom discipline
  referrals
 Attendance data
 Suspension and expulsion data
 Bullying and harassment incidents
  and/or surveys
 Safety or climate surveys
 Academic achievement
 Surveys – parent, student,
  community
Outcome Assessment – Office
  and Classroom Discipline
            Data
 Link to consequence system
 Office discipline referral (ODR)
  form or behavior incident form
 Data entry
 Data analysis
    Behavior Incident Data
 Consequence system to define
  procedures for documenting
  behavioral incidents
  – May collect data on major/office and
    minor/classroom behaviors
  – Each team must decide what level(s)
    of behavioral infractions will be
    documented
  – Tip: Documenting only major
    infractions will probably not provide
    enough information to calibrate your
      Behavior Incident Data
 Create a referral form that is easy
  to use
  –   Circle or use checkmarks
  –   Limited writing
  –   Limited space for comments
  –   Easy to compile data (e.g., frequency
      counts)
 Provides data for the system as
  well as individual students
 Information captured by the
          ODR form
 Staff referral vs.    Location
  Office referral       Homeroom
 Student                teacher
 Grade                 Referring staff
 Date                  Adult response
 Time of day            (consequence)
 Behavior              Others involved
                        Function
                                                         n
                                        Kirkpatrick Ele me tary School - Te nnessee
                                          OFFICE DISCIPLINE REFERRAL FORM

Student : ___________________________                   Referring Staff: __________________ _________ Date: ______ Time: ______
Grade : PreK K 1 2 3 4
                                                             Location of Rule Violation

   Classroom                        Cafete ria                    Libra ry                                         Sevier Street to Cente r and
                                                                                                                       behind
   Playground                       Bathroom                      Portable P ath
                                                                                                                      Field T rip
   Hallway/breezeway                Center                        Specia l event (Audit orium)/as sembly           Other
                                                                                                                       _____________ ______
                                                   Problem Behavi or (check the most intrusive)

   Abusive language /inappropri ate language                           Skip class/truan cy                       Proper ty damage
   Fighting/p hy sical aggression                                      Forger y/theft                            Bomb thre at
    Studen t Staf f       (please circle whi ch one)                    Ly ing/chea ting                          Arson
   Defianc e/disrespect/insubordina tion/                              Alcoho l/drugs/toba cco                   Weapons
    non -complia nt                                                     Combus tibles                             Other
    Harassment/bu llying/threatening                                                                                _____________ ___________
                                                                       Tantrum
    Studen t Staf f     (please circle whi ch one)
                                                                        Inappropri ate touching
   Severe Disruption

                                                                 Possible M otivation

          Obtain pee r attention                            Avoid tasks/activities                       Avoid adu lts
          Obtain adu lt atten tion                          Avoid work                                   Impulsivity
          Obtain items/ac tivities                          Avoid peer(s)                                Other
                                                                                                            _____________ __________ ____

                                                                   Others Involved

   None      Peers           Voluntee rs            Teacher/ Staff     Substitute            Unknown         Other ___________________


                                                              Administrative Decision

   Time i n Office                        Parent Co ntact                    In-class beha vior suppo rt              Isolation in anoth er
                                                                                                                          classroom
   Los s of Privilege                     Parent Co nfere nce                Out o f School Suspen sion
                                                                                                                         Other ________________
   Conferenc e with Stude nt              Restitution                        Zero D ay Suspen sion
Comment s:




Administrator’s Signature: ____________ _____________ Student’s Signature: __________________________
Parent’s Signature: _______ ___________________ Date: _____________________
T
    Behavior Incident Data

           Remember!
 ODRs are not “treatment”
  interventions
 ODRs are completed for
  documentation purposes only
 Simply filling out the form is not a
  consequence
 TEACHING changes behavior!
        Team Discussion
 Do you already have a behavior
  incident form?
 If you do, do you collect all of the
  information suggested?
            Data Entry
 Need to have a person(s) to enter
  the data into the data system
 Person should be trained on how to
  enter data
 Person should be trained on issues
  related to confidentiality
 Need to data entry in a regular and
  on-going way
         Data Analysis
 Need to have a person(s) who is
  able to analyze the data
 Person should be trained on issues
  related to confidentiality
 Person should be trained on how to
  access data needed to answer
  identified questions
   Data Needed for Analysis
 The BIG 5 –
   –   # referrals per day per month
   –   # referrals by student
   –   # referrals by location
   –   #/kinds of problem behaviors
   –   # problem behaviors by time of day
 Average referrals per day per 100 students
 System status (e.g., % of students in Core)
 Students with 3-5 office referrals and 6 or
  more office referrals
 Individual student reports
What Does the Office Referral
      Data Tell You?
 Areas of need by behavior,
  location, time of day, etc.
 Current overall status of the
  system
 Changes in office referrals over
  time
  – Total number across years
  – Changes in referral patterns
  – Changes in behavior severity
                                                                                          Number of Referrals
                                      Ab
                                            us
                       D                       i    ve
                           es
                              t   ru                      la




                                                                                    100
                                                                                            200
                                                                                                  300
                                                                                                        400
                                                                                                              500
                                                                                                                    600
                                                                                                                          700




                                                                                0
                                    ct                      ng
                                         io                        ua
                                            n                        ge
                                                of
                                                     pr
                                                       op
                                                         er
                                                                   ty

                                                         D
                                                             isr
                                                                  up
                                                                       tio
                                                     H                    n
                                                         ar
                                                           as
                                                                sm
                                            N                          en
                                                on                       t
                                                   c     om
                                       O                   pl
                                        ve                    ia
                                                                nc
                                                rt                         e
                                                     di
                                                        s  re
                                  Ph                            sp
                 Po                    ys                         ec
                      ss
                        es
                                           ica                      t
                                                    la
                           s io                          gg
                                  n                         re
                                      of                       s
                                         a   lco
                                                                   sio
                                                                       n
                                                                                                                                ODR by Behavior




                           Po                        ho




    Be hav ior
                                ss                        l/d
                                   es                         r ug
                                      s   io                           s
                                                n
                                                    of
                                                       w
                                                             ea
                                                                p   on
                                                                      s

                                                                   Ta
                                                                      r    dy

                                                                   Th
                                                                      e    ft
                                                             Tr
                                                               ua
                                                                        nc
                                                                          y
2004-2005
                                                                Number of Referrals




                                                          100
                                                                 200
                                                                       300
                                                                             400
                                                                                      500
                                                                                            600
                                                                                                  700




                                                      0
                                             Bu
                                                  s
                             C
                                 la
                                    ssr
                                       oo
                                         m
                      C
                          om
                             m
                                  on
                                         Ar
                                             ea


                                   H
                                        al
                                           lw
                                             ay
                          Lo
                            ck
                               e   rR
                                         oo
                                           m
                            Lu
                              nc
               O                    hr
                ff                        oo
                     Sc
                       ho                    m
                            ol
                               G
                                   ro




    Location
                                     un
                                             ds
                                                                                                        Office Referrals by Location




                                        O
                                         f fi
                                                ce
                            Pl
                                 ay
                                    g   ro
                                             un
                                                d
                             R
                                 es
                                    t   ro
                                          om
                     Sc
                       ho
                         ol
                                 G
                                  ro
                                     u     nd
                                             s
2004-2005
                                               ODR by Month

                      350



                      300



                      250
Number of Referrals




                      200



                      150



                      100



                       50



                        0
                            Sept   Oct   Nov   Dec     Jan    Feb   March   April    May
                                                     M onth                         2004-2005
                                                                                                    Number of Referrals
                                             Ab
                                                   us
                              D                       i   ve
                                  es
                                     t   ru                    la




                                                                                              100
                                                                                                      200
                                                                                                            300
                                                                                                                  400
                                                                                                                        500
                                                                                                                              600
                                                                                                                                    700




                                                                                          0
                                           ct                    ng
                                                io                       ua
                                                   n                       ge
                                                       of
                                                            pr
                                                               op
                                                                 er
                                                                   ty

                                                              D
                                                                  isr
                                                                      up
                                                                             tio
                                                          H                     n
                                                              ar
                                                                 a   ss
                                                                         m
                                                   N                      en
                                                       on                   t
                                                          c    om
                                              O                       pl
                                               ve                        ia
                                                                           nc
                                                       rt                    e
                                                          di
                                                             s   re
                                         Ph                           sp
                        Po                    ys                        ec
                             ss
                               es
                                                  ica                            t
                                                        la
                                  s io                        gg
                                         n                       r   es
                                             of
                                                a   lco
                                                                          sio
                                                                                 n
                                  Po                        ho




    Be hav ior
                                       ss                        l/d
                                                                     r
                                                                                                                                          Office Referrals by Behavior




                                          es                            ug
                                             si
                                                o                            s
                                                    n
                                                        of
                                                           w
                                                                  ea
                                                                     p    on
                                                                            s

                                                                         Ta
                                                                            r    dy

                                                                          Th
                                                                             e       ft
                                                                   Tr
                                                                     ua
                                                                             nc
                                                                                y
2005-2006
                 2004-2005
          Data Systems
 High impact decisions require high
  quality data
 Good data system have these
  qualities
  – Real time capabilities
  – Graphic displays of data
  – Manipulate and summarize data by
    target behavior, location, etc.
           To Graph, or Not To Graph
                                                        Baseline      Student   Base Yr1
                       ODR for Students
                                                        Year 1        Brian     12   8
      20                                                              Kyle 0    1
                                                                      Mary      8    8
      15
                                                                      Sue       5    2
ODR




      10                                                              Amy 2     2
                                                                      Jim       10   5
       5
                                                                      Joe       0    1
       0                                                              Erik      16   12
           Ky le Joe Jane Amy      Sue Mary Jim Brian Erik
                                                                      Jane      1    0
                                Student
Central Data System Options
 School-Wide Information System
  (SWIS)
 HEART - Behavior Module
 Information Management Systems
  (IMS)
 Excel
              SWIS
 Stores major and minor behavior
  incident data
 Real-time capability
 Will automatically graph and
  generate reports
 $250.00 per year
 www.swis.org
 HEART - Behavior Module
 Stores major and minor behavior
  incident data; may also store
  academic data
 Real-time capability
 Will automatically graph and
  generate reports
 Free to schools within Heartland
                IMS
 JMC, Power School, Infinite
  Campus
 Typically will store information
  about major behavior incidents
  (e.g., those that lead to
  suspensions/expulsion)
 Will store attendance data
 Typically will generate summary
  reports of the data; doesn’t usually
  create graphs of the data
               Excel
 Will store data categories selected
  by user
 Can be programmed to generate
  graphs
 Difficult to create graphs across
  multiple years of data
             Activity
 Review the data system your
  building currently uses to store
  behavior incident data
 Discuss whether or not your
  system will meet your data analysis
  needs within PBS and, if not,
  identify what system will you use
  instead
 Resources – Do you have people to
  do data entry and analysis? Do
Outcome Assessment – Other
      Data Sources
 Iowa Youth Survey
 Bullying and harassment incidents
  and/or surveys
 Safety or climate surveys
           Iowa Youth Survey
 Given every 3 years by Iowa Department
  of Public Health
 Shorter version available to give annually
 Given to students in 6th, 8th, & 11th
  grade
 Areas of assessment:
  –   Secure and supportive family
  –   Safe and supportive school climate
  –   Safe and supportive community
  –   Healthy youth-avoidance of risky behavior
  –   Socially competent youth
  –   Youth successful at school
  –   Youth prepared for a productive adulthood
            Bully Surveys
 Olweus Bully-Victim Questionnaire
  – 29 items
  – Grades 3 and higher
  – Questions read out loud
  – Computer program available to analyze
  – $280 for questionnaire and statistical
    program
  – Questions address being bullied, being a
    bully, types of bullying, locations, and
    outcomes
      Bully Surveys (cont.)
 Bully-Proofing Your Schools (Sopris-
  West)
  – Student, parent, and staff surveys available
  – Elementary and secondary
  – Online survey available through AEA
  – Briefer than Olweus survey
  – Questions address being bullied, locations,
    and bystander behavior
  – Some questions about overall school climate
Safety and Climate Surveys
 School Safety Survey [Sprague, Colvin, & Irvin
  (1995)] (available at www.pbis.org)
   – Assesses risk factors for school safety and violence
   – Assesses response plans for school safety and violence
 California School Climate and Safety Survey
  (available in Safe, Supportive, & Successful
  Schools)
   – Elementary and secondary versions
   – Questions address safety, violence, bullying, peer and
     adult support
 School Culture Triage Survey
   – 17 questions address school culture (e.g., collaboration,
     collegiality, efficacy, self-determination)
   – Given to staff members
Outcome Assessment – Other
      Data Sources
   Attendance
   Suspensions and expulsions
   Drop-out/graduation rates
   Academic achievement
   Demographic data
   Surveys – parent, student,
    community

    May all be useful to help determine
         Evaluation Plan
 Determine evaluation measures
  before implementing
  – Collect baseline data
  – Determine goal
  – Collect data throughout year of
    implementation
  – Evaluate performance compared to
    goal
  – Use data to make decision(s) about
    the core program
           SMART Goals
   S - specific
   M - measurable
   A - attainable
   R - realistic
   T - time bound
         Evaluation Plan
Outcome Area: Implementation
Assessment: SET
Baseline: 50% overall mean, 20%
  expectations taught
Goal: By May 2008, score 80% on the
  overall mean and on the Teaching
  Expectations categories of the SET
Ending Performance: 75% overall mean,
  75% expectations taught
            Evaluation Plan
Outcome Area: Implementation
Assessment: EBS Survey
Baseline: 85% of staff rate school-wide as
 a high area of improvement and 25% of
 staff indicated that the school-wide
 system is “in place”
Goal: By May 2008, at least 75% of staff
 indicate that the school-wide system
 is “in place”
Ending Performance: 65% indicated “in
  place”
          Evaluation Plan
Outcome Area: Student Outcomes
Assessment: Office Referral Data
Baseline: 1250 office referrals in
 2006-07 and 1500 in 07-08
Goal: By May 2009, have a 5% overall
 decrease in office referrals for the
 school year
Ending Performance: 7% decrease
       Team Discussion
 What outcomes are you interested
  in monitoring as part of PBS
  implementation? What is important
  to you as a staff member? What is
  important to parents? What is
  important to stakeholders?
       Team Work Today
 Before you leave
  – Schedule a baseline SET visit
  – Schedule a time for staff members to
    complete the EBS Self-Assessment
    survey
   On-Going Team Work –
       Data System
 Determine which data system you
  will use to house your behavior
  incident data and inform PBS
  Consultant
 Identify who will do data entry and
  data analysis
  – Identify training needs
  – Identify needed resources
    On-Going Team Work –
    Evaluation Plan for PBS
 Write at least one PBS goal for
  implementation and at least one
  goal for PBS outcomes
  – Evaluation Plan form
 Collect baseline data during the
  2009-10 school year
    On-Going Team Work –
   Additional Things to Think
            About…
 Policy
  – Confidential data entry and data use

 Communication
  – How are you going to communicate
    the new data system to staff?
  – How will you communicate
    assessment results to all staff
    members?
 Team Work - Data System
 Professional Development
  – How will staff be trained to do data
    entry and data analysis?
  – How will staff be trained regarding
    confidentiality of data?
  – When will staff be trained?

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:3
posted:8/31/2012
language:Unknown
pages:71