IES Presentation

							Are We Making Progress Yet?


       Linda A. Reddy, Ph.D.
         Rutgers University
A rogue wave smashed the bridge of the
ship, destroying all the instruments except
the speedometer. The captain announced
to the badly shaken crew, "We have no idea
where we are and we don't have a clue
where we are going, but we do know we are
making record time."
Progress Monitoring is All About
 Change and Change is Hard
What is Progress Monitoring?
 “A scientifically based practice that is used to assess students’
  academic performance and evaluate the effectiveness of
  instruction” (National Center on Students Progress Monitoring)

 A “seamless and flexible” approach ….. that can be used across
  students of different age and skill levels, different settings and
  curricula, and across school years (Wallace et al., 2007)

 “A set of assessment procedures for determining the extent to
  which students are benefiting from classroom instruction and for
  monitoring effectiveness of curriculum” (The National Research
  Center on Learning Disabilities)
    Historical Context of Progress Monitoring

   Traditional progress monitoring has been used to identify one’s relative
    position within a group rather than evaluating individual progress across time
    (Deno, 1997)

   The early practice was a means of establishing eligibility for certain
    classifications/programs (e.g., special education, gifted)

   Originally, progress monitoring was based on “mastery measurement”
      Criterion-referenced tests used to assess the mastery of specific
        curriculum objectives
      Following mastery of specific objectives (skills), the teacher would provide
        instruction and assess the skill taught
      This method of progress monitoring was reflected in common teaching
        practices (e.g., Wisconsin Instructional Design System, Precision
        Teaching)
    Historical Context of Progress Monitoring
■   Questions arose - mastery measurement's assumption that a series of short-term
    objectives directly leads to broad-based competence (false sense of progress)

   Resulted alternative ways of conceptualizing progress monitoring

   Curriculum-Based Measurement (Deno, 1985)
      Requires students to simultaneously integrate the various skills required for
        competent yearend performance on weekly test
      As students learn the necessary components of the curriculum, their CBM score
        gradually increases
      Slope can be used to quantify rate of learning and gauge students’ responsiveness
        to an instructional program or OTL
      When “inadequate” responsiveness is revealed, program can be revised

   NCLB (2001)

   IDEA (2004) – LD assessment - removal of discrepancy criteria
         Advent or “rebirth” of RtI
What is Student Social Behavior?

   Broad global construct
   Heterogeneous
   Complex in nature and context
   Lack of agreement
    How “Student Social Behavior” is
    Defined?
■ School and interpersonal connectedness (attachments) (Bond et al., 2007)

   Cooperation, assertion, self-control (Rutherford, DuPaul, & Jitendra, 2008)

   Self and social awareness, responsible decision-making, self-management and relationship-
    management skills (Zins et al., 2004)

   Physical and intellectual actions (e.g., hygiene, nutrition, physical activity, avoiding harmful substances,
    decision-making skills, creative thinking), social/emotional actions for managing oneself responsibly
    (e.g., self-control, time management), getting along with others (e.g., empathy, altruism, respect,
    conflict resolution), self-honesty, integrity, self-appraisal and continuous self-improvement (e.g., goal
    setting, problem solving, courage to try new things, persistence) (Snyder et al., 2010)

   Cooperation (i.e., puts work materials or school property away), Assertion (introduces him / herself to
    new people without being told), and Self-control (control temper in conflict situations with adults). Level
    of aggression towards peers, helpfulness with other children, interactions with peers, involvement or
    exclusion by peers, level of anxious-fearfulness, and hyperactivity-distractibility (Rimm-Kaufmann &
    Chiu, 2007)

   Awareness of self and others, positive attitudes and values, responsible decision making, social
    interaction skills (e.g. active listening, expressive communication, cooperation, negotiation, refusal,
    help seeking) (Payton et al., 2000)
        Student Social Behavior and
           Academic Outcomes
 A comprehensive K-5 school-based program targeting student
  behavior and character was linked to academic achievement,
  school attendance, and positive disciplinary outcomes (Snyder
  et al., 2010)

 ADHD 1-4 grade students - Teacher ratings of reading gains
  following academic interventions related to improvements in
  social skills on the SSRS
    Peer-mediated math interventions produced improvements
       in math fluency and self-control (Rutherford, DuPaul, &
       Jitendra, 2008)

 Social problem-solving skills are related to cognitive problem-
  solving skills (Ben-Avie, & Ensign, 2003)
         Student Social Behaviors and
           Mental Health Outcomes
 Low school connectedness and high interpersonal conflict in early
  secondary school are linked to increased mental health problems and
  substance use in later schooling.
    The presence of both school and social connectedness was
     associated with the lowered risk of depressive symptoms in
     students (Bond et al., 2007)

 Good Behavior Game (GBG) universal prevention intervention in first
  grade classrooms predicted lower levels of antisocial behavior by
  middle school (Kellam & Anthony, 1998)

 GBG intervention provided in 1-2 graders (n=1,196) significantly
  reduced the risk (50%) drug abuse/dependence, smoking, and
  antisocial behavior for adolescent and young adult (Kellam et. al.,
  2008)
Three Presentations: Some Shared
Ingredients
 Comprehensive and rigorous programs of research
 Emphasis on student social behavior in schools
 Informant is the teacher (general education or special
  education)
 Reliance on teacher accuracy and cooperation
 All tools are brief, user friendly and easy to administer
 Present integrative assessment-intervention models
 Assessment is dynamic, fluid, and continuous
 Framed in a three tiered RtI model
 Emphasis test scores (raw or scale score) used as a method for
  determining efficacy of interventions
Enhancing Individual Education Plans for Children
with Attention-Deficit/Hyperactivity Disorder Using
a Daily Report Card

   Disorder-specific tool
   ADHD representing a highly comorbid population
   Assessment is fluid for daily use
   Students receive immediate teacher feedback on their efforts
   Can be used to develop positive student-teacher and student-
    parents interactions/relationships
   Fosters parent and teacher communication and partnerships
   Scores linked to home-based privileges
   Tailored for individual student needs – linked to IEP and 504 Plan
    goals – critical for buy in
   Assesses targeted behaviors within specific academic lessons (e.g.,
    art, math, language arts, science etc.)
   Includes benchmarks of specific social behavior within context
   DRC raw scores discriminate between treatment conditions
      Development and Validation of Progress Monitoring
      Tools for Social Behavior:
      Lessons from Project VIABLE

■   DBR - undergone significant development and investigation, until now there has been limited
    attention to the psychometric properties

■ Highly reliable and valid alternative to SDO, interviews, and behavior rating scales

■ Emphasis on specific time, place (context) and intended rater for assessing the behavior

■ Excellent tool for IEP Goals and 504 Plans for K-12 grade students

■ Terrific tool for minimizing effects of rater inference and retrospective judgments about student
  social behavior

■ Considerations for developing other DBR measures:
         Rater bias; rater skill (error)
         Feasibility
         Consideration of base rates within context
         Wording (molar/molecular, negative/positive)
         Visual cues
         Anchors
         General outcome or individualized targets
Scale gradients – first study to examine a single item scale compared
   three scale gradients (categories – 6, 10, 14) Chafouleas et al. (2009) Educational
   and Psychological Measurement


Illustrates that DBR are highly useful for measuring the effects of
    intervention on individual and classroom behavior problems

DBR-SIS: 2 part web-based training program – frame of reference and
  rater error (accuracy)

DBR-BASIS: web-based program for data entry, analysis, and graphing
       Developing Change Sensitive Brief Behavior Rating
       Scales as Progress-Monitoring Tools for Social
       Behavior

■ Significant research on social skills (SSRS) serves as the spring board for the
   development of 4 brief behavior ratings scales for social skills, conduct, anxiety,
   and ADHD

■ Construct specificity - detect progress in specific area (skills) of social emotional
  functioning

■ Emphasis on change-sensitivity of constructs and items

■ Proposes that change sensitive item are identified by four data analytic methods
       [Odds ratio, Effect sizes, Paired and 2-sample T-tests,
       Interaction effect from a mixed-factorial ANOVA]

■ Emphasizes the assessment of both strength and problem behaviors
      “Dual lens” assessment approach – particularly important in the area of ED
  Where to go from here?

 Opportunities for measure development are
  endless!

 Adopt a broad iterative test development and
  validation approach:
   Unified model of validation (Messick, Cronbach) and…
   Rigorously examine and validate test utility
      Utility of test score(s) for informing decision making for
       intervention plans that improve students’ lives (meaningfully)
Test Utility Framework for Future Test
Development and Validation

 “In developing a test, one must first decide
  on the proposed interpretations and uses -
  and the assumptions embedded in the
  proposed interpretations and uses. The
  chosen interpretations and uses then
  provide guidance on how to proceed in
  developing the test” (Kane, 2008; pp. 77).
    Before Test Development and Validation

   What is the specific construct(s) we intended to measure?

   What is the intended purpose of the test?

   Include a strong consumer participatory model for test development

   What is the "end game” goal?

   Test score utility (inference):

        How can the test score(s) generalize to the function and process of the targeted social
         behavior in the classroom?

        How can the test score(s) be used by the consumers to inform decision making for
         intervention plans:
              individual student
              classrooms
              schools
              school district level

   We must go beyond student level assessment and focus on system level assessment and
    change
Where to go from here?

   Adopt a Strong Program of External Validation

      Acceptability and usefulness of consumers

      Carefully examine consumers’ intended and unintended test score
       inferences

      What are the extraneous factors that impact test score utility
       (inferences) for the consumer?

      Do test scores change with targeted practical school interventions
       and how do test scores change?

      Incremental validity over business as usual

      What are the pragmatic factor (barriers) that may impede
       development and validation (e.g., time, cost, teacher sensitivity or
       bias towards constructs and items)?
Example Barriers for Future Measurement
Development
   Barriers:
      Consumers’ perceptions of time, work, and cost
      Consumers’ perceived utility
      Progress monitoring instruments may not be sufficiently sensitive
       to monitor progress among high school students because of small
       growth rates in this age range
      Progress monitoring tools must be validated for different age
       groups (e.g., secondary students)
      Absence of school staff buy-in
      School personnel misunderstanding and ineffective use of data
Example Facilitators for Future Measurement
Development

 Facilitators:
     Committed administrative leadership and support
     District-wide support
     Ongoing professional development
     Cohesive team membership
     Team efficiency (e.g., clear operating standards)
     Stakeholder agreement and shared vision regarding the change
     Autonomous teachers
     School psychologists serving as leaders, change agents, and
      consultants
     Financial resources committed to the cause
     Organizational restructuring – system change
                  Where to go from here?:
                How about teacher behavior?
   What teachers do in the classroom influences students’ learning and behavior.

   GBG intervention with 570 2-3 graders - reduced use of negative teacher remarks
    predicted improvements in on-task behavior and talking out behavior.
      Improved student behavior mediated the impact of the intervention on the
        development of hyperactive and oppositional behavior (Leflot et al., 2010).

   Teachers’ social and emotional competence are key to “creating a classroom climate
    that is more conducive to learning and that promotes positive developmental outcomes
    among students” (Jennings & Greenberg, 2009).

   Teachers who interact with their students in supportive ways promote positive student
    social behavior (Wentzel, et al., 2010).

   Problematic relationships between Kindergarten teachers and their students with
    behavior problems predict academic and behavior problems through eighth grade
    (Hamre & Pianta, 2001).

   Students with ED are significantly influenced by teacher-student interactions
    (Sutherland et al., 2008; Reddy et al., 2009).
     Future Measurement Development Projects

   Tools for preschool, elementary school, and middle and high school that inform supports
    and interventions

   Tools that are dynamic and flexible for 6-12 grade - multiple teachers and the complexity
    of settings (classroom, hallway, study hall, lunch, after school activities such as clubs and
    sports)

   Tools tailored for specific disorders and conditions - example Aspergers, Autism, and
    Anxiety/Depression

   Brief tools for teacher and student interactions (specific aspects of classroom and school
    climate)

   Tools for teacher classroom practices – general and special education settings

   Tools assess aspects of teachers' social emotional competency (e.g., anger management,
    efficacy, perceived risk of violence)

   Tools that can link meaningful goals for IEPs and 504 plans
Future Measurement Projects -
Opportunity for School Reform
 By 2015 over 1/3 of the nation’s veteran teachers and
  school administrators (baby boomers) will be leaving
  the field of education.

 This change in school personnel offer the nation
  tremendous opportunity for comprehensive school
  reform and innovation in teacher preparation and
  training across the nation.

 The time is right for new progress monitor tools for
  students and teachers.
Thank you!