Section I: Basic Information by cbtms2B9

VIEWS: 5 PAGES: 27

									                     NATIONAL CENTER ON INTENSIVE INTERVENTION




 PROTOCOL FOR EVALUATING
PROGRESS MONITORING TOOLS



                                            May 2012




The National Center on Intensive Intervention defines progress monitoring as repeated measurement
of academic performance for the purpose of helping schools individualize instructional programs for
students in grades K-12 who have intensive instructional needs. For this purpose, progress monitoring
is collected weekly to assess whether student progress is adequate to meet the student’s instructional
goal. If not, the teacher adjusts the instructional program to better meet the student’s needs and
continues to monitor progress. This process recurs throughout intervention to formatively develop an
effective, individually tailored instructional program. In the Center and in the document, we refer to
this use of progress monitoring as data-based individualization.
                             Please Read Before You Start
Q1.   My tool does not have enough alternate forms to conduct progress monitoring at
      least weekly, as required for data-based individualization. Can I still submit my tool
      for review?

         No. The TRC will only review tools that meet its definition of progress monitoring, as follows:

          The National Center on Intensive Intervention defines progress monitoring as repeated
          measurement of academic performance for the purpose of helping schools individualize
          instructional programs for students in grades K-12 who have intensive instructional needs.
          For this purpose, progress monitoring is collected weekly to assess whether student progress
          is adequate to meet the student’s instructional goal. If not, the teacher adjusts the instructional
          program to better meet the student’s needs and continues to monitor progress. This process
          recurs throughout intervention to formatively develop an effective, individually tailored
          instructional program. In the Center and in the document, we refer to this use of progress
          monitoring as data-based individualization.

          Because this definition requires that progress monitoring be conducted at least weekly, the
          TRC requires that tools submitted for review have at least 20 alternate forms. Please
          include with this submission documentation of at least 20 alternate forms. The Center will not
          distribute your protocol for review unless you demonstrate evidence of having at least 20
          alternate forms. If your tool is computer delivered, please note how the test forms are derived
          instead of providing alternate forms.


Q2.   My progress monitoring tool assesses multiple domains of academic performance
      (e.g., reading vs. mathematics or math computation vs. concepts/applications). Do
      I need a separate protocol for each domain?

         Yes. The Center recognizes that for products designed to measure progress in multiple
          academic domains, some of the information to be submitted in the protocol will be the same.
          However, the tool for each academic domain or subcomponent within a domain will be
          evaluated and reported separately on our tools chart. Therefore, if your tool assesses more
          than one domain/subcomponent, you MUST submit separate protocols for EACH
          domain/subcomponent. For example, if your tool measures sub-components of reading, such
          as letter name fluency, letter sound fluency, and passage reading fluency, you must submit a
          separate protocol for each.


Q3.   The protocol requires information that is already included in a technical report or
      research study. Can I submit this study instead of filling out the protocol?

         No. Technical reports and relevant research papers may be submitted as supporting
          information, but you MUST COMPLETE THE FULL PROTOCOL. Reviewers will use the
          information in the protocol to make their judgments. They are not expected to search for and
          find additional information in accompanying materials.
Q4.   The protocol requires information that is not currently available. Can I still submit
      my progress-monitoring tool?

         Yes. The Protocol for Evaluating Progress Monitoring Tools is designed to collect
          comprehensive and detailed information on the submitted progress-monitoring tools to ensure
          rigorous evaluation of tools. Therefore, tools that are undergoing improvements or are in an
          early phase of development may not have all the information the protocol asks for. Please
          provide as much information available as possible.

         If it is found that your submission packet needs substantial amount of supplemental
          information or is missing critical information, the entire packet will be returned to you. A
          revised protocol packet with additional information may be re-submitted.


Q6.    Can I submit a tool to be evaluated for BOTH General Outcome Measurement
      (GOM) and Mastery Measurement (MM)?

         Yes. However, fill out BOTH GOM and MM sections in the protocol. Also, provide TWO
          separate sets of evidence – one set which addresses the standards for GOM and another set
          which addresses the standards for MM.


Q7.   I am not familiar with some of the terms in the protocol, and thus, I am not sure
      what information I should provide. What should I do?

         Center staff are available to answer your questions or to assist you in completing the protocol
          for submission. Please contact the National Center on Intensive Intervention:

                     National Center on Intensive Intervention
                     American Institutes for Research
                     1000 Thomas Jefferson Street, NW
                     Washington, DC 20007
                     E-mail: NCII@air.org
                     Phone: 866-577-5787


Q8.   Can I withdraw my tool from the review process?

         No. Results of the review will be posted on the Center’s website, in the Progress
          Monitoring Tools Chart. Once the review has begun, withdrawal from the process
          is not permitted.
                             SECTION I: BASIC INFORMATION

A. TOOL
        Title: Click here to enter text.
        Developer: Click here to enter text.
        Publisher: Click here to enter text.
        Publication Date: Click here to enter text.
        Contact Person: Name: Click here to enter text.
                            Telephone: Click here to enter text.
                            E-mail Address: Click here to enter text.



B. DESCRIPTIVE INFORMATION
Description of tool: Click here to enter text.


        1. The tool is intended for use in grade(s) (Check all that apply):

                    Kindergarten                 Fifth Grade            Tenth Grade

                    First Grade                  Sixth Grade            Eleventh Grade

                    Second Grade                 Seventh Grade          Twelfth Grade

                    Third Grade                  Eighth Grade

                    Fourth Grade                 Ninth Grade


        2. The tool assesses one or more of the following dimensions (Check all that apply):
           □ READING

               Global Indicator of Reading Competence                         Decoding

               Listening Comprehension                                        Passage Reading

               Vocabulary                                                      Word Identification

               Phonemic Awareness                                             Comprehension

               Other:   Click here to enter text.

           □ MATHEMATICS
               Global Indicator of Mathematics Competence                     Mathematics Computation
               Early Numeracy                                                 Mathematic Application

               Mathematics Concepts                                           Fractions

                                                                              Algebra
               Other:   Click here to enter text.


                                                                                                        1
                              SECTION I: BASIC INFORMATION
           □ SPELLING ACHIEVEMENT

              Global Indicator of Spelling Competence

              Other (List specific skills or subtests): Click here to enter text.


           □ WRITTEN EXPRESSION

              Global Indicator of Written Expression
              Competence
              List specific dimension, skills, or subtests:
                                                              Click here to enter text.


           □ OTHER

              List specific skills or subtests:
                                                  Click here to enter text.


 Notes:
 Click here to enter text.




The tool provides information on student performance in:

          English          Spanish       Other    Click here to enter text.
Acquisition Information:
  Where to Obtain: Click here to enter text.
  Address Click here to enter text.
  Phone: Click here to enter text.
  Web Site: Click here to enter text.


  Cost per student for Year 1: Click here to enter text.
          Including:
          $ Click here Complete Kit (describe contents)Click here to enter text.
          $ Click here Manuals and Test Materials Click here to enter text.
          $ Click here Directions for Administration Click here to enter text.
          $ Click here Test Forms
          $ Click here Technical Manuals
          $ Click here Protocol per Student
          Other (Describe below):
                 $ Click here
                 $ Click here
                                                                                          2
                                SECTION I: BASIC INFORMATION
                 $ Click here
                 $ Click here


  Cost per student for subsequent years: Click here to enter text.
          Including:
          $ Click here Complete Kit (describe contents) Click here to enter text.
          $ Click here Manuals and Test Materials Click here to enter text.
          $ Click here Directions for Administration Click here to enter text.
          $ Click here Test Forms
          $ Click here Technical Manuals
          $ Click here Protocol per Student


          Other (Describe below):
                 $ Click here
                 $ Click here


          Online Costs:
                       $ Click here Start-up
                       $ Click here Per Student cost
                       Other (Describe below):
                             $ Click here
                             $ Click here




Additional Information on the tools:
Describe basic pricing plan and/or structure of the tools. Also, provide information on what is included in the
published tools, including information about special accommodations for students with disabilities.
Click here to enter text.




                                                                                                                  3
         SECTION II: DEVELOPMENT AND ADMINISTRATION
A. TIME, ADMINISTRATION, AND FREQUENCY
   Assessment format (Check all that apply):

                         Individual        Group       Computer-administered

   Administration time: Click here minutes
   Additional scoring time: Click here minutes
   Discontinue Rules:

          No

          Yes:    Basal    Ceiling   Other:
                  s        s                  Click here to enter text.
   How many alternate forms available? (Please attach each alternate form.)
   # of forms per grade, test level, or unit: Click here


B. TRAINING
   Time required for training tester:

                        Less than 1 hour of training         1-4 hours of training

                        4-8 hours of training                Information not available

   Minimum qualifications of the examiner:

                        Professional                         Paraprofessional

                        Information not available

   Training manuals and materials available:

                        Yes        No

   Training manuals/materials field-tested:

                        Yes        No

   Training manuals/materials included in cost of tool:

                         Yes          No

   Sources for ongoing technical support available:

                        Yes        No

                        If yes please explain:
                                                 Click here to enter text.




                                                                                         4
           SECTION II: DEVELOPMENT AND ADMINISTRATION
C. SCORING
Types of scores available:
     1. Do you provide scores on level performance?

                         Yes          No

                       If yes, please check all that apply:

                             raw score                normal curve equivalents         composite scores
                             standard score           stanines                         error analysis

                             percentile score         developmental benchmarks         grade equivalents
                             IRT-based score          subscale/subtest
                                                      scores
                             other (specify):
                                                Click here to enter text.

             Basis for calculating level standard & percentile scores:

                             age norms           grade norms        stanines     normal curve equivalents


             Scoring Structure (specify how raw scores are calculated and what comprises cluster/composite
             score): Click here to enter text.


     2. Do you provide the basis for calculating slope? (i.e. amount of improvement per unit in time)

                             Yes       No

         Do you provide benchmarks for the slopes?

                             Yes       No

         Do you provide percentile ranks for the slopes?

                             Yes       No


         Basis for calculating slope standard & percentile scores:

                             age norms          grade norms         stanines     normal curve equivalents


         Scoring Structure (specify how raw scores are calculated and what comprises cluster/composite score):
         Click here to enter text.




                                                                                                                 5
            SECTION II: DEVELOPMENT AND ADMINISTRATION

Describe the tool’s approach to progress monitoring, behavior samples, test format, and/or scoring practices.
Click here to enter text.




Describe the tool’s methods for avoiding cultural or linguistic bias.
Click here to enter text.




                                                                                                                6
                       SECTION III: TECHNICAL INFORMATION

What approach to progress monitoring is used?
        General outcome measurement (Complete GOM 1-12.)

               With GOM, alternate forms of the progress monitoring are of comparable difficulty or IRT-based
               methods are used with each testing sampling the construct in the same way. With GOM, progress
               toward a year-end goal is monitored.

        Mastery Measurement (Complete MM 1-9.)

               With MM, the objectives are targeted for mastery changes. That is, criterion-referenced
               assessment on an objective continues with alternate forms of a test (each test has one type of item
               on it) until mastery is achieved. Then, a new objective (the next one in the sequence) is targeted for
               monitoring, etc.


        General outcome measurement and Mastery Measurement (Complete BOTH GOM 1-12 and MM 1-MM9.)

        Please provide TWO separate sets of evidence – one set which addresses the standards for GOM and
        another set which addresses the standards for MM.


General Outcome Measures


A.   Psychometric Standards
     GOM 1. In the chart below, report the reliability of the performance level score (e.g., internal
            consistency, stability, test-retest reliability, alternate form reliability).
          Type of         Age or       n           Coefficient         SEM      Information (including
         Reliability      Grade     (range)                                     normative data) / Subjects
                                                range      median




     Manual cites other published reliability studies:    Yes    No
     *Provide citations for additional published studies.



                                                                                                                   7
                  SECTION III: TECHNICAL INFORMATION
GOM 1a. Do you provide, in your user’s manual, reliability for the performance level score that is
        disaggregated by race/ethnicity? If so, complete below for each race/ethnicity for which you
        provide disaggregated reliability for the performance level score data.

     Type of         Age or       n           Coefficient               Information (including normative
                                                                 SEM
    Reliability      Grade     (range)     range      median                    data) / Subjects




Manual cites other published reliability studies:    Yes    No
*Provide citations for additional published studies.


GOM 2. In the chart below, report reliability for the slope (e.g., ratio of true slope variance to total
       slope variance) by grade level (if relevant).

     Type of         Age or       n           Coefficient               Information (including normative
                                                                 SEM
    Reliability      Grade     (range)     range      median                    data) / Subjects




Manual cites other published reliability studies:    Yes    No
*Provide citations for additional published studies.




                                                                                                           8
                  SECTION III: TECHNICAL INFORMATION
GOM 2a. Do you provide, in your user’s manual, reliability for the slope that is disaggregated by
        race/ethnicity? If so, complete below for each race/ethnicity for which you provide
        disaggregated reliability for the slope data.

     Type of            Age or      n          Coefficient                     Information (including normative
                                                                       SEM
    Reliability         Grade    (range)    range       median                         data) / Subjects




Manual cites other published reliability studies:      Yes       No

*Provide citations for additional published studies.


GOM 3.      In the chart below, report validity information for the performance level score (e.g., content,
            concurrent, predictive, and/or construct***)

 Type of       Age or         Test or         n              Coefficient       Information (including normative
 Validity      Grade         Criterion     (range)      range         median           data) / Subjects




Other forms of validity: Click here to enter text.

Manual cites other published reliability studies:    Yes         No
*Provide citations for additional published studies.
*** Validity information may also include: evidence based on response processes, evidence based on internal
structure, evidence based on relations to other variables, and/or evidence based on consequences of testing.
NOTE: To support validity, the TRC prefers and strongly encourages criterion measures that are external to
the progress monitoring system. If internal measures are used, please include a description of what
provisions have been taken to address the limitations of this method, such as possible method variance or
overlap of item samples: Click here to enter text.

                                                                                                              9
                 SECTION III: TECHNICAL INFORMATION

GOM 3a. Do you provide, in your user’s manual, validity for the performance level score that is
        disaggregated by race/ethnicity? If so, complete below for each race/ethnicity for which you
        provide disaggregated validity for the performance level score data.

 Type of       Age or       Test or          n           Coefficient      Information (including normative
 Validity      Grade       Criterion      (range)      range    median            data) / Subjects




Other forms of validity: Click here to enter text.

Manual cites other published reliability studies:    Yes       No
*Provide citations for additional published studies.
*** Validity information may also include: evidence based on response processes, evidence based on internal
structure, evidence based on relations to other variables, and/or evidence based on consequences of testing.
NOTE: To support validity, the TRC prefers and strongly encourages criterion measures that are external to
the progress monitoring system. If internal measures are used, please include a description of what
provisions have been taken to address the limitations of this method, such as possible method variance or
overlap of item samples: Click here to enter text.


GOM 4. In the chart below, report predictive validity information for the slope of improvement
       (correlation between the slope and achievement outcome).
            **Please note, the TRC suggests controlling for initial level when the correlation for slope
            without such control is not adequate.

 Type of      Age or       Test or         n           Coefficient        Information (including normative
 Validity     Grade       Criterion     (range)      range     median              data)/Subjects




Other forms of validity: Click here to enter text.

                                                         Yes         No
Manual cites other published validity studies:
*Provide citations for additional published studies.
NOTE: To support validity, the TRC prefers and strongly encourages criterion measures that are external to
the progress monitoring system. If internal measures are used, please include a description of what

                                                                                                             10
                       SECTION III: TECHNICAL INFORMATION
     provisions have been taken to address the limitations of this method, such as possible method variance or
     overlap of item samples: Click here to enter text.


     GOM 4a. Do you provide, in your user’s manual, predictive validity information for the slope of
             improvement that is disaggregated by race/ethnicity? If so, complete below for each
             race/ethnicity for which you provide disaggregated predictive validity for the slope of
             improvement data.

      Type of      Age or        Test or         n          Coefficient         Information (including normative
      Validity     Grade        Criterion     (range)     range    median                data)/Subjects




     Other forms of validity: Click here to enter text.

     Manual cites other published reliability studies:    Yes      No
     *Provide citations for additional published studies.
     NOTE: To support validity, the TRC prefers and strongly encourages criterion measures that are external to
     the progress monitoring system. If internal measures are used, please include a description of what
     provisions have been taken to address the limitations of this method, such as possible method variance or
     overlap of item samples: Click here to enter text.




B.   PROGRESS MONITORING STANDARDS
     GOM 5. Provide evidence that alternate forms are of equal and controlled difficulty or if IRT based
            please provide evidence of item or ability invariance (attach documentation of direct
            evidence).


                 What is the number of alternate forms of equal and controlled difficulty? Click here


                 If IRT based, provide evidence of item or ability invariance (attach documentation): Click here

                 If computer administered, how many items are in the item bank for each grade level? Click here


                 If your tool is computer administered, please note how the test forms are derived instead of
                 providing alternate forms: Click here


     GOM 6. Is minimum acceptable growth (slope of improvement or average weekly increase in score
            by grade level) specified in your manual or published materials?

                                                                                                                   11
               SECTION III: TECHNICAL INFORMATION
            Yes          No

         Specify the growth standards: Click here to enter text.
         What is the basis for specifying minimum acceptable growth?

            Norm-referenced        Criterion-referenced

            Other (please describe):
                                        Click here to enter text.


GOM 6a. If norm-referenced, describe the normative profile:
         NOTE: If your tool is posted on our online tools chart, the Center will make the information
         you provide below on the normative profile available to consumers.
         National representation?                         Local representation?

             Yes         No                                  Yes          No

         Date: Click here to enter text.
         Number of States: Click here to enter text.
         Size: Click here to enter text.
         Regions: Click here to enter text.

         Gender (Percent):                  Male              Female               Unknown
         SES (Percent, check all reported)

                  Low            Parents did not graduate high school

                  Middle         Parents graduated high school

                  High           Parents had 1-3 years of college

                                 Parents had 4 or more years of college

         Other SES Indicators: Click here to enter text.
         Race/Ethnicity (Percent):

                     White, Non-Hispanic                  American Indian/Alaska Native

                     Black, Non-Hispanic                  Asian/Pacific Islander

                     Hispanic                             Other

                     Unknown
         ELL (Percent): Click here to enter text.
         Disability classification (Percent): Click here to enter text.


GOM 6b. Do you provide, in your user’s manual, norms which are disaggregated by race or ethnicity?
        If so, for which race/ethnicity? (Check all that apply)
         Race/Ethnicity:
                                                                                                    12
               SECTION III: TECHNICAL INFORMATION
               White, Non-Hispanic         American Indian/Alaska Native

               Black, Non-Hispanic         Asian/Pacific Islander

               Hispanic                    Other

               Unknown

GOM 6c. If criterion-referenced, describe procedure for specifying criterion for adequate growth
        (attach documentation): Click here to enter text.


GOM 6d. Please describe other procedures for specifying adequate growth: Click here to enter text.


GOM 7. Are benchmarks for minimum acceptable end-of-year performance specified in your manual
       or published materials?

                      Yes        No

         Specify the end-of-year performance standards: Click here to enter text.
         What is the basis for specifying minimum acceptable end-of-year performance?

                       Norm-referenced        Criterion-referenced

                       Other (please describe):
                                                   Click here to enter text.


         Specify the benchmarks: Click here to enter text.
         What is the basis for specifying these benchmarks?

                       Norm-referenced        Criterion-referenced

                       Other (please describe):
                                                   Click here to enter text.


GOM 7a. If norm-referenced, describe the normative profile:

         If same as GOM 2, check here:        . Otherwise, complete below.


         National representation?                        Local representation?

             Yes       No                                   Yes       No

         Date: Click here to enter text.
         Number of States: Click here to enter text.
         Size: Click here to enter text.
         Regions: Click here to enter text.



         Gender (Percent):                 Male              Female              Unknown

                                                                                                     13
                    SECTION III: TECHNICAL INFORMATION
              SES (Percent, check all reported)

                       Low           Parents did not graduate high school

                       Middle        Parents graduated high school

                       High          Parents had 1-3 years of college

                                     Parents had 4 or more years of college

              ELL (Percent) Click here to enter text.
              Disability classification (Percent) Click here to enter text.


     GOM 7b. If criterion-referenced, describe procedure for specifying benchmarks for end-of-year
             performance levels (attach documentation): Click here to enter text.


     GOM 7c. Please describe any other procedures for specifying minimal acceptable end-of-year
             performance: Click here to enter text.


     GOM 8. Describe evidence that the monitoring system produces data that are sensitive to student
            improvement (i.e., when student learning actually occurs, student performance on the
            monitoring tool increases on average): Click here to enter text.




C.   DATA-BASED INDIVIDUALIZATION STANDARDS

     GOM 9. Does your manual or published materials specify validated decision rules for when changes
            to instruction need to be made?

                 Yes          No


              Specify the decision rules: Click here to enter text.
              What is the evidentiary basis for these decision rules? Click here to enter text.


     GOM 10. Does your manual or published materials specify validated decision rules for when changes
             to increase goals?

                 Yes          No


              Specify the decision rules: Click here to enter text.
              What is the evidentiary basis for these decision rules? Click here to enter text.


     GOM 11. Describe evidence that teachers’ use of the tool results in improved student achievement.
             For this GOM, please describe and attach an empirical study that provides this evidence.
              Study Citation: Click here to enter text.

                                                                                                         14
                SECTION III: TECHNICAL INFORMATION

GOM 11a.Study Sample
          Number of students in product condition: Click here to enter text.
          Number of students in control condition: Click here to enter text.
          Describe characteristics of students in sample and how they were selected for participation in
          study: Click here to enter text.


GOM 11b. Study Design
           Was random assignment used?

                        Yes       No

           If not, please specify the type of design: Click here to enter text.
           What was the unit of assignment?

                     Schools                Teachers                 Students
           What was the unit of analysis?

                     Schools                 Teachers                 Students
           What was the duration of product implementation? Click here to enter text.
           Describe analysis: Click here to enter text.


GOM 11c. Study Fidelity
           Describe when and how fidelity of treatment information was obtained:
           Click here to enter text.


           What were the results on the fidelity of treatment implementation measure:
           Click here to enter text.


GOM 11d. Study Measures
In the table below, please list external outcome measures used in the study, along with psychometric
properties (e.g., Cronbach’s alpha, IRT reliability, temporal stability, inter-rater)
 Measure name         Reliability statistics (specify type of reliability, e.g. Cronbach’s alpha, IRT
                      reliability, temporal stability, inter-rater)




GOM 11e. Study Results
           Describe the results of the study:
           Click here to enter text.


                                                                                                           15
               SECTION III: TECHNICAL INFORMATION

Report effect sizes for each outcome measure:
 Measure name                  Effect size




         Summarize conclusions and explain conditions to which effects should be generalized:
         Click here to enter text.


         Other related references or information:
         Click here to enter text.


GOM 12. Describe evidence that teachers’ use of the tool results in improved planning. For this GOM,
        please describe and attach an empirical study that provides this evidence.
         Study Citation: Click here to enter text.


GOM 12a. Study Sample
          Number of students in product condition: Click here to enter text.
          Number of students in control condition: Click here to enter text.
          Describe characteristics of students in sample and how they were selected for participation in
          Study: Click here to enter text.


GOM 12b. Study Design
          Was random assignment used?

                       Yes       No

          If not, please specify the type of design: Click here to enter text.
          What was the unit of assignment?

           Schools                   Teachers              Students
          What was the unit of analysis?

          Schools                Teachers                 Students
          What was the duration of product implementation? Click here to enter text.
          Describe analysis: Click here to enter text.


GOM 12c. Study Fidelity
          Describe when and how fidelity of treatment information was obtained:
           Click here to enter text.
                                                                                                           16
                SECTION III: TECHNICAL INFORMATION

           What were the results on the fidelity of treatment implementation measure:
           Click here to enter text.


GOM 12d. Study Measures
In the table below, please list external outcome measures used in the study, along with psychometric
properties (e.g., Cronbach’s alpha, IRT reliability, temporal stability, inter-rater)
 Measure name                   Reliability statistics (specify type of reliability, e.g. Cronbach’s alpha, IRT
                                reliability, temporal stability, inter-rater)




GOM 12e. Study Results
           Describe the results of the study: Click here to enter text.
Report effect sizes for each outcome measure:
 Measure name                             Effect size




           Summarize conclusions and explain conditions to which effects should be generalized:
           Click here to enter text.


           Other related references or information: Click here to enter text.




                                                                                                             17
                     SECTION III: TECHNICAL INFORMATION
Mastery Measures
A.   Psychometric Standards

     MM 1.      Report reliability coefficients for the tests incorporated in the system:

       Type of      Age or        n         Coefficient                Information (including normative data) /
                                                                SEM
      Reliability   Grade      (range)   range     median                              Subjects




     Manual cites other published reliability studies:    Yes     No
     *Provide citations for additional published studies.

     How many items comprise a single test?


     MM 1a.     Do you provide, in your user’s manual, reliability coefficients for the tests incorporated in
                the system that are disaggregated by race/ethnicity? If so, complete below for each
                race/ethnicity for which you provide disaggregated reliability data.

       Type of      Age or       n         Coefficient                  Information (including normative data) /
                                                                SEM
      Reliability   Grade     (range)    range    median                                Subjects




     Manual cites other published reliability studies:    Yes     No
     *Provide citations for additional published studies.



                                                                                                                   18
                 SECTION III: TECHNICAL INFORMATION
MM 2.       What is the correlation between number of skills mastered over the course of an academic
            year with an important end-of-year outcome? Report these validity coefficients:
  Type                                                 Coefficient
              Age or      Test or          n                          Information (including normative data) /
   of
              Grade       Criterion     (range)      range   median                   Subjects
 Validity




Other forms of validity: Click here to enter text.

Manual cites other published reliability studies:    Yes      No
*Provide citations for additional published studies.
*** Validity information may also include: evidence based on response processes, evidence based on internal
structure, evidence based on relations to other variables, and/or evidence based on consequences of testing.
NOTE: To support validity, the TRC prefers and strongly encourages criterion measures that are external to
the progress monitoring system. If internal measures are used, please include a description of what
provisions have been taken to address the limitations of this method, such as possible method variance or
overlap of item samples: Click here to enter text.


MM 2a.      Do you provide, in your user’s manual, validity coefficients that are disaggregated by
            race/ethnicity? If so, complete below for each race/ethnicity for which you provide
            disaggregated validity data.
  Type                                                 Coefficient
              Age or      Test or           n                         Information (including normative data) /
   of
              Grade       Criterion      (range)     range   median                   Subjects
 Validity




Other forms of validity: Click here to enter text.

Manual cites other published reliability studies:    Yes      No
*Provide citations for additional published studies.
*** Validity information may also include: evidence based on response processes, evidence based on internal
structure, evidence based on relations to other variables, and/or evidence based on consequences of testing.
NOTE: To support validity, the TRC prefers and strongly encourages criterion measures that are external to
the progress monitoring system. If internal measures are used, please include a description of what
provisions have been taken to address the limitations of this method, such as possible method variance or
overlap of item samples: Click here to enter text.

                                                                                                            19
                    SECTION III: TECHNICAL INFORMATION


B.   PROGRESS MONITORING STANDARDS

     MM 3.    Type of evidence for skill sequence (instructional hierarchy) on which MM system is based:

                 Logical

                     Describe Logic: Click here to enter text.

                 Expert judgment

                     Describe procedure for deriving these judgments: Click here to enter text.

                 Tied to curriculum

                     Describe evidence that this curriculum/program is research validated; attach journal
                     articles): Click here to enter text.


     MM 4.    Describe evidence that the monitoring system produces data that are (a) sensitive to
              children’s development of academic competence in this area and/or (b) sensitive to the
              effects of effective interventions (attach journal articles).
              Click here to enter text.


     MM 5.    What is the basis for defending the pass/fail (mastered/nonmastered) decisions in the
              system:
              Click here to enter text.
              Describe the sensitivity/specificity of these pass/fail decisions: Click here to enter text.
              How are false negatives and false positives assessed for the benchmarks? (Criterion and
              grade/age) Click here to enter text.


                                                Negative                 Positive

                              False
                               True


              EVIDENCE OF SENSITIVITY
              Odds ratios and conditional probabilities if given: Click here to enter text.


     MM 5A.   Do you provide, in your user’s manual, sensitivity and specificity data that are
              disaggregated by race/ethnicity? If so, complete below for each race/ethnicity for which you
              provide disaggregated sensitivity and specificity data.
                                                Negative                 Positive

                              False
                               True


                                                                                                             20
                SECTION III: TECHNICAL INFORMATION
          RELIABILITY OF DECISIONS
          Calculations:
          Specificity: TN / (TN + FP) = Click here to enter text.
          Sensitivity: TP / (TP + FN) = Click here to enter text.
          Hit rate: (TP + TN) / N = Click here to enter text.




C. DATA-BASED INDIVIDUALIZATION STANDARDS

 MM 6.    Does your manual or published materials specify validated decision rules for when changes
          to instruction need to be made?

                 Yes       No

          Specify the decision rules here: Click here to enter text.
          What is the evidentiary basis for these decision rules? Click here to enter text.


 MM 7.    Does your manual or published materials specify decision rules for when to increase goals?

                 Yes       No

          Specify the decision rules here:     Click here to enter text.
          What is the evidentiary basis for these decision rules?    Click here to enter text.

 MM 8.    Describe evidence that teachers’ use of the tool results in improved student achievement.
          For this GOM, please describe and attach an empirical study that provides this evidence.
          Study Citation: Click here to enter text.


 MM 8a.   Study Sample
          Number of students in product condition: Click here to enter text.
          Number of students in control condition: Click here to enter text.
          Describe characteristics of students in sample and how they were selected for participation in
          study: Click here to enter text.


 MM 8b.   Study Design
          Was random assignment used?

                 Yes       No

          If not, please specify the type of design: Click here to enter text.


          What was the unit of assignment?

          Schools                Teachers                 Students

                                                                                                           21
                SECTION III: TECHNICAL INFORMATION

          What was the unit of analysis?

          Schools               Teachers                 Students


          What was the duration of product implementation? Click here to enter text.


          Describe analysis: Click here to enter text.


MM 8c.    Study Fidelity
          Describe when and how fidelity of treatment information was obtained:
          Click here to enter text.

          What were the results on the fidelity of treatment implementation measure:
          Click here to enter text.

MM 8d.    Study Measures
In the table below, please list external outcome measures used in the study, along with psychometric
properties (e.g., Cronbach’s alpha, IRT reliability, temporal stability, inter-rater)
Measure name                               Reliability statistics (specify type of reliability, e.g. Cronbach’s
                                           alpha, IRT reliability, temporal stability, inter-rater)




MM 8e.    Study Results
          Describe the results of the study:   Click here to enter text.
Report effect sizes for each outcome measure:
Measure name                               Effect size




          Summarize conclusions and explain conditions to which effects should be generalized:
          Click here to enter text.

          Other related references or information:
                                                                                                              22
                 SECTION III: TECHNICAL INFORMATION
           Click here to enter text.

MM 9.      Describe evidence that teachers’ use of the tool results in improved planning. For this MM,
           please describe and attach an empirical study that provides this evidence.
           Study Citation:   Click here to enter text.

MM 9a.     Study Sample
           Number of students in product condition: Click here to enter text.
           Number of students in control condition: Click here to enter text.

           Describe characteristics of students in sample and how they were selected for participation in
           Study: Click here to enter text.


MM 9b.     Study Design
           Was random assignment used?

              Yes       No

           If not, please specify the type of design: Click here to enter text.


           What was the unit of assignment?

           Schools                Teachers                Students


           What was the unit of analysis?

           Schools                Teachers                Students


           What was the duration of product implementation?: Click here to enter text.


           Describe analysis: Click here to enter text.


MM 9c.     Study Fidelity
           Describe when and how fidelity of treatment information was obtained:
           Click here to enter text.


           What were the results on the fidelity of treatment implementation measure:
           Click here to enter text.


MM 9d.     Study Measures
In the table below, please list external outcome measures used in the study, along with psychometric
properties (e.g., Cronbach’s alpha, IRT reliability, temporal stability, inter-rater)

                                                                                                            23
                SECTION III: TECHNICAL INFORMATION
Measure name                             Reliability statistics (specify type of reliability, e.g. Cronbach’s
                                         alpha, IRT reliability, temporal stability, inter-rater)




MM 9e.    Study Results
          Describe the results of the study:   Click here to enter text.
Report effect sizes for each outcome measure:
Measure name                             Effect size




          Summarize conclusions and explain conditions to which effects should be generalized:
           Click here to enter text.

          Other related references or information:
          Click here to enter text.




                                                                                                            24

								
To top