The Process of Developing Performance Assessments

Document Sample
The Process of Developing Performance Assessments Powered By Docstoc
					            Enriching
           Assessment
           of the Core
Albert Oosterhof, Faranak Rohani, & Penny J. Gilmer
               Florida State University
Center for Advancement of Learning and Assessment
      Issues to Be Addressed

• Significant science competencies remain
  excluded from statewide assessments.
• Assessing samples of students could help
  expand types of science competencies
  assessed.
• Formative assessments are an integral part of
  the larger assessment picture.



                    cala.fsu.edu/ies
      Issues to Be Addressed

• Significant science competencies remain
  excluded from statewide assessments.
• Assessing samples of students could help
  expand types of science competencies
  assessed.
• Formative assessments are an integral part of
  the larger assessment picture.



                    cala.fsu.edu/ies
       Examples of Competencies
          That Go Unassessed

Nature of Science
• Student can formulate a scientifically testable question(s)
  that relates to the context or data provided.
• Student can create a plan for carrying out a scientific
  investigation, including what, when, and how to measure
  variables.
• Student can make inferences and predictions and use
  the data to defend or refute [the] conclusion.


                         cala.fsu.edu/ies
       Examples of Competencies
          That Go Unassessed

Life Science
• Student can observe and describe a local ecosystem.
• Student can determine potential limiting factors for
  specified populations in a local ecosystem.
• Student can explain why a species must continue to
  adapt when its environment changes in order for that
  species to continue to have an advantage.



                        cala.fsu.edu/ies
         Assessment Format

• Competencies such as those we listed can be
  assessed, just not with multiple-choice and other
  item formats that can be efficiently administered
  and scored.
• Efficient item formats, when well constructed,
  can measure important competencies, just not
  competencies such as those we listed.
• These more complex competencies require the
  use of performance assessments and similar
  formats.
                     cala.fsu.edu/ies
 Strategy We Are Investigating

• Teachers and the state education agency both
  develop assessments of complex competencies.
• The state’s assessments are administered to
  carefully selected samples to establish proficiency of
  students at the group level.
• Teachers establish the proficiency of individual
  students.
• Scores on assessments developed by teachers and
  the state agency are used to help cross-validate
  each other.
                       cala.fsu.edu/ies
Participants




   cala.fsu.edu/ies
               Specification

• Competency
• Evidence
• Example Task
• Scoring Plan for the Example Task
• Procedure for Creating Parallel Tasks
• Scoring Plan for the Parallel Tasks



                     cala.fsu.edu/ies
                  Specification for Assessment #1
Formulating a Scientifically Testable Question

• Competency
  Student can formulate a scientifically testable question(s) that relates to the
  context or data provided.

• Evidence
  Students are provided a short list of variables that are related to a scientific topic or
  concept with which the students are familiar. The variables are listed without reference to
  how they could be observed, measured, or quantified. Approximately half of the variables
  are objectively measurable while the other variables are less objectively measurable.
  Students are asked to create and write down a scientifically testable research question
  that relates two of these variables in some way. To facilitate scoring, students also are
  asked to explain how each variable could be observed or measured. If a student selects a
  variable that is less objectively measurable, the student must describe a specific process
  to quantify observations of the variable that clearly is both objective and appropriate. A
  student’s performance is scored with respect to whether a scientifically testable question is
  provided that relates two of the listed variables.



                                     cala.fsu.edu/ies
                  Specification for Assessment #1
Formulating a Scientifically Testable Question

• Example Task
  In this exercise, you will create a                  Choose two variables from
  scientifically testable question related to          the list that you think might
  behaviors and other characteristics of flowers.      have a relationship with one
  Use two of the following seven variables             another. Only pick variables
  when you create your scientifically testable         you (or another scientist)
  question:                                            could actually observe or
  – How beautiful a flower looks                       measure scientifically.
  – How often insects (pollinators) visit a flower     Explain how each variable
  – How many flowers there are in a certain area       could be observed or
  – How much nectar a flower produces                  measured.
  – How much a person enjoys the smell of a
    flower                                             Write a scientifically
  – How much time insects spend visiting a             testable question that relates
    flower                                             your two variables to each
  – How much familiarity a person has with a           other.
    flower

                                    cala.fsu.edu/ies
         Specification for Assessment #1
Formulating a Scientifically Testable Question




                   cala.fsu.edu/ies
              Specification for Assessment #1
  Formulating a Scientifically Testable Question

Procedure for Creating Parallel Tasks




                         cala.fsu.edu/ies
             Specification for Assessment #1
 Formulating a Scientifically Testable Question

Scoring Plan for Parallel Tasks




                        cala.fsu.edu/ies
Goals of Project




     cala.fsu.edu/ies
                 Goals of Project

• Develop example specifications relevant to science
  competencies not measurable with present statewide
  assessments.
• Determine if teachers and an external agent can independently
  develop comparable assessments based on these specifications.
• Using a generalizability theory, obtain estimates of score
  reliability.
• Examine the formative use of these assessments including
  teachers’ development of learning progressions.
  Note: Our project is not investigating the use of sampling, but
  instead is addressing issues highly relevant to the assessment
  model we propose.
                            cala.fsu.edu/ies
Some Preliminary Results Related
      to Generalizability
    G-coefficients with Design of Rater : Agent




                   cala.fsu.edu/ies
 Some Preliminary Results Related
        to Comparability
Frequency Distributions of Average Scores Assigned by Raters
           Formative Assessment
                   Black and Wiliams (1998)

“We start from the self-evident proposition that
teaching and learning must be interactive. Teachers
need to know about their pupils’ needs – needs that
are often unpredictable and that vary from one pupil to
another.”

Black, P., & Wiliam, D. (1998). Inside the black box: Raising standards through
classroom assessment. Phi Delta Kappan, 80(2), 139-148.




                                cala.fsu.edu/ies
         Formative Assessment
                 Some Characteristics

• Occurs in real-time during learning to redirect
  instructional activities so as to improve learning.
• Is a package deal that involves substantially more than
  administering and scoring tests.
• Includes feedback, not only teacher to student, but also
  vice versa and student to student.
• Involves teacher and student activities that are
  responsive to this feedback.
• Proceeds through a deliberate learning progression.


                         cala.fsu.edu/ies
           Formative Assessment
                     Learning Progression

Popham (2007) defines learning progressions as a
“carefully sequenced set of building blocks that
students must master en route to a more distant
curricular aim. The building blocks consist of sub skills
and bodies of enabling knowledge” (p. 83).

Popham, J. W. (April 2007). The lowdown on learning progressions.
Educational Leadership, 64(7), 83-84.




                               cala.fsu.edu/ies
            Formative Assessment
                       Learning Progression

Heritage (2008) states that a “well-constructed learning
progression presents a number of opportunities to
teachers for instructional planning. It enables teachers
to focus on important learning goals in the domain,
centering their attention on what the student will learn
rather than what the student will do (i.e., the learning
activity)” (p. 5).

Heritage, M. (2008). Learning progressions: Supporting instruction and formative
assessment. Washington, DC: A paper published by the Council of Chief School
Officers.


                                  cala.fsu.edu/ies
Formative Assessment
      Worksheet
Formative Assessment
      Worksheet
Formative Assessment
      Worksheet
        Formative Assessment
                       Feedback

Black, P., & Wiliam, D. (1998). Assessment and classroom
learning. Assessment in Education, 5(1), 7-74.

Shute, V. J. (2008). Focus on formative feedback. Review
of Educational Research, 78(1), 153-189.




                       cala.fsu.edu/ies
    Interactions with Teachers

• Summer training workshop
• Follow-up workshop
• Review of teachers’ formative materials
• Class visitations
• Administration of external summative
  assessments



                      cala.fsu.edu/ies
Enriching assessments is a critical component to
enriching learning.




                    cala.fsu.edu/ies
Center for Advancement of
Learning and Assessment
 Florida State University

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:1
posted:4/20/2014
language:English
pages:29