Open Response Questions by pLdevf2h


									    Open Response Questions

 State ORQ Development PowerPoint
 KASC Information
 Other State Information
          Essential Question

   What information do I need to help
    me design well-written open-response
    questions that will assess what I am
    currently teaching?
Why Teach Open-Response?

Open-response questions require
  students to:
 Process information
 Use critical thinking
 Communicate what they know in writing
(Read information from the Partnership
  for Kentucky Schools)
   Kentucky Core Content Test
Content Advisory Committee (CAC)

    Pretest Item Development
   Kentucky Core Content Test
Test Item Development

   General test development tips

   Developing multiple-choice items

   Developing open-response items
    General Item-Writing Guidelines
 Make sure the item assesses knowledge or
  skills described by the Core Content and
  Academic Expectations.
 Make sure the item reflects the Core Content
  at the appropriate grade level.
 Use simple, basic vocabulary instead of
  technical vocabulary unless you are
  assessing the students’ knowledge of the
  meaning of the technical word/phrase.
Sample Item
 When you plant a seed, the roots grow
 downward. This is called geotropism.
 Which factor is responsible for
 When you plant a seed, the roots grow
 downward, and the stem grows upward.
 Which factor is responsible for the roots
 growing downward?
General Item-Writing Guidelines
   Use grade-appropriate vocabulary as much
    as possible.
   Eliminate irrelevant information.
   Include clear, correct, easily understood
    graphics as required by the item.
   Follow the Guidelines for Handling
    Sensitive Issues in Kentucky’s State
    Assessment Development.
Bias should be avoided in:

   •   question prompts,
   •   identified situations,
   •   graphics, or
   •   reading selections

 The emphasis should be on developing items
 that reflect academic versus life experiences.
   Guidelines for Handling Sensitive
   Issues in Assessment Development
            Protecting Privacy and Avoiding
                   Offensive Content

The content of the text or test item(s) will not intrude on the
privacy of the values and beliefs of students or their families,
or offend students, parents, or the public of Kentucky.
Guidelines for Handling Sensitive
Issues in Assessment Development
           Opportunity and Access
   The content of the text or test item(s) will provide
students with a fair opportunity to demonstrate what
they know, regardless of their race, ethnicity, gender,
religion, disability, socioeconomic status, or region in
which they live.
Guidelines for Handling Sensitive
Issues in Assessment Development

      Portrayal of Groups Represented

   Although text or test item(s) may focus on one group
within Kentucky’s diverse population, issues and/or
themes are approached in a manner that do not demean,
offend, or inaccurately portray any race, ethnicity,
religious group, disability, culture, gender, social group,
or region.
Guidelines for the Development of
     Open-Response Items
Suggestions for Item Format

 Present the prompt in paragraph form.
 Use bullets to emphasize the details in the
 If the students are required to respond to
  multiple parts of a question, label each part
  separately (a, b, c).
Sample Item
Martin said, “I am thinking of a whole number
between 100 and 300.
  The number is divisible by three but not by 9.
  The ones digit is the sum of the hundreds digit
   and the tens digit.”
A. Show why 153 cannot be Martin’s number.
B. Find all the numbers that match Martin’s clues.
   Show all your work.
C. Write one more clue that would limit the answer in
   part b to one and only one correct number.
   Make sure that each item has an item
    prompt (i.e., one or more introductory
    sentences) that precedes the directions.
    Even if the item has a graphic, there still
    must be a prompt that describes or provides
    information related to the graphic and/or
    item directions.
Is there any extraneous information
in the stem? The stem of an item
should only include information that
is essential to answering the item.
Does the item stem include all
information essential to answering
the item? Just as some items include
too much information, other items
include too little. It is very important
for an item stem to include all of the
information necessary to answer the
item or necessary for the answer to be
Are the item directions directly tied to
the content of the item stem? In an
open-response item, it is important that
the item directions be directly tied to the
content of the stem. When the stem’s
content is disjointed from the item
directions, it may cause confusion for
students who look to the stem for clues
as to how to answer the item.
Make sure the directions are
complete and specify what is desired
from the students.

 If you want the students to provide examples,
  state in the directions that the students should
  provide examples and how many.
 As a general rule, if students are asked to
  generate a certain number of ideas on their
  own without benefit of a passage or graphic,
  then there should be at least twice that
  number of possible answers. You will be
  asked to supply the possible answers.
Make sure that the task is actually

If the item asks students to read a passage or
examine a graphic and then “give three ways”
or “explain two reasons” based on the
material given, make sure there are three ways
or two reasons found in the material provided.
Is the item specific enough?
Sometimes items are written that are
not specific enough. Such items are
usually difficult to score and unfair to
Write a 4 Response to the ORQ’s

 Once an open response question is
  written, write a response to the question
  so that you can determine if the
  question needs revisions.
 If there is more than one correct
  answer, it needs to be stated.
When drafting open-response
items . . .
   Write open-response items on butcher
   Critique each item as a group—for the
    match to Core Content, grade-level
    appropriateness, vocabulary, and whether
    answerable within the one-page limit.
   Brainstorm examples of information that
    would be included in an acceptable
There are Five Types of ORQ’s.

   Scaffolded
    Have multiple parts with each question
    or direction the student is to address
    presented and labeled separately. The
    order is arranged so that successive
    questions depend upon the response to
    the previous question.
    Single Dimension/Component

   Asks a straightforward question which
    requires explanation, examples,
    descriptions, or evidence as support.
        Two or More Relatively
       Independent Components
   Are signaled by A, B, etc. and may be
    about the same prompt, but have little in
    relation to one another. Response to
    one question is not dependent upon the
    response to the other.
             Student Choice

   Provides topics or options, which allow
    the student to choose from the
    selections. Have many more “correct”
    answers and may be more difficult to
         Response to Provided
   May include data, readings, or graphics.
    Requires students to be able to
    manipulate raw materials in order to
    respond to specific questions.
Guidelines for the Development
  of Multiple-Choice Items
 Identify whether the following
ORQ’s are strong, weak or not an
1. Several students in your class have
  trouble getting along with others. Using
  your experience, write an article for your
  class newsletter explaining how to make
  and keep friends.
   Be sure there is only one right answer.
   Develop parallel response options.
    Options should be parallel with respect
    to content, structure, and length.
   State the item stems in positive terms (if
A class had a pizza party. They ate most of the
pizza, but wanted to make “whole” pizzas from
the leftover slices. Some leftovers are cut into
halves, some are cut into fourths and some are
cut into eighths. There are enough left overs to
make two whole pizzas.
A. DRAW two ways to make a whole pizza
    from the leftover slices using at least two
    different fractions per pizza.
B. LABEL EACH slice ½, ¼, OR 1/8.
C. Explain how your choices equal a whole
Let’s try one more…

Draw a diagram of the water cycle and tell
how one of the steps applies to your life.
   Avoid the use of absolute terms in
    response options (always, never, all,
    none, only).
   Avoid using the options “all of the
    above” and “none of the above.”
When writing response options…

 Make the incorrect response options plausible.
 Common misconceptions or errors of students
  are good response options to include in an item.
 Use other concepts or terms from the grade-
  appropriate Core Content.
 Avoid humorous or nonsensical response
Training Examples for
Multiple-Choice Items
Do all four response options make
sense? Each response option should
make sense both grammatically and
with respect to content.
Is the vocabulary used in the stem
and response options grade-level
appropriate? It is very important
that the wording used in the stem and
response options be grade-level
appropriate, otherwise the item may
be assessing a student’s knowledge of
vocabulary rather than content
Is there one – and only one – right
answer? Each item must have only
one right answer.
Is there any inadvertent cueing
going on in the item? Sometimes
items are written in such a way
that one of the response options
contains an important word or
phrase from the stem.
Does one response option stand out
from the others in any way? Some
items are written so that one of the
response options clearly stands out
from the other response options. There
are many ways that a response option
may stand out from the other options.
One way is if one of the options begins
with a different word than the other
three options.
Another way that an option can stand
out is in length. If an option is much
longer or shorter than the other three
options, it stands out.
Is the item format an appropriate
one? Items should not be written in
certain formats. For example, “fill in
the blank format” should be avoided,
as should the analogy format (except in
reading, grade 10). Analogies in
particular are very difficult for many
When drafting multiple-choice

 Develop items based on need according to the Test
 Review existing items with the same Content Code
  to avoid redundancy.
 Draft items using the development form.
 Prepare for item review.
Training Examples for
Open-Response Items
Does the item stem inadvertently cue
students to one or more answers?
Just as with multiple choice items, it is
important to make sure that the stem
of an open-response item does not cue
students to the answer(s).
   Write “final version” of the item on the
    development form.
   Include examples as part of the
    development form.
When drafting items . . .
    Complete the information requested on the
     development forms.
    Provide complete copyright information if
     using material from another source.
    Complete a graphic submission form if
     including a graphic as part of the question.
Item Development Goals

 80 multiple-choice Items
 20 open-response Items

 10 multiple-choice items per passage
 3 open-response items per passage

To top