Documents
Resources
Learning Center
Upload
Plans & pricing Sign in
Sign Out

multiple-choice

VIEWS: 85 PAGES: 39

									   Writing Strong
Multiple-Choice Items

     Alan Carter, IT Instructor
     acarter@greenriver.edu




                                  1
                     Agenda
 9:00 – 9:45    Welcome
                 Presentation on Writing Strong
                 Multiple-Choice Exam Items
 9:45 – 10:30   Break, Participants Write 3 Items
10:30 – 11:00   Meet as a Group
                 Introduce Multiple-Choice Item Rubric
11:00 – 11:30   Break, Participants Analyze Their
                 Own Items, Groups Analyze Each
                 Others’ Items Against the Rubric
11:30 – 11:45   Meet as a Group to Discuss Reflection/
                 Reactions/Results, Q & A, Wrap-Up

                                                          2
           In the Beginning,
       There Was Assessment...
 What Is Assessment?
  According to Stephen Brookfield, “assessment is
  a value-free ascertainment of the extent to which
  objectives determined at the outset of a program
  have been attained by participants. [It] requires no
  value judgment as to their worthwhileness. It is
  simply a nonjudgmental checking as to whether or
  not certain purposes have been attained.”



                                                         3
Why Do We Assess Student Learning?

 To determine whether students have met the
  learning outcomes specified in the CAR

 For grading purposes

 For accreditation purposes




                                               4
          Exams: A Common
         Assessment Technique
 Exams are often used to assess student learning

 Discussion Question:
     What general types of questions or elements
     do you use in your exams (multiple-choice,
     perform a hands-on task, etc.)?




                                                    5
Focus of This Session: Writing Strong
    Multiple-Choice Test Items
 Multiple-choice items are test items in which
  students select the correct answer from a list of
  several choices. (Haladyna, 1994)

 Multiple-choice items can be used to “measure
  knowledge and complex mental acts, such as
  reasoning, critical thinking, and problem solving”
  (Haladyna, 1994).



                                                       6
Components of a Multiple-Choice Item
                  Stem
 What is H2O?

 A.   Fire       Correct Answer
 B.   Water
 C.   Air
                  Distractors
 D.   Earth




                                       7
           Components of a Complex
             Multiple-Choice Item
Scenario
   Jack, who is 48, took his 78-year-old mother to see a
   movie. General admission tickets cost $10, and the
   cost of senior admission is $8.
   How much did Jack spend for movie tickets?
   A.   $ 10                                    Stem
   B.   $ 16                  Correct Answer
   C.   $ 18
                              Distractors
   D.   $ 20

                                                           8
             What Is a Strong
           Multiple-Choice Item?
 Measures the learning outcomes for the course

 Follows the best practices for writing multiple-
  choice items

 Assesses student learning at the appropriate level
  of Bloom’s taxonomy (which we will discuss later
  in this session)


                                                       9
             Learning Outcomes
 The course content learning outcomes in the
  Course Adoption Revision document (CAR)
  should drive assessment

 For example, if an outcome states:
  “Student identifies the functions of the four
   classes of biomolecules”
   then the exam items relating to this outcome
   should require students to identify the functions,
   as opposed to analyzing or evaluating the
   functions

                                                        10
    Best Practices for Constructing
        Multiple-Choice Items
 There should be only one correct answer
 Avoid “trick” questions
 The wording in the stem should not
  telegraph the correct answer
 Avoid negatives (no, not, etc.) in the scenario
  and stem. If they cannot be avoided, consider
  calling attention to the word by using bold and/or
  italic font

                                                       11
    Best Practices for Constructing
        Multiple-Choice Items
 All distractors should be plausible answers
 Distractors should be of parallel construction and
  length
 Avoid “all of the above,” “none of the above,” and
  “A and B, but not C”
 Avoid testing trivial material
 Items should focus on a single problem, concept,
  or idea
                                                       12
    Best Practices for Constructing
        Multiple-Choice Items
 Balance the answer key: If there are four possible
  answers, approximately 25% of the correct
  answers should be A, 25% should be B, and so on
 Items should not provide the answer for other
   items on the exam
 When possible, the stem should be a
  question, and end with a question mark
 Avoid textbook verbatim phrasing unless asking
  a knowledge-based question
                                                       13
                  You Be the Judge
Which sport involves players shooting a ball through
a hoop?

A.   Soccer
B.   Cricket
C.   Ping pong
D.   Basketball

           Which best practice of item writing
                does this item violate?

                                                       14
              You Be the Judge
What is a paraplegic?

A. An animal
B. A fish
C. A person suffering from paralysis of both lower
   limbs
D. A plant

         Which best practice of item writing
              does this item violate?

                                                     15
                 You Be the Judge
The textbook, Operations Management, was written
in:

A.   2000
B.   2002
C.   2004
D.   2006

            Which best practices of item writing
                  does this item violate?

                                                   16
  Introduction to Bloom’s Taxonomy

 Benjamin Bloom, an educational psychologist,
  proposed this classification of learning levels in
  1956
 Breaks cognitive thinking into six categories

 These categories can be thought of as
  degrees of difficulty



                                                       17
           Levels of Bloom’s Taxonomy

Category/Level     Description
Evaluation         Make judgments

Synthesis          Put parts together to form a whole

Analysis           Distinguishes between facts and
                   inferences
Application        Use a concept in a new situation
Comprehension      Understand the meaning
Knowledge          Recall data or information

Clark, D. (2001)
                                                        18
      Bloom’s Examples and Key Words
Knowledge          Examples: Recite a policy. Quote prices from memory
                   to a customer. List the safety rules.
                   Key Words: defines, describes, identifies, knows,
                   labels, lists, matches, names, outlines, recalls,
                   recognizes, reproduces, selects, states.
Comprehension      Examples: Explain in your own words the steps for
                   performing a complex task. Translate an equation into
                   a computer spreadsheet.
                   Key Words: comprehends, converts, defends,
                   distinguishes, estimates, explains, extends,
                   generalizes, gives examples, infers, interprets,
                   paraphrases, predicts, rewrites, summarizes,
                   translates.
Clark, D. (2001)
                                                                           19
     Bloom’s Examples and Key Words
Application        Examples: Use a manual to calculate an employee’s
                   vacation time. Apply laws of statistics to evaluate the
                   reliability of a written test.
                   Key Words: applies, changes, computes, constructs,
                   demonstrates, discovers, manipulates, modifies,
                   operates, predicts, prepares, produces, relates,
                   shows, solves, uses.
Analysis           Examples: Troubleshoot a piece of equipment by
                   using logical deduction. Recognize logical fallacies in
                   reasoning. Gather information from a department and
                   select the required tasks for training.
                   Key Words: analyzes, breaks down, compares,
                   contrasts, diagrams, deconstructs, differentiates,
                   discriminates, distinguishes, identifies, illustrates,
                   infers, outlines, relates, selects, separates.
Clark, D. (2001)
                                                                             20
        Bloom’s Examples and Key Words
Synthesis          Examples: Write a company operations or process manual.
                   Design a machine to perform a specific task. Revise a process
                   to improve the outcome.
                   Key Words: categorizes, combines, compiles, composes,
                   creates, devises, designs, explains, generates, modifies,
                   organizes, plans, rearranges, reconstructs, relates,
                   reorganizes, revises, rewrites, summarizes, tells, writes.
Evaluation         Examples: Select the most effective solution. Hire the most
                   qualified candidate. Explain and justify a new budget.
                   Key Words: appraises, compares, concludes, contrasts,
                   criticizes, critiques, defends, describes, discriminates,
                   evaluates, explains, interprets, justifies, relates, summarizes,
                   supports.

Clark, D. (2001)
                                                                                      21
                Bloom’s Level?
The Social Security Act of 1935 provides:

A.   Emergency relief to farmers
B.   Funding for small businesses
C.   Financial assistance to retired people
D.   A guaranteed minimum wage for workers




                                              22
                Bloom’s Level?
Your car suddenly stops running. When you attempt
to restart the car by turning the key in the ignition to
the start position, nothing happens.
What is the most likely cause of the problem?
A. The gas tank is empty
B. The spark plugs are fouled
C. The battery is dead
D. The fuel injectors are clogged


                                                           23
         What About Test Banks?
 What is your experience with test banks?
  Do you use them? Do you like the items?



 Test bank items often violate many of the best
  practices for writing strong multiple-choice items
 You may find that you need to edit or revise test
  bank questions


                                                       24
  How Many Questions Per Exam?

 Depends on:
  Complexity of exam items
  Length of exam items
  Amount of analysis required
  Amount of time available for exam
 Consider taking the exam yourself, and
  then allowing students 2–3 times as long
  as it took you to complete the exam

                                             25
        Assignment: Item Writing
 Take a short break!

 Write three multiple-choice items:
  Create a scenario for each item, or one
    scenario that is used for multiple items
  Try to target the Application or Analysis
    level of learning
  Use best practices for item writing
 Reconvene at 10:30 a.m.

                                               26
             Item Writing Debrief

 Was everyone able to write three items?
 Did you find it difficult to write items that followed
  the best practices?
 Did you find it difficult to write items that targeted
  the higher levels of learning?




                                                           27
        Using Multiple-Choice Items to
           Assess Critical Thinking
Critical thinking is one of the college’s campus-wide
learning outcomes:
 “Critical thinking finds expression in all disciplines
and everyday life. It is characterized by an ability to
reflect upon thinking patterns, including the role of
emotions on thoughts, and to rigorously assess the
quality of thought through its work products. Critical
thinkers routinely evaluate thinking processes and
alter them, as necessary, to facilitate an improvement
in their thinking and potentially foster certain
dispositions or intellectual traits over time.”
(Green River Community College, 2005, Critical Thinking Community Rubric)
                                                                            28
     Using Multiple-Choice Items to
        Assess Critical Thinking

 You can use multiple-choice items to assess a
  student’s ability to think critically about a topic
 Items that assess critical thinking will fall in the
  Application, Analysis, Synthesis, or Evaluation
  level of Bloom’s taxonomy




                                                         29
        Must We Always Assess
          Critical Thinking?
 In general, not all multiple-choice items should
  assess critical thinking, because not all course
  learning outcomes are at the Application, Analysis,
  Synthesis, or Evaluation levels
 For example, if a learning outcome says the
  student will comprehend a topic, an item at the
  Comprehension level would be appropriate
  This item should not measure critical thinking



                                                        30
A Rubric for Assessing How Well Multiple-
 Choice Items Measure Critical Thinking
Competency          Poor/No             Adequate/Easy Good/                 Excellent/
                    Critical                          Intermediate          Difficult
                    Thinking
                    Required
2.1 Apply           The item has        The item has      The item has      The item has
relevant criteria   only 1 discrete     two discrete      three discrete    four or more
and standards       fact or statement   facts or          facts or          discrete facts or
when                that must be        statements that   statements that   statements that
evaluating          evaluated when      must be           must be           must be
information,        determining how     evaluated when    evaluated when    evaluated when
claims, and         to answer the       determining how   determining how   determining how
arguments           question            to answer the     to answer the     to answer the
                                        question          question          question


                                                                                                31
 A Rubric for Assessing How Well Multiple-
  Choice Items Measure Critical Thinking
Competency       Poor/No             Adequate/Easy Good/                   Excellent/
                 Critical                          Intermediate            Difficult
                 Thinking
                 Required
2.2 Use          The item:           The item:          The item:          The item:
appropriate      Is knowledge       Requires          Requires          Requires
reasoning to     based               students to        students to        students to
evaluate         Has only one       analyze multiple   analyze multiple   analyze multiple
problems, make   fact or statement   facts or           facts or           facts or
decisions, and   for students to     statements that    statements that    statements that
formulate        analyze             are interrelated   are partially      are discrete
solutions.                           Partially or      discrete           Does not
                                     directly           Partially         telegraph the
                                     telegraphs the     telegraphs the     correct answer
                                     correct answer     correct answer
                                                                                              32
                You Be the Judge

What is H2O?

A.   Fire
B.   Water
C.   Air
D.   Earth

        Using the rubric, does this item do a poor,
           adequate, good, or excellent job of
     assessing critical thinking for each competency?
                                                        33
               You Be the Judge
Your car suddenly stops running. When you attempt
to restart the car by turning the key in the ignition to
the start position, nothing happens.
What is the most likely cause of the problem?
A. The gas tank is empty
B. The spark plugs are fouled
C. The battery is dead
D. The fuel injectors are clogged
      Using the rubric, does this item do a poor,
          adequate, good, or excellent job of
   assessing critical thinking for each competency?
                                                           34
  Assignment: Determining Whether
    Items Assess Critical Thinking

 As we discussed previously, not all multiple-
  choice items should assess critical thinking
 However, in this exercise we will practice
  determining how well a multiple-choice item
  assesses critical thinking




                                                  35
   Assignment: Determining Whether
     Items Assess Critical Thinking
 Form teams of two persons
 First, individually evaluate the three items you
  wrote by using the rubric. Rate each item as Poor,
  Adequate, Good, or Excellent for each competency.
 Next, exchange items with your teammate and
  evaluate his or her items using the rubric.
 Meet with your teammate and discuss any
  differences in your ratings.
 If time, take a short break!
 Reconvene at 11:30 a.m. for discussion and
  final wrap-up
                                                       36
Reflections, Reactions,
    and Questions


                          37
           Resources for Writing
          Multiple-Choice Test Items

Bloom, B. S. (1956). Taxonomy of educational objectives,
  Handbook I: The cognitive domain. New York: David
  McKay Co. Inc.
Haladyna, T. M. (1994). Developing and validating
  multiple-choice test items. Hillsdale, NJ: Lawrence
  Erlbaum Associates.



                                                           38
                    References
Brookfield, S. D. (1986). Understanding and facilitating
  adult learning: A comprehensive analysis of principles
  and effective practices. San Francisco: Jossey-Bass.
Clark, D. (2001). Learning domains or Bloom’s taxonomy.
  Retrieved September 9, 2006, from
  http://www.nwlink.com/~donclark/hrd/bloom.html
Haladyna, T. M. (1994). Developing and validating
  multiple-choice test items. Hillsdale, NJ: Lawrence
  Erlbaum Associates.

                                                           39

								
To top