326

Document Sample
326 Powered By Docstoc
					WHAT MAKES A GOOD
TEST QUESTION?
Karen E. Rambo-Hernandez, Assistant Professor
School of Education & School of Teacher Education and Principal Preparation
  Introductions
     Name
     School
     Position




Introductions  CTT Test Common Sense  Using data Recommendations
  Expectations
     You are going to be asked to do things 
     Feel free to ask questions
       Understand   I may have to delay answering some if we
        start running behind
     You may have to do some simple mental
      mathematics




Introductions  CTT Test Common Sense  Using data Recommendations
First:
A VERY brief introduction to Classical Test Theory


                      X= T + E
  Actual Test Score                                  Error

                   True Score if measured precisely
            (average of an infinite number of observations)

        Every observed score (X) is made up of two parts:
             •The person’s true score (real ability) (T)
                             •Error (E)
Introductions  CTT Test Common Sense  Using data Recommendations
 Things to keep in mind
    Error can be both positive and negative
    If the error is small, the true score and the observed
     score will be closer to each other
    There is always some error, so we can never know
     the true score
    We are going to look at ways to improve items
            Common sense
      First-

      Second- Item data

    I use the terms item and question interchangeably
Introductions  CTT Test Common Sense  Using data Recommendations
 The Test of Franzipanics
    Please answer every item on the “Test of
     Franzipanics”




Introductions  CTT Test Common Sense  Using data Recommendations
  What are some ways to reduce error
  on your multiple choice assessments?
     Avoid cueing
       Ending statements with “an”, a, etc.
       Giving away answers in other items

     Make all responses roughly the same length
     Remove extremes
       Always,   never, etc.




Introductions  CTT Test Common Sense  Using data Recommendations
  How would you improve this item?
  In objective testing, the term objective:
      a.   refers to the method of    identifying the learning
           outcomes.
      b.   refers to the method of    selecting the test content.
      c.   refers to the method of    presenting the problem.
      d.   refers to the method of    scoring the answers.




Introductions  CTT Test Common Sense  Using data Recommendations
  A better item
  In objective testing, the term objective refers to
      the method of...
      a.   identifying the learning outcomes.
      b.   selecting the test content.
      c.   presenting the problem.
      d.   scoring the answers.




Introductions  CTT Test Common Sense  Using data Recommendations
  How would you improve this item?
  A table of specifications...
      a.   indicates how a test will be used to improve
           learning.
      b.   provides a more balanced sampling of content.
      c.   arranges the instructional objectives in order of
           their importance.
      d.   specifies the method of scoring to be used on a
           test.


Introductions  CTT Test Common Sense  Using data Recommendations
  A better item
  What is the main advantage of using a table of
    specifications when preparing an
    achievement test?
      a.   It reduces the amount of time required.
      b.   It improves the sampling of content.
      c.   It makes the construction of test items easier.
      d.   It increases the objectivity of the test.



Introductions  CTT Test Common Sense  Using data Recommendations
  Common Sense
     Make sure all of the needed material is present in
      the stem
     “Front load” the stem
       Remove   as much as possible from the options




Introductions  CTT Test Common Sense  Using data Recommendations
  What about this item?
      Which of these opening sentences to a
       mathematics word problem is most likely to
       constitute unfair penalization?
  a)    Mary and Chris were at the mall to shop for new
        shoes, and they each had $40 to spend.
  b)    Albert’s batting average was .267 after his first
        four baseball games.
  c)    Latoya started working on her science fair
        project at 7 p.m.
  d)    Scott averaged reading 25 pages a night in his
        new book.
Introductions  CTT Test Common Sense  Using data Recommendations
  Context matters!
     I used this item on an assessment
     The boys missed the item at much higher rates than
      the girls!
     My item about unfair penalization actually unfairly
      penalized the boys in my class!
     Choose neutral context as much as possible




Introductions  CTT Test Common Sense  Using data Recommendations
 Identifying problems with multiple
 choice items… the data!
 1.    Look at percent correct
       Ifthere are 4 responses and less than 25% got it right,
        you likely have a problem.




Introductions  CTT Test Common Sense  Using data Recommendations
 Identifying problems with multiple
 choice items
 1.        Look a discrimination values
      a)       Sort students from highest total score to lowest total
               score
      b)       Divide class into thirds: highest, middle, lowest
      c)       Calculate percent correct for the highest and lowest
      d)       Subtract:
                Percent correct (highest)- percent correct (lowest)
              If the difference is less than or near zero (<.2) for any
               item, you may have a problem.


Introductions  CTT Test Common Sense  Using data Recommendations
 An example
 Item     P Value      Upper 23        Lower 23     Discrimination
      1    0.96          1.00            0.91            0.09
      2    0.58          0.51            0.65            -0.14
      3    0.80          0.87            0.57            0.30
      4    0.68          0.65            0.61            0.04
      5    0.51          0.78            0.22            0.57
      6    0.75          1.00            0.70            0.30
      7    0.90          0.96            0.87            0.09
      8    0.96          1.00            0.96            0.04
      9    0.67          0.96            0.43            0.52
     10    0.67          0.83            0.43            0.39
Introductions  CTT Test Common Sense  Using data Recommendations
 Breaking it down farther
    Which question is more likely to have more error
     associated with it?
                                         Q2
                     Q1
                40% A.             17%   A.
                51% B.             22%   B.
                  8% C.            37%   C.
                0.5% D.            12%   D.
                                   12%   E.




Introductions  CTT Test Common Sense  Using data Recommendations
 Identifying problems with multiple
 choice items
    3.       Examine your distractors
             If you have low percentages of students
              choosing a particular option, then the item is
              actually an easier item
             Improving your distractors reduces the amount
              of error




Introductions  CTT Test Common Sense  Using data Recommendations
 Sample Analysis 1

                     A            B            C            D*

   Upper 10          2            0            1            7
   students
   Lower 10          2            4            2            2
   students

      What is the percent correct?
      What is the discrimination value?
      Based on this information, does this item seem to need
       improvement?
      What other recommendations do you have for this
       item?
Introductions  CTT Test Common Sense  Using data Recommendations
 Sample Analysis 1

                     A            B            C            D*

   Upper 10          2            0            1            7
   students
   Lower 10          2            4            2            2
   students

      Total percent correct (of shown): 9/20= 45%
      Upper students correct : 7/10= 70%
      Bottom Students % correct: 2/10=20%
      Discrimination: 70%-20%=50%
Introductions  CTT Test Common Sense  Using data Recommendations
  Sample Analysis 2
                     A            B*           C            D
   Upper 10          2            7            1             0
   students
   Lower 10          4            5            0            1
   students

      What is the percent correct?

      What is the discrimination value?
      Based on this information, does this item seem to
       need improvement?
      What other recommendations do you have for this
       item?
Introductions  CTT Test Common Sense  Using data Recommendations
  Sample Analysis 2
                     A            B*           C            D
   Upper 10          2            7            1             0
   students
   Lower 10          4            5            0            1
   students

      Total percent correct (of shown): 12/20= 60%

      Upper students correct : 7/10= 70%

      Bottom Students % correct: 5/10=50%

      Discrimination: 70%-50%=20%

Introductions  CTT Test Common Sense  Using data Recommendations
  Other tips
     Avoid using negative words: not, no, never
       If you must use it, EMPHASIZE negative wording

     Put most of the information needed for the item in the
      stem.
     State the stem of the item in simple, clear language.
        Limit or eliminate extraneous material from the
         stem.
     Apply these guidelines to other types of multiple
      choice items
       True/false,   matching

Introductions  CTT Test Common Sense  Using data Recommendations
  Constructed response items
     They come in various forms
       Essays

       Short   answer*
       Fill-in the blank*

       Computation problems*
        *  follow a lot of the same guidelines as selected response
          items
         Essays, well, that is another issue for another day




Introductions  CTT Test Common Sense  Using data Recommendations
Constructed response items
   If time allows…
     Lookat the list of recommendations
     Which ones apply to constructed response?
  Ok, so when should I actually do this

     General common sense guidelines
       Every   time!
     Item level data
       For   MAJOR assessments: finals, midterms, etc.




Introductions  CTT Test Common Sense  Using data Recommendations
  And remember…

   Break  (or bend) any rules if necessary to improve the
   effectiveness of the item!




Introductions  CTT Test Common Sense  Using data Recommendations
  That’s all folks!
     Questions?
     Thanks for coming!




Introductions  CTT Test Common Sense  Using data Recommendations

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:1
posted:11/26/2012
language:English
pages:29