# 326

Document Sample

```					WHAT MAKES A GOOD
TEST QUESTION?
Karen E. Rambo-Hernandez, Assistant Professor
School of Education & School of Teacher Education and Principal Preparation
Introductions
   Name
   School
   Position

Introductions  CTT Test Common Sense  Using data Recommendations
Expectations
   You are going to be asked to do things 
   Feel free to ask questions
 Understand   I may have to delay answering some if we
start running behind
   You may have to do some simple mental
mathematics

Introductions  CTT Test Common Sense  Using data Recommendations
First:
A VERY brief introduction to Classical Test Theory

X= T + E
Actual Test Score                                  Error

True Score if measured precisely
(average of an infinite number of observations)

Every observed score (X) is made up of two parts:
•The person’s true score (real ability) (T)
•Error (E)
Introductions  CTT Test Common Sense  Using data Recommendations
Things to keep in mind
   Error can be both positive and negative
   If the error is small, the true score and the observed
score will be closer to each other
   There is always some error, so we can never know
the true score
   We are going to look at ways to improve items
Common sense
 First-

 Second- Item data

   I use the terms item and question interchangeably
Introductions  CTT Test Common Sense  Using data Recommendations
The Test of Franzipanics
Franzipanics”

Introductions  CTT Test Common Sense  Using data Recommendations
What are some ways to reduce error
   Avoid cueing
 Ending statements with “an”, a, etc.
 Giving away answers in other items

   Make all responses roughly the same length
   Remove extremes
 Always,   never, etc.

Introductions  CTT Test Common Sense  Using data Recommendations
How would you improve this item?
In objective testing, the term objective:
a.   refers to the method of    identifying the learning
outcomes.
b.   refers to the method of    selecting the test content.
c.   refers to the method of    presenting the problem.
d.   refers to the method of    scoring the answers.

Introductions  CTT Test Common Sense  Using data Recommendations
A better item
In objective testing, the term objective refers to
the method of...
a.   identifying the learning outcomes.
b.   selecting the test content.
c.   presenting the problem.

Introductions  CTT Test Common Sense  Using data Recommendations
How would you improve this item?
A table of specifications...
a.   indicates how a test will be used to improve
learning.
b.   provides a more balanced sampling of content.
c.   arranges the instructional objectives in order of
their importance.
d.   specifies the method of scoring to be used on a
test.

Introductions  CTT Test Common Sense  Using data Recommendations
A better item
What is the main advantage of using a table of
specifications when preparing an
achievement test?
a.   It reduces the amount of time required.
b.   It improves the sampling of content.
c.   It makes the construction of test items easier.
d.   It increases the objectivity of the test.

Introductions  CTT Test Common Sense  Using data Recommendations
Common Sense
   Make sure all of the needed material is present in
the stem
 Remove   as much as possible from the options

Introductions  CTT Test Common Sense  Using data Recommendations
    Which of these opening sentences to a
mathematics word problem is most likely to
constitute unfair penalization?
a)    Mary and Chris were at the mall to shop for new
shoes, and they each had \$40 to spend.
b)    Albert’s batting average was .267 after his first
four baseball games.
c)    Latoya started working on her science fair
project at 7 p.m.
d)    Scott averaged reading 25 pages a night in his
new book.
Introductions  CTT Test Common Sense  Using data Recommendations
Context matters!
   I used this item on an assessment
   The boys missed the item at much higher rates than
the girls!
   My item about unfair penalization actually unfairly
penalized the boys in my class!
   Choose neutral context as much as possible

Introductions  CTT Test Common Sense  Using data Recommendations
Identifying problems with multiple
choice items… the data!
1.    Look at percent correct
 Ifthere are 4 responses and less than 25% got it right,
you likely have a problem.

Introductions  CTT Test Common Sense  Using data Recommendations
Identifying problems with multiple
choice items
1.        Look a discrimination values
a)       Sort students from highest total score to lowest total
score
b)       Divide class into thirds: highest, middle, lowest
c)       Calculate percent correct for the highest and lowest
d)       Subtract:
     Percent correct (highest)- percent correct (lowest)
        If the difference is less than or near zero (<.2) for any
item, you may have a problem.

Introductions  CTT Test Common Sense  Using data Recommendations
An example
Item     P Value      Upper 23        Lower 23     Discrimination
1    0.96          1.00            0.91            0.09
2    0.58          0.51            0.65            -0.14
3    0.80          0.87            0.57            0.30
4    0.68          0.65            0.61            0.04
5    0.51          0.78            0.22            0.57
6    0.75          1.00            0.70            0.30
7    0.90          0.96            0.87            0.09
8    0.96          1.00            0.96            0.04
9    0.67          0.96            0.43            0.52
10    0.67          0.83            0.43            0.39
Introductions  CTT Test Common Sense  Using data Recommendations
Breaking it down farther
   Which question is more likely to have more error
associated with it?
Q2
Q1
40% A.             17%   A.
51% B.             22%   B.
8% C.            37%   C.
0.5% D.            12%   D.
12%   E.

Introductions  CTT Test Common Sense  Using data Recommendations
Identifying problems with multiple
choice items
    If you have low percentages of students
choosing a particular option, then the item is
actually an easier item
    Improving your distractors reduces the amount
of error

Introductions  CTT Test Common Sense  Using data Recommendations
Sample Analysis 1

A            B            C            D*

Upper 10          2            0            1            7
students
Lower 10          2            4            2            2
students

   What is the percent correct?
   What is the discrimination value?
   Based on this information, does this item seem to need
improvement?
   What other recommendations do you have for this
item?
Introductions  CTT Test Common Sense  Using data Recommendations
Sample Analysis 1

A            B            C            D*

Upper 10          2            0            1            7
students
Lower 10          2            4            2            2
students

   Total percent correct (of shown): 9/20= 45%
   Upper students correct : 7/10= 70%
   Bottom Students % correct: 2/10=20%
   Discrimination: 70%-20%=50%
Introductions  CTT Test Common Sense  Using data Recommendations
Sample Analysis 2
A            B*           C            D
Upper 10          2            7            1             0
students
Lower 10          4            5            0            1
students

   What is the percent correct?

   What is the discrimination value?
   Based on this information, does this item seem to
need improvement?
   What other recommendations do you have for this
item?
Introductions  CTT Test Common Sense  Using data Recommendations
Sample Analysis 2
A            B*           C            D
Upper 10          2            7            1             0
students
Lower 10          4            5            0            1
students

   Total percent correct (of shown): 12/20= 60%

   Upper students correct : 7/10= 70%

   Bottom Students % correct: 5/10=50%

   Discrimination: 70%-50%=20%

Introductions  CTT Test Common Sense  Using data Recommendations
Other tips
   Avoid using negative words: not, no, never
 If you must use it, EMPHASIZE negative wording

   Put most of the information needed for the item in the
stem.
   State the stem of the item in simple, clear language.
 Limit or eliminate extraneous material from the
stem.
   Apply these guidelines to other types of multiple
choice items
 True/false,   matching

Introductions  CTT Test Common Sense  Using data Recommendations
Constructed response items
   They come in various forms
 Essays

 Fill-in the blank*

 Computation problems*
*  follow a lot of the same guidelines as selected response
items
 Essays, well, that is another issue for another day

Introductions  CTT Test Common Sense  Using data Recommendations
Constructed response items
   If time allows…
 Lookat the list of recommendations
 Which ones apply to constructed response?
Ok, so when should I actually do this

   General common sense guidelines
 Every   time!
   Item level data
 For   MAJOR assessments: finals, midterms, etc.

Introductions  CTT Test Common Sense  Using data Recommendations
And remember…

 Break  (or bend) any rules if necessary to improve the
effectiveness of the item!

Introductions  CTT Test Common Sense  Using data Recommendations
That’s all folks!
   Questions?
   Thanks for coming!

Introductions  CTT Test Common Sense  Using data Recommendations

```
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
 views: 1 posted: 11/26/2012 language: English pages: 29