How to Write Exams
An Office of Learning Excellence Workshop
Art Hill, OMS Department Steve Huchendorf, OMS Department
February 8, 2006
Agree or disagree?
1. Student learning depends heavily on the assessment method. 2. Ungraded assessments can be even more important than graded assessments in the student learning experience. 3. Assessments should always be aligned with course goals. 4. It is almost impossible to measure higher order thinking. 5. An in-class test is usually the best way to assess learning. 6. It is usually best to have only one type of assessment method. 7. Assessments should be primarily at the end of the course.
How to Write Exams – an OLE Workshop by Art Hill & Steve Huchendorf, OMS Department
Slide 2
McKeachie‟s Nine Assertions
McKeachie’s Nine Assertions: 1. Student learning depends on assessment. What is measured is valued. 2. Tests are for facilitating learning, not merely assigning grades 3. Include informal (non-graded) Classroom Assessment Techniques (CATs) for feedback about learning. 4. Match assessment methods with course goals. If higher order thinking is a goal, is it being assessed? 5. Some goals (motivation, attitudes, values, ethics) may not be measurable by conventional means. Develop other methods for assessment 6. Testing is not the only assessment technique. Look for other classroom and out-of-class activities: Fits, Portfolios, Peer Assessment 7. Develop student self-assessment skills through training and practice. 8. Provide a variety of assessments, don‟t rely solely on mid-term and final exams. 9. Assessment is not only an end-of-class activity. Assessment can be an active learning technique throughout the course.
Source: McKeachie's’ Teaching Tips, 12th edition, Chapter 7, pgs 74-86. How to Write Exams – an OLE Workshop by Art Hill & Steve Huchendorf, OMS Department Slide 3
Outline
Warm-up exercise How to write multiple-choice questions (Art) On-line exams at the U of M (Art) Classroom assessment techniques – CATs (Steve) Conclusions (Art)
How to Write Exams – an OLE Workshop by Art Hill & Steve Huchendorf, OMS Department
Slide 4
How to write multiple-choice questions
Multiple-choice items have three components: 1. Item stem Question to be answered Problem to be solved Incomplete statement to be completed 2. The item key – the response with the correct answer. 3. The item distracters – the incorrect responses.
How to Write Exams – an OLE Workshop by Art Hill & Steve Huchendorf, OMS Department
Slide 5
How to write multiple-choice questions
Types of MC questions: One correct answer – One unambiguous answer. One best answer – Best answer based on judgment. Multiple answers – Multiple correct answers. Negative stem – Which one of is NOT correct? Combined response – Pick combinations of alternatives that are correct (I only, II only, III only, I and II, etc.). True-false (yes/no or right/wrong).
How to Write Exams – an OLE Workshop by Art Hill & Steve Huchendorf, OMS Department
Slide 6
How to write multiple-choice questions
Require clarity – Define the context, make all necessary qualifications, use precise meanings of words in standard English, and avoid superfluous information. Focus on the content that matters – not peripheral or trivial. Randomize the order of the responses. Use parallel grammatical construction for responses. Do not repeat words from the stem in the responses. Avoid the words all, none, certainly, never, and always. Avoid double barreled questions – especially for true/false questions. Avoid negative questions. Make the distracters plausible and avoid silly or absurd distracters. Use “none of the above” and “all of the above” rarely and with caution.
How to Write Exams – an OLE Workshop by Art Hill & Steve Huchendorf, OMS Department
Slide 7
How to write multiple-choice questions
1. Assemble in groups of 2 or 3 people. (This might be a good chance to make some new friends.) Evaluate this question. A good job description usually contains all of the following except: a. The performance level of the current employee b. A general job summary c. An identification of the job d. The duties and responsibilities of the job e. All of the above are included 3. Write one or two multiple choice questions each for your discipline – and then have your other group members evaluate your questions.
2.
How to Write Exams – an OLE Workshop by Art Hill & Steve Huchendorf, OMS Department
Slide 8
Outline
Warm-up exercise How to write multiple-choice questions (Art) On-line exams at the U of M (Art) Classroom assessment techniques – CATs (Steve) Conclusions (Steve)
How to Write Exams – an OLE Workshop by Art Hill & Steve Huchendorf, OMS Department
Slide 9
On-line exams at the U of M
My experience: Take as many times as they want. Any time until the day before grades are due. Open book and open notes, but must work individually. One-hour time limit. Exactly 50 test items randomly selected from database. Database is organized into 14 categories, with a set number for each. Students see their score immediately after completing the exam. Students get extensive feedback immediately after the exam. Only the highest exam score is recorded in the grade book. All scores, previous exams, and previous exam feedback is available.
Slide 10
How to Write Exams – an OLE Workshop by Art Hill & Steve Huchendorf, OMS Department
On-line exams at the U of M
Benefits: Frees up significant class time for class discussion and interaction. Provides real accountability for learning the basic material. Gets the “basic” material out of the way so that the classroom can focus on the higher-order critical thinking skills. Takes a lot of the stress out the testing process – which is appropriate for adult learners. Maximizes the learning experience – students can keep trying and keep learning until they get the grade they want (or run out of time).
How to Write Exams – an OLE Workshop by Art Hill & Steve Huchendorf, OMS Department
Slide 11
On-line exams at the U of M
Some challenges: Only for the U of M community Slightly difficult for data entry and editing Somewhat difficult data management (e.g., import/export to Excel was nearly impossible). Very limited test structure – only one level. Extremely limited test statistics – wanted response level and item statistics.
How to Write Exams – an OLE Workshop by Art Hill & Steve Huchendorf, OMS Department
Slide 12
Outline
Warm-up exercise How to write multiple-choice questions (Art) On-line exams at the U of M (Art) Classroom assessment techniques – CATs (Steve) Conclusions (Art)
How to Write Exams – an OLE Workshop by Art Hill & Steve Huchendorf, OMS Department
Slide 13
Classroom Assessment Techniques (ALT-CATs)
Angelo & Cross, Classroom Assessment Techniques McKeachie, Teaching Tips
Active Learning Techniques (ALTs)
How to Write Exams – an OLE Workshop by Art Hill & Steve Huchendorf, OMS Department
Slide 14
Classroom Assessment Techniques (ALT-CATs)
Angelo & Cross, Classroom Assessment Techniques McKeachie, Teaching Tips
Active Learning Techniques (ALTs)
+
Classroom Assessment Techniques (CATs)
How to Write Exams – an OLE Workshop by Art Hill & Steve Huchendorf, OMS Department Slide 15
Classroom Assessment Techniques (ALT-CATs)
Angelo & Cross, Classroom Assessment Techniques McKeachie, Teaching Tips
Active Learning Techniques (ALTs)
+
=
ALT-CATs
Classroom Assessment Techniques (CATs)
How to Write Exams – an OLE Workshop by Art Hill & Steve Huchendorf, OMS Department
Slide 16
Classroom Assessment Techniques (ALT-CATs)
ALT-CATs: Informal assessments of classroom learning Did the students get it? Find out! Bush Grant “Improving Student Learning in Large Lectures” ALT-CATs in MBA 6120 Data Analysis and Statistics for Managers, Online Section – Spring 2006
How to Write Exams – an OLE Workshop by Art Hill & Steve Huchendorf, OMS Department
Slide 17
Classroom Assessment Techniques (ALT-CATs)
Instructor
Final Exam Mid-Term Exam Quiz Assignment
Active Learning
Lecture Pre-read Student Learning
How to Write Exams – an OLE Workshop by Art Hill & Steve Huchendorf, OMS Department
Slide 18
Classroom Assessment Techniques (ALT-CATs)
Instructor
Final Exam Mid-Term Exam Quiz Assignment
Active Learning
Lecture Pre-read Student Learning
How to Write Exams – an OLE Workshop by Art Hill & Steve Huchendorf, OMS Department
Slide 19
Classroom Assessment Techniques (ALT-CATs)
Incorporating ALT-CATs into course design:
ALT-CAT opportunities provided throughout the semester Each ALT-CAT completed in a team environment Teams of 2, 3 or 4 Team members sign their names ALT-CATs are turned in and scanned for informal assessment Recorded as 5 points Participation points Begin the next class period with an overview „Teachable Moment‟ - Most common mistakes Bridge from “Where we‟ve been” to “Where we‟re going”
How to Write Exams – an OLE Workshop by Art Hill & Steve Huchendorf, OMS Department
Slide 20
Classroom Assessment Techniques (ALT-CATs)
Informal (non-graded) Classroom Assessment Techniques (ALT-CATs):
Angelo & Cross Classroom Assessment Techniques Think-Pair-Share, Write-Pair-Share One Minute Write / Muddiest Point Buzz Groups Focused Listings Shared brainstorming Applications cards Background knowledge probe Concept maps Invented dialogues Analytic Memos Empty Outlines
How to Write Exams – an OLE Workshop by Art Hill & Steve Huchendorf, OMS Department Slide 21
Classroom Assessment Techniques (ALT-CATs)
My Experience with ALT-CATS: Additional incentive for attendance Additional incentive for participating in Active Learning Additional incentive for cooperative learning Assists with the two endpoints of student participation Isolates the „dominator‟ More opportunities for the non-participant Rapid Response - More immediate feedback to the student Comfort level of class Working in same small group Connects the „islands‟
How to Write Exams – an OLE Workshop by Art Hill & Steve Huchendorf, OMS Department Slide 22
An ALT-CAT for the ALT-CATs
1. Assemble in groups of 2 or 3 people. 2. Create an Applications Card ALT-CAT. Select an ALT-CAT and a concept that you teach in your course Identify how you would adapt the ALT-CAT in creating an active learning experience for your students 3. Buzz Groups – discuss with your group.
How to Write Exams – an OLE Workshop by Art Hill & Steve Huchendorf, OMS Department
Slide 23
Outline
Warm-up exercise How to write multiple-choice questions (Art) On-line exams at the U of M (Art) Classroom assessment techniques – CATs (Steve) Conclusions (Art)
How to Write Exams – an OLE Workshop by Art Hill & Steve Huchendorf, OMS Department
Slide 24
Conclusions
Formal (graded) assessments Exams, homework assignments, quizzes, case studies, projects, portfolios, etc. Informal (non-graded) assessments Think-pair-share, buzz groups, problem-based learning, concept maps, analytic memos, etc. Online assessment tools Writing multiple-choice exams is really hard! The vision …
How to Write Exams – an OLE Workshop by Art Hill & Steve Huchendorf, OMS Department
Slide 25