Docstoc

Writing MCQ

Document Sample
Writing MCQ Powered By Docstoc
					Writing MCQs

Susan Chamberlain
Dept of OBS/GYN
Queen‟s University
Writing MCQs

At the end of this session, the participant will
  be able to
 Identify different types of MCQ items
 Describe the structure of an MCQ
 List the technical flaws common to MCQ‟s
 write one well-structured MCQ testing
  appropriate content
Writing MCQs

At the end of this session, the participant will
  be able to
 Identify different types of MCQ items
 Describe the structure of an MCQ
 List the technical flaws common to MCQ‟s
 write one well-structured MCQ testing
  appropriate content
 All of the above
MCQ Check-list

 Why we write them
 What type is best
 What flaws are to be avoided
 What content to test
 How to construct it
Why do we test?

 Communicate to the students what material
  is important (formative vs summative)
 Motivate students to study
 Identify areas of deficiency in need of
  remediation or further learning
 Determine final grades or make promotion
  decisions
 Identify areas where the course/curriculum
  is weak
What Should Be Tested?

 Exam content should match course
  objectives
 Important topics should be weighted more
  heavily than less important topics
 The testing time devoted to each topic
  should reflect the relative importance of the
  topics
 The sample of items should be
  representative of the instructional goals
What Should Be Tested?

   Sampling
      “Performance on the sample provides a basis
      for estimating achievement in the broader
      domain that is actually of interest.”
   Questions sample:
    – TOPICS
    – SKILLS ie determining diagnosis, next step in
      management, etc.
MCQ „Items‟

 Should test important information – either
  common or life-threatening
 Should be well-structured
    – Flaws benefit test-wise examinees
    – Irrelevant difficulty does not test examinees‟
      specific knowledge
Anatomy of an MCQ Item

 Stem
 Lead-in
 Options
    – One correct answer
    – Distractors
Types of MCQ‟s

True/False family
 Assess recall of isolated facts
    – Stems must be clear and unambiguous
    – Options must be absolutely true or false or
      examinee must decide how true options are


   Generally not recommended – usually have
    subtle flaws
Types of MCQ‟s: True/False
Which of the following statements is true regarding
   contraception?
A) oral contraceptives should not be initiated until six weeks
   after therapeutic abortion to avoid the risk of
   hypercoagulability
B) an intrauterine device can provide good contraceptive
   protection 5 years after insertion
C) oral contraceptives should not be prescribed to individuals
   under the age of 15 without parental consent
D) lesbian women do not require counseling about contraception
   or sexually transmitted diseases
E) oral contraceptives should not be prescribed for women over
   the age of 35
 Types of MCQ‟s

 One-Best-Answer family (A type)
  Options may be diagrammed:




 D C A          E                     B
Least correct                       Most correct


    Correct answer is the most likely correct
    Better application of knowledge, integration,
     synthesis and judgment
Types of MCQ‟s:
   One-Best-Answer family
A 24 year old university student came to see you because she is
missing several days of school each month because of severe
dysmenorrhea unresponsive to the OCP and NSAIDS. She also
complains of deep dyspareunia and has a tender uterosacral
nodule on the pelvic examination. Which one of the following
conditions is most likely responsible for her symptoms?
   A) Chronic pelvic inflammatory disease
   B) Endometriosis
   C) Adenomyosis
   D) Fibroids
MCQ Check-list

Why we write them
What type is best
 What flaws are to be avoided
 What content to test
 How to construct it
MCQ Technical Flaws

 Issues Related to Test-Wiseness
 Issues Related to Irrelevant Difficulty
MCQ Technical Flaws

   Issues Related to Test-Wiseness
    – provide clues to the right answer
    – examinee can eliminate options based on flaws
      and not on knowledge


   Issues Related to Irrelevant Difficulty
Item Flaws: Test-wiseness

 Grammatical cues
 Logical cues
 Absolute terms
 Long correct answer
 Word repeats
 Convergence strategy
Item Flaws: Test-wiseness

Grammatical cues
 One or more distractors don‟t follow
  grammatically from the stem
Item Flaws: Test-wiseness

Grammatical cues
A 24 year old patient presents with a one year
   history of dysmenorrhea, mid-cycle pain,
   dyspareunia, and painful bowel movements with
   her periods. The most likely diagnosis is:
     A)   Fibroid uterus
     B)   Adenomyosis
     C)   Endometriosis
     D)   Psychosomatic condition
Item Flaws: Test-wiseness

Logical cues
 A subset of the options are collectively
  exhaustive
Item Flaws: Test-wiseness

Logical cues
   Arterial blood gases in a normal pregnant
   woman, compared to her non-pregnant state may
   show which of the following?
     A)   Decreased pO2
     B)   Increased pCO2
     C)   Decreased pCO2
     D)   Decreased pH
     E)   No difference in pCO2
Item Flaws: Test-wiseness

Logical cues
 One or more distractors don‟t follow
  logically from the stem
Item Flaws: Test-wiseness

Logical cues
An important tool for assessing fetal well-being is
    the Non Stress test. A well baby demonstrates
    the following fetal heart rate pattern:
      A)   A baseline heart rate above 120 bpm
      B)   Evidence of fetal movement
      C)   Baseline variability
      D)   Two fetal heart rate accelerations in 20 minutes
      E)   Fetal breathing movements
Item Flaws: Test-wiseness

Absolute terms
 Always, never
 Not a problem if stem is long and lead-in
  asks a precise question
Item Flaws: Test-wiseness

Absolute terms
In patients with a previous Cesarean section:
   A) Vaginal birth is NEVER an option
   B) Repeat Cesarean section is less dangerous to the
      mother
   C) The uterine scar may rupture during labour
   D) Care should ALWAYS be supervised by an
      obstetrician
   E) No further pregnancies should EVER be considered
Item Flaws: Test-wiseness

Long correct answer
 Correct answer is longer, more specific, or
  more complete than other options
Item Flaws: Test-wiseness

Long correct answer
   Which of the following problems in vaginal
   breech delivery is most likely to cause serious
   long-term complications?
     A) Failure to descend
     B) Cord prolapse
     C) Nuchal arms
     D) Entrapment of the after coming head with an
        incompletely dilated cervix
     E) 1 minute Apgar score of 4
Item Flaws: Test-wiseness

Word repeats
 A word or phase is included in the stem and
  in the correct answer
Item Flaws: Test-wiseness

Word repeats
  A 24 year old with a previous history of chronic
   PID presents with primary infertility. Select the
   procedure that would be most appropriate to
   evaluate the status of her tubes and pelvic
   anatomy
      A) Post-coital test
      B) Endometrial biopsy
      C) Laparoscopy with dye insufflation of the fallopian
         tubes
      D) Hysterosalpingogram
Item Flaws: Test-wiseness

Convergence strategy
 The correct answer includes the most
  elements in common with the other options
Item Flaws: Test-wiseness

Convergence strategy
   Normal bladder neck mobility is measured by:
     A) A Q-tip angle measurement under 40 degrees
     B) A Q-tip angle measurement under 30 degrees
     C) A bladder neck ultrasound angle less than 30
        degrees
     D) A Q-tip angle measurement under 20 degrees
MCQ Technical Flaws

   Issues Related to Test-Wiseness

   Issues Related to Irrelevant Difficulty
    – do not assess knowledge of intended topic
Irrelevant Difficulty

   Options are long, complicated or double
    Extraneous details in stem
   Numeric data not stated consistently
   Vague frequency terms
   Non-parallel language in options
   Stems are tricky or unnecessarily complicated
   „None of the above‟
Irrelevant Difficulty

Options are long, complicated or double
Extraneous details in stem
 Shifts difficulty from content knowledge to
  speed reading
Irrelevant Difficulty
Options are long, complicated or double
     Mrs. Brown is a 70 year old woman who had a stroke 6 months
     ago and is still impaired. Her daughter tells you that her
     mother’s recent urinary incontinence makes her care
     unmanageable. She is seeking permanent placement. Mrs.
     Brown complains of urinary frequency, nocturia, and urge
     incontinence. Which of the following statements is true?

A)   Neurological conditions are involved in the majority of women with
     symptoms like those presented by Mrs. Brown
B)   Diabetes has a greater impact on quality of life than Mrs. Brown’s
     condition.
C)   The pharmacological blockage of muscarinic receptors in the detrusor
     muscle controls symptoms compatible with Mrs. Brown’s condition
D)   Overactive bladder is defined as urinary incontinence accompanied by
     or immediately preceded by urgency
E)   Alpha stimulants are effective in treating this condition
Irrelevant Difficulty
Extraneous details in stem
   You see in your office a 24 year old woman for her routine postpartum
   visit, at 6 weeks. Her baby is irritable and cries “all the time’ (including
   during her visit.) As she is changing the diaper during her visit, you
   notice that the child has a very red bum but upon questioning, she tells
   you she did not notice anything abnormal. Her partner is away in
   Afghanistan and she does not have a lot of friends, as they just recently
   were posted in Kingston. She states that she has a hard time getting
   dressed in the morning, cries easily and feels like she does not love her
   child. She denies any medical or psychiatric history. Her exam is
   essentially normal. She weighs 2 kg less than at her first antenatal visit
   at 10 weeks. You most likely clinical diagnosis is:
       A) Postpartum hypothyroidism
       B) Hyperthyroidism
       C) Postpartum blues
       D) Postpartum depression
       E) Psychotic depression
Irrelevant Difficulty

Numeric Data
 Not stated consistently
 Mixed format
 Illogical order
Irrelevant Difficulty

Numeric Data not stated consistently
   Following a second episode of salpingitis, what
   is the likelihood that a woman is infertile?
     A)   Less than 20%
     B)   20 – 30 %
     C)   Greater than 50 %
     D)   90%
     E)   75%
Irrelevant Difficulty

Vague frequency terms
 Often, usually, rarely, frequently
 Not consistently defined


*may be used in stem if otherwise carefully
 constructed unambiguous question
Irrelevant Difficulty

Vague frequency terms
Gonorrhea is
     A) sometimes asymptomatic
     B)   treated with cefixime 450 mg orally
     C)   caused by a gram positive intracellular diplococcus
     D)   diagnosed by a DNA test
     E)   usually associated with vulvar ulcers
Irrelevant Difficulty

 Non-parallel language in options
 Non-logical order
Irrelevant Difficulty
Non-parallel language in options
In a vaccine trial, 200 2-year-old boys were given a vaccine
     against a certain disease and then monitored for five
     years for occurrence of the disease. Of this group, 85%
     never contracted the disease. Which of the following
     statements concerning these results is correct?
    A) No conclusion can be drawn, since no follow-up was
        made of non-vaccinated children
    B) The number of cases (ie 30 cases over five year) is
        too small for statistically meaningful conclusions
    C) No conclusions can be drawn because the trial
        involved only boys
    D) Vaccine efficacy (%) is calculated as 85-15/100
Irrelevant Difficulty

“None of the above”
 Problematic where judgment is involved or
  if options not absolutely true or false
 Turns question into true/false question
 Confuses very knowledgeable students
Irrelevant Difficulty

“None of the above”
   The main reason that fertility is reduced in a
   woman in her early 40‟s is:
   A) Irregular cycles make it hard to time intercourse
   B) Chromosomal damage adversely affects oocytes‟
      genetic complement
   C) Male erectile difficulties increase
   D) Hormone imbalance leads to poor endometrial
      development
   E) None of the above
Irrelevant Difficulty
Stems are tricky or unnecessarily complicated
   Arrange the parents of the following children with Down’s syndrome
   in order of the highest to lowest risk of recurrence. Assume that the
   maternal age in all cases is 22 years and that a subsequent pregnancy
   occurs within 5 years. The karyotypes of the daughters are:
      I: 46, XX, –14,+T (14q21q) pat
      II: 46, XX, –14,+T (14q21q) de novo
      III: 46, XX, –14,+T (14q21q) mat
      IV: 46, XX, –21,+T (14q21q) pat
      V: 46, XX, –21,+T (21q21q) (parents not typed)
           A) III, IV, I, V, II
           B) IV, III, V, 1, II
           C) III, I, IV, V, II
           D) I, V, III, I, V, II
           E) III, IV, I, II, V
MCQ Check-list

Why we write them
What type is best
What flaws are to be avoided
 What content to test
 How to construct it
What to test?

Items classified by cognitive processes required to
   answer the question:
 Recall (Memory)– tests knowledge of isolated facts
 Interpretation (Comprehension)– review information
   to reach conclusion, eg diagnosis
 Problem solving (Reasoning) – situation requires
   action, eg next step in management
   – Process depends on background of trainee
What to test?
Simpler classification based on task:
 Application of knowledge item **
    – Reach conclusions
    – Make predictions
    – Select course of action
   Recall item
    – Test knowledge of isolated facts
What to test?

   Test application of knowledge using clinical
    vignettes
    – Adds „face-validity‟
    – Usually selects more important, less trivial
    – Identifies student who‟ve memorized but do not
      understand

    BUT experience in patient care should not
     be necessary
What to test - summary
   Focus item on key concepts and principles that are
    essential information (without access to
    references) for all examinees to understand
   Test material that is relevant to learning in clinical
    clerkships and beyond
   Avoid items that only require recall of isolated
    facts
   Avoid esoteric or interesting topics that are not
    essential
MCQ Check-list

Why we write them
What type is best
What flaws are to be avoided
What content to test
 How to construct it
Writing the Questions

 Construct stem –include all necessary
  information to arrive at the right answer
 Choose lead-in – pose a clear question
 Choose distractors
Stem*

 Usually a clinical scenario
 Clear, unambiguous
 Should be long relative to options
 Includes all pertinent information
    –   Patient‟s age/gender
    –   Clinical setting
    –   Complaint
    –   Other important info – history, physical
        findings, test results
Lead-in*

 Follows stem
 In the form of a question
 Should relate to the stem
 Should be answerable without looking at
  options
    – “Cover the Options test”
Distractors

 Each should be selected by some, therefore
  all plausible, none obviously incorrect
 3 or 4 choices
    – Common misconceptions
    – Faulty reasoning
Distractors

   Homogeneous in content to the correct answer
   Should be in same category as correct answer–
    diagnosis, treatment, diagnostic test
   Clearly incorrect or inferior to the correct answer
   Plausible and attractive to the uninformed
   Similar to the correct answer in construction and
    length
   Grammatically consistent and logically compatible
    with the stem
General Guidelines for Item
Construction
   Include as much of the item as possible in the
    stems: Long stems, short options
   Avoid negatively phrased items (except or not in
    the lead-in). If you must, use only with short
    options
   Avoid writing question of the form:
    – Which of the following statements is correct?
    – Each of the following statements is correct EXCEPT
       » Unfocussed
       » Heterogeneous options
“Top-down” versus “Bottom-up”

Top-Down
 Knowledge or recall question
 Cite a disease and then ask what patient
  findings are expected
 Structured like textbooks
            Clinically backward
“Top-down” versus “Bottom-up”

Bottom-up
 Application of knowledge
 Gives findings and asks examinee to
  indicate the disease
 Examinees need to be able to synthesize
  information from several pages of a
  textbook (or notes) to answer these
  questions
            Clinically realistic
Guidelines for Clinical Sciences
ie Clerkship and beyond…
 Test application of knowledge using clinical
  vignettes to pose medical decisions in
  patient care situations
 Focus items on common or potentially
  catastrophic problems
 Pose clinical decisions that would be
  expected of a successful examinee
 Avoid clinical situations that would be
  handled by a (sub)specialist
MCQ Check-list

 Why we write them
 What type is best
 What flaws are to be avoided
 What content to test
 How to construct it
Basic Rules for MCQ‟s (A type)*

 Focus item on important concept
 Item should assess application of
  knowledge, not recall of an isolated fact
 Stem must pose clear question – „Cover the
  options‟ test
 All distractors should be homogeneous
 Avoid technical item flaws
Reference
Constructing Written Test Questions For the
 Basic and Clinical Sciences
  – Case & Swanson, NBME 1996

				
DOCUMENT INFO