Learning Center
Plans & pricing Sign in
Sign Out




       Alfred Kobsa
University of California, Irvine
                       When to chose a survey

• Scope of problem very clear
• Questions can be clearly formulated at a concrete level
• Complex/holistic answers not needed. Checking pre-
  formulated answers is sufficient (plus occasionally one or
  two free-text sentences).
• Mailing lists with current addresses available
• Specific location not required
• Answers from many users needed
• Answers needed quickly
• Broad geographic distribution of respondents
• Anonymity required
• Identify the objectives of the study (in writing)
• Select the target audience
• Select the form of announcement and response collection
 (email, physical mail, web [☛survey websites], in person, phone)
• Decide how to analyze the data (software, statistical tests if any)
• Brainstorm questions (closed questions, open questions)
• Formulate questions (mind double-barreled, double negatives,
 leading or loaded questions, and questions with self-image,
 acquiescence and social desirability bias)
• Use existing surveys (e.g. System Usability Scale SUS        )
• Reduce question set
• Pilot test and revise your questions!
                            Problems to consider
• Non-representative recipient population (☛ compare
  demographics of respondents with demographics of names in
  mailing list and normalize answers accordingly)
• Low response rate (☛ pre-announcement, personalized cover
  letter, reward, short survey, few open-ended questions, self-
  addressed envelope w/ stamp, multi-mode reminders)
• Self-selection bias of respondents based on topic (☛ hide your
  study goal among more general questions)
• Honesty (☛ reverse-code questions, check consistency, discard
• Self-incrimination (☛ emphasize “harmlessness”, impunity,
• Social desirability bias, self-image bias (☛ phrase questions
  carefully, use several questions for same concept, go for facts
  and not habits, attitudes or intentions)
• Sensitive questions (☛ explain their purpose, do not make
  answers mandatory, offer broad answer bins)
                                                        Response format
• Closed-ended questions
     • Single-choice or multiple-choice (include “other” + “explain”)
     • scalar value (“on a scale from 1 to 5, how would you rate…”)
     • Likert* scale: [1 … n] or [-n -n+1…. n-1 n] (typically n = 5, 6 or 7)
            with mid-point, or without (forces respondents to take a stance)
            with verbal “anchors” for first and last value only, or for
            all/some values including first and last**
     • Ranking scales (e.g., sorting by priority)
• Open-ended questions
  Should be kept to a minimum and optional since
   • they may cause users not to fill in or abandon a questionnaire
   • if users answer them, they only provide brief answers
   • answers are hard to evaluate
*) pronounced like “ike” in the US, but like “lickert” elsewhere (also by Mr Likert himself and his son)
**) First and last value anchors should be “extreme”

To top