Christine Coombe Teacher Tester Toastmaster

Document Sample
Christine Coombe Teacher Tester Toastmaster Powered By Docstoc
					Questionnaire Development

            Christine Coombe
           Dubai Men‟s College
Presentation Agenda
• What are questionnaires and what do
  they measure?
• Objectives in questionnaire research
• Advantages/Disadvantages
• Main parts and formats
• Do‟s/Don‟ts of questionnaire writing
• Questionnaire administration
• Issues in questionnaire research
• Ethical principles of questionnaire
What are Questionnaires?
• Questionnaires are any written
  instruments that present respondents
  with a series of questions or statements
  to which they are to react either by
  writing out their answers or selecting
  them from among existing answers.
  (Brown, 2001:6)
• Referred to by different names
  – Inventories, forms, opinnionaires, tests,
    batteries, checklists, scales, surveys,
    profiles, indexes (Aiken, 1997)
What do Questionnaires
• Questionnaires (Qs) can yield different
  types of data about respondents
  (Dornyei, 2003)
  – Factual questions
     • Find out information about who respondents are;
  – Behavioral questions
     • Find out what respondents are doing or have done
       in the past
  – Attitudinal questions
     • Find out what people think
     • Attitudes, opinions, beliefs, interests & values
Objectives in Questionnaire
 To obtain accurate and relevant
• To maximise the response rate (the
  proportion of subjects answering
  the Q)
  – To do this we must carefully consider
    how we administer the instrument,
    establish rapport, explain the purpose
    of the survey and remind those who
    have not as yet responded
Deciding What to Ask
• Information we are primarily
  interested in (dependent variables)
• Information which might explain the
  dependent variables (independent
• Other factors related to DV and IV
  factors which may distort the results
  and have to be adjusted for
  (confounding variables)
• Unprecedented efficiency in terms of
  researcher time, effort and financial resources
  (Dornyei, 2003)
• Versatility– can be used successfully with a
  variety of people in a variety of contexts
  targeting a variety of topics
• Collect a large amount of data in a short time
  (Brown, 1988)
• Easier and less expensive than other forms of
  data collection (Seliger & Shohamy, 1989)
• Can be used to research any aspect of teaching
  and learning (Nunan, 1989)
• Can be easily used in field settings such as
  classrooms (Nunan, 1992)
• Simplicity and superficiality of answers
   – Qs must be understood by all; written in
     simplistic way; can‟t probe deeply into an
     issue; results in superficial data (Moser &
     Kalton, 1971)
• Unreliable and unmotivated respondents
   – Results vary greatly from person to person
• Respondent literacy problems
   – Assumption that all can read and write well
• Social desirability or prestige bias
   – People do not always provide true answers
     about themselves
   – Happens because Qs are often transparent
     (i.e. Rs have a fairly good idea of what the
     desirable/acceptable answer is)
      • Some will provide this response even if it is not
   – Presenting ourselves in a good light is a
     natural human tendency
• Self-deception
• Acquiescence bias
   – Tendency for people to agree with sentences when
     they are unsure or ambivalent
   – Go along with anything that sounds good
• Halo effect
   – Concerns the human tendency to overgeneralize
   – If our overall opinion about someone/thing is positive
     we are disinclined to say anything negative
• Fatigue effects
   – Kicks in if Q is overly long or monotonous
Main Parts of Questionnaires

• Title
   – To provide Rs with initial orientation & activate
     content schemata (Dornyei, 2003)
   – Qs with titles are generally perceived as more
     serious/credible than those without
• Instructions: Two Types
   – General: opening greeting at beginning of Q
   – Should contain
          •   What the study is about and why it‟s important
          •   Organization responsible for conducting study
          •   Emphasis that there are no „right‟ or „wrong‟ answers
          •   Promise of confidentiality
          •   A „thank you‟
Main Parts of Questionnaires

• Instructions
   – Specific
   – Explains and demonstrates how Rs should
     answer questions
   – Each new task type requires instructions
• Questionnaire Items
• Additional Information
   – Contact details of researcher; how Qs
     should be returned; note promising a copy of
     results; invitation for follow-up interview
• Final „thank you‟
Open or Closed Formats?

• Responses can be in open or
  closed formats
  – Open-ended formats
     • Rs formulate their own answers
  – Closed formats
     • Rs are forced to choose between several
       given options
• It is possible to use a mixture of
  these formats
Open or Closed Formats:
• Open-ended formats      • Closed (or forced)
• Allows exploration of     choice format
  the range of possible   • Easy and quick to fill
  themes arising from       in
  an issue                • Minimise
• Can be used if a          discrimination
  comprehensive             against the less
  range of alternative      literate or the less
  choices cannot be         articulate
  compiled                • Easy to code, record,
                            analyze results
                          • Easy to report results
Closed or Forced Choice
• Choice of Categories
   – What is your marital status?
   – Circle/tick: single, married, divorced,
• Likert Scale
   – Language assessment is an interesting
   – Circle/tick SD D NA or D A SA
• Differential Scale
   – How would you rate this presentation?
   – Very Boring 1 2 3 4 5 6 7 8 9 10 Very
Closed or Forced Choice
• Checklists
  – Circle the classroom activities you are
    particularly interested in
     •   Drill and practice
     •   Cloze
     •   Dictation
     •   Pair work
     •   Group work
     •   Silent reading
Closed or Forced Choice
• Ranking
  – Please rank your interest in the following
    subjects. (1=most interesting; 5=least
     •   Listening
     •   Reading
     •   Writing
     •   Speaking
     •   Grammar
     •   Vocabulary
  – Probably the least used format because of
    difficulty with analysis
Open-ended Questions
• Specific open questions
   – Ask about concrete pieces of information
     (i.e. facts, past activities, preferences)
   – How many languages have you studied?
• Clarification questions
   – Questions that are so important you need
     follow-up information
   – If you rated the textbook you are using as
     „poor‟ or „very poor‟, please briefly explain
Open-ended Questions

• Sentence completion questions
  – One thing I liked about this test
  – I found this activity……
• Short-answer questions
  – Exploratory inquiry about an issue
  – What was it that you found most
    useful about this workshop?
• General temptation is always to cover
  too much ground by asking everything
  that might turn out to be interesting
• Experts agree that anything over 4-6
  pages and requires more than 30 min to
  complete is too long
• As a general rule, long questionnaires
  get fewer responses than short Qs
Question Arrangement

• Go from general to particular
• Go from easy to difficult
• Go from factual to abstract
• Start with closed format questions
• Start with questions relevant to the
  main subject
• Do not start with demographic and
  personal questions
How to Write Good Items
• Most problems with Qs can be traced back to the design
  phase of the project.
    – Ask only Qs that address the study goals
• Use short and simple sentences.
    – Generally less confusing & ambiguous than long, complex
    – As a rule of thumb, most sentences should contain one or
      two clauses.
        • Sentences with more than three clauses should be rephrased.
    – Should not exceed 20 words
• Ask for only one piece of information at a time.
    – “Please rate the lecture in terms of its content and
      presentation” asks two questions.
• Avoid negatives, if possible or use them sparingly.
    – Double negatives can be very problematic.
    – Do you agree with this statement, “team teaching should not
      be abolished.”
How to Write Good Items
• Ask precise questions.
    – Qs may be ambiguous because a word or a term may have
      a different meaning.
• Ask for the appropriate level of detail.
• Keep the Q brief and concise.
    – Avoid peripheral Qs and one to find out “something that
      might be nice to know”
    – clear cut need for every question should be established
• Get feedback on your initial list of Qs
    – Include experts in the Q design process
    – feedback may be obtained from a small but representative
      sample of potential respondents
    – field test of instrument is also desirable
• Locate personal or confidential questions at the end of
  the Q
    – early appearance of unsettling Qs may result in
      respondents discontinuing the Q
How to Write Good Items
• Order categories
   – when response categories represent a progression between
     the lower level of response and a higher one, it is usually
     better to list them from the lower level to the higher level in
     left-to-right order
• Consider combining categories
   – seldom or never/occasionally/frequently
   – combining is desirable if respondent would be unlikely to
     mark „never‟; if you really want to distinguish between
     seldom and never
       • like in the Q “how often do you drink alcohol?”
• Ask respondents to rate both positive and negative
   – sometimes difficult with „apple pie‟ problem where
     respondents are asked to rate items for which the general
     level of approval is high
   – tendency to mark every item at the same end of the scale
   – by offering positive/negative responses Rs must evaluate
     every statement
How to Write Good Items
• Begin with non-threatening and interesting
• Leave adequate space for comments
• Place most important Qs in the first half of Q
   – Ss often send back partially completed Qs
• Hold the Ss interest
   – provide a variety of item types; this will prevent
     „response sets‟
• Provide incentives as a motivation for
  completion (offering a free summary is often an
• Make it convenient
   – envelopes with self-addressed return postage stamps
     get better response rates
What to Avoid
• Avoid the response option „other‟
    – some will overlook the option they should have designated
      and mark „other‟ or will be hairsplitters and reject an option
      for some trivial reason
• Avoid category proliferation
    – Ex: single (never married)
        •   Married
        •   Widowed
        •   Divorced
        •   Separated
    – Unless research deals with conjugal relationships,
      distinctions are not useful
    – Usually this demographic info only needs to distinguish
      between a conventional family setting or anything else so
      you can use
        • Married and living with spouse
        • Other
What to Avoid
• Avoid scale point proliferation
   – Never/rarely/occasionally/fairly often/often/very
     often/almost always/always
• Psychometric research has shown that most Ss
  cannot reliably distinguish between more than
  six or seven levels of response
   – four to five scale points are usually quite sufficient
• Distinguish between a neutral and no response
   – when you don‟t want to have a neutral option you can
     sometimes have a “prefer not to answer”, “not
     applicable”, or “no basis for judgement”
What to Avoid

• Avoid asking Rs to rank
  – Rs cannot be reasonably expected to
    rank more than about six things at a
    time; confusion sometimes occurs
  – Try the following
    • Which color do you like best?
    • Which color do you like second best?
    • Which color do you like least?
Qualities of Good
• Evokes the truth
  – Qs must be non-threatening
  – if your Q has sensitive items, be sure to
    clearly state your policy on confidentiality
• Ask for an answer on only one
  – ex. Where you satisfied with the time and
    location of your class?
• Can accommodate all possible answers
• Has mutually exclusive options
Qualities of Good
• Produces variability of responses
• Follows comfortably from the previous
  – transitions between Qs should be smooth
  – grouping Qs that are similar will make the Q
    easier to complete
• Do not presuppose a certain knowledge
• Do not imply a desired answer
  – “Don‟t you think that the textbook should be
• Do not use emotionally loaded or
  vaguely defined words; unfamiliar words
  or abbreviations
Questionnaire Administration

• Sample
  – Difference between sample and population
• Different types of samples
  – Convenience/opportunity: most common in
    L2 research
  – Snowball: where researcher asks a group to
    recommend others who have similar
  – Quota: defines certain subgroups within a
    population and samples according to
    population (i.e. male vs females in UAE
    tertiary education)
  – Random: selection by random basis
Questionnaire Administration

• How large should the sample size be?
  – No hard/fast rules
  – 10% of population is generally ideal
  – Basic requirement is normal distribution
    which equates to at least 30 Ss (Hatch &
    Lazarton, 1991)
  – For statistical significance at least 50
  – For multivariate statistical procedures like
    factor analysis at least 100
Issues in Questionnaire
• Most common way to administer is by
• The cover letter
  – Necessary to „sell‟ the Q & create rapport
    with Rs
  – What they should include?
  – Research on cover letters
     • Literature regarding personalization is mixed
     • Ethnic sounding names and status of researcher
       has no effect on response rate
     • Handwritten vs scanned signatures no effect
Ethical Principles of
Questionnaire Research
• Five Ethical Principles (Oppenheim,
  1992; Sudman & Bradburn, 1983)
  – No harm should come to Rs as a result of
    their participation
  – Rs right to privacy should always be
  – Rs should be provided with sufficient info to
    complete the Q
  – In the case of children, permission should be
    sought from caretakers
  – Researcher should not promise a higher
    degree of privacy and confidentiality than
    he/she can deliver
Issues in Questionnaire
•   Rs that tend to “sit the fence”
     – relevant for Likert scales with odd numbered response options
•   Sensitive issues
     – Difficult to obtain truthful answers to sensitive questions.
     – Consider the example on cheating:
          • “Have you ever copied other students‟ answers on an exam?”
                 – will most likely produce either no response or negative
          • The Casual Approach: “By the way, do you happen to have
             copied other students‟ answers on an exam?”
          • The Numbered Card Approach: “Please tick one or more of the
             following items which correspond to how you have answered
             exam questions in the past”
                 – In the list of items, include “copy from other student” as
                   one of the list
          • The Everybody Approach: “As we all know, most students
             have copied other students‟ answers on exams. Do you
             happen to be one of them?”
          • Other People Approach: Ss given a scenario, “John copies
             answers on an exam from Jean.”
                 – then asked, “Do you feel John is wrong, what penalty
                   should be imposed for John, and have you done or would
                   you consider doing the above?”
• Qs must be carefully designed to
  yield valid information.
• Attention must be paid to ensure
  that individual questions are
  relevant, appropriate, intelligible,
  precise and unbiased.
• The order of Qs must be carefully
  arranged and the layout of the Q
  must be clear.
Final Thoughts

• In writing questionnaire items

”no amount of textbook admonition
  can take the place of common
  sense.” (Moser & Kalton, 1971:310)