Document Sample
Questionnaires Powered By Docstoc
					             The following lecture has been approved for
         University Undergraduate Students

This lecture may contain information, ideas, concepts and discursive anecdotes
                that may be thought provoking and challenging

       It is not intended for the content or delivery to cause offence

  Any issues raised in the lecture may require the viewer to engage in further
                thought, insight, reflection or critical evaluation
Questionnaires & Surveys

 Improving Response rates

         Prof. Craig Jackson
 Head of Psychology Division

Education Law & Social Sciences
Are Postal Questionnaires Dead Yet?

IT predicted death of postal surveys
Use of IT at home and work increases survey methods

Comparison of surveys using      WWW      or   EMAIL     or     POSTAL

Subjects – UK university staff

200 email questionnaires
200 emails with www url
100 postal questionnaires
Asked 3 questions:       teeth cleaning
                                                  Jones, R. and Pitt, N. 1999
Are Postal Questionnaires Dead?

Days after sending     email          www           post
                       n=200          n=200         n=100
                               numbers responding

3                      25             31            0
5                      59             34            16
9                      61             35            53
10                     63             35            60
17                     68             37            72
17 day response rate   34%            19%           72%

                                 cost per reply

Actual cost            35p            41p           92p
With 100% response     19p            7.5p          72p
What this means for surveys

Golden age of communication?

Postal methods still much better

        Novelty value of email is dead - therefore lower response rates

        Junk mail perceptions

        Email Filters are improving

        Postal letters demonstrate & emphasize their importance

www surveys & email allow immediate data processing (software)

Email & www have potential for low cost regular user surveys

Intranet users benefit from e-surveys

Questionnaire is a fundamental component of most research projects
Most MSc / PhD projects use questionnaire methods
Largest reason for criticising projects - weak / dubious questionnaires
Can be very efficient

A Questionnaire does:                                  A Questionnaire is not:
Take planning to get right                          Constructed in 10 minutes
Reward time spent on it                                        An easy option

Capture information                           A collection of simple questions
Basics of Measurement

Measurement tools must be appropriate
       Psychometrics -- personality, attitudes, stress, symptoms, Physical
measurements -- working, environment, symptoms
       Exposure assessments -- hazards, risks, ppm3, duration

When measuring...
       Take multiple measurements (and take the mean)
       Under same conditions, but if not….
               Statistical remedies to adjust e.g. age, time of day etc.
       Reliable & Validated tool
       Defined and Regular variables
       Well-defined standard of reference
     Achieving a high response rate to a questionnaire is vital
     But does not promise a normal distribution of responses?

Postal questionnaires rarely get a response rate > 40%.
Unless respondents have a vested interest in the outcome.

Most efficient (best) response rates usually happen when respondents have to
do very little to take part in the study.
Multiple phase projects see a depletion in numbers at every stage.
Quick “in and out” one-stop approach is best
Know your enemy
Diminishing returns of multi-stage recruitment

           Researcher                   Potential Sample
                                          1000 people

                                         540 consents

                  540 questionnaires

Under-powered study
      n = 210
Response rate of 21%                   210 questionnaires

Identifying Items



Branch Instructions

Research Items

does not need to be honest
some deception is necessary
shorter is better
Identifying items

Preliminary questions

Collecting info necessary for screening:
recording keeping
 data manipulation

Ask only for relevant info - unethical

Fewer items minimizes chance of alienating respondent

       The need for rich info                  Participant‟s need for
       to improve the study                    privacy & anonymity
Encryption devices

Secret communication of a message by hiding it‟s existence
Steganos, meaning covered. Gk
Graphein, meaning to write, Gk
If message is discovered it is easily read because of no encryption

Secret communication of a message by hiding it‟s meaning
Kryptos, meaning hidden. Gk
Message established using a known protocol, to be decrypted by the receiver

Steganography & Cryptography can be combined together if needed
Steganogrpahy arouses less suspicion in questionnaire respondents
Encryption devices

“This example of steganogrpahy may not work very well when projected onto a
large screen, but it works very well on paper, such as questionnaires.
Hopefully many of you will not notice the method of steganography used in
this piece of text. Gosh how clever I am……”

drbjh kbdltpo, jotujuvuf pg pddvqbujpobm ifbmui

The above text (containing an encrypted name and address) looks suspicious
and may be obliterated by the respondent

                                                             drbjh kbdltpo, jotujuvuf pg pddvqbujpobm ifbmui

                                               An important introduction

                                                      Frame of reference

     Without a FOR a respondent may base their answer on a wrong context

“We would like to know if you have had any medical complaints
and how your health has been in general, over the past few weeks.
Please answer ALL the questions on the following pages by ticking
the answer which best applies to you. Remember that we want to
know about present and recent complaints, not those that you had
in the past. It is important that you try to answer all questions”

Not to be underestimated – try to be user-friendly

Sets the tone for the respondent

The General Health Questionnaire vs The GHQ 28

                                    “Simple language for simple people”

Beware of abbreviated titles - may alienate respondents
e.g.    Agricultural Satisfaction Scale
        HAD etc
Research items

Set(s) of items to collect the “real” research data

Take many forms:

                 Item Type                  Data type

                 Fixed option items         (ordinal or nominal)
                 Rating scale items         (likert)
                 Yes / No items             (binary)
                 True / False items         (binary)
                 Likert scale items         (likert)
                 Diagrams                   binary, ordinal, or nominal
Branch Instructions

Guide the respondent through the items

Maximises the efficiency of respondent‟s efforts

Avoids redundant items

Focuses on worthwhile items

Instructions can sometimes confuse the respondent so think carefully

“If you answered „NO‟ to question 9, ignore 9b, 9c, and 9d, go to question 10.”
Ah, the Truth
Are your respondents telling the truth?

Use a sprinkling of LIE DETECTOR items to assess reliability

“Have you ever taken anything without asking permission?”
“Did you ever lie to your parents as a child?”                        increases
                                                                       with item
“Have you ever visited a pornographic web-site?”                       intimacy

        Decide what to do with any respondent who “fails” the lie detector

       Keep them in?     Exclude immediately?       Retain but “flag” them?
Funnel Items

Set funnel questions to try to narrow down a respondent to specific details


6a “Did you write essays in college?”    Yes / No

6b “What type of essays were they mostly?”

                                         Narrative / Descriptive / Persuasive

6c “How often did you write them?”       ___ per term

                                         1 per term / 2-3 per term / 3 or more
Response Set Bias

Enables collection of detailed data easily

Build a detailed data set from a simple binary item

Randomise the layout / order of responses “true / false”

        True         
        True     False
                     Respondents ticking the same items

                Swap the “True & False” responses sequence
Likert scales

A visual linear scale for rating purposes

                                                X                    sad

              Swap around the words to avoid response-set bias

           Avoid using numbers on the scale -- can be (mis)leading
Likert scales

  XX    X       XX
                   X   X XX X   X   XXX X
  X               X                 X
happy                                       sad
Likert scales

    XX X        XX   XX X X   X
  happy                           sad
Design Pointers

Avoid colloquialisms or abbreviations

Beware of local expressions

Avoid words with double meanings         e.g.”Fair....Dip....Lie....Well”

Set a definition of specific terms       e.g. “OK.....Average”

Avoid long questions

Specify exact time, place and context    e.g “At school, did you ever...”

Phrase items to make denial impossible e.g “When did you first...”

Avoid numbers on your responses / scales

Have items seemingly related to the research topic
Design Pointers

Clearly phrase items
Make items unambiguous
Avoid leading items                                 e.g. “Many people think...”
Ask only what a respondent is qualified to answer
Avoid socially loaded items                         e.g. “beggars” or “junkies”
Do not use socially / religiously biased items
Avoid phrases, clichés or sooth-sayings
Ask for ages as a true number / D.O.B - - NOT in age groups
Most common errors

  “In your organization, do women have the same
  responsibilities as men, and should women have more?”

  “Preventing accidents in the workplace is vital, and more
  money should be spent on prevention.”

  “Training in risk assessment is something I would like to do,
  and I would like to see my colleagues do it to.”

                    Only ask one question per item
 Most common errors

 What is your age ? (please tick)
 16 - 26         26 - 36          36 - 46           46 - 56

 What is your marital status ? (please tick)
 Married     Single

 How many blood splashes have you had ? (please tick)
 1-5    6 - 10   11 - 15  15 - 20   20 plus

 Are life preservers and flares essential on-deck equipment ?

agree                                                           disagree
Factorial approach

The GHQ 28
A self-completion questionnaire assessing mental health

28 items
7 about Anxiety                           Anxiety score               3
7 about Severe depression                 Depression score            4
7 about Dysfunction                       Dysfunction score           5
7 about Somatic symptoms                  Somatic score               2
                                          Global score               14

By summing the 4 factors there is a Global Mental Health score

Statistics are performed on the factor scores and the global score

NOT on each individual item
Designing factors

Piloting items
Set down as many items as
possible concerning the topic

Limit the questionnaire to 30

Give the questionnaire to at
least 20 people

Score responses and place
into SPSS spreadsheet

Factor Analysis
use statistics to “group” items together into factors
(will do this as a group exercise in the SPSS lecture)
Creating factors

Factor Analysis
        Results in manageable number of factors instead of 30 items

        Each factor comprised of a number of items that had similar answers
        from your pilot sample

        The similarity of responses to particular items is what SPSS uses to
        group items together into factors

Identifying Items
As a matter of routine:
Get as much information as possible
         may be vital in later (unforeseen analyses)
         DOB or Age (not age groups)
         Marital status / Domestic arrangements
Increasing Response Rates







  Increasing Response Rates - Incentives



  Prize draw

  Ethical aspects


  Monetary incentive vs. None                 2.02
  Incentive with Q. vs. Incentive on return   1.71
  Non-monetary incentive vs. No incentive     1.19
Edwards et al. 2002
  Increasing Response Rates - Appearance


  Shorter format vs. Longer format     1.86

  Brown envelope vs. White             1.52

  Coloured ink vs. Black               1.39

  Folder / Booklet vs. Stapled pages   1.17

  Personalised vs. Not personalised    1.16

  ID feature on return vs. No ID       1.08

  Coloured Q vs. White Q               1.06
Edwards et al. 2002
  Increasing Response Rates – Delivery methods


  Recorded delivery vs. Standard                         2.21

  Stamped return envelope vs. Business reply / franked   1.26

  Q sent to work vs. Q sent to home                      1.16

  1st class outward mail vs. Other class                 1.12

  Pre-paid return envelope vs. Not pre-paid              1.09

  Stamped outward envelope vs. Franked                   0.95

  Commemorative stamp vs. Ordinary stamp                 0.92

Edwards et al. 2002
  Increasing Response Rates – Origin & Contact

  Origin                                            O.R
  University vs. Other organisation                 1.31
  Sent by senior persons vs. Juniors                1.13
  Ethnically ambiguous name vs. Non-white name      1.11

  Contact                                           O.R
  Pre-contact vs. No contact                        1.54
  Follow up vs. No follow up                        1.44
  Postal follow up with Q vs. Without Q             1.41
  Mentioning follow up vs. None                     1.04
  Pre-contact by telephone vs. Postal pre-contact   0.90
Edwards et al. 2002
  Increasing Response Rates - Content


  More interesting vs. Less interesting             2.44

  User-friendly vs. Standard                        1.46
  Factual items only vs. Factual & attitude items   1.34
  Relevant items first vs. Other items              1.23
  Demographic items first vs. Other items           1.04
  “Don‟t know” boxes vs. no “Don‟t know” boxes      1.03
  Sensitive items vs. No sensitive items            0.92
  General items first vs. Last                      0.80

Edwards et al. 2002
  Increasing Response Rates - Communication


  Explain drop-out required vs. Not           1.32

  Stresses benefit to respondent vs. Others   1.06

  Stresses benefit to sponsors vs. Others     1.01

  Stresses benefit to society vs. Others      1.00

  Response deadline given vs. No deadline     1.00

  Instructions given vs. No instructions      0.89

  Choice to opt out given vs. No opt out      0.76

Edwards et al. 2002
Questionnaire Summary

1. Postal questionnaires widely used in data collection

2. Mark pre-pay or addressed envelopes

3. Use steganography over cryptography

4. Perform analyses on factors not individual items

5. Identifying items may be useful in later analyses

6. Think about scoring items before rushing ahead with the questionnaire

7. Store questionnaires securely until passing any viva voce

8. Collect as much info at source, parse it down later at discretion
Questionnaire Summary cont.

9. Perfect for epidemiological studies and health research

10. Non-response to postal questionnaires reduces sample & introduces bias

11. Identification of effective ways to increase postal response rates

12. Use existing metrics - pilot items if making new questionnaire

13. Use factor analysis

14. Keep it all as brief as possible

15. Don‟t alienate respondents

16. Alternate types of items

Shared By: