Docstoc

Questionnaires

Document Sample
Questionnaires Powered By Docstoc
					             The following lecture has been approved for
         University Undergraduate Students

This lecture may contain information, ideas, concepts and discursive anecdotes
                that may be thought provoking and challenging


       It is not intended for the content or delivery to cause offence


  Any issues raised in the lecture may require the viewer to engage in further
                thought, insight, reflection or critical evaluation
Questionnaires & Surveys

                 Delivery
                  Design
             Development
 Improving Response rates




         Prof. Craig Jackson
 Head of Psychology Division

Education Law & Social Sciences
                           BCU
Are Postal Questionnaires Dead Yet?

IT predicted death of postal surveys
Use of IT at home and work increases survey methods


Comparison of surveys using      WWW      or   EMAIL     or     POSTAL

Subjects – UK university staff


200 email questionnaires
200 emails with www url
100 postal questionnaires
Asked 3 questions:       teeth cleaning
                         fruit
                         walking
                                                  Jones, R. and Pitt, N. 1999
Are Postal Questionnaires Dead?
Results

Days after sending     email          www           post
                       n=200          n=200         n=100
                               numbers responding

3                      25             31            0
5                      59             34            16
9                      61             35            53
10                     63             35            60
17                     68             37            72
17 day response rate   34%            19%           72%

                                 cost per reply

Actual cost            35p            41p           92p
With 100% response     19p            7.5p          72p
What this means for surveys

Golden age of communication?

Postal methods still much better

        Novelty value of email is dead - therefore lower response rates

        Junk mail perceptions

        Email Filters are improving

        Postal letters demonstrate & emphasize their importance

www surveys & email allow immediate data processing (software)

Email & www have potential for low cost regular user surveys

Intranet users benefit from e-surveys
Introduction

Questionnaire is a fundamental component of most research projects
Most MSc / PhD projects use questionnaire methods
Largest reason for criticising projects - weak / dubious questionnaires
Can be very efficient




A Questionnaire does:                                  A Questionnaire is not:
Take planning to get right                          Constructed in 10 minutes
Reward time spent on it                                        An easy option

Capture information                           A collection of simple questions
Basics of Measurement

Measurement tools must be appropriate
       Psychometrics -- personality, attitudes, stress, symptoms, Physical
measurements -- working, environment, symptoms
       Exposure assessments -- hazards, risks, ppm3, duration

When measuring...
       Take multiple measurements (and take the mean)
       Under same conditions, but if not….
               Statistical remedies to adjust e.g. age, time of day etc.
       Reliable & Validated tool
       Defined and Regular variables
       Well-defined standard of reference
Dilemma
     Achieving a high response rate to a questionnaire is vital
     But does not promise a normal distribution of responses?




Postal questionnaires rarely get a response rate > 40%.
Unless respondents have a vested interest in the outcome.
Bias?

Most efficient (best) response rates usually happen when respondents have to
do very little to take part in the study.
Multiple phase projects see a depletion in numbers at every stage.
Quick “in and out” one-stop approach is best
http://onlinestatbook.com/stat_sim/sampling_dist/index.html
Know your enemy
Diminishing returns of multi-stage recruitment


           Researcher                   Potential Sample
                                          1000 people




                                         540 consents

                  540 questionnaires



Under-powered study
      n = 210
Response rate of 21%                   210 questionnaires
Structure

Identifying Items

Title

Preamble

Branch Instructions

Research Items




does not need to be honest
some deception is necessary
shorter is better
Identifying items

Preliminary questions

Collecting info necessary for screening:
recording keeping
tracking
 tracing
 data manipulation

Ask only for relevant info - unethical

Fewer items minimizes chance of alienating respondent

       The need for rich info                  Participant‟s need for
       to improve the study                    privacy & anonymity
Encryption devices

Steganography
Secret communication of a message by hiding it‟s existence
Steganos, meaning covered. Gk
Graphein, meaning to write, Gk
If message is discovered it is easily read because of no encryption

Cryptography
Secret communication of a message by hiding it‟s meaning
Kryptos, meaning hidden. Gk
Message established using a known protocol, to be decrypted by the receiver

Steganography & Cryptography can be combined together if needed
Steganogrpahy arouses less suspicion in questionnaire respondents
Encryption devices

Steganography
“This example of steganogrpahy may not work very well when projected onto a
large screen, but it works very well on paper, such as questionnaires.
Hopefully many of you will not notice the method of steganography used in
this piece of text. Gosh how clever I am……”


Cryptography
drbjh kbdltpo, jotujuvuf pg pddvqbujpobm ifbmui


The above text (containing an encrypted name and address) looks suspicious
and may be obliterated by the respondent


                                                             drbjh kbdltpo, jotujuvuf pg pddvqbujpobm ifbmui
                                                            Preamble

                                               An important introduction

                                                      Frame of reference

     Without a FOR a respondent may base their answer on a wrong context



“We would like to know if you have had any medical complaints
and how your health has been in general, over the past few weeks.
Please answer ALL the questions on the following pages by ticking
the answer which best applies to you. Remember that we want to
know about present and recent complaints, not those that you had
in the past. It is important that you try to answer all questions”
Title

Not to be underestimated – try to be user-friendly

Sets the tone for the respondent

The General Health Questionnaire vs The GHQ 28

                                    “Simple language for simple people”

Beware of abbreviated titles - may alienate respondents
e.g.    Agricultural Satisfaction Scale
        LEI
        BDI
        MMPI
        HAD etc
Research items

Set(s) of items to collect the “real” research data

Take many forms:

                 Item Type                  Data type

                 Fixed option items         (ordinal or nominal)
                 Rating scale items         (likert)
                 Yes / No items             (binary)
                 True / False items         (binary)
                 Likert scale items         (likert)
                 Diagrams                   binary, ordinal, or nominal
Branch Instructions

Guide the respondent through the items

Maximises the efficiency of respondent‟s efforts

Avoids redundant items

Focuses on worthwhile items

Instructions can sometimes confuse the respondent so think carefully

e.g
“If you answered „NO‟ to question 9, ignore 9b, 9c, and 9d, go to question 10.”
Ah, the Truth
Are your respondents telling the truth?

Use a sprinkling of LIE DETECTOR items to assess reliability

e.g
                                                                     Respondent
“Have you ever taken anything without asking permission?”
                                                                      “Outrage”
“Did you ever lie to your parents as a child?”                        increases
                                                                       with item
“Have you ever visited a pornographic web-site?”                       intimacy



        Decide what to do with any respondent who “fails” the lie detector

       Keep them in?     Exclude immediately?       Retain but “flag” them?
Funnel Items

Set funnel questions to try to narrow down a respondent to specific details

e.g

6a “Did you write essays in college?”    Yes / No



6b “What type of essays were they mostly?”

                                         Narrative / Descriptive / Persuasive


6c “How often did you write them?”       ___ per term

                                         1 per term / 2-3 per term / 3 or more
Response Set Bias

Enables collection of detailed data easily

Build a detailed data set from a simple binary item

Randomise the layout / order of responses “true / false”

e.g
        True
                     
                 False
        True         
                 False
        True
        True
                    
                 False
                 False
        True     False
                     Respondents ticking the same items

                Swap the “True & False” responses sequence
Likert scales

A visual linear scale for rating purposes




  happy
                                                X                    sad




              Swap around the words to avoid response-set bias

           Avoid using numbers on the scale -- can be (mis)leading
Likert scales




  XX    X       XX
                 X
                   X   X XX X   X   XXX X
                                            X
                                            X
  X               X                 X
happy                                       sad
Likert scales




                          X
    XX X        XX   XX X X   X
                 X
  happy                           sad
Design Pointers

Avoid colloquialisms or abbreviations

Beware of local expressions

Avoid words with double meanings         e.g.”Fair....Dip....Lie....Well”

Set a definition of specific terms       e.g. “OK.....Average”

Avoid long questions

Specify exact time, place and context    e.g “At school, did you ever...”

Phrase items to make denial impossible e.g “When did you first...”

Avoid numbers on your responses / scales

Have items seemingly related to the research topic
Design Pointers

Clearly phrase items
Make items unambiguous
Avoid leading items                                 e.g. “Many people think...”
Ask only what a respondent is qualified to answer
Avoid socially loaded items                         e.g. “beggars” or “junkies”
Do not use socially / religiously biased items
Avoid phrases, clichés or sooth-sayings
Ask for ages as a true number / D.O.B - - NOT in age groups
Font
Pictures
Most common errors

  “In your organization, do women have the same
  responsibilities as men, and should women have more?”


  “Preventing accidents in the workplace is vital, and more
  money should be spent on prevention.”


  “Training in risk assessment is something I would like to do,
  and I would like to see my colleagues do it to.”

                    Only ask one question per item
 Most common errors

 What is your age ? (please tick)
 16 - 26         26 - 36          36 - 46           46 - 56


 What is your marital status ? (please tick)
 Married     Single



 How many blood splashes have you had ? (please tick)
 1-5    6 - 10   11 - 15  15 - 20   20 plus


 Are life preservers and flares essential on-deck equipment ?

agree                                                           disagree
Factorial approach

The GHQ 28
A self-completion questionnaire assessing mental health

How?
28 items
7 about Anxiety                           Anxiety score               3
7 about Severe depression                 Depression score            4
7 about Dysfunction                       Dysfunction score           5
7 about Somatic symptoms                  Somatic score               2
                                          Global score               14

By summing the 4 factors there is a Global Mental Health score

Statistics are performed on the factor scores and the global score

NOT on each individual item
Designing factors

Piloting items
Set down as many items as
possible concerning the topic

Limit the questionnaire to 30
items

Give the questionnaire to at
least 20 people

Score responses and place
into SPSS spreadsheet

Factor Analysis
use statistics to “group” items together into factors
(will do this as a group exercise in the SPSS lecture)
Creating factors

Factor Analysis
        Results in manageable number of factors instead of 30 items

        Each factor comprised of a number of items that had similar answers
        from your pilot sample

        The similarity of responses to particular items is what SPSS uses to
        group items together into factors

Identifying Items
As a matter of routine:
Get as much information as possible
         may be vital in later (unforeseen analyses)
         Sex
         DOB or Age (not age groups)
         Marital status / Domestic arrangements
Increasing Response Rates

Incentives

Appearance

Delivery

Origin

Contact

Content

Communication
  Increasing Response Rates - Incentives

  Money

  Vouchers

  Prize draw

  Ethical aspects

  Bias
                                              O.R

  Monetary incentive vs. None                 2.02
  Incentive with Q. vs. Incentive on return   1.71
  Non-monetary incentive vs. No incentive     1.19
Edwards et al. 2002
  Increasing Response Rates - Appearance

                                       O.R

  Shorter format vs. Longer format     1.86

  Brown envelope vs. White             1.52

  Coloured ink vs. Black               1.39

  Folder / Booklet vs. Stapled pages   1.17

  Personalised vs. Not personalised    1.16

  ID feature on return vs. No ID       1.08

  Coloured Q vs. White Q               1.06
Edwards et al. 2002
  Increasing Response Rates – Delivery methods

                                                         O.R

  Recorded delivery vs. Standard                         2.21

  Stamped return envelope vs. Business reply / franked   1.26

  Q sent to work vs. Q sent to home                      1.16

  1st class outward mail vs. Other class                 1.12

  Pre-paid return envelope vs. Not pre-paid              1.09

  Stamped outward envelope vs. Franked                   0.95

  Commemorative stamp vs. Ordinary stamp                 0.92


Edwards et al. 2002
  Increasing Response Rates – Origin & Contact

  Origin                                            O.R
  University vs. Other organisation                 1.31
  Sent by senior persons vs. Juniors                1.13
  Ethnically ambiguous name vs. Non-white name      1.11


  Contact                                           O.R
  Pre-contact vs. No contact                        1.54
  Follow up vs. No follow up                        1.44
  Postal follow up with Q vs. Without Q             1.41
  Mentioning follow up vs. None                     1.04
  Pre-contact by telephone vs. Postal pre-contact   0.90
Edwards et al. 2002
  Increasing Response Rates - Content

                                                    O.R

  More interesting vs. Less interesting             2.44

  User-friendly vs. Standard                        1.46
  Factual items only vs. Factual & attitude items   1.34
  Relevant items first vs. Other items              1.23
  Demographic items first vs. Other items           1.04
  “Don‟t know” boxes vs. no “Don‟t know” boxes      1.03
  Sensitive items vs. No sensitive items            0.92
  General items first vs. Last                      0.80

Edwards et al. 2002
  Increasing Response Rates - Communication

                                              O.R

  Explain drop-out required vs. Not           1.32

  Stresses benefit to respondent vs. Others   1.06

  Stresses benefit to sponsors vs. Others     1.01

  Stresses benefit to society vs. Others      1.00

  Response deadline given vs. No deadline     1.00

  Instructions given vs. No instructions      0.89

  Choice to opt out given vs. No opt out      0.76



Edwards et al. 2002
Questionnaire Summary

1. Postal questionnaires widely used in data collection

2. Mark pre-pay or addressed envelopes

3. Use steganography over cryptography

4. Perform analyses on factors not individual items

5. Identifying items may be useful in later analyses

6. Think about scoring items before rushing ahead with the questionnaire

7. Store questionnaires securely until passing any viva voce

8. Collect as much info at source, parse it down later at discretion
Questionnaire Summary cont.

9. Perfect for epidemiological studies and health research

10. Non-response to postal questionnaires reduces sample & introduces bias

11. Identification of effective ways to increase postal response rates

12. Use existing metrics - pilot items if making new questionnaire

13. Use factor analysis

14. Keep it all as brief as possible

15. Don‟t alienate respondents

16. Alternate types of items

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:38
posted:11/15/2011
language:English
pages:41