Document Sample
Surveying Powered By Docstoc
         Data collection methods

• Interviews
• Focus groups
• Surveys/Questionnaires
When we Use Surveys
•   Requirements specification
•   User and task analysis
•   Prototype testing
•   User feedback

• Principles, methods of survey research in
• Content of surveys for needs and usability
• Survey methods for needs, usability
• Survey:
   – (n): A gathering of a sample of data or opinions
      considered to be representative of a whole.
   – (v): To conduct a statistical survey on.
• Questionnaire: (n) A form containing a set of questions,
  especially one addressed to a statistically significant
  number of subjects as a way of gathering information
  for a survey.
• Interview
   – (n): A conversation, such as one conducted by a
      reporter, in which facts or statements are elicited
      from another.
   – (v) To obtain an interview from.
                                    – American Heritage Dictionary
    Surveying Steps
•   Sample selection
•   Questionnaire construction
•   Data collection
•   Data analysis
          Surveys – detailed steps

• Determine purpose, information needed
• Identify target audience(s)
• Select method of administration
• Design sampling method
• Design prelim questionnaire
   – including analysis
   – Often based on unstructured or semi-
     structured interviews with people like your
• Pretest, revise
• Administer: draw sample, administer q’aire,
  follow-up non-respondents
• Analyze results
                 Why surveys?
• Answers from many people, including those
  at a distance
• Relatively easy to administer
• Can continue for a long time
• Easy to analyze
• Yield quantitative data
  – Incl. Comparable x time
Ways of Administering Surveys
 •   In person
 •   Phone
 •   Mail
 •   Paper, in person
 •   Email (usually with a link)
 •   Web
                  Possible Data
• Facts
  – Characteristics of respondents
  – Self-reported behavior
     • This instance
     • Generally/usually
     • Past
• Opinions and attitudes:
  – Preferences, opinions, satisfaction, concerns
          Some Limits of Surveys
• Reaching users easier than non-users,
  members/non-members, insiders/outsiders
• Relies on voluntary cooperation, possibly
  biasing responses
• Questions have to be unambiguous,
  amenable to short answers
• Self-reports
• Only get answers to the questions you ask
• The longer, more complex, more sensitive the
  survey the less cooperation
                Some sources of error
•   Sample, respondents
•   Question choice
•   Question wording
•   Method of administration
•   Inferences from the data
•   Users’ interests in influencing results
    – “vote and view the results”
    CNN quick vote:
          When to do interviews?
• Need details that can’t get from survey
• Need more open-ended discussions with
• Small #s OK
• Can identify and gain cooperation from target
• Sometimes: want to influence respondents as
  well as get info from them
Sample selection
           Targeting respondents
• What info do you need?
• From whom can you get the information you
  – E.g. non-users are hard to reach
  – Can’t ask 5-year-olds; what do their parents
• Probability samples – random selection
  –   SimpleRandom Sampling
  –   Stratified Random Sampling
  –   Systematic Random Sampling
  –   Cluster (Area) Random Sampling
  –   Multi-Stage Sampling
• Non-probability – not random selection
  – Quota samples
       • Proportional; nonproportional
  – Convenience samples
  – Purposive samples
  – Snowballing
              Sampling terminology
• Sampling Element: the unit about which info is
  collected; unit of analysis. E.g., members of households
  with access to the internet.
• Universe: hypothetical aggregation of all elements. All
  US households with access to the Internet.
• Population: theoretically specified aggregation of
  survey elements. I.e., next slide.
• Survey or study population: aggregate of elements
  from which the sample is actually selected. Households
  in US etc. etc. with phones…if a telephone survey.
                             Internet use

• A Nation Online:                        • Nielsen//NetRatings
  – Individuals age 3+                      – “all members (2 years of age
                                              or older) of U.S. households
  – “Is there a computer or laptop            which currently have access
    in this household?”                       to the Internet.”
  – “Does anyone in this household          – “Internet usage estimates
    connect to the Internet from              are based on a sample of
    home?”                                    households that have access
                                              to the Internet and use the
  – “Other than a computer or
                                              following platforms:
    laptop, does anyone in this               Windows 95/98/NT, and
    household have some other                 MacOS 8 or higher”
    device with which they can              – Sept. 2001: 168,600,000
    access the Internet, such as:
                                                • (+18%)
      • cellular phone or pager
      • a personal digital assistant or
        handheld device
      • a TV-based Internet device
      • something else/ specify”
  – Sept. 2001: 143,000,000
                 Terminology, cont.

• Sampling unit: element considered for selection. E.g.,
  household. Census tracts and then households.
• Sampling frame: list of units composing population
  from which sample is selected.   E.g, phone book
• Observation unit: unit from which data is collected.
  E.g. one person (observational unit) may be asked about
  the household or all members of the household. A
  person may be asked about a transaction or event.
• Sample: aggregation of elements actually included in
               Terminology, cont.

• Variable: a set of mutually exclusive
  characteristics such as sex, age, frequency of use.
• Parameter: summary description of a given
  variable in a population.
• Statistics: summary description of a given
  variable in a sample.
                   Sample design
• Probability samples
  –   random
  –   stratified random
  –   cluster
  –   Systematic
  –   GOAL: Representative sample
• Non-probability sampling
  – Convenience sampling – many web surveys
  – Purposive sampling
  – Quota sampling
          Representative samples
• Which characteristics matter?
• Want the sample to be roughly proportional
  to the population in terms of
  groups/characteristics that matter
  – Exception: oversampling small groups
• E.g., students by gender and grad/undergrad
  status; students by major
                   Sample size
• Formulas for sample sizes are based on
  probability samples from very large
  – Size: if 10/90% split, 100; if 50/50, 400;
    If a table, 30-50 in each cell
• To break down responses x groups, need
  large enough sample in each cell
  – Oversample small groups – e.g., Internet use
    surveys and Hispanics
  – Later, correct for oversampling by weighting in
    data analysis

            Undergrads    Grads     Total
            (n=120)       (n=200)   (n=320)
            %             %         %
Satisfied   60%           13%       31%
            (71)          (25)      (96)

Dissatis.   40            87        69
            (47)          (165)     (127)

Total       100%          100%      100%
            n = 118       n = 190   n = 308
No ans.     n=2           n = 10    n= 12
      Needs, usability, and sampling
• Requirements specification
  – Convenience sample of current users
  – Purposive sample of employees, users
  – Quota sample
     • E.g., x from each location, department
• Prototype evaluation
  – Questionnaire as a way of getting consistent
    data from test population – probably in
    entirety; but could be any of the above
• User feedback
  – User surveys; comments solicitations
          Active vs passive sampling
• active: solicit respondents
   – Send out email, letters, phone
      • Use sampling frame to develop a sample, I.e. list
   – Keep track of who responds
   – Follow up on non-respondents if possible
   – Compare respondents/non-respondents
     looking for biases
• Passive
   Popup box: “would you take a few minutes to help
                Response Rates

• % of sample who actually participate
• low rates may indicate bias in responses
   – Whom did you miss? Why?
   – Who chose to cooperate? Why?
• How much is enough?
   – Babbie: 50% is adequate; 70% is very good
           Increasing response rates

• Harder to say ‘no’ to a person
• Captive audience
• NOT an extra step
• Explain purpose of study
   – Don’t underestimate altruism
• Why you need them
• Incentives
   – Reporting back to respondents as a way of
     getting response
   – Money; entry in a sweepstakes
• Follow up (if you can)
              Web survey problems
• Loss of context – what exactly are you asking
  about, what are they responding to?
  – Are you reaching them at the appropriate point in their
    interaction with site?
• Incomplete responses
• Multiple submissions
      Passive: problems may include
• Response rate probably unmeasurable
• May be difficult to compare respondents to
  population as a whole
• Likely to be biased (systematic error)
  – Frequent users probably over-represented
  – Busy people probably under-represented
  – Disgruntled and/or happy users probably over-
Questionnaire construction
        Questionnaire construction
• Content
  – Goals of study: What do you need to know?
  – What can respondents tell you?
• Conceptualization
• Operationalization – e.g., how exactly do you
  define “household with access to internet”?
• Question design
• Question ordering
       Topics addressed by surveys
• Respondent characteristics
• Sampling element characteristics
  – “Tell me about every member of this household…”
• Respondent/sampling element behavior
• Respondent opinions, perceptions,
  preferences, evaluations
         Respondent characteristics
• Demographics: what do you need to know?
  How will you analyze data?
  – Age, sex, education, occupation, year in
    school, race/ethnicity, type of employer…
  – Equal intervals
• User role (e.g., buyer, browser…)
• Expertise – hard to ask
  – Subject domain
  – Technology
  – System/site
• Tasks (e.g., what did you do today?)
• Site usage, activity
  – Frequency; common functions – hard to answer
  – Self-reports vs observations
• Web and internet use: Pew study
        Opinions, preferences, concerns
• About the site: Content, organization, architecture,
• Ease of use
• Perceived needs
• Preferences
• Concerns
   – E.g., security

• Success, satisfaction
   – Subdivided by part of site, task, purpose…
• Other requirements
• Suggestions

Shared By: