Introduction to Usable Security Reasoning About the Human in

Document Sample
Introduction to Usable Security Reasoning About the Human in Powered By Docstoc
					Surveys, interviews, and focus
groups

Lorrie Faith Cranor
September 2009


                 CyLab Usable Privacy and Security Laboratory
                 http://cups.cs.cmu.edu/
           CyLab Usable Privacy and Security Laboratory   http://cups.cs.cmu.edu/   1
Surveys, interviews, and focus groups
 Surveys
   – Ask people set list of questions (possibly with conditional
     questions or branching) with multiple choice or free response
     answers
   – Can be conducted in person, by postal mail, by phone, online
 Interviews
   – Ask people open-ended questions, with follow-up questions
     depending on their responses
   – Usually conducted in person or by telephone
 Focus groups
   – Group interview, usually with 5-10 participants
   – Usually conducted in person



                CyLab Usable Privacy and Security Laboratory   http://cups.cs.cmu.edu/   2
What method should I use?
   Survey
     – Allows you to collect data from a lot of people relatively quickly
     – Easy to collect data in standardized format, ready for analysis
     – But, not very good for answering “why” questions unless you have already developed
       hypothesis through other studies
     – Useful for validating hypotheses from smaller studies with a larger population
   Interview
     – Allows you to probe mental models, what people think and why
     – Semi-structured interviews allow you to explore issues as they are raised
     – Allows you to clarify if people don’t understand a question
   Focus group
     – Many of the advantages of interviews, but more efficient as you can interview 5-10 people all
       at once
     – Less detail from any interviewee than you would get in an individual interview
     – Not that great for testing usability because participants probably won’t all get hands on
       experience with software
     – Sometimes an opinionated individual can dominate a focus group
     – Hard to publish papers unless focus groups are used along with other methods or you report
       on a large number of focus groups together



                        CyLab Usable Privacy and Security Laboratory       http://cups.cs.cmu.edu/     3
SURVEYS


      CyLab Usable Privacy and Security Laboratory   http://cups.cs.cmu.edu/   4
Designing good survey questions
 Word questions clearly, without jargon or
  undefined abbreviations
 Avoid leading questions, ambiguous terms, or
  emotionally-loaded terms
 Design questions to evoke truthful responses
  – Non-threatening, don’t bias participants to provide
    what they think you want, protect confidentiality
 Probe one dimension at a time
 Design questions such that respondents are likely
  to provide a range of answers


             CyLab Usable Privacy and Security Laboratory   http://cups.cs.cmu.edu/   5
Multiple-choice answers
 Make sure answer choices are clear, mutually
  exclusive, cover entire space of possible answers
 Make sure answer choices are at appropriate
  level of granularity
 Where appropriate, allow respondents to indicate
  they don’t know, don’t have an opinion, or the
  question is not applicable to them
 Use consistent rating scales throughout your
  survey
 Be aware that the rating scale can skew
  responses (people like to think they are normal)

            CyLab Usable Privacy and Security Laboratory   http://cups.cs.cmu.edu/   6
More survey design tips
 Cluster similar questions together
 Do not ask respondents to perform cognitively
  difficult tasks (unless you are testing their
  ability to perform these tasks) such as ranking
  more than 5 items
 Use a clear and attractive layout
 Pilot, pilot, pilot!



            CyLab Usable Privacy and Security Laboratory   http://cups.cs.cmu.edu/   7
INTERVIEWS


      CyLab Usable Privacy and Security Laboratory   http://cups.cs.cmu.edu/   8
Developing an interview script
 Keep your questions fairly open-ended
   – You can follow-up with specific probes (“What files do you
     have on your computer that you consider valuable?”
     Follow up with “Do you have valuable photos? Videos?
     School work? Letters?”)
 Start with general questions and get more specific so
  you get their unbiased impressions before you direct
  their thinking to particular details
 A semi-structured approach allows you to adjust as
  needed
 Write-out all your questions and follow-up probes

               CyLab Usable Privacy and Security Laboratory   http://cups.cs.cmu.edu/   9
Preparing for an interview
 The day before the interview:
  – Print out:
     • Protocol – including detailed interview script
     • Official consent form
     • Payment sheet
  – Prepare:
     • Compensation payment
     • Audio/video recording (devices, batteries, extension cords,
       etc.)‫‏‬
     • Additional material
  – Send an email to the interviewee to remind him/her
    of the date/time/place


               CyLab Usable Privacy and Security Laboratory   http://cups.cs.cmu.edu/   10
During the interview
 Explain the purpose of the study (unless you need
  to hide that)‫‏‬
 Ask to read and sign the consent form
 If recording/video taping, turn it on!
 Perform the study!
 Debrief the person (if applicable)‫‏‬
 Ask to sign the signing sheet
 Give the payment
 Thank the person!

            CyLab Usable Privacy and Security Laboratory   http://cups.cs.cmu.edu/   11
More interview recommendations
 Schedule meetings at least one week in
  advance
 Be mindful of your subject/interviewee time
 Keep a careful record of your interviews and
  keep it secure!




           CyLab Usable Privacy and Security Laboratory   http://cups.cs.cmu.edu/   12
CyLab Usable Privacy and Security Laboratory   http://cups.cs.cmu.edu/   13
CyLab Usable Privacy and Security Laboratory   http://cups.cs.cmu.edu/   14
Analyzing interview data
 Interview can be formally analyzed through a coding process
    – Transcribe interview data (yourself or with hired transcription service)
    – Iteratively review transcripts and create code for concepts mentioned by
      participants
    – As new codes are added, check to see whether those concepts were
      mentioned in previously analyzed transcripts
    – Keep track of how many participants mentioned each concept to find
      concepts that resonate with a lot of participants
    – Group similar codes together into categories
    – Note interesting quotes
 Qualitative approach
    – Use concepts and categories to develop theory (Grounded theory approach,
      does not start with hypotheses)
 Quantitative approach
    – With large number of interviewees (~30), and questions that ask participants
      to provide numerical ratings, quantitative analysis may be feasible
    – Important to validate for coder reliability
 Can be used to develop mental models

                   CyLab Usable Privacy and Security Laboratory   http://cups.cs.cmu.edu/   15
FOCUS GROUPS


     CyLab Usable Privacy and Security Laboratory   http://cups.cs.cmu.edu/   16
Advice on conducting focus groups
   Develop detailed script to guide you
   Pre-screen participants to get a good mix of people who meet your criteria
   Record everything, but don’t have cameras pointed in people’s faces so they feel
    like they are under surveillance
   Helpful to have at least 2 people, a moderator and a note taker
   Give people name tags with their first name only
   Make the session fun, informal, relaxed feel
     – Provide drinks and snacks
   Promote a free flowing conversation that engages all participants
     –   Ask open ended questions
     –   Show people multiple things and ask them to compare
     –   Give demos or show videos to start-off discussion
     –   Give people handouts and ask them to circle things they like/don’t like, or jot down first
         impressions before discussing with the group
   Plan to do multiple focus groups to mitigate effects of dominant participant
    steering conversation




                        CyLab Usable Privacy and Security Laboratory          http://cups.cs.cmu.edu/   17
Using videos in focus groups
 Videos can provide concrete scenarios for
  people to discuss
 Videos from L. Little, E. Sillence and P. Briggs.
  Ubiquitous Systems and the Family: Thoughts
  about the Networked Home. SOUPS 2009




            CyLab Usable Privacy and Security Laboratory   http://cups.cs.cmu.edu/   18
Cylab Usable Privacy and Security
          Laboratory
      http://cups.cs.cmu.edu/

  CyLab Usable Privacy and Security Laboratory   http://cups.cs.cmu.edu/   19