National Survey of Child and Adolescent Well-Being _NSCAW_

Document Sample
National Survey of Child and Adolescent Well-Being _NSCAW_ Powered By Docstoc
					Cognitive Interviewing and
  Questionnaire Design



     Gordon Willis, Ph.D.
      National Cancer Institute
     National Institutes of Health
        willisg@mail.nih.gov
Preface: Why worry about questionnaire
          design problems?
 On the one hand:
  –   A self-report sample survey is a ‘blunt instrument’
  –   There is bound to be some ‘slop’
  –   To the extent questionnaire design IS rocket science,
      we’re content to just hit the moon, somewhere…
 On the other hand:
  – Of the various types of errors afflicting surveys –
    Sampling error, Non-response error, Processing
    error…:
  – Response Error (e.g., bias or unreliability) is often the
    largest category ->

                    G Willis 2012 Cog Interviewing USUHS        2
  Do small differences in wording…
    produce differences in data?
VERSION 1 (No filter)
 On a typical day, how much time do you
 spend doing strenuous physical activities
 such as lifting, pushing, or pulling?
 __ None __ Less than 1 hour __ 1-4 hours
 __ 5 + hours
VERSION 2 (Filtered)
 On a typical day, do you spend any time
 doing strenuous physical activities such as
 lifting, pushing, or pulling?
 IF YES: Read Version 1
               G Willis 2012 Cog Interviewing
                           USUHS                3
   Evaluation study: Embed target
  questions, using three split-sample
         survey experiments
1) Field Pretest of NCHS National Health
 Interview Survey: 78 adult household
 respondents
2) Women’s Health Study: 191 women, 18-41,
 in contractor offices
3) Random-Digit Dial (RDD) Survey: 989
 respondents in Maryland Omnibus telephone
 survey


  (Willis & Schechter, 1997)
                    G Willis 2012 Cog Interviewing
                                USUHS                4
 Survey experiment results: Screening
         for physical activity
On a typical day, how much time do you spend
doing strenuous physical activities such as
lifting, pushing, or pulling?

                                 0         <1    1-4   5+
FIELD PRETEST (n=78)
 No filter                      32% 32% 35% 0%
 Filtered                       72% 18% 10% 0%
WOMEN’S HEALTH (n=191)
 No filter             4%  42% 50% 4%
 Filtered              49% 16% 27% 8%
RDD SURVEY(n=989)
  No filter                     -          45% 34% 22%
  Filtered                      -          62% 29% 9%

                G Willis 2012 Cog Interviewing
                            USUHS                           5
 Survey experiment results: Screening
         for physical activity
On a typical day, how much time do you spend
doing strenuous physical activities such as
lifting, pushing, or pulling?

                                 0         <1    1-4   5+
FIELD PRETEST (n=78)
 No filter                      32% 32% 35% 0%
 Filtered                       72% 18% 10% 0%
WOMEN’S HEALTH (n=191)
 No filter             4%  42% 50% 4%
 Filtered              49% 16% 27% 8%
RDD SURVEY(n=989)
  No filter                     -          45% 34% 22%
  Filtered                      -          62% 29% 9%

                G Willis 2012 Cog Interviewing
                            USUHS                           6
     The cognitive testing process
            in a nutshell
Develop a questionnaire or material (advance letter,
etc.) to be evaluated
Recruit (paid) members of the targeted population (e.g.,
recipients of home loans, people without employment,
cancer survivors)
Conduct one-on-one interviews, in “laboratory” or
other location:
– Home
– Homeless shelter
– Health clinic
– Elderly center

                   G Willis 2012 Cog Interviewing USUHS   7
  The cognitive testing process
         in a nutshell
THEN: Make informed decisions, with cognitive
testing as one source of input –
  We also rely on –

  a) Expert Review (Topic area and/or
  questionnaire design);
  b) Review of measurement objectives;
  c) Consideration of past practice, need for
    comparability




              G Willis 2012 Cog Interviewing USUHS   8
  The cognitive testing process, in a
              nutshell
To conduct the cognitive interview, we use verbal
probing techniques to elicit thinking about
question
 – Finding: Note apparent problems related to
    question wording, ordering, format
 – Fixing: Suggest modifications that address
    problems
Cognitive testing is best done as an iterative
process (multiple small testing ‘rounds’)
                                                       Analyze   Design



                                                             Test

                G Willis 2012 Cog Interviewing USUHS                      9
Tourangeau (1984) cognitive model
Encoding of question (understanding it)
 – Have you ever received care from a podiatrist?
Retrieval of information (knowing/remembering)
 – Do you approve of the Budget Reconciliation Act of
   1993?
 – How many times have you ridden in a passenger
   airplane?
Decision and judgment processes (truth, adequacy)
 – How many sex partners have you had in the past 12
   months?
Response (matching internal representation to given
categories)
 – “Would you say your health is excellent, good, fair,
   or poor?”
                G Willis 2012 Cog Interviewing USUHS   10
Focus: Cognitive and other problems
Focus is NOT just on “cognitive” problems:
 (1) Problems with question assumptions
     Do you own or rent your house?
 (2) Problems for interviewer
     Question is tongue-twister, etc.
 (3) Format issues
     Where do I go next, for self-administered
       questionnaire (“skips”)
 (4) Issues that influence flow of the interview
     Subject thinks question is over and interrupts
       with answer
 As such, has been called “Intensive interviewing”
(Cantril & Fried, 1944(!); Royston, 1989)
                G Willis 2012 Cog Interviewing USUHS   11
        Think Aloud interviewing
§ Mainly advocated by Ericsson and Simon (1980,
  1984) as a means to study cognitive processes
  used in problem solving (chess…)
§ The interviewer requests a verbal “train” as
  subject completes the task
§ The interviewer interjects little except to say
  “Keep talking” or “Tell me what you’re thinking”
  Ø In the extreme case, avoid the use of references to
    “me” the interviewer(!)



                  G Willis 2012 Cog Interviewing USUHS    12
        Example of think-aloud
INTERVIEWER: How many times have you talked
to a doctor in the last 12 months?
SUBJECT: I guess that depends on what you mean
when you say “talked.” I talk to my neighbor, who is
a doctor…I go to my doctor about once a year, for a
general check-up, so I would count that. I’ve been
to…a specialist a couple of times in the past year -
once to get a bad knee diagnosed, and I also saw
an ENT about a coughing thing, which I’m pretty
sure was in the past year, although I wouldn’t swear
to it. I also talked to doctors when I brought my kids
to the pediatrician - I assume that you don’t want
that, though I’m not sure. Also, I saw a chiropractor,
but I don’t know if you consider that to be a
doctor…. So, I’m not sure what number to give
you…               G Willis 2012 Cog Interviewing USUHS 13
Limitations of “pure” Think-Aloud
  Many subjects are not good at this
  ØMy experience: male, teenage smokers…
  Was not designed for a task that involves
 social interaction (survey…)
  Is very different from the normal question
 asking-answering sequence
  Analysis of think-aloud protocol is not
 straightforward
  ØWe can use a coding system (e.g., Bickart &
   Felcher, 1996), but this is complex and
   burdensome

             G Willis 2012 Cog Interviewing USUHS   14
   The alternative to Think-Aloud:
    Verbal Probing Techniques
Ask follow-up questions –
Ø About the questions
Ø About the answer
Ø About the general context
Not new: Used by Cantril (1944), Belson
(1980)…
Verbal probing is more active than think-aloud
Ø Requires decision by the interviewer about how to
  probe -> This is what makes the activity
  interesting!

               G Willis 2012 Cog Interviewing USUHS   15
   Basic varieties of (“classic”) verbal
                probes
Comprehension probe: What does the term “dental
                     sealant” mean to you?
Paraphrase:              Can you repeat the question in
                         your own words?
Confidence judgment: How sure are you that your
                     health insurance covers…
Recall probe:            How do you know that you went
                         to the doctor 3 times…?
Specific probe:          Why do you think that breast
                         cancer is the most serious
                         health problem?
‘General’ probe:         How did you arrive at that
                         answer?
                   G Willis 2012 Cog Interviewing USUHS   16
     The combined approach
Cognitive interviews are normally done as
combination of Probing and Think-Aloud
(DeMaio & Landreth, 2004)
But, the relative mix of these may vary
across type of investigation




            G Willis 2012 Cog Interviewing USUHS   17
Verbal probing techniques: Concurrent
    versus retrospective approach
 Concurrent probing: Probe immediately after the
   subject has answered each survey question
   Advantage: Probing when the memory still exists
   Disadvantage: Measurement process interrupts
   normal flow
 Retrospective probing: Wait until after the interview,
  and then go back to probe
   Advantage: Mirrors “field” procedures
   - Is particularly useful for self-administered Q’s
   Disadvantage: Subject may have forgotten key
   information

                    G Willis 2012 Cog Interviewing USUHS   18
Thesis:
 – C.I. depends on more than just the process of
   asking probe questions
 – We need to start with a working knowledge of
   questionnaire design
   • We can’t just make up probes haphazardly
   • We can’t just leave it to the subject to tell us
     something is wrong!!!
 – Active probing requires the targeting and
   recognition of problems
 – Model: Question Appraisal System ->
   (Willis & Lessler, 1999; Willis, 2009)


                   G Willis 2012 Cog Interviewing USUHS   19
G Willis 2012 Cog Interviewing   20
            USUHS
More evidence that we should
     do ‘Expert Review’




        G Willis 2012 Cog Interviewing   21
                    USUHS
Fashioning probes, in
       depth



       G Willis 2012 Cog Interviewing USUHS   22
   Tested (“classic”) question:
       Pain in the abdomen
“In the last year have you been bothered by
  pain in the abdomen?”
  What (Anticipated) probes make sense
  here?
  § What time period are you thinking about,
    exactly?
  § What does “bothered by pain” mean to you?
  § Where is your “abdomen?” è

                G Willis 2012 Cog Interviewing USUHS   23
G Willis 2012 Cog Interviewing USUHS   24
G Willis 2012 Cog Interviewing USUHS   25
           Revised question:
           Pain in abdomen
(Please look at this diagram. During the past 12
   months, have you had pain in this area…)

  Remember to check out S’s interpretation of
  vague reference periods like “year,” “month,”
  “week,” or no reference period at all (e.g., “How
  often do you X?”).
  Argument: “Lose this example” – It’s obvious
  that “Abdomen” is a problem
  Response: Yes, but sometimes we need to
  demonstrate that which we think we know

                 G Willis 2012 Cog Interviewing USUHS   26
             TV, Smoking…
During the past 30 days, about how
often have you seen anti-smoking
commercials on TV?
  Probe that was used (Paraphrase): What
  was that question asking you?
  n Subject: “About smoking commercials – I
    don’t think there are any”
  n So, he heard “anti-smoking” and thought
    “smoking” -- NO GOOD

è A straightforward answer can hide a “Silent
  Misinterpretation” (DeMaio and Rothgeb,
  1996)
                G Willis 2012 Cog Interviewing USUHS   27
     Fixing… potential alternatives
  During the past 30 days, have you seen any TV
  commercials about the dangers of cigarette smoking?
  During the past 30 days, have you seen any TV
  commercials that warned people NOT to smoke?
  During the past 30 days, have you seen any
  messages on TV warning people NOT to smoke?


->  Which is best? Depends on objective – may need to
  consult with sponsor/client
-> Cognitive interviewing is often a “vagueness
  detector” which brings out hidden ambiguity

                  G Willis 2012 Cog Interviewing USUHS   28
 Beyond Proactively searching
Illustrative example of tested question:
Interviewer: “Inside your home, are there any
 walls that have peeling paint?”
Subject: No… not on the walls, anyway…
I (Probe): Is there any paint that’s peeling?
S: Yeah, the window frame…
Problem wasn’t anticipated beforehand –
so neither was the probe
Cog Interviewing needs a capacity not
only for problem Verification, but also for
reaction to the unexpected: Discovery
             G Willis 2012 Cog Interviewing USUHS   29
    Easy default: “tell me more…”
n   Example:
    I:How often is your household trash collected?
     Twice a week.
    S:
    I:Tell me more about your trash pickup.
    S:The garbage gets picked up on Monday, and
      the recycling on Wednesday.”
   § NOT GOOD: The objective concerns trash, not
     recycling
   § Probing revealed that the tested version
     doesn’t work
-> Good example of a Silent Misinterpretation

                 G Willis 2012 Cog Interviewing USUHS   30
          How would you probe?
What types of smokeless tobacco products
 have you used in the past year?
n   Proactive forms:
    • To you, what’s a smokeless tobacco product?
    • When did you last use (X)?; How sure are
      you that this was within the past year?

n   Reactive forms:
    • It depends on what we hear ->


                   G Willis 2012 Cog Interviewing USUHS   31
          How would you probe?
What types of smokeless tobacco products
 have you used in the past year?
§ How about if given this response:
    “Well, I used to chew tobacco…”
n   Does he mean in the past year, or longer ago?
    • I would first follow up on the Reactive part,
      before Proactively “fishing” -
    -> Follow problems before searching for them
    • So: How long ago did you last chew tobacco?
    • You can then follow up with other (Anticipated)
      probes: Besides chewing tobacco, what do you
      consider to be a “smokeless tobacco product”

                    G Willis 2012 Cog Interviewing USUHS   32
An alternative to probing… listening
When you first thought you were
 pregnant, how did you feel? Were you -
   Happy,
   Unhappy,
   Both happy and unhappy, or
   Neither happy nor unhappy

  Say that we observe a favorite subject ploy:
   She jumps in at “how did you feel” with “I felt
   awful!”
  So, the respondent thinks the question is over
   (or that it should be…)

                 G Willis 2012 Cog Interviewing USUHS   33
  So, we change the question…
When you first thought you were
 pregnant, were you -
   Happy,
   Unhappy,
   Both happy and unhappy, or
   Neither happy nor unhappy

Subject: “I don’t get all those different happy-
  unhappy things”
-> Again, we need only to listen – no probing
  may be necessary

                 G Willis 2012 Cog Interviewing USUHS   34
  So, we change the question…
When you first thought you were
 pregnant, were you…
   Happy,
   Unhappy,
   or did you have mixed feelings?

Lessons:
1) There may be multiple problems
2) Iterative testing is good
3) We may not need to probe – but, we react
  differently from a field interviewer (who is
  motivated to “make it work”)
                 G Willis 2012 Cog Interviewing USUHS   35
Avoiding Probing
     Pitfalls



    G Willis 2012 Cog Interviewing
                USUHS                36
                   Probing “fouls”
Ø   I: …what about contamination of the ground water.
    Would you say it’s a very important problem, a
    somewhat important problem, a small problem, or
    not a problem?
Ø   S: “I guess… hmmm. I’m not sure how to answer
    that, really.”
Ø   I: “Why don’t you know how to answer it?”
    Problems with probe?
    Ø   Kind of accusatory
    Ø   I’m not sure that a person who can’t answer can
        further report why they can’t answer
    Ø   I would prefer “Ok, tell me what you’re thinking”
                      G Willis 2012 Cog Interviewing
                                  USUHS                     37
        Probing fouls: continued
Ø (I: “Why don’t you know how to answer it?”)
Ø S: “Well, I don’t know how to answer it, because I
        don’t know really what your mean… um…”
Ø I: OK, whatever it means to you. Again, please
       indicate whether it’s a very important problem, a
       somewhat important problem, a small problem,
       or not a problem”
Ø S: “Oh, it’s a very important problem”


     Problems with probe? ->

                   G Willis 2012 Cog Interviewing
                               USUHS                   38
          Probing fouls: continued
(I: OK, whatever it means to you. Again, please indicate
    whether it’s a very important problem, a somewhat
    important problem, a small problem, or not a problem” )
(S: “Oh, it’s a very important problem”)
     PROBLEMS WITH PROBING:
1)   “Whatever it means to you” is a field interviewer
     activity – designed only to “get through the
     questionnaire”
2)   Do we know that S is still thinking about the actual
     topic of the question (what is “it” that’s an important
     problem?)
3)   Responses to probing are subject to social
     desirability effects (just as survey questions)
                      G Willis 2012 Cog Interviewing
                                  USUHS                   39
           Rule of probing flaws:
Ø Probes follow all the usual rules of
 questionnaire design
  l   They may be too long
  l   They may be vague
  l   They may produce “huh?” response
  l   They can be biasing
  l   They can be irrelevant to the exchange
  l   If so, they must be fixed, or abandoned!

                   G Willis 2012 Cog Interviewing
                               USUHS                40
       Modifying probes to recover from
     communication failure: Wellens (1994)
I:   Are you a citizen of the United States?
S:   No.
I:   In your own words, what does the term “citizen” mean
     to you?
S:   I don’t… I don’t understand
I:   I just want to know what you think the definition of the
     word “citizen” is.
S:   A citizen is a person who belongs to this country. That
     person has the right to vote or join the government.
I:   Can you tell me more about why you are not a
     citizen…?
S:   A person must live here for five years, take a test,
     obtain history course, then the person can become
     citizen.”
                        G Willis 2012 Cog Interviewing
                      G Willis 2012 Cog Interviewing
                                 USUHS                      41
Adapting to Survey
Administration Mode



      G Willis 2012 Cog Interviewing USUHS   42
Adapting cognitive interviewing to
   Face-to-face administration
§ This is the default – the situation that is most
  similar to the in-person cognitive interview
§ So, there’s no not much to adapt to
§ The main thing to keep in mind is that the
  cognitive interviewer is not the same as the
  field interviewer
   -> So, pay attention to problems for the interviewer
     as well as for the subject



                  G Willis 2012 Cog Interviewing USUHS    43
 Adapting cognitive interviewing to
    Telephone administration
§ Important issue:
  § Can C.I. be done over the phone?
  § Should C.I. be done over the phone?
     - We miss out on body language
     - But, phone may be the mode of fielded survey
       administration
§ NCHS procedure:
     § Put subject in separate room and call him/her
     § Conduct interview over phone, but with camera
     § Go to Subject location, debrief in-person
                   G Willis 2012 Cog Interviewing USUHS   44
 Adapting cognitive interviewing to
    Telephone administration
§ Alternative -
  § Conduct an initial interviewing round face-to-
    face, and a second round over the telephone
    (e.g., CDC Behavior Risk Factor Surveillance Survey)
  § We are likely to find that the telephone
    produces impediments to question functioning




                   G Willis 2012 Cog Interviewing USUHS   45
    Adapting cognitive interviewing to
           Self-administration
§ Another interesting/unresolved area:
  § The cognitive demands of the S/A task are
    different from those of face-to-face and phone:
     § S/A relies on visual rather than auditory
       processing
     § S/A sometimes requires navigational activities
       (find the starting point, follow skip patterns…)
  § So, for the C.I., it may not make sense to follow
    the usual procedure of having the interviewer
    read the questions to the subject
                      G Willis 2012 Cog Interviewing USUHS   46
 Adapting cognitive interviewing to
        Self-administration
§ So… It makes sense in the C.I. to have the
  subject read the form
§ But, do we then do the cog interview
  concurrently, or retrospectively?
  § If concurrent, should we use (a) uninterrupted
    think-aloud, (b) interruption with probes?, or (c)
    both
§ Issue: Which procedure produces more
  distortion of the question answering process,
  and therefore produces reactivity effects?


                   G Willis 2012 Cog Interviewing USUHS   47
   Example: Self-administered teen
       tobacco questionnaire
Do you think young people who smoke
  cigarettes have more friends?
    [ ] Definitely yes
    [ ] Probably yes
    [ ] Probably not
    [ ] Definitely not
§ Retrospective probe: What does “young people” mean?
  S: “13 year old”
§ Observation: S left the answer blank – objects that “It
  doesn’t make any difference” - but there’s no way to say
  that
§ Expert judgment: Seems classically biased to me
  (presents one possibility, but not others)
                     G Willis 2012 Cog Interviewing USUHS   48
             Revised question
Do you think that people your age who smoke
  cigarettes have more friends, fewer friends, or the
  same number of friends as those who don’t smoke?
   [ ] People who smoke have MORE friends
   [ ] People who smoke have FEWER friends
   [ ] People who smoke and those who don’t have the
      SAME number of friends

è “Fixes” may be appropriate for one mode but not
  another. The current example lengthens the question
  and seemed to be ok for self-administration, but may be
  too much for the phone.


                     G Willis 2012 Cog Interviewing USUHS   49
          Adapting C.I. to Web surveys
§ The Web Usability Testing area is huge, and well
  entrenched (Mick Couper)
  § It shares MANY commonalities with cognitive
    interviewing of S/A questionnaires
  § But there is a major emphasis on use of the system
    – navigation, problem-solving (e.g., how do I go
    back and change an answer, “skip around”)
  § For this reason, it bears similarities to the classic
    Ericsson-Simon think-aloud task
     § Cleo Redline (NSF), Jennifer Crafts (Westat): Think-
       aloud works for usability testing- we observe how
       the subject handles a complex, non-verbal activity


                       G Willis 2012 Cog Interviewing USUHS   50
      Adapting C.I. to Web surveys

§ Challenge: Integrating C.I. and Usability testing
  § We need to consider both (a) the questionnaire
    and (b) Usability of the computer system
  § Makes sense to (a) test questionnaire prior to
    programming, but to (b) include “regular”
    cognitive testing during Usability test (combine
    these so we don’t forget about the questions!)
  § Example of a complex Web questionnaire ->



                   G Willis 2012 Cog Interviewing USUHS   51
1) Where do users look?
2) Can they tell what the panel on the left is for?
3) Do they go directly to the question…?
4) Can they tell what the colors signify?

        G Willis 2012 Cog Interviewing USUHS      52
                                       1)   Does it matter that
                                            there are no
                                            question numbers?
                                       2)   Are the questions
                                            understandable?




G Willis 2012 Cog Interviewing USUHS                       53
           Examples of Cognitive
              Testing Reports

Many cognitive testing reports are retrievable from the
Web, through the Q-BANK database
  Miller, et al. (National Center for Health Statistics,
  Centers for Disease Control and Prevention):
  Q-Bank Home:
  http://wwwn.cdc.gov/qbank/Home.aspx

-> Consult these reports for examples of how cognitive
testing results are written up in U.S. Federal Agencies

                      G Willis 2012 Cog Interviewing
                                  USUHS                    54
  Logistics of the
Cognitive Interview
     Process



      G Willis 2012 Cog Interviewing USUHS   55
           Practical considerations
► 1)   How long should a Cog Interview be?
  § One hour is reasonable
  § Avoids fatigue (for Interviewer, and Subject)


► 2)   How long a questionnaire can be tested in 1 hr?
  § Best measured in terms of administration time, not
    pages
  § Depends on probing intensity:
     ►DeMaio and Landreth: 3 lab minutes for each
      questionnaire minute
     ►Me: 2 lab minutes for 1 questionnaire minute
                     G Willis 2012 Cog Interviewing USUHS   56
            Practical considerations
► 3)   How many interviews can you do in one day?
  §    A LOT – but not for too many days!
  §    I have done up to 6
  §    Plan a maximum of three, on a regular basis


► 4)   What is appropriate payment (“remuneration”)?
  §    Industry standard is US $25 - $50 for one-hour
  §    May need to pay more for special populations
  §    Interestingly, paying more brings in lower-income
       (NCHS finding), as they’re not just doing “hobby
       research”
                      G Willis 2012 Cog Interviewing USUHS   57
           Practical considerations
► 5)   How many interviews should be conducted?
  § The question sub-divides into (a) How many
    rounds; and (b) How many per round?
  § For pretesting purposes, I like doing no more than
    12 before stopping to review and modify
  § Note that different problems require different
    effective sample sizes to detect and to correct
  § If we stop to change half the questions, and do
    another round, the other half get two rounds worth
    of unaltered testing
  § A nice plan = three iterative rounds of 10 subjects
  § Normally, we run out of time before running out of
       problems
                    G Willis 2012 Cog Interviewing USUHS   58
             Practical considerations
► 6)How much time is required for the various
 activities involved?
  § For every 1-hour interview, plan another 3 hours
    for preparation, processing, and writing up results
       ►It is a good idea to listen to the interview again
       ►It takes awhile to make sense of and write up results
       ►You may have several debriefing meetings with
        project staff
  § Concerning clock time, two activities tend to take
    time, and spread out interviewing rounds:
       ►Recruitment  (can take several weeks)
       ►Getting feedback from client/team before next
        testing round
                      G Willis 2012 Cog Interviewing USUHS   59
           Practical considerations
► 7)   What staffing do you need to run a lab?
  § Few people will have a true “Cognitive Lab”
  § So, you can do everything yourself, and still get
    something out of it
  § For a Lab:
       ►Severalstaff should be trained as interviewers
       ►Good to have a staff member who is a Laboratory
        Manager (advertisements, recruitment, logistics)




                      G Willis 2012 Cog Interviewing USUHS   60
           Practical considerations
► 8)   Do you need multiple interviewers?
  § Some successful interviewing rounds have been
    done with one cognitive interviewer
       ►May   be necessary: As for multiple-language testing
  § I prefer 2-3 interviewers:
       ►Get  differing approaches/perspectives on
        interviewing
       ►Have variation in questionnaire design expertise
       ►Can look for consistency across interviewers in
        results


                       G Willis 2012 Cog Interviewing USUHS    61
            Practical considerations
► 9)   Is there value in very limited efforts?
  § Good question: What if I can only do 3 or 4
    interviews, myself --
  § Is this better (or worse) than doing nothing?
  § I find it hard to believe that this can be harmful
  § The smaller the sample, the more conservative we
    need to be in accepting results as “real”
  § But, if you administer a questionnaire to 4 people
    and none of them have any idea what you’re
    asking… it’s time to revisit the approach

                      G Willis 2012 Cog Interviewing USUHS   62
             Practical considerations

► 10)   What equipment do I need for my “lab”
  § You don’t need a “lab”…
  § You do need:
        ►A private, quiet place to interview
        ►An audio recorder




                        G Willis 2012 Cog Interviewing USUHS   63
            Practical considerations

► 11)   What about confidentiality?
  § Cognitive Interviewing is research, so pay attention
    to issues of Human Subjects Protection
        ►Always  good to use a consent form
        ►Don’t keep forms with subject identifiers laying
         around
        ►Eventually, destroy tapes/videos
        ►If you plan to use interviews for demonstration
         purposes, have subjects sign a separate release for
         that purpose


                       G Willis 2012 Cog Interviewing USUHS    64
            Practical considerations
► 12)   How can clients/team be integrated into the
        process?
  § Don’t isolate recipients of information – include
    them!
  § Seeing is believing – Stakeholders can be influenced
    by observing a single interview
        ►Contractors normally invite clients to observe
         through video setup/one-way mirror, or watch tapes




                       G Willis 2012 Cog Interviewing USUHS   65
          Useful Internet Resources
►   1) US Census Bureau guide to self-administration:
    http://www.census.gov/srd/Economic_Directorate_Guidelines_
    on_Questionnaire_Design.pdf
►   2) NCI resources:
►   a) The Question Appraisal System (Full version)
►   http://appliedresearch.cancer.gov/areas/cognitive/qas99.pdf
►   b) Cognitive Interviewing: A How-To Guide
►   http://appliedresearch.cancer.gov/areas/cognitive/qas99.pdf
►   c) An Introduction to Modern Measurement Theory
►   http://appliedresearch.cancer.gov/areas/cognitive/immt.pdf




                        G Willis 2012 Cog Interviewing USUHS      66
In closing…

 “The uncreative mind can spot wrong
   answers,
 but it takes a very creative mind to spot
   wrong questions.”

 - Anthony Jay


               G Willis 2012 Cog Interviewing   67
                           USUHS

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:7
posted:9/9/2013
language:English
pages:67