Wendy Hicks_ITSEW_0521

							A Tale of Two Methods:
Comparing mail and
RDD data collection for
the Minnesota Adult
Tobacco Survey III
Wendy Hicks and David Cantor
Westat
Ann St. Claire,
ClearWay Minnesotasm
Rebecca Fee,
Blue Cross and Blue Shield of Minnesota
Peter Rhode,
Minnesota Department of Health
Overview of Presentation

• Report results of a feasibility study comparing mail
  and telephone data collection methods
    Seeming declines in RDD response rates (Curtin et al,
     2005)
    Changes in coverage of the telephone population
     (Blumberg et al, 2006, 2007)
    Increasing costs associated with RDD surveys
• Findings mostly replicate those reported by Link et
  al. (2006)
• Some suggestion that an “all adult” selection
  method in a mail survey results in biased
  estimates for young adults (18-24 year olds)

                                                             2
Methodological Approach


• Mail survey has small sample and large
  confidence intervals
    Contrast results found in a parallel mail vs. telephone
     study for which mail study selected individuals rather than
     addresses
    Bring in similar findings from another pilot study (Health
     Information National Trends Survey, HINTS)




                                                                   3
MATS Overview

• Minnesota Adult Tobacco Survey (MATS)
   ClearWay Minnesotasm
   Blue Cross and Blue Shield of Minnesota
   Minnesota Department of Health
• Major objective of MATS: collect a diverse set of
  public health data about adult pop in MN, focusing
  on tobacco and cigarette use
• MATS samples from two frames
   RDD
   Blue Cross member list frame




                                                       4
 Overview of MATS 2007 samples by mode and frame

                      Telephone                    Mail
                 (Feb – June, 2007)      (March – June, 2007)
General      • targeted 7500 completes  • sampled 1200
population   • oversampled young adults addresses
samples      & African Americans        • no oversampling
             • selected one individual   • “all adults” selected
             within household            to respond
BCBS        • targeted 5000 completes • sampled 1200
member list • oversampled young adults members

            • selected individual BC   • no oversampling
             member                      • selected individual
                                         BC member

                                                                   5
USPS Mail Data Collection


• All adult respondent selection mechanism
• Mailing package included 4 identical
  questionnaires and return envelopes
       “Each adult living at this address should complete one of the
       enclosed questionnaires. Please give one of the enclosed
       questionnaires and an envelope to each person 18 years old
       or older living at this address.”
• Some differences from the CATI instrument to
  make it more suitable to paper, self-administration




                                                                       6
Response rates by frame and mode


                             MAIL      TELEPHONE

General Population Frame   31% - 33%      41%
(USPS and RDD-only)


BCBS Member List             56%          48%




                                                   7
Nonresponse and coverage error by mode


• 9.3% weighted estimate of cell-phone only households in
  the USPS survey

• Generally, the demographic distributions for mail and
  telephone respondents parallel one another
    Overall, more missing data in mail than telephone

• USPS vs. RDD-only respondents (differences of 3% plus)
    USPS has fewer HS grads, larger underestimate relative to CPS
    USPS has fewer 65+, smaller overestimate relative to CPS
    USPS has fewer 1-adult households and more 4+ adult
      households, closer distribution to CPS




                                                                     8
Nonresponse and coverage error by mode


• Similar story for the BC member list, generally the
  distributions for mail and telephone parallel
    BC-M had fewer “high school grads” and “some college”
    BC-M had more “married” than phone respondents




                                                             9
Comparing survey estimates by mode


• Socially sensitive items (risk behaviors)
    Smoking prevalence,
    Binge drinking
• Factual items
    Exposure to media messages regarding tobacco use
    Workplace smoking policies
• Items assessing beliefs
    Harm in smoking an occasional cigarette
    Smoking increases comfort in social situations



                                                        10
Estimates of risk behaviors

Hypothesis: mail > reporting of risk behaviors

                      USPS       RDD      BC mail BC phone
                        %         %          %       %
Smoking prevalence    14.32     17.01      11.90    10.86
(current smokers)     ± 4.66     ± 1.39    ± 2.85   ± 0.92


Binge drinking        19.80     15.44      21.68    14.56
                      ± 4.78     ± 1.21    ± 3.78   ± 1.03




                                                             11
Percent current smokers, by age, mode: General pop

          25
                     *
          20
 Current 15
 smokers
   (%)   10

           5
                *
           0
                18-24       25-44     45-64   65 and
                                               older

                         USPS   RDD

               * p >0.03
                                                       12
Percent binge drinkers, by age, mode: General pop.


        30
        25

 Binge 20
drinkers 15
   (%)   10
         5
         0
              18-24   25-44     45-64    65 or
                                         older

                  USPS    RDD


                                                     13
Percent current smokers, by age, mode: Member frame


        20

        15
Current
smokers 10
  (%)
         5

         0
             18-24      25-44   45-64   65 or
                                        older

              BC-Mail     BC-Phone


                                                      14
Percent binge drinkers, by age, mode: Member frame


         45
         40
         35
 Binge 30
         25
drinkers
         20
   (%)   15
         10
          5
          0
              18-24      25-44   45-64   65 or
                                         older

               BC-Mail     BC-Phone


                                                     15
More suggestive evidence . . .


• Health Information National Trend Survey (HINTS)
      Small pilot
      USPS frame
      National sample
      “all adults” selection method


 Replicated the smoking result: Estimates for
  18-24 year old mail respondents significantly lower




                                                        16
What about other types of items?


• Factual items
    Reported exposure to media messages
    Work place policy regarding smoking
      • No differences by mode, for either frame
      • No differences by mode for 18-24 years old


• Items assessing beliefs and attitudes
    Some support for age effect for one of the two items…




                                                             17
Any harm in smoking an occasional cigarette? Gen. pop.


          100

          80
 Yes,     60
harmful
  (%)     40

          20

           0
                18-24   25-44    45-64    65 or
                                          older

                    USPS   RDD


                                                         18
Any harm in smoking an occasional cigarette? Member list


         90
         80
         70
  Yes, 60
         50
 harmful
         40
   (%)   30
         20
         10
          0
              18-24      25-44   45-64    65 or
                                          older

               BC-Mail     BC-Phone


                                                       19
Summary and Discussion


• Response rate roughly comparable
• Mail mode includes coverage of cell-only
  population
• Demographic comparison by mode-
   USPS (all adult) to RDD (next birthday) – very similar
    distributions, some differences in educ, adults in HH,
   Member list – also very similar, differences as anticipated
• Some suggestion that 18-24 year olds not
  representative for USPS “all adult” method
   Lower reports of risk behaviors in mail mode



                                                                  20
Household type for 18-24 year olds, by mode


• In USPS sample
    Almost 80% live in family-type households
    About 6% live alone
    About 3% live in un-related households

    Only 13% of 18-24 year olds were cell-only households
In the RDD sample,
    Almost 65% live in family-type households
    About 4% live alone
    Almost 25% live in un-related households



                                                             21
Next steps


• Incorporate methods to broaden “type” of young
  adult respondents
    Advance letter
    Incentive
    Fed-ex the second package


• Explore other respondent selection mechanism
    Next birthday method
    Other methods



                                                   22