National Survey of Recent College Graduates by xuk33092

VIEWS: 39 PAGES: 25

									 Analyzing the Effects of a Data Collection
Mode Change in the 2003 National Survey of
   Recent College Graduates (NSRCG)
 2009 American Association for Public Opinion Research
                 Annual Conference
                      Hollywood, FL
                      May 15, 2009


            Donsig Jang and Zhanyun Zhao
           Mathematica Policy Research, Inc.

                    Kelly H. Kang
          National Science Foundation (NSF)
                Disclaimer
The opinions and assertions in this presentation are
those of the authors and do not reflect the views or
policies of the National Science Foundation (NSF).
                      Outline
   Overview of NSRCG
   Data collection mode change in 2003
   Issues related to mode change/differences
   Random experiment
   Results
   Summary
                        NSRCG
 Sponsored by NSF
 Conducted every two or three years since 1974
 Researchers collect information about education, employment,
  and demographics from recent graduates with bachelor’s or
  master’s degrees in science, engineering, or health fields
 Two-stage sample design: school sample at the first stage and
  graduate sample at the second stage
 For more information on NSRCG:
  www.nsf.gov/statistics/srvyrecentgrads
  Data Collection Mode Change in 2003
 Prior to 2003, the primary means of data collection was
  interviewer-administered computer-assisted telephone
  interviewing (CATI) with a small mail followup.

 In 2003, the data collection protocol changed to multimode
  administration (mail and Web with CATI followup).
   – Self-administration (mail and Web) became the primary
     mode of data collection.
       Why Multimode Collection?
 Declining response rates in telephone surveys
 Seven percent of the total completed surveys in 2001
    NSRCG were via mail (not CATI)
 Characteristics of NSRCG sample persons
    – Frequent Internet users
    – Highly mobile
   Providing additional response options helps increase the
    likelihood of participation
 Since Web completes are less costly, offering Web mode to
    everyone increases the potential savings
     Issues Associated with
 Data Collection Protocol Change
 Time series analysis
 Data quality
 Mode effect
          Experimental Samples

 Study to assess the effects of the data collection protocol
  changes in the 2003 NSRCG estimates

 Randomly split the entire sample of 18,000 into two groups:
   – 16,000 had multimode survey administration (main
     mode of data collection)
   – 2,000 had CATI-only survey administration (same as the
     previous mode)
           Outcome Measures

   Unit response rates
    – A measure of the quality of the survey and the
      potential nonresponse bias

   Item nonresponse rates
    – To evaluate the quality of the survey data

   Survey estimates
Unit Response Rates (Unweighted)
Response Rate for 2003 NSRCG

%
      Response Rates
    by Mode and Gender
%
         Response Rates
    by Mode and Race/Ethnicity
%
        Response Rates
    by Mode and Field of Major
%
        Response Rates
    by Mode and Degree Level
%
Item Nonresponse Rates
       Item Nonresponse Rates
for Race and Salary Data Items by Mode
    Data     CATI Only                   Multimode
    Item
                           Overall     CATI        Mail       Web

    Race*        11.2%        9.4%      12.5%          5.6%   8.7%

    Salary        4.2%        4.0%       5.2%          2.3%   4.3%



    * Indicates the statistical significance at =0.1
              Survey Estimates
 Treated the CATI-only and multimode groups as two
  independent samples and reweighted each group so
  that they represent the same population

 Constructed replicate weights for each sample group
  to calculate standard errors
       Survey Estimates (cont’d.)
 Compared survey estimates in the following five
  categories:
   – Degree levels and field of major
   – Demographic characteristics such as age, gender,
     and race/ethnicity
   – Academic outcomes such as undergraduate GPA,
     major field of highest degree major, and whether an
     associate’s degree was received
   – Financial information such as type/amount of loans
     and other financial support
   – Employment outcomes such as employment field,
     labor force status, and primary work activity
  Percentages of Graduates in Each Bachelor’s Degree Field of Major
            (Before and After Nonresponse Adjustments)
                                     Before Nonresponse        After Nonresponse
                                        Adjustments               Adjustments
       Field of Major                CATI    Multi    RD      CATI   Multi    RD
       Sciences                      77.9    76.8     1.4%    72.6    72.7    -0.1%
       Life and related sciences     15.7    17.7    -12.7%   16.1    16.1     0.0%
       Computer and information
       sciences                      12.2      9.6   21.3%     8.6     9.1    -5.8%
       Physical and related
       sciences                        5.9     4.7   20.3%     3.9     3.8     2.6%
       Psychology                    14.7    16.5    -12.2%   16.4    16.4     0.0%
       Social and related sciences   26.7    25.3     5.2%    24.8    24.6     0.8%
       Engineering                   12.4    13.2     -6.5%   11.8      12    -1.7%

RD stands for relative difference between multimode group estimate and CATI group
estimate.
  Percentages of Graduates by Labor Force Status and Degree Level
            (Before and After Nonresponse Adjustments)
                                             Before Nonresponse           After Nonresponse
                                                Adjustments                  Adjustments

Labor Force Status                          CATI       Multi     RD     CATI Multi        RD
Bachelor’s
In labor force**                               88.8     86.8 2.3%          89.2    86.6     2.9%
In labor force and employed                    82.8     82.0 0.9%          83.4    82.0     1.7%
In labor force and unemployed                   6.0      4.7 21.7%          5.8     4.6    20.7%
Not in labor force**                           11.2     13.2 -18.3%        10.8    13.4   -24.1%
Full-time employed, including all jobs         68.8     68.7 0.1%          69.4    68.8     0.9%
Part-time employed, including all jobs         14.0     13.3 4.8%          14.0    13.2     5.7%
Master’s
In labor force**                               87.2     92.0 -5.5%         87.2    92.0   -5.5%
In labor force and employed                    85.7     87.5 -2.0%         85.8    87.1   -1.5%
Not in labor force**                           12.8      8.0 37.8%         12.8     8.0   37.5%
Full-time employed, including all jobs         75.5     76.0 -0.7%         74.9    75.6   -0.9%
Part-time employed, including all jobs         10.2     11.5 -12.1%        10.8    11.5   -6.5%
 ** Indicates the statistical significance at =0.05; RD stands for relative difference
 between multimode group estimate and CATI group estimate.
                      Summary
 Unit response rates
   – The self-administered data collection mode (mail and
     web) helped improve the response rate.
   – The non-Hispanic Asian group was less likely to respond
     to the survey by CATI only compared to other race/ethnic
     groups.

 Item nonresponse rates
   – In general, the data collection mode did not greatly affect
     overall data quality.
   – Specifically, we found no statistically significant
     differences on a variety of survey items, including
     demographic, degree, financial, and employment
     information.
             Summary (cont’d.)
 Survey estimates
  The CATI-only sample and multimode sample provided
  significantly different survey estimates for proportions
  of graduates in each field of major before nonresponse
  adjustments.
   – Correlation with response propensity (certain major
     groups are more likely to respond to certain modes
     of data collection)
   – Such differences became negligible with
     nonresponse adjusted weights (weighting cells
     include field of major)
            Summary (cont’d.)
 Differences between CATI-only and multimode samples
  existed even after nonresponse adjustments for other
  employment-related survey estimates.
   – Implies that respondents answer the same questions
     differently when interviewed in different data
     collection modes
   – Must be cautious when comparing the results from
     multiple rounds of NSRCG data collection that
     involved different data collection methods

								
To top