Introducing… by sofiaie


									A poor prognosis for survey methods? Assessing the
health of survey methods in social research

                                McCaughey Centre
                              Survey Methods Seminar
                                     May 2008
                      The Topic

Survey methods have been a stalwart tool for social science research, however,
traditional approaches to survey methods are becoming increasingly difficult to
undertake. Letterboxes are overflowing with „junk‟ mail, households are fatigued by
unsolicited telephone calls and electronic home protection systems are turning
homes into fortresses. With low response rates - what does this mean for ideals of
„representativeness‟ and what does „representativeness‟ mean in contexts of
increasing diversity within populations?

For survey methods to adapt to these changing contexts for social science
research and to continue to provide reliable and valid insights into people‟s lives
there is a need for innovative approaches.

                                 Sample frames

                                Response Rates

                            Methodological Innovations

Credentials              About the Social Research Centre

    •   Established in 2000 as a specialist provider of survey research
        services to government and academia

    •   Operates as a social research supplier under the umbrella of the
        market and social research industry
         • 25 permanent staff
         • 180 telephone interviewers
         • 110 seat Call Centre

    •   Our aim is to enhance knowledge and contribute to informed
        decision-making by undertaking high quality social and health

                    About the Social Research Centre

Services provided

 •   large-scale survey design, conduct and management
 •   longitudinal surveys
 •   population health research
 •   community attitudes research
 •   campaign evaluation research
 •   research amongst persons of culturally and linguistically
     diverse backgrounds
 •   Programme evaluation and policy development research

                      About the Social Research Centre

Research Areas
   social capital and community   environmental attitudes and
   strength                       behaviours
   community well-being and       welfare reform and income support
   population health monitoring   poverty and social exclusion
   health promotion and social    parenting and families
   cancer research                tobacco, drugs and alcohol
   employment and unemployment    mental health

   education and training         Racism
   skills shortages               immigration and multiculturalism
   carers and persons with        violence
   Indigenous research            crime, justice and victimisation
   road safety
Random Digit Dialling (RDD)

RDD                      Background

      The rise of RDD and the demise of the Electronic White Pages (EWP)

  •     Up until quite recently the Electronic White Pages was
        considered complete and accurate for most survey research

  •     The final version of the Desktop Marketing Systems (DtMS)
        Electronic White Pages (EWP) product was released in July
        2004 and is still widely used by market research companies
        but is no longer used for high quality social research
  •     Random Digit Dialing (RDD) methodologies are now the norm
        rather than the exception
  •     One of the main barriers to the efficient use of RDD has been
        overcome by the development of automatic dialing

EWP versus RDD         Methods of RDD generation

  •   Generation process involves
       Launched in 2003
       •   Undertaking a random selection of “seed” numbers from
       Government‟s 5 year plan for Queensland women
           the 2004 Electronic White Pages
       Initiatives to address areas where women experience inequity,
        •    Retaining the and digit
       under-representationeightneed stem and deleting the last two

       •   Randomly generating two numbers to append to the
           eight digit stem

       •   Washing the resultant randomly generated ten digit
           numbers against business listings

       •   Send those records for which we get a confirmed
           address a primary approach letter

EWP versus RDD Selected characteristics

  Selected demographic characteristics of persons with listed and unlisted phone numbers

  Characteristics                                      Listed             Unlisted
  VPHS 2005 (n=7,560)                                  5,449                 2,111
  Employed                                              50.7                 60.3
  Retired                                               33.2                 15.4
  University educated                                   22.9                 28.3
  Household income <$40k pa                             49.7                 46.6
  Language spoken at home - English only                89.8                 84.5
  Dependent children in household                       32.0                 42.3
  Divorced / separated                                  11.0                 16.4
  Resident in area less than one year                    1.7                 18.9
  Renting                                               10.6                 30.9

EWP versus RDD Selected characteristics

Selected survey measures for persons with listed and unlisted phone numbers

 Characteristics                                                                    Listed   Unlisted
 VPHS 2005 (n=7,560)                                                                5,449     2,111
 Holds private health insurance                                                      52.5      43.6
 Daily smoker                                                                        14.6      22.0
 Has an alcoholic drink every day                                                    18.9      13.6
 Sought professional help for mental health related illness                          8.6       12.2
 Could raise $2,000 within 2 days in an emergency                                    85.0      79.4
 ICVS 2004 (n=6,000)                                                                4,730     1,270
 Victim of at least one crime in last 5 years                                        51.3      58.0
 Victim of violent crime in last 5 years                                             17.5      24.4
 Feels safe waiting for public transport after dark                                  60.7      55.9

 All of the results are statistically significant at the 95% confidence interval.
RDD                   Improved representation

  •   RDD samples can access different persons in terms of:
       Launched in 2003
       •    Age distribution
       Government‟s 5 year plan for Queensland women
       •    Demographic characteristics
       Initiatives to address areas where women experience inequity,
       under-representation and need behavioural variables
        •    Key usage, attitudinal or

  •   Inclusion of persons with unlisted numbers in the sample
      universe by using RDD techniques can achieve a more
      representative sample of the general population

RDD                           Limitations of RDD frames

  •   Little or no coverage of areas poorly represented in
      source listing e.g. emerging growth areas
       Launched in 2003

  •    Government‟s 5 year „mobile only‟ households. Non
      The emergence of plan for Queensland women
       Initiatives of households without a experience inequity,
      coverage to address areas where women landline (used to be
      1-2% but now probably around 13%)1,2
       under-representation and need

  •   Decreasing ability to obtain address matches and send
      out primary approach letters

  1   12.8% of households in the US in 2006. National Health Interview, Public Opinion Quarterly,
      Vol. 71
  2   One in four adults aged 18 to 24 years, Ibid.

Response Rates         Are response rates declining?

     Generally, probably yes but not inevitably so
       Launched in 2003
     In market research it has been predicted that …
       Government‟s 5 year plan for Queensland women
          “Reach is increasing but co-operation is declining and non-
          contact is address areas where women experience inequity,
       Initiatives to increasing, leading to a 3% decline in response rate
       under-representation and need
          per annum” (Bednall, 2000).

    Is the same true in social research …Possible yes but not
    inevitably so and certainly not to the same extent …. It
    depends on how hard you work…

Response Rates              Selected findings

                          2005 2006
    Victorian Population Health Survey          2007
     Response Rate (%) 67.9        61.5         69.3

    Health system       Feb-05 May-06 Aug-06 Nov-06 Feb-07 May-07 Aug-07 Nov-07
             Government‟s 67.9 plan for Queensland women
     Response Rate (%)    5 year 61.5    59.3   59.2  60.8  54.7   65.9   61.6

   Smoking and Health Survey        2005        2006     2007
    Response Rate (%)      55.2     54.9        43.3       51

    Community Attitudes to Road Safety          2003     2004    2005    2006
     Response Rate (%)                            68       64      73      66

    NTS                      2002        2003     2004    2005    2006    2007
     Response Rate (%)       43.4        43.4     52.3    44.2    30.8    40.2

    National Drug Survey            2004         2007
     Response Rate (%)                38           42

Response Rates              The current state of play

         Overall only marginally lower than they were several years ago
         or at least not substantially lower. Certainly no precipitous
         decline. This in 2003
                        has been achieved by:

     •       Government‟s 5 year
          Extended call regimes plan for Queensland women

     •    Longer fieldwork periods

     •    Increased use of Primary Approach Letters

     •    Increased reliance on bilingual interviewing

     •    More sophisticated interviewer training

     •    Leaving messages on answering machines

     •    Increased reliance on survey sponsors to promote their surveys

     •    Creating an internet presence for surveys

     •    More calls to mobiles to obtain interviews with selected respondents
Population Research                      The future?

       • Access to the Integrated Public Number Data Base is needed to
         overcome these deficiencies
       • Mixed / multi modes of data collection women
           Government‟s 5 year plan for Queensland
       • On line access panels1
       • Access to mobile phone directories

      •         Advocacy is needed

      •         Further research into advances in survey methodology is

      •         Survey methods are certain to adapt to the changed
                environment as the alternative is sub standard population

          1.   2006 Census, 58% of households had an internet connection, 37% had broadband.

Population Research      The future?

      A poor prognosis for survey methods?

       No, just continued innovation …

CATI              Computer Assisted Telephone Interviewing


       Economical                              

       Good response rates                     ??

       Direct data entry
       Computerised survey management          
       system (SMS)
       Random Digit Dialling (RDD)
       Reaches silent numbers and recent       ?
       More disconnected, non-household, fax   __

       Not completely random                   ??

RDD               Random Digit Dialing

 The proceedings from the first CATI Population Health
 Forum in 1998 record the following advantages of CATI and
     quick
     economical
     good response rates
     direct data entry
     computerised survey management system (SMS)
 Random Digit Dialling (RDD)
     reaches silent numbers and recent connections
     more disconnected, non-household, fax
     not completely random


To top