Document Sample
pitfalls Powered By Docstoc

                                                                                    April 2003

                                                                                    Informational Brochure

                     Common Pitfalls in Conducting a Survey

 Table of Contents
                                    The intent of this brochure is to provide a brief review of potential problems that
                                    may occur during the survey research process. All of these common pitfalls can
  Time and Cost             1
                                    bias or invalidate the survey results. For a more thorough discussion of these
                                    pitfalls, please consult the list of technical references shown in the box below.
  Initial Planning          2

  Questionnaire Design      3
                                    Time and Cost Requirements
  Sampling Procedures       4
                                    One of the more              quite time consuming. By not
  Pretesting                5
                                    common pitfalls is to        allowing adequate time, it might
  Nonrespondents            6       underestimate the            be tempting to skip one or
                                    amount of time required      more of the stages and
  Processing the Data       7
                                    to properly conduct a        inadvertently introduce errors
  Glossary                  7       survey. There are many       or bias into the results.
                                    different stages involved
                                    in conducting a survey—      The amount of time
      References:                   identifying the              needed to conduct a survey depends on the
                                    information needs;           complexity and type of survey. According to the
American Statistical Association
      1429 Duke Street              defining and locating        American Statistical Association, it may be possible
    Alexandria, VA 22314            respondents; deciding        to conduct a simple, brief telephone survey in two
       703-684-1221           how the data will be         or three weeks, but most surveys will take
                                    collected; designing and     considerably more time. In their pamphlet, How to
American Association for Public
      Opinion Research              pretesting the               Plan a Survey, they state that several months to a
       P.O. Box 14263               questionnaire; hiring        year is a more typical timeline for most survey
     Lenexa, KS 66285
       913-310-0118                 and training staff or        projects.
                                    contractors; determining
  Council of American Survey        how to handle non-           As a general rule, surveys are an expensive method
    Research Organizations
         3 Upper Devon              response; and coding,        of obtaining information. Sponsors of surveys
   Port Jefferson, NY 11777         cleaning, tabulating, and    should be wary of firms offering to conduct surveys
        631-928-6954           analyzing the data.          at prices markedly below those of other firms.
                                    Although some of these       Often these “deals” result in second-rate work
      Federal Committee on
     Statistical Methodology        steps can be performed       because the firm skips important steps in the
Statistical Policy Working Papers   simultaneously, the          process, violates proper sampling procedures, and
                                    entire process can be        does not employ adequate quality control
Page 2                                                                  Common Pitfalls in Conducting a Survey

procedures. The cost of a survey generally increases with the length and complexity of the
questionnaire, the size and complexity of the sample, and the amount and complexity of the data

Initial Planning
Not clearly defining the purpose of a survey is    the desired population. However, if the desired
a common pitfall. Before beginning a survey,       population is a small proportion of the general
it is strongly recommended that the following      population, a very large sample would be needed
questions be answered:                             and this may not be financially feasible.
1. What is the purpose of the survey?              4. If a survey is needed, who will use the
                                                      information and how will it be used?
The goal could be to seek specific information,
predict future behavior, or test a particular      If the survey results will be made available to the
hypothesis. Whatever the purpose, it needs to      public, if subpopulations are to be analyzed
be clearly and concisely stated.                   separately, or if a crucial decision will be based
                                                   on the information, the issue of an acceptable
2. Can the information be
                                                               margin of error or precision needs to be
   obtained by other means?
                                                               addressed. Once the required accuracy
It is quite possible that another                              level is decided, it will have a direct
organization has the desired                                   bearing on the sample size and the
information. Perhaps, published                                proper design of the questionnaire.
documents from the Federal
                                                             5. Are specific cross-tabulations on
Government or private
                                                             certain characteristics desired?
institutions may suffice.
                                                             If so, the questionnaire should be
3. Can the intended audience be
                                                             designed in such a way that the desired
                                                   information can be extracted easily from the
If there is no information that allows the         results. The analyst doing the computer
researcher to define and select the members        programming should be consulted about whether
of the population to be surveyed, then it may      the desired cross-tabulations can be performed
not be possible to conduct a statistically         based on the questionnaire's design.
reliable study that can be generalized to the
                                                   6. Given the resources available, is the study
entire population. For example, there may be
no information on where persons with a
specific characteristic live. Without location     The scope of the project may have to be
information, a simple random sample can be         diminished or the project postponed if money,
conducted in the hope of reaching enough of        manpower, and time constraints are binding.
                                                                                                    Page 3

Questionnaire Design
The questionnaire should be          may be so varied that any
designed with the study              comparisons will be
objectives in mind. The              meaningless. Open-ended
challenge for the researcher is      questions cannot be handled
to design the questions and          easily by computer processing
word them in such a way as to        methods, especially with a
obtain unbiased information.         large number of questions or a
This sounds much easier than it      large sample size. If open-
is in practice. Careful attention    ended questions are used with
must be paid to avoid                personal interviews, the            experiences. A question
sentences with multiple              interviewer may not code the        asking how frequently family
meanings; ones containing            verbatim response and the           members drink soda may be
technical jargon; and lengthy,       failure to do so could change       interpreted differently
difficult, or sensitive questions.   the entire meaning of the           depending on the background
It is not uncommon for the           answer. It is strongly              of the respondent. One
sponsor conducting the survey        suggested that open-ended           respondent may interpret this
to inadvertently introduce bias      questions be used only when         question as asking about soft
in the question design through       the question cannot be written      drinks while another may
the wording or ordering of the       in a multiple choice format.        interpret the question as asking
questions. For example, a            The obvious advantages of           about ice cream drinks.
series of questions querying         using multiple choice questions
the respondents on their             are the ease in answering,          Asking too much information in
concerns about crime followed        coding, and analyzing the           a single question can result in
by a question asking if more         responses. However, care            problems. Questions that
tax dollars should be spent on       must be taken to see that all the   require a respondent to make
public safety is likely to result    possible response alternatives      calculations such as the percent
in a biased assessment. To           are included and that these         of income spent on housing
counter this tendency, the           alternatives do not overlap.        costs are likely to elicit
survey should be reviewed by                                             incorrect answers from
a diverse group.                     When developing questions for       respondents who do not know
                                     a survey, abbreviations and         how to calculate a percent. It is
Another common pitfall               colloquialisms should be            preferable to ask two
involves the use of open-ended       avoided. Abbreviations and          questions—one for annual
questions rather than multiple       colloquialisms often are            income and a second for
choice. With open-ended              misunderstood or interpreted        monthly housing costs. The
questions, the range of answers      differently depending on one’s      analyst tabulating the data can
Page 4                                                                  Common Pitfalls in Conducting a Survey

  then calculate the desired percent from these         been to a county park. This may happen
  two questions.                                        because the respondent may not know which
                                                        parks are county parks and which are private,
  Requiring respondents to answer questions on          federal, state, or regional parks but they do
  items for which they have incomplete or little        know that they recently visited a park.
  knowledge is a common pitfall. People often
  answer questions even if they do not have the         In mail or written surveys, interviewer bias is
  knowledge the researcher requires. This               not a concern. When conducting a personal
  happens because respondents define the                interview survey or telephone survey, the
  question differently than the researcher,             interviewers must be carefully trained so as to
  because respondents do not wish to appear             avoid possible bias and avoid influencing the
  ignorant or because respondents do not want to        responses. For example, if it is crucial to
  disappoint the researcher. A question asking a        interview a particular respondent, like the head
  respondent if they have been to a Fairfax             of the household, the interviewer must be
  County park during the past year may elicit a         informed of the possible bias of using a
  “yes” answer even if the respondent has not           substitute respondent.

  Sampling Procedures
  It is rarely feasible to survey    made available to users of
  the entire population and, thus,   the data so they can
  a sample of the population is      evaluate the usefulness of
  selected to be surveyed.           the results. Because
  Probability samples are the        nonprobability samples
  only type of samples where the     are easier to conduct, it is
  results of the survey can be       often tempting to use a
  generalized to the entire          nonprobability sample
  population. In a probability       when a probability sample
  sample, all the members of the     is what is really required.        cases, judgment sampling,
  target population have a           Examples of nonprobability         where interviewers or "experts"
  known, non-zero chance of          convenience samples are            choose those believed to be the
  being included in the sample.      stopping customers in a store      most representative of the
  Only probability samples allow     or surveying visitors to a web     population, should be avoided.
  the researcher to calculate the    site. With these methods, valid    Different researchers are very
  precision of the estimates and     generalizations cannot be          likely to have different ideas
  to specify sampling error.         made beyond the people             about who the typical subject
  These statistics should be         interviewed. Also in most          should be.
                                                                                                 Page 5

A probability sample,             that may affect the probability    Often these adjustments are not
however, is only as good as the   of being included in a sample.     done and this introduces bias
population from which it is                                          in the data and conclusions.
drawn. It is a common pitfall
not to spend the time                                                Another commonplace pitfall is
necessary to ensure that the                                         the substitution of available
population from which the                                            respondents when the selected
sample is drawn is complete                                          respondents cannot be easily
and well defined. Often                                              reached. Unfortunately when
members that should be part of                                       this is done, the sample ceases
the population are excluded                                          to be a true random probability
because of the difficulty of                                         sample. Substitution builds
finding or reaching those         Using a telephone survey as an     sample size without
members. For example, a           example once again, most           diminishing nonresponse bias.
telephone survey cannot reach     telephone samples are drawn
all households because some       using a list of telephone          A similar sampling pitfall
households do not have            numbers or through random          involves enlarging the sample
telephones. Therefore, a          digit dialing techniques.          size to get “enough returns”
telephone survey can only be      Households that have more          despite an anticipated low
generalized to those              than one telephone line have a     response rate. A sample of
households who have               greater probability of inclusion   10,000 that elicits 1,000
telephones unless other means     in the sample than those           responses is less reliable than
are used to reach households      households with only one           a sample of 200 that elicits 190
without telephones.               telephone line. Thus, the          responses. With low response
                                  researcher must adjust for         rates, the researcher cannot be
In addition, the researcher       these different probabilities of   sure that those who do not
needs to be aware of other        inclusion when sampling or         respond are not very different
characteristics of a population   else when analyzing the data.      from those who do respond.

Following the questionnaire       information. Although               arise from the questions. In
design and sample selection, a    pretesting is frequently over-      addition, pretesting is the only
pretest should be conducted on    looked, it is the only way to       way to discover problems that
a subset of the target            discover possible                   may arise from the field
population to find out if the     misinterpretations and              procedures to be used.
questions elicit the desired      misunderstandings which may         Ideally, the pretest should test
Page 6                                                                   Common Pitfalls in Conducting a Survey

  all phases of the survey           techniques. With a mailed            debriefings, interviewer
  methodology—the question           pretest, be especially alert to a    debriefings, split panel tests,
  wording, the procedures to be      large number of "don’t know"         and analyzing item
  used to conduct the survey, and    responses, which might signal a      nonresponse. Because each of
  the procedures to be used to       question which is too lengthy or     these pretesting techniques has
  analyze the results. Even the      difficult. Pretesting can be         strengths and weaknesses, it is
  most experienced researcher        done a variety of ways. Some         not unusual to combine several
  cannot foresee all the potential   of these techniques include          of these methods when
  problems that may occur when       using focus groups, laboratory       conducting a pretest.
  using new questions or             interviews, respondent

  Another commonly overlooked area concerns              should not be considered a complete solution to
  the issue of nonrespondents. According to the          the problem. A diligent effort should be made
  American Statistical Association, a low response       to obtain a high response rate to the survey.
  rate produces more questionable results than a         High response rates are usually obtained by
  small sample, since there is no scientifically           employing rigorous follow-up procedures as
  valid way to infer the characteristics of                 part of the survey methodology.
  the population that the nonrespondents
  represent. So, rather than simply ignoring               For item nonresponse, where some, but not
  the nonrespondents, every attempt                           all, of the questions are answered, various
  should be made to get them to respond                        imputation methods can be used in an
  to the survey. If further                                    attempt to compensate for possible bias.
  correspondence or callbacks do not                         These methods use the responses to other
  substantially increase the response rate,                  items in the questionnaire as information in
  weighting adjustments can be used to                             assigning values to the nonresponse
  help compensate for potential                                       items. A major benefit of imputation is
  nonresponse bias. For instance, if the                              that a data set is constructed with
  response rate is lower among low                                  fewer or no missing values. However,
  income groups, weighting up the low                           users of the data should be alerted that the
  income respondents to conform to a known               data set contains imputed values, so that greater
  income distribution might help eliminate               precision will not be attributed to the survey
  nonresponse bias. However, if there are                estimates than is warranted. The techniques
  differences between the respondents and                discussed in this section can be useful, but they
  nonrespondents in each income group, then              can be extremely difficult to implement.
  nonresponse bias will remain or increase               Therefore, only the experienced surveyor
  despite attempts to reduce it through data             should attempt to employ these methods.
  weighting. Therefore, weighting adjustments
                                                                                                      Page 7

Processing the Data
The process of data entering, editing, and           confusion on the questionnaires.
analyzing the completed questionnaires is subject
to a great deal of human error. Maintaining a high   Ideally, all of the questionnaires should be
level of quality control during these processes      checked to see if the responses match what was
takes time, and it is very tempting to skip steps              entered in the computer. Many
that ensure that the data are reliable.                              professional data entry organizations
Before the questionnaires are entered                                   will double enter the data to
into a database, the returned                                             ensure accuracy. In addition,
questionnaires should                                                       further editing must be
be scanned to                                                                performed to check for
determine if                                                                 impossible or inconsistent
respondents are                                                              answers. This editing usually
leaving comments that                                                        is accomplished by using
could indicate that they                                       computerized crosschecks. Also,
misunderstood a question                                    surprising or unintuitive results should be
or are answering a question in                            explored, since these could be caused by
an inappropriate manner. If                              faulty computer programming or respondents
pretesting was performed adequately,                      who did not understand what was being
there should be very little evidence of                 sought.

•   Bias (error): Distorted or          inappropriate techniques to        (i.e., person, business,
    unreliable survey results.          analyze the data.                  household, car, dog, et
    All surveys contain some                                               cetera).
                                    •   Census: A study using all
    bias. Bias is increased
                                        available elements or          •   Heterogeneous: A population
    when the respondents
                                        members of a population.           whose elements have
    (persons answering the
                                                                           dissimilar characteristics.
    survey) are not                 •   Data: The collection of
                                                                           Heterogeneity is the state of
    representative of the               observations and
                                                                           being dissimilar.
    population being                    information resulting from
    questioned, when questions          the survey process.            •   Homogeneous: A population
    are poorly written or                                                  whose elements have similar
                                    •   Element (member): The
    misunderstood, and when                                                characteristics. Homogeneity
                                        basic unit about which
    the researcher uses                                                    is the state of being similar.
                                        survey information is sought
    Page 8                                                                                         Common Pitfalls in Conducting a Survey

•      Instrument: The tool or device                                cetera) being described or            •    Sample: Any portion of the
       used for survey measurement;                                  measured by a sample.                      population, less than the total.
       usually a questionnaire.
                                                                 •   Pretest: An initial evaluation of     •    Sampling Frame: An
•      Nonresponse: Unit                                             the survey design by using a               exhaustive list of all members
       nonresponse refers to the                                     small, subsample of the                    of the population from which a
       refusal of persons selected to                                intended population for                    sample can be drawn.
       be sampled to participate in a                                preliminary information.
                                                                                                           •    Survey: A process of inquiry
       survey (i.e., person does not
                                                                 •   Questionnaire: A measuring                 for the purpose of data
       return the mail questionnaire).
                                                                     device used to query a                     collection and analysis using
       Item nonresponse refers to
                                                                     population/sample in order to              observation, polls,
       selected questions left
                                                                     obtain information for                     questionnaires, and/or
       unanswered by the person
                                                                     analysis.                                  interviews.
                                                                 •   Respondent: An element or             •    Statistics: Descriptive
•      Population: The universe or
                                                                     member of the population                   measures based upon a
       collection of all elements
                                                                     selected to be sampled.                    probability sample.
       (persons, businesses, et

    Fairfax County Department of Systems
    Management for Human Services

      Economic and Demographic Research
      12011 Government Center Parkway
      Suite 222
      Fairfax, Virginia 22035

      Phone: 703-324-4519
      TTY: 711 (Virginia Relay)
      Fax: 703-803-8598

    Building Upon the Strengths of a Diverse Community

                                   We’re on the Web!