FAIRFAX COUNTY DEPARTMENT OF SYSTEMS MANAGEMENT FOR HUMAN SERVICES
Common Pitfalls in Conducting a Survey
Table of Contents
The intent of this brochure is to provide a brief review of potential problems that
may occur during the survey research process. All of these common pitfalls can
Time and Cost 1
bias or invalidate the survey results. For a more thorough discussion of these
pitfalls, please consult the list of technical references shown in the box below.
Initial Planning 2
Questionnaire Design 3
Time and Cost Requirements
Sampling Procedures 4
One of the more quite time consuming. By not
common pitfalls is to allowing adequate time, it might
Nonrespondents 6 underestimate the be tempting to skip one or
amount of time required more of the stages and
Processing the Data 7
to properly conduct a inadvertently introduce errors
Glossary 7 survey. There are many or bias into the results.
different stages involved
in conducting a survey— The amount of time
References: identifying the needed to conduct a survey depends on the
information needs; complexity and type of survey. According to the
American Statistical Association
1429 Duke Street defining and locating American Statistical Association, it may be possible
Alexandria, VA 22314 respondents; deciding to conduct a simple, brief telephone survey in two
http://www.amstat.org how the data will be or three weeks, but most surveys will take
collected; designing and considerably more time. In their pamphlet, How to
American Association for Public
Opinion Research pretesting the Plan a Survey, they state that several months to a
P.O. Box 14263 questionnaire; hiring year is a more typical timeline for most survey
Lenexa, KS 66285
913-310-0118 and training staff or projects.
Council of American Survey how to handle non- As a general rule, surveys are an expensive method
3 Upper Devon response; and coding, of obtaining information. Sponsors of surveys
Port Jefferson, NY 11777 cleaning, tabulating, and should be wary of firms offering to conduct surveys
http://www.casro.org analyzing the data. at prices markedly below those of other firms.
Although some of these Often these “deals” result in second-rate work
Federal Committee on
Statistical Methodology steps can be performed because the firm skips important steps in the
Statistical Policy Working Papers simultaneously, the process, violates proper sampling procedures, and
entire process can be does not employ adequate quality control
Page 2 Common Pitfalls in Conducting a Survey
procedures. The cost of a survey generally increases with the length and complexity of the
questionnaire, the size and complexity of the sample, and the amount and complexity of the data
Not clearly defining the purpose of a survey is the desired population. However, if the desired
a common pitfall. Before beginning a survey, population is a small proportion of the general
it is strongly recommended that the following population, a very large sample would be needed
questions be answered: and this may not be financially feasible.
1. What is the purpose of the survey? 4. If a survey is needed, who will use the
information and how will it be used?
The goal could be to seek specific information,
predict future behavior, or test a particular If the survey results will be made available to the
hypothesis. Whatever the purpose, it needs to public, if subpopulations are to be analyzed
be clearly and concisely stated. separately, or if a crucial decision will be based
on the information, the issue of an acceptable
2. Can the information be
margin of error or precision needs to be
obtained by other means?
addressed. Once the required accuracy
It is quite possible that another level is decided, it will have a direct
organization has the desired bearing on the sample size and the
information. Perhaps, published proper design of the questionnaire.
documents from the Federal
5. Are specific cross-tabulations on
Government or private
certain characteristics desired?
institutions may suffice.
If so, the questionnaire should be
3. Can the intended audience be
designed in such a way that the desired
information can be extracted easily from the
If there is no information that allows the results. The analyst doing the computer
researcher to define and select the members programming should be consulted about whether
of the population to be surveyed, then it may the desired cross-tabulations can be performed
not be possible to conduct a statistically based on the questionnaire's design.
reliable study that can be generalized to the
6. Given the resources available, is the study
entire population. For example, there may be
no information on where persons with a
specific characteristic live. Without location The scope of the project may have to be
information, a simple random sample can be diminished or the project postponed if money,
conducted in the hope of reaching enough of manpower, and time constraints are binding.
The questionnaire should be may be so varied that any
designed with the study comparisons will be
objectives in mind. The meaningless. Open-ended
challenge for the researcher is questions cannot be handled
to design the questions and easily by computer processing
word them in such a way as to methods, especially with a
obtain unbiased information. large number of questions or a
This sounds much easier than it large sample size. If open-
is in practice. Careful attention ended questions are used with
must be paid to avoid personal interviews, the experiences. A question
sentences with multiple interviewer may not code the asking how frequently family
meanings; ones containing verbatim response and the members drink soda may be
technical jargon; and lengthy, failure to do so could change interpreted differently
difficult, or sensitive questions. the entire meaning of the depending on the background
It is not uncommon for the answer. It is strongly of the respondent. One
sponsor conducting the survey suggested that open-ended respondent may interpret this
to inadvertently introduce bias questions be used only when question as asking about soft
in the question design through the question cannot be written drinks while another may
the wording or ordering of the in a multiple choice format. interpret the question as asking
questions. For example, a The obvious advantages of about ice cream drinks.
series of questions querying using multiple choice questions
the respondents on their are the ease in answering, Asking too much information in
concerns about crime followed coding, and analyzing the a single question can result in
by a question asking if more responses. However, care problems. Questions that
tax dollars should be spent on must be taken to see that all the require a respondent to make
public safety is likely to result possible response alternatives calculations such as the percent
in a biased assessment. To are included and that these of income spent on housing
counter this tendency, the alternatives do not overlap. costs are likely to elicit
survey should be reviewed by incorrect answers from
a diverse group. When developing questions for respondents who do not know
a survey, abbreviations and how to calculate a percent. It is
Another common pitfall colloquialisms should be preferable to ask two
involves the use of open-ended avoided. Abbreviations and questions—one for annual
questions rather than multiple colloquialisms often are income and a second for
choice. With open-ended misunderstood or interpreted monthly housing costs. The
questions, the range of answers differently depending on one’s analyst tabulating the data can
Page 4 Common Pitfalls in Conducting a Survey
then calculate the desired percent from these been to a county park. This may happen
two questions. because the respondent may not know which
parks are county parks and which are private,
Requiring respondents to answer questions on federal, state, or regional parks but they do
items for which they have incomplete or little know that they recently visited a park.
knowledge is a common pitfall. People often
answer questions even if they do not have the In mail or written surveys, interviewer bias is
knowledge the researcher requires. This not a concern. When conducting a personal
happens because respondents define the interview survey or telephone survey, the
question differently than the researcher, interviewers must be carefully trained so as to
because respondents do not wish to appear avoid possible bias and avoid influencing the
ignorant or because respondents do not want to responses. For example, if it is crucial to
disappoint the researcher. A question asking a interview a particular respondent, like the head
respondent if they have been to a Fairfax of the household, the interviewer must be
County park during the past year may elicit a informed of the possible bias of using a
“yes” answer even if the respondent has not substitute respondent.
It is rarely feasible to survey made available to users of
the entire population and, thus, the data so they can
a sample of the population is evaluate the usefulness of
selected to be surveyed. the results. Because
Probability samples are the nonprobability samples
only type of samples where the are easier to conduct, it is
results of the survey can be often tempting to use a
generalized to the entire nonprobability sample
population. In a probability when a probability sample
sample, all the members of the is what is really required. cases, judgment sampling,
target population have a Examples of nonprobability where interviewers or "experts"
known, non-zero chance of convenience samples are choose those believed to be the
being included in the sample. stopping customers in a store most representative of the
Only probability samples allow or surveying visitors to a web population, should be avoided.
the researcher to calculate the site. With these methods, valid Different researchers are very
precision of the estimates and generalizations cannot be likely to have different ideas
to specify sampling error. made beyond the people about who the typical subject
These statistics should be interviewed. Also in most should be.
A probability sample, that may affect the probability Often these adjustments are not
however, is only as good as the of being included in a sample. done and this introduces bias
population from which it is in the data and conclusions.
drawn. It is a common pitfall
not to spend the time Another commonplace pitfall is
necessary to ensure that the the substitution of available
population from which the respondents when the selected
sample is drawn is complete respondents cannot be easily
and well defined. Often reached. Unfortunately when
members that should be part of this is done, the sample ceases
the population are excluded to be a true random probability
because of the difficulty of sample. Substitution builds
finding or reaching those Using a telephone survey as an sample size without
members. For example, a example once again, most diminishing nonresponse bias.
telephone survey cannot reach telephone samples are drawn
all households because some using a list of telephone A similar sampling pitfall
households do not have numbers or through random involves enlarging the sample
telephones. Therefore, a digit dialing techniques. size to get “enough returns”
telephone survey can only be Households that have more despite an anticipated low
generalized to those than one telephone line have a response rate. A sample of
households who have greater probability of inclusion 10,000 that elicits 1,000
telephones unless other means in the sample than those responses is less reliable than
are used to reach households households with only one a sample of 200 that elicits 190
without telephones. telephone line. Thus, the responses. With low response
researcher must adjust for rates, the researcher cannot be
In addition, the researcher these different probabilities of sure that those who do not
needs to be aware of other inclusion when sampling or respond are not very different
characteristics of a population else when analyzing the data. from those who do respond.
Following the questionnaire information. Although arise from the questions. In
design and sample selection, a pretesting is frequently over- addition, pretesting is the only
pretest should be conducted on looked, it is the only way to way to discover problems that
a subset of the target discover possible may arise from the field
population to find out if the misinterpretations and procedures to be used.
questions elicit the desired misunderstandings which may Ideally, the pretest should test
Page 6 Common Pitfalls in Conducting a Survey
all phases of the survey techniques. With a mailed debriefings, interviewer
methodology—the question pretest, be especially alert to a debriefings, split panel tests,
wording, the procedures to be large number of "don’t know" and analyzing item
used to conduct the survey, and responses, which might signal a nonresponse. Because each of
the procedures to be used to question which is too lengthy or these pretesting techniques has
analyze the results. Even the difficult. Pretesting can be strengths and weaknesses, it is
most experienced researcher done a variety of ways. Some not unusual to combine several
cannot foresee all the potential of these techniques include of these methods when
problems that may occur when using focus groups, laboratory conducting a pretest.
using new questions or interviews, respondent
Another commonly overlooked area concerns should not be considered a complete solution to
the issue of nonrespondents. According to the the problem. A diligent effort should be made
American Statistical Association, a low response to obtain a high response rate to the survey.
rate produces more questionable results than a High response rates are usually obtained by
small sample, since there is no scientifically employing rigorous follow-up procedures as
valid way to infer the characteristics of part of the survey methodology.
the population that the nonrespondents
represent. So, rather than simply ignoring For item nonresponse, where some, but not
the nonrespondents, every attempt all, of the questions are answered, various
should be made to get them to respond imputation methods can be used in an
to the survey. If further attempt to compensate for possible bias.
correspondence or callbacks do not These methods use the responses to other
substantially increase the response rate, items in the questionnaire as information in
weighting adjustments can be used to assigning values to the nonresponse
help compensate for potential items. A major benefit of imputation is
nonresponse bias. For instance, if the that a data set is constructed with
response rate is lower among low fewer or no missing values. However,
income groups, weighting up the low users of the data should be alerted that the
income respondents to conform to a known data set contains imputed values, so that greater
income distribution might help eliminate precision will not be attributed to the survey
nonresponse bias. However, if there are estimates than is warranted. The techniques
differences between the respondents and discussed in this section can be useful, but they
nonrespondents in each income group, then can be extremely difficult to implement.
nonresponse bias will remain or increase Therefore, only the experienced surveyor
despite attempts to reduce it through data should attempt to employ these methods.
weighting. Therefore, weighting adjustments
Processing the Data
The process of data entering, editing, and confusion on the questionnaires.
analyzing the completed questionnaires is subject
to a great deal of human error. Maintaining a high Ideally, all of the questionnaires should be
level of quality control during these processes checked to see if the responses match what was
takes time, and it is very tempting to skip steps entered in the computer. Many
that ensure that the data are reliable. professional data entry organizations
Before the questionnaires are entered will double enter the data to
into a database, the returned ensure accuracy. In addition,
questionnaires should further editing must be
be scanned to performed to check for
determine if impossible or inconsistent
respondents are answers. This editing usually
leaving comments that is accomplished by using
could indicate that they computerized crosschecks. Also,
misunderstood a question surprising or unintuitive results should be
or are answering a question in explored, since these could be caused by
an inappropriate manner. If faulty computer programming or respondents
pretesting was performed adequately, who did not understand what was being
there should be very little evidence of sought.
• Bias (error): Distorted or inappropriate techniques to (i.e., person, business,
unreliable survey results. analyze the data. household, car, dog, et
All surveys contain some cetera).
• Census: A study using all
bias. Bias is increased
available elements or • Heterogeneous: A population
when the respondents
members of a population. whose elements have
(persons answering the
survey) are not • Data: The collection of
Heterogeneity is the state of
representative of the observations and
population being information resulting from
questioned, when questions the survey process. • Homogeneous: A population
are poorly written or whose elements have similar
• Element (member): The
misunderstood, and when characteristics. Homogeneity
basic unit about which
the researcher uses is the state of being similar.
survey information is sought
Page 8 Common Pitfalls in Conducting a Survey
• Instrument: The tool or device cetera) being described or • Sample: Any portion of the
used for survey measurement; measured by a sample. population, less than the total.
usually a questionnaire.
• Pretest: An initial evaluation of • Sampling Frame: An
• Nonresponse: Unit the survey design by using a exhaustive list of all members
nonresponse refers to the small, subsample of the of the population from which a
refusal of persons selected to intended population for sample can be drawn.
be sampled to participate in a preliminary information.
• Survey: A process of inquiry
survey (i.e., person does not
• Questionnaire: A measuring for the purpose of data
return the mail questionnaire).
device used to query a collection and analysis using
Item nonresponse refers to
population/sample in order to observation, polls,
selected questions left
obtain information for questionnaires, and/or
unanswered by the person
• Respondent: An element or • Statistics: Descriptive
• Population: The universe or
member of the population measures based upon a
collection of all elements
selected to be sampled. probability sample.
(persons, businesses, et
Fairfax County Department of Systems
Management for Human Services
Economic and Demographic Research
12011 Government Center Parkway
Fairfax, Virginia 22035
TTY: 711 (Virginia Relay)
Building Upon the Strengths of a Diverse Community
We’re on the Web!