SURVEY RESEARCH

SURVEY

RESEARCH

Survey Research

 a.k.a. “paper-pencil” measures or

“self-report” measures

 represents the dominant paradigm for

social science research in the last 30

years

 advantages:

 expedience in gathering data

 yields interval level data

Do subjects know their own minds?

 problems:

 problem of “mindfulness,” verbal

reports of mental processes are

suspect

 problem of “non-attitudes,” people

tend to venture an opinion whether

they have one or not

 unreliability of self-reports data:

even factual information can be

misreported

Will subjects tell you if they know?

 intentional misreporting of

information.

 social desirability bias

 reasons for non-disclosure

 embarrassment

 fear of negative sanctions

 shame, public humiliation

 privacy

 sometimes helpful to include a

“lie scale”

Questionnaire Design

and Construction

 phrasing of questions is  avoid lengthy questions;

critical keep questions short,

 avoid vague, nebulous succinct

questions  Bad example: Sara Palin

 Bad example: “What do would be the first female

you think about abortion?” Vice President Secretary

 Bad example: “How many of the United States. Do

children do you have?” you think her gender

wuold be a problem in

 questions must be clear, negotiating with the Arab

unambiguous world, which is a male-

 Good example: “Which of dominated culture?

the following statements  Good example: Would

best represents your Hillary Clinton make a

attitude toward elective good Supreme Court

abortions in the first justice?

trimester?”

 Conduct a “readability

analysis”

questionnaire design and construction--

continued

 avoid double-barreled  avoid slang, jargon,

questions (compound abbreviations and

questions) acronyms

 Bad example: “Do you  Bad example: “Should

favor stricter hand gun

controls and mandatory states regulate PETA and

minimum sentences for the ALF?”

carjackers?”  Bad example: “Do you

 avoid loaded language think hip hop is wack?”

(push polling)  avoid or minimize negative

 Bad example: “Don’t you wording

think that...?” “Isn’t it

true that...?”  Bad example: Is the Bush

 Bad example: Emotionally administration right in not

charged words: “ gang establishing a firm

member,” “welfare deadline for withdrawing

mother,” “extremist from Iraq?

groups,” “spin doctor,”

etc.

open-ended versus close-ended

questions

 Schuman, Ludwig, & Krosnick (1986): 60% of

respondents selected one of four options in a

close-ended format, but only 2.4% mentioned any

of the same four responses in an open-ended

format.

 open-ended: allows subjects more leeway,

flexibility

 “What is your primary ethnic/cultural

background?”

 requires a content analysis of responses



 close-ended or “forced-choice” ties respondents’

hands somewhat

 easy to code the data

 always include an “other_____________”

category

phrasing of questions is critical

 even slight variations in wording can alter

respondents’ answers.

 “occupied territory” versus “contested

territory”

 “assisted suicide” versus “mercy killing”

 “fetus” versus “unborn child”

 make questions concrete, come down the

“ladder of abstraction”

 Use negatively worded questions or

statements sparingly

 sometimes necessary to include

reverse-valenced items to identify a

“response set”

…more on phrasing questions

 Avoid evaluative language

 Phrasing of questions should not imply approval,

disapproval

 Follow-ups should not suggest surprise, liking,

disliking, etc.

 Bad example: Do you think the Democratic

dominated Congress should lift the harsh

restrictions on stem cell research?

 Bad example: Despite its poor track record in

crisis intervention, do you think the United

Nations should intervene in Darfur, Sudan?

respondents’ abilities

 Respondents must be capable and willing

to answer the questions

 May be unaware of topic/issue

 May be unwilling to disclose

 May lie, exaggerate, under-report or over-report

 example: If you ask people what nonverbal cues

they leak when they lie, people may not know.

 example: if you ask people what they would do in

a hostage crisis, how would they know?

additional considerations

 order effects: sequencing of questions can be a

factor

 “halo effect”: carry-over from one item to the

next

 use multiple versions of questionnaire

 “response set”: checking the same response

all the way down a questionnaire

 use “reverse scaling” to detect this

additional considerations

 gender of researcher/respondent can influence

responses

 ethnicity of researcher/respondent can influence

responses

 medium can influence responses

 face-to-face interview, telephone survey, mail survey

 anonymous versus non-anonymous questionnaire

 fatigue: minimize length of surveys

Characteristics of types

of surveys

Type of survey Obtaining Cooperation Cost per advantages disadvantages

sample rate respondent





in person difficult medium high interviewer respondent

rapport, apprehension,

nonverbal cues expensive



telephone easy high medium Fast Limited time,

nonrandom

sample



mall intercept easy medium medium fast, in-person Nonrandom

sample





mail easy low low expensive Nonrandom

sample,

respondent

errors

computer- easy high low automatic data Requires

assisted entry computer

literacy