Survey exercise follow-up
Small, bounded population
All the people who work here
Non-response bias an issue in each case
Survey element – unit ABOUT WHICH information is
collected; unit of analysis
E.g., Person; household; event
Survey universe – theoretical and hypothetical
aggregation of elements
E.g., Americans; users; visits
Population – specified aggregation of survey
Households in the US, defined as one person, or a
collection of related or unrelated persons…as of
Visits to any web page in defined domain during
Sample Design, cont.
Survey population -- Aggregation of elements from
which survey sample is actually selected
Households in 50 states with listed telephone numbers
Sampling unit – elements consider for selection at some
listed residential phone numbers
If multi-stage sample:
State; telephone directory; phone numbers
Sampling frame – list of sampling units
Telephone directories for specified places
Observation unit – element from which info collected
person over 18 who either answers the phone or is
fetched by a person under 18 answering the phone
Consider possible biases of sample design
Mixed methods may give uninterpretable samples
E.g. mail to a group AND hand out in Sproul Plaza –
so what population does your sample represent?
And people on the mailing list who walk through Sproul
have *2* chances of being sampled
Consider practical issues
Confidentiality often precludes getting a list, e.g.
from the university
If you can’t quantify units (people, events), you
can’t sample proportionately
Non-users or non-affiliated users (e.g., potential
students; web users with no particular connection to
the site) are hardest to track down
Purpose of introduction
Justify your asking
Induce their cooperation
Allay possible fears
“this is not a test…”
Help them understand domain of survey, questions
Describing purpose of survey
Enough to gain cooperation, not enough to bias
E.g. “a study of compensation in this field,” not “a study of
whether women’s salaries lag behind men’s in this field”
SHORT but not at the expense of clarity.
Worthwhile topic, survey
Not a waste of time
Interesting to them
If you are not credible, not worth their
Short and easy to answer
Short and clear
The more you ask of them, the more likely
they are to mess up
“please rate from 1, most important, to 7, least
For skip patterns
“Did you use the library today? If not, go to
question 3. If yes,…
For what to do if question does not apply
Choice of questions
Think through: what will you do with
If you don’t know, don’t ask
If you do know, word in a way that
E. g. “Have you ever used a library?”
What difference will it make if they
Choice of questions
Respondents must be qualified to answer
“What would be the best way to bring peace to
the Middle East?”
Respondents must think the topics of the
Not invasion of privacy
Not “dumb questions”; worth their time
Not a test
Of some idea or proposal they may not like
(unless you mean it)
“if we had to cut the budget, should we lay people
off or reduce salaries?”
Time dimension to questions
General, usual, most often:
What do you usually do?
What did you do today?
What have done in the last month?
What do you use this site for? (today? Ever?)
What’s important to you may not be to them
“what did you do the last time you used this site?”
Time and frequency are hard to remember
“Approximately how many times in the last month…”
Resolve any possible ambiguity!
Definitions, illustrations, examples
Have you ever used a library? What’s a library?
“Did you use the library’s homepage?”
Which page is that?
What proportion of your income do you pay for housing?
Individual or household income?
Housing: rent or mortgage; taxes? Insurance?
You cannot use “etc.” unless the reader can complete the
NO: “borrow library books, ask ref questions, etc.”
YES: “borrow library books, ask reference questions, or
otherwise use the library’s resources or staff, in person or
Use parallel construction
What did you use this site for today?
Searching for information
Placing an order
Checking a previous order
Searched for information
I wanted to check on an earlier order
What did you use this site for today? If
What was the question to which “yes” was the
Closed-ended are easier (which is usually,
but not always, a good thing)
Easier for users (recognition, not recall; less
They are less likely to think of different choices
Easier for you to summarize
Be as exhaustive as possible with choices
If you leave out an answer that a user might
reasonably give, they may or may not think to
They may become confused about the question
or the survey
Allow degrees of answers
Yes/ No, always/never are rarely the case
E.g., Usually; Sometimes; Rarely
Allow multiple dimensions of answers
How satisfied are you with your visit today?
Very: found what I was looking for.
Not very: did not find..
What if I found what I needed but with
difficulty? I may have been successful but am
dissatisfied with the experience.
Give clear instructions
Check one? Check the one that best describes…?
Check as many as apply?
Lay out clearly
E.g. Ordinal scales all on one line
How satisfied are you?
Very satisfied somewhat satisfied not at all
Very somewhat not at all
Satisfied satisfied satisfied
NOT: How satisfied are you? Very satisfied
Somewhat satisfied Not at all satisfied
Ranges and intervals
Want answers to be easy to for your reader
Generally, use either:
Intervals of equal size
OR Intervals reflecting meaningful groupings
Age: under 18 18-25 25-40 40-55…
OR intervals get larger when you expect fewer
How many years have you worked here?
Less than 1 1-3 4-10 10-20 more than 20
Years worked here
1 year 10%
2-5 years 15%
6-7 years 20%
8-15 years 50%
Have you checked out a book from
the UC Berkeley Library system?
If you use Pathfinder, Gladis, or
Melvyl to find books or periodicals,
where do you use them?
Terminal at main library
Terminal at my departmental library
I don’t use these services
If you could allocate funds for more salary and had
to pick from the following places, how would you
prioritize where you would obtain these funds?
(1 – first choice to find money through 7 – last place
you would go)
Decreased health coverage
Less vacation/sick leave
How long has your library offered live
online reference service?
How long have you been a Reference
Librarian? (providing reference services
comprises at least 40% of your day)
Up to 5 years
Between 5 and 10 years
Between 10 and 15 years
Over 15 years
Which part(s) of this web site
interests you the most?
Which part(s) of this web site do you
find the most uninteresting?
What parts of this web site do you
like the most?
Which parts of this web site do you
dislike the most?
1. Have you ever used the following online
2. If you checked any box in question 1,
how important do you think online help
would be in each of these activities? (only
respond as applicable)
Absolutely unimportant |--|--|--||--|--|--| absolutely important