Writing Good Questions: Part 2
Slide 1 (No Audio)
Slide 2
Another rule for writing good questions is to be certain that you are asking only one question at
a time. The question that asks more than one question at a time is called a double barreled
question. Here are two examples of double barreled questions that you should avoid. The first
one, “Do you believe that McDonald‟s has fast and courteous service?” Well, I may believe it‟s
fast, but I may also believe it is rude. How do I answer that question? I may believe it‟s slow, but
reasonably courteous. Because there are two questions here, I don‟t know which question to
focus on and my answer will be muddled. The second question; “Are you in favor of mandatory
seat belt regulations in order to reduce the incidents of automobile fatalities or are you opposed
to mandatory seat belt regulations even though this would lead to more automobile fatalities?”
At first blush this may not seem to be a double barreled question but it is because it assumes,
automatically for respondents, that seat belts reduce automobile fatalities. Although the
statistics strongly bear that out, some people don‟t believe that seat belts increase safety and
reduce automobile fatalities. In this particular, rather long winded question, there‟s actually a
hidden second question; do you believe that seat belts reduce automobile fatalities?
Slide 3
Here‟s an example of a double barreled question and three different ways to fix it. The double
barreled version is; “Should the city build a new swimming pool that includes lanes for
swimming laps, that is not enclosed for winter use?” As you can see, there is really two
questions here, hence the revision below; “Should the city build a new swimming pool that
includes lanes for swimming laps?” Yes or no? Then; “Should the city build a new swimming
pool that is enclosed for winter use?” That‟s one perfectly reasonable way to divide the original
double barreled question into two relatively simple and straightforward questions. If you were
insistent on the revision that left but one question, you might consider one of the following two
alternatives, although I wouldn‟t suggest either one because they are much wordier and more
complex, but nonetheless, one could use one of these other alternatives if it was important to
leave the question as a single question.
Slide 4
The following examples are not double barreled, but they are somewhat a kin to them.
Unfortunately, I‟ve been guilty of doing this kind of thing in the past in order to conserve the
number of pages I needed to print. As you can see, these formats are fairly condensed and
sometimes it seems critical to save photocopying and postage costs for mail questionnaires.
Nonetheless, the author of this example is correct that one should only ask one question at a
time. In the first case; “Please indicate the extent that each of the following services of our
organization are important to you and for those that are very important, please indicate how
frequently you have used that service during that past six months.” What someone does is to
first consider the service, then consider how important. If they answered „very‟, then they‟re
supposed to indicate by writing in a number of how often they‟ve used that service in the last six
months. This seems like a reasonable way to do it, but it‟s convoluted and it requires people to
think in terms of answering two questions at the same time. A better revision is shown below.
Splitting question 16, above, into 16 and 17 below, makes much more sense, as far as easing
respondent‟s burdens and making the task simpler. Instead of the convoluted initial question,
Page | 1
the revised question first asks; “How often have you used each of these services from our
organization during the last six months?” People can consider how many times they‟ve used
customer credit, next day delivery, and the „no questions asked‟ return policy. Then they can
think about how important each of those services is to them. It‟s better to ask one question at a
time because that reduces the amount of cognitive stress on respondents and should improve
the quality of the answers they provide us.
Slide 5
Another rule for writing good questions is stating the alternatives explicitly. You‟ll notice that
version A; “Would you normally buy pasta in a jar if available in a store where you normally
shop?” This is not nearly as explicit as version B; if pasta in the jar and the canned pasta
product that you are using were both available in the store where you normally shop, would you
buy only the canned pasta product, buy only the pasta in the jar product, or buy both products.
When possible be certain to state the alternatives as explicitly as possible.
Slide 6
Good questions are both relevant to the respondent and should not overly tax that respondent‟s
memory. As I‟ve discussed in other lectures, respondent‟s memory is faulty and can be stressed
beyond the point where the respondent can give reliable answers. As a result, the respondent is
asked a question that overtaxes his or her memory and may forget having done certain things or
may have remembered doing it more recently than what was the case or may even,
unintentionally, create memories of having done things that were never done. For example, you
may recall that you went to the movies last month when you actually went three months ago.
Fortunately, the way memories work; we can create things that seem to be actual behaviors,
when in fact those behaviors never occurred.
Slide 7
Here are some examples dealing with memory as it relates to time by helping the respondents
by providing the appropriate time reference. Here we have three examples; how many times in
the last three years have you gone to see a doctor or other health care practitioner? Asking me
to recall how many times in doing so, accurately, is very small. Instead, the revised question
asks about the same behavior but over the last six months. Over a shorter period of time, I‟m
much more likely to remember accurately. The second example; how many times in the last six
months have you eaten away from home? Unlike physician visits, dining out is a relatively
frequent occurrence and not especially memorable. As a result, people will probably construct
models of their behaviors and answer this question consistent with those models. For example,
if I believe I eat away from home twice a week and there are 26 weeks in 6 months, then in the
last 6 months I‟ve eaten away from home 52 times. That may be totally unrepresentative to the
true number of time I ate away from home. A revision would ask about a shorter period of time
because dining out is not an especially memorable occurrence. Hence the revision; how many
times per week do you eat a meal away from home? Although that particular solution asks
people to model their behaviors and I advocate that you merely ask someone how many times
in the last week they did eat away from home and assume that that is going to be, on average,
more reflective of their true behaviors than any models of their behaviors they can develop. That
would be my solution to the problem posed in the second example. In the third problem; how
many times have you played golf so far this year? Do you mean the last 12 months? It‟s
ambiguous if we‟re talking about calendar years or some period of time. Hence the revision;
during the last calendar year of 1998 did you play golf? At least that time reference is
Page | 2
unambiguous, although it is a somewhat long period of time upon which to ask people about
their golfing behaviors.
Slide 8
A good question will certainly avoid taxing respondents by asking them to make unnecessary
calculations. Look at the problem version of the sample question; what percent of time, on trips
spent away from home, in 1998 were for business reasons? Then some percent is to be
indicated. To answer that question, I first have to think about how many nights I was away from
home during 1998, and then come up with that number. Then I have to figure out the number of
nights I was away from home on business during 1998. Then I have to divide that second
number by the first number to indicate a percentage. That‟s far too complex a question for
someone to answer meaningfully. A better, but not great solution is provided in the revision; how
many nights did you spend away from home on trips of any kind in 1998? Indicate the number
of nights. And; how many nights away from home were because of business trips? Then you
indicate whatever smaller number and then leave it to the analyst to indicate what percentage or
compute the number of nights away for business trips is relative to the total number of nights.
My concern with this question has nothing to do with unnecessary calculations. It‟s just the
ability of the respondent to recall all the trips away from home during 1998. Regardless, the
revised version is certainly superior to the first version because it does not require respondents
to make a calculation that the researcher can easily make from the data provided by the
respondent.
Slide 9
This is an example of the type of convoluted questions and answers you should avoid. In the
truly problematic example; which of these five statements best describes this dean; innovative
but lacking leadership qualities, about the same on innovation and leadership qualities, stronger
on leadership than innovation, a born leader, a real innovator? These choices are so muddled, I
couldn‟t quite figure out whether or not they are exhaustive or mutually exclusive. This thing is
just a tremendous muddle. Here are two different revisions of this totally muddled question. In
the first revision, the question is divided into two different questions; to what extent has the dean
demonstrated strong leadership qualities and frequency of occurrence and then, to what extent
has the dean demonstrated an ability to innovate. This eliminates the double barreled nature of
the original question. Alternatively, one could try the solution at the bottom of the page; which
one of the following do you feel best describes the dean; a strong leader, a strong innovator,
both a strong leader and innovator, or neither a strong leader nor innovator. In this case, each of
these answers is mutually exclusive and exhaustive.
Slide 10
This pointer for writing sound questionnaires suggests that we could ask respondents questions
that are so specific that they could not possibly have an accurate readymade answer. The
problematic version here is; about how many books have you read for leisure over the past
year? In assuming the number of the year is unambiguous, we‟ll talk about the previous twelve
month period. It would be almost impossible for someone, unless they‟ve read very little for
leisure, to indicate anything approaching an accurate number for books. Instead we might try
one of the two revisions below. We might ask about how many books have you read for leisure
over the past year and then provide some broad categories. Alternatively, you might try the last
version; about how many books have you read for leisure during the past year? In which case,
you‟ve got a much larger set of books in each of the options and the version above, we assume,
Page | 3
has light readers and the version below, we‟re assuming, heavy readers. Remember that the
choices do tend to hint at what the appropriate answer is. We have to be careful that we don‟t
hint at what the appropriate number of books read for leisure is or otherwise respondents will
bias their responses accordingly.
Slide 11
Another good rule for writing good questions is to avoid order bias. At first blush, this might not
seem to be a problem. What difference would it make in an examination if the right answer were
A, B, C, or D. One would hope that a well prepared student would be equally likely to provide
the correct answer, regardless of where it was in the sequence of possible answers.
Unfortunately, that is not the case for survey research. Here‟s an example that illustrates the
problem. In this case people are asked to rate three analgesics and use some constant sum
scale for the points to sum to 100 and the greater the preference for the analgesic, the more
points it receives. From this example, you can see that this is mean point totals for groupings of
ten people, as there were 60 respondents in total. We can see the average points are vastly
different for Tylenol, if it‟s the first choice or if it‟s the last choice. When it‟s the first choice, it only
receives 10 and 15 points on average. When it‟s the last choice it receives 30 and 35 points. In
order to overcome this bias one would have to create multiple versions of the questionnaire, in
which the options are rotated or balanced. This is far easier to do if the questionnaire is being
administered by a computer than if it‟s being administered via mail. Nonetheless, it‟s important
to try to avoid order bias. I can assure you that none of my consulting clients has ever been
concerned about this. I suspect the detriment of the kinds of decisions or at least the quality of
decisions they could make on the data I provided them.
Slide 12
For those of you with an interest in the causes of order bias, this slide reviews several reasons.
Slide 13 (No Audio)
Slide 14
I have just a few more general guidelines to help you improve the quality of the questions that
you write for your questionnaires. One thing you might think about is avoiding unequal
comparisons. Looking at the problematic version of this question, #19; which one of the
following do you feel is responsible for recent outbreaks of violence in American schools?
Irresponsible parents are a different kind of entity that school policies or television programs and
irresponsible parents imply something negative. Policies and television programs are neutral.
One way to revise that question so that the comparisons are equal is to, instead of asking about
irresponsible parents; ask the way children are raised by parents. That is neither a positive nor a
negative, but it could be an influence regarding violence in schools. Another possible revision;
which one of the following do you feel is most responsible? The choices are; parents, schools,
and television. Again, those choices are neutral. Another alternative; to what extent do you feel
that the way children are raised by parents is responsible for recent outbreaks of violence in
American schools; completely responsible, all the way to not responsible. All three of these
alternatives is better than the original in which policies and television programs are couched in a
neutral form, but parents are couched in a negative form.
Page | 4
Slide 15
Here‟s a very simple rule: Be certain to state both sides of the attitude scales and the question
stems. So if we have a question stem, rather than ask; to what extent do you agree with this
statement? You would ask; to what extent do you agree or disagree with the statement.
Alternatively, in the second revision; do you feel it is easier, the same, or more difficult for
people to find work in this community than it was about a year ago? Just be certain that when
you ask the question, you ask both sides.
Slide 16
Try to ask questions as complete sentences. Think of even a paper and pencil survey as a
surrogate for a personal interview. The person reading yourself administered questionnaire is
having a conversation with you, although they move one step from you because you are now
translated into that paper the person is reading. As you don‟t speak in incomplete sentences,
you would prefer your questionnaire questions not be composed in incomplete sentences either.
Instead of the three questions—number of years lived in Idaho, a blank and years, your city or
town, a blank, and your county, with a blank—you would ask these questions as follows: How
many years have you lived in Idaho? In what city or town do you live? In what Idaho County do
you live? By making the questionnaire more conversational you will encourage people to think
more carefully about their answers and treat it more as if it‟s a conversation between you, the
researcher, and them.
Slide 17
As I mentioned in the lecture regarding question format, you have decisions to make regarding
whether scales are balanced or unbalanced and you also have questions that may be either
forced choice or non forced choice. The reason for allowing non forced choice answers is that
people will not respond in the intermediate „neither agree nor disagree‟ (that neutral point), as a
replacement for having no opinion. In this example, we can see that if there is a „no opinion‟
option or if we have a non forced choice question, we would like that choice to be the last one.
In the problem version, you can see the „undecided‟ is the midpoint. Does undecided mean I
have no opinion? Or does undecided mean I‟ve thought about the issue but I neither agree nor
disagree? We would rather not have it ambiguous between being undecided and having
considered the issue but still not having finalized an opinion.
Slide 18
Assuming that you are interested in an accurate answer to a single question or our respondents‟
unwillingness to provide accurate answers to that one question, here‟s an alternative; you could
ask respondents to randomly select which of two questions, one an innocuous question and one
a sensitive question, and you will not be aware of which of the two is chosen. Some sort of
random mechanism, like flipping a coin or picking a white versus black marble out of a jar,
would be used and then the respondent will provide an answer and you will not know to which
question that respondent is answering. Perhaps the numerical example will explain this better; is
the last digit of your SS# odd, yes or no. Then some sensitive question might have to do with
consumption of controlled substances, sexual behaviors, or any kinds of questions that people
may be reluctant to answer honestly. Let‟s assume we were to survey 1000 people. 300 people
answered yes and 700 people answered no. We would know, from doing some simple math, the
question somebody is answering is at random. So 500 people are answering the sensitive
question and 500 people are answering the innocuous question, the odds are 50/50 that they
Page | 5
will answer yes to the innocuous question. There‟s a 50/50 chance they will answer yes to the
last four digits of their SS# is odd. You are going to get 250 out of 500 yes‟ from the first
innocuous question. If you had 300 total yes‟, that would mean that 50 of the 500 people who
answered the sensitive question, answered yes to it. This would be a way to query people about
a sensitive question, where the answer would be either yes or no.
Slide 19
As an alternative to the technology suggested in the previous slide, one could just merely soften
the impact of potentially objectionable questions. Here are two examples: What was this
person‟s total income from all sources in1999? People are very reluctant to provide income
data. A somewhat softer version of the income question is the revised version; which of the
following broad categories best describes this person‟s total income from all sources in 1999
and then you allow a person to select from one of those categories. People are much more
willing to answer questions of the second type, relative to the first type, when we ask them
about things like income. The second example of the problematic question revision has to do
with shoplifting. In the problem version, you are asking someone; have you ever shoplifted
something from a store? Many people will be reluctant to admit that they‟ve committed such a
crime. Instead of using the word shoplifting, which is somewhat of a jarring word, we might
revise the question; have you ever taken anything from a store without paying for it? It means
the same thing but somehow if we‟re not using the word shoplifting, the questions does not
seem to be asking about whether or not someone has committed a crime in the past. As a
result, they are much more likely to answer the second version much more honestly.
Slide 20 (No Audio)
Slide 21
The next three examples provide examples of the types of lousy questions that I hope you
avoid.
Slide 22
In most cases, these questions are lousy because they provide respondents with ambiguous
choices or the question itself was asked in an ambiguous way. In a few cases, however, the
questions are also leading.
Slide 23
This very brief questionnaire is so lousy it would probably take an hour to really explain all the
problems with it, but I‟ll do so in just a couple of minutes. The first question; how much money
do you make is an incredibly poor question. Are we talking about profit, gross, weekly, monthly,
or annually? We have no time frames, we have no financial frames, and there is no way I can
know what I‟m supposed to answer with that question. The second question; how many
hamburgers do you sell in a typical month? The odds that the stand would know that number
are slim to none. How many hamburgers does your nearest competitor sell in a typical month?
There is no way that I, as the owner of a hamburger stand, would know what my competitor
sells in a typical month. Even if I knew the answer to that question, why would you want to know
that from me? The fourth question; do you get a lot of complaints about your hamburgers being
greasy? What if I say yes, what does that mean? Are those complaints founded? What do you
mean by greasy? What do you mean by complaints? Is it an off handed comment? Did
Page | 6
somebody refuse to pay the bill? The question is just incredibly vague. Another question; do you
put any foreign substances on your hamburgers? If I did I would certainly not admit that. There
is a tremendous social desirability bias both to this question and to some extent to question four.
Do you and your wife have marital problems relating to your work? This is a personal question
that I could not understand it being on this survey. How is this related in any way to the research
problem and to your previous questions? I would resent this and immediately wonder why I‟m
being asked this question and what the true purpose of this survey is. Suddenly I‟m being asked
personal questions about my age and the categories are certainly unequal and that is the
problem. Category B is 30-55 and category C is 55-65. Unless I have a good reason for
believing that hamburger stand ownership is skewed so that there is a tendency for older people
to own such stands, it makes no sense to format the age question this way.
Page | 7