United Nations Economic Commission for Europe
Statistical Division
The problem of identifying
persons with disabilities – the
importance of questionnaire
design
Angela Me, Chief Social and
Demographic Statistics Section
Challenge
How to measure a wider experience of
disability through a limited number of
questions?
- UNECE Statistical Division Slide 2
Bishkek, 13-15 December 2006
Design an instrument to identify the
defined population with disability
The difficult part is:
To logically convert/translate objectives
into measurement instruments and to
link definitions with questions
- UNECE Statistical Division Slide 3
Bishkek, 13-15 December 2006
Design an instrument to identify the defined
population with disabilities - a Census
Example
Definition:
• Any restriction or lack (resulting from an
impairment) of ability to perform an activity in the
manner or within the range considered normal for a
human being
Question:
• Is there anybody in the household who is disabled?
- UNECE Statistical Division Slide 4
Bishkek, 13-15 December 2006
Definition-measurement
instrument
BFS limitations
Paralyzed …..
Deaf
Activity limitations
…..
- UNECE Statistical Division Slide 5
Bishkek, 13-15 December 2006
Design an instrument to identify the defined
population with disabilities - a Census
Example
Despite a definition based on activity
limitations, the questions identified only
persons with most severe impairments
2.5%
- UNECE Statistical Division Slide 6
Bishkek, 13-15 December 2006
From theory to practice
Estimates of prevalence of disability are
highly sensitive to the measures used
- UNECE Statistical Division Slide 7
Bishkek, 13-15 December 2006
Measurement Issues
Design of the study
• Method of data collection
• Question design (wording, place, length, …)
• Interview process:
o interviewer effect
o Respondent effect
Socio-cultural Determinants
- UNECE Statistical Division Slide 8
Bishkek, 13-15 December 2006
US Survey Example
The following questions and results were obtained in an American survey
% 'Yes'
Have you ever heard the word AFROHELIA? 8
(no such word!)
Have you ever heard of the famous writer, John Woodson?
16
(no such writer!)
Have you ever heard of the Midwestern Life Magazine?
(no such magazine!) 25
Do you recall that, as a good citizen you voted last December in the
special election for your state representative?
(no election!) 33
Have you ever heard of the Taft-Pepper Bill concerning veteran's housing
(no such bill!) 53
- UNECE Statistical Division Slide 9
Bishkek, 13-15 December 2006
Example: Australian Survey
Average number of sex partners reported
• By women who were watched as they filled in their
survey answers: 2.6;
• By women who knew they were completely
anonymous: 3.4;
• By women who thought they were attached to a lie
detector: 4.4
Sydney Morning Herald, August 31, 2003
- UNECE Statistical Division Slide 10
Bishkek, 13-15 December 2006
Wording
The most detailed disability survey, using
a carefully designed and relatively
complete set of questions covering a
wide range of topics, is limited when the
initial questions used to identify the
persons with disability is poorly designed
- UNECE Statistical Division Slide 11
Bishkek, 13-15 December 2006
Developing instruments to
identify persons with disabilities
Disability is a dynamic complex related to:
• Individual attributes
• Environment
• Time
Two persons with the same impairment may
have a different perception of disability
- UNECE Statistical Division Slide 12
Bishkek, 13-15 December 2006
Issues that we need to
consider
Particular attention is needed to measure
disability through an interview process
• People may be unwilling to talk about their
problems
• Difficulty in defining what is meant by
disability and its various aspects
• Stigma
- UNECE Statistical Division Slide 13
Bishkek, 13-15 December 2006
Issues that we need to
consider
In an interview process:
• Easier to measure activity limitations (day-
to-day activities) and participation
- UNECE Statistical Division Slide 14
Bishkek, 13-15 December 2006
Developing instruments to identify
the complexity of disability
Requirement:
• Multiple questions to set context, clarify
terminology, define multiple domains
Resource availability
• Short questions
- UNECE Statistical Division Slide 15
Bishkek, 13-15 December 2006
Developing instruments to identify
the complexity of disability
Long instruments/modules
• High number of questions, more opportunities to
capture the different dimensions, intensity
Short instrument/modules
• 1-5 questions to identify persons with disabilities
• Careful design of the question(s) to make sure that
all persons with disabilities that we want to identify
can indeed be properly identified
- UNECE Statistical Division Slide 16
Bishkek, 13-15 December 2006
What defines a good
instrument
The instrument measures the concept it
is supposed to measure (Validity-
Accuracy)
Repeated measurements of the same
instruments give the same results
(Reliability-Precision)
- UNECE Statistical Division Slide 17
Bishkek, 13-15 December 2006
Design an instrument to identify the target
population- A Census Example
Definition:
• Any restriction or lack (resulting from an
impairment) of ability to perform an activity in the
manner or within the range considered normal for a
human being
Question:
• Is there anybody in the household who is disabled?
- UNECE Statistical Division Slide 18
Bishkek, 13-15 December 2006
An example: U.K. Census 1991
Do you have any long-standing illness, health
problem or handicap which limits your daily
activities or the work you can do? Include
problems which are due to old age
- UNECE Statistical Division Slide 19
Bishkek, 13-15 December 2006
Conditions that effect the
output of a question
Wording
Context
Self/Proxy
Response categories
Mode of data collection
Method of data collection
Overall survey topics
Survey sponsor
- UNECE Statistical Division Slide 20
Bishkek, 13-15 December 2006
Wording: what does affect
comprehension?
Ambiguous syntax
Complicated syntax
Unfamiliar terms
Vague concepts
Assumptions about respondent’s
knowledge
- UNECE Statistical Division Slide 21
Bishkek, 13-15 December 2006
Wording
Language:
• Clear
• Unambiguous
• Simple
Terms such as long-term, disabilities, handicaps are
viewed as extremely negative and tend to
underreport disabilities (Langlois, 2001)
- UNECE Statistical Division Slide 22
Bishkek, 13-15 December 2006
2000 US Census
Because of a physical, Multiple health domains
mental, or emotional (explicit enumeration)
condition lasting 6 Duration
months or more, does “Difficulty”
this person have any
capacity
difficulty in doing any of
the following activities… participation
“Working”
d. Working at a job or
business?
- UNECE Statistical Division Slide 23
Bishkek, 13-15 December 2006
Question components for a
short question
Preamble
Health condition (as cause)
Duration (long/short term disability?)
ICF domain
Functioning
Activities
Participation
- UNECE Statistical Division Slide 24
Bishkek, 13-15 December 2006
Question components for a
short question
Keep it relevant and valid but
SIMPLE
If possible split the different components
of the measure into different questions
- UNECE Statistical Division Slide 25
Bishkek, 13-15 December 2006
Question components for a
short question
Introduction: make the respondent think about the
outputs of an health condition and set the duration
(conditions that last for 3 months, 6 months, 12
months, …)
Depending on the domain we want to identify:
Do you have difficulties walking?
Do you have difficulties concentrating?
Do you have difficulties participating in daily activities
…..
- UNECE Statistical Division Slide 26
Bishkek, 13-15 December 2006
Question components for a
short question
Questions based on activities and
participation rather than impairments
provide a broader view of disability and
they facilitate the identification of
persons with disabilities by providing a
more simple and natural language
- UNECE Statistical Division Slide 27
Bishkek, 13-15 December 2006
Response process: judgment
and response formulation
Evaluation of retrieved information
• Perception of accuracy
• Motivation
Evaluation of response options
Communication of response
- UNECE Statistical Division Slide 28
Bishkek, 13-15 December 2006
Response categories
Response categories set the context of
the question
Avoid to force the respondent to identify
him/herself with a socially-defined label
(stigma)
Scale Response instead of a dichotomy
• None/A little/A lot
• Yes, sometimes/Yes, often/No
- UNECE Statistical Division Slide 29
Bishkek, 13-15 December 2006
Response categories
It has been proven that scaled responses
improve the respondents’ ability to report
having disabilities (Statistics Canada, Austrian
Bureau of Statistics, Research in the USA)
If the respondent can not choose among
multiple dimensions, he/she is likely to
misreport his/her disability status
Disability is not a yes/no phenomenon but
rather a status that varies on a continuum
in terms of intensity and time
- UNECE Statistical Division Slide 30
Bishkek, 13-15 December 2006
Mode of data collection
Self reported or Proxy?
Avoid if possible proxy responses
The disability process relates to the
individual’s experience and can be
accurately described only by the
individual him/her self
- UNECE Statistical Division Slide 31
Bishkek, 13-15 December 2006
Type of question
Avoid household-based questions (Is
there anybody in the household who has
difficulties walking?)
Use person-based question (Do you
have difficulties walking?)
It has been proven that person-based
questions identify more persons with
functional limitations (USA, UN)
- UNECE Statistical Division Slide 32
Bishkek, 13-15 December 2006
Context
Place of the question in the
questionnaire
Example: disability and economic
characteristics questions
Introduction to the question
- UNECE Statistical Division Slide 33
Bishkek, 13-15 December 2006