Impact on Questionnaire Psychometrics Among Substance Abuse Treatment Clients
E. Michael Bohlig, Thomas M. Bohman, Kelly Alanis, Michelle Steinley-Bumgarner, Richard T. Spence
University of Texas at Austin, Addiction Research Institute
Introduction Results Key Findings
Self-administered client surveys are an important source of information for quality improvement efforts in substance Table 1. Number of items per scale with reading levels Table 2. Number of items per scale with significant Over half of the items on one third of the scales contain items that would be difficult for a client with a 5th
abuse treatment. However, clients’ literacy levels vary widely and self-report instruments and instrument items exceeding 5th and 9th grade differences in response between high and low grade reading level to understand (see Table 1).
themselves vary in their reading difficulty. In 2003, the US Department of Education sponsored the National No. of Items, Reading Level education groups.
Assessment of Adult Literacy (NAAL) which utilized a nationally representative sample (n=19,714) of all persons in the Above Grade Level
Clients with different levels of education tend to respond differently to more than one third of the items
No. of Items and these items are concentrated in 9 of the 17 CEST scales.
United States aged 16 and older. This study also included a representative sample (n=1,173) of incarcerated persons No. of with Kruskal-
Scale Grade 5 Grade 9
aged 16 and older. The results of the study found that as many as 43 percent of the U.S. population aged 16 and Items No. of Wallace
Scale Scale reliability is significantly different across education levels for 8 of the 17 CEST Scales.
older possessed, at best, basic literacy skills (Kirsch et al., 2002; Kutner et al., 2005). In a multi-site study of clients in Desire for Help 6 1 0 Items p-value < .05
substance abuse treatment programs across the United States, Johnson and colleagues estimated that 70 percent of Treatment Readiness 8 2 1 Desire for Help 6 5 Compared to the high education group, the low education group was more likely to choose Item Value =
the clients would have difficulty understanding materials written at or above a fourth grade reading level (M. E. External Pressure 6 2 1 Treatment Readiness 8 7 3 and less likely to choose endpoints on the scale when reading difficulty increased.
Johnson et al., 1996). Self Esteem 6 2 0 External Pressure 6 1
Depression 6 2 0 Self Esteem 6 1 Education level, serving as a proxy for reading ability, does have an impact on item response and
Anxiety 7 2 1 Depression 6 2 reliability of the CEST items and scales.
Decision Making 9 6 1 Anxiety 7 1
Self Efficacy 7 1 0 Decision Making 9 5
Research Question - Importance Hostility
A common oversight of document creation, including surveys and questionnaires, is the literacy level of the target
audience. Treatment Satisfaction 7 5 2 Treatment Needs 5 4
Counseling Rapport 13 11 7 Treatment Satisfaction 7 5 In a recently released report on the quality of health care for metal health and substance abuse treatment patients, the
Treatment Participation 12 8 3 Counseling Rapport 13 11 Institute of Medicine recommends actions to improve health care in these areas. One recommendation specifies that
Two possible threats to the validity of survey data Peer Support 5 4 0
The inability of respondents to understand what survey items are asking can affect survey data in two serious ways. Treatment Participation 12 8 treatment providers include “patients and their families in the design, administration, and delivery of treatment and
Social Support 9 5 0 Peer Support 5 4
First, the respondent may not answer the item at all. A second outcome would be the introduction of large amounts of recovery services” (Institute of Medicine (U.S.). Committee on Crossing the Quality Chasm: Adaptation to Mental
Table 1 shows that the majority of items in six of the 17 CEST scales exceeded a Social Support 9 3 Health and Addictive Disorders, 2006, p. 12). Client feedback, including responses to questionnaires about their
random noise into the data because the respondents selects randomly among the response options or simply selects fifth grade reading level (light blue highlighting). This means that a person with a
Table 2 presents a summary of the results of a nonparametric test of
the middle response. Data from such instruments therefore, could be misleading, resulting in inaccurate diagnosis or fifth grade education would be able to read and comprehend what the item was
differences in responses by the two education groups to the items on the
treatment such as the CEST, will be an important source of information as treatment providers address this
asking, but a person with a lower education level would have difficulty recommendation. The results from the National Adult Literacy Study (Kutner et al., 2005) and Johnson and colleagues
assessment of client needs. At the program level, use of noisy and unreliable data to drive decision-making could comprehending the item. Approximately half of the items on the Counseling
CEST using the NPAR1WAY procedure in SAS. The light yellow
result in the inadvertent misallocation of scarce resources. Therefore, it is imperative to understand the overall literacy Rapport scale were rated at or above a ninth grade reading level.
highlighting in Table 2 identifies the scales where more than half of the (1996) which found that large numbers of US adults and substance abuse treatment clients have poor literacy skills
items were found to have significantly different median responses.
level of the target population and design instruments to maximize the accuracy of survey data. become more pertinent in light of this call for increased client input into treatment decisions. Respondents must
understand the questions they are responding to or their responses, if they respond at all, may be unreliable as the
Table 3. Cronbach’s Alpha for each scale by education level Table 4. Score Statistics For GEE Analysis testing respondent may not answer the question the questionnaire designer intended.
This study looks at the impact of education level – as a proxy for reading level – and its impact on the Client Evaluation
Cronbach’s Alpha unique, joint effect
of Self and Treatment as well as the effect of reading ease on response characteristics by education level. We look
specifically at the following questions: Education Degrees of As this study’s results demonstrated, item reading difficulty is related to how people with differential education levels
No. of All Less than 10th Grade Source Freedom Chi-Square Pr > ChiSq respond to items on the CEST. Scale developers in the substance abuse area do need to pay attention to item wording
1. Is there a significant relationship between education level of the respondent and the reliability of Items Respondents 10th Grade or Higher to ensure that responses are not biased by educational levels. A mismatch between the literacy level of the
Desire for Help 6 .729 .626 .708 Item Value 4 10.64 0.0309
the scales on the CEST? respondents and the reading level of questionnaire items has negative implications not only for treatment planning at
Treatment Readiness 8 .774 .665 .780** Reading Ease 1 0.16 0.6878
2. Is there a significant relationship between education level and the item-level responses of the the individual client level, but also for resource allocation across the clinic.
External Pressure 6 .495 .495 .495
clients? Reading Ease* Item 4 14.88 0.0050
Self Esteem 6 .768 .650 .789*** value interaction
Depression 6 .789 .669 .795***
3. Is the variability in responses to each item related to item readability?
Anxiety 7 .823 .779 .838* Table 4 shows the overall results of the GEE analysis. As expected, the
interaction effect (Item Value by Reading Ease) was statistically significant
4. Are differences in responses to individual item values related to education level of clients? Decision Making 9 .790 .737 .813**
Self Efficacy 7 .646 .588 .676
(p < .01). This shows that the effect of Reading Ease on the conditional
probability of detecting a group difference between the high and low
Hostility 8 .797 .811 .786 education groups on the Z test of independent proportions depended on the
Risk Taking 7 .764 .680 .764* specific item value.
Social Conformity 8 .520 .498 .537 Writing items at a low reading level is one possible approach to improving respondents’ ability to understand
Treatment Needs 5 .800 .758 .811 questionnaire items and thus possibly improving the reliability of self-assessment instruments. However, research has
Table 5. Score Statistics For GEE Analysis testing
Method Treatment Satisfaction
unique, joint effect
demonstrated that listening comprehension tends to be more advanced than reading comprehension (D. J. Johnson,
1993) so another possible solution is to write all items at a medium reading level such as 6th grade and administer the
Treatment Participation 12 .878 .872 .878 Standard survey via computer where the respondents have the option to hear the item read via a sound file (e.g., .wav files).
Peer Support 5 .826 .752 .833** Parameter Estimate Error Pr > |Z| Odds Ratio This approach has the additional benefits that such a program would eliminate the need to have clinic staff enter the
Subjects. This study included 529 substance abuse treatment program clients from a large urban setting. Social Support 9 .817 .765 .838**
Approximately 20 percent reported completing less than 10 years of formal education. The majority of the Intercept -0.5395 0.3941 0.1710 .58 data, and it could automatically score the instrument and produce reports for the counselors for treatment purposes
* p < .10; ** p < .05; *** p < .01.
Item Value = 1 -0.2067 0.4617 0.6544 .81
and clinic administrators for management and planning purposes.
respondents with available demographic data (n=425) were White (69%) and 24 percent were African American.
Approximately 25 percent reported being of Hispanic origin. Table 3 shows that there are significant differences in the reliability of eight of the
Item Value = 2
CEST subscales. In all cases, the reliability was greater for the higher education -0.7093 0.5306 0.1813 .49
group than for the respondent group with lower educational attainment. This
The participants completed the Client Evaluation of Self and Treatment (CEST) questionnaire (Joe et al., 2002) which pattern, while not significant held for all other subscales except Hostility where the
Item Value = 3
has 130 items comprising 17 scales. reliability was higher for the lower education group than for the higher education
group, although this difference was not significant.
-0.3561 0.4743 0.4528 .70
Item Value = 4
Analyses. Responses to each item were analyzed for differences between the two groups (less than 10th grade vs. -1.8183 0.5952 0.0023 .16
10th grade or higher education) and were compared to see if there was a pattern of significant differences by the Figure 1. Predicted probabilities for Interaction Education level was used as a proxy for reading ability in this study. This has been shown to be a poor proxy variable
reading level of the items. Each of the items in the CEST was tested for reading level using Flesch Scores; both between Item Value and Reading Ease Item Value = 5 (Reference
. 1.00 inasmuch as several studies have pointed out that actual reading ability tends to range from two to six years below
reading ease and Flesch-Kincaid grade level scores were computed. The Flesch reading ease scores were reversed reported years of education completed (Christensen & Grace, 1999; Currier et al., 2001; O'Bryant et al., 2005; Parker,
for this analysis so higher ratings represented more difficult reading. Reliability (Cronbach’s Alpha) of the 17 subscales Reading Ease -0.0103 0.0078 0.1872 .90 2000).
was also analyzed by education level and the Cronbach’s α for each group on each scale was compared for significant Reading Ease * Item Value = 1 -0.0030 0.0095 0.7536 .97
differences using the Alfatest utility developed by Hox (Hox, 2005). Item-level responses of the sample were tested for
Reading Ease * Item Value = 2 0.0073 0.0091 0.4202 1.08
differences between education level using the PROC NPAR1WAY procedure in SAS to test the null hypothesis that
there are no differences between the respondents with low educational attainment and those with higher levels of Reading Ease * Item Value = 3 0.0195 0.0086 0.0233 1.22
education. Reading Ease * Item Value = 4 0.0380 0.0113 0.0008 1.46 References
Reading Ease * Item Value = 5
To examine the effect of item reading level on variation in responses to each item, the standard deviation of each item 0.0000 0.0000 . 1.00
(Reference) Christensen, R. C., & Grace, G. D. (1999). The prevalence of low literacy in an indigent psychiatric population. Psychiatric Services, 50(2), 262-263.
was correlated with the reading ease score for each item. Currier, G. W., Sitzman, R., & Trenton, A. (2001). Literacy in the psychiatric emergency service. Journal Of Nervous And Mental Disease, 189(1), 56-58.
Table 5 shows the parameter estimates and their transformation into odds Hox, J. (2005). Alfatest. Amsterdam, NL: TT Software.
ratios for all effects in the model. For the Item Value effect, statistically Institute of Medicine (U.S.). Committee on Crossing the Quality Chasm: Adaptation to Mental Health and Addictive Disorders. (2006). Improving the
To examine whether the two education groups differed in use of item response values across the 130 items, the significant differences between groups were less likely to be detected for quality of health care for mental and substance-use conditions. Washington, DC: National Academy Press.
percentage of each group’s responses falling into each of the five response categories was calculated and then an Item Value = 4 than Item Value = 5. For the Reading Ease by Item Value Joe, G. W., Broome, K. M., Rowan-Szal, G. A., & Simpson, D. D. (2002). Measuring patient attributes and engagement in treatment. Journal Of
interaction, higher Item values showed a stronger positive effect of Reading Substance Abuse Treatment, 22(4), 183-196.
independent z-test was used to determine if those percentages differed between groups. The p-value for each of these Ease indicating that greater item reading difficulty was more likely associated
Johnson, D. J. (1993). Relationships between oral and written language. School Psychology Review, 22(4), 595-609.
tests was transformed into a dichotomous outcome: 1 = p value < .05 and 0 if >= .05. Using the items as the unit of with statistically significant group differences in the percentage of responses
Johnson, M. E., Fisher, D. G., Davis, D. C., Cagle, H. H., Rhodes, F., Booth, R., et al. (1996). Assessing reading level of drug users for HIV and AIDS
for that value.
analysis, a generalized estimating equations (GEE) logistic regression model was used to test whether responses prevention purposes. Aids Education And Prevention, 8(4), 323-334.
differed based on the response level and reading ease of each item. The GEE model was utilized to take into account Kirsch, I. S., Jungeblut, A., Jenkins, L., & Kolstad, A. (2002). Adult literacy in America: A first look at the results of the national adult literacy survey. In U.
S. D. o. E. National Center for Educational Statistics (Ed.) (pp. 178): U.S. Government Printing Office, Superintendent of Documents,
that each item has five observations (one for each response value) and these transformed p-values may not be Washington, DC 20402 (Stock No. 065-000-00588-3).
independent of each other. The multiplicative interaction term between reading ease and item response value tested Kutner, M., Greenberg, E., & Baer, J. (2005). A first look at the literacy of America’s adults in the 21st century. In N. C. f. E.
whether the probability of group differences jointly depended on the response value and item reading level. Statistics (Ed.): Institute of Education Science, US Dept. of Education.
O'Bryant, S. E., Schrimsher, G. W., & O'Jile, J. R. (2005). Discrepancies between self-reported years of education
and estimated reading level: Potential implications for neuropsychologists. Applied Neuropsychology,
Another way of looking at the interaction results is to graph the predicted values of the Item Value by Reading Ease interaction. Figure 1 shows that for values 3 and 4, 12(1), 5-11.
We would like to thank our colleagues at the Texas Department of State Health Services, Mental Health and Substance Abuse Services Division increased reading difficulty is related to greater probabilities of detecting group differences while for values 1 and 5, greater difficulty is associated with detecting fewer Parker, R. (2000). Health literacy: A challenge for American patients and their health care providers.
for their collaborative support of this study, as well as the treatment providers that participated in this study. group differences. Further descriptive analysis indicated that the low education group was more likely to choose value 3 as reading difficulty increased and values 1 and 5 Health Promotion International, 15(4), 277-283.
as reading difficulty decreased, while respondents from the higher education group were more likely to pick response 4 as the reading difficulty of the items increased.
Presented at the College on Problems of Drug Dependence 68th Annual Meeting, Scottsdale, Arizona, June 20, 2006