Internet Mediated Psychological Assessment: Problems With Use of Normative Data Tom Buchanan University of Westminster Poster presented at the British Psychological Society Annual Conference, 13th March 2003, Bournemouth, UK. Can Traditional Norms Be Used With Internet Tests? Case 2: Self-Monitoring Scale (SMS-R; Snyder, 1987) •Use of Internet- based questionnaire assessment is increasing in many fields of psychology (e.g. research, occupational, clinical). •Reflects tendencies to attend •There is evidence that, while psychometric properties of online tests to social cues and moderate Self-Monitoring status according to Snyder's (1987) Criteria 160 cannot be taken for granted, they can be reliable and valid as well as behaviour in response. 140 Low 120 High conferring some practical advantages. •High Self-Monitors (upper Number 100 Self-Monitors Self-Monitors •However, the ways in which they are used require careful thought, given quartile) pay considerable of People 80 60 differences reported in distributions of scores that people achieve in online attention to such cues, Low 40 and offline assessments (especially on measures of negative affect, which is Self-Monitors (lower quartile) 20 particularly relevant to clinical applications - Buchanan, in press). behave more in accord with 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 •To test the hypothesis that traditional paper-and-pencil norms are internal states or enduring traits Self-Monitoring Score and attitudes. unsuitable* for use with online versions of the same tests, data from three •Snyder (1987) suggests score online research projects were compared with such normative data. of 13 as cut-off point for HSMs Figure 2: High and Low Self-Monitors according to Snyders’s cutting scores *In some cases (e.g. Bartram & Brown, 2003) norms may be interchangeable. This, however, needs to be empirically established rather and 7 as cut-off for LSMs than assumed. (based on traditional college Self-Monitoring status according to actual quartiles student samples) 160 •However, these cutting scores 140 Low 120 High Case 1: The Hospital Anxiety and Depression Scale do not accurately reflect 100 Self-Monitors Self-Monitors Number quartiles of combined sample of of People 80 (HADS; Zigmond & Snaith, 1983) 1299 respondents (Buchanan & 60 40 Smith, 1999; Buchanan, 2000) 20 who completed SMS-R via the 0 •102 Participants recruited via www.personalitytest.org.uk Internet. 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Self-Monitoring Score •Self-selected sample with an interest in personality, but there is no reason to •For that combined sample, the believe they differed from a “normal” population in terms of the prevalence of top quartile best approximated anxiety or depression related disorders. by scores of 12 and above Figure 3: High and Low Self-Monitors according to •Completed online version of Zigmond and Snaith’s (1983) Hospital Anxiety (20.8%) and lower quartile quartiles and Depression Scale (HADS). Conclusion: Application of approximated by 6 and below (24.2%). Snyder’s classification criteria •Proportion of respondents scoring in probable / possible clinical disorder ranges higher than sample of cancer patients reported by Moorey et al to this Internet dataset would have led to misclassification (1991). [Table 1] of 234 people (18% of the sample). •Differences in mean scores consistent with medium effect sizes for both anxiety (d=.50), and depression (d=.40). [Fig. 1] Scale Score Meaning % in current sample % in Moorey et al.’s Case 3: The Everyday Memory Questionnaire (EMQ; cancer sample “possible clinical Sunderland, Harris & Baddeley, 1984). 8-10 disorder” •28-item instrument that measures self-reported everyday memory problems A 32.4% •Cornish (2000) administered EMQ to 277 students, using traditional paper- 11+ “probable clinical disorder” 13.7% } 46.1% } 27% and-pencil techniques, and reported mean score of 90.2 (SD=25.8). •Rodgers et al (2003) administered EMQ, along with other instruments, in a 8-10 “possible clinical web-based study of the cognitive effects of recreational drug use. Two disorder” hundred and forty two of their 763 participants claimed never to have taken D 14.7% 11+ “probable clinical disorder” 4.9% } 19.6% } 8.7% the recreational drugs Cannabis and Ecstasy. For these 242 drug-free participants - many of whom were students - the mean score on the EMQ was 70.4 (SD=26.2). Table 1: Percentage of sample with possible / probably clinical disorder. •This mean score is substantially lower than that reported by Cornish. The 8 difference (19.8 scale points) corresponds to a large effect size (d = .76). 7 Current Online Sample Conclusion: Use of norms based on Cornish’s offline 6 Moorey et al (people diagnosed with sample would be inappropriate in trying to interpret 5 cancer) the online scores of Rodgers et al’s sample. 4 3 Conclusions 2 1 •Established norms and cutting scores based on 0 Anxiety Depression offline samples may not be suitable for use with online versions of tests. •Problems especially important when online tests used in Figure 1: Mean HADS anxiety and depression scores. “real life” contexts such as behavioural telehealth, career counselling or personnel selection. •Reasons for differences are an important priority for •Seems very unlikely the current participants really more anxious and future research. depressed than people who had received a diagnosis of cancer. •Seems very unlikely almost 14% of respondents suffering from clinically significant level of anxiety. References Contact Details •Bartram, D., & Brown, A. (2003, January). Online testing: Mode of administration and the stability of OPQ 32i scores. Paper presented at the British Psychological Society Occupational Psychology Conference, Bournemouth, UK, 8-10 January, 2003. •Buchanan, T. (2000). Internet Research: Self-monitoring and judgments of attractiveness. Behavior Research Methods, Instruments, & Computers, 32, 521-527. Tom Buchanan •Buchanan, T. (in press). Internet based questionnaire assessment: Appropriate use in clinical contexts. Cognitive Behaviour Therapy. •Buchanan, T., & Smith, J. L. (1999). Using the Internet for psychological research: Personality testing on the World-Wide Web. British Journal of Psychology, 90, Department of Psychology Conclusion: Use of published offline normative data (cut- 125–144. •Cornish, I. M. (2000). Factor Structure of the Everyday Memory Questionnaire. British Journal of Psychology, 91, 427-438. •Moorey, S., Greer, S., Watson, M., Gorman, C., Rowden, L., Tunmore, R., Robertson, B., & Bliss, J. (1991). The factor structure and factor stability of the Hospital University of Westminster 309 Regent Street Anxiety and Depression Scale in patients with cancer. British Journal of Psychiatry, 158, 255-259. off points) likely to lead to faulty inferences being made •Rodgers, J., Buchanan, T., Scholey, A. B., Heffernan, T. M., Ling, J., & Parrott, A. C. (2003). Patterns of drug use and the influence of gender on self-reports of memory ability in ecstasy users: a web-based study. Manuscript submitted for publication. •Snyder, M. (1987). Public Appearances / Private Realities. New York: W. H. Freeman. London W1B 2UW buchant @ wmin.ac.uk •Sunderland, A., Harris, J. E., & Baddeley, A. D. (1983). Do laboratory tests predict everyday memory? A neuropsychological study. Journal of Verbal Learning and http://www.wmin.ac.uk/~buchant about current online sample. Verbal Behaviour, 22, 341-357. •Zigmond, A. S., & Snaith, R. P. (1983). The Hospital Anxiety and Depression Scale. Acta Psychiatrica Scandinavica, 67, 361-370.
Pages to are hidden for
"Internet Mediated Psychological Assessment Problems With Use of"Please download to view full document