Internet Mediated Psychological Assessment: Problems With Use of Normative Data
University of Westminster
Poster presented at the British Psychological Society Annual Conference, 13th March 2003, Bournemouth, UK.
Can Traditional Norms Be Used With Internet Tests? Case 2: Self-Monitoring Scale (SMS-R; Snyder, 1987)
•Use of Internet- based questionnaire assessment is increasing in many
fields of psychology (e.g. research, occupational, clinical).
•Reflects tendencies to attend
•There is evidence that, while psychometric properties of online tests to social cues and moderate
Self-Monitoring status according to Snyder's (1987) Criteria
cannot be taken for granted, they can be reliable and valid as well as behaviour in response. 140
conferring some practical advantages. •High Self-Monitors (upper Number 100
•However, the ways in which they are used require careful thought, given quartile) pay considerable
differences reported in distributions of scores that people achieve in online attention to such cues, Low 40
and offline assessments (especially on measures of negative affect, which is Self-Monitors (lower quartile) 20
particularly relevant to clinical applications - Buchanan, in press). behave more in accord with 0
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
•To test the hypothesis that traditional paper-and-pencil norms are internal states or enduring traits Self-Monitoring Score
unsuitable* for use with online versions of the same tests, data from three •Snyder (1987) suggests score
online research projects were compared with such normative data. of 13 as cut-off point for HSMs Figure 2: High and Low Self-Monitors according to
Snyders’s cutting scores
*In some cases (e.g. Bartram & Brown, 2003) norms may be interchangeable. This, however, needs to be empirically established rather
and 7 as cut-off for LSMs
than assumed. (based on traditional college Self-Monitoring status according to actual quartiles
student samples) 160
•However, these cutting scores 140
Case 1: The Hospital Anxiety and Depression Scale do not accurately reflect 100
quartiles of combined sample of of People
(HADS; Zigmond & Snaith, 1983) 1299 respondents (Buchanan & 60
Smith, 1999; Buchanan, 2000) 20
who completed SMS-R via the 0
•102 Participants recruited via www.personalitytest.org.uk Internet.
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
•Self-selected sample with an interest in personality, but there is no reason to •For that combined sample, the
believe they differed from a “normal” population in terms of the prevalence of top quartile best approximated
anxiety or depression related disorders. by scores of 12 and above Figure 3: High and Low Self-Monitors according to
•Completed online version of Zigmond and Snaith’s (1983) Hospital Anxiety (20.8%) and lower quartile quartiles
and Depression Scale (HADS). Conclusion: Application of
approximated by 6 and below
Snyder’s classification criteria
•Proportion of respondents scoring in probable / possible clinical disorder
ranges higher than sample of cancer patients reported by Moorey et al to this Internet dataset would have led to misclassification
(1991). [Table 1] of 234 people (18% of the sample).
•Differences in mean scores consistent with medium effect sizes for both
anxiety (d=.50), and depression (d=.40). [Fig. 1]
Scale Score Meaning % in current sample % in Moorey et al.’s Case 3: The Everyday Memory Questionnaire (EMQ;
Sunderland, Harris & Baddeley, 1984).
disorder” •28-item instrument that measures self-reported everyday memory problems
•Cornish (2000) administered EMQ to 277 students, using traditional paper-
} 46.1% } 27% and-pencil techniques, and reported mean score of 90.2 (SD=25.8).
•Rodgers et al (2003) administered EMQ, along with other instruments, in a
8-10 “possible clinical web-based study of the cognitive effects of recreational drug use. Two
disorder” hundred and forty two of their 763 participants claimed never to have taken
} 19.6% } 8.7%
the recreational drugs Cannabis and Ecstasy. For these 242 drug-free
participants - many of whom were students - the mean score on the EMQ
was 70.4 (SD=26.2).
Table 1: Percentage of sample with possible / probably clinical disorder.
•This mean score is substantially lower than that reported by Cornish. The
8 difference (19.8 scale points) corresponds to a large effect size (d = .76).
Current Online Sample
Conclusion: Use of norms based on Cornish’s offline
Moorey et al (people diagnosed with
sample would be inappropriate in trying to interpret
the online scores of Rodgers et al’s sample.
1 •Established norms and cutting scores based on
offline samples may not be suitable for use with
online versions of tests.
•Problems especially important when online tests used in
Figure 1: Mean HADS anxiety and depression scores. “real life” contexts such as behavioural telehealth, career
counselling or personnel selection.
•Reasons for differences are an important priority for
•Seems very unlikely the current participants really more anxious and future research.
depressed than people who had received a diagnosis of cancer.
•Seems very unlikely almost 14% of respondents suffering from clinically
significant level of anxiety. References Contact Details
•Bartram, D., & Brown, A. (2003, January). Online testing: Mode of administration and the stability of OPQ 32i scores. Paper presented at the British
Psychological Society Occupational Psychology Conference, Bournemouth, UK, 8-10 January, 2003.
•Buchanan, T. (2000). Internet Research: Self-monitoring and judgments of attractiveness. Behavior Research Methods, Instruments, & Computers, 32, 521-527. Tom Buchanan
•Buchanan, T. (in press). Internet based questionnaire assessment: Appropriate use in clinical contexts. Cognitive Behaviour Therapy.
•Buchanan, T., & Smith, J. L. (1999). Using the Internet for psychological research: Personality testing on the World-Wide Web. British Journal of Psychology, 90, Department of Psychology
Conclusion: Use of published offline normative data (cut- 125–144.
•Cornish, I. M. (2000). Factor Structure of the Everyday Memory Questionnaire. British Journal of Psychology, 91, 427-438.
•Moorey, S., Greer, S., Watson, M., Gorman, C., Rowden, L., Tunmore, R., Robertson, B., & Bliss, J. (1991). The factor structure and factor stability of the Hospital
University of Westminster
309 Regent Street
Anxiety and Depression Scale in patients with cancer. British Journal of Psychiatry, 158, 255-259.
off points) likely to lead to faulty inferences being made •Rodgers, J., Buchanan, T., Scholey, A. B., Heffernan, T. M., Ling, J., & Parrott, A. C. (2003). Patterns of drug use and the influence of gender on self-reports of
memory ability in ecstasy users: a web-based study. Manuscript submitted for publication.
•Snyder, M. (1987). Public Appearances / Private Realities. New York: W. H. Freeman.
London W1B 2UW
buchant @ wmin.ac.uk
•Sunderland, A., Harris, J. E., & Baddeley, A. D. (1983). Do laboratory tests predict everyday memory? A neuropsychological study. Journal of Verbal Learning and http://www.wmin.ac.uk/~buchant
about current online sample. Verbal Behaviour, 22, 341-357.
•Zigmond, A. S., & Snaith, R. P. (1983). The Hospital Anxiety and Depression Scale. Acta Psychiatrica Scandinavica, 67, 361-370.