Internet Mediated Psychological Assessment Problems With Use of by whitecheese


									   Internet Mediated Psychological Assessment: Problems With Use of Normative Data
                                                                                                                       Tom Buchanan
                                                                                                                    University of Westminster

                                                   Poster presented at the British Psychological Society Annual Conference, 13th March 2003, Bournemouth, UK.

  Can Traditional Norms Be Used With Internet Tests?                                                                                             Case 2: Self-Monitoring Scale (SMS-R; Snyder, 1987)
  •Use of Internet- based questionnaire assessment is increasing in many
  fields of psychology (e.g. research, occupational, clinical).
                                                                                                                                         •Reflects tendencies to attend
  •There is evidence that, while psychometric properties of online tests                                                                 to social cues and moderate
                                                                                                                                                                                                                                                                   Self-Monitoring status according to Snyder's (1987) Criteria

  cannot be taken for granted, they can be reliable and valid as well as                                                                 behaviour in response.                                                                                                         140
                                                                                                                                                                                                                                                                        120                                                                 High
  conferring some practical advantages.                                                                                                  •High Self-Monitors (upper                                                                                         Number      100
                                                                                                                                                                                                                                                                                         Self-Monitors                                  Self-Monitors

  •However, the ways in which they are used require careful thought, given                                                               quartile) pay considerable
                                                                                                                                                                                                                                                            of People
  differences reported in distributions of scores that people achieve in online                                                          attention to such cues, Low                                                                                                     40

  and offline assessments (especially on measures of negative affect, which is                                                           Self-Monitors (lower quartile)                                                                                                  20

  particularly relevant to clinical applications - Buchanan, in press).                                                                  behave more in accord with                                                                                                        0
                                                                                                                                                                                                                                                                                0    1     2   3   4   5    6    7   8    9 10 11 12 13 14 15 16 17 18

  •To test the hypothesis that traditional paper-and-pencil norms are                                                                    internal states or enduring traits                                                                                                                                     Self-Monitoring Score

                                                                                                                                         and attitudes.
  unsuitable* for use with online versions of the same tests, data from three                                                            •Snyder (1987) suggests score
  online research projects were compared with such normative data.                                                                       of 13 as cut-off point for HSMs                                                                                  Figure 2: High and Low Self-Monitors according to
                                                                                                                                                                                                                                                          Snyders’s cutting scores
  *In some cases (e.g. Bartram & Brown, 2003) norms may be interchangeable. This, however, needs to be empirically established rather
                                                                                                                                         and 7 as cut-off for LSMs
  than assumed.                                                                                                                          (based on traditional college                                                                                                   Self-Monitoring status according to actual quartiles
                                                                                                                                         student samples)                                                                                                               160

                                                                                                                                         •However, these cutting scores                                                                                                 140
                                                                                                                                                                                                                                                                        120                                                                 High

      Case 1: The Hospital Anxiety and Depression Scale                                                                                  do not accurately reflect                                                                                                      100
                                                                                                                                                                                                                                                                                         Self-Monitors                                  Self-Monitors

                                                                                                                                         quartiles of combined sample of                                                                                    of People
              (HADS; Zigmond & Snaith, 1983)                                                                                             1299 respondents (Buchanan &                                                                                                    60
                                                                                                                                         Smith, 1999; Buchanan, 2000)                                                                                                    20

                                                                                                                                         who completed SMS-R via the                                                                                                       0
  •102 Participants recruited via                                                                             Internet.
                                                                                                                                                                                                                                                                                0    1    2    3   4   5    6    7   8    9 10 11 12 13 14 15 16 17 18

                                                                                                                                                                                                                                                                                                                Self-Monitoring Score
  •Self-selected sample with an interest in personality, but there is no reason to                                                       •For that combined sample, the
  believe they differed from a “normal” population in terms of the prevalence of                                                         top quartile best approximated
  anxiety or depression related disorders.                                                                                               by scores of 12 and above                                                                                        Figure 3: High and Low Self-Monitors according to
  •Completed online version of Zigmond and Snaith’s (1983) Hospital Anxiety                                                              (20.8%) and lower quartile                                                                                       quartiles

  and Depression Scale (HADS).                                                                                                          Conclusion: Application of
                                                                                                                                         approximated by 6 and below
                                                                                                                                                                    Snyder’s classification criteria
  •Proportion of respondents scoring in probable / possible clinical disorder
  ranges higher than sample of cancer patients reported by Moorey et al                                                                 to this Internet dataset would have led to misclassification
  (1991). [Table 1]                                                                                                                     of 234 people (18% of the sample).
  •Differences in mean scores consistent with medium effect sizes for both
  anxiety (d=.50), and depression (d=.40). [Fig. 1]

   Scale          Score           Meaning               % in current sample % in Moorey et al.’s                                                Case 3: The Everyday Memory Questionnaire (EMQ;
                                                                              cancer sample
                              “possible clinical
                                                                                                                                                     Sunderland, Harris & Baddeley, 1984).
                                 disorder”                                                                                                      •28-item instrument that measures self-reported everyday memory problems
  A                                                       32.4%
                                                                                                                                                •Cornish (2000) administered EMQ to 277 students, using traditional paper-
                   11+            “probable
                              clinical disorder”
                                                                         }       46.1%        }         27%                                     and-pencil techniques, and reported mean score of 90.2 (SD=25.8).
                                                                                                                                                •Rodgers et al (2003) administered EMQ, along with other instruments, in a
                   8-10       “possible clinical                                                                                                web-based study of the cognitive effects of recreational drug use. Two
                                 disorder”                                                                                                      hundred and forty two of their 763 participants claimed never to have taken
  D                                                       14.7%
                   11+            “probable
                              clinical disorder”
                                                                         }       19.6%        }            8.7%
                                                                                                                                                the recreational drugs Cannabis and Ecstasy. For these 242 drug-free
                                                                                                                                                participants - many of whom were students - the mean score on the EMQ
                                                                                                                                                was 70.4 (SD=26.2).
  Table 1: Percentage of sample with possible / probably clinical disorder.
                                                                                                                                                •This mean score is substantially lower than that reported by Cornish. The
  8                                                                                                                                             difference (19.8 scale points) corresponds to a large effect size (d = .76).
                                                                                      Current Online Sample
                                                                                                                                        Conclusion: Use of norms based on Cornish’s offline
                                                                                      Moorey et al (people diagnosed with
                                                                                                                                        sample would be inappropriate in trying to interpret
  5                                                                                   cancer)
                                                                                                                                        the online scores of Rodgers et al’s sample.


  1                                                                                                                                     •Established norms and cutting scores based on
                    Anxiety                            Depression
                                                                                                                                        offline samples may not be suitable for use with
                                                                                                                                        online versions of tests.
                                                                                                                                        •Problems especially important when online tests used in
 Figure 1: Mean HADS anxiety and depression scores.                                                                                     “real life” contexts such as behavioural telehealth, career
                                                                                                                                        counselling or personnel selection.
                                                                                                                                        •Reasons for differences are an important priority for
  •Seems very unlikely the current participants really more anxious and                                                                 future research.
  depressed than people who had received a diagnosis of cancer.
  •Seems very unlikely almost 14% of respondents suffering from clinically
  significant level of anxiety.                                                                                                                                                                            References                                                                                                        Contact Details
                                                                                                                                        •Bartram, D., & Brown, A. (2003, January). Online testing: Mode of administration and the stability of OPQ 32i scores. Paper presented at the British
                                                                                                                                        Psychological Society Occupational Psychology Conference, Bournemouth, UK, 8-10 January, 2003.
                                                                                                                                        •Buchanan, T. (2000). Internet Research: Self-monitoring and judgments of attractiveness. Behavior Research Methods, Instruments, & Computers, 32, 521-527.                      Tom Buchanan
                                                                                                                                        •Buchanan, T. (in press). Internet based questionnaire assessment: Appropriate use in clinical contexts. Cognitive Behaviour Therapy.
                                                                                                                                        •Buchanan, T., & Smith, J. L. (1999). Using the Internet for psychological research: Personality testing on the World-Wide Web. British Journal of Psychology, 90,               Department of Psychology
Conclusion: Use of published offline normative data (cut-                                                                               125–144.
                                                                                                                                        •Cornish, I. M. (2000). Factor Structure of the Everyday Memory Questionnaire. British Journal of Psychology, 91, 427-438.
                                                                                                                                        •Moorey, S., Greer, S., Watson, M., Gorman, C., Rowden, L., Tunmore, R., Robertson, B., & Bliss, J. (1991). The factor structure and factor stability of the Hospital
                                                                                                                                                                                                                                                                                                                         University of Westminster
                                                                                                                                                                                                                                                                                                                         309 Regent Street
                                                                                                                                        Anxiety and Depression Scale in patients with cancer. British Journal of Psychiatry, 158, 255-259.

off points) likely to lead to faulty inferences being made                                                                              •Rodgers, J., Buchanan, T., Scholey, A. B., Heffernan, T. M., Ling, J., & Parrott, A. C. (2003). Patterns of drug use and the influence of gender on self-reports of
                                                                                                                                        memory ability in ecstasy users: a web-based study. Manuscript submitted for publication.
                                                                                                                                        •Snyder, M. (1987). Public Appearances / Private Realities. New York: W. H. Freeman.
                                                                                                                                                                                                                                                                                                                         London W1B 2UW
                                                                                                                                                                                                                                                                                                                         buchant @
                                                                                                                                        •Sunderland, A., Harris, J. E., & Baddeley, A. D. (1983). Do laboratory tests predict everyday memory? A neuropsychological study. Journal of Verbal Learning and      
about current online sample.                                                                                                            Verbal Behaviour, 22, 341-357.
                                                                                                                                        •Zigmond, A. S., & Snaith, R. P. (1983). The Hospital Anxiety and Depression Scale. Acta Psychiatrica Scandinavica, 67, 361-370.

To top