The Equivalence of Internet and Paper-and-pencil Personality Assessments
Chuah Siang Chee
Fritz Drasgow
Brent Roberts
Society for Industrial and Organizational Psychology, Toronto, April 12-April 14, 2002
Internet Assessment
• Access from more locations and to more people • Convenience for the user and test administrator
• User access from Internet enabled computers
• Electronic data entry
• Relative speed and low cost for large test administrations
Database Driven Website
Are Internet Assessments Equivalent to Other Approaches?
• Measurement Equivalence
– Mean-Level – Differential Item Functioning (DIF)
• Examinees of equal ability but from separate populations differ in their probability of giving a particular answer.
Previous research on meanlevel differences across modality
• Mean-level
– Social Desirability Distortions
Meta-analysis of non-cognitive tests across mediums Richman, Kiesler, Weisband and Drasgow (1999)
– Anonymity
Lower Social Anxiety and Social Desirability with Internet Johnson (1999)
– Pasveer & Ellard (1998)
Self-Trust – No differences
– Davis (1999)
Ruminative Responses – No differences
Problems with previous research on mean levels
• Comparison groups were not equivalent.
– Different recruiting techniques for groups – Different samples sizes
• Unknown internet structure to assessment
Previous research on DIF
• No published research with DIF analysis
The Present Study
• Examined mean-level differences on the Big Five across testing medium
– Equivalent groups across modalities
• Paper-and-Pencil
Traditional administration under experimenter supervision.
• Computer Laboratory
Group administration of WWW questionnaires in laboratory under experimenter supervision.
• Internet
E-mail notification of WWW questionnaire. Administered over WWW for 1 week.
The Present Study …
• Random Assignment to Conditions
No Self-selection effects
• Identification of Subjects
Unique Password – security/integrity data Academic – course credit Industry – aberrant responses, linking information
• Multiple-access for Internet Questionnaire
No Duplicate Responses
DIF Analyses
• SIBTEST (Stout & Roussos, 1995) SIB Statistic β & Mantel-Haenszel • Iterlink DIF (Steve Stark) Lord’s Chi-Square
Big 5 Personality Scale
• Trait Descriptive Adjective (Goldberg) • Neuroticism, Extroversion, Openness, Agreeableness, Conscientiousness • 5 point Likert Scale (Strongly Disagree – Strongly Agree) “How well do the adjectives describe you?”
Big 5 Personality Items
Neuroticism
1 2 3 4 5 6 7 8 9 10 11 12 13 Unenvious Unemotional Relaxed Imperturbable Unexcitable Anxious* Moody* Temperamental* Envious* Irritable* Jealous* Emotional* Fretful*
Extroversion
Extraverted Talkative Assertive Verbal Energetic Introverted* Shy* Quite* Reserved* Untalkative*
Openness
Intellectual Creative Complex Imaginative Bright Unintellectual* Unintelligent* Unimaginative* Uncreative Simple*
Agreeableness
Kind Cooperative Sympathetic Warm Trustful Cold* Unkind* Unsympathetic* Distrustful* Harsh*
Conscientiousness
Organized Systematic Through Practical Neat Disorganized* Careless* Unsystematic* Undependable* Inefficient*
* Reverse Scored Items
Sample
• Undergraduates Psychology Students participating in the study for course credit
Condition N Total
Paper-and-pencil 266 Computer Lab Internet 222 240 728
SIB Statistic p<.05
Paper Vs Internet
Neuroticism Extroversion Openness
Unenvious Unexcitable
p
Paper Vs Computer
p
Computer Vs Internet
p
.042
.047 Imperturbable .008 Complex
.000 Imperturbable
.005
Sympathetic
.042
Conscientio- Inefficient usness
.018 Systematic
Unsystematic
.041
.006
Mantel-Haenszel p<.05
Paper Vs Internet
Neuroticism Extroversion Openness
Unexcitable
p
Paper Vs Computer
Unexcitable
p
.000 .039
Computer Vs Internet
p
.037 Imperturbable
Conscientio- Inefficient usness
.019 Systematic
Unsystematic
.041
.006
Lord’s Chi-Square p<.05
Paper Vs Internet
Neuroticism Extroversion Openness Conscientiousness
Systematic Neat Disorganized Unsystematic .001 .025 .011 .036
p
Paper Vs Computer
p
Computer Vs Internet
p
Expected # of “False Alarms”
95% Confidence Interval 5% Comparisons ± (α X SD) α X SD = 1.96 X (30 X .05 X .95)½ = 2. 34 α X SD = 1.96 X (123 X .05 X .95)½ = 4.74 Per Dimension 2.05 ± 2.735 = (0, 4.39) All 4 Dimensions 6.15 ± 4.74 = (1.41, 10.89)
Summary Analysis p<.05
SIBStatistic MantelHaenszel Lord’s Chi-Square
Neuroticism
Extroversion
4
1
3
Agreeableness
Conscientious -ness
1 3 9 3 6 4 4
Total
SIB Statistic Bonferroni Corr.
Paper Vs Internet
Neuroticism Extroversion Openness p
Paper Vs Computer
Imperturbable
p
.000
Computer Vs Internet
P
Complex
.005
\
Conscientiousness
Mantel-Haenszel Bonferroni
Paper Vs Internet
Neuroticism Extroversion Openness p
Paper Vs Computer
Imperturbable
p
.000
Computer Vs Internet
p
Conscientiousness
Lord’s Chi-Square Bonferroni
Paper Vs Internet
Neuroticism Extroversion Openness p
Paper Vs Computer
p
Computer Vs Internet
p
Conscientiousness
Systematic
.001
Mean-Level Comparisons
• DIF analyses established that Personality Items are functioning equivalently across media
• Is there a shift in Trait Levels as a consequence of medium?
– mean-levels comparisons using ability/theta estimated by Bilog
Mean Trait Levels Effect Size
1
Mean Theta/Ability Scores
0.5
p<.033
0
-0.5
-1
Emotional Stability Extroversion Agreeableness Conscientiousness
Paper-and-pencil
Computer Laboratory
Internet
Paper-and-Pencil Vs Both Computer Administrations Emotional Stability d’ = .19
Conclusion
• Personality Tests are Equivalent • Features of Internet Questionnaire Important