Interviewer Evaluation Form

Document Sample
Interviewer Evaluation Form Powered By Docstoc
					Estimating Validity at the Level of the
 Interviewer:The Case for Individual
             Differences
           Robert L. Dipboye
           Kenneth E. Podratz
             Rice University
        The interview is the SAME
         as other selection tools

• It leads to a quantifiable decision
• It should be evaluated on the basis of
  – Validity
  – Reliability
  – Fairness
  – Utility
        The interview is UNIQUE
           as a selection tool

• The tool is the individual interviewer
  who gathers information and evaluates
  applicants.
• The validity of interview depends on the
  validity of the individual interviewers
  who evaluate applicants.
      Objectives of the present study

• Assess the extent of variations among
  individual interviewers in the validity of
  their judgments.
• Explore potential sources of these
  variations in interviewer validity.
     The typical interview validation

• Involves more than one interviewer who
  assesses more than one applicant.
• Validity is assessed by aggregating
  interviewer judgments of applicants and
  computing correlations with
  performance of the applicant on the
  criteria.
  The assumption underlying the typical
          interview validation

• Individual differences among
  interviewers are irrelevant.
• Interviewers are equivalent and
  substitutable as instruments of
  selection.
  Several researchers have questioned
           these assumptions
• Zedeck, Tziner & Middlestadt, Personnel
  Psychology, 1983.
• Dreher, Ash, & Hancock, Personnel
  Psychology, 1988.
• Kinicki, Lockwood, Hom, & Griffeth,JAP,
  1990.
• Gehrlein, Dipboye, & Shahani, 1993, Ed.
  Psych. & Measurement
• Dipboye, Gaugler, & Hayes, Business
  Psychology, 2001
  The implication of these studies…….

• The failure to consider the validity of
  individual interviewers may lead to
  inaccurate estimation of the interview’s
  validity.
• Specifically, aggregation without
  accounting for the variations, leads to
  an underestimation of the interview’s
  validity.
   Pulakos, Schmitt, Whitney, & Smith
     (1966) cast doubt on previous
              conclusions
• The BEST study on this issue so far…
• Concurrent validation of the evaluations by 62
  interviewers who each evaluated a mean of
  25 employees (11 - 48)
• Evaluations were conducted in three person
  panels and the final evaluation was the
  product of consensus decision making.
• The criterion was the performance rating by
  current supervisor of the employee
        Findings and conclusions

• Validities at the level of the individual
  interviewers ranged from a low of -.10 to
  a high of .65.
• Weighted average of the individual
  validities was .31 (p < .05).
• Pulakos et al concluded that “all the
  observed variance in the validities could
  be attributed to sampling error (p. 98).”
       Potential limitations of this study

•   Relatively small samples of applicants.
•   Concurrent validation design.
•   Lack of corrections for artifacts.
•   Use of one criterion measure.
•   Independence of the interviewer ratings
    can be doubted.
    Our study investigated variations among
               interviewers using...
•   ...larger samples of applicants.
•   ...a predictive validation design.
•   ...more than one criterion.
•   ...independent interviewer evaluations.
•   ...correction for restriction of range in
    interviewer judgments.
        We also explored potential
          sources of variation
• Proximal determinants of interviewer
  differences in validity can be distinguished
  from distal determinants.
• We only looked at proximal factors in the form
  of differences in information gathered and
  judgment policies.
             Research questions

• Are variations in interviewer validity
  attributable to sampling error and artifactual
  sources - mainly restriction in range?
• Is validity related to the extent to which
  interviewer judgments are influenced by the
  race and sex of the applicant?
• What dimensions of applicant qualifications
  do the more valid interviewers take into
  account?
                    Samples

• All applicants for one specific position during
  a three year period (n = 25,511) - the
  unrestricted sample.
• A random sample of 2,400 of those hired
  during this three year period (n = 2,400) - the
  restricted sample.
• A sample of 490 of those hired used in further
  exploration of interviewer policies.
            The selection process

• Applicants for the position were first screened
  on the basis of tests, a criminal investigation,
  and employer references.
• A 15 - 20 interview was conducted using a
  highly standardized procedure containing
  some elements of behavioral interviewing.
• Final selection decision was based on the
  interviewers assessment of the applicants
  from the interview and their application file.
         The interviewer evaluation

• Interviewer evaluation: ratings on three-point
  scales of nine dimensions with weights
  assigned to each dimension on the basis of
  relative importance.
• Dimensions were based on a job analysis.
• Interviewers were given a detailed guide for
  conducting the interviews and evaluating the
  applicants.
• Interviewers were trained in these
  procedures.
          The criterion measures

• Annual performance appraisal by
  supervisor at the end of the first year.
• The number of disciplinary actions on
  record against the employee during
  their first year of employment.
• Involuntary departure: resignation in lieu
  of firing, dismissal for cause, training
  school dismissals.
 The validity of interviewer evaluations at
            the aggregate level
• The interviewer evaluation was a statistically
  significant predictor of disciplinary record and
  the annual performance appraisal.
• The more positive the interviewer evaluation,
  the fewer disciplinary actions and the better
  the performance appraisal.
Corrected and Unco rrected Validities at the
             Aggregate Level

                     Annu al Discipl. Invo lunt.
Validity Coefficient Eval. Record Turn.
Weighted Avg.
Validity
(uncorrected)          0.089 -0.124       -0.037
Weighted Avg.
Validity (corrected)   0.107 -0.147       -0.043
Unw eighted Avg.
Validity
(uncorrected)          0.076 -0.124       -0.037
Unw eighted Avg.
Validity (corrected)    0.09 -0.147       -0.043
          Effects of standardizing
          within each interviewer
• Examined the aggregate validities before and
  after standardizing the interviewer evaluation
  within each interviewer.
• Contrary to Dreher and Hancock (1988),
  validity did not differ as a function of
  standardization.
• Probably reflects the high standardization of
  the procedure.
Regression of the criteria on applicant race and
  sex in addition to interviewer evaluations

• Frequency of disciplinary actions: positively related to
  interviewer evaluation and higher for Blacks than
  Whites and Hispanics.
• Annual performance appraisal: positively related to
  interviewer evaluation and higher for men than
  women and Hispanics than Whites and Blacks.
• Involuntary turnover: No significant relationships.
   Regression o f Disciplinary Record on
       Interviewer Evaluation, and
         Applicant Sex and Race.


   Predictor Beta t-test Sig.
   Int. Eval. -.118 -5.809 .000
    Gender .018 .860 .390
       Black .157 6.715 .000
   Hispanic .018 .783 .434

Note: R2 = .038, F(4, 2321) = 22.85,
p < .001.
Regression o f Invo luntary Departure on
     Interviewer Evaluation and
       Applicant Sex and Race.

        Predictor     Beta t-test    Sig.
        Int. Eval.   -.037 -1.794   .073
         Gender      -.038 -1.843   .065
            Black     .003 .136     .892
        Hispanic     -.037 -1.560   .119

    Note: R2 = .004, F(4, 2321) =
    2.53, p < .05.
Regression o f Performance Appraisal
   on In terviewer Evaluation, and
       Applicant Sex and Race.

   Predictor     Betas t-test    Sig.
   Int. Eval.     .074 2.617    .009
   Gender         .098 3.475    .001
   Black         -.056 -1.705   .088
   Hispanic       .078 2.394    .017

    Note: R2 = .027, F(4, 1233) =
    9.55, p < .001.
Regression of Discipli nary Record on Interview
Evaluation Dimensions, Applicant Sex & Race

               Predictor        Beta      t-test    Sig.
             Motivation         .016       .472    .637
         Perceptiveness         .032       .969    .333
        Commun. Skill s        -.066    -1.924     .055
        Learning Abilit y       .017       .505    .614
         Emot. Stabilit y      -.011     -.342     .733
         Interper. Skills      -.012     -.358     .721
              Judgment/         .028       .843    .400
          Responsibilit y      -.071    -2.175     .030
          Pers. Integrity      -.072    -2.190     .029
                 Gender         .012       .357    .721
                   Black        .166     4.421     .000
                Hispanic        .046     1.232     .218
      R2 = .028, F(12, 928) = 3.24, p < .001.
 Regression of Disciplinary Record on
Sex, Race, and Individual Dimensions of
      the Interviewer Evaluation
• The dimensions of the interviewer evaluation
  that were most predictive were responsibility
  and personal integrity.
• Communication skills was related to
  disciplinary record at p < .06.
  Individual validities of the interviewers

• Computed the validities separately for 16 of
  the 17 interviewers.
• Corrected for range restriction on the
  interviewer evaluation at the level of each
  interviewer.
• The estimated validities varied considerably
  across the 16 interviewers.
The Validity of Each Interviewer
 in Predicting Discipline Record

       Interv. Eval. Validi ty Coeff.
Int     n Mean S. D. Uncorr      Corr.
  1    43 72.74 6.00 -0.332 -0.357
  2   186 70.84 5.94 -0.141 -0.175
  4    49 70.04 5.87 -0.188 -0.219
  5   299 72.50 6.05 -0.101 -0.124
  6   465 73.23 5.60 -0.108 -0.127
  7   245 73.40 5.57 -0.045 -0.053
  8    94 73.12 6.16 0.010 0.010
  9   170 71.38 5.97 -0.100 -0.124
10    122 72.21 6.07 -0.200 -0.224
11    139 73.10 5.57 -0.074 -0.094
12      6 71.00 5.18 0.852 0.902
13    157 70.74 5.86 -0.272 -0.326
14     55 74.31 4.96 0.004 0.006
15    187 72.22 5.92 -0.176 -0.206
16     62 73.37 5.83 -0.294 -0.329
17     12 70.33 5.30 -0.122 -0.184
Meta-analysis of the interviewer validities

• 100% of the observed variance in interviewer
  validities could be attributed to sampling error
  for the performance appraisal and involuntary
  turnover.
• 42.8% of the observed variance could be
  attributed to sampling error in the prediction
  of disciplinary record
• Subsequent analyses focused on exploring
  potential sources of these variations
             Subgroup analysis

Data from interviewers having corrected
 validities at above the mean of the correct
 validities were assigned to the high validity
 group; the remainder to the low validity group.
 Regression of interviewer evaluation on
interviewer evaluation and applicant race
    and sex for the two validity groups
• Sex was not significantly related to
  interviewer evaluation in either group.
• Race was related to interviewer evaluation in
  the low group at p<.10.
• This regression was performed for each
  individual interviewer. High and low validity
  interviewers did not differ in the influence of
  race and sex on their ratings.
      Regression of Interviewer Evaluation on
                App. Sex and Race

Low Validit y Group

   Predictor Beta      t-test Sig.
     Gender .009        .349 .727
       Black -.051 -1.677 .094
   Hispanic -.053 -1.742 .082
 2
R = .003, F(3, 1463) = 1.33, p = .26.

High Validit y Group

   Predictor Beta      t-test Sig.
     Gender .000       -.001 .999
       Black -.055 -1.391 .165
   Hispanic -.055 -1.384 .167
 2
R = .003, F(3, 820) = 0.87, p = .45.
Regression of Interviewer Evaluation on App.
   Gender and Race for each Interviewer

Group Int. Sex Black Hispanic n R-sq
Low        5 -.049 -.024        -.103 299 .012
Validity 6 -.021 -.038           .009 465 .002
           7 .055 -.105         -.211 245 .036
           8 .152 -.211         -.168 94 .063
           9 .037 .089           .086 170 .009
         11 .013 -.222          -.053 139 .038
         14 .061 -.003           .128 55 .024
           1 -.432 -.414        -.238 43 .268
High       2 .021 .076          -.013 186 .007
Validity 4 -.230 -.121           .215 49 .120
         10 -.099 -.011          .034 122 .012
         13 .146 -.166          -.068 157 .045
         15 .032 -.046          -.003 187 .003
         16 .113 .022           -.311 62 .105
         17 .344 -.194          -.065 12 .120
The Betas that are shaded were significant at p < .10.
  Are race and sex related to interviewer
      evaluations after controlling for
            other information?
• Used a sample of 490 applicants.
• Coded for information that was cited by
  interviewers as the basis for their evaluations.
• Coded the quality of the descriptions by
  applicants of their previous jobs.
• Coded the skill level of the applicants’
  previous jobs.
  Regressed interviewer evaluations on

• …the number of items of information cited by
  the interviewer to back each of the
  evaluations.
• ….the quality of the descriptions applicants
  provided on their previous experience.
• …..the skill level of the applicant’s previous
  work experience.
• …..applicant race and sex.
Regression of Interviewer Evaluation on App. Race,
Sex, Qualit y of Job Description, Previous Job Skill
         Level, and Information Gathered
  on Nine Dimensions: High Validit y Subgroup

 Predictor                        Beta t-test    Sig.
 motivation                      -.194 -2.213   .029
 perceptiveness/alertness         .075 .730     .467
 communication skills            -.175 -1.896   .061
 learning ability                 .082 .779     .438
 interpersonal skills            -.158 -1.875   .063
 judgment/common sense            .057 .659     .511
 responsibility                  -.352 -3.953   .000
 integrity                       -.120 -1.372   .173
 BLACK                           -.013 -.118    .906
 HISPANIC                        -.107 -1.045   .298
 SEX                              .072 .821     .413
 quality of the job descriptions .128 1.361     .176
 skill level of previous work    -.101 -1.144   .255
R2 = .255, F(13, 110) = 2.90, p < .001.
Regression of Interviewer Evaluation on Applicant
Race, Gender, Quality of Job Descriptions, Previous
    Job Skill L evel, and Information Gathered
   on Nine Dimensions: Low Valid ity Subgroup

 Predictor                     Beta t-test Sig.
 motivation                   -.201 -4.023 .000
 perceptiveness                .033 .671 .503
 communication skills         -.199 -3.913 .000
 interpersonal skills         -.157 -3.216 .001
 judgment                      .043 .865 .388
 responsibility               -.303 -6.141 .000
 integrity                    -.391 -7.718 .000
 BLACK                        -.146 -2.608 .010
 HISPANIC                     -.136 -2.467 .014
 SEX                           .075 1.497 .136
 quality of job descriptions -.018 -.331 .741
 skill level of previous work .043 .805 .422
Note: R2 = .461, F(12, 234) = 16.70, p < .001.
    Factors associated with interviewer
        evaluations within the high
              validity group
• R square of .255
• Information gathered on motivation and
  responsibility (p < .05)
• Information gathered on communications and
  interpersonal skills to a lesser extent (p < .10)
• Applicant race and sex were not related to
  interviewer evaluations.
   Factors associated with interviewer
       evaluations within the low
             validity group
• R square of .46.
• Information gathered on motivation,
  communications, interpersonal skills,
  integrity, and responsibility (p < .05).
• Whether the applicant was Hispanic or not
  (being Hispanic was associated with lower
  evaluations).
• Whether the applicant was Black or not
  (being Black was associated with lower
  evaluations).
       A correlational analysis at the
          level of the interviewer
• Using the original large sample computed
   – Indicator of sex bias -- the correlation of
     each individual interviewer’s evaluation
     with applicant sex.
   – Indicator of race bias -- the correlation of
     each individual interviewer’s evaluation
     with applicant race (Caucasian vs. non-
     Caucasian).
• Correlation of corrected validity of each
  interviewer with each indicator.
• Correlation between corrected validity and
  the sex bias indicator: - .54 (p < .05, two-
  tailed, n = 16).
• Correlation between corrected validity and
  the race bias indicator: -.51 (p < .06, two-
  tailed, n = 16).
• The more biased the interviewer in favor of
  males over females, and in favor of Whites
  over non-whites, the lower the validity.
                   Limitations

•   The size of the validities were SMALL.
•   Probably reflects non-optimal procedures.
•   Samples were opportunistic.
•   Nonrandom assignment.
•   Small number of interviewers.
•   Crudity of the subgroup analyses.
•   The inability to code information for all
    interviewers in the sample.
               Conclusions
• The variations among interviewers in the
  validity of their judgments exceeds what one
  would expect on the basis of sampling error.
• Variations in interviewer validity were criterion
  specific.
• Interviewers who achieve higher levels of
  validity tend to be less biased in their
  evaluations against women and ethnic
  minorities.
           Conclusions (cont)
• Interviewers who achieve lower levels of validity in
  the prediction of disciplinary record tend to be
  influenced by the same factors as those achieving
  higher levels of validity but,
• …..their evaluations tend to be influenced by more
  factors, some of which are not related to the criterion
  once objective information is controlled (e.g., race).

				
DOCUMENT INFO
Description: Interviewer Evaluation Form document sample