Your Federal Quarterly Tax Payments are due April 15th Get Help Now >>

Current Methods, Problems, and Solutions by iht11609

VIEWS: 0 PAGES: 38

									                               Current Methods,
                            Problems, and Solutions
                                            Paul Barrett
                                             Paul Barrett
                                 email: p.barrett@liv.ac.uk
                                  email: p.barrett@liv.ac.uk
                       http://www.liv.ac.uk/~pbarrett/paulhome.htm
                        http://www.liv.ac.uk/~pbarrett/paulhome.htm
                     Affiliations: Chief Scientist at The State Hospital, Carstairs
                      Affiliations: Chief Scientist at The State Hospital, Carstairs
                    Senior Research Fellow at the Dept. of Clinical Psychology
                     Senior Research Fellow at the Dept. of Clinical Psychology
                                           Univ. Of Liverpool
                                            Univ. Of Liverpool
                                                                       March 30th , ,2001
                                                                        March 30th 2001
Integrity Testing                                                            BCS-ISSG Oxford, March 2001
     The Collins English Dictionary, 3rd edition (1991)
     defines an honest person as one “not given to
     lying, cheating, stealing etc., trustworthy; not
     false or misleading, genuine; and characterised
     by sincerity and candour”. Integrity is defined as
     “adherence to moral principles, honesty; the
     quality of being unimpaired”.




Integrity Testing                            BCS-ISSG Oxford, March 2001
                                                                             Integrity
                                                                            Assessment




                               Psycho-
                                                                Interview                         Biodata                     Psychometrics
                              physiology



            CQT           GKT          P300       Arousal                  Covert and Overt                                 Covert            Overt




                                                                                                                  Personality Trait
   Control Question Test.                                                                                             measures
                                                                                                                                        Specialist Integrity
        Measures the                                                                                               re-weighted to
                                                                                                                                       tests that are highly
   difference in response     Guilty Knowledge Test.                                                             maximise prediction
                                                                                                                                        face-valid (in that
   to a "critical" question   Measures the response                                                               of some criterion.
                                                                                                                                          what is being
     vs a more "neutral"            to information       The P300 is a positive
                                                                                                                                       assessed is obvious
           question           embedded in a question    voltage evoked potential
                                                                                                                                         to the test-taker
                               - the Peak of Tension     - related to information
                                                                                    Psychopaths and some
                                (POT) method is an       processing of features
                                                                                     individuals associated
                               ordered series of such   of a stimulus - at around
                                                                                          with delinquent
                              information-embedded           300ms after the
                                                                                       behaviours show a
                                       questions            stimulus has been
                                                                                         reduced level of
                                                                 presented
                                                                                       arousal/reactivity to
                                                                                    "startle-inducing" stimuli




Integrity Testing                                                                                                          BCS-ISSG Oxford, March 2001
               8

               6                                         P2 or P200
                        Brain-Stem
               4        activity
                                                                        P3 or P300
               2                   P1                                   Late positive
  Microvolts




                                                                        complex
               0

               -2

               -4

               -6
                                        N1 or N100
               -8
                    0   50   100   150    200      250    300   350   400       450       500
                                                Milliseconds

Integrity Testing                                                           BCS-ISSG Oxford, March 2001
      Quantitative Science

     ! Investigative methodology that is characterised by
     the construction of variables which possess a
     quantitative structure.
     ! A variable which possesses quantitative structure is
     required to satisfy the 9 conditions of ordinality and
     additivity (the 9 uniformities of co-existence from
     J.S. Mill)




Integrity Testing                                BCS-ISSG Oxford, March 2001
  The 9 axioms of Quantitative Measurement .1

   From Michell (1990, p.52):
   Let X, Y, and Z be any three values of a variable Q. Then
   Q is ordinal if and only if:

   1. If X ≥ Y and Y ≥ Z then X ≥ Z (transitivity)
   2. If X ≥ Y and Y ≥ X then X = Y (antisymmetry)
   3. Either X ≥ Y or Y ≥ X (strong connexity)

   A relation possessing these three properties is called a
   simple order, so Q is ordinal if and only if ≥ is a simple
   order on all its values.
Integrity Testing                                   BCS-ISSG Oxford, March 2001
  The 9 axioms of Quantitative Measurement .2
 All quantitative variables are simply ordered by ≥ , but not
 every ordinal variable is quantitative, for quantity involves
 more than order, it involves additivity.
                               additivity
 Additivity involves a ternary relation, symbolized as “X+Y=Z”. Let Q
 be any ordinal variable such that for any of its values X, Y, and Z …

 4. X+(Y+Z) = (X+Y)+Z (associativity)
 5. X+Y = Y+X (commutativity)
 6. X ≥ Y if and only if X+Z ≥ Y+Z (monotonicity)
 7. If X > Y then there exists a value of Z such that X=Y+Z (solvability)
 8. X+Y > X (positivity)
 9. There exists a natural number n such that nX ≥ Y
   (where 1X = X and (n +1)X = nX + X) (Archimedean condition)
Integrity Testing                                        BCS-ISSG Oxford, March 2001
  ! Measurement in quantitative science is defined as the
  identification of a magnitude of a quantitative variable
  relative to some standard unit magnitude of that variable.
  ! This measurement uses numerical relations to express
  the ratio of magnitudes.
  ! These numerical relations are the real-valued, positive
  number system.
  ! There is thus an isomorphic (one-to-one) relation
  between the numbers used to represent magnitudes, and
  the standard unit for a variable.
  ! The concatenation of standard units for a variable is
  additive – given a fixed-property unit.

Integrity Testing                              BCS-ISSG Oxford, March 2001
  So?
 ! Integrity measurement has no standard unit.
 ! Any measurement that uses conventional, classical
 psychometric quantitative procedures is required to
 assume that such a standard unit exists.
 ! Science requires that the meaning of the proposed
 standard unit is subsequently explored, tested, and better
 understood.
 ! This does not mean that integrity cannot, or is not,
 being measured by various procedures, but rather, the
 laissez-faire attitude toward measurement taken by most
 psychologists renders the measurement of integrity as a
 somewhat ambiguous issue to be approached with care.
Integrity Testing                              BCS-ISSG Oxford, March 2001
   Reliability
   The measurement remains stable over time (test-retest),
   and the components of a composite measure (test score)
   all measure the same attribute (internal consistency).
   Associated with internal consistency reliability is the
   concept of a measure being a measure of a single,
   unidimensional attribute

   Validity
   Does the test actually measure what it purports to
   measure?


Integrity Testing                              BCS-ISSG Oxford, March 2001
   Corrections for Unreliability and Restriction of Range


   Most quoted correlations between Integrity measures
   and Job Performance, offences, or counter-productive
   behaviour are corrected for unreliability of
   measurement in the criterion or test and/or restricted
   range of measurement in the test and/or the criterion.




Integrity Testing                              BCS-ISSG Oxford, March 2001
                          Measures                       Uncorrected       Corrected
                                                           (actual)       (True-Score)
               Integrity Test vs Job Performance
                                                           0.33              0.47
       Personality Composite vs Job Performance
                                                           0.25              0.39
  *Personality Composite = agreeableness, conscientiousness, and emotional stability


     Taken from:
     Ones, D.S., Viswesvaran,C., and Schmidt, F.L (1993) Comprehensive Meta-
     Analysis of Integrity Test Validities: Findings and Implications for Personnel
     Selection and Theories of Job Performance, Journal of Applied Psychology,
     vol.78 (4) 679-703.

     Ones, D.D., Schmidt, F.L. and Viswesvaran, C. (1993) Nomological net for
     measures of integrity and conscientiousness. Paper presented at the 8th annual
     conference of the Society for Industrial and Organizational Psychology , San
     Francisco.
Integrity Testing                                                   BCS-ISSG Oxford, March 2001
                             N=2000 cases, Bivariate Normal sample, Correlation=0.43
                      24

                      22

                      20

                      18
    Job Performance




                      16

                      14

                      12
                                                                        Region of Interest
                                                                        Individuals with low
                      10
                                                                        Dishonesty scores possess
                                                                        the same Job
                       8                                                Performance rating as
                                                                        those with high scores
                       6
                        25    35          45        55         65        75          85            95
                                   Dishonesty Score (high score = more dishonest)
Integrity Testing                                                               BCS-ISSG Oxford, March 2001
                             Filtered Dataset (no case above population mean score)
                                            N=1023, Correlation = 0.34
                      22

                      20

                      18
    Job Performance




                      16

                      14

                      12

                      10

                       8

                       6
                        25    35        45         55        65        75          85             95
                                Dishonesty Score, High Score = more Dishonesty)
Integrity Testing                                                             BCS-ISSG Oxford, March 2001
                            Filtered Dataset - Integrity Scores between 46 and 64
                                         N=1400, Correlation = 0.32
                     22

                     20

                     18
   Job Performance




                     16

                     14

                     12

                     10

                      8

                      6
                       25   35        45        55         65         75         85             95
                             Dishonesty Score (High Score - more Dishonesty)
Integrity Testing                                                           BCS-ISSG Oxford, March 2001
  Corrected vs Uncorrected Correlations


                                  Uncorrected    Corrected
   Population Correlation
                                   0.43             -
   High Scorers Clipped
                                   0.34           0.52
   Low and High Scorers Clipped
                                   0.32           0.49


Integrity Testing                               BCS-ISSG Oxford, March 2001
     But, do we want to use correlations at all as
     an indicator of the “utility” or “validity” of a measure?

     Let’s look at some simulated Integrity Score prediction
     data – where we have a base rate of 10% “undesirables”
     – i.e. those who are classified by their behaviours as
     “lacking integrity”.

     First, we see how well we can predict the undesirables if
     we were to just randomly reject 1 out of 10 candidates.


Integrity Testing                                 BCS-ISSG Oxford, March 2001
      *Simulated data, 10% base rate, no test – random selection




Integrity Testing                                    BCS-ISSG Oxford, March 2001
      *Simulated data, 10% base rate, Integrity Test Selection




Integrity Testing                                     BCS-ISSG Oxford, March 2001
      *Simulated data, 10% base rate, Integrity Test Selection




Integrity Testing                                     BCS-ISSG Oxford, March 2001
     If the cost of a False Negative (someone we predict to be
     “honest” and so employ, but in fact they turn out to commit
     dishonest behaviours) is £10,000 on average (taking into
     account all the costs associated with fraud detection, HR
     issues, and job-replacement), then by selecting an
     instrument solely upon its “validity” correlation
     coefficient, we might choose the one with a validity
     coefficient of 0.3325 (Pearson r) – which would cost us
     £290,000 in terms of “failure”. If we chose the other one
     with a coefficient of 0.3331, it would cost us £110,000 –
     a cost saving of £180,000. As the costs of fraud near
     £25,000 per individual, savings approach £½million.

Integrity Testing                                  BCS-ISSG Oxford, March 2001
      Hard and Soft Criteria?

      Hard … Stealing of goods, Absenteeism, Aggressive
      Incidents, Financial Fraud, Shrinkage

      Soft … Supervisor Ratings, Workplace Attitudes,
      Performance Targets, Dress




Integrity Testing                              BCS-ISSG Oxford, March 2001
  Social Desirability not a problem?

     o When candidates distort their responses, this can be
       systematic in that scale scores are elevated by some
       constant across all candidates (everybody tends to
       increase their scores say on conscientiousness).
       This kind of distortion has no effect upon the
       affected trait scale score and some criterion ….




Integrity Testing                                BCS-ISSG Oxford, March 2001
                           Systematic Variation: Hypothetical Applicant and Non-Applicant Data
                                       Correlation between Job Criterion score in each group = 0.52
                          14
                                       Non-Applicants
                          12           Applicants

                          10
    Job Criterion Score




                           8

                           6

                           4

                           2

                           0
                               0   1          2         3      4        5        6        7           8          9          10
                                                        Conscientiousness Sten score                      Brown and Barrett, 1999

Integrity Testing                                                                                 BCS-ISSG Oxford, March 2001
  Social Desirability not a problem?

 o Alternatively, the distortion may be non-systematic,
                                                systematic
   with certain candidates obtaining elevated scores whilst
   others remain static. This kind of distortion has
   unpredictable consequences upon trait-criterion
   correlations. An example below shows what happens
   when “true low-scorers on conscientiousness” tend to
   fake-good at a rate relative to the size of their low
   scores, whilst average to high scorers maintain their
   “true” score.



Integrity Testing                             BCS-ISSG Oxford, March 2001
                                  Non-Systematic Distortion of Conscientiousness scores (Faking Good)
                                     Non-Applicant correlation = 0.52, Applicant Correlation = -0.08
                         14
                                     Non-Applicants
                         12          Applicants


                         10
   Job Criterion Score




                          8

                          6

                          4

                          2

                          0
                              0     1        2        3       4        5        6       7        8          9          10
                                                       Conscientiousness Sten Score                  Brown and Barrett, 1999

Integrity Testing                                                                                BCS-ISSG Oxford, March 2001
   Key “competing” References:

   Ones, D. S., Viswesvaran, C. & Reiss, A. D. (1996).
   Role of social desirability in personality testing for
   personnel selection: the red herring. Journal of Applied
   Psychology, 81, 660-679.

   Barrett, P.T. and Hutton, R. (2000) Personality and
   Psychometrics. Selection and Development Review, 16, 2,
   5-9



Integrity Testing                                BCS-ISSG Oxford, March 2001
          A simple question that should be asked by anyone
          thinking of using any psychological test …

                    How accurate is this test?

               And …
               Is a correlation/validity coefficient a
               sufficient answer to this question?
               Answer = a resounding No!

Integrity Testing                                 BCS-ISSG Oxford, March 2001
                                                             Dose-Response Curve for a Continuous/Ordered-Category criterion variable
                                                   1.0
   Probability of a Counter-Productive Behaviour




                                                   0.9

                                                   0.8

                                                   0.7

                                                   0.6

                                                   0.5

                                                   0.4

                                                   0.3

                                                   0.2

                                                   0.1

                                                   0.0
                                                         1           2         3         4           5         6        7            8            9
                                                                                             Integrity Score
Integrity Testing                                                                                                           BCS-ISSG Oxford, March 2001
                        Predicting risk of violent recidivism in forensic mental health
                                              Rates of Violent Recidivism for subjects at each of 9 risk levels
                                                        7 year recidivism follow-up - Taken from Rice (1997)
                                    1.1
                                                                                                                              9
                                    1.0
      Rate of Violence Recidivism




                                    0.9                  These are the numbers
                                    0.8                  of patients who obtained                                  29
                                                         the VRAG score
                                    0.7
                                    0.6                                                                74
                                    0.5                                                       96
                                    0.4                                              116
                                    0.3
                                                                            111
                                    0.2                            101
                                                          71
                                    0.1
                                                  11
                                    0.0
                                          0        1       2        3        4        5        6       7           8          9
                                                                 Violence Risk Level (VRAG)
Integrity Testing                                                                                              BCS-ISSG Oxford, March 2001
   The VRAG dichotomous/binary outcome decision table




Integrity Testing                         BCS-ISSG Oxford, March 2001
  " ROC analysis permits an investigator to evaluate a fixed
  or “to be inferred” criterion value to make accurate
  discrimination between two outcomes (whether they be 2
  different stimuli, stimulus vs no-stimulus, violent offence vs
  no violent offence, or risk factor vs no risk factor). The
  criterion value may be a test score, level of risk factor, drug
  dosage, cognitive judgement, or any other kind of variable
  that can possess at least a binary categorical, multiple
  ordinal, or even equal-interval magnitudes).
  " The ROC curve, in every case, consists of plotting the
  True Positive rate on the ordinate (vertical, y-axis) and the
  False-Alarm rate (False Positive Proportion or rate) on the
  abscissa (horizontal, x-axis).
Integrity Testing                                  BCS-ISSG Oxford, March 2001
                                                                                            ROC Curve for Quinsey et al (1994) VRAG instrument
                                                                                                     Using 7 yr recidivism probabilities
                                                                          1.0
                    Sensitivity (Prob. of correctly predicting offence)


                                                                                   VRAG Score = 5       VRAG Score = 4
                                                                          0.9

                                                                          0.8

                                                                          0.7

                                                                          0.6

                                                                          0.5

                                                                          0.4
                                                                                                                               50/50 Chance level
                                                                          0.3

                                                                          0.2

                                                                          0.1

                                                                          0.0
                                                                             0.0      0.1      0.2    0.3     0.4    0.5     0.6    0.7     0.8     0.9    1.0
                                                                                               False Alarm, False Positive Rate (1-Specificity)
Integrity Testing                                                                                                                                 BCS-ISSG Oxford, March 2001
  Interpreting a classification function
                                           A Classifier function as utilised in the ATH-1

                       The General Public                                               The Prisoner
                       Score Distribution                                               Score Distribution
                                                                    xc




                       True Negatives                                                           True Positives
         Probability




                                                                          False Positives
                                        False Negatives




                                                                                      xc The decision
                                                          The Honesty Variable        criterion value

Integrity Testing                                                                                BCS-ISSG Oxford, March 2001
  The weighting of Integrity Test Scores in a
  Multi-Attribute Selection Battery

  ! That is, is it the case that regardless of any other test
  scores, if the integrity score signals “low degree of
  honesty”, then the candidate’s application is rejected?
  ! Psychologists talk about an integrity test as just one of
  many indicators of a prospective employee’s attributes
  – but this is not how they are used. If you are told that a
  candidate’s “honesty” score is very low, would you
  want to employ them, even if they meet other targets on
  your person specification?

Integrity Testing                                BCS-ISSG Oxford, March 2001
   The Meaning of Integrity Test Scores?

        ! Conscientiousness and Integrity: Ones et al (1993) …
        she demonstrated that although Conscientiousness and
        Integrity were moderately correlated (0.26 uncorrected
        and 0.39 corrected for unreliability and restriction of
        range in both tests), job performance was best
        predicted by integrity test scores.
        ! Eysenck Personality Questionnaire – Tough-Minded
        vs ATH-1 Integrity = 0.62 (uncorrected) – a study
        carried out by myself, and published by Permetric (the
        original ATH-1 test publisher) as a data addendum in 1989


Integrity Testing                                   BCS-ISSG Oxford, March 2001
        ! Permetric also found that many entrepreneurial
        managers and executives had ATH-1 scores much
        closer to prisoners than to the general public.
        However, this makes a great deal of sense in that
        entrepreneurs are mould-breakers by definition – they
        are not compliant, and do not always follow others’
        rules except if they are congruent with their own
        business needs. This interpretation is borne out by the
        EPQ P-scale correlation noted above.



Integrity Testing                                  BCS-ISSG Oxford, March 2001
   ! All interventions within a business environment must
   be subject to explicit quantitative evaluation of outcome
   ! The optimal information about integrity tests is
   embodied in decision-theoretic statistics and full
   prediction error diagnostics.
   ! Stay away from correlations and validity coefficients,
   unless they also come with a portfolio of diagnostic
   statistics that allow you to properly evaluate a cost-
   benefit analysis for the assessment strategy.
   ! Think how you will provide feedback to a rejected
   candidate – for you will have to (APA and BPS guidelines).
   As you do this, you will suddenly realise just how
   critical it is to be able to justify any form of cut-off or
   region of rejection.
Integrity Testing                                BCS-ISSG Oxford, March 2001

								
To top