Document Sample

                Rubik’s cubes of colour,
                     shape and size:
              Ethical psychological testing
                 in a multicultural work

               Department of Industrial & Organisational
                           18 August 2010
Morning session: 08h30 - 13h00
• Welcome by Prof Dirk Geldenhuys
• Introduction: Welcome to the Rubik’s Cube
• A short history of psychological assessment
• The areas of psychological assessment
• Psychological assessment myth busters
Tea break: 11h00
• Legislation and ethics
• Ethical freeze role play
Lunch: 13h00 - 13h45
• Current topics in psychological assessment
• Psychological assessment Crosswords
• The future of psychological assessment
Conclusion: 15h00
A short history of Psychological Assessment
                   Why look at the past?
 History influences current practice

     Relevance
     Evolved and progressed

 History is important

     Helps explain current practice
     Strengths and weaknesses
     Prevent repetitions of the ‘wrongs of the past’
    History of testing: An International Perspective
 Over the years many authors, philosophers and scientist have explored
  various avenues in an attempt to assess human attributes. To mention a
 Astrology (positions of planets to describe the personality of individuals
  and predict what may happen in their lives)

 Physiognomy (judging the character of a person from external features of
  the body, especially the face.)

 Graphology (the systematic study of handwriting and the notion that it
  can be used as an expression of personality characteristics)

 Humorology

 Phrenology
(Foxcroft, 2009)
         Testing in South Africa: The Early Years

• Psychology emerged as an academic discipline in South Africa after
  World War 1.

• Many of the influential figures in the development of intellectual tests
  in South Africa where trained at American institutes.

• Therefore tests which where typically used in the early phases
  where also American such as; Goddards 1911 revision of the
  Binet-Simons test, Termans 1916 Stanford revision, the Army Beta,
  the Porteus Maze Test etc.

• In the period between the two world wars, social and human
  sciences in South Africa became important contributors to debates
  on crucial social issues including ‘the Native question”

(Louw, 1997; Foxcroft, 2009; Classen, 1997)
      Testing in South Africa: The Early Years

 Psychological knowledge was used as a tool legitimise a social
  order based on race as the perception was that psychological
  testing produced empirical data which supported certain
  explanations of this order.

 There was an increase in the use of science as a factor in regulating
  aspects of social economic life.

 Differential performance of black and white reinforced the idea of
  hierarchy of human societies and consequently differential treatment
  in terms of education, employment etc.

  (Louw, 1997)
Key notes
                Psychometric Testing

Psychometric testing is applied worldwide and used in various
industries for recruitment, selection and counselling purposes
(Gregory, 2000).

For the purpose of this seminar four important areas for
assessment have been identified.

    Cognitive assessment
    Personality assessment
    Behavioral assessment
    Interest
       Range of tests and distributors - Cognitive
      Range of tests and distributors availableavailable -
MEASUREMENT     CONCEPT                    TEST             DISTRIBUTORS            APPLICATION IN
 CONSTRUCT                                                                           INDUSTRIAL/
Cognitive     IQ,                  WAIS, SSAIS, ASAT,      Mindmuzik Media       Personnel screening and
              General              RPM,                                          selection,
              cognitive,           DAT Forms R, S, K & L                         Career guidance and
              Multiple aptitude,                                                 counseling,
              Learning             CPP, CTA                Magellan Consulting   Personnel development
                                   APIL-B,                 AproLab
                                   TRAM 1 & 2,             M&M Initiatives

                                   LPCAT                   M&M Initiatives

                                   TST                     Thomas

                                   COPAS                   Integrity
       Range of tests and distributors - Cognitive
      Range of tests and distributors availableavailable -
MEASUREMENT     CONCEPT                    TEST             DISTRIBUTORS            APPLICATION IN
 CONSTRUCT                                                                           INDUSTRIAL/
Cognitive     IQ,                  WAIS, SSAIS, ASAT,      Mindmuzik Media       Personnel screening and
              General              RPM,                                          selection,
              cognitive,           DAT Forms R, S, K & L                         Career guidance and
              Multiple aptitude,                                                 counseling,
              Learning             CPP, CTA                Magellan Consulting   Personnel development
                                   APIL-B,                 AproLab
                                   TRAM 1 & 2,             M&M Initiatives

                                   LPCAT                   M&M Initiatives

                                   TST                     Thomas

                                   COPAS                   Integrity
                         Cognitive Assessment

   • Some consensus
      – IOPs agree that cognitive ability test are valid and fair
      – Cognitive ability tests provide good but inadequate
        measures of intelligence
      – Other attributes are necessary to account for multi
        dimensional nature of performance (90% of

(Murphy, Cronin & Tam)
                            Cognitive Assessment

• Some controversy
   - Societal concerns
      - Cultural bias
      - Education
      - Socio-economic status
      - Education levels of parents
      - Effects of negative stereotypes
   - The g-Ocentric vs MI

(Murphy, Cronin & Tam)
(McKay, Doverspike, Bowen-Hilton & McKay)
                     Cognitive Assessment

 • Unifying theoretical approach
    – 3 approaches
       • Structural approach (factor analytic)
       • Information-processing
       • Dynamic (based on learning theory)

           “Learning as the critical factor underlying cognitive competence
                    and the mastery of problems and challenges”

Structural - Spearman (g – one general factor) and Thurstone 7&9
Primary mental abilities
        - fluid & crystallised (Cattell)
        - mostly power tests

Info processing – e.g. Ravens progressive matrices
               receiving, processing & retrieval speeds
               computer admin essential

Learning / dynamic approach
• Vygotsky – Zone of proximal development (ZPD)
•less susceptible to cultural bias
•3 phases of learning – conceptual understanding, compilation of
execution procedures & automatization of processing.
•Test – teach – test or pre-test – mediation – post-test

Emphasis on potential and not achievement
      Range of tests available: Personality

MEASUREMENT    CONCEPT                 TEST                 DISTRIBUTORS            APPLICATION IN
 CONSTRUCT                                                                           INDUSTRIAL/

Personality   Personality     MMPI, TAT, SORT, VS ,      Mindmuzik Media           Personnel selection,
              preferences,    WLQ                                                  Career counselling,
              Management                                                           Academic counselling,
              styles,         OPQ 32                     SHL                       Personal development,
              Communication                                                        Psychological growth,
              styles,         15 FQ, OPP, Jung           Psytech                   Candidate - job match,
              Interpersonal                                                        Management
              styles,                                                              development,
              Coping styles   MBTI, Big Five, WPI, LSP   Jopie van Rooyen          Stress management
                              MSP, CPI

                              PAW                        Integrity International
      Range of tests available: Personality
 CONSTRUCT                                                                  INDUSTRIAL/
Personality   Locus of       LCI               Jopie van Rooyen           Personnel selection
              control,       AMP                                          Career counselling
              motivation                                                  Academic counselling,
              and autonomy                                                Personal development
                                                                          Psychological growth,
              Emotional      MP                SHL, Magellan Consulting   Candidate - job match,
              Intelligence                                                Management
                             EQI               Jopie van Rooyen           development,
                                                                          Stress management
                             15 FQ EI          Psytech

                             TEI               Thomas International
              Integrity      Giotto            Mindmuzik Media

                             PSI               Jopie van Rooyen

                             IP-200            Integrity International
      Range of tests available: Behavioral
 CONSTRUCT                                                            INDUSTRIAL/

Behavioural   Underlying    BESS              Jopie van Rooyen   Screening and selection
              emotional                                          Training and development
              issues,                                            Promotion
              Stress,                                            Management potential
              Anxiety,      BAI               Mindmuzik Media    Group-role analysis
              Depression,   BDI-II            Jopie van Rooyen   Interpersonal relationships
              Conflict      BDI Fast screen                      Career psychology
                                                                 Personnel psychology
                                                                 Organizational psychology
                            SWSI              Jopie van Rooyen

                            SP                Mindmuzik Media

                            TKI               Jopie van Rooyen   Conflict management
                                                                 Personal development
           Range of tests available: Interest

 CONSTRUCT                                                        INDUSTRIAL/

Interest      Career      SDS              Jopie van Rooyen   Career guidance and
                          CVS                                 counselling
                          CIP                                 Personal development and

                          19 FII           Mindmuzik Media

                          IDEAS workbook   ??
                Cost Pricing Analysis
• Option 1: Purchasing a “report”
   – Advised for non-accredited users and for individual testing on
     once-off basis
   – Some test can be expensive (up to R3,000 per report)
   – Suppliers include SHL, Jopie van Rooyen, and Thomas
   – Benefit: initial costs is low
       Computer generated reports is time efficient
   – Downfall: ongoing high costs
       Potential misuse of reports due to lack of control
                Cost Pricing Analysis

• Option 2: Purchasing Material: Software or Paper-and-Pencil version
   – Advised for accredited users and group or large-scale testing on
     ongoing basis
   – Initial purchasing of material and software can be expensive
     (need manuals, booklets, answer sheets, scoring stencils, etc –
     between R2,000 – R25,000)
   – Suppliers include Minimize, M&M Initiatives, PsyTech
   – Benefit: ongoing assessments at minimal fee
   – Downfall: initial start-up fees can be high
       Software need ongoing purchasing of credits for reporting
       Manual interpretation, scoring and reporting is time consuming
Key notes
Legal aspects of Psychological Assessment
The Constitution of the Republic of South

–   Chapter 2: Bill of Rights “is a cornerstone of democracy in South
    Africa - democratic values of human dignity, equality and freedom”.

–   Section 9(3): “The state may not unfairly discriminate directly or
    indirectly against anyone on one or more grounds, including race
    gender, sex, and pregnancy, marital status, ethical or social origin,
    colour, sexual orientation, age, disability, religion, conscience,
    belief, culture, language and birth.

–   Section 9 (4): “No person may unfairly discriminate directly or
    undirectly against anyone on one or more grounds in terms of
    subsection (3).
    Employment Equity Act, No.55
• Section 8: “Psychological testing and other
  similar assessments of an employee are
  prohibited unless the test or assessment being
  used –
  – Has been scientifically shown to be valid
      and reliable;
  – Can be applied fairly to all employees;
  – Is not biased against any employee or
               EEA (continued)
•   Section 11: Burden of proof.

•   Section 15: Affirmative action.

•   Subsection 20(3): “capacity to acquire, within a
    reasonable time, the ability to do the job”

•   Code of Good Practice on the Integration of
    Employment Equity into Human Resources
    Policies and Practices: Section 10.
       The Labour Relations Act,
             No. 66 (1995)
• Chapter VIII: Unfair discrimination and unfair
  labour practices – same grounds as the
• Chapter VIII: Unfair discrimination and unfair
  labour practices – same grounds as the
                       Legal Cases
•   Association of Test Publishers of South Africa Saville &
    Holdsworth (SA) and the Chairperson of the Professional
    Board for Psychology

•   Dispute: ATP: Any individual (unregistered) should be able to
    conduct psychometric assessments? HPCSA: Only registered
    professionals !

•    Ruling (19 February 2010): In favour of ATP SA. “The court order
    declares the notice published on the 10th November 2008, which
    stated that “it is not permissible to use unregistered persons to
    render psychological services including the administration of tests,
    instruments or techniques”, to be void and of no force and

•   Current status: This ruling will stand until further notice. WATCH
    THIS SPACE!!!!
          Legal Cases (continued)

• Buthelezi v Amalgamated Beverage
  [1999] JOL 5086 (LC)

• Hendricks v SA Airways
  [2002] JOL 10382 (LC)

• NUM obo Moeng / Douglas Colliery
  [2007] 7 BALR 647 (CCMA)
What will the long arm of the law do
               to you?

• Fines: Employment Equity Act

• Costs: CCMA, Labour Court,
  Constitutional Court
               Closing statement
• Assessments play an important role in the measuring of
  performance, selection, recruitment, etc. in the
  workplace, schools and various other institutions.

• Assessment Practitioners should today more than ever
  be aware of the responsibilities that the legal use of
  these powerful instruments require.

• Our history has not been without bumps, hurdles and
  mountains, but it is our responsibility to promote a fair
  and unbiased future and set an example for the rest of
  the world. The current limitations and challenges that we
  face can only be achieved through co-operation,
  adhering to legislation and high ethical standards.
We rest our case…
The Regulation of Psychological Assessment

                Regulation by

                Regulation by

 Employment Equity Act (1998) – different to all other countries, in
  that we are asked to proactively defend our position with regards to
  psychological assessment

• (Responsibilities: Training, Accreditation, Test classification)

 HPCSA -- might not be in a position to efficiently regulate access to
  and use of tests
 Psychometric Committee of the board of Psychology – falls under
 International Test Commission (2001)
 Society for Industrial and Organisational Psychology of South Africa
 People Assessment in Industry
 Human Sciences Research Council

   South African Board for People Practices
   Association of test publishers
   Test Developers/Publishers
   Any others? What about universities?


 Strategy of self-regulation (Paterson & Uys, 2005) – practitioners
  empowered to make informed decisions through access to
  information and adequate training
 Training = better decision making with regards to selection of
  (registered) tests
 Continuing professional development
 Personal moral compass

 Conflicting policies (e.g. Employment Equity Act (1998) –
  assessments must not discriminate against groups, but
  can be used to assist with addressing previously
  disadvantaged groups)

 Personal morals vs Ethical policies

 Canadian Psychological Association – ethical principle
  ranking to assist decision making
Competency Based Assessment -
    Competency Based Assessment
• What is competency-based testing?
  – Competencies
    • Knowledge, Skills, Abilities
  – Job Analysis
  – Effective in Application
  – Movement from intelligence
Defining Competency Based Assessment

• A way to monitor and assess the competences a person
  has, no matter where or how these competences were

• Three major principles differentiating competence-based
  assessment from other methods used to assess are:
   – Assessment of evidence
   – Current abilities
   – Standards or competence
Benefits of Competency Based Assessment

• An assessment of past behaviour can assist you in
  predicting future behaviour

• Multiple assessment techniques are
  used to verify the results obtained

• The competencies measured are related to successful
  job execution

• Objectivity of evaluation is ensured through the utilisation
  of trained assessors
          Recognition of Prior Learning

“Recognition of prior learning is a process whereby, through
  assessment, credit is given to learning which has already been
  acquired in different ways”. – SAQA

• Discuss practical example of the RPL process
  (real life case study)
                     Benefits of RPL

RPL is seen to have the capacity:
• to contribute to redress equity by opening up more ways for
  people to attain qualified status;
• enable more people to reach higher levels of qualification and
  expertise by beginning with an acknowledgement of existing
  skills and knowledge;
• contribute to enhancing international economic
  competitiveness by building on often invisible and
  unacknowledged workforce skills;
• and offer the first step in attaining the goal of developing a
  multiskilled and flexible workforce by acting as an auditing tool
  to quantify existing competence.
 Competency Based Assessment Series (CAS)

• Designed by JvR professionals and associates
• Purpose is to provide their clients with a range of uniquely South
  African assessment centre exercises
• the exercises are designed to evaluate performance against certain
  job relevant competencies and can contribute to an informed
  selection decision.
• The exercises can be categorised into four groups, namely:
   1.   In-basket Exercises
   2.   Group Exercises
   3.   Strategy and/or Presentation Exercises
   4.   Role Play Exercises
• All the exercises are job relevant and assess current as well as
  future potential.
 Examples of Competency Based Assessment


The aims of the CAS exercises include:
• To identify behavioural competencies needed for a specific job.
• To assist the assessor in gaining a comprehensive understanding of
  the candidates’ behavioural competencies.
• Provide additional, job-relevant, “tangible” information to supplement
  that gathered with traditional psychometric tools.
• To determine the participants’ strengths and development areas.
• To use the information in combination with collateral information for
  selection and development purposes in the workplace.
Examples of Competency Based Assessment

•   Case studies
•   In-basket exercises
•   Job sample tests/Skills Tests
•   Direct Observation
•   Oral/Written Tests
•   Portfolios
•   Simulation
Designing a Competency Based Battery

• Step 1: Planning
  – Purpose
• Step 2: Establish Competencies
  – Job Analysis
• Step 3: Develop assessment battery
  – Structure
• Step 4: Implement and evaluate battery
Uses of Competency Based Assessment

• Within in the organisation
• In the South African context
  – Advantages
  – Disadvantages
• Fairness
  – Unbiased, ethical
Key notes
Computerised Assessment
           Before we start

•   CAPA
•   CBA
•   CBT
•   CAT
•   CBTI
•   IDT
Computer-based Assessment

   Assessment      Assessment
   Generation       Delivery

     Storage,       Assessment
   Retrieval and    Scoring and
   Transmission    Interpretation
     Taking a step back in History

• International:
    Taking a step back in History

• South-Africa
  Late 1970’s and early 1980’s
  Advantages and Disadvantages

• Advantages:
  –   More enjoyable
  –   Ultimate levels of standardization
  –   Biasing effect is eliminated
  –   Reduction of time
  –   More information about test-takers
  –   Spatial and perceptual measured to greater extend
   Advantages and Disadvantages

• Advantages (continued)
  – Voice activated and touch screen responses are
  – Assessments can be individually tailored
  – Greater element of control
  – Number of assessment practitioners and assistants
  – Scoring by a computer
  – Increased test security
    Advantages and Disadvantages
• Disadvantages
   – Eish Eskom
   – Hardware / software
     compatibility & capability
   – Screen size
   – Machine availability
   – Socio-cultural and linguistic
   – System failure
   Advantages and Disadvantages

• Disadvantages (continued)
  –   Copy right violation
  –   Lack of security
  –   Problems with confidentiality
  –   Skilled clinical judgement is overlooked
  –   Difficult to detect certain problems in software
  –   High costs in item development
  –   Costly
   Advantages and Disadvantages

• Disadvantages (continued)
  – Vital qualitative information
    not assessed
  – Human-computer interface
  – Lack of computer literacy
   Computer-based Interpretation

• Four approaches to CBTI are recognized:
  – Scoring reports
  – Descriptive reports
  – Actuarial reports
  – Computer-assisted clinical reports

• What is it

• Purpose

• Characteristics

• Measurement Advantages
         Dynamic Assessment

• What is it?

• Increased attention

• What does it involve?

• South Africa
           Assessing Potential

• Effective management tool

• Time and cost effective

• Usage
              Assessing Potential
• Why will we use it?

   • Present and potential

   • Opportunity to be measured fairly.

   • Is suitable at all levels

   • Scientifically proven valid and reliable

   • Learning takes place during the assessment

   • Is extremely time-saving

   • Uses computerised adaptive testing
           Assessing Potential

• Advantages:
  – Fair
  – Suitable for use at all levels of ability;
  – Valid and reliable indication of potential to
    cope with training;
  – Time saving;
  – Uses CAT which allows for all ability levels.
         Item Response Theory

• Definition

• Higher an individual’s ability level, the
  greater the chance of getting an item

• Each item is referenced
            Item Response Theory

 5                                          Item 2
 4                                          Item 1
     Low Ability   Average   High Ability
Good practice and Ethical considerations

• The International Test Commission (ITC)
  – Give due regard to technological
  – Attend to quality issues in CBT and internet
  – Provide appropriate levels of control
  – Make appropriate provision for security
Good practice and Ethical considerations

• Minimum professional and ethical
  – Competence
  – Potential utility
  – Choose a technically (psychometrically)
    sound computer-based test
  – Equivalence of paper and computer-based
  – Consider the human factors and issues of
Good practice and Ethical considerations

  – Prepare test takers
  – Verify the identity of test-takers
  – Closely supervise the administration
  – Unsupervised psychological test
  – Contingency
  – Securely stored.
  – Computer scoring system
  – Interpretation of results
  – Debrief test-takers
  The future – high definition video and
               virtual reality
• Multimedia
• A five-dimension framework
  – Innovations in item format
  – Innovations in response
  – Innovations in media inclusion
  – Innovations in the extent of interactivity
  – Innovations in scoring
  – Authenticity
The Future of Psychological Assessments
  Research building on existing knowledge

•Enhancing consumer psychology

•Reduced costs

•System generated reports

•Dynamic assessment of intelligence

•Construction of new adaptive tests
         The Future Role of the HPCSA

•Need for an all encompassing body to:
    Monitor test use
    Advise practitioners
    Research and review tests with information centrally
    To monitor and coordinate test development,
    adaptation, and updating (Foxcroft, 2004)
• Control and regulation
•Increased availability of information via internet
                Artificial Intelligence

•Virtual Reality – a new tool for psychological assessment
(Fernandez-Ballesteros, 2009)

•Computer assisted assessment

•Ethical considerations

•Control and access to electronic assessments

•Closer examination of the tester-testee relationship
 The Future of Assessments in South Africa

• Locally developed tests appropriate for all cultures within
• Leadership tests appropriate for SA context
• Explore other intelligences – cosmic, spiritual, moral,
emotional, transcendental contributions
• Stimulating use of assessments by smaller organisations
and NGOs
• Industrial psychologists making
significant contributions locally and
 The Future of Assessments in South Africa

• Psychometric tests – most effective of predicting
  behaviour (Mittner, 1998)
• Tests 4 X more effective than screening interviews (Van
  der Walt, 1998)
• Employment Equity Act – definite move towards making
  assessment techniques more scientific (Eckstein, 1998)
• To be used as additional aid in decision making i.e. not in
• Retain professional level for conducting psychometric
The Future of Assessments in South Africa

• (Bartram & Coyne, 1998) in a recent world-wide survey
  found lack of psychologists involvement in testing and
  use of tests

   – Found 60% of test users were not specifically trained
     in use of psychological tests.

   – Therefore recommend training and increasing public

   – Gregoire, (1999), recommend academic education,
     continued education, and more publications on tests
     and assessment methods.
 The Future of Assessments in South Africa

• Cook (1997), trend towards assessment centres and
  competency based assessment in changing socio-
  political and economic context. [some psychological
  factors that competence measures cannot provide, i.e.
  learning and potential]

• Consideration should be given to development of new
  tests with urgent special emphasis on culture-fairness.

• Validation of tests in different organisations that they are
  used in. Cook (1997),
               Conclusion: History

 In the past the same tests were
  used for everyone.
 Psychologists used international
 Testing was for a long time viewed
  as discriminatory and unjust.
 Practitioners are becoming aware
  of the advantages of fair
 The implementation of cross-
  culturally fair tests.
Conclusion: Assessment Areas



             Conclusion: Legislation

 Changes in legislation.
 The EEA has been established in
  1998 to ensure more equitable
  and fair Psychological testing.
 Tests need to be scientifically valid
  and reliable;
 tests should be able to be applied
  fairly to all employees; and
 tests should not be biased against
  any employee or group.
               Conclusion: Ethics

 Psychological tests can act as
  a disabling factor.
 Practitioners therefore have to
  be more cautious and informed
  and should take responsibility
  to ensure that a test can be
  fairly applied.
 Practitioners should act in an
  ethical way before, during and
  after psychological tests.
 Conclusion: Competency Based Assessment

 Focus on the essential
  behaviours required to perform a
  specific job.

 Focussing on individual
  differences in terms of work-
  related constructs relevant to job

 This contributes to keep tests fair
  and relevant as required by the
   Conclusion: Computerised Assessment

 In our modern society there is new
  technology on the market every
  day and everything is being
  computerised in one way or
  another as well as psychological
 There is a lot of advantages to
  computerized testing we just need
  to make sure that it remains
  unbiased, ethical, valid and
  reliable for our rainbow nation.
               Conclusion: Challenges

 According to research the most
  frequently cited hindrance to the
  administration of psychological
  tests is language.
 When administering individual
  intelligence tests, psychologists
  often argue that it is justifiable to
  administer the measure in
  English, irrespective of whether
  English is the first or second
  language of the test-taker.
              Conclusion: Challenges

 It is important to determine
  whether the performance on the
  test reflects the testee’s actual
  ability and not the testee’s
  competence in the test language.
 Translation of a test is not a quick-
 Changing some wording in an
  item/question can draw a question
  mark over the construct, score and
  predictive comparability.
             Conclusion: Challenges

 Other factors that need to be taken
  into account: cultural and
  environmental factors, age and
 Test anxiety has an effect on test
 Anxiety can hamper a person’s test
 Employees may also start to view tests
  as unfair and invalid.
 Explain everything in simple terms.
“If you talk to a man in a language he understands,
    that goes to his head. If you talk to him in his
language, that goes to his heart.” Nelson Mandela

“ The most important single ingredient in the
formula of success is knowing how to get
along with people” Theodore Roosevelt.
Crossword Competition

Shared By: