Foxcroft_Teaching psychological assessment

Document Sample
Foxcroft_Teaching psychological assessment Powered By Docstoc
   What, How and
Psychological assessment is one of the core
competencies of all levels of psychology professionals.

 Even undergraduate psychology students who do not
continue with psychology develop basic measurement
competencies in their studies which could prove to be
useful in the world of work.

Consequently, the curricula of undergraduate and
postgraduate psychology programmes routinely
include modules/courses in psychometrics,
measurement and evaluation, psychological
assessment, and so on.
  This paper aims to raise three questions about the
  teaching of psychological measurement, testing and
  assessment, namely:

1. What should we be teaching?
2. How should we be teaching psychological testing and
3. Who should be teaching psychological testing and
What should we be teaching?
The context (local, national, international and legal) in
which we find ourselves constantly shapes what we
should be teaching.
The HR needs of the world of work and the
competencies required of workers in a knowledge
society shapes the type of graduates that we need to
deliver and, consequently, impacts on the teaching and
learning that we facilitate in our programmes.
Advances in the science and art of psychological
testing and assessment shape what we teach.
It is thus opportune to constantly reflect on whether
what we teach matches up with what the world of work
requires from our graduates and advances in the field
of testing and assessment.
Ways of reflecting
Look at test use patterns and needs of practitioners in
the world of work – generally and for specific categories
of professionals.

Look at what is being taught in programmes around the
country and internationally.

Look at advances in testing and assessment and the
way in which the world of work is changing and then
evaluate whether a learning programme adequately
prepares the learner for the world of work.
HSRC Test Use Survey (2004)

 Tests are used with clients across the age spectrum.

 Tests are used for various purposes (identifying &
 diagnosing psychiatric conditions, describing
 intellectual/cognitive or personality functioning, for
 selection & development purposes, to perform
 specialist forensic and psycholegal assessments.

 Most psychologists, especially those in private practice,
 tend to use tests for similar purposes, irrespective of
 their registration categories.
Purposes for different professional categories
Purpose                            Clin    Couns      Indus      Educ    Res        Psychom
                                   % use   % of use   % of use   % use   % of use   % of use

Psycho-educational                 54.6    64.9       27.2       94.5    30.3       45.7
School readiness assessment        45.8    53.5       10.4       80.9    27.3       40.3
Assessment of learning             50.4    60.0       16.8       92.7    33.3       45.0
Intellectual assessment            76.9    83.8       74.4       94.9    54.5       76.0

Assessment of potential            51.7    59.2       87.2       87.7    45.5       80.6
Career assessment                  50.4    85.4       93.6       84.5    42.4       82.9

Assessment of personality          72.7    83.8       85.6       81.7    54.5       72.9
Employment selection               26.1    38.9       86.4       24.2    39.4       58.1
Selection of people for training   19.7    31.4       72.8       12.3    30.3       55.0
in employment setting

Neuropsychological                 50.4    33.0        8.8       47.5    30.3       17.1
Child custody assessment           21.4    22.0        4.8       33.8    18.2        6.2
Forensic assessment                29.4    28.1       18.4       20.5    27.3       10.9
Types of Tests Used and Patterns of Test Use

 Tests of intellectual ability, personality functioning, and
 interests represent the bulk of the frequently used tests.

 Test-use patterns were particularly influenced by
 whether or not a test has been classified by the Prof.

 Test-use patterns were influenced by the tests
 practitioners were exposed to during their training.

 A major concern is that the majority of the tests being
 used frequently are in need of adapting for our
 multicultural context or required updating.
Most Frequently Used Tests
Test name (Top 13 tests)                   Postal   Focus    Indiv        SA or
                                           survey   Groups   Interviews   International
16 Personality Factor Inventorya               *       *         *        SA
19 Field Interest Inventory (19 FII) a         *       *                  SA
APIL                                           *       *         *        SA
Bender Visual Motor Gestalt Test               *       *         *        International
California Psychological inventory             *       *         *        International
Children’s Apperception Test (CAT)             *       *         *        International
Developmental Test of Visual-Motor             *                 *        International
Integration (Beery)
Differential Aptitude Tests (DAT) a            *       *         *        SA
Goodenough Harris Draw-A-Person Test           *       *         *        International
High School Personality Questionnaire          *       *                  SA
(HSPQ) a
Jung Personality Questionnaire (JPQ) a         *       *         *        SA
Junior South African Individual Scales         *       *         *        SA
Learning Potential Computerised Adaptive       *       *         *        SA
Test (LPCAT)
Most Frequently Used Tests
Test name (top 14 to 25 tests)                Postal   Focus    Indiv        SA or
                                              survey   Groups   Interviews   International
Minnesota Multiphasic Personality Inventory       *        *                 International
Myers-Briggs Type Indicator                       *        *        *        International
Occupational Personality Questionnaire            *        *        *        Adapted for SA

Potential Index Batteries (PIB)                   *        *        *        SA

Rorschach cards                                   *        *        *        International
SA Vocational Interest Inventorya                 *        *        *        SA

Self-Directed Search Questionnaire (SDS) a        *        *        *        SA
Senior Aptitude Tests (SAT) a                     *        *        *        SA
Senior South African Individual Scale -           *        *        *        SA
Revised (SSAIS-R) a
South African Wechsler Adult Intelligence         *        *        *        SA
Scale (SAWAIS) a
TAT (cards)                                       *        *        *        International
Wechsler Intelligence Scale for Children          *        *        *        International

60% = SA dev or adapted. 40% = HSRC. About 25% = up to date.
Top 10 Tests for different categories
Couns Psychologists      Psychometrists     Indus Psychologists
16 PF (SA 92)         16 PF (SA 92)         16 PF (SA 92)
19 Field Int Inv      19 Field Int Inv      MBTI
SSAIS-R               MBTI                  19 Field Int Inv
Bender                SSAIS-R               OPQ
JSAIS                 Bender                SAT
HSPQ                  SAT                   SDS
JPQ                   JSAIS                 JPQ
TAT                   SDS                   RPM
SAWAIS                JPQ                   15FQP+
SAT                   SAWAIS                SAWAIS
                                            High Level battery
                   80%   Independent Prac? 60%
Survey of Training in Tests (2007)
Bender          9
                     • Tests for RCs and
SAIS-R          9
                     • 12 responses.
16PF            7    • Min = 4, Max = 48, M = 17.
SDS             7    • 73% ex-HSRC tests problematic.
DAT             7    • Current test publishers have
PHSF            5
                       R&D sections to adapt & update
JSAIS           5
                    • Focus on screening tests
Beery           5     and checklists
19 FII          5   • Focus on comprehensive
HSPQ            4     training in one IQ, aptitude,
CPQ             4     interest and personality test
                      and limited exposure to
ASB             4
                      other tests.
RPM             4   • Trend – focus less on
WAIS III        4     administration and more on
QNST            4     interpretation and reporting.
Knowledge & Skills – Schepers, 1998
Knowledge of psychometric concepts.
Knowledge of test construction.
Knowledge of psychological theories.
Legal issues.
Best practices guidelines regarding the ethical and fair use of
Cross-cultural testing issues [and language issues]
Knowledge of the factors that impact on test scores.
Skilled in test administration, scoring, interpretation and reporting -
for tests appropriate for level of user and specific contexts and
using different modes of test delivery (e.g., paper- & computer-
based and Internet-delivered testing).
Skilled in test evaluation and knowing when to test and when not
Skilled in developmentally-focused assessment (as opposed to
only diagnostically-focused) and linking assessment results to
developmental/training opportunities.
Skilled in application of measurement principles in variety of work-
related contexts (includes policy-making & implementation).
Classical Testing Tradition
In the classical tradition of testing, psychological assessment practitioners
had to control and have expertise in all aspects of the testing and
assessment process in terms of:
• Having expertise in the underlying cognitive and personality theories of
  the measures that they used.
• Their choice of tests.
• Being well-trained in how to administer the tests that they used and to
  supervise psychometrists and psychotechnicians who performed group
  or individual testing.
• Being well-trained to score tests correctly, convert raw scores to norm
  scores, and interpret test performance with the appropriate use of
  information from various sources and in the light of the theoretical
  underpinnings of the test.
• Being able to compile reports for a variety of different audiences.

In view of the extensive role that assessment practitioners (psychologists
and psychometrists) played in all aspects of the assessment process and
the consequences for test-takers if the assessment was flawed, it was
imperative that the use of psychological measures was strictly controlled.
Modern Testing Era
Impact of globalisation and WWW on assessment.

Mobility of test users and cross-border access to
materials, especially in EU.

Test delivery has become internationalised, in that tests
can be delivered from almost anywhere in the world to
almost anywhere. This implies that individual countries
& their testing regulations can no longer operate as
closed systems.
   Testing Scenario
An Italian job applicant is assessed at a test centre in France
using an English language test. The test was developed in
Australia by an international test developer and publisher, but
is running from an ISP located in Germany.

The testing is being carried out for a Dutch-based subsidiary
of a US multi-national. The position the person is applying for
is as a manager in the Dutch company’s Tokyo office.

The report on the test results, which are held on the multi-
national’s Intranet server in the US, are sent to the applicant’s
potential line-manager in Japan having first been interpreted
by the company’s out-sourced HR consultancy in Belgium.

Which country’s standards apply?
How does supplier decide on user qualification?
Which country should the user be in?
Who is responsible for checking cultural adaptation of
Whose chooses the language?
What norms should be used?
How would the test taker seek redress for unfairness,
and where?
Modern Testing Era
Explosion of CBT and IDT.
The face of testing and assessment has changed
rapidly over the past few decades. Much of this change
has been brought about by the rapid advent of
computer-based (CBT) and Internet-delivered (IDT) tests
and testing.
According to Kriek and Whitford (2006), about 300,000
Internet-delivered tests are administered per month
across the world, and there is a 40% increase in IDT in
South Africa over the past two years. Indeed, CBT and
IDT have revolutionized all aspects of testing – from the
design of tests, through to their administration
(delivery), scoring, and reporting.
Explosion of CBT and IDT
  Automated test administration, which has reduced the need for a
  highly trained test administrator to be present as the
  administrator only needs to verify the identity of the test-taker,
  establish rapport and settle test takers in, and take note of any
  untoward happenings during the session. Consequently, the role
  of the administrator has changed to a more supervisory one.
  While such supervisors need to be trained, the argument that
  such a person has to be a highly trained psychology
  professional no longer holds water.

  A reconceptualisation of the degree of the control and
  supervision required during test administration. CBT and IDT
  have not only resulted in a reconsideration of who supervises
  test administration but it has raised questions about whether a
  supervisor always needs to be present.
 Internet Test Administration modes                        Bartram, 2001

                Anyone can access     No user               No human session

1. Open
                               Insecure mode
                and complete the
                                      identification or

                Access is             Known user is sent    No human session
                           Moderately secure
                controlled by login
                                      specific login &
2. Controlled
                Access controlled     Supervisor has        Human session
                by login process,     candidate login &     supervision
                location insecure     password
3. Supervised
                Location controlled
                                      Maximum security      Human session
                (e.g. Test Centre)    for item content      supervision
4. Managed
Explosion of CBT and IDT
  Automated scoring and the availability of expert
  interpretation systems, which requires a rethink of the
  role of the psychologist.

  Computerized reports, which need to meet certain
  standards and should be tailored by psychologists so
  as to provide appropriate feedback to the relevant
  target audience(s). Thus, in terms of reporting, the
  role of the psychologist is also under scrutiny. Is
  main role to provide input to the development of
  expert systems and how to create user friendly
Explosion of CBT and IDT
The conversion of inputs based on psychological traits
into outputs based on competencies required in the
work place (Bartram, 2003). By translating
psychological test information into the language used
in selecting and developing people in the workplace
this information becomes more accessible for HR and
other line managers to use. This in turn, starts the
debate on how to define a “test user” and narrow
definitions used in the past (e.g., when it comes to
psychological tests in SA, a test user is defined as a
registered psychology professional) no longer seem to
be in step with who uses assessment results in the
everyday world.
Psychological & Competency-based assessment

Psychological assessment requires expertise in
psychology and psychological theories to ensure that
measures of cognitive, aptitude, personality, etc.
functioning are used in an ethical and fair manner, right
from the choice of which tests to use through to
interpretation and feedback. Furthermore, the outputs
of psychological assessment are in the form of
psychological traits/constructs (such as personality
and ability). The expertise to perform psychological
assessment is clearly embodied in an appropriately
registered psychology professional.
Psychological & Competency-based assessment

Competency-based assessment focuses on the skills, behaviours,
knowledge, and attitudes/values required in the work place or in
educational settings (e.g., decision-making, critical thinking,
leadership, influence). The assessment measures used are as
directly linked as possible to competencies acquired in the work
place or educational setting. Indirect methods such as simulations
and Assessment Centres are used to conduct competency-based
assessment. As the outputs of such assessment are directly linked
to the language of the work place or educational settings, the test
user does not need expertise in psychological theories to be able
to apply the results of competency-based assessments. What is
required, however is that competency-based assessment needs to
be performed by people with expertise in this area of assessment
(e.g., skilled in competency-based interviews).
Should traditional Psychology Departments broaden who they
train? For example, offer an advanced diploma in competency-
based assessments for managers, HR practitioners, etc.?
Grey Area
In accordance with the simple definitions provided above, the
obvious rule of thumb to use is that psychological assessment
should only be performed by appropriately registered psychology
professionals and competency-based assessment could be
performed by “non-psychologists”.

Grey Area: In view of the online management of the assessment
process, it is possible to use psychological tests, or even the
selection of item sets from various psychological tests that tap the
required competencies to be measured and to then produce test
output that is solely in terms of competencies and not
psychological traits and constructs. In this instance, the “online
test user may never see the test in any physical form, will never
have to score it, look up norms, or carry out an interpretation”
(Bartram, 2003, p. 4). According to Bartram (2003), if the report that
is generated focuses solely on work place competencies, then
there is no reason why a non-psychologist cannot use it and the
non-psychologist should not be required to undergo training in the
psychological test(s) used to generate the output (report).
Psychology professionals role in assisting in application of results
w.r.t. implications for selection and/or training.

Our training programmes may not be in step with the
modern testing era.
Only 25% of tests on Top 25 list of frequently used tests
(HSRC survey) are CBT or have a CBT version.
Only 25% of the tests included for RC and
psychometrist training are CBT.
Not enough of the tests that are currently undergoing
the most adaptation and development are included in
the list (PsyTech, SHL, Jopie van Rooyen).
Are we preparing students for the world of CBT & IDT
and the changing roles of psychology professionals?
Or are we training them in the classical model?
Classic versus modern
How should we be teaching & training?

Outcomes-based education in SA.

Internationally there has been a move to a competency-
based approach to training in psychological testing and
assessment. We can learn from this and adapt the
model for our context
EFPA-EAWOP Task Force on Test User Qualification

  Aims to:
 • define standards of competence in test use
 • facilitate the recognition of qualifications that meet those standards
 • assist countries in developing such qualifications
 • audit proposed qualifications to see if they are in line with the EFPA-
   EAWOP standards.

  Consider Knowledge and Skills in relation to:
 • General – common across all areas of testing
 • Domain related – related to general area of application
 • Specific–instrument related

  Three broad domains: Clinical, Educational and Work
 • For example, Work qualifications could be divided into three sub-
   - Selection and recruitment
   - Individual development
   - Organizational development
Standards structure: Initial content input from BPS
standards and ITC Guidelines

    Standards of          Evidence            Learning
    competence            requirements        specifications

    Performance           Rules of evidence   Knowledge and skill
    requirements set by   and assessment      requirements
    EFPA-EAWOP with       specifications      defined by EFPA-
    consultation.         defined by EFPA-    EAWOP
    Context defines area                      Learning input
    and range of         Methods of           methods locally
    application          assessing            defined
                         competence left to
                         local decision
How should we be teaching & training?
Need to develop a competency-based model nationally. The Board is our
SGB and quality assurance body.

Some pointers:
• Get clarity about the roles and competencies of different categories/levels
  of psychology professionals.
• Contemplate training RCs and psychometrists in a problem-based way,
  e.g., training them in specific batteries of tests for specific contexts/client
  needs (e.g., school readiness screening, screening for depression) –
  instead of providing them with generalist assessment training.
• Emphasis needs to be on what competence must be demonstrated in and
  the evidence of such competence, instead of the inputs and the specific
  tests trained in. But need to decide on the scope of measures in which
  competence should be demonstrated.
• Achieve a balance between developing knowledge competencies, applied
  (skill) competencies and attitudinal competencies (values, ethics).
• Do not isolate training in testing and assessment from other areas of
  psychology (theories and intervention).
Who should be teaching?
Junior versus senior staff?
Eyde and Childs (1998) surveyed the qualifications of
those who teach assessment courses in the USA and
found that senior faculty tended to teach assessment
courses and that 76% of them had doctorates.
Situation in SA – senior staff are involved at PG (and
especially masters level) but less experienced staff are
sometimes involved at an UG and 4th year level.
From a QA perspective, should we not be setting
minimum qualification levels and levels of
measurement & assessment knowledge and applied
experience needed?
Use of people in the field as guest lecturers?
Closing Thoughts
Questions and reflection have value as they move us
out of our comfort zones.

But now action is necessary to move our testing and
assessment teaching training from the classical
tradition to the modern era.

How? Establish a task force under the leadership of the
Education/Psychometrics Committee of the
Professional Board, or both?