Teaching Psychological Assessment: What, How and Who? Cheryl.Foxcroft@nmmu.ac.za www.intestcom.org Introduction Psychological assessment is one of the core competencies of all levels of psychology professionals. Even undergraduate psychology students who do not continue with psychology develop basic measurement competencies in their studies which could prove to be useful in the world of work. Consequently, the curricula of undergraduate and postgraduate psychology programmes routinely include modules/courses in psychometrics, measurement and evaluation, psychological assessment, and so on. Introduction This paper aims to raise three questions about the teaching of psychological measurement, testing and assessment, namely: 1. What should we be teaching? 2. How should we be teaching psychological testing and assessment? 3. Who should be teaching psychological testing and assessment? What should we be teaching? The context (local, national, international and legal) in which we find ourselves constantly shapes what we should be teaching. The HR needs of the world of work and the competencies required of workers in a knowledge society shapes the type of graduates that we need to deliver and, consequently, impacts on the teaching and learning that we facilitate in our programmes. Advances in the science and art of psychological testing and assessment shape what we teach. It is thus opportune to constantly reflect on whether what we teach matches up with what the world of work requires from our graduates and advances in the field of testing and assessment. Ways of reflecting Look at test use patterns and needs of practitioners in the world of work – generally and for specific categories of professionals. Look at what is being taught in programmes around the country and internationally. Look at advances in testing and assessment and the way in which the world of work is changing and then evaluate whether a learning programme adequately prepares the learner for the world of work. HSRC Test Use Survey (2004) Tests are used with clients across the age spectrum. Tests are used for various purposes (identifying & diagnosing psychiatric conditions, describing intellectual/cognitive or personality functioning, for selection & development purposes, to perform specialist forensic and psycholegal assessments. Most psychologists, especially those in private practice, tend to use tests for similar purposes, irrespective of their registration categories. http://www.hsrc.ac.za/research/outputDocuments/1716_Foxcroft_Psychologicalassessment20SA.pdf Purposes for different professional categories Purpose Clin Couns Indus Educ Res Psychom % use % of use % of use % use % of use % of use Psycho-educational 54.6 64.9 27.2 94.5 30.3 45.7 School readiness assessment 45.8 53.5 10.4 80.9 27.3 40.3 Assessment of learning 50.4 60.0 16.8 92.7 33.3 45.0 problems Intellectual assessment 76.9 83.8 74.4 94.9 54.5 76.0 Assessment of potential 51.7 59.2 87.2 87.7 45.5 80.6 Career assessment 50.4 85.4 93.6 84.5 42.4 82.9 Assessment of personality 72.7 83.8 85.6 81.7 54.5 72.9 Employment selection 26.1 38.9 86.4 24.2 39.4 58.1 Selection of people for training 19.7 31.4 72.8 12.3 30.3 55.0 in employment setting Neuropsychological 50.4 33.0 8.8 47.5 30.3 17.1 assessment Child custody assessment 21.4 22.0 4.8 33.8 18.2 6.2 Forensic assessment 29.4 28.1 18.4 20.5 27.3 10.9 Types of Tests Used and Patterns of Test Use Tests of intellectual ability, personality functioning, and interests represent the bulk of the frequently used tests. Test-use patterns were particularly influenced by whether or not a test has been classified by the Prof. Board. Test-use patterns were influenced by the tests practitioners were exposed to during their training. A major concern is that the majority of the tests being used frequently are in need of adapting for our multicultural context or required updating. Most Frequently Used Tests Test name (Top 13 tests) Postal Focus Indiv SA or survey Groups Interviews International 16 Personality Factor Inventorya * * * SA 19 Field Interest Inventory (19 FII) a * * SA APIL * * * SA Bender Visual Motor Gestalt Test * * * International California Psychological inventory * * * International Children’s Apperception Test (CAT) * * * International Developmental Test of Visual-Motor * * International Integration (Beery) Differential Aptitude Tests (DAT) a * * * SA Goodenough Harris Draw-A-Person Test * * * International High School Personality Questionnaire * * SA (HSPQ) a Jung Personality Questionnaire (JPQ) a * * * SA Junior South African Individual Scales * * * SA (JSAIS) a Learning Potential Computerised Adaptive * * * SA Test (LPCAT) Most Frequently Used Tests Test name (top 14 to 25 tests) Postal Focus Indiv SA or survey Groups Interviews International Minnesota Multiphasic Personality Inventory * * International (MMPI) Myers-Briggs Type Indicator * * * International Occupational Personality Questionnaire * * * Adapted for SA Potential Index Batteries (PIB) * * * SA Rorschach cards * * * International SA Vocational Interest Inventorya * * * SA Self-Directed Search Questionnaire (SDS) a * * * SA Senior Aptitude Tests (SAT) a * * * SA Senior South African Individual Scale - * * * SA Revised (SSAIS-R) a South African Wechsler Adult Intelligence * * * SA Scale (SAWAIS) a TAT (cards) * * * International Wechsler Intelligence Scale for Children * * * International (WISC-III) 60% = SA dev or adapted. 40% = HSRC. About 25% = up to date. Top 10 Tests for different categories Couns Psychologists Psychometrists Indus Psychologists 16 PF (SA 92) 16 PF (SA 92) 16 PF (SA 92) 19 Field Int Inv 19 Field Int Inv MBTI SSAIS-R MBTI 19 Field Int Inv Bender SSAIS-R OPQ JSAIS Bender SAT HSPQ SAT SDS JPQ JSAIS JPQ TAT SDS RPM SAWAIS JPQ 15FQP+ SAT SAWAIS SAWAIS High Level battery 80% Independent Prac? 60% Survey of Training in Tests (2007) Bender 9 • Tests for RCs and Psychometrists. SAIS-R 9 • 12 responses. 16PF 7 • Min = 4, Max = 48, M = 17. SDS 7 • 73% ex-HSRC tests problematic. DAT 7 • Current test publishers have PHSF 5 R&D sections to adapt & update JSAIS 5 • Focus on screening tests Beery 5 and checklists 19 FII 5 • Focus on comprehensive HSPQ 4 training in one IQ, aptitude, CPQ 4 interest and personality test and limited exposure to ASB 4 other tests. RPM 4 • Trend – focus less on WAIS III 4 administration and more on QNST 4 interpretation and reporting. Knowledge & Skills – Schepers, 1998 Knowledge of psychometric concepts. Knowledge of test construction. Knowledge of psychological theories. Legal issues. Best practices guidelines regarding the ethical and fair use of tests. Cross-cultural testing issues [and language issues] Knowledge of the factors that impact on test scores. Skilled in test administration, scoring, interpretation and reporting - for tests appropriate for level of user and specific contexts and using different modes of test delivery (e.g., paper- & computer- based and Internet-delivered testing). Skilled in test evaluation and knowing when to test and when not to. Skilled in developmentally-focused assessment (as opposed to only diagnostically-focused) and linking assessment results to developmental/training opportunities. Skilled in application of measurement principles in variety of work- related contexts (includes policy-making & implementation). Classical Testing Tradition In the classical tradition of testing, psychological assessment practitioners had to control and have expertise in all aspects of the testing and assessment process in terms of: • Having expertise in the underlying cognitive and personality theories of the measures that they used. • Their choice of tests. • Being well-trained in how to administer the tests that they used and to supervise psychometrists and psychotechnicians who performed group or individual testing. • Being well-trained to score tests correctly, convert raw scores to norm scores, and interpret test performance with the appropriate use of information from various sources and in the light of the theoretical underpinnings of the test. • Being able to compile reports for a variety of different audiences. In view of the extensive role that assessment practitioners (psychologists and psychometrists) played in all aspects of the assessment process and the consequences for test-takers if the assessment was flawed, it was imperative that the use of psychological measures was strictly controlled. Modern Testing Era Impact of globalisation and WWW on assessment. Mobility of test users and cross-border access to materials, especially in EU. Test delivery has become internationalised, in that tests can be delivered from almost anywhere in the world to almost anywhere. This implies that individual countries & their testing regulations can no longer operate as closed systems. Testing Scenario An Italian job applicant is assessed at a test centre in France using an English language test. The test was developed in Australia by an international test developer and publisher, but is running from an ISP located in Germany. The testing is being carried out for a Dutch-based subsidiary of a US multi-national. The position the person is applying for is as a manager in the Dutch company’s Tokyo office. The report on the test results, which are held on the multi- national’s Intranet server in the US, are sent to the applicant’s potential line-manager in Japan having first been interpreted by the company’s out-sourced HR consultancy in Belgium. Questions? Which country’s standards apply? How does supplier decide on user qualification? Which country should the user be in? Who is responsible for checking cultural adaptation of test? Whose chooses the language? What norms should be used? How would the test taker seek redress for unfairness, and where? Modern Testing Era Explosion of CBT and IDT. The face of testing and assessment has changed rapidly over the past few decades. Much of this change has been brought about by the rapid advent of computer-based (CBT) and Internet-delivered (IDT) tests and testing. According to Kriek and Whitford (2006), about 300,000 Internet-delivered tests are administered per month across the world, and there is a 40% increase in IDT in South Africa over the past two years. Indeed, CBT and IDT have revolutionized all aspects of testing – from the design of tests, through to their administration (delivery), scoring, and reporting. Explosion of CBT and IDT Automated test administration, which has reduced the need for a highly trained test administrator to be present as the administrator only needs to verify the identity of the test-taker, establish rapport and settle test takers in, and take note of any untoward happenings during the session. Consequently, the role of the administrator has changed to a more supervisory one. While such supervisors need to be trained, the argument that such a person has to be a highly trained psychology professional no longer holds water. A reconceptualisation of the degree of the control and supervision required during test administration. CBT and IDT have not only resulted in a reconsideration of who supervises test administration but it has raised questions about whether a supervisor always needs to be present. Internet Test Administration modes Bartram, 2001 Anyone can access No user No human session 1. Open Insecure mode and complete the test identification or checking supervision Access is Known user is sent No human session Moderately secure controlled by login process specific login & password supervision 2. Controlled mode Access controlled Supervisor has Human session by login process, candidate login & supervision location insecure password 3. Supervised Secure Location controlled modes Maximum security Human session (e.g. Test Centre) for item content supervision 4. Managed Explosion of CBT and IDT Automated scoring and the availability of expert interpretation systems, which requires a rethink of the role of the psychologist. Computerized reports, which need to meet certain standards and should be tailored by psychologists so as to provide appropriate feedback to the relevant target audience(s). Thus, in terms of reporting, the role of the psychologist is also under scrutiny. Is main role to provide input to the development of expert systems and how to create user friendly reports? Explosion of CBT and IDT The conversion of inputs based on psychological traits into outputs based on competencies required in the work place (Bartram, 2003). By translating psychological test information into the language used in selecting and developing people in the workplace this information becomes more accessible for HR and other line managers to use. This in turn, starts the debate on how to define a “test user” and narrow definitions used in the past (e.g., when it comes to psychological tests in SA, a test user is defined as a registered psychology professional) no longer seem to be in step with who uses assessment results in the everyday world. Psychological & Competency-based assessment Psychological assessment requires expertise in psychology and psychological theories to ensure that measures of cognitive, aptitude, personality, etc. functioning are used in an ethical and fair manner, right from the choice of which tests to use through to interpretation and feedback. Furthermore, the outputs of psychological assessment are in the form of psychological traits/constructs (such as personality and ability). The expertise to perform psychological assessment is clearly embodied in an appropriately registered psychology professional. Psychological & Competency-based assessment Competency-based assessment focuses on the skills, behaviours, knowledge, and attitudes/values required in the work place or in educational settings (e.g., decision-making, critical thinking, leadership, influence). The assessment measures used are as directly linked as possible to competencies acquired in the work place or educational setting. Indirect methods such as simulations and Assessment Centres are used to conduct competency-based assessment. As the outputs of such assessment are directly linked to the language of the work place or educational settings, the test user does not need expertise in psychological theories to be able to apply the results of competency-based assessments. What is required, however is that competency-based assessment needs to be performed by people with expertise in this area of assessment (e.g., skilled in competency-based interviews). Should traditional Psychology Departments broaden who they train? For example, offer an advanced diploma in competency- based assessments for managers, HR practitioners, etc.? Grey Area In accordance with the simple definitions provided above, the obvious rule of thumb to use is that psychological assessment should only be performed by appropriately registered psychology professionals and competency-based assessment could be performed by “non-psychologists”. Grey Area: In view of the online management of the assessment process, it is possible to use psychological tests, or even the selection of item sets from various psychological tests that tap the required competencies to be measured and to then produce test output that is solely in terms of competencies and not psychological traits and constructs. In this instance, the “online test user may never see the test in any physical form, will never have to score it, look up norms, or carry out an interpretation” (Bartram, 2003, p. 4). According to Bartram (2003), if the report that is generated focuses solely on work place competencies, then there is no reason why a non-psychologist cannot use it and the non-psychologist should not be required to undergo training in the psychological test(s) used to generate the output (report). Psychology professionals role in assisting in application of results w.r.t. implications for selection and/or training. Reflection Our training programmes may not be in step with the modern testing era. Only 25% of tests on Top 25 list of frequently used tests (HSRC survey) are CBT or have a CBT version. Only 25% of the tests included for RC and psychometrist training are CBT. Not enough of the tests that are currently undergoing the most adaptation and development are included in the list (PsyTech, SHL, Jopie van Rooyen). Are we preparing students for the world of CBT & IDT and the changing roles of psychology professionals? Or are we training them in the classical model? Classic versus modern How should we be teaching & training? Outcomes-based education in SA. Internationally there has been a move to a competency- based approach to training in psychological testing and assessment. We can learn from this and adapt the model for our context EFPA-EAWOP Task Force on Test User Qualification Aims to: • define standards of competence in test use • facilitate the recognition of qualifications that meet those standards • assist countries in developing such qualifications • audit proposed qualifications to see if they are in line with the EFPA- EAWOP standards. Consider Knowledge and Skills in relation to: • General – common across all areas of testing • Domain related – related to general area of application • Specific–instrument related Three broad domains: Clinical, Educational and Work • For example, Work qualifications could be divided into three sub- domains: - Selection and recruitment - Individual development - Organizational development Standards structure: Initial content input from BPS standards and ITC Guidelines Standards of Evidence Learning competence requirements specifications Performance Rules of evidence Knowledge and skill requirements set by and assessment requirements EFPA-EAWOP with specifications defined by EFPA- consultation. defined by EFPA- EAWOP EAWOP. Context defines area Learning input and range of Methods of methods locally application assessing defined competence left to local decision How should we be teaching & training? Need to develop a competency-based model nationally. The Board is our SGB and quality assurance body. Some pointers: • Get clarity about the roles and competencies of different categories/levels of psychology professionals. • Contemplate training RCs and psychometrists in a problem-based way, e.g., training them in specific batteries of tests for specific contexts/client needs (e.g., school readiness screening, screening for depression) – instead of providing them with generalist assessment training. • Emphasis needs to be on what competence must be demonstrated in and the evidence of such competence, instead of the inputs and the specific tests trained in. But need to decide on the scope of measures in which competence should be demonstrated. • Achieve a balance between developing knowledge competencies, applied (skill) competencies and attitudinal competencies (values, ethics). • Do not isolate training in testing and assessment from other areas of psychology (theories and intervention). Who should be teaching? Junior versus senior staff? Eyde and Childs (1998) surveyed the qualifications of those who teach assessment courses in the USA and found that senior faculty tended to teach assessment courses and that 76% of them had doctorates. Situation in SA – senior staff are involved at PG (and especially masters level) but less experienced staff are sometimes involved at an UG and 4th year level. From a QA perspective, should we not be setting minimum qualification levels and levels of measurement & assessment knowledge and applied experience needed? Use of people in the field as guest lecturers? Closing Thoughts Questions and reflection have value as they move us out of our comfort zones. But now action is necessary to move our testing and assessment teaching training from the classical tradition to the modern era. How? Establish a task force under the leadership of the Education/Psychometrics Committee of the Professional Board, or both?