VIEWS: 13 PAGES: 69 POSTED ON: 5/16/2012
Schedule Today and Wednesday: Lecture Monday, 4/11: Exam 1 For many years, tests have been classified as either achievement tests or aptitude tests Definitions of “achievement” and “aptitude” Achievement The act of accomplishing or finishing something successfully, especially by means of exertion, skill, practice, or perseverance Aptitude A natural or acquired talent or ability or inclination; Quickness in learning and understanding - intelligence Mind-body dualism typical of traditional testing (skipping SOs 1&2, straightforward; in this material, GF&B argue that the terms “achievement test” and “aptitude test” are inappropriate and should be replaced with the term “ability test” - and I agree with them; excellent material) 2 Achievement tests (supposedly) measure What a person learned as a result of a specific structured educational/training experience/course Scores are interpreted to be a measure of how much an individual knows as a result of the education or training English grammar, math, science, social studies, etc. These are the types of tests used in grade school and high school to measure student learning/proficiency ▪ In Michigan, MEAP tests: Michigan Educational Assessment Program (based on the definitions and distinction between achievement and aptitude, then..) 3 Aptitude tests (supposedly) measure Accumulation of learning from a number of diverse and usually informal learning experiences Although not emphasized by GFB, genetic implication as well ▪ You have artistic ability or you don’t ▪ You have mechanical aptitude or you don’t ▪ Women don’t have an aptitude for math ▪ Men don’t have good spatial aptitude Said to measure potential to learn, or the potential to develop new skills and acquire new knowledge Intelligence tests, SATs, GREs, Mechanical Aptitude, Artistic Aptitude These are the tests that you are told you can’t study for (hog wash - most people don’t say that any more) 4 All tests measure what a person has learned up to the time he or she takes the test and that is the only thing a test can measure They cannot and do not measure innate or unlearned potential Thus, the distinction between achievement tests and aptitude tests is arbitrary and We should use the term “ability tests” for both types of tests Ability in the sense of competence or proficiency, regardless of how you have acquired the ability/skill 5 Notice that GFB are not denying that “innate” potential exists - but that really is their position (or I should say, they believe most, if not all of what is measured on tests is learned; which is exactly the position we should take). Look at their use of quotes: “large degree”. Also, “innate” is in quotes but unlearned is not - that is a subtle, but important distinction they are making with their use of quotation marks. They have crafted this argument based on objections/conversations with traditional testing folk. (aptitude measuring genetic/innate potential; pretty much direct quotes from G&F, including quotation marks) 6 Tests can and do measure the prerequisites that are necessary for further learning in an specified area, and thus can predict future learning/performance If students do not do well in PSY 3600, Concepts and Principles of Behavior Analysis, they cannot do well in PSY 4600, Survey of Behavior Analysis Research, thus a student’s grade in PSY 3600 can predict his or her performance in PSY 4600 You can’t balance an equation in chemistry unless you know algebra, thus a test of algebra can predict performance in a chemistry class (not in text, but important to understand) 7 Mental ability tests were at the center of early critical Supreme Court decisions regarding unfair discrimination Thus, many companies stopped using them However, there is a lot of research in selection that indicates that mental ability tests are related to almost all jobs Validity correlations are often quite high, and higher than correlations of other tests Many companies are now using them again Remember, however, if you use one of these, you must conduct an empirical validity study (or use validity generalization) (as a behavior analyst, I still have trouble with the term “mental” ability since it still implies mind-body dualism; I’m more comfortable, but not completely with “cognitive” ability; but haven’t been able to come up with anything different that and certainly like those terms better than “intelligence” tests) 8 A rose is not a rose is not a rose A mental ability test is not a mental ability test is not a mental ability test Mental ability tests measure a collection of abilities - a learned repertoire Verbal, math, memory, and reasoning abilities 14 different abilities are often measured in some combination by mental ability tests (next slide) Different mental ability tests often measure a different set of these abilities Thus a person may score differently on different tests of mental ability (FE: main abilities include some form of verbal, math, memory, and reasoning abilities) 9 Memory span Figure classification Numerical fluency Spatial orientation Verbal comprehension Visualization Conceptual classification Intuitive Reasoning Semantic relations Ordering General reasoning Figure identification Conceptual foresight Logical evaluation and deduction (that is why if you use the PAQ you must take great care in selecting tests that are similar to the GATB tests that are recommended) 10 The term mental ability makes it explicit that these tests measure various cognitive abilities of the applicant (and not some innate, unlearned, hypothetical construct called “intelligence”) These cognitive abilities are most directly identified by the what is measured (some combination of the 14 abilities listed earlier) and from the content of the items themselves They should be thought of the same way the other abilities discussed in the book are thought of e.g., mechanical ability, clerical ability In other words, they are disputing the traditional view that there is something called “intelligence” (the answer is similar to SO7) 11 I am going to show you some examples of mental ability tests at the end of class, just to “de-mystify” them a bit The authors describe the Wonderlic Personnel Test which is probably the most popular Given to all players at the NFL Scouting Combine and scores are reported to NFL teams before the annual draft For a moment, look at items in the text that are similar to the ones on the Wonderlic Personnel Test 12 1. Which of the following months has 30 days? (a) February (b) June (c) August (d) December 2. Alone is the opposite of: (a) happy (b) together (c) single (d) joyful 3. Which is the next number in this series: 1, 4, 16, 4, 16, 64, 16, 64, 256, (a) 4 (b) 16 (c) 64 (d) 1024 (Two slides - Note: all six items are different types of items: general knowledge, opposites (verbal comprehension and vocabulary), numerical reasoning and ordering) 13 4. Twilight is to dawn as autumn is to: (a) winter (b) spring (c) hot (d) cold 5. If Bob can outrun Junior by 2 feet in every 5 yards of a race, how much ahead will Bob be at 45 yards? (a) 5 yards (b) 6 yards (c) 10 feet (d) 90 feet 6. The two words relevant and immaterial mean: (a) the same (b) the opposite (c) neither same nor opposite (again, notice the type of questions: semantic or verbal reasoning, numerical fluency/reasoning, verbal comprehension - opposites) 14 What have the validity studies uniformly concluded? Mental ability tests are among the most valid of all selection instruments (next slide on validity of mental ability tests as well) 15 Differences in the actual tasks that a person performs as part of a job have very little effect on the magnitude of the validity coefficients for mental ability tests In other words, mental ability tests are valid predictors of performance for a wide variety of jobs 16 They have repeatedly been shown to have adverse impact on protected classes, particularly blacks and hispanics This led to the notion that these types of test might have differential validity - next 17 11A: What is meant by differential validity? Notion/hypothesis that tests are less valid for minority groups than for nonminorities ▪ That is, a test may be significantly more valid for whites than for blacks ▪ Term is related to test bias regarding ability tests, particularly mental ability tests ▪ This claim is made over and over again with respect to SATs and GREs - that those tests are more predictive of the performance of white students than they are of the performance of minority students (extremely important; and mentioned often in selection as well as admissions to colleges and universities,- and is still very controversial) 18 The argument is that the content of ability tests is based on content/items related to the white middle-class (e.g., vocabulary and grammar), and thus the scores of the minorities are lower than what they should be 19 The data are very clear about this issue Differential validity does not exist • That is, tests are equally valid for whites and other ethnic/racial groups • It makes sense – Verbal comprehension skills are verbal comprehension skills – Verbal reasoning skills are verbal reasoning skills – Math skills are math skills, etc. • Thus if any of these skills are required by the job, they should be “equally required” by whites and members of other ethnic/racial groups 20 Meta-analyses have been consistent – there are significant differences in mean test scores among racial/ethnic groups Ranking: Asians whites Hispanics blacks 21 Cognitive ability tests have a high correlation with job performance and academic performance They have a disproportionate impact on Hispanics and blacks Often result in adverse impact as legally defined when used for selection (important, difficult issue arises) 22 13A: Adverse impact, however, does not mean that unfair discrimination has occurred; if the tests are job related then fair discrimination has occurred 13B: Two things that make a defense against adverse impact likely: They are usually valid – and are among the most valid and least expensive tests Differential validity does not exist The result of that is that a large percentage of the jobs are likely to be filled with whites and much smaller percentage by Hispanics and blacks 23 It is not appropriate to conclude from these studies that differences are due to genetic differences educational differences cultural differences Studies do not address the reasons (the authors want to caution any one making any general conclusions as to why differences exist; particular concern about race-based genetic arguments as advanced in the Bell Curve, one of the most recent books/articles on this issue) 24 Cognitive ability tests measure verbal fluency, math, spatial relations, reasoning skills, and memory These are skills taught in schools These skills also, particularly verbal fluency and math, have a high correlation with socio-economic status I firmly believe that if we are able to improve the quality of education that all of our country’s children receive, then we would eliminate most, if not all, of the differences we are seeing in the cognitive abilities of various demographic groups As a society, that needs to be one of our top, if not our top, priority (clearly, NFE) 25 Cognitive ability tests are among the most valid tests for a large number of jobs (and some selection specialists would say for all jobs) Evidence also indicates they result in mean score differences for demographic groups and thus adverse impact is highly likely (I am skipping to SO17 for the moment, I’ll return to 15) 26 Because they are so valid, some selection specialists believe cognitive ability tests should be used extensively in selection Some, however, have expressed deep reservation about using them because of the social implications of the disqualification of larger proportions of minorities (very nice discussion of this in text; directly quoting GFB here; cont. on next slide) 27 To some extent, the decision may reflect the values/goals of the organization If goal is to maximize individual performance with minimal cost, cognitive ability tests will do this If the organization has multiple goals of sustaining high performance while maintaining a broad representation of minorities, then it would be better to limit the use of cognitive ability tests and use other, generally more expensive and almost equally valid instruments ▪ biodata inventories (I don’t like these as you will see next unit) ▪ structured interviews ▪ work sample tests ▪ assessment centers (that’s the rub - the expense of those other instruments) 28 If an organization has diversity as a selection goal and wants to use cognitive ability tests because of their validity and the fact that other options are much more expensive, what is the main/best option? Vigorous recruitment of minority applicants (now back to SO15: remember race norming is not legal; often a problem because selection specialists are typically not the ones who are responsible for recruitment –selection specialists really need to work with the HR staff) 29 The authors describe several very popular tests Refer to this material if you are ever looking for tests in these categories I am not going to have you learn anything specific about these tests 30 Physical ability tests do have high validity (not surprising if the job is physically demanding), but of course you have to be particularly concerned legally about: Females, disabled, and older workers Tests must be clearly linked to job tasks that require those abilities 31 Height and weight requirements have often been challenged in court Adverse impact on females and Asians The courts have rarely let them stand The rationale for using these measures is that they are substitute measures for strength But courts have consistently held that if strength is the job requirement, then it should be measured directly (physical ability test) (a lot of organizations in the past; police and fire) 32 The data and information on personality tests is difficult For many years, companies used personality tests that were developed by clinical psychologists, and some of those tests are still popular and being used by organizations One is the California Personality Inventory Have not had good validity historically In prior editions of the book, GFB advised against their use But, there is some good work going on right now, however, the field is in a bit of flux right now Intuitively we know that “personality” influences how effective a person is at work, we just haven’t tapped into what the relevant KSAs really are, or what the relevant clusters of behaviors are 33 There has been some work on what has been called the “Big Five” personality dimensions or traits that are being assessed in work environments rather than taken from earlier clinical tests Extroversion ▪ Being sociable, gregarious, assertive, talkative, and active Agreeableness ▪ Being courteous, flexible, trusting, good natured, cooperative, forgiving, softhearted, and tolerant Conscientiousness ▪ Being responsible, organized, dependable, planful, willing to achieve, and persevering Emotional stability (only one described in negative terms) ▪ Being emotional, tense, insecure, nervous, excitable, apprehensive, and easily upset Openness to experience (also called intellect or culture) ▪ Being imaginative, cultured, curious, intelligent, artistically sensitive, original and broad minded (we will see where it goes - we are empiricists) 34 I am not going to discuss the ways that personality has been measured for selection, but I want to deal with the Meyers Briggs (self report inventory) ▪ It is very popular in business and industry for leadership training, work group development and career counseling (it is based on psychological types developed by Carl Jung) ▪ It is completely useless for selection ▪ Very low validity in studies ▪ Test-retest reliability coefficients range from .69 to .77 which partially accounts for the low validities (remember .85 is considered to be the minimum acceptable) (Leslie Braksick mentioned this when she was here; friend in the business college who swore by it, DBA from Harvard; Very popular in the Public Administration Dept. as well Brethower story, 9 years, Center for HR development) 35 “Personality should be a worker characteristic related to performance in many jobs.” However, validity data have not been high That may be because the methods of personality assessment in selection often are inadequate Use with caution!!! (needless to say, this is not a strong endorsement of using personality tests) 36 If you do use a personality test, you must use a criterion- related validity study to support it because personality traits cannot be directly observed Concurrent validity Predictive validity Validity generalization (in other words you cannot use content validity: also have some legal issues to be aware of) 37 ADA (dealt with this previously in U3) If a test can and is used to diagnose mental/psychiatric disorders, then it will probably be considered a medical examination under ADA If it deals with other personality traits (the Big 5, for example) then it probably will not be considered a medical examination Nonetheless, my strong advice to you is to treat every personality test as a medical examination until things are clarified more by the courts Which means you should only administer personality tests post- offer and keep the results in a file that is separate from the personnel file 38 Clarifying court case, 2005, 7th Circuit Court MMPI is a medical examination and thus illegal for pre- employment use (certainly that was expected) Psychological tests that measure personal traits such as honesty, integrity, preferences and habits do not constitute medical examinations 39 Right to privacy (be able to explain this as well) Although a right to privacy is not explicitly guaranteed under the US Constitution, individuals are protected from unreasonable intrusions and surveillance Personality tests, by their nature, reveal an individual’s thoughts and feelings Several states have laws that explicitly guarantee a right to privacy ▪ To date, litigation has occurred about questions relating to sexual inclinations and orientation, and religious views (second thorny issue) 40 Soroka v. Dayton Hudson (1991) California Court of Appeals stopped Dayton Hudson’s Target stores from requiring applicants for store security positions to take a personality test that contained questions about sexual practices and religious beliefs The court also stated that employers must restrict psychological testing to job-related questions The ruling was later dismissed because the parties reached a court- approved settlement ▪ Dayton-Hudson agreed to stop using the personality test ▪ Divided $1.3 million dollars among the estimated 2,500 members of the plaintiff class who had taken the test 41 Performance or work sample tests are excellent and I highly recommend their use when you can do them Typing test Having candidates write a computer program to solve a specific problem Role playing a sales situation with an applicant for a sales position Having mechanics trouble shoot a problem with an engine You are getting an actual sample of behavior under controlled testing conditions (which permits you to easily compare performance across applicants) (this slide NFE) 42 From a technical perspective, they have Very high validity, and often much higher than pencil and paper tests designed to measure the same KSAs Show less adverse impact that pencil and paper tests They have some disadvantages, which I will get to in a moment, but They reduce two limitations of other selection procedures, and both are related to verbal behavior Most selection procedures rely heavily on verbal behavior Written answers to questions (ability tests) Oral descriptions of abilities/skills (interviews, training and evaluation assessments) (This slide NFE) 43 Willful distortion and faking (people want to look good) This varies dependent upon the selection procedure ▪ Reports about past experiences (interviews, T&Es) where the information is difficult to confirm - most susceptible ▪ Personality and honesty inventories, next susceptible ▪ Ability tests, least susceptible 44 Relationship between verbal behavior and actual behavior is not perfect (as we behavior analysts well know) Much of our behavior is contingency-shaped, not rule-governed This is particularly a problem for exemplar performers who are not verbally fluent ▪ Automobile mechanic ▪ Plumbers ▪ Machine operator It can also be a problem for employees who are exemplar performers but can’t describe what makes them exemplary performers – sales representatives 45 Many consulting firms use stress interviews as part of their selection process Stress interviews Interviewer creates a stressful situation, often by asking many questions rapidly, not allowing much time for the applicant to respond, interrupting the applicant frequently, acting in a semi-hostile manner, or in a cool aloof manner 46 SO26: Why aren’t stress interviews representative of actual job activities? Even if the job is one of high work demands that produce stress, rarely is the situation staged in the interview representative of the actual work demands that produce the stress In very few jobs, is the stress related to a semi-hostile or cool/aloof stranger rapidly firing questions The behavior of the applicant doesn’t readily generalize to the job and thus should not be used as a predictor (press secretary, perhaps; next slide also) 47 This situation illustrates one of the main limitations of performance tests… Difficulty of accurately simulating complex, multiple-demand, types of tasks Also sometimes difficult to construct a representative test that does not involve skills/knowledge learned on the job General sales skills OK, but questions that deal with specific company-related products and pricing may not be 48 Cost They tend to be very expensive to develop Equipment and materials ▪ Do-nothing machine that tests the mechanical aptitude of mechanics ▪ Program it for many common problems that mechanics encounter on assembly lines Often one-on-one or small group administration is required Development of test and scoring 49 Validity They tend to have very good validity ▪ Often equal to or greater than mental ability tests Adverse impact Many studies have found they have less adverse impact than mental ability and pencil and paper tests 50 Assessment centers or even the use of some of the exercises often included in assessment centers have been highly successful In-basket tests Leaderless group interaction tests Case analyses Main problem is their time and expense to both develop and administer 51 Refer you to the Minnich & Komaki article in U7 in the course pack from the OBM Network News The article describes the use of a validated in-basket test to assess the effectiveness of managers based on Komaki’s Operant Supervisory Taxonomy and Index This is one of the best examples I have ever seen of the intersection of behavior analysis and traditional I/O Psychology Operant supervisory taxonomy and index Assessed the difference between high performing and low performing managers Found that work sampling and type of consequence following performance distinguished between high and low performing managers (Gives a detailed description of the instrument, some of the actual items, and responses, along with analysis of responses I haven’t asked her whether it is commercially available) 52 During the introduction to the course, I provided some information about graphology Kalamazoo Gazette Headline Dow uses handwriting to determine aptitude Used as a selection tool in/by (very popular in Europe): ▪ 2,500 US companies ▪ 68% of Swiss companies ▪ 50% of French companies ▪ 80% of French selection consultants ▪ 80% of Western European countries I am appalled, as are the authors, that a section on graphology has to be included in a legitimate text on personnel selection and placement (couldn’t resist including this; this slide NFE) 53 Graphology has no validity whatsoever as a selection tool or as GFB state, “it flat out doesn’t work.” Neter & Ben-Shakhar conducted a meta-analysis of 17 graphology studies: one of my personal favorites ▪ Trained graphologists were no better than nongraphologists in predicting performance based on hand writing samples ▪ Psychologists with no knowledge of graphology outperformed graphologists on all dimensions! 54 (NFE) Just for fun, look at Table 15.4 Gatewood sent a handwriting sample to a graphologist who graduated from the program conducted by the International Graphoanalysis Society Four times (for each edition of the book), they calculated reliability (same graphologist) and reported the results with commentary by GBF Read pages 662-664 (love the way the authors handle this - humor and irony) 55 For all practical purposes, it is illegal Federal law, Employee Polygraph Protection Act of 1988 It can be used in some specific employment situations for selection (there are other requirements for use with current employees) ▪ Private employers whose primary business purpose is to provide security services (e.g., protection of nuclear power facilities, public water supply facilities, shipments or storage of radioactive or other toxic waste materials, public transportation) ▪ Employers involved in the manufacture, distribution, or dispensing of controlled substances ▪ Federal, state and local government employers; also private consultants or experts under contract to governmental depts. and agencies (e.g., Defense Dept., Energy Dept., National Security Agency, CIA, FBI) (spies and spooks) 56 Frequency of false positives; that is, there is a high degree of error with respect to finding that an individual is lying when in fact, the individual is not (details below, NFE) Assume 90 percent accuracy (high end estimate) Assume rate of stealing is 5% of the working population If 1,000 polygraphs were given, we would expect 50 individuals would be lying, and given 90% accuracy, 45 of those would be detected However, the problem lies with the other 950 individuals 95 (950 X .10) would be identified as lying when they had not Thus, 140 individuals would be identified as having lied, with 68% of them being false positives Not good (text actually gives 3, I am asking you to learn the major one; (a) other reactions than guilt can trigger an emotional response; (b) there are countermeasures that can be used to avoid detection - I am sure you can find them on the web) 57 There are two basic types of paper and pencil integrity tests Overt integrity tests ▪ Self-report inventories that measure a job applicant’s “attitudes” and “cognitions” toward theft that might predispose him/her to steal at work Personality-based measures ▪ Self-report inventories that measure integrity as part of a larger syndrome of antisocial behavior or organizational delinquency and thus not only measure theft but things like drug and alcohol abuse, vandalism, sabotage, assaultive actions, insubordination, absenteeism, excessive grievances, bogus worker compensation claims and violence (this slide NFE) 58 Pencil and paper integrity tests were developed to replace polygraph testing after the Employee Polygraph Protection Act was passed in 1988 A few states have passed laws against the use of these tests as well, so be careful and check the state laws Once again the reason for concern is the high number of false positives that occur Because of the concern about theft by employers, the use of integrity tests is on the rise and thus more validity studies have been conducted recently (this slide NFE) 59 These tests indicate that these measures do correlate with measures of theft, detrimental behaviors (grievances filed, absenteeism, disciplinary actions, etc.), and overall job performance They appear to be OK to use in a selection program, however, at the current time, many still oppose their use False positives and the social implications of that – how would you like to be identified as a liar and cheater when you were not? Frequency of false positives is unknown (and even if it were, how would you like to be a statistic?) 60 Use of drugs and alcohol have been a major concern since the 1960s (casting dispersions on my generation, the hippie generation) NFE, but paper and pencil drug tests - see items, p. 654,1 No public studies that have evaluated either the reliability of validity of these tests! In one court case, the court ruled that these were illegal based on the Fifth Amendment’s prohibition against involuntary self- incrimination (I find it hard to believe anyone would answer these types of questions honestly!) 61 The legal status of drug testing is unclear, particularly given the new medical marijuana laws Organizations face less risk using drug testing for pre- employment selection (testing individuals who are applying for a job) They face considerably more risk if they test existing workers for promotions (or transfers) or testing workers to detect drug users for disciplinary or counseling purposes NFE, but why? Applicants cannot take advantage of collective bargaining or challenge employment at-will principles, as can employees who feel they have been wrongly treated 62 Basically, Do NOT institute drug testing until you consult with a very, very good employment lawyer 63 Questions??? 64 Drug testing is not considered a medical test under ADA You can administer a drug test before an offer is made Why? Those using illegal drugs are excluded from coverage under ADA. Thus, while many would consider drug testing a medical test, it is not considered a medical test under ADA 65 15 states and the DC have passed medical marijuana laws If a person has a disability and uses medical marijuana, what about drug testing? Many laws protect employers with clauses like “employers are not required to accommodate the medical use of marijuana in any workplace.” However, laws are varied and there have not yet been many cases 66 California Supreme Court, 2008 OK to fire a worker after drug test Employers are under no obligation to accommodate medical marijuana on or off the job The law protects the individual from criminal prosecution but provides no protection on the job Why? Marijuana remains classified as an illegal substance under federal law (I dealt with this previously, but want you to learn this point now; so drug test away) 67 Agreed to review a case in which a customer service consultant was “fired” (not hired) for her legal, at-home use of marijuana Applicant disclosed her use during the hiring process Gave the company a copy of her physician’s authorization Was not hired after a pre-employment drug screen when she tested positive for THC. 68 Don’t think so, but who knows? (no one knows where this is going) 69
"PowerPoint Presentation - Alyce Dickinson PhD.ppt"