MeritTrac follows a standard methodology to arrive at a Cut (passing) Score. A passing score is the minimum score a test taker should requires to score in a test in order to be eligible to go to the next level, in a hiring scenario. The methods of setting the standards are based on the concept of considering responses from the "borderline" test taker. This test taker is someone whose knowledge and skills are on the borderline area between the upper group and the lower group. These methods are based on the idea that, once the test takers, who belong to the upper group actually start scoring higher than those who belong to the lower group, the passing score should be that which is expected from a person whose skills are just on the borderline. The judgments these methods require are made in terms of the specific questions on the test. Judges: Each team has to identify judges who have the capability to make this educated approximation. A minimum of 3 to be identified for this activity – the more the number of judges, the better would it be for the process. Nedelsky's Method: This method, suggested by Leo Nedelsky in 1954, can be used only for multiple-choice tests, since it requires a judgment about each possible wrong option. The judge's task is to look at the question and identify the options that a borderline test taker would be able to recognize as wrong. For example: a) Who is the Chief of Microsoft? (i) Steve Jobs (ii) Richard Branson (iii) Michael Dell (iv) Bill Gates A judge might decide that the borderline test-taker would be able to eliminate wrong options (i). But the judge might decide that the choice between wrong answer (ii) and (iii) is too difficult for the borderline test taker. In this case the number of answer options that a borderline test taker would not be able to eliminate is 2. For a particular test, the judges have to provide information for each question in the test. Then we can compute the score without correcting for guessing as shown below: Answer Options ABCD ABCD ABCD Total Answer Key an option the borderline test taker would eliminate Number of answers options not eliminated 2 3 1 Expected Score 1/2 = 0.5 1/3 = 0.33 1/1 = 1 1.83
Questions 1 2 3
Rules: a) According to Nedlesky's method, the borderline test taker's expected score for any question is 1 divided by the number of answer options the test takers has to guess (answer) from. b) To find the borderline test taker's expected score for the whole test, add up that test taker's expected scores for all the individual questions (for the entire test) The expected score derived by this method is the score that is typical for a borderline test taker's performance. If you choose this score as the cut (passing) score, a borderline test taker should have a 50 percent chance of passing the test. Therefore, in a fairly large group of borderline test-takers, about half would pass the test and half would fail. Angoff's Method: This method, suggested by Angoff is similar to Nedelsky's method, but it can be used with tests that are not purely MCQs by nature. In this method, the passing score is computed from the expected scores of the individual questions, as in Nedelsky's method. However, the judge has to consider each question as a whole and makes a judgment of the probability that a borderline test taker would answer the question correctly. Similar to the Nedelsky's method, passing score is computed by adding up the probabilities for all the questions in the test. Questions 1 2 3 Total Probability of the correct answer 0.7 0.2 0.8 1.7
We have one more method that can be considered to decide on setting Cut – Scores: Ebel's method - which is a two-stage procedure. I have not detailed it because of its complexity. To run an exercise following the two methods listed above let us have a look at the steps to be followed: a) Identify a Standard Test. b) Nominate Judges from your team. c) Record the Judges inputs based on any of the two methods mentioned above d) Compute the Cut Score.