BASIC PRINCIPLES OF ASSESSMENT 1 Clarify the purpose What is the assessment for? 2 Define what is to be tested What knowledge is needed to be assessed? 3 Select appropriate test methods Find an appropriate assessment method to fit the purpose you have in mind What is ‘appropriate’? 4 Address practical and technical issues of administration and scoring Ensure reliability Ensure validity Scoring 5 Set standards for performance Determine type of standard Choose standard setting method The following is based on a UMAP presentation given to University of Bristol, February 5th 2004 by Professor David Newble. David Newble is Professor of Medical Education at University of Sheffield, one of the five founding UMAP partner schools. Basic Principles of Assessment 1 Clarify the purpose What is the assessment for? Feedback i.e. for formative purposes Measure progress i.e. to track individual or cohort improvement Ranking or grading students i.e. by norm or criterion referencing Quality control i.e. to assess students against a standard set internally or externally Evaluation of teaching or curriculum i.e. feedback to tutors and programme coordinators Assessments should be designed with a single purpose in mind. If assessments are developed for more than one purpose their effectiveness can be reduced. For example, an assessment constructed to deliver feedback for formative purposes which is also used for quality control will be confounded as while the exam promises to provide opportunities for learning on the areas assessed, it simultaneously delivers a critical judgement against a required standard of performance. If more than one purpose exists, separate assessments should be considered. 2 Define what is to be tested What knowledge is needed to be assessed? Create a blueprint - define the range of competences - build a grid or ‘blueprint’ - decide what weightings to give the different cells in your blueprint (i.e. greater weighting could mean greater numbers of questions or greater numbers of marks assigned) 3 Select appropriate test methods Find an appropriate assessment method to fit the purpose you have in mind Be aware there are a range of assessment methods with various strengths and limitations - MCQ, EMQ, short answer essay questions, modified essay, OSCE, projects and reports, portfolios, log books are just a few - be prepared to accept that you may need a range of assessments in order to meet your various purposes You must allow purpose to drive choice! Basic Principles of Assessment What is ‘appropriate’? Let us consider a written assessment and evaluate one possible method, the true-false examination. o Advantages of true-false examinations Popular with question writers Easy to construct Evidence of reliability if enough items are used o Disadvantages of true-false examinations Not so easy to construct unambiguous items Does not allow for wide sampling as many subject areas do not lend themselves to ‘black and white’ statements (low content validity) Restricts to testing of factual recall rather than applied knowledge (little diagnostic assessment possible) High chance of false positives which cannot be reliably corrected for On this basis, only very few purposes for setting a written assessment would be met. Because of these factors UMAP do not write true-false items for the UMAP bank. UMAP writes one-best-answer-from-five MCQs and one-best-answer EMQs as these two item types have greater evidence of both validity and reliability, therefore meeting a larger number of purposes of assessment. 4 Address practical and technical issues of administration and scoring Ensure reliability Inter-rater comparison Reproducability (depends on sample size) Ensure validity Method of scoring performance as accurate as possible, removing any marker-bias - Rating forms - Check lists - Multiple answer options Scoring Consider weightings Methods of combining elements of an exam to produce a score Scoring keys Basic Principles of Assessment 5 Set standards for performance Determine type of standard Relative (norm referenced) Absolute (criterion referenced) Choose standard setting method Written exams - Anghoff - Ebel - Nedelsky - Hofstee Clinical exams - Anghoff - Borderline - Contrasting groups No test is 100% reliable or valid, therefore there is always a measure of error around the standard cut point. Degree of error can be calculated as follows: Standard error of measurement: SEM = SD 1-R R = alpha or another relevant statistic.