Docstoc

BASIC PRINCIPLES OF ASSESSMENT

Document Sample
BASIC PRINCIPLES OF ASSESSMENT Powered By Docstoc
					             BASIC PRINCIPLES OF ASSESSMENT


1 Clarify the purpose
   What is the assessment for?

2 Define what is to be tested
   What knowledge is needed to be assessed?

3 Select appropriate test methods
   Find an appropriate assessment method to fit the purpose you have in mind
   What is ‘appropriate’?

4 Address practical and technical issues of administration and scoring
   Ensure reliability
   Ensure validity
   Scoring

5 Set standards for performance
   Determine type of standard
   Choose standard setting method




The following is based on a UMAP presentation given to University of Bristol, February
5th 2004 by Professor David Newble. David Newble is Professor of Medical Education at
University of Sheffield, one of the five founding UMAP partner schools.
                                                               Basic Principles of Assessment


1 Clarify the purpose

   What is the assessment for?

      Feedback
        i.e. for formative purposes
      Measure progress
        i.e. to track individual or cohort improvement
      Ranking or grading students
        i.e. by norm or criterion referencing
      Quality control
        i.e. to assess students against a standard set internally or externally
      Evaluation of teaching or curriculum
        i.e. feedback to tutors and programme coordinators

   Assessments should be designed with a single purpose in mind. If assessments are
   developed for more than one purpose their effectiveness can be reduced. For
   example, an assessment constructed to deliver feedback for formative purposes
   which is also used for quality control will be confounded as while the exam promises
   to provide opportunities for learning on the areas assessed, it simultaneously
   delivers a critical judgement against a required standard of performance.

   If more than one purpose exists, separate assessments should be considered.

2 Define what is to be tested

   What knowledge is needed to be assessed?

           Create a blueprint
             - define the range of competences
             - build a grid or ‘blueprint’
             - decide what weightings to give the different cells in your blueprint
             (i.e. greater weighting could mean greater numbers of questions or greater
             numbers of marks assigned)

3 Select appropriate test methods

   Find an appropriate assessment method to fit the purpose you have in mind

         Be aware there are a range of assessment methods with various strengths
           and limitations
           - MCQ, EMQ, short answer essay questions, modified essay, OSCE, projects
           and reports, portfolios, log books are just a few
           - be prepared to accept that you may need a range of assessments in order
           to meet your various purposes
         You must allow purpose to drive choice!
                                                            Basic Principles of Assessment


   What is ‘appropriate’?

         Let us consider a written assessment and evaluate one possible method, the
           true-false examination.

              o   Advantages of true-false examinations
                      Popular with question writers
                      Easy to construct
                      Evidence of reliability if enough items are used

              o   Disadvantages of true-false examinations
                       Not so easy to construct unambiguous items
                       Does not allow for wide sampling as many subject areas do
                         not lend themselves to ‘black and white’ statements
                         (low content validity)
                       Restricts to testing of factual recall rather than applied
                         knowledge
                         (little diagnostic assessment possible)
                       High chance of false positives which cannot be reliably
                         corrected for

       On this basis, only very few purposes for setting a written assessment would
         be met. Because of these factors UMAP do not write true-false items for the
         UMAP bank.
       UMAP writes one-best-answer-from-five MCQs and one-best-answer EMQs as
         these two item types have greater evidence of both validity and reliability,
         therefore meeting a larger number of purposes of assessment.

4 Address practical and technical issues of administration and scoring

      Ensure reliability

         Inter-rater comparison
         Reproducability (depends on sample size)

      Ensure validity

         Method of scoring performance as accurate as possible, removing any
           marker-bias
           - Rating forms
           - Check lists
           - Multiple answer options

      Scoring

         Consider weightings
         Methods of combining elements of an exam to produce a score
         Scoring keys
                                                            Basic Principles of Assessment




5 Set standards for performance

      Determine type of standard

         Relative (norm referenced)
         Absolute (criterion referenced)

      Choose standard setting method

         Written exams
           - Anghoff
           - Ebel
           - Nedelsky
           - Hofstee

         Clinical exams
           - Anghoff
           - Borderline
           - Contrasting groups

      No test is 100% reliable or valid, therefore there is always a measure of error
      around the standard cut point. Degree of error can be calculated as follows:

      Standard error of measurement:

           SEM = SD     1-R

      R = alpha or another relevant statistic.

				
DOCUMENT INFO
Shared By:
Categories:
Stats:
views:81
posted:3/9/2010
language:English
pages:4
Description: BASIC PRINCIPLES OF ASSESSMENT