Standard Setting

							   Performance-based
standard setting methods

     Katharine Boursicot
       Trudie Roberts
    Performance-based standard
          setting methods
   Borderline group method

   Contrasting group method

   Regression based standard method
          Kramer A, Muijtjens A, Jansen K, Düsman H, Tan L, van der   Vleuten
          C Comparison of a rational and an empirical     standard setting
          procedure for an OSCE, Medical Education, 2003 Vol 37 Issue 2, Page
          132

          Kaufman DM, Mann KV, Muijtjens AMM, van der Vleuten CPM. A
          comparison of standard-setting procedures for an OSCE in
          undergraduate medical education. Acad Med 2000; 75:267-271.
        The examiner’s role in
          standard setting
• Uses the examiner’s clinical expertise to judge the candidate’s
  performance

• Examiner allocates a global judgement based on the
  candidate’s performance at that station

• Remember the level of the examination

                                                Pass
                         Borderline
       Fail
     Borderline Group Method
                                    Test score distribution

Checklist
1. Hs shjs sjnhss sjhs sjs sj

                            
2. Ksks sksmsiqopql qlqmq q q qkl

3. Lalka kdm ddkk dlkl dlld
                            
4. Keyw dd e r rrmt tmk

5. Jfjfk dd                 
6. Hskl;s skj sls ska ak akl
                            ald

7. Hdhhddh shs ahhakk as
                                          Borderline score distribution
TOTAL                       
Pass, Fail, Borderline

                                    Passing score
   Contrasting groups method
                                           Test score distribution

 Checklist
 1. Hs shjs sjnhss sjhs sjs sj
                                     
 2. Ksks sksmsiqopql qlqmq q q qkl
                                     
 3. Lalka kdm ddkk dlkl dlld

 4. Keyw dd e r rrmt tmk

 5. Jfjfk dd
                                     
                                              Fail    Pass
 6. Hskl;s skj sls ska ak akl ald
                                     
 7. Hdhhddh shs ahhakk as


 TOTAL                               

Pass, Fail, Borderline P/B/F




                                         Passing score
     Regression based standard

Checklist                                                          X = passing score
1. Hs shjs sjnhss sjhs sjs sj
                                             Checklist
2. Ksks sksmsiqopql qlqmq q q qkl
                                               Score
3. Lalka kdm ddkk dlkl dlld

4. Keyw dd e r rrmt tmk

5. Jfjfk dd

6. Hskl;s skj sls ska ak akl ald
                                                         X
                                    
7. Hdhhddh shs ahhakk as


TOTAL                               
                                                               1          2          3          4        5
Overall rating 1 2 3 4 5                                      Clear   Borderline    Clear   Excellent Outstanding
                                                              fail                 pass
                                1   =   Clear fail
                                2   =   Borderline
                                3   =   Clear pass
                                4   =   Excellent
                                5   =   Outstanding
Performance-based standard
         setting
 • Advantages
 • Utilises the expertise of the examiners
   • they are observing the performance of the
     students
   • they are in a position to make a (global)
     judgement about the performance
      • based on
         • their clinical expertise
         • expected standards for the level of the test
         • knowledge of the curriculum/teaching
      Borderline/ contrasting/
     regression-based methods
• Advantages
• Large number of examiners set a collective
  standard while observing the candidates – not
  just an academic exercise
• Reliable: cut-off score based on large sample of
  judgments
• Credible and defensible: based on expert
  judgment in direct observation
       Performance-based standard
                setting

• Disadvantages
• Requires large cohort of candidates to achieve
  enough numbers in the ‘borderline’ group
• Passing score not known in advance
• Judgments not independent of checklist scoring
• Requires expert processing of marks immediately
  after the exam
   •   Checking of results
   •   Delay in producing results