“Criterion Based” vs.
“Norm-Based” Evaluation
David L Gaspar MD
October 18, 2008
UCD School of Medicine
Evaluation of Students
Goals- This session is designed to enable
participants to:
1. Understand the definition and purpose of
evaluation
2. Develop a framework to make evaluation
efficient and useful
3. Understand and apply criterion-based
evaluation – the “R.I.M.E.” method
UCD School of Medicine
Evaluation Basics
Definition
Evaluation is the process of making judgments based on
factual information and observations in order to rate,
rank, or assess an individual’s status at a given point.
Purposes of Evaluation
1. Summarize performance at a given point in time.
2. Provide information for planning future educational
experiences.
3. Communicate summary information to other parties.
UCD School of Medicine
Framework for Planning
Clerkship Evaluation
Before Clerkship
1. Understand medical school expectations and “forms”.
2. Review clerkship goals and objectives.
During Clerkship
1. Gather information from multiple sources & provide feedback.
3. Use systematic method of recording.
End of Clerkship
1. Prepare for final evaluation.
2. Schedule and conduct summary meeting.
3. Complete and submit final evaluation.
UCD School of Medicine
Tips to Facilitate Evaluation
1. Dictate or write a student progress note. (end
of day, when procedures occur-behaviours)
2. Use student log books.
3. Review student’s written record. (copy
progress notes at the beginning, middle, and
end of clerkship)
4. Use student self-assessment forms.
5. Use computer printout of student’s patient
profile.
UCD School of Medicine
Evaluate…
End of Clerkship
1. Review evaluation criteria and form.
2. Review student performance data.
3. Schedule and conduct final evaluation session
with student.
4. Have some “backbone”
5. Complete and return the evaluation form
promptly. (Keep a Copy!!!)
UCD School of Medicine
“Norm”-Based vs. Criterion-
Based
Any ideas????
UCD School of Medicine
“Norm”-Based vs. Criterion-
Based
NORM-BASED EVALUATION
• Norm-based assessment compares
individuals’ performances to one another,
resulting in a ranking of individuals.
Medical Knowledge
1 2 3 4 5 6
Poor Excellent
UCD School of Medicine
“Norm”-Based vs. Criterion-
Based
NORM-BASED EVALUATION
• This traditional form of assessment
has been widely used to select
individuals for further education or for
jobs and is what most students,
residents, fellows and faculty have
been exposed to throughout their
training
UCD School of Medicine
“Norm”-Based vs. Criterion-
Based
NORM-BASED EVALUATION
• Since norm-based assessment provides no
information about the observed quality of
the performance, it isn’t useful for providing
feedback or improving performance.
• The rater’s task ends up distinguishing one
individual’s performance from another’s.
UCD School of Medicine
“Norm”-Based vs. Criterion-
Based
NORM-BASED EVALUATION
• Raters will vary in their assessments
depending on their experience with using a
norm-based scale, and their experience with
the group being evaluated
1 2 3 4 5 6
Poor Excellent
UCD School of Medicine
NORM-BASED EVALUATION
Dr. Thomas, who has 15 years of experience working with and
evaluating students, is due to evaluate John, his student over the last
2 weeks. He has observed John performing a brief history and
physical examination, has heard daily oral presentations, and has
interacted with him on multiple occasions to assess his fund of
knowledge. Dr. Thomas has a clear expectation based on this
experience for what an “honors student” is. He believes that John is a
very good student with great enthusiasm, but not quite functioning at
an honors level and will therefore complete his evaluation in a manner
that reflects what he thinks will result in a “high pass” grade- on a 6-
point scale. mostly 5s and some 6s.
1= poor performance and 6=excellent performance,
UCD School of Medicine
NORM-BASED EVALUATION
Dr. Thomas’ evaluation:
1= poor performance and 6=excellent performance,
Poor Excellent
Medical Knowledge 1 2 3 4 5 6
History Taking 1 2 3 4 5 6
Physical 1 2 3 4 5 6
Examination
Oral Presentations 1 2 3 4 5 6
Professionalism 1 2 3 4 5 6
UCD School of Medicine
NORM-BASED EVALUATION
The next attending to work with John, Dr. Smith, has just completed
residency. She too has been working with John for 2 weeks and has
observed him doing a brief history and physical examination, has
heard daily oral presentations, and has interacted with him on multiple
occasions to assess his fund of knowledge. She is an enthusiastic
teacher and believes that students should get honors if they are active
participants in rounds, enthusiastic about learning and generally
helpful. She went to a medical school where most students got honors
if they did these things and she thinks this is a good policy. She has
heard from the medical students and residents that she has worked
with that to get honors, she must give John all 6s on a 6 -point scale
1= poor performance and 6=excellent performance,
UCD School of Medicine
NORM-BASED EVALUATION
Dr. Thomas’ evaluation:
1= poor performance and 6=excellent performance,
Poor Excellent
Medical Knowledge 1 2 3 4 5 6
History Taking 1 2 3 4 5 6
Physical 1 2 3 4 5 6
Examination
Oral Presentations 1 2 3 4 5 6
Professionalism 1 2 3 4 5 6
UCD School of Medicine
NORM-BASED EVALUATION
What grade does John deserve?
More importantly, what is John’s level of
competence?
UCD School of Medicine
NORM-BASED EVALUATION
Answer:
Hard to tell…..This example provides insight
into some of the challenges of norm -based
assessment. Now let’s learn about an
alternative means of assessment.
UCD School of Medicine
How do we evaluate the learner
with a criterion-based system?
• There have been a variety of methods
used over the past several decades
• In 1999 L. Pangaro developed RIME – a
new tool for evaluating the level of your
learner – more specifically for medical
students
UCD School of Medicine
RIME
• Reporter
• Interpreter
• Manager
• Educator
• Before RIME – Observer/Pre-Reporter
(pre-clinical)
UCD School of Medicine
RIME
• Reporter - Consistently good in interpersonal
skills, reliably obtains and communicates
clinical findings. Written and spoken
presentations are clear and organized.
• Interpreter – Able to prioritize and analyze
patient problems, develop a differential
diagnosis and next steps in workup. Data is not
just data (reporter), but has a clinical
significance.
UCD School of Medicine
RIME
• Manager – Consistently proposes reasonable
diagnostic and therapeutic options,
incorporating patient preferences. They can
take their knowledge, integrate it with the
findings for a given patient and form a plan
that prioritizes within and amongst the problem
list.
• Educator – Consistent level of knowledge of
current medical evidence; demonstrates self-
directed learning (practice based improvement
and learning) and contributes to others’
education.
UCD School of Medicine
RIME Changes with Time
• You can use RIME to track the progress/growth
of a student over the course of their clinical
years or within a given year.
• Early MS3 – Should report well and begin to
interpret.
• Late MS3 – Should be a good interpreter
• Sub-I should have the above and begin to
manage patients appropriately.
• Interns should become astute managers
• R2s and beyond – progressive depth of
management and becoming educators
UCD School of Medicine
RIME: Criterion-based evaluation
• Criterion -based assessment begins with an
established set of criteria for successful performance
and compares an individual’s performance to these
criteria.
UCD School of Medicine
RIME: Criterion-based evaluation
• Rating an individual’s performance against the criteria
provides feedback on the quality of performance.
Multiple individuals may successfully meet the criteria
as one individual’s performance is not constrained by
another’s performance. Agreement between raters is
high as the standard for comparison is stable and based
on observable behaviors. In other words, the rater is
responsible for documenting what the specific skills of
the individual are based on their observed
performance. They are NOT responsible for comparing
them to anyone else, or for trying to determine a
grade.
UCD School of Medicine
RIME: Criterion-based evaluation
In the scenario just described, both Dr. Smith and Dr. Thomas have observed
John’s oral presentations. When they are asked to complete the evaluation of
John they are asked to rate his abilities based on observable behaviors related
to his oral presentations. John’s oral presentations are accurate and objective.
But, they sometimes contain extraneous information and important information
gets lost in the presentation.
Please complete the evaluation below as if you were Dr. Thomas, with 15 years
of experience teaching medical student:
UCD School of Medicine
RIME: Criterion-based evaluation
Would you evaluate the student any differently
if you were Dr. Smith?
UCD School of Medicine
RIME: Criterion-based evaluation
John is able to derive a rudimentary differential diagnosis and
assessment for patients presenting with shortness of breath and chest
pain with only minor assistance from an intern or resident. But, he has no
idea how to approach the presenting problems of acute renal failure and
hyponatremia. For each of these latter clinical situations, the intern
needed to completely develop the differential diagnosis and teach it to
John, who was then able to repeat it at attending rounds. He is only able
to develop management plans with significant help from the intern. Each
of these 4 conditions is common on your rotation and required clinical
entities for the clerkship.
How would you rate him on the following scale?
UCD School of Medicine
RIME: Criterion-based evaluation
John is able to derive a rudimentary differential diagnosis and assessment for
patients presenting with shortness of breath and chest pain with only minor
assistance from an intern or resident. But, he has no idea how to approach the
presenting problems of acute renal failure and hyponatremia. For each of these
latter clinical situations, the intern needed to completely develop the differential
diagnosis and teach it to John, who was then able to repeat it at attending
rounds. He is only able to develop management plans with significant help from
the intern. Each of these 4 conditions is common on your rotation and required
clinical entities for the clerkship.
How would you rate him on the following scale?
UCD School of Medicine
And Then Its in My Court…
UCD School of Medicine