Evaluation - Harvard Medical School by wuyunyi

VIEWS: 8 PAGES: 76

									            26 September 2007




  Student Assessment:
New Ideas and Old Basics

                  Louis Pangaro, MD
               Professor and Vice-chair
              for Educational Programs
           Department of Internal Medicine
  Uniformed Service University of the Health Sciences
         “clinical assessment”
•   By teachers (house staff and faculty)
•   On clinical rotations
•   Based on words = descriptors
•   Using words = descriptive

• In vivo (vs. in vitro end of course/year)


                                              2
                  What’s old?
• Suspicion of grades by teachers
• Teachers reluctance to be direct
  (honest?)
• Belief that grading by teachers is
  subjective
• Lawsuits about low grades
    • Jamieson, Guidebook for Clerkship Directors, 2005.



                                                           3
              What’s basic?
• We have an obligation
• Fairness
• Mentoring
  – The Hippocratic Oath
• Professionalism
  – Duty and expertise (Pellegrino)



                                      4
          How it looked in 1987
• lack of meaningful comments by
  evaluators
• insufficient definition of evaluation criteria
• too much inter-observer variability
• late submission of evaluations
• delay in feedback to students

          Tonesk X, Buchanan RG. J Med Ed. 1987
                                                  5
        How it looked in 2001
• “Areas of weakness in current
  [clinical] evaluation models include
  psychometric properties associated
  with the tools, namely their
  questionable reliability and validity.”

• Turnbull, International Handbook of Research in
  Medical Education, 2002.
                                                    6
        How it looks in 2007
• “Constructive criticism is hard to come
  by..”
• “…Candor is at least as painful to the
  provider as to the recipient…”
• “…faculty members feel uncomfortable
  inflicting pain even in a good cause like
  student improvement.”

• HMS Student Handbook, 2006-2007
                                          7
             focus

• The emotional issues for
  teachers and learners in the
  grading process.
• Dealing with barriers to
  candor.

                                 8
• Quality: minimizing unwarranted
  variation in physician performance.
  – Wennberg
• Faculty Development: minimizing
  unwarranted variation in teacher
  performance.



                                        9
             What’s new
• A believe that quality methods can
  apply to teaching.
• Search for more rigor.
• “Best Evididence Med Ed” (BEBM)
• “Med Ed Research Certificate”
• MPH, MHPE programs


                                       10
Today: describing
 success in clinical
 evaluations


                       11
    New focus on evaluation
       in clincial setting:
• Attention to professional
  traits (Papadakis)
• Lawsuits by patients




                              12
        Context of new methods
•   Objective Structured Clinical Exams
•   3600 assessments
•   Portfolios
•   Descriptive vocabularies
    – To get more inter-rater agreement
    – To calibrate differences between levels of
      performance


                                                   13
  FOCUSED PHYSICAL EXAMINATION

• [Norm]
• Exam is generally appropriate in scope
  and technique. Identifies major
  abnormalities and pertinent normal
  findings, only occasionally missing
  elements. Exam linked to history.
  Appropriate for level of training.

• HMS Clerkship Grading Form
                                       14
FOCUSED PHYSICAL EXAMINATION (2)

• [Poor]
• Consistently misses important findings
  and often does not make appropriate
  connection between history and
  physical. Often uses faulty or
  inappropriate technique. Not organized
  or thorough.


                                       15
FOCUSED PHYSICAL EXAMINATION (3)

• [Excellent]
• Exam is consistently superior.
  Uncovers subtle and important
  findings, incorporating advanced
  techniques where appropriate.
  Exceptionally organized and
  thorough, even on difficult cases.

                                       16
American Board of Internal Med

                      Average and 2 SDs



1   2   3   4     5    6     7   8   9



            Minimal acceptable
                                         17
   ACGME Outcomes Project
• New emphasis on outcomes (vs
  curriculum)
• “long-term effort designed to
  emphasize educational outcome
  assessment in residency programs
  and in the accreditation process.”

                                   18
ACGME: Outcomes: a cultural shift
• Designing Curriculum is no longer
  enough
• Results must be demonstrated.

• A shift from process to product.
• Content expertise not enough; need
  pedagogic expertise

                                       19
ACGME “Competencies”(1999)
•   Medical Knowledge
•   Interpersonal & communication skills
•   Professionalism
•   Patient Care
•   System-based Practice
•   Practice-based learning & Improvement


                                        20
the ACGME has spoken !




                         21
 CanMEDS (2005)
• Medical Expert
• Communicator
• Collaborator
• Professional
• Manager
• Health Advocate
• Scholar

                    22
    “R.I.M.E. Scheme”
– Reporter
– Interpreter
– Manager/Educator

         Pangaro, Academic Med, 1999

                                       23
     Acceptance (1) Ob-Gyn
“The RIME method is a valid, logistically
  feasible and acceptable way of assessing
  medical student clinical performance…

…..minimizes disadvantages of descriptive
 evaluation, and maximizes the opportunity
 for accurate observations and helpful
 feedback.”
         APGO UME Taskforce:
         Espey et al, Am J Ob Gyn, 2007
                                             24
  Fairness
Consistency
Expectations
               25
          Principle 1 : Fairness
   to society :
     valid(not arbitrary) and
     sensitive to detect marginal performers
   to students :
     know  what’s expected,
     timely feedback
   to teachers :
     know  what observations to make
     protected (legally, emotionally)
                                                26
       Principle 2 : Consistency
Reliability
  - Within teacher
  - Between teachers
    - In same rotation or across blocks
    - Between sites in same clerkship
    - Between disciplines
  - suitable for high-stakes decisions
                                          27
             strategy

•   Simplicity leads to
    acceptance and use
•   Acceptance to consistency
•   Consistency to fairness

                                28
  Fairness & Consistency
    Reliability &Validity =
Stability of measurement and
  strength of inferences from
          observations


                            29
   “ Old” beliefs in our culture
• Grading by teachers is subjective,
  Examinations are objective

• Measurement > description,
  Numbers are > words


                                   30
Framing the question more simply:

• What do we expect of students?
• Can we get all teachers to have the
  same expectations, and apply them
  consistently?
• [A question of words and of
  conceptual frameworks]

                                    31
        Educational Goals
community

            Curriculum


            Evaluation
                         Depends on
                         expectations

       Feedback / Grading
                                    32
The goal: progressive independence of the learner



    Teacher/                    Learner
    program




                 Content – Goals
                   (Patients)
                                     after SFDP
                                                  33
Frames of reference
for expressing goals
1.   Analytic
2.   Developmental
3.   Synthetic


                       34
goals for education (generic):

 • attitudes/behavior
 • skills

 • knowledge
                   “KSA”
                             35
       Bloom’s Taxonomies

Cognitive Domain



Psychomotor Domain



Affective Domain

                            36
    1. Analytic expression of Goals
–   “ana-lytic”: takes the learner “apart”
–   into domains, categories
–   “attitude”, “skills”, “knowledge”
–   considered separately
–   generic terms
–   useful for discrete assessments
                                        37
           Curricular Goals: KNOWLEDGE
The School of Medicine will ensure that before graduation a student will have
  demonstrated, to the satisfaction of the faculty, the following:
• The capacity to recognize the limitations in one’s knowledge and clinical skills and
  to make a commitment to engage in lifelong learning
• Knowledge of the normal structure and function of each of the major organ
  systems of the body and the current basic scientific mechanisms operative at the
  systemic, cellular, and molecular levels
• Knowledge of the various causes (genetic, developmental, metabolic, toxic,
  microbiologic, immune, psychosocial, neoplastic, traumatic, and degenerative) of
  illnesses and diseases
• Knowledge of the altered structure and function of the body and its major organ
  systems that are seen in various illnesses and diseases
• Knowledge of the scientific method in establishing the causation of disease and
  efficacy of traditional and non-traditional therapies
• Knowledge of health care policy and the economic, psychological, social, and
  cultural factors that affect health and health care delivery
• Knowledge of the most frequent clinical, laboratory, radiographic, and pathologic
  manifestations of common as well as life threatening diseases
• Knowledge about relieving pain and ameliorating the suffering of patients
• Knowledge of the epidemiology of diseases and the systematic approaches useful
  in promoting health
• Knowledge of techniques of patient education and counseling in basic lifestyle
  changes/prevention                                                             38
• Knowledge of and approaches to reduce the psychological and physical risks and
                     Curricular Goals: SKILLS
•   The School of Medicine will ensure that before graduation a student will have
    demonstrated, to the satisfaction of the faculty, the following:
•   The ability to obtain an accurate medical history and the ability to perform both a complete
    and an organ specific examination, including a mental status examination (See appendix A)
•   The ability to perform routine technical procedures (See appendix B)
•   The ability to interpret the results and be aware of the indications, complications, and
    limitations of commonly used diagnostic procedures (See appendix C)
•   The ability to demonstrate knowledge of theories and principles that govern ethical decision
    making
•   The ability to reason deductively and inductively in solving clinical problems
•   The ability to construct appropriate differential diagnoses and treatment plans for patients
    with common conditions, both acute and chronic, including medical, psychiatric, and surgical
    conditions, and those requiring short- and long-term rehabilitation
•   The ability to recognize patients with immediate life threatening conditions regardless of
    etiology, and to institute appropriate initial therapy
•   The ability to recognize and outline an initial course of management for patients with serious
    conditions requiring critical care
•   The ability to communicate effectively, both orally and in writing, with patients, patients’
    families, colleagues, and others with whom physicians must exchange information in carrying
    out their responsibilities
•   The ability to select appropriate tests for detecting patients at risk for specific diseases and to
    determine strategies for responding appropriately
•   The ability to retrieve, critically review, and effectively utilize biomedical information from
    electronic databases and other resources for solving problems and making decisions that are
    relevant to the care of individuals and populations
•   The ability to evaluate the economic, psychosocial, and cultural factors that impact the39    health
    of patients and families and to incorporate these into assessment and treatment plan
          Curricular Goals: ATTITUDES
• The School of Medicine will ensure that before graduation
  a student will have demonstrated, to the satisfaction of
  the faculty, the following:
• A commitment to advocate the interests of one’s patients
• Compassionate treatment of patients, and respect for their
  privacy and dignity
• Honesty and integrity and dutifulness in all interactions with
  patients, their families, colleagues, and others with whom
  physicians interact
• An understanding of, and respect for, the roles of other health
  care professionals, and the need to collaborate with others in
  caring for patients and promoting health
• A commitment to provide care to patients who are unable to
  pay and to advocate for access to health care for members of
  underserved populations

                                                            40
         2. Developmental Terms:

•   “Novice”
    can identify absence of pulse
•    “Advanced learner”
    can distinguish specific arrythmias
•    “Expert”
    can manage ventricular fibrillation

                                          41
        Dreyfus and Dreyfus
   Novice
   Advanced beginner            students
   Competent performance
                                 residents
   Proficient performance
   Intuitive expert             facutly
   Master

              Mind Over Machine (1986)
                                           42
 Developmental vs. Analytic
• time-line, progression included
    levels of function
• essential for multi-year training
• although the terms (“novice”,
 “master”…) remain generic

                                      43
Goals: ACGME “Competencies”
•   Medical Knowledge
•   Interpersonal & communication skills
•   Professionalism
•   Patient Care
•   Practice-based learning
•   System-based Practice
         analytic?         What about
         developmental? the last three??
                                           44
   A cube to encompass competence
                UME --> GME  CME
       school
  hospital                          Medical Knowledge
clinic
                                    Patient Care

                                    Interpersonal skills

                                    Professionalism

                                    Practice-based
                                    learning/improv.
                                    System-based
                                    Practice


                                                     45
Resistance is futile




                       46
        Educational Goals
community

            Curriculum


            Evaluation
                         Depends on
                         expectations

       Feedback / Grading
                                    47
3. the “Synthetic”framework
•   “syn-thetic” -putting the learner
    back together
•   “K S A” are all required, integrated
•   terms are a bit less generic, more
    behavioral

                                      48
    “R.I.M.E. Scheme”
– Reporter
– Interpreter
– Manager/Educator

         Pangaro, Academic Med, 1999

                                       49
  a framework to classify level of function

• rudimentary reporting: “My patient has
 a fever, cough and a bad rash - it’s
 vesicular or pustular ….”
• rudimentary interpreting: “I think it
 might be due to chicken pox or
 herpes.”

                                          50
• rudimentary manager/educator:
     “I’d consider a smear of the
  fluid and a chest x-ray . …We
  might observe or treat with
  acyclovir, …but I’m not sure. I’ll
  have to look this up.”



                                       51
  Alternative model R.I.M.E.
• Framework that is
  – developmental
  – behavioral
   • student can visualize,
     framework for observer
 – “synthetic”
                               52
                “RIME”
• Reporter: takes ownership of getting
  the facts on every patient (“what”?)
• Interpreter: takes ownership of thinking
  and explaining (“why”?)…
• Manager: takes ownership of planning
  with patient (“how?)…
• Educator: takes ownership of
  developing and sharing expertise…
                                         53
      Matrix: transition to higher expectations
                  I   II   III   IV   PGY1   PGY2/3   Practice

EDUCATOR
              I            R     R    R       P        M
MANAGER                           I    R      P        M
                                                       M
INTERPRETER           I    R     P
                                                       M
REPORTER      I       R    P

 I = introduced R = repetition P = proficiency


 M = mastery in practice
                                                                 54
Duty and Expertise (Pellegrino)
• Each RIME “level” is a way of asking,
  does the student fulfill that promise
• Making a diagnosis, not “giving” it.
• “Objective”
• Does it affect teacher ratings?



                                      55
  Example: Construct Validity
Grade Distributions Univ. of Utah




     Low Pass Pass   HP   Honors
             Battistone Acad. Med. , 2001   56
Grade Distributions Univ. of Utah
     after RIME methods




  O     R    I       M       E
             Battistone Acad. Med. , 2001   57
            Case using RIME:

W.O., student, “presents” Mrs. Jones:
• 45yo woman with acute lower back pain
• gives detailed description clinical
  picture suggestive of acute lumbar
  strain
• through physical examination,
  – blood pressure 130/80, heart rate 80
  – left-sided para-spinal tenderness L2 – L5
                                                58
     Case using RIME (continued)
• while student is writing up findings in the
  patient’s record, you interview and
  examine patient.
• Mrs. Jones asks: “Doctor, can you take
  my blood pressure since no one has?”

• at what “RIME” level is this student?


                                           59
Is the student ready for more
  responsibility – yes or no?


RIME is a razor.

                                60
    Complimentary Approaches

Synthetic (“steps”)    Analytic (domains)
  • (non-reporter)

  • Reporter              • Attitude
  • Interpreter           • Skills
  • Manager/              • Knowledge
    Educator
                      ACGME “Professionalism”

                                            61
 The RIME rhythm is familiar:

H&P…………. …….S.0…..       Reporter
Assessment..   ….A………… Interpreter
Plan………….      ….P…………. Manager/
                         Educator



                                     62
 Reliability versus validity
X X X
 X   X
X X X
                       X X X
                       X XX
                       X


   Reliable            Valid   63
              reliability

• Computed estimate of whether an
  assessment tool is testing a single
  construct.
• A reliability of 0.8 is considered
  needed for high stakes decisions
  (80% signal).


                                        64
      Typical Reliabilities

 exam type             alpha
• “shelf”-100 MCQ      0.75 - .8
• Step 1 USMLE         0.9 - .95
• OSCE (6 stations)    0.5 - 0.6
• OSCE (12 stations)   0.7 - 0.9
                                   65
       Third-year performance
            Level = Grade

• (Observer)    Low Pass    D   1.0


• Reporter      Pass        C   2.0

• Interpreter   High Pass   B   3.0

• Manager/      Honors      A   4.0
  Educator

                                      66
 USU Medicine clerkship

(students)          Reliability
  n = 467             0.83


       Roop, Amer J Med, 2001

                                  67
  validity

What can you infer from your
evaluation?
Are you measuring what is
important?                     X X X
                               X XX
                               X


                               Valid   68
     EDP Evaluation System
• RIME Vocabulary
• Formal Evaluation Sessions
  – Sit down with teachers every few weeks
  – Noel, J Med Ed, 1987 (detecting marginal
    students)
  – “Frame of reference training”




                                               69
       Content Validity: Detecting Deficiencies in
                    Professionalism

% of professionalism domains rated less than acceptable




                         Hemmer, Academic Medicine, 2000.
                                                     70
    Predictive Validity: Sensitivity in
    predicting Internship Problems
      (PGY1 supervisor surveys)
                                      USU classes
percent                               of 86 - 93




          Low Ratings   Bad Comment

           Lavin, Academic Medicine 1998            71
    Acceptance (2): Medicine
• 93% teachers’ evaluations
• 81% NBME subject examination
• 42 % RIME vocabulary
• 32% OSCE
• 22% direct observation (mini-CEX)

             Hemmer, Teach Learn Med, 2007
                                             72
    Limitations of our system
• RIME is sometimes taken as
  developmental scheme.
• The analytic model is very strong, and
  RIME is used for cognitive growth only.
• Wanting to skip the Evaluation
  Sessions - they take time!


                                        73
     Basics              Innovations
• Clinicians are good   • Descriptors provide
  diagnosticians          patterns and
                          exemplars
• We are mentors for    • Descriptive
  our students            frameworks allow
                          behavioral feedback
• We promise society    • Synthetic frameworks
  duty and expertise      make this simpler


                                                 74
             strategy

•   Simplicity leads to
    acceptance and use
•   Acceptance to consistency
•   Consistency to fairness

                                75
 Thanks for coming!

lpangaro@USUHS.mil



                      76

								
To top