Document Sample
clinical Powered By Docstoc
					           Evaluating Clinical Performance and the Core Competencies
                     CORD Faculty Development Conference
                                   February 2003
                                 Washington, D.C.
                             Nicholas J. Jouriles, M.D.
                          Akron General Medical Center

Learning Objectives

At the end of this session, the learner will be able to:
    1. List the ACGME core competencies
    2. Describe 3 techniques for bedside evaluation
    3. Discuss the pros and cons of bedside evaluation techniques
    4. Describe which bedside evaluation techniques work best for EM

Lecture Outline
   1. Adult learning and evaluation theories
   2. The core competencies
   3. Bedside evaluation techniques
   4. The core competencies applied to EM


The ACGME has offered all residency programs the opportunity to make
improvements. This opportunity is named the core competencies. Why is this an
opportunity? Because the root cause is neither medicine nor education, but
business. This is an opportunity to apply a bottom line, results oriented approach
to medical education.

The core competency movement is a paradigm shift for residency programs,
from demonstrating compliance with requirements to showing the ability to
evaluate competence in several domains. It is also a revolutionary change for the
ACGME, from one of proscribing behavior based on rigid, well defined standards
to one where program initiative and creativity will be encouraged then judged
against an as yet to be defined standard.

 The core competencies include patient care, medical knowledge,
professionalism, systems-based practice, practice-based learning and
improvement and communication and interpersonal skills. They are supposed to
increase resident education and accreditation effectiveness by emphasizing
outcomes [Swing]. By July 2006 every residency program must demonstrate
The core competency movement once again shows that Emergency Medicine is
a leader in medical education. In March 2002, CORD sponsored a conference
“The ACGME CORE Competencies: Getting Ahead of the Curve” which brought
together EM educators to define the core competencies for our specialty.
Participants came from residencies, medical schools, the ACGME, ABEM,
EMRA, SEAM, AEM and the RRC-EM. This conference offered the specialty a
proactive approach to the core competencies – a chance us to define what is
important for our specialty. The results of this conference are summarized in the
November 2002 special consensus issue of AEM [Volume 9, Number 11].

Teaching and Evaluation
The most important concept is that teaching and evaluation are inexorably linked.
You teach and evaluate to the same end – the terminal objectives of your
educational program. This is true for learners at all levels and includes curricula
ranging from 30 minute seminars to entire residency programs.

There are many ways to categorize traditional teaching approaches and types of
teachers. The master crafts person teaches by modeling exemplary behavior. In
contract, the Socratic teacher encourages thinking in the learners by the use of
probing questions. The "pimp" also asks questions but usually manages to
embarrass the learner. Finally there is the coach whose supportive manner in
both questioning and supervision encourages the learner to take the risks that
are necessary when learning new skills.

Several concepts have emerged for improving teaching in adult learners. The
most important is for the teacher to be available. Showing interest and
enthusiasm and support for the adult learner will go a long way toward achieving
a positive result. The instructor should be very clear about his or her
expectations for the learner and clarify those expectations as needed. The
successful performance should be rewarded. [Epstein]

One medical educator [Irby, 1994] studied the educators at his institution that
received the most praise for their teaching. Several themes emerged. They: 1.
Actively involve learners. 2. Capture attention and have fun. 3. Connect the
case to broader concepts. 4. Go to the bedside. 5. Meet individual needs. 6.
Be practical and relevant. 7. Be selective and realistic. 8. Provide feedback
and evaluation.

But to be effective in the core competency era, we must demonstrate the
outcome of our teaching. The first step is to define the outcome. This is well
described in the November 2002 issue of AEM. The next step is to assess
whether the learner has met the objective. This is the Holy Grail of education.

Any evaluation tool must be reliable and valid. The closer the assessment
technique resembles the real situation, the more likely it will be to predict future
performance. Assessments that predict future performance exhibit predictive
reliability. [Swing] Well defined scoring criteria, training evaluators to use these
criteria and assessment of specific tasks rather than a subjective global appraisal
all improve the reliability of the evaluation. [Swing]

Evaluation tools include: global assessment, checklists, 360 degree evaluations,
structured oral exams or cases, simulators, standardized patients, direct
observation and portfolios.

Global assessment: This is the most common assessment tool in use today. It
consists of a scoring sheet that is completed based on the overall impression of
the evaluator. There are usually criteria set out, sometimes defined, and space
for free form comments. The reliability and validity of this tool varies.

Checklists: This tool may be used for any complex task that can be divided into
component parts. This works best when the performance of each part is
necessary to competently perform complete task. Procedures are well suited to
this type of evaluation.

360 degree assessment: This tool involves an assessment being performed by
all the individual that a learner interacts with, presumably demonstrating a variety
of relationships, each thought to be relevant to the overall outcome.

Structured oral exams or cases: These are reliable and valid tools where the
evaluator creates and controls content so as to assess specific areas. These can
include standardized patients, role playing or other stimuli.

Simulators: This tool can include models, mannequins or computer generated
material that can be used as a substitute for a patient or clinical scenario. Like
structured cases, content can be controlled by the evaluator.

Standardized patients (SP): This tool is becoming more available as it is
embraced by the USMLE and ECFMG. The use of SP’s has a long history and
there are many studies showing the reliability and validity of their use.

Direct observation: This tool places the evaluator and the learner at the bedside
together where actual desired performance is witnessed by an evaluator skilled
in the area being demonstrated by the observer.

Portfolios: This tool is similar to what artists have used for centuries – show your
work product. It can also be a case log or a collection of reports about different
educational opportunities that the learner has experienced.

Teaching and evaluation are linked. Define your educational end point and teach
and evaluate to that end. The paradigm shift to the core competencies
represents a paradigm shift in medical education – and therefore an opportunity.
Emergency Medicine is on the cutting edge – yet again! – of this shift. It’s all
summarized in the November 2002 issue of AEM. And the best is yet to come.


1.    WF Bond, L Spillane, for the CORD Core Competencies Simulation
      Group. The use of simulators for Emergency Medicine resident
      assessment. Acad Emerg Med 9 : 1295 - 1299, 2001.
1.    SA McLaughlin, D Dozeman, DP Sklar. Human simulation in Emergency
      Medicine: A model curriculum. Acad Emerg Med 9 : 1310 – 1318, 2001.
2.    W. P. Burdick, et al. Reliability of Performance-Based Clinical Skills
      Assessment. Academic Emergency Medicine 3:1119-1123, 1996.
3.    W. P. Burdick, et al. Standardized Patients and MS1 Emergency
      Medicine Course. Journal of Emergency Medicine 10:627-629, 1991.
4.    R. K. Cydulka, C. L. Emerman, N. J. Jouriles. Evaluation of Resident
      Performance and Intensive Bedside Teaching during Direct Observation.
      Academic Emergency Medicine. 3:345-351,1996.
5.    R. Davidhizar, R. Shearar. Improving Your Bedside Manner. Journal of
      Practical Nursing. 48:10-14,1998.
6.    R. M. Epstein, E. M. Hundert. Defining and Assessing Professional
      Competence. JAMA. 287:226-244,2002.
7.    H. L. Copeland, M. G. Hewson. Developing and Testing an Instrument to
      Measure the Effectiveness of Clinical Teaching in an Academic Medical
      Center. Academic Medicine. 75:161-166,2000.
8.    D. Irby. What Clinical Teachers in Medicine Need to Know. Academic
      Medicine. 69:333-342,1994.
9.    D. Irby. Teaching and Learning in Ambulatory Care Settings: Thematic
      Review of the Literature. Acadmic Medicine. 70:898-931,1995.
10.   SR Swing. Assessing the core competencies: general considerations and
      assessment methods. Acad Emerg Med 9 : 1278 – 1288, 2001
11.   RW King, F Schiavone, FL Counselman, EA Panacek. Patient care
      competency in Emergency Medicine graduate medical education: results
      of a consensus group on patient care. Acad Emerg Med 9 : 1227 - 1235,
12.   MJ Wagner, HA Thomas. Application of the medical knowledge core
      competency to Emergency Medicine. Acad Emerg Med 9 : 1236 - 1241,
13.   SR Hayden, S Dufel, R Shih. Definitions and competencies for practice
      based learning and improvement. Acad Emerg Med 9 : 1242 - 1248, 2001.
14.   GL Larkin, LS Binder, D Houry, J Adams. Defining and evaluating
      professionalism: a core competency for graduate Emergency Medicine
      education. Acad Emerg Med 9 : 1249 - 1256, 2001.
15.   C Hobgood, R Riviello, NJ Jouriles, G Hamilton. Assessment of
      communication and interpersonal skills. Acad Emerg Med 9 : 1257 - 1269,
16.   PL Dyne, RW Strauss, S Rinnert. Systems based practice: the sixth core
      competency. Acad Emerg Med 9 : 121270 - 1277, 2001.
17.   KG Rodgers, C Manifold. 360 – degree feedback: possibilities for
      assessment of the ACGME core competencies for Emergency Medicine
      resident competencies. Acad Emerg Med 9 : 1300 – 1304, 2001.
18.   OW Hayes, EJ Reisdorf, GL Walker, DJ Carlson, B Reinoehl. Using
      standardized oral examinations to evaluate general competencies. Acad
      Emerg Med 9 : 1334 - 1337, 2001.
19.   NJ Jouriles, C Emerman, RK Cydulka. Direct observation for assessing
      Emergency Medicine core competencies: interpersonal skills. Acad Emerg
      Med 9 : 1338 - 1341, 2001.

NJJ 12/28/02

Shared By: