ACGME Outcome Project Update:
Phase III and Beyond
CREOG Competency Task Force
March 2007
CREOG Competency Task Force
Diane M. Hartmann, MD, Chair University of Rochester
Jessica Bienstock, MD, MPH Johns Hopkins University
Mary Ciotti, MD University of California, Davis
Dee Fenner, MD University of Michigan
Tina Foster, MD, MPH Dartmouth University
Lee Learman, MD, PhD University of California, San Francisco
Rebecca McAlister, MD Washington University
Mark Woodland, MS, MD Drexel University
ACGME Outcome Project
Overview
Diane M. Hartmann, MD
ACGME Outcome Project:
What’s New?
All RRC Development Courses
completed
New Institutional Requirements
New Common Program
Requirements
New Common PIF
ACGME Resident Portfolio Project
ACGME Outcome Project:
New Institutional Requirements
(effective 7/07)
I.B.5.b.
“The Sponsoring Institution must ensure that
program directors have sufficient support and
protected time to effectively carry out their
educational and administrative responsibilities
to their respective programs.”
ACGME Outcome Project:
New Institutional Requirements
(effective 7/07)
IV.4.
“The internal review should assess each program’s:
e. Effectiveness in providing learning experiences that
lead to achievement of educational outcomes in
ACGME general competencies;
f. Effectiveness in using evaluation tools and outcome
measures to assess a resident’s level of competence
in each of the ACGME general competencies;
g. Effectiveness in linking educational outcomes with
program improvement.”
ACGME Outcome Project:
Common Program Requirements
(effective 7/07)
New specific language in sections IV. (Educational
Program) and V. (Evaluation) about teaching and
Assessment of the 6 competencies
ACGME Outcome Project:
Evaluations for most programs
should include:
Global assessments
In-training exams
Focused assessments (surgical,
medical, communication)
Professional associate/patient
assessments
Methods to record attainment of PBLI,
SBP skills
ACGME Outcome Project:
New Common PIF (effective 7/07)
Will align requirements and PIF
Asks specific questions about core
competency teaching, assessment
and faculty development
Competency Methods of Evaluation Evaluator(s)
Global evaluation of Faculty attending
competency-based physicians
Patient Care objectives
Observed history & Teaching attending on
physical using a the general inpatient
checklist service
Review of clinic and Clinic preceptors and
procedure logs Program Director,
respectively
Global evaluation of Faculty attending
competency-based physicians
Example response Medical Knowledge objectives
demonstrating Observation of a Program Director,
Journal Club with Associate Program
substantial predetermined criteria Director and Adolescent
compliance as for assessment Attending
suggested by American Board of
Pediatrics In-Training
Scores determined by
Board but PD discusses
Pediatrics Examination scores and study plan
with residents
Practice-based Global evaluation of Faculty attending
learning & competency-based physicians
Improvement objectives
Review of a quality Faculty expert in QI who
improvement activity serves as mentor for
with predetermined resident QI projects
criteria for assessment
Documented Self- Resident
assessment
Annual documented Resident with mentor
learning plans guidance
Observation and Primary care faculty
documentation of
teaching skills using a
checklist
Cont‟d
Competency Methods of Evaluation Evaluator(s)
Interpersonal & Global evaluation of Faculty attending
Communication Skills competency-based physicians
objectives
Example response
Evaluation by patients Patients, nurses and
demonstrating and allied health other support staff in
professionals using a continuity clinic, PICU
substantial checklist and NICU
compliance as Professionalism Global evaluation of Faculty attending
competency-based physicians
suggested by objectives
Pediatrics Evaluation by patients Patients, nurses and
and allied health other support staff in
professionals continuity clinic, PICU
and NICU
Systems-Based Global evaluation of Faculty attending
Practice competency-based physicians
objectives
Review of written Faculty expert in
reflections managed care who
teaches this segment of
curriculum
Review of a system Director of PICU and M
errors activity using & M activities and
predetermined criteria Program Director
Other helpful resources for assessment
from Pediatrics:
http://www.acgme.org/acWebsite/downloads/RRC_progReq/320pedSubs01012007.pdf
See pages 21-25
ACGME Outcome Project:
What’s Next?
Reconciliation of Common and
Specialty Program Requirements and
PIF
More web-based reporting
Common wording for citations among
specialty review committees
More consistency among specialty
review committees
WebADS
Program
Common
Requirements
Specialty Specific Resident Surveys
Institutional
Common
Common
Specialty Specific
Specialty Specific RRC Accreditation
Decisions & Citations
Common
Specialty Specific
PIF Site Visitor’s Survey
Common Common
Specialty Specific Specialty Specific
ACGME Outcome Project:
What‟s Next?
Resident Portfolio Project
Learning Portfolio
ACGME Style
A learning portfolio can be used to support reflective
Learning and practice, to document assessments and
To provide evidence for program and institutional
Accreditation decisions. . .It may eventually facilitate
Individual physician credentialing by various agencies;
e.g., certifying boards, licensing authorities and
Hospital credentialing committees.
ACGME Outcome Project
Patient Care, Surgery
Dee Fenner, MD
Surgical Skills
Current PIF
Web Accreditation Data System (Web ADS)
Provide a brief description of two things you have
done since your last review that have most improved
your teaching or assessment of the General
Competencies.
Anticipated 2007
“New PIF”
“Prove to us that you are
confident that residents can
do the standard procedures
needed to graduate.”
Teaching Methods
Clinical Teaching
Teaching that occurs in the clinic, EDs, ORs, laboratories, or other
medical settings and addresses issues related to residents‟ current patient
cases or clinical responsibilities.
Clinical Experiences
Direct, hands-on clinical or patient care activities. This may include
surgery, patient exams, the reading of radiographs and preparation of
pathology assays.
Performance Feedback
Information provided to a resident that describes what (s)he has done well
or poorly and provides specific guidance as to how performance might be
improved.
Teaching Methods
High-Tech Simulators/Simulations
3-dimensional, high tech, computerized devices
that represent human anatomy and physiological
responses (simulators) are used by residents to
learn procedures and operations. Or realistic
patient care scenarios are generated using high
tech/virtual reality devices (simulations). Residents
engage in the scenario as in real life to learn or
apply clinical or teamwork skills.
Anatomic or Animal Models
Non-computerized, 3-dimensional devices that
replicate the properties of human anatomical
structures are used by residents to learn
procedures.
Assessment Methods
Focused Observation and Evaluation -
Supervisor/attending observation of individual
resident-patient encounters, operations, specimen
preparation, etc., and concurrent (same day)
evaluation
High Tech Simulators/Simulations - Residents'
performance of procedures on a high-tech
simulator (e.g., Harvey) is evaluated; this may
involve built-in evaluation by the simulator or
observation and concurrent evaluation.
Methods
Anatomic or Animal Models -
Residents' performance of procedures
on non-computerized, 3-dimensional
models that replicate the properties of
human anatomical structures is
observed and evaluated concurrently
Review of Case or Procedure Log -
Review of number of cases or
procedures performed and comparison
against minimum numbers required
Identify Key Procedures
University of MI
HO – 1 Cases
Midline episiotomy repair
Cesarean Section
Diagnostic laparoscopy/TL
HO- 2 Cases
Operative Hysteroscopy
First trimester OB USN and GYN USN
Endometrial Biopsy (Office)
Sonohysteroscopy (Office)
Identify Key Procedures
HO-3 Cases
Abdominal Hysterectomy
Operative Laparoscopy/Ovarian Cystectomy
IUD insertion (Mirena or Cu) (Office)
Office hysteroscopy
HO-4 Cases
Vaginal Hysterectomy
TVT
Total laparoscopic hysterectomy
Urogyn Office Assessment
Sample Procedure Form
Surgical Competency
Ob/Gyn - UCSF
Evaluator: Date:
Resident: PGY: 1 2 3 4
Rotation:
Diagnosis:
Procedure: Cesarean Delivery
Operative Checklist
Rating X = not seen or indicated 1 = performed but poorly
Key: 0 = not performed but indicated 2 = performed correctly
1. Describe indication for cesarean delivery X 0 1 2
2. Select appropriate skin incision X 0 1 2
3. Separate fascia from abdominal muscle X 0 1 2
Office/Urogyn Evaluation
Rotation:
Diagnosis:
Rating X = not seen or indicated 1 = performed but poorly
Key: 0 = not performed but indicated 2 = performed correctly
Procedure: Urogynecology Assessment - POP-Q, PVR, Review Diary, Simple Cystometry
Procedure Checklist
1. Describe the symptoms of SUI and Urge Inc. X 0 1 2
2. Understands associated risks and contraindications
3. Explains procedure to the patient/ counseling X 0 1 2
4. Perform POP -Q - all points, assess Kegel X 0 1 2
5. Fill in grid X 0 1 2
6. Detrimine prolapse stage X 0 1 2
7. Clean Urethra X 0 1 2
8. Insert Catheter (obtain PVR) X 0 1 2
9. Fill bladder and assess first urge and capacity. X 0 1 2
10. Perform supine and standing stress test X 0 1 2
11. Perform supine and standing stress test X 0 1 2
12. Diary: Determine functional capacity, number voids, avg. X 0 1 2
13. Evaluate incontinence episodes and type of leakage X 0 1 2
Strengths:
Suggestions for Improvement:
Evaluator Signature:
Resident Signature:
Summary
Must have a documented assessment of
surgical competency.
OR
Simulation
Models
“Validate numbers”
ACGME Outcome Project
Interpersonal & Communication Skills
Jessica Bienstock, MD, MPH
Residents must be able to demonstrate
interpersonal and communication skills that
result in effective information exchange and
teaming with patients, their patients families,
Interpersonal and Communication Skills
and professional associates.
Residents are expected to:
1. create and sustain a therapeutic and ethically sound relationship
with patients
2. use effective listening skills and elicit and provide information
using effective nonverbal, explanatory, questioning, and writing
skills
3. work effectively with others as a member or leader of a health
care team or other professional group
ACGME
Teaching Interpersonal and
Communication Skills
Content Setting
Clinical teaching
Communication with
Role modeling
Patients and Families
Case based teaching
Interactive workshops or
seminars using role-
plays
Teaching Oral Communication Skills
Med Teams
Team training for all staff on L&D
Didactics and role playing exercises
Continuing assessment of skills through
Professional Associate Questionnaires
Teaching Oral Communication
Standardized Communication Exercises
Consent & Disclosure
Breaking Bad News
C-FAC form
Frequency
SCE = 3 scenarios/yr
Assessing Oral Communication Skills
360º Assessment
Professional Associate Questionnaire
Patient Assessment of Communication Skills (Patient Satisfaction
Questionnaire)
Evaluation by
• Medical Students
• Junior Residents
• Senior Residents & Faculty
Frequency
Patient Assessment – 3 per ½ day clinic
Professional Assessment – Every rotation (q 6 wks)
Assessing Interpersonal and
Communication Skills
Content Setting
Scholarly Grand Rounds
communication Presenting
lectures/seminars/
conferences
Writing abstracts
Presenting a
poster
Scholarly articles
Oral Communication
Evaluation of Grand Rounds Talks
Evaluation of M&M Conference
6-8 evaluations over the course of 4 yrs
45 evaluations per presentation
Teaching Written Communication
Skills
Didactic session on “How & What to
Chart & Dictate – Effective Medical
Communication”
Developed by Residents/Faculty/Patient
Safety Nurse
Assessment of Written
Communication
Charts reviewed by Patient Safety Nurse
Legibility
Completeness
Feedback & suggestions for improvement provided
Publications
Book Chapters
Abstracts & Papers
1 on 1 mentoring / editing from faculty
Medical Records
Weekly report produced by medical
records dept and sent to Chair‟s office
>5 records not dictated results in an e-
mail to the resident and, if not corrected
immediately, a note to their learning
portfolio
Summary
Interpersonal & Communication Skills
Valid tools have been developed
Need to use multiple tools & repeat
assessments
Need input from more than just your faculty
ACGME Outcome Project
Professionalism
Mary Ciotti, MD
Professionalism
Resident must demonstrate professionalism as
manifested through a commitment to carrying
out professional responsibilities adherence to
ethical principles and sensitivity to a diverse
patient population
Professional behavior
Ethical principles
Cultural competence
Professionalism
Professional Behavior-Education
Clinical teaching
Expectations and measures
Case based teaching
Mentoring
Role modeling
Role plays and clinical vignettes
Professionalism
Professional Behavior-Evaluation
Global assessment
360 degree evaluation
Patient evaluation
OSCE
Portfolio
Self reflection
PROFESSIONAL ASSOCIATE QUESTIONNAIRE
The Department of OBG places a high value on impersonal skills and humanistic health care. In order to assess these qualities in our housestaff
we are asking you to complete this form for the resident named below. We are interested in your perception of their behavior during their last
rotation relationship with other professionals and with patients and family members.
Resident Name________________________________Date_____________________
Please indicate the clinical setting you have interacted with the resident ___Labor and Delivery___ Clinic_____Inpatient
On average how many clinical observations did you have of the resident? 20 _____
Professional role: □Nurse □ Resident □Attending □Med. Student □ Other_______________
Unsatis Satisfa Excelle Unable
factory ctory nt assess
1. Communication Co mmunicates clearly, is willing to answer questions and provide explanations, willing to listen to patients 1 2 3 4 5 6 7 8 9
and families
Consistently demonstrates willingness to listen to nursing and allied staff 1 2 3 4 5 6 7 8 9
2. Respectfulness Treats others with respect does not demean or make others feel inferior, Provides equitable care to patients, 1 2 3 4 5 6 7 8 9
uses respectful language when discussing patients, is sensitive to cultural needs of patients.
Consistently courteous and receptive to nursing and allied health staff, acknowledges and respects roles of 1 2 3 4 5 6 7 8 9
other health care professionals in patient care.
3. Compassion Is kind to patient and families, appreciates patients and families special needs, and accepts inconvenience 1 2 3 4 5 6 7 8 9
when necessary to meet the needs of the patient, consistently attentive to details of patient comfo rt
4. Reliability Co mpletes and fulfills responsibilit ies, responds promptly when on call or when paged, assists and fills in for 1 2 3 4 5 6 7 8 9
other when needed.
5.Honesty/Integrity Knows limits of ability and asks for help when appropriate, is honest and trustworthy, does not falsify 1 2 3 4 5 6 7 8 9
informat ion, co mmitted to ethical principles
6. Responsibility Accepts responsibility (does not blame others or the system), co mmitted to self-assessment, responds to 1 2 3 4 5 6 7 8 9
feedback, co mmitted to excellence and self- learn ing.
7. Altruism Places patients needs at the forefront of care. 1 2 3 4 5 6 7 8 9
8. Advocate An advocate for patient needs, effectively accesses and coordinates resources to optimize patient care, seeks 1 2 3 4 5 6 7 8 9
to find and correct system causes of medical error.
Please provide comments concerning the resident’s relationships with patients, families and other healthcare professionals
Patient Satisfaction Questionnaire
Please complete the following questionnaire about your recent visit with your doctor.
Your answers will be confidential.
Name of Physician ____________________________Date _______________________
HOW IS THIS DOCTOR AT……… Poor Fair Good Excellent
Telling you everything; being truthful, upfront and frank;
not keeping things from you that you should know
Greeting you warmly: calling you by the name you prefer;
being friendly never crabby or rude
Treating you like you’re on the same level: never talking
down to you or treating you like a child
Letting you tell your story: listening carefully: asking
thoughtful question: not interrupting you while you are
talking
Showing interest in you as person; not acting bored or
ignoring what you have to say
Warning you during the physical exam about what he/she is
going to do and why: telling you what he/she finds
Discussing options with you: asking you opinion: offering
choices and letting you help decide what to do: asking what
you think before telling you what to do
Encouraging you to ask question: answering them clearly:
never avoiding your questions or lecturing you
Explaining what you need to know about your problems,
how and why they occurred and what to expect next
Using words you can understand when explaining your
problem and treatment: explaining any technical medical
terms in plain language.
Comments:
Project Professionalism- ABIM - 1999
360 degree evaluation
Nursing staff
Peers Students
Resident
Faculty Self
PRAIS E CARD
To: Faculty
Fro m Program Director
Re: Praise card about Performance of Resident
Please comp lete and submit this card to me when you wish to praise the performance and/or profess ional
behavior of a resident. This info rmation will be conveyed to the physician and noted in the departmental file.
Name of Physician:_________________ PGY_______________Date__________________
My praise about the performance of this physician is based on his/her demonstration of exceptional ability in the
following:
Please (X)
______Clin ical Judgement _____Professionalism
______Clin ical Skills _____Humanistic qualities
______Medical Knowledge _____Team management
______Communication Skills _____Team leadership
______Teaching ______Research
Co mments:
Name:________________________________ Phone_______________________
Behavior-specific
Punctuality
Appearance
Honesty/Trust
accountability /trustworthiness
Compulsiveness
Responsibility/Sense of Duty
Response to Criticism
Initiative
Self regulation /ability to self assess
Respect for others
Interprofessional relationship
Reflection
Critical Incident
Concern card
Remediation-How?
Behavior identified
Discussed at teaching committee
Plan made
Discussion with resident
Plan developed/Desired outcome determined
Self reflection
Portfolio
Meetings set
Follow up
Professionalism
Ethics based practice-Education
Clinical teaching
Case based teaching/reading/discussion
Ethics committee
Ethics lecture/case interactive session/
Humanistic seminars-self reflection
Mentoring/Role modeling
Professionalism
Ethics Based practice-Evaluation
Simulation cases OSCE
360 degree evaluation
Portfolio/Self reflection exercise
Patient evaluation
Cultural Competence
Clinical teaching
Cased based teaching
Interactive workshops
Lecture/conference/seminar
Institutional initiative
Role modeling
Mentoring
Professionalism
Cultural competence-Evaluation
360 degree evaluation
Portfolio
Self reflection
OSCE
Patient evaluation
Evaluating professionalism
Assessment of Program
Professionalism climate
Evaluation of environment from students,
and visiting resident/interns
Feedback from Faculty
Feedback from residents re:
environment and education while on a
service
ACGME Outcome Project
Practice-Based Learning & Improvement
Lee Learman, MD, PhD
C. Practice-based Learning
and Improvement
1. Self-assessment and life-long learning
2. Quality improvement
3. Evidence-based practice
4. Teaching skills
1. Self-assessment & LLL
Mentorship, including mentor selection,
faculty development, mentoring process
and topics.
Identify specific ways in which the
program fosters self-reflection, self-
assessment and practice improvement
for residents.
Use of individualized learning plans for
residents.
Mentoring Roles: 1 person or 3?
Academic Progress (summative)
semi-annual meetings, advancement,
disciplinary issues, etc.
Career Development (formative)
coach, advocate, professional society
sponsor, advisor for career decisions
Research Mentor
specifically skilled to help with project
Self-reflection, Self-assessment
An Evaluation Warehouse
Reflection is scored as
evidence of practice-
based learning &
Resident receives, improvement
instructions and selects
experience
Discusses reflection
Reflects and
with mentor
summarizes
Reflection Assignments
Select one experience that taught you the most about
[insert name of competency]
• Critical appraisal of the literature
• Practice improvement
• Communication, professionalism, SBP, etc…
Write a short letter for each reflection
• Detailed vignette, linkage to past/future
Provide evidence of learning
• Journal club review or clinical topic review log
• Practice or systems improvement project
• Note from a patient or colleague
Scoring „Rubric‟
more
superficial
1. Lists evidence of progress from experience
2. Evaluates evidence of progress from experience
3. Monitors evidence of progress
deeper
Professional Development of Reflective Ability. (Davis, personal communication).
UCSF Adaptation
1. Describes encounter only.
more
superficial 2. Unsupported opinions about lessons learned.
3. Superficial justification of lessons learned.
4. Discussion well-supported with examples of
challenges, techniques and lessons learned.
5. Analyzes factors from experience that contribute
to progress.
6. Justifies strategies used and evidence for
deeper effectiveness.
Learman LA, O‟Sullivan P. Does Reflective Ability Grow with Experience and Vary
by Skill? Oral presentation. AAMC Annual Meeting 2006.
Depth of Reflection: Description
• This was a valuable learning experience to
me to not only have continuity of care . .
.but also to develop patient-physician
relationship with her (Communication
skills).
• Based on the articles reviewed we knew that
there was insufficient data to make reliable
conclusions about maternal morbidity and
mortality (Critical appraisal skills).
Depth of Reflection: Analysis
• My natural inclination was to feel frustrated and
defensive when this patient was so angry with me from
the beginning, and I know I can be more empathetic
(Communication skills).
• This was one of the first times that I felt there was so
much discrepancy that there was no right answer. This
was also the first time the way I managed this topic in
practice changed so much after reviewing the evidence.
This experience further solidified the importance of
continuing evidence-based practice when I leave my
training (Critical appraisal skills).
Individualized Learning Plans
Setting: semi-annual meeting
Procedure: in advance the resident . . .
• reviews the evaluation data
• completes reflection exercises
• defines strengths and improvement goals in each area
of competency
Feedback: mentor makes sure assessment and
learning goals are appropriate
Product: 6-month learning plan
2. Quality Improvement
List the activities in which residents
actively participate to learn and apply QI
principles, and identify those who
oversee these activities.
Example of a QI activity/project that
residents have been involved: its
development, goal, implementation, and
evaluation of success.
Active Participation in QI
Leads M&M presentations using a
systematic approach
Unit, department or hospital QI meetings
Mentored project evaluating current
quality indicators or an intervention to
improve them, e.g.:
C/S rate, surgical site infection rate
Post-operative pain management
Pap smear and mammogram ordering
3. Evidence-based Practice
How are residents taught to examine
their current patient care practices in the
context of scientific evidence and how
does the program instill this skill as a
habit of practice?
Use a specific clinical example to
illustrate how the program meets this
requirement.
PBLI: Evidence-based
Practice
CREOG
Clinical Topic Review Log Competency
Task Force
Name ___________________________________
Role (Student/Resident)_____________________
Date
Do you plan
Search Terms, to change
Access Resources (MeSH headings, your
Clinical (MedLine, Cochrane, Methods of Limiting Level of practice?
Question (s) Textbook, on-line ref) Literature etc.) Evidence What was learned How?
I Large randomized trials with clear-cut results (and low
risk of error)
II Small randomized trials with uncertain results (and
moderate to high risk of error)
III Nonrandomized trials with concurrent or
contemporaneous controls
IV Nonrandomized trials with historical controls
V Case series with no controls
Modified by UCSF Ob/Gyn from the
Critical Appraisal Exercise (PBLI) CREOG Competency Task Force
Name of Presenter _____________________________________________
Date _____________________________
Clinical Search Strategy and Search Terms Magnitude of Expected Effect Level of
Question(s) (MedLine, Cochrane, Textbook, on-line ref) (e.g. # needed to treat for benefit & harm) Evidence*
What was learned Do you plan to change your practice? How?
*LEVEL OF EVIDENCE
GOOD Large randomized trials with clear-cut results (and low risk of error)
FAIR Small randomized trials with uncertain results (and moderate to high risk of error)
or nonrandomized trials with concurrent or contemporaneous controls
POOR Nonrandomized trials with historical controls or case series with no controls
Goal Met
1 - Refined question to be clinically focused and relevant Yes No
2 - Used logical, focused search strategy Yes No
3 - Summarized findings into clinically relevent metric (e.g. NNT) Yes No
4 - Considered health policy implications of findings (e.g. feasibility, cost, harms) Yes No
5 - Discussed limitations of current evidence Yes No
6 - Discussed areas of future research Yes No
Evaluator ____________________________________ Signature _______________________________
4. Teaching Skills
Describe how residents learn teaching
skills, opportunities for teaching, and
how the residents‟ skills are assessed,
including specific tools if used.
Indicate (yes/no) whether residents
participate in the teaching of other
Ob/Gyn residents, residents in other
specialties, medical students, nurses,
other medical personnel.
Teaching Residents to Teach
New intern orientation
Resident core lectures, retreats
Institutional programs, “fellowships”
CREOG/APGO Resources
REFEREE
Resident Workshops
Observation and feedback
Workshops
Real-time
Resident Educators/Faculty Educators:
Recommendations for Educational Excellence
(REFEREE)
Resident Educators/Faculty Educators: Recommendations for Educational
Excellence (REFEREE) is a new program that is being developed by the
Committee on Resident Education in Obstetrics and Gynecology (CREOG) and
the Association of Professors in Gynecology and Obstetrics (APGO).
The first module in this highly interactive online program is a course on how to
provide feedback. It includes virtual situations where the user, who may be a
resident, medical student, fellow, or faculty member, reviews interactions and
techniques presented in slice-of-life scenarios. Options are provided for answering
questions, giving input, challenging assumptions, and suggesting solutions.
REFEREE provides answers or suggestions for improvement to the user, and then
moves on to new scenarios for learning as previous ones are mastered.
Planning is underway to secure funding for additional modules.
http://www.ja-online.com/referee
.
CREOG LEADERSHIP WORKSHOP FOR
RESIDENTS
The purpose of this program is to equip senior residents with the knowledge and
skills necessary to serve as leaders and exemplary teachers for the residents and
medical students for whom they are responsible. It is expected that by the end of the
program participants will have learned general principles of leadership and education,
as well as specific tips for immediate practice and application.
TOPICS TO BE COVERED
• Role of the Resident
• Establishing Climate
• Physician as a Coach
• Clinical Teaching Skills
• Leadership/Working Styles
• Creating Productive Teams
• Best Practices
• Teaching in the Ambulatory Setting
• Giving Feedback on Performance
• Conflict Resolution: Acting as a liaison between faculty and residents
• Preparing for Life after Residency-Practice and Fellowship issues addressed
University of California, San Francisco
OB-Gyn Clerkships
Analysis of Educator Performance
Evaluation Type: Medical Student of Resident
Time Period: 11/24/2006 to 02/24/2007
Time Period Type: Request Date
Report Date: 02/24/2007
Expanded View To Educator Comments To Time Series
Question ID Question Zero Count Applicable Answers Mean Scale Std
Conveyance of Information
29551 Convey information clearly. 0 116 4.72 1 to 5 0.54
Teaching Enthusiasm
Provide enthusiastic and stimulating
29552 teaching. 0 116 4.72 1 to 5 0.54
Approachability
29553 Be approachable and available. 1 115 4.72 1 to 5 0.54
Direction and Feedback
29554 Provide direction and feedback. 6 110 4.55 1 to 5 0.66
Caring/Ethical Behaviors
29556 Model caring and ethical behavior. 1 115 4.87 1 to 5 0.34
Cultural Sensitivity
29557 Model culturally sensitive patient care. 8 108 4.89 1 to 5 0.32
Promotion of Critical Thinking
29558 Promote critical thinking. 3 113 4.6 1 to 5 0.58
Observation
My preceptor/attending observed me with
29559 patients 12 104 4.55 1 to 5 0.67
Treat me with Respect
29560 I was treated with respect by this individual 0 116 4.91 1 to 5 0.35
Treat Others with Respect
I observed others (students, residents, staff,
patients) being treated with respect by this
29561 individual 0 116 4.9 1 to 5 0.38
Teaching Skills, Overall
29562 Overall teaching effectiveness. 2 114 4.67 1 to 5 0.56
UCSF Academy of Medical Educators TOP Observation Form: Lecture/Seminar
NAME: _____________________________________ OBSERVER: _________________________________
TOPIC: _____________________________________________________________________________________
FOCUS OF OBSERVATION (discuss w/ mentee in advance): __________________________________________
____________________________________________________________________________________________
INTRODUCTION OBSERVATIONS
1. Introduced topic, stated objectives, offered preview.
2. Gained attention and motivated learning.
3. Established climate for learning and for participation
BODY OF LECTURE OBSERVATIONS
1. Presented 3-5 main points in clear and organized fashion.
2. Provided supporting materials, examples, and summaries
3. Content level
4. Effectively used visuals, handouts, and/or demonstartions. Include AV
problems (if any), effective use of slides (set stage for each slide,
focused audience on important parts of slides), use of pointer.
5. Varied presentations (used blackboard, slides, visuals)
6. Transitions between topics
CONCLUSION OBSERVATIONS
1. Summarized major principles, key points without introducing new
materials.
2. Provided closure or stimulated further thought
TEACHER DYNAMICS OBSERVATIONS
1. Exhibited enthusiasm and stimulated interest in content.
2. Used appropriate voice, gestures, movement, and eye contact.
Avoidance of unconscious use of repeated words (eg “um”,
“ok”).
3. Encouraged active participation.
4. Used questions to stimulate thought and discussion.
5. Response to questions (repeated or rephrased question, concise
answer).
PBLI Key Points
1. Self-assessment and life-long learning
• Mentorship, reflection, individualized learning plans
2. Quality improvement
• Active participant in supervised QI, specific project example
3. Evidence-based practice
• Examine patient care in the context of scientific evidence, specific
clinical example
4. Teaching skills
• Describe how teaching skills are learned and assessed (tools), list
who residents teach
ACGME Outcome Project
Systems-Based Practice
Rebecca McAlister, MD
D. Systems-Based Practice
1. Teaching and Evaluation
Address how the elements of SBP are
TAUGHT by describing content and methods
of teaching.
SBP Content Areas
Business of Medicine
Medico-legal aspects of medicine
Cost effectiveness
Patient advocacy / patient safety
Team building / team leading
Root cause systems-based analysis of
error
Business of Medicine
Billing and coding curriculum
Practice management seminars
HIPAA training
Medico-legal Aspects of OBGYN
Risk management discussions/ mtgs
Exercises in proper documentation/ charting/
dictations
Mock depositions / trials
Practice management discussions of types of
professional liability insurance
Cost Effectiveness
Reference listings of costs of common:
Drugs
Tests
Procedures
Case based discussions of alternative tx
plans
Pt Advocacy / Pt Safety
Didactic sessions on legislative efforts in
Women‟s Health issues
Discharge planning with interdisciplinary teams
JCAHO training
Didactic sessions on physician fatigue /
sleepiness
Team Building / Leading
Leadership skills for residents
Discharge planning with interdisciplinary
teams
Interdisciplinary conferences
Root Cause Analysis of Error
2. Reflective Practice
Describe resident activity (in)
experimental learning to address
system-causes of error
State who guides / supervises in this
activity
Critical Incident
Hypothetical or real clinical scenarios
Anonymous or personal involvement
Analyze system based issues
Devise safeguards to prevent
Review by attending and place in portfolio
ANALYSIS OF MEDICAL ERROR
The resident is to identify an adverse patient outcome or a “near miss” event that
would have led to an adverse outcome. The resident then analyses the event from
a systems based approach, identifying the “latent” causes for error in the systems
that allowed the individuals involved to err. The resident then identifies possible
changes that could be made in the system to prevent a recurrence of a similar
adverse event. The resident may utilize a “fishbone” diagram to organize the
analysis of the medical error. The resident may either submit a written summary
of the above analysis to be included in the portfolio or present the analysis in a
departmental conference such as Morbidity and Mortality.
PROCESS PEOPLE
ADVERSE
PATIENT
OUTCOME
UNDERLYING
DRIVERS
COUNTERMEASURE
POLICIES ENVIRONMENT
ACGME Outcome Project
Faculty Development and Evaluation
Mark Woodland, MS, MD
Outcomes Project
Faculty Education
Program Level
Program Annual Review
Departmental Meetings
• Faculty Meetings
• Education Committee Meetings
• Resident Education Meetings
Institutional Level
GMEC Emphasis
Internal Review Process
Institutional Meetings
Evaluation Tools Development & Utilization
Faculty Development Outcomes Project
Why competency based education initiative?
What is the outcomes project and how does it
affect us?
How to we evaluate resident competency?
How do we measure achievement of
competency?
How do we change or modify our programs to
produce better outcomes?
What are our next steps?
Example: Development of A
Competency Based Curriculum
Work directly with our residents and faculty.
Critically Assess each rotation/education
experience
Developed a template for each rotation
Re-organize goals and objectives in the 6 areas of
competency
Developing assessment methods in achievement of
competency specific to goals an objectives
Resources from the ACGME Website
Educating Physicians for the 21st Century
“Educating Physicians for the 21st Century,” a series of five PowerPoint presentations with a Facilitator‟s Manual, is a new
educational resource from the Research and Education Department. The Facilitator‟s Manual contains speaker notes and
discussion questions to help Program Directors give these presentations during faculty meetings or educational retreats. If you
wish to provide feedback, click here.
Educating Physicians for the 21st Century - Introduction (web-based learning module)
Facilitator's Manual
Module 1 - Introduction to Competency-Based Resident Education (PowerPoint)
Facilitator's Manual
Module 2 - Practical Implementation of the Competencies (PowerPoint)
Facilitator's Manual
Module 3 - Developing an Assessment System (PowerPoint)
Facilitator's Manual
Curriculum Template (MS Word)
Module 4 - Developing a Competency-based Curriculum (PowerPoint)
Coming Soon:
Facilitator's Manual
Module 5 - Educational Quality Improvement (PowerPoint)
ACGME Curriculum Template
Description of Rotation or Educational Experience
Insert brief description of rotation or educational experience
Include rotational goals
PATIENT CARE
Goal
Residents must be able to provide patient care that is compassionate, appropriate, and effective for the treatment of health
problems and the promotion of health. Residents are expected to:
Competencies
[Insert specialty specific requirements]
Objectives
Insert specialty specific requirements or insert measurable objectives that align with specialty specific curriculum
MEDICAL KNOLEDGE
Goal
Residents must demonstrate knowledge of established and evolving biomedical, clinical, epidemiological, and social-behavioral
sciences, as well as the application of this knowledge to patient care. Residents are expected to:
Competencies
[Insert specialty specific requirements]
Objectives
Insert specialty specific requirements or insert measurable objectives that align with specialty specific curriculum
Teaching Methods
What teaching methods are you using on this rotation or educational experience?
Assessment Method (residents)
How do you measure the resident’s performance on this rotation or educational experience?
Example: Implementation Competency
Assessment Tools
Annual Program Evaluation
Fully Implemented
Global Assessment Tools
Fully Implemented
Focused Assessment Tools
Partially Implemented
360 Degree Assessment Tools
Under Construction
Partially Implemented
Systems Based Feed Back Tools
Partially Implemented
ACGME Curriculum Template
Description of Rotation or Educational Experience
Insert brief description of rotation or educational experience
Include rotational goals
PATIENT CARE
Goal
Residents must be able to provide patient care that is compassionate, appropriate, and effective for the treatment of health
problems and the promotion of health. Residents are expected to:
Competencies
[Insert specialty specific requirements]
Objectives
Insert specialty specific requirements or insert measurable objectives that align with specialty specific curriculum
MEDICAL KNOWLEDGE
Goal
Residents must demonstrate knowledge of established and evolving biomedical, clinical, epidemiological, and social-behavioral
sciences, as well as the application of this knowledge to patient care. Residents are expected to:
Competencies
[Insert specialty specific requirements]
Objectives
Insert specialty specific requirements or insert measurable objectives that align with specialty specific curriculum
Teaching Methods
What teaching methods are you using on this rotation or educational experience?
Assessment Method (residents)
How do you measure the resident’s performance on this rotation or educational experience?
Introduction to Competency-Based
Residency Education
Joyce, 2006
Objectives
Upon completion of this module, Program Directors and
faculty will be able to:
State foundational concepts of the Outcome Project
State the requirements related to the general
competencies in the Common Program Requirements
Describe expectations of the Outcome Project timelines
What is Competency-based Education?
Competency-based education focuses on learner
performance (learning outcomes) in reaching specific
objectives (goals and objectives of the curriculum).
Developing an Assessment System
B. Joyce, PhD 2006
Objectives
Identify components of an assessment system
Identify important criteria for an assessment system
Identify a set of assessment tools for use in your
program
Why assess?
Whatever we measure, we tend to improve
D. Leach
Practical Implementation of the
Competencies
B. Joyce, 2006
Objectives
• Identify examples of educational content in your
program for each of the competencies
• Identify settings currently used in your program
to teach each of the competencies
• Identify an educational improvement plan for
teaching the competencies
Developing A Competency-based
Curriculum
B. Joyce, Ph.D.
Objectives
• Develop a competency-based curriculum for a
rotation or educational experience
• Develop goals and objectives for the rotation or
educational experience
• Appreciate the importance of linking objectives
with assessment methods
Six Steps to Developing a Competency-
based Curriculum
1. Conduct needs assessment
2. Identify competencies addressed by this rotation
or experience
3. Write goals and objectives
4. Determine teaching methods
5. Determine assessment methods
6. Determine program improvement methods
Faculty Development
We have found…..
This is the most difficult issue for us to date.
It has been important for us to address from both the
departmental and institutional level through…
• Internal Reviews
• GMEC Initiatives
• Institutional Initiatives
o 360 Degree Evaluation
• DrexelMed GME Journal
ACGME Outcome Project
Portfolios – What Might We Expect from
the ACGME?
Tina Foster, MD, MPH
ACGME and Portfolios
Moving ahead with design of web-based
portfolio
Will start “alpha test” this spring – small
scale use in several specialties
Will scale up following this to larger pool
of “beta test” sites
As part of this team, I REALLY DO want
your input.
Why a Portfolio?
• Need to shift accreditation from process to
outcomes
• Need for developmental assessment
• Need for national assessment tools
• Need to reduce or at least understand burden
• Need to foster better conversation
• Need to collaborate with other organizations
• And TO FOSTER BETTER LEARNING (for
residents, programs, institutions, and RRCs)
So What Does All That Mean?
One place for record of resident experience:
evaluations, case logs, other work. Portions of
record can move forward with resident after
completion of training.
With a better record of what residents have
done and how they have demonstrated
competency, PDs can more easily assess their
program outcomes and DIOs their institutional
outcomes. RRCs can benefit from aggregated
data; aspects of portfolio could replace some of
PIF narrative
My Experience
Primarily in an unusual combined Prev Med
residency, but many aspects generalizable to
our ob-gyn program
Evaluations – similar benefits to other systems
Collection of resident work and prompts for
reflection/evaluation – residents like having a
record of what they have done and appreciate
the feedback
Threaded discussion/critical incidents can be
very powerful
Known Challenges
No added burden – or if burden is added
here, where do we take it away?
If we have to add a little burden…how do
we know it‟s worth it? What‟s the added
value?
Link to post-residency activities will be
important - -this is a “lifelong” portfolio
PIF Section 998/999
Mentoring – easy to show that residents have mentors; process
and level of activity visible
Individualized learning plan can be aprt of portfolio; who looks
at it and when they are updated easy to see
QI activities: demonstrated by QI learning activity in portfolio
Evidence-based practice: ditto
Teaching Skills: record of resident teaching activites; evaluation
and reflections
Evaluation of professionalism: 360 instrument in portfolio
Reflective practice: critical incidents or threaded discussion,
prompts for reflection in portfolio
Work in interdisciplinary teams, errors and systems – “product”
in portfolio
Evaluation distribution and collection
Review of performance