360 Rating Form 360 Evaluation Instrument 360 Assessment Tool by ezm24188


									    360 Rating Form
360 Evaluation Instrument
  360 Assessment Tool
                A Word of Caution
                in the First Steps!

1/24/2007            Malakan Rad      2
                 The elements are reliability
               and responsibility, honesty and
           integrity, maturity, respect for others,
           critique, altruism, interpersonal skills,
     and [absence of ] impairment (i.e., psychological/

1/24/2007                 Malakan Rad                     3
   humanistic qualities; honor, integrity,
      ethical and moral standards;
     excellence; and duty/advocacy.

1/24/2007          Malakan Rad               4
                    demands that the best interests
            of patients, not self-interest, guide physicians.
                           Honor and integrity
                entail the highest standards of behavior
                    and the refusal to violate one’s
                   personal and professional codes.
                     Duty is the free acceptance of
                         commitment to service.

1/24/2007                     Malakan Rad                       5
    Society expects physicians to act professionally.
                        In response
 to recent criticism regarding unprofessional behavior
                      in medicine,
  some argue that improving medical professionalism
   can only occur through changes in teaching and
                      .assessing it

1/24/2007               Malakan Rad                      6
       There are at least three types of
     studies, however, that may point the
       way for future evaluation thrusts.

1/24/2007           Malakan Rad             7
              work evaluates professional behavior as
                    part of clinical performance.
            studies evaluate only professional behavior,
                    as a comprehensive entity in
                             and of itself.
                    Still other research evaluates
              single elements of professional behavior
                such as humanism, self-assessment,
                   dutifulness, altruism, empathy
              and compassion, honesty, integrity, and
            ethical behavior, as well as communication.

1/24/2007                    Malakan Rad                   8
   Although assessing professionalism poses many
    challenges,gauging and detecting changes in
professionalism is impossible without measurement.
           There was a review of techniques
  used to assess professionalism during the past 20

1/24/2007              Malakan Rad                    9
  The authors searched five electronic databases and
                    reference lists
                  from 1982 to 2002.
     Eighty-eight assessments were retained and
                     organized into :

1/24/2007              Malakan Rad                     10
1/24/2007   Malakan Rad   11
            content area:
3- comprehensive professionalism
4- diversity

1/24/2007       Malakan Rad        12
Diversity pertained to assessments that addressed cultural
issues, socioeconomic status, gender, age or disability.
    Validity and reliability were examined for five of 10

1/24/2007                Malakan Rad                     13
      Type of outcome examined:
2- cognitive
3- behavioral

1/24/2007       Malakan Rad       14
                           personal        comprehensive
                ethics                                       diversity
                         Characteristics   professionalism





1/24/2007                 Malakan Rad                                    15

Forty-nine assessments addressed various aspects of
 ethics such as morality, ethical principles, honor
                   codes, social
 norms, deception, abuse or mistreatment, cheating,
               and sexual misconduct.

1/24/2007             Malakan Rad                 16
            Personal characteristics
          The category labeled ‘personal characteristics’ included
        assessments that addressed attributes such as emotional
          intelligence, personal values, empathy, trustworthiness,
       cynicism and dogmatism. All of the personal characteristics
     instruments assessed affective outcomes. Validity and reliability
                    were examined for 13 of 15 (Table 1a).

1/24/2007                       Malakan Rad                              17
   Balance between commitment at hospital and commitment at

1/24/2007                  Malakan Rad                        18
1/24/2007   Malakan Rad   19
                              Tip 1
     Build a positive view of the importance of training in
      attitudes with all those involved in the curriculum

1/24/2007                   Malakan Rad                       20
                               Tip 2
            Use examples of good practice from elsewhere

1/24/2007                     Malakan Rad                  21
                             Tip 3
  Ensure that professional development objectives are set
                   the training curriculum

1/24/2007                 Malakan Rad                       22
                               Tip 4
            Use learning methods that match the objectives

1/24/2007                      Malakan Rad                   23
                         Tip 5
            Tutors must be good role models

1/24/2007              Malakan Rad            24
                             Tip 6
    Create a learning culture that examines shortcomings

1/24/2007                 Malakan Rad                      25
                          Tip 7
            Use peer influence systematically

1/24/2007               Malakan Rad             26
                             Tip 8
            Assess the learning objectives formally

1/24/2007                  Malakan Rad                27
                            Tip 9
 Support the learner in vulnerability and strengthen coping

1/24/2007                 Malakan Rad                     28
                           Tip 10
        Reward good performance by students and staff

1/24/2007                 Malakan Rad                   29
                          Tip 11
            Secure necessary resources and skills

1/24/2007                 Malakan Rad               30
                              Tip 12
            Use levers for change within the curriculum

1/24/2007                    Malakan Rad                  31
                          As the saying goes,
   ‘‘It is easier to kill a man than to change his ways’’. The
      hopes these tips may be useful to those who wish to
                 preserve life in all its fullness.

1/24/2007                   Malakan Rad                          32
  recommendations for

1/24/2007   Malakan Rad   33
            •Encourage formative and
                   Start early
            Summative assessment, often undertaken to classify learners,
             dominates medical education. Using assessment to improve
                learning is gaining credibility, however, with reports of
            formative assessments appearing more often in the literature.
             Some evidence suggests that focused teaching can improve
                 moral reasoning (Self et al., 1992; Goldie et al., 2002);
              more specifically, that feedback derived from assessment
               may improve professional behaviors (Phelan et al., 1993;
                  Papadakis et al., 2001). These findings suggest that
            professionalism should be formatively assessed. This means
              that assessment should begin early (Lowe et al., 2001) be
                 conducted frequently, be implemented long term, and
               provide learners with opportunities to change (Van Luijk
                                      et al., 2000).

1/24/2007                            Malakan Rad                             34
           The educational environment, whether through formal
        or informal curricula, appears to influence learner attitudes
       and behavior (Stern, 1996). One study discerned relationships
          between the ethical environment and medical students’
         ethical behavior (Feudtner et al., 1994). In another study,
         residents reported learning most about professionalism
            from observing role models (Brownell & Cote, 2001).

1/24/2007                        Malakan Rad                            35
    Research suggests that the business (Freeman et al., 1999)
          and cultural environment (Hoffmaster et al., 1991)
      influence professionalism among practicing physicians.
     Consequently, assessments that gauge professionalism in
   the environment may provide insight into the professionalism
                            of individuals.

1/24/2007                   Malakan Rad                           36
          Systematic assessment of professionalism should also
       include many different assessors, more than one assessment
       method and assessment in different settings (Littlefield et al.,
             1996; Van Luijk et al., 2000; Ginsburg et al., 2002).
         Research indicates that different assessors offer different
         perspectives, thus enhancing the breadth of assessment
         (Wooliscroft et al., 1994) and multiple assessors enhance
                        reliability (Swanson, 1987).
                       Each assessment method has
                        strengths and weaknesses.

1/24/2007                        Malakan Rad                              37
              Rating forms are considered
relatively easy to use, but are plagued with the ‘halo’ or
               ‘horns’ effect (Gray, 1996).

1/24/2007                Malakan Rad                    38
            THE HALO EFFECT
   The halo effect results when a supervisor
              overrates employee by:
• Failing to see deteriorating performance
  because of a good past record or personal
• Rewarding those with similar beliefs or
  background as his or her own.
• Believing that, because performance is
  outstanding in one area, there are no
  problems in other areas.
1/24/2007           Malakan Rad                39
            THE HORNS EFFECT

   The horns effect results when a supervisor
         underrates employees who:
• Does not meet the impossible high standards
  the supervisor sets.
• Achieves results by different methods that
  the supervisor uses.
• Fails at one thing – and find that this one
  instance is influencing the supervisor’s
  future evaluations.
1/24/2007          Malakan Rad              40
           Using more than
  one assessment method may help to
         compensate for the
 weaknesses associated with any single

1/24/2007        Malakan Rad         41
 Using more thanBecause professionalism is a
 complex construct, it is unlikely that a single
   assessment will adequately measure it.
Using a combination of assessments, however,
such as a moral reasoning assessment
together with a behavioral assessment,
may be adequate.

1/24/2007           Malakan Rad                42
               Different Settings
       Assessment in different settings can help to
        determine the generalizability of learners’
      and identify context specific issues relevant to
       Suggestions for assessing medical students,
                residents, and practicing
              physicians Medical students.

1/24/2007                 Malakan Rad                    43
                         Assessment of medical
       student professionalism is often delayed until clerkship
      rotations. The Physicianship Evaluation Form (Papadakis
       et al., 2001), however, indicates that it is both desirable
      and possible to begin assessing student professionalism
                 during the first year of medical school.

1/24/2007                      Malakan Rad                           44
                  Initial use of this
   assessment is formative; students are invited to
                   present their
perspective and data are used to provide feedback and

1/24/2007              Malakan Rad                  45
           Persistent patterns of unprofessional behavior,
      despite remediation, may provide grounds for dismissal.
    This performance-based, longitudinal approach helps to set
    professionalism expectations early, both for learners and for
     faculty who must commit to addressing and attempting to
              improve students’ professional behavior.

1/24/2007                     Malakan Rad                           46
Arnold suggested that assessments could be improved
   focusing on professionalism as a separate entity,
including rigorous qualitative approaches, exploring
                 the extent to which
      the environment supports assessment, and
                   determining the
usefulness of developmental expectations in assessing

1/24/2007              Malakan Rad                  47
                       Ginsburg and colleagues (2000)
        examined existing assessments in terms of types of raters,
                   such as faculty supervisors and peers.
                              They reasoned that
       difficulties involved in assessing professionalism are due to
  the frequent use of abstract idealized definitions, the context specific
    nature of professionalism, and evaluator reluctance to address
                          relatively minor lapses.
                         The same authors proposed
       that professionalism assessments should address cognitive
                          and behavioral outcomes.

1/24/2007                        Malakan Rad                                 48
                       For Students
                                Students should
       also be prepared to address ethical issues in clinical settings.
       Behavioral simulations, such as the Moral Behavior Analysis
          (Sheehan et al., 1987) and the Ethics OSCE (Singer et al.,
         1996), in which ethical dilemmas are role-played, are ideal
         for this purpose, especially if they include post-encounter
           oral or written assessments that probe reasoning used
           during encounters. To obtain reliable data, many cases
         would be required, thus, these simulations should be used
                        for formative assessment only.

1/24/2007                         Malakan Rad                             49
                   For Residents
         Residents typically rotate through many different
   settings and interact with a broad range of personnel. These
  characteristics of residency training may present challenges to
     coordinating comprehensive assessment and providing
   opportunities for the development of long-term relationships
               helpful to assessing professionalism.

1/24/2007                    Malakan Rad                            50
                  On the other hand,
  changing rotations provide the option of obtaining
                    input from
 various observers (e.g. nurses, patients, supervising
and peers), and assessing the extent to which resident
       professionalism varies across settings.

1/24/2007              Malakan Rad                   51
                               To assess residents’
        knowledge of professionalism and to stimulate instructional
            discussion, the Barry’s Challenges to Professionalism
                Questionnaire may be used (Barry et al., 2000).
                              This self-administered
               questionnaire consists of vignettes that address
        conflict of interest, gifts and physician impairment. Research
         results indicated that the questionnaire was able to detect a
            broad range of knowledge about professionalism and
           discriminate among respondents depending on years of

1/24/2007                         Malakan Rad                            52
            For Practicing physicians
  Assessments of ethics knowledge by surveys and of comprehensive
 professionalism by patient questionnaires were the most frequently
 used, and are probably the most feasible, approaches to assessing
              professionalism in practicing physicians.
         For instance, the Defining Issues Test, a written test in
     which ethical dilemmas are presented in vignettes, is capable
      of discriminating different levels of moral reasoning among
      practicing physicians (Baldwin & Bunch, 2000). The Barry’s
      Challenges to Professionalism Questionnaire would also be
               suitable for use with practicing physicians.

1/24/2007                     Malakan Rad                             53
Examples of patient questionnaires designed to assess
   practicing physician professionalism include the
    Scale (Hauck et al., 1990) and the Wake Forest
 Trust Scale (Hall et al., 2002). Both have yielded valid
                      reliable data.

1/24/2007               Malakan Rad                    54
     Patient questionnaires have advantages: they
   obtain information about physicians’ behaviors in
practice, they seem to be acceptable to physicians, and
     may prompt changes in behaviors assessed.

1/24/2007               Malakan Rad                   55
  At least 88 professionalism assessments have been
            in medical education since 1982.

1/24/2007              Malakan Rad                    56
                  For Student
    Useful approaches to assessing medical student
  professionalism include a longitudinal, performance-
        method and behavioral simulations with

1/24/2007               Malakan Rad                      57
               For Resident
 Useful approaches to assessing resident physician
professionalism include a 360-degree assessment and
     a cognitive assessment of professionalism.

1/24/2007             Malakan Rad                 58
            For Practicing Physician

 Useful approaches to assessing practicing physician
 professionalism include patient questionnaires and a
      cognitive assessment of professionalism.

1/24/2007              Malakan Rad                      59
   No single method exists for
     the reliable and valid
   evaluation of professional

1/24/2007     Malakan Rad        60
                      more studies
on the predictive validity of assessments and their use
                      as part of
   formative evaluation systems are recommended

1/24/2007               Malakan Rad                   61
        Instead of creating new
       assessments should be

1/24/2007        Malakan Rad      62
            A Word of Caution!

        The new
professionalism needs
   to be tackled at a
   number of levels.

1/24/2007          Malakan Rad   63
Teaching :unprofessionalism(by role
  Absolute unprofessionalistic act

1/24/2007      Malakan Rad        64
1/24/2007   Malakan Rad   65
1/24/2007   Malakan Rad   66


1/24/2007   Malakan Rad               67


1/24/2007   Malakan Rad               68

1/24/2007   Malakan Rad               69
                          0 degree


1/24/2007   Malakan Rad              70
1/24/2007     Malakan Rad   71

To top