Research Report on 360 Feedback in Bank by hli11975

VIEWS: 36 PAGES: 24

More Info
									                                               NHS




                360° Appraisal

      “To See Ourselves as
        Others See Us”

                Heather Ainscough
           Medical Education Manager
     Southport & Ormskirk Hospital NHS Trust




February 2004
                       ACKNOWLEDGEMENTS


I would like to thank Matthew Serlin, Director of Medical Education, and
Sharon Partington, Director of Human Resources, Southport & Ormskirk
Hospital NHS Trust, for allowing me the time to complete this project.

I would like to thank Jonathan Parry, Chief Executive and Michael Mason,
Trust Library Manager, for their encouragement. I would also like to thank
Delwyn Addis, Ann Allman and Sheila Rodd (my fellow colleagues on the
evidence based course) for all their cheerful support.

Thanks also to Dee Snape, Research and Development Co-ordinator,
Southport and Ormskirk Hospital NHS Trust, Barbara Jack and Janet Taylor,
Senior Lecturers at the School of Health Sciences, Edge Hill College.




                                                                        1
                               CONTENTS

Acknowledgements                                                 1

Contents                                                         2-3

Abstract                                                         4

1.    BACKGROUND                                                 5

2.    LITERATURE SEARCH                                          6

      2.1 Key Terms
      2.2 Databases
      2.3 Dates

3.    KEY THEMES                                                 7

4.    WHAT IS 360° APPRAISAL?                                    8

5.    MULTI-SOURCE FEEDBACK                                      9 - 10

      5.1   Number of Feedback providers
      5.2   Categories for performance
      5.3   Objective feedback
      5.4   Questionnaire based survey
      5.5   Changes in practice
      5.6   Enhancing departmental performance

6.    LEARNING OPPORTUNITIES                                     11 - 12

      6.1 Major benefit
      6.2 Encouraging openness
      6.3 Transferable skills

7.    DEALING WITH PERSONALITY CONFLICTS                         13 - 14

      7.1   Potential Bias
      7.2   Law suits
      7.3   Nurse study
      7.4   Feedback providers
      7.5   Employee behaviour
      7.6   Awareness of conflict

8.    EVALUATION OF SKILLS & COMPETENCIES                        15 - 16

      8.1   What to evaluate
      8.2   When to evaluate
      8.3   Why to evaluate
      8.4   Performance and competence (e.g. Clinical Skills Centre)
                                                                           2
9.    SCORING SYSTEM              17 – 18

      9.1   Behaviour
      9.2   Validity
      9.3   Law suits
      9.4   Reliability

10.   RESOURCES                   19 - 20

      10.1 Physical resources
      10.2 Software systems

11.   CONCLUSIONS                 21

      11.1 Perspectives
      11.2 Implementation
      11.3 Training individuals

12.   RECOMMENDATIONS             22

13.   REFERENCES                  23




                                            3
                                ABSTRACT


I attempt to identify an effective and constructive system of 360° appraisal as

a possible tool for assessing behavioural patterns. Also to determine how

people develop skills and competencies that are key to their role.        360°

appraisal is being considered as part of a robust system relevant to

‘Modernising Medical Careers’ and as part of a health care system

increasingly provided by multi-disciplinary teams.



The report details the results of a literature review looking into 360°

appraisal. The report looks at multi-source feedback, learning opportunities,

evaluation, personality conflicts, scoring systems and resources.



The review concluded that, while training individuals may temporarily

increase the expense associated with 360° feedback programmes, the gains

seem to outweigh the ‘cost’ of feedback to participants as they become more

focused.




1. BACKGROUND


                                                                             4
Record of In-Training Assessment (RITA) for Senior House Officers was

introduced in the Mersey Deanery in 2000. Southport & Ormskirk Hospital

NHS Trust (which is split-site) was one of the two pilot sites.          I was

Management Lead for the pilot in Southport and Ormskirk. The system was

successful and was subsequently rolled out to the whole of the Mersey

Deanery. It is now used in other deaneries in the UK.



Currently, Senior House Officers (SHOs) have three assessments in a six

month period – an initial (to set objectives with their supervising consultant),

mid-point (to ensure satisfactory progress) and a final assessment. The final

assessment is in the presence of an external assessor, acting as the Royal

College representative. Both the SHO and the post are assessed. The SHO

receives feedback with respect to their personal development and training for

their chosen career path. It is also important, as these are recognised training

posts, that the Director of Medical Education, the Postgraduate Dean and the

Chief Executive of the Trust receive feedback from the SHO regarding the

support they have received while training in the Trust.



Junior doctors no longer work exclusively for a particular consultant or firm.

They are part of a multidisciplinary team. 360° appraisal is being considered

as a possible source of evidence, which will form part of the portfolio

required, for General Medical Council purposes.




                                                                              5
2. LITERATURE SEARCH



2.1      Key Terms used:

•     360 degree

•     360 degree evaluation

•     360 degree appraisal and doctor

•     Various versions of the above



2.2      Databases accessed:

•     Ebsco

•     Proquest

•     Psy.Info

•     Ingenta

•     Emerald



2.3      Dates

Most searches were done within the dates 1996-2003. The majority of the

information used is from 2002-2003.




                                                                     6
3   KEY THEMES

During my search the following terms occur as key themes:

•   Multi Source Feedback

       The view of a variety of individuals highlighting areas of strength and

       those in need of improvement.

•   Learning opportunities

       Improving performance and enabling individuals to learn and grow,

       whether appraisee or coach.

•   Dealing with personality conflicts

       Employees may feel feedback on performance is ‘biased’ due to a

       personality conflict.

•   Evaluation

       Evaluation should be consistent across departments or specialties.

•   Scoring system

        360° appraisal requires a scoring system in order to measure

       behaviour. A scoring system is also required to measure competence.

•   Resources

       An organisation needs to consider both the physical and financial

       resources required to introduce a 360° appraisal system.



I will be using the above key themes to give a positive and critical appraisal,

in order to review the literature findings.




4   WHAT IS 360° APPRAISAL?

                                                                             7
4.1     It is an appraisal system that includes the view of a variety of

        colleagues – using it to highlight areas of strength and those in need of

        improvement.     It is usually questionnaire-based and a ‘score’ or

        ‘rating’ is given to measure relevant behavioural patterns.



4.2     The term originates from the commercial sector and refers to ‘full

        circle’.   It is also referred to as Multi Source Feedback (MSF).

        Research suggests that, due to potential bias, most organisations no

        longer use it for performance management and recognise it as a

        developmental tool.



4.3     360° appraisal began to take hold in American businesses in 1996

        (Antonioni, 2001) including Chrysler, GTE, Barclays and the Bank of

        America. In the UK it is believed that 65% of companies used 360°

        appraisal in 2000, compared to 40% in 1995 (Pfau and Kay, 2002).



4.4     A doctor has to provide evidence of working relationships with

        patients and colleagues to comply with General Medical Council

        (GMC) standards of ‘Good Medical Practice’ (King, 2003). 360°

        appraisal is suggested as a source of evidence which will form part of

        a portfolio required for GMC purposes.




5     MULTI SOURCE FEEDBACK (MSF)


                                                                               8
5.1   There does not appear to be an ideal number of feedback providers –

      typically 4-10 per recipient. Most external assessment consultants say

      that who gives the feedback is more important than how many

      (Vinson, 1996). Brotherton, (2003) recommends that the appraisee

      has a say in who the individuals, who regularly work with them,

      should be and that there should be two evaluators – one inside the

      department and the other from another section. Thus, Vinson argues

      that feedback should be anonymous and confidential, so that people

      can be candid. Valid feedback depends on people having worked with

      someone long enough to get to know them. If less than six months,

      feedback from a previous group can serve as a benchmark for the next

      appraisal. Feedback should be clear, as most people will not act on

      something they do not understand.      Follow up is essential.    It is

      commonly believed that if appraisers, and those being appraised, fail

      to follow up after feedback, performance and attitudes do not change.

      360° appraisals should not be used to determine salary or promotion;

      the aim is to open up dialogue, change behaviour and improve

      performance.

5.2   Typically a 360° feedback questionnaire would include key categories

      such as communication and team work. It would be developed to

      reflect performance criteria e.g. ‘Good Medical Practice’. There is, as

      yet, no standard method in the National Health Service. However the

      NHS is committed to introducing it and some hospitals have begun to

      use 360° appraisal where members of staff are willing to learn from

      feedback and use it to enhance team performance (King, 2003). The


                                                                           9
      onus is on the person being appraised to gather data and bring along to

      the appraisal to help inform the Personal Development Plan (PDP).

5.3   Green and Griffin, (2003) consider 360° feedback extremely useful for

      all types of leaders and teams, precisely because the feedback is

      objective.   They support the argument that, for this tool to be

      effective, the data collected must be confidential and the responses

      should be benchmarked.         Top-down approach highlights any

      performance gaps and motivates managers to improve.

5.4   Rodgers and Manifold, (2002) also found that most 360° appraisal

      tools use a survey or questionnaire to gather information in several

      areas, e.g. knowledge based, skills and task proficiency, team work,

      communication,       management       skills,     decision    making,

      professionalism, use of information technology, teaching skills and

      facilitation of learning by other team members.

5.5   Research studies carried out show that reliable data can be generated

      with a reasonable number of respondents and physicians will use

      feedback to contemplate and initiate changes in practice (Lockyer,

      2003). Multi-source feedback is not a replacement for audit when

      clinical outcomes need to be addressed.

5.6   Feedback should contribute to enhancing departmental performance

      over time if used by managers as a guide to development and

      behaviour change (Adsit, London, Crom and Jones, 1996).




                                                                          10
6     LEARNING OPPORTUNITIES



6.1     A major benefit of using 360° appraisal is that it provides a learning

        opportunity for appraisees and those coaching the appraisee develop

        highly transferable skills.



6.2     Storr, (2002) described 360° appraisal as being a valuable learning

        opportunity when it was introduced in 1996 at the Humberside

        Training and Enterprise Council. Its purpose is stated: to improve

        performance and enable people to learn and grow. The system is now

        standard for all 200 staff. Storr argues that this system encourages

        openness with line managers, or groups with a facilitator (up to seven

        people chosen by the appraisee, from whom they wish feedback).

        Any material produced, e.g. Personal Development Plan, belongs to

        the appraisee.     It is suggested that the 360° appraisal process

        maximises a person’s learning opportunities. This is supported by

        Antonioni, (2001). When employees/trainees excel in some areas, but

        are weak in others, 360° feedback can help them realise their potential

        and prevent career derailment (Wimer, 2002).



6.3     It is also believed that facilitators and those ‘coaching’ develop highly

        transferable skills, such as listening and questioning. It has been

        established that ‘coaching’ is important in helping appraisees interpret




                                                                              11
     results, set improvement goals with appraisees and follow up on

     action plans.    Thus, training for coaches might include active

     listening, focused interviewing, dealings with feelings and reactions to

     unexpected negative appraisal feedback, setting specific goals and

     action planning, follow-up and ways of shaping the development of

     new behavioural skills.




7.   DEALING WITH PERSONALITY CONFLICTS


                                                                          12
7.1   As managers confront todays highly competitive health care service

      market, performing effective appraisal has become more complex

      (Arnold and Pulich, 2003). To carry out appraisals accurately on a

      ‘one-to-one’ basis there is an increased likelihood of personality

      conflicts, as employees may feel that feedback about performance is

      biased by a manager’s ‘apparent’ dislike of them.



7.2   Furthermore, for top management to be involved in a personality

      conflict situation means that intense feeling could increase the

      probability that the organisation might be involved in law suits over

      appraisal. Research studies reveal that individual differences in age,

      race and gender may bias an appraisal.



7.3   A study of registered nurses illustrated how age can bias managers.

      Those nurses aged 30-39 evaluated their performance similarly to the

      rating of their manager; those aged 21-29 rated their performance

      lower and those aged 40-61 rated their performance higher. An

      interpretation was that the manager tended not to give older nurses

      credit for their success.



7.4   Therefore a significant difference in the input of others, when

      compared to one manager’s, could be an indication that objectivity is

      a problem (Arnold and Pulich, 2003). Using 360° appraisal, a nurse

      appraisal might include input from an immediate manager, doctors,


                                                                         13
      peers, patients or family members, just as a doctor’s appraisal might

      include their supervising consultant, peers, nurses, patients. However,

      several studies indicated that patient feedback could be the most

      difficult to capture due to literacy, language, cultural and personality

      concerns.



7.4   A benefit of the 360° appraisal system is that others interact with the

      employee whose performance is being appraised, at a time when the

      immediate manager is not present. This is a view held by Pfau and

      Kay (2000) who believe that people who work most closely with an

      employee see the person’s behaviour in settings and circumstances a

      supervisor/manager may not. However, Pfau and Kay also argue that

      unless every participant in 360° appraisal is trained in the art of giving

      and receiving feedback, both positive and negative, the process can

      lead to uncertainty and conflict amongst team members.



7.6   In conclusion, managers can improve the quality of the appraisal by

      increasing their awareness of personality conflicts in evaluating

      results, incorporating input from others. Arnold and Pulich, (2003)

      argue that an objective appraisal would result in a ‘win-win’ situation

      for all parties concerned.




                                                                             14
8.      EVALUATION           OF     SKILLS       AND      COMPETENCIES



8.1   In considering what we evaluate, attitudes, skills and knowledge are

      important. We also need to establish different levels of proficiency at

      different levels of training. Pangaro, (2003) argues that a novice

      could identify a life threatening wound or infection, an advanced

      learner could manage the crucial wound and an expert could manage a

      team.   Pangaro also suggests that to succeed, we have to be able to

      specify what the level of expectation is and the performance level for

      the trainee. There is a relationship between trainer and trainee, but

      eventually the trainer has to ‘disappear’ so that independency can be

      achieved, which is the ultimate goal for training (e.g. medicine).

8.2   An evaluation during training to give feedback is called ‘formative’,

      whereas evaluating after training, to render a pass/fail, certify, etc is

      ‘summative’. Formative evaluation is very important for fairness to

      learners, whereas summative evaluation is vital, particularly in the

      NHS, to be fair to patients to ascertain whether sustained competency

      of a doctor or health care professional has been achieved. This is a

      determination made at the end of an experience.

8.3   ‘Fairness’ is at the heart of evaluation (Pangaro, 2003). Rodger and

      Manifold, (2002) agree with this statement.              In a hospital

      environment, the stakes are high for patient, students and residents.

      Evaluation is important and we have to ensure expectations are clear

      and trainees are given feedback during the process. Part of the idea of

      fairness to society is to ensure there is validity to the process of


                                                                            15
      evaluation. Measurements during training provide some prediction of

      whether a doctor is ‘competent’ later.        Also doctors need to be

      competent to use equipment relevant to their patient’s care.

      Brotherton (2003) reveals that at all levels in an organisation, the first

      step for every department is to determine core competencies. Staff in

      the organisation need to revise job descriptions or job plans and to list

      specific tasks associated with every competency, whether they are a

      manager, senior accountant or a director.



8.4   Pangaro, (2003) suggests that evaluation should be consistent between

      specialties. A clinical example that Pangaro relates to is a ‘simulator’

      (for example, in a Clinical Skills Centre), where performance is

      observed and standards are set for trainees to meet, which generate a

      rating or score. This is then evaluated to determine the value. The

      trainee is expected to recognise a problem and manage it. If a doctor

      only recognises the problem, but not more than this, the doctor would

      not be judged as competent, and action might result in them repeating

      the curriculum or certification being delayed or denied.             The

      distinction between performance and competence is that performance

      is what the learner does under test conditions (e.g. clinical exam),

      whereas competence is what the learner does habitually under day to

      day conditions with no one watching. It is an important distinction.




                                                                             16
9. SCORING SYSTEM



9.1   Every 360° appraisal system has to have an appropriate scoring

      system to determine behaviours or competencies that are key to a job.

      Behaviour, for example, is often rated as ‘needs mentoring’, ‘meets

      expectations’, ‘not meeting expectations’ and ‘constantly surpasses

      expectations’ and would use a scoring system 1-5 for example. It

      would be determined whether 1 was high or low. Furthermore, the

      Accreditation Council for Graduate Medical Education suggests that

      appraisal   looks   at   medical   knowledge,    patient   care,   skills,

      professionalism, practise based learning and system based practice.



9.2   If using an observer, for example in a Clinical Skills Centre, to

      provide ratings, reliability and validity of raters should be considered

      (Pangaro, 2003). Reliability is a property of making an observation,

      whereas validity is the main property of evaluation. Reliability is the

      first step in making a rigorous evaluation and can be defined as the

      consistency, or stability of results. Validity, on the other hand, is

      measuring what you want to measure, compared to something else.

      Face validity is the method that seems most appropriate for measuring

      competency.    Pangaro’s study reveals that managers at ‘General

      Electric’ are taking one day training courses on how to give ratings.

      Ratings might include ‘exceed expectations' based on progress toward

      clearly stated and pragmatic company goals and ‘meeting deadlines’.




                                                                             17
9.3   ‘Ford’, on the other hand, scrapped rankings in 2002 after an age

      descrimination suit. The flurry of law suits is prompting many other

      companies to make similar changes (Clark, 2003). The law suits are

      not slowing down the rate of change; instead, employers are making

      changes to ‘jury proof’ their appraisal system by backing ratings with

      evidence and data. These changes not only prevent law suits, but

      make appraisal fairer and more accurate.



9.4   Adsit, London, Crom and Jones, (1996) believe that information from

      multiple sources when aggregated within source (e.g. the average of

      all the subordinates ratings) should provide reliable information.

      Ratings can also reflect elements of unit performance. Ratings should

      ultimately enhance both individual and organisational development.




                                                                           18
10.    RESOURCES



10.1    A further theme that emerged was that of resources which would be

        required in order to implement 360° appraisal. Training Facilitators

        and Coaches can be time consuming, which is a cost to the

        organisation. There is also a cost involved in one day training courses

        to train individuals in giving both positive and negative feedback and

        how to give ratings. However, Reichheld and Detrick, (2003) argue

        that the 360° feedback system should be tested, starting with senior

        managers and rolled out to all employees, in order to retain

        stakeholders     in   the   organisation.   In    business,   retaining

        stakeholders/customers/employees can help reduce costs. In financial

        services a 5% increase in customer retention produces a 25% increase

        in profit.     Reichheld and Detrick claim that every stakeholder

        (employee, supplier, customer) is a potential cost reduction crusader.



10.2    Rodgers and Manifold, (2002) state that most 360° assessments are

        computer based software systems that generate timely and easily

        correlated results. It is estimated that 8% of major companies in the

        U.S. are currently using this method of feedback for all levels of

        employees. Many companies provide the software to run the system

        and a set of validated questionnaires examining common workplace

        behaviours and providing targeted feedback.        Most systems have

        designed their templates or questionnaires so that they can be

        customised.     Advantages of the 360° assessment tool, using an

                                                                             19
electronic database for documentation, results in rapid access for

raters and rapid turnaround for feedback and analysis.    This is

supported by Antonioni, (2001) who reports that appraisers can set

their improvement goals and action plans on line. However, there

does not appear to be a method of evaluation uniquely suited to

medical education as yet.




                                                               20
11. CONCLUSION



11.1   Researchers look at outcomes from different perspectives and

       different stakeholders determine their own important outcomes.



11.2   Several studies have looked into 360° appraisal. Some authors are

       positive about the process; others have some reservations. Despite

       some reservations, the National Health Service is clearly committed to

       the introduction of the 360° appraisal system. (Cheshire & Merseyside

       Strategic Health Authority is an early implementer of the process and

       is introducing the system beginning with Chief Executives).        The

       Royal College of Physicians is also piloting the process for Specialist

       Registrars and Senior House Officers.



11.3   The key issue appears to be training individuals to give positive and

       negative feedback.    Individuals/appraisers find it difficult to give

       ‘negative’ feedback to appraisees for fear of demoralising or

       demotivating staff or causing any conflict between manager and

       employee or within the team. While training individuals to give and

       receive feedback may temporarily increase the expense associated

       with 360° feedback programmes, the gains seem to outweigh the

       ‘cost’ of feedback to participants as they become more focused. The

       goal would be to create a culture in which individuals feel comfortable

       giving and receiving feedback, both positive and negative.




                                                                           21
12. RECOMMENDATIONS

    Having reviewed the available literature, the following appear to be

    key to the success of implementing a 360o appraisal programme:

    •   It must be for the right reasons – consider why you (the

        organisation) are doing it

        Have you got a strong business case to support the introduction of

        it? (If not, perhaps it should not be introduced)

    •   Assess the costs to the organisation (financial and physical

        resources)

        Ensure this will achieve the best possible results to achieve goals

    •   Focus on goals and strategy for the organisation’s appropriate

        training needs

    •   Get support at all levels of the organisation

        Make sure executives play a key and visible role. Also that line

        managers/employees are included in appropriate discussion

        relevant to their department to have ‘ownership’ of new system

    •   Train people in giving and receiving feedback (both positive and

        negative)

        Ensure people feel comfortable with the process

    •   Develop a list of core competencies for all staff

        Link competencies to specific job-related skills

    •   Evaluate the process

        Identify areas for improvement

    •   Monitor the process



                                                                          22
13. REFERENCES

Antonioni, D. (1996) ‘Designing an Effective 360-Degree Appraisal
      Feedback Process’, Organisational Dynamics, 25(2): 24-38.

Arnold, E and Pulich, M.(2003) ‘Personality conflicts and objectivity in
       appraising performance’, The Health Care Manager, 22(3): 227.

Adsit, D. London, M. Crom, S. and Jones, D. (1996) ‘Relationships between
       employee attitudes, customer satisfaction and departmental
       performance’, Journal of Management Development, 15(1): 62-75.

Brotherton, P. (2003) ‘Meyners Pays for Performance’, Journal of
       Accountancy, 196(1), 41-46.

Clark, K. (2003) ‘Judgement Day’, U.S News & World Report, 134(1): 31-32.

Epstein, R. and Hundert, E. (2002) ‘Defining and Assessing Professional
       Competence’, Journal of the American Medical Association, 287(2):
       226

Green, B. and Griffin, N. (2003) ‘Personalize Your Management
      Development’, Harvard Business Review, 81(6): 130.

Lockyer, J. (2003) ‘Multi-source feedback in the assessment of physician
      competencies’, Journal of Continuing Education in the Health
      Professions, 23(1): 4-12.

King, J. (2003) ‘360° appraisal’, British Medical Journal, 324(7352).

Pangaro, L. (2003) ‘Assessment and outcomes in medical education’,
      Military Medicine, 168(9): 21.

Pfau, B. and Kay, I. (2002) ‘Does 360-Degree Feedback Negatively Affect
       Company Performance?’, H.R Magazine, 47(6).

Reichheld, F. and Detrick, C. (2003) ‘Loyalty: A Prescription for Cutting
      Costs’, Marketing Management, 12(5): 24-26.

Rodgers, K.G. (2002) ‘360-degree feedback: Possibilities for assessment of
      the ACGME core competencies for emergency medicine residents’,
      Academic Emergency Medicine, 9(11): 1300-1305.

Storr, F. (2000) ‘360-Degree Appraisal’, People Management, 6(10).

Vinson, M. (1996), ‘The Pros and Cons of 360-Degree Feedback: Making it
      Work’, Training & Development, 50(4).

Wimer, S. (2002) ‘The Dark Side of 360-Degree Feedback’, Training &
      Development, 56(9): 37-42.

								
To top