NHS 360° Appraisal “To See Ourselves as Others See Us” Heather Ainscough Medical Education Manager Southport & Ormskirk Hospital NHS Trust February 2004 ACKNOWLEDGEMENTS I would like to thank Matthew Serlin, Director of Medical Education, and Sharon Partington, Director of Human Resources, Southport & Ormskirk Hospital NHS Trust, for allowing me the time to complete this project. I would like to thank Jonathan Parry, Chief Executive and Michael Mason, Trust Library Manager, for their encouragement. I would also like to thank Delwyn Addis, Ann Allman and Sheila Rodd (my fellow colleagues on the evidence based course) for all their cheerful support. Thanks also to Dee Snape, Research and Development Co-ordinator, Southport and Ormskirk Hospital NHS Trust, Barbara Jack and Janet Taylor, Senior Lecturers at the School of Health Sciences, Edge Hill College. 1 CONTENTS Acknowledgements 1 Contents 2-3 Abstract 4 1. BACKGROUND 5 2. LITERATURE SEARCH 6 2.1 Key Terms 2.2 Databases 2.3 Dates 3. KEY THEMES 7 4. WHAT IS 360° APPRAISAL? 8 5. MULTI-SOURCE FEEDBACK 9 - 10 5.1 Number of Feedback providers 5.2 Categories for performance 5.3 Objective feedback 5.4 Questionnaire based survey 5.5 Changes in practice 5.6 Enhancing departmental performance 6. LEARNING OPPORTUNITIES 11 - 12 6.1 Major benefit 6.2 Encouraging openness 6.3 Transferable skills 7. DEALING WITH PERSONALITY CONFLICTS 13 - 14 7.1 Potential Bias 7.2 Law suits 7.3 Nurse study 7.4 Feedback providers 7.5 Employee behaviour 7.6 Awareness of conflict 8. EVALUATION OF SKILLS & COMPETENCIES 15 - 16 8.1 What to evaluate 8.2 When to evaluate 8.3 Why to evaluate 8.4 Performance and competence (e.g. Clinical Skills Centre) 2 9. SCORING SYSTEM 17 – 18 9.1 Behaviour 9.2 Validity 9.3 Law suits 9.4 Reliability 10. RESOURCES 19 - 20 10.1 Physical resources 10.2 Software systems 11. CONCLUSIONS 21 11.1 Perspectives 11.2 Implementation 11.3 Training individuals 12. RECOMMENDATIONS 22 13. REFERENCES 23 3 ABSTRACT I attempt to identify an effective and constructive system of 360° appraisal as a possible tool for assessing behavioural patterns. Also to determine how people develop skills and competencies that are key to their role. 360° appraisal is being considered as part of a robust system relevant to ‘Modernising Medical Careers’ and as part of a health care system increasingly provided by multi-disciplinary teams. The report details the results of a literature review looking into 360° appraisal. The report looks at multi-source feedback, learning opportunities, evaluation, personality conflicts, scoring systems and resources. The review concluded that, while training individuals may temporarily increase the expense associated with 360° feedback programmes, the gains seem to outweigh the ‘cost’ of feedback to participants as they become more focused. 1. BACKGROUND 4 Record of In-Training Assessment (RITA) for Senior House Officers was introduced in the Mersey Deanery in 2000. Southport & Ormskirk Hospital NHS Trust (which is split-site) was one of the two pilot sites. I was Management Lead for the pilot in Southport and Ormskirk. The system was successful and was subsequently rolled out to the whole of the Mersey Deanery. It is now used in other deaneries in the UK. Currently, Senior House Officers (SHOs) have three assessments in a six month period – an initial (to set objectives with their supervising consultant), mid-point (to ensure satisfactory progress) and a final assessment. The final assessment is in the presence of an external assessor, acting as the Royal College representative. Both the SHO and the post are assessed. The SHO receives feedback with respect to their personal development and training for their chosen career path. It is also important, as these are recognised training posts, that the Director of Medical Education, the Postgraduate Dean and the Chief Executive of the Trust receive feedback from the SHO regarding the support they have received while training in the Trust. Junior doctors no longer work exclusively for a particular consultant or firm. They are part of a multidisciplinary team. 360° appraisal is being considered as a possible source of evidence, which will form part of the portfolio required, for General Medical Council purposes. 5 2. LITERATURE SEARCH 2.1 Key Terms used: • 360 degree • 360 degree evaluation • 360 degree appraisal and doctor • Various versions of the above 2.2 Databases accessed: • Ebsco • Proquest • Psy.Info • Ingenta • Emerald 2.3 Dates Most searches were done within the dates 1996-2003. The majority of the information used is from 2002-2003. 6 3 KEY THEMES During my search the following terms occur as key themes: • Multi Source Feedback The view of a variety of individuals highlighting areas of strength and those in need of improvement. • Learning opportunities Improving performance and enabling individuals to learn and grow, whether appraisee or coach. • Dealing with personality conflicts Employees may feel feedback on performance is ‘biased’ due to a personality conflict. • Evaluation Evaluation should be consistent across departments or specialties. • Scoring system 360° appraisal requires a scoring system in order to measure behaviour. A scoring system is also required to measure competence. • Resources An organisation needs to consider both the physical and financial resources required to introduce a 360° appraisal system. I will be using the above key themes to give a positive and critical appraisal, in order to review the literature findings. 4 WHAT IS 360° APPRAISAL? 7 4.1 It is an appraisal system that includes the view of a variety of colleagues – using it to highlight areas of strength and those in need of improvement. It is usually questionnaire-based and a ‘score’ or ‘rating’ is given to measure relevant behavioural patterns. 4.2 The term originates from the commercial sector and refers to ‘full circle’. It is also referred to as Multi Source Feedback (MSF). Research suggests that, due to potential bias, most organisations no longer use it for performance management and recognise it as a developmental tool. 4.3 360° appraisal began to take hold in American businesses in 1996 (Antonioni, 2001) including Chrysler, GTE, Barclays and the Bank of America. In the UK it is believed that 65% of companies used 360° appraisal in 2000, compared to 40% in 1995 (Pfau and Kay, 2002). 4.4 A doctor has to provide evidence of working relationships with patients and colleagues to comply with General Medical Council (GMC) standards of ‘Good Medical Practice’ (King, 2003). 360° appraisal is suggested as a source of evidence which will form part of a portfolio required for GMC purposes. 5 MULTI SOURCE FEEDBACK (MSF) 8 5.1 There does not appear to be an ideal number of feedback providers – typically 4-10 per recipient. Most external assessment consultants say that who gives the feedback is more important than how many (Vinson, 1996). Brotherton, (2003) recommends that the appraisee has a say in who the individuals, who regularly work with them, should be and that there should be two evaluators – one inside the department and the other from another section. Thus, Vinson argues that feedback should be anonymous and confidential, so that people can be candid. Valid feedback depends on people having worked with someone long enough to get to know them. If less than six months, feedback from a previous group can serve as a benchmark for the next appraisal. Feedback should be clear, as most people will not act on something they do not understand. Follow up is essential. It is commonly believed that if appraisers, and those being appraised, fail to follow up after feedback, performance and attitudes do not change. 360° appraisals should not be used to determine salary or promotion; the aim is to open up dialogue, change behaviour and improve performance. 5.2 Typically a 360° feedback questionnaire would include key categories such as communication and team work. It would be developed to reflect performance criteria e.g. ‘Good Medical Practice’. There is, as yet, no standard method in the National Health Service. However the NHS is committed to introducing it and some hospitals have begun to use 360° appraisal where members of staff are willing to learn from feedback and use it to enhance team performance (King, 2003). The 9 onus is on the person being appraised to gather data and bring along to the appraisal to help inform the Personal Development Plan (PDP). 5.3 Green and Griffin, (2003) consider 360° feedback extremely useful for all types of leaders and teams, precisely because the feedback is objective. They support the argument that, for this tool to be effective, the data collected must be confidential and the responses should be benchmarked. Top-down approach highlights any performance gaps and motivates managers to improve. 5.4 Rodgers and Manifold, (2002) also found that most 360° appraisal tools use a survey or questionnaire to gather information in several areas, e.g. knowledge based, skills and task proficiency, team work, communication, management skills, decision making, professionalism, use of information technology, teaching skills and facilitation of learning by other team members. 5.5 Research studies carried out show that reliable data can be generated with a reasonable number of respondents and physicians will use feedback to contemplate and initiate changes in practice (Lockyer, 2003). Multi-source feedback is not a replacement for audit when clinical outcomes need to be addressed. 5.6 Feedback should contribute to enhancing departmental performance over time if used by managers as a guide to development and behaviour change (Adsit, London, Crom and Jones, 1996). 10 6 LEARNING OPPORTUNITIES 6.1 A major benefit of using 360° appraisal is that it provides a learning opportunity for appraisees and those coaching the appraisee develop highly transferable skills. 6.2 Storr, (2002) described 360° appraisal as being a valuable learning opportunity when it was introduced in 1996 at the Humberside Training and Enterprise Council. Its purpose is stated: to improve performance and enable people to learn and grow. The system is now standard for all 200 staff. Storr argues that this system encourages openness with line managers, or groups with a facilitator (up to seven people chosen by the appraisee, from whom they wish feedback). Any material produced, e.g. Personal Development Plan, belongs to the appraisee. It is suggested that the 360° appraisal process maximises a person’s learning opportunities. This is supported by Antonioni, (2001). When employees/trainees excel in some areas, but are weak in others, 360° feedback can help them realise their potential and prevent career derailment (Wimer, 2002). 6.3 It is also believed that facilitators and those ‘coaching’ develop highly transferable skills, such as listening and questioning. It has been established that ‘coaching’ is important in helping appraisees interpret 11 results, set improvement goals with appraisees and follow up on action plans. Thus, training for coaches might include active listening, focused interviewing, dealings with feelings and reactions to unexpected negative appraisal feedback, setting specific goals and action planning, follow-up and ways of shaping the development of new behavioural skills. 7. DEALING WITH PERSONALITY CONFLICTS 12 7.1 As managers confront todays highly competitive health care service market, performing effective appraisal has become more complex (Arnold and Pulich, 2003). To carry out appraisals accurately on a ‘one-to-one’ basis there is an increased likelihood of personality conflicts, as employees may feel that feedback about performance is biased by a manager’s ‘apparent’ dislike of them. 7.2 Furthermore, for top management to be involved in a personality conflict situation means that intense feeling could increase the probability that the organisation might be involved in law suits over appraisal. Research studies reveal that individual differences in age, race and gender may bias an appraisal. 7.3 A study of registered nurses illustrated how age can bias managers. Those nurses aged 30-39 evaluated their performance similarly to the rating of their manager; those aged 21-29 rated their performance lower and those aged 40-61 rated their performance higher. An interpretation was that the manager tended not to give older nurses credit for their success. 7.4 Therefore a significant difference in the input of others, when compared to one manager’s, could be an indication that objectivity is a problem (Arnold and Pulich, 2003). Using 360° appraisal, a nurse appraisal might include input from an immediate manager, doctors, 13 peers, patients or family members, just as a doctor’s appraisal might include their supervising consultant, peers, nurses, patients. However, several studies indicated that patient feedback could be the most difficult to capture due to literacy, language, cultural and personality concerns. 7.4 A benefit of the 360° appraisal system is that others interact with the employee whose performance is being appraised, at a time when the immediate manager is not present. This is a view held by Pfau and Kay (2000) who believe that people who work most closely with an employee see the person’s behaviour in settings and circumstances a supervisor/manager may not. However, Pfau and Kay also argue that unless every participant in 360° appraisal is trained in the art of giving and receiving feedback, both positive and negative, the process can lead to uncertainty and conflict amongst team members. 7.6 In conclusion, managers can improve the quality of the appraisal by increasing their awareness of personality conflicts in evaluating results, incorporating input from others. Arnold and Pulich, (2003) argue that an objective appraisal would result in a ‘win-win’ situation for all parties concerned. 14 8. EVALUATION OF SKILLS AND COMPETENCIES 8.1 In considering what we evaluate, attitudes, skills and knowledge are important. We also need to establish different levels of proficiency at different levels of training. Pangaro, (2003) argues that a novice could identify a life threatening wound or infection, an advanced learner could manage the crucial wound and an expert could manage a team. Pangaro also suggests that to succeed, we have to be able to specify what the level of expectation is and the performance level for the trainee. There is a relationship between trainer and trainee, but eventually the trainer has to ‘disappear’ so that independency can be achieved, which is the ultimate goal for training (e.g. medicine). 8.2 An evaluation during training to give feedback is called ‘formative’, whereas evaluating after training, to render a pass/fail, certify, etc is ‘summative’. Formative evaluation is very important for fairness to learners, whereas summative evaluation is vital, particularly in the NHS, to be fair to patients to ascertain whether sustained competency of a doctor or health care professional has been achieved. This is a determination made at the end of an experience. 8.3 ‘Fairness’ is at the heart of evaluation (Pangaro, 2003). Rodger and Manifold, (2002) agree with this statement. In a hospital environment, the stakes are high for patient, students and residents. Evaluation is important and we have to ensure expectations are clear and trainees are given feedback during the process. Part of the idea of fairness to society is to ensure there is validity to the process of 15 evaluation. Measurements during training provide some prediction of whether a doctor is ‘competent’ later. Also doctors need to be competent to use equipment relevant to their patient’s care. Brotherton (2003) reveals that at all levels in an organisation, the first step for every department is to determine core competencies. Staff in the organisation need to revise job descriptions or job plans and to list specific tasks associated with every competency, whether they are a manager, senior accountant or a director. 8.4 Pangaro, (2003) suggests that evaluation should be consistent between specialties. A clinical example that Pangaro relates to is a ‘simulator’ (for example, in a Clinical Skills Centre), where performance is observed and standards are set for trainees to meet, which generate a rating or score. This is then evaluated to determine the value. The trainee is expected to recognise a problem and manage it. If a doctor only recognises the problem, but not more than this, the doctor would not be judged as competent, and action might result in them repeating the curriculum or certification being delayed or denied. The distinction between performance and competence is that performance is what the learner does under test conditions (e.g. clinical exam), whereas competence is what the learner does habitually under day to day conditions with no one watching. It is an important distinction. 16 9. SCORING SYSTEM 9.1 Every 360° appraisal system has to have an appropriate scoring system to determine behaviours or competencies that are key to a job. Behaviour, for example, is often rated as ‘needs mentoring’, ‘meets expectations’, ‘not meeting expectations’ and ‘constantly surpasses expectations’ and would use a scoring system 1-5 for example. It would be determined whether 1 was high or low. Furthermore, the Accreditation Council for Graduate Medical Education suggests that appraisal looks at medical knowledge, patient care, skills, professionalism, practise based learning and system based practice. 9.2 If using an observer, for example in a Clinical Skills Centre, to provide ratings, reliability and validity of raters should be considered (Pangaro, 2003). Reliability is a property of making an observation, whereas validity is the main property of evaluation. Reliability is the first step in making a rigorous evaluation and can be defined as the consistency, or stability of results. Validity, on the other hand, is measuring what you want to measure, compared to something else. Face validity is the method that seems most appropriate for measuring competency. Pangaro’s study reveals that managers at ‘General Electric’ are taking one day training courses on how to give ratings. Ratings might include ‘exceed expectations' based on progress toward clearly stated and pragmatic company goals and ‘meeting deadlines’. 17 9.3 ‘Ford’, on the other hand, scrapped rankings in 2002 after an age descrimination suit. The flurry of law suits is prompting many other companies to make similar changes (Clark, 2003). The law suits are not slowing down the rate of change; instead, employers are making changes to ‘jury proof’ their appraisal system by backing ratings with evidence and data. These changes not only prevent law suits, but make appraisal fairer and more accurate. 9.4 Adsit, London, Crom and Jones, (1996) believe that information from multiple sources when aggregated within source (e.g. the average of all the subordinates ratings) should provide reliable information. Ratings can also reflect elements of unit performance. Ratings should ultimately enhance both individual and organisational development. 18 10. RESOURCES 10.1 A further theme that emerged was that of resources which would be required in order to implement 360° appraisal. Training Facilitators and Coaches can be time consuming, which is a cost to the organisation. There is also a cost involved in one day training courses to train individuals in giving both positive and negative feedback and how to give ratings. However, Reichheld and Detrick, (2003) argue that the 360° feedback system should be tested, starting with senior managers and rolled out to all employees, in order to retain stakeholders in the organisation. In business, retaining stakeholders/customers/employees can help reduce costs. In financial services a 5% increase in customer retention produces a 25% increase in profit. Reichheld and Detrick claim that every stakeholder (employee, supplier, customer) is a potential cost reduction crusader. 10.2 Rodgers and Manifold, (2002) state that most 360° assessments are computer based software systems that generate timely and easily correlated results. It is estimated that 8% of major companies in the U.S. are currently using this method of feedback for all levels of employees. Many companies provide the software to run the system and a set of validated questionnaires examining common workplace behaviours and providing targeted feedback. Most systems have designed their templates or questionnaires so that they can be customised. Advantages of the 360° assessment tool, using an 19 electronic database for documentation, results in rapid access for raters and rapid turnaround for feedback and analysis. This is supported by Antonioni, (2001) who reports that appraisers can set their improvement goals and action plans on line. However, there does not appear to be a method of evaluation uniquely suited to medical education as yet. 20 11. CONCLUSION 11.1 Researchers look at outcomes from different perspectives and different stakeholders determine their own important outcomes. 11.2 Several studies have looked into 360° appraisal. Some authors are positive about the process; others have some reservations. Despite some reservations, the National Health Service is clearly committed to the introduction of the 360° appraisal system. (Cheshire & Merseyside Strategic Health Authority is an early implementer of the process and is introducing the system beginning with Chief Executives). The Royal College of Physicians is also piloting the process for Specialist Registrars and Senior House Officers. 11.3 The key issue appears to be training individuals to give positive and negative feedback. Individuals/appraisers find it difficult to give ‘negative’ feedback to appraisees for fear of demoralising or demotivating staff or causing any conflict between manager and employee or within the team. While training individuals to give and receive feedback may temporarily increase the expense associated with 360° feedback programmes, the gains seem to outweigh the ‘cost’ of feedback to participants as they become more focused. The goal would be to create a culture in which individuals feel comfortable giving and receiving feedback, both positive and negative. 21 12. RECOMMENDATIONS Having reviewed the available literature, the following appear to be key to the success of implementing a 360o appraisal programme: • It must be for the right reasons – consider why you (the organisation) are doing it Have you got a strong business case to support the introduction of it? (If not, perhaps it should not be introduced) • Assess the costs to the organisation (financial and physical resources) Ensure this will achieve the best possible results to achieve goals • Focus on goals and strategy for the organisation’s appropriate training needs • Get support at all levels of the organisation Make sure executives play a key and visible role. Also that line managers/employees are included in appropriate discussion relevant to their department to have ‘ownership’ of new system • Train people in giving and receiving feedback (both positive and negative) Ensure people feel comfortable with the process • Develop a list of core competencies for all staff Link competencies to specific job-related skills • Evaluate the process Identify areas for improvement • Monitor the process 22 13. REFERENCES Antonioni, D. (1996) ‘Designing an Effective 360-Degree Appraisal Feedback Process’, Organisational Dynamics, 25(2): 24-38. Arnold, E and Pulich, M.(2003) ‘Personality conflicts and objectivity in appraising performance’, The Health Care Manager, 22(3): 227. Adsit, D. London, M. Crom, S. and Jones, D. (1996) ‘Relationships between employee attitudes, customer satisfaction and departmental performance’, Journal of Management Development, 15(1): 62-75. Brotherton, P. (2003) ‘Meyners Pays for Performance’, Journal of Accountancy, 196(1), 41-46. Clark, K. (2003) ‘Judgement Day’, U.S News & World Report, 134(1): 31-32. Epstein, R. and Hundert, E. (2002) ‘Defining and Assessing Professional Competence’, Journal of the American Medical Association, 287(2): 226 Green, B. and Griffin, N. (2003) ‘Personalize Your Management Development’, Harvard Business Review, 81(6): 130. Lockyer, J. (2003) ‘Multi-source feedback in the assessment of physician competencies’, Journal of Continuing Education in the Health Professions, 23(1): 4-12. King, J. (2003) ‘360° appraisal’, British Medical Journal, 324(7352). Pangaro, L. (2003) ‘Assessment and outcomes in medical education’, Military Medicine, 168(9): 21. Pfau, B. and Kay, I. (2002) ‘Does 360-Degree Feedback Negatively Affect Company Performance?’, H.R Magazine, 47(6). Reichheld, F. and Detrick, C. (2003) ‘Loyalty: A Prescription for Cutting Costs’, Marketing Management, 12(5): 24-26. Rodgers, K.G. (2002) ‘360-degree feedback: Possibilities for assessment of the ACGME core competencies for emergency medicine residents’, Academic Emergency Medicine, 9(11): 1300-1305. Storr, F. (2000) ‘360-Degree Appraisal’, People Management, 6(10). Vinson, M. (1996), ‘The Pros and Cons of 360-Degree Feedback: Making it Work’, Training & Development, 50(4). Wimer, S. (2002) ‘The Dark Side of 360-Degree Feedback’, Training & Development, 56(9): 37-42.
Pages to are hidden for
"Research Report on 360 Feedback in Bank"Please download to view full document