“To See Ourselves as
Others See Us”
Medical Education Manager
Southport & Ormskirk Hospital NHS Trust
I would like to thank Matthew Serlin, Director of Medical Education, and
Sharon Partington, Director of Human Resources, Southport & Ormskirk
Hospital NHS Trust, for allowing me the time to complete this project.
I would like to thank Jonathan Parry, Chief Executive and Michael Mason,
Trust Library Manager, for their encouragement. I would also like to thank
Delwyn Addis, Ann Allman and Sheila Rodd (my fellow colleagues on the
evidence based course) for all their cheerful support.
Thanks also to Dee Snape, Research and Development Co-ordinator,
Southport and Ormskirk Hospital NHS Trust, Barbara Jack and Janet Taylor,
Senior Lecturers at the School of Health Sciences, Edge Hill College.
1. BACKGROUND 5
2. LITERATURE SEARCH 6
2.1 Key Terms
3. KEY THEMES 7
4. WHAT IS 360° APPRAISAL? 8
5. MULTI-SOURCE FEEDBACK 9 - 10
5.1 Number of Feedback providers
5.2 Categories for performance
5.3 Objective feedback
5.4 Questionnaire based survey
5.5 Changes in practice
5.6 Enhancing departmental performance
6. LEARNING OPPORTUNITIES 11 - 12
6.1 Major benefit
6.2 Encouraging openness
6.3 Transferable skills
7. DEALING WITH PERSONALITY CONFLICTS 13 - 14
7.1 Potential Bias
7.2 Law suits
7.3 Nurse study
7.4 Feedback providers
7.5 Employee behaviour
7.6 Awareness of conflict
8. EVALUATION OF SKILLS & COMPETENCIES 15 - 16
8.1 What to evaluate
8.2 When to evaluate
8.3 Why to evaluate
8.4 Performance and competence (e.g. Clinical Skills Centre)
9. SCORING SYSTEM 17 – 18
9.3 Law suits
10. RESOURCES 19 - 20
10.1 Physical resources
10.2 Software systems
11. CONCLUSIONS 21
11.3 Training individuals
12. RECOMMENDATIONS 22
13. REFERENCES 23
I attempt to identify an effective and constructive system of 360° appraisal as
a possible tool for assessing behavioural patterns. Also to determine how
people develop skills and competencies that are key to their role. 360°
appraisal is being considered as part of a robust system relevant to
‘Modernising Medical Careers’ and as part of a health care system
increasingly provided by multi-disciplinary teams.
The report details the results of a literature review looking into 360°
appraisal. The report looks at multi-source feedback, learning opportunities,
evaluation, personality conflicts, scoring systems and resources.
The review concluded that, while training individuals may temporarily
increase the expense associated with 360° feedback programmes, the gains
seem to outweigh the ‘cost’ of feedback to participants as they become more
Record of In-Training Assessment (RITA) for Senior House Officers was
introduced in the Mersey Deanery in 2000. Southport & Ormskirk Hospital
NHS Trust (which is split-site) was one of the two pilot sites. I was
Management Lead for the pilot in Southport and Ormskirk. The system was
successful and was subsequently rolled out to the whole of the Mersey
Deanery. It is now used in other deaneries in the UK.
Currently, Senior House Officers (SHOs) have three assessments in a six
month period – an initial (to set objectives with their supervising consultant),
mid-point (to ensure satisfactory progress) and a final assessment. The final
assessment is in the presence of an external assessor, acting as the Royal
College representative. Both the SHO and the post are assessed. The SHO
receives feedback with respect to their personal development and training for
their chosen career path. It is also important, as these are recognised training
posts, that the Director of Medical Education, the Postgraduate Dean and the
Chief Executive of the Trust receive feedback from the SHO regarding the
support they have received while training in the Trust.
Junior doctors no longer work exclusively for a particular consultant or firm.
They are part of a multidisciplinary team. 360° appraisal is being considered
as a possible source of evidence, which will form part of the portfolio
required, for General Medical Council purposes.
2. LITERATURE SEARCH
2.1 Key Terms used:
• 360 degree
• 360 degree evaluation
• 360 degree appraisal and doctor
• Various versions of the above
2.2 Databases accessed:
Most searches were done within the dates 1996-2003. The majority of the
information used is from 2002-2003.
3 KEY THEMES
During my search the following terms occur as key themes:
• Multi Source Feedback
The view of a variety of individuals highlighting areas of strength and
those in need of improvement.
• Learning opportunities
Improving performance and enabling individuals to learn and grow,
whether appraisee or coach.
• Dealing with personality conflicts
Employees may feel feedback on performance is ‘biased’ due to a
Evaluation should be consistent across departments or specialties.
• Scoring system
360° appraisal requires a scoring system in order to measure
behaviour. A scoring system is also required to measure competence.
An organisation needs to consider both the physical and financial
resources required to introduce a 360° appraisal system.
I will be using the above key themes to give a positive and critical appraisal,
in order to review the literature findings.
4 WHAT IS 360° APPRAISAL?
4.1 It is an appraisal system that includes the view of a variety of
colleagues – using it to highlight areas of strength and those in need of
improvement. It is usually questionnaire-based and a ‘score’ or
‘rating’ is given to measure relevant behavioural patterns.
4.2 The term originates from the commercial sector and refers to ‘full
circle’. It is also referred to as Multi Source Feedback (MSF).
Research suggests that, due to potential bias, most organisations no
longer use it for performance management and recognise it as a
4.3 360° appraisal began to take hold in American businesses in 1996
(Antonioni, 2001) including Chrysler, GTE, Barclays and the Bank of
America. In the UK it is believed that 65% of companies used 360°
appraisal in 2000, compared to 40% in 1995 (Pfau and Kay, 2002).
4.4 A doctor has to provide evidence of working relationships with
patients and colleagues to comply with General Medical Council
(GMC) standards of ‘Good Medical Practice’ (King, 2003). 360°
appraisal is suggested as a source of evidence which will form part of
a portfolio required for GMC purposes.
5 MULTI SOURCE FEEDBACK (MSF)
5.1 There does not appear to be an ideal number of feedback providers –
typically 4-10 per recipient. Most external assessment consultants say
that who gives the feedback is more important than how many
(Vinson, 1996). Brotherton, (2003) recommends that the appraisee
has a say in who the individuals, who regularly work with them,
should be and that there should be two evaluators – one inside the
department and the other from another section. Thus, Vinson argues
that feedback should be anonymous and confidential, so that people
can be candid. Valid feedback depends on people having worked with
someone long enough to get to know them. If less than six months,
feedback from a previous group can serve as a benchmark for the next
appraisal. Feedback should be clear, as most people will not act on
something they do not understand. Follow up is essential. It is
commonly believed that if appraisers, and those being appraised, fail
to follow up after feedback, performance and attitudes do not change.
360° appraisals should not be used to determine salary or promotion;
the aim is to open up dialogue, change behaviour and improve
5.2 Typically a 360° feedback questionnaire would include key categories
such as communication and team work. It would be developed to
reflect performance criteria e.g. ‘Good Medical Practice’. There is, as
yet, no standard method in the National Health Service. However the
NHS is committed to introducing it and some hospitals have begun to
use 360° appraisal where members of staff are willing to learn from
feedback and use it to enhance team performance (King, 2003). The
onus is on the person being appraised to gather data and bring along to
the appraisal to help inform the Personal Development Plan (PDP).
5.3 Green and Griffin, (2003) consider 360° feedback extremely useful for
all types of leaders and teams, precisely because the feedback is
objective. They support the argument that, for this tool to be
effective, the data collected must be confidential and the responses
should be benchmarked. Top-down approach highlights any
performance gaps and motivates managers to improve.
5.4 Rodgers and Manifold, (2002) also found that most 360° appraisal
tools use a survey or questionnaire to gather information in several
areas, e.g. knowledge based, skills and task proficiency, team work,
communication, management skills, decision making,
professionalism, use of information technology, teaching skills and
facilitation of learning by other team members.
5.5 Research studies carried out show that reliable data can be generated
with a reasonable number of respondents and physicians will use
feedback to contemplate and initiate changes in practice (Lockyer,
2003). Multi-source feedback is not a replacement for audit when
clinical outcomes need to be addressed.
5.6 Feedback should contribute to enhancing departmental performance
over time if used by managers as a guide to development and
behaviour change (Adsit, London, Crom and Jones, 1996).
6 LEARNING OPPORTUNITIES
6.1 A major benefit of using 360° appraisal is that it provides a learning
opportunity for appraisees and those coaching the appraisee develop
highly transferable skills.
6.2 Storr, (2002) described 360° appraisal as being a valuable learning
opportunity when it was introduced in 1996 at the Humberside
Training and Enterprise Council. Its purpose is stated: to improve
performance and enable people to learn and grow. The system is now
standard for all 200 staff. Storr argues that this system encourages
openness with line managers, or groups with a facilitator (up to seven
people chosen by the appraisee, from whom they wish feedback).
Any material produced, e.g. Personal Development Plan, belongs to
the appraisee. It is suggested that the 360° appraisal process
maximises a person’s learning opportunities. This is supported by
Antonioni, (2001). When employees/trainees excel in some areas, but
are weak in others, 360° feedback can help them realise their potential
and prevent career derailment (Wimer, 2002).
6.3 It is also believed that facilitators and those ‘coaching’ develop highly
transferable skills, such as listening and questioning. It has been
established that ‘coaching’ is important in helping appraisees interpret
results, set improvement goals with appraisees and follow up on
action plans. Thus, training for coaches might include active
listening, focused interviewing, dealings with feelings and reactions to
unexpected negative appraisal feedback, setting specific goals and
action planning, follow-up and ways of shaping the development of
new behavioural skills.
7. DEALING WITH PERSONALITY CONFLICTS
7.1 As managers confront todays highly competitive health care service
market, performing effective appraisal has become more complex
(Arnold and Pulich, 2003). To carry out appraisals accurately on a
‘one-to-one’ basis there is an increased likelihood of personality
conflicts, as employees may feel that feedback about performance is
biased by a manager’s ‘apparent’ dislike of them.
7.2 Furthermore, for top management to be involved in a personality
conflict situation means that intense feeling could increase the
probability that the organisation might be involved in law suits over
appraisal. Research studies reveal that individual differences in age,
race and gender may bias an appraisal.
7.3 A study of registered nurses illustrated how age can bias managers.
Those nurses aged 30-39 evaluated their performance similarly to the
rating of their manager; those aged 21-29 rated their performance
lower and those aged 40-61 rated their performance higher. An
interpretation was that the manager tended not to give older nurses
credit for their success.
7.4 Therefore a significant difference in the input of others, when
compared to one manager’s, could be an indication that objectivity is
a problem (Arnold and Pulich, 2003). Using 360° appraisal, a nurse
appraisal might include input from an immediate manager, doctors,
peers, patients or family members, just as a doctor’s appraisal might
include their supervising consultant, peers, nurses, patients. However,
several studies indicated that patient feedback could be the most
difficult to capture due to literacy, language, cultural and personality
7.4 A benefit of the 360° appraisal system is that others interact with the
employee whose performance is being appraised, at a time when the
immediate manager is not present. This is a view held by Pfau and
Kay (2000) who believe that people who work most closely with an
employee see the person’s behaviour in settings and circumstances a
supervisor/manager may not. However, Pfau and Kay also argue that
unless every participant in 360° appraisal is trained in the art of giving
and receiving feedback, both positive and negative, the process can
lead to uncertainty and conflict amongst team members.
7.6 In conclusion, managers can improve the quality of the appraisal by
increasing their awareness of personality conflicts in evaluating
results, incorporating input from others. Arnold and Pulich, (2003)
argue that an objective appraisal would result in a ‘win-win’ situation
for all parties concerned.
8. EVALUATION OF SKILLS AND COMPETENCIES
8.1 In considering what we evaluate, attitudes, skills and knowledge are
important. We also need to establish different levels of proficiency at
different levels of training. Pangaro, (2003) argues that a novice
could identify a life threatening wound or infection, an advanced
learner could manage the crucial wound and an expert could manage a
team. Pangaro also suggests that to succeed, we have to be able to
specify what the level of expectation is and the performance level for
the trainee. There is a relationship between trainer and trainee, but
eventually the trainer has to ‘disappear’ so that independency can be
achieved, which is the ultimate goal for training (e.g. medicine).
8.2 An evaluation during training to give feedback is called ‘formative’,
whereas evaluating after training, to render a pass/fail, certify, etc is
‘summative’. Formative evaluation is very important for fairness to
learners, whereas summative evaluation is vital, particularly in the
NHS, to be fair to patients to ascertain whether sustained competency
of a doctor or health care professional has been achieved. This is a
determination made at the end of an experience.
8.3 ‘Fairness’ is at the heart of evaluation (Pangaro, 2003). Rodger and
Manifold, (2002) agree with this statement. In a hospital
environment, the stakes are high for patient, students and residents.
Evaluation is important and we have to ensure expectations are clear
and trainees are given feedback during the process. Part of the idea of
fairness to society is to ensure there is validity to the process of
evaluation. Measurements during training provide some prediction of
whether a doctor is ‘competent’ later. Also doctors need to be
competent to use equipment relevant to their patient’s care.
Brotherton (2003) reveals that at all levels in an organisation, the first
step for every department is to determine core competencies. Staff in
the organisation need to revise job descriptions or job plans and to list
specific tasks associated with every competency, whether they are a
manager, senior accountant or a director.
8.4 Pangaro, (2003) suggests that evaluation should be consistent between
specialties. A clinical example that Pangaro relates to is a ‘simulator’
(for example, in a Clinical Skills Centre), where performance is
observed and standards are set for trainees to meet, which generate a
rating or score. This is then evaluated to determine the value. The
trainee is expected to recognise a problem and manage it. If a doctor
only recognises the problem, but not more than this, the doctor would
not be judged as competent, and action might result in them repeating
the curriculum or certification being delayed or denied. The
distinction between performance and competence is that performance
is what the learner does under test conditions (e.g. clinical exam),
whereas competence is what the learner does habitually under day to
day conditions with no one watching. It is an important distinction.
9. SCORING SYSTEM
9.1 Every 360° appraisal system has to have an appropriate scoring
system to determine behaviours or competencies that are key to a job.
Behaviour, for example, is often rated as ‘needs mentoring’, ‘meets
expectations’, ‘not meeting expectations’ and ‘constantly surpasses
expectations’ and would use a scoring system 1-5 for example. It
would be determined whether 1 was high or low. Furthermore, the
Accreditation Council for Graduate Medical Education suggests that
appraisal looks at medical knowledge, patient care, skills,
professionalism, practise based learning and system based practice.
9.2 If using an observer, for example in a Clinical Skills Centre, to
provide ratings, reliability and validity of raters should be considered
(Pangaro, 2003). Reliability is a property of making an observation,
whereas validity is the main property of evaluation. Reliability is the
first step in making a rigorous evaluation and can be defined as the
consistency, or stability of results. Validity, on the other hand, is
measuring what you want to measure, compared to something else.
Face validity is the method that seems most appropriate for measuring
competency. Pangaro’s study reveals that managers at ‘General
Electric’ are taking one day training courses on how to give ratings.
Ratings might include ‘exceed expectations' based on progress toward
clearly stated and pragmatic company goals and ‘meeting deadlines’.
9.3 ‘Ford’, on the other hand, scrapped rankings in 2002 after an age
descrimination suit. The flurry of law suits is prompting many other
companies to make similar changes (Clark, 2003). The law suits are
not slowing down the rate of change; instead, employers are making
changes to ‘jury proof’ their appraisal system by backing ratings with
evidence and data. These changes not only prevent law suits, but
make appraisal fairer and more accurate.
9.4 Adsit, London, Crom and Jones, (1996) believe that information from
multiple sources when aggregated within source (e.g. the average of
all the subordinates ratings) should provide reliable information.
Ratings can also reflect elements of unit performance. Ratings should
ultimately enhance both individual and organisational development.
10.1 A further theme that emerged was that of resources which would be
required in order to implement 360° appraisal. Training Facilitators
and Coaches can be time consuming, which is a cost to the
organisation. There is also a cost involved in one day training courses
to train individuals in giving both positive and negative feedback and
how to give ratings. However, Reichheld and Detrick, (2003) argue
that the 360° feedback system should be tested, starting with senior
managers and rolled out to all employees, in order to retain
stakeholders in the organisation. In business, retaining
stakeholders/customers/employees can help reduce costs. In financial
services a 5% increase in customer retention produces a 25% increase
in profit. Reichheld and Detrick claim that every stakeholder
(employee, supplier, customer) is a potential cost reduction crusader.
10.2 Rodgers and Manifold, (2002) state that most 360° assessments are
computer based software systems that generate timely and easily
correlated results. It is estimated that 8% of major companies in the
U.S. are currently using this method of feedback for all levels of
employees. Many companies provide the software to run the system
and a set of validated questionnaires examining common workplace
behaviours and providing targeted feedback. Most systems have
designed their templates or questionnaires so that they can be
customised. Advantages of the 360° assessment tool, using an
electronic database for documentation, results in rapid access for
raters and rapid turnaround for feedback and analysis. This is
supported by Antonioni, (2001) who reports that appraisers can set
their improvement goals and action plans on line. However, there
does not appear to be a method of evaluation uniquely suited to
medical education as yet.
11.1 Researchers look at outcomes from different perspectives and
different stakeholders determine their own important outcomes.
11.2 Several studies have looked into 360° appraisal. Some authors are
positive about the process; others have some reservations. Despite
some reservations, the National Health Service is clearly committed to
the introduction of the 360° appraisal system. (Cheshire & Merseyside
Strategic Health Authority is an early implementer of the process and
is introducing the system beginning with Chief Executives). The
Royal College of Physicians is also piloting the process for Specialist
Registrars and Senior House Officers.
11.3 The key issue appears to be training individuals to give positive and
negative feedback. Individuals/appraisers find it difficult to give
‘negative’ feedback to appraisees for fear of demoralising or
demotivating staff or causing any conflict between manager and
employee or within the team. While training individuals to give and
receive feedback may temporarily increase the expense associated
with 360° feedback programmes, the gains seem to outweigh the
‘cost’ of feedback to participants as they become more focused. The
goal would be to create a culture in which individuals feel comfortable
giving and receiving feedback, both positive and negative.
Having reviewed the available literature, the following appear to be
key to the success of implementing a 360o appraisal programme:
• It must be for the right reasons – consider why you (the
organisation) are doing it
Have you got a strong business case to support the introduction of
it? (If not, perhaps it should not be introduced)
• Assess the costs to the organisation (financial and physical
Ensure this will achieve the best possible results to achieve goals
• Focus on goals and strategy for the organisation’s appropriate
• Get support at all levels of the organisation
Make sure executives play a key and visible role. Also that line
managers/employees are included in appropriate discussion
relevant to their department to have ‘ownership’ of new system
• Train people in giving and receiving feedback (both positive and
Ensure people feel comfortable with the process
• Develop a list of core competencies for all staff
Link competencies to specific job-related skills
• Evaluate the process
Identify areas for improvement
• Monitor the process
Antonioni, D. (1996) ‘Designing an Effective 360-Degree Appraisal
Feedback Process’, Organisational Dynamics, 25(2): 24-38.
Arnold, E and Pulich, M.(2003) ‘Personality conflicts and objectivity in
appraising performance’, The Health Care Manager, 22(3): 227.
Adsit, D. London, M. Crom, S. and Jones, D. (1996) ‘Relationships between
employee attitudes, customer satisfaction and departmental
performance’, Journal of Management Development, 15(1): 62-75.
Brotherton, P. (2003) ‘Meyners Pays for Performance’, Journal of
Accountancy, 196(1), 41-46.
Clark, K. (2003) ‘Judgement Day’, U.S News & World Report, 134(1): 31-32.
Epstein, R. and Hundert, E. (2002) ‘Defining and Assessing Professional
Competence’, Journal of the American Medical Association, 287(2):
Green, B. and Griffin, N. (2003) ‘Personalize Your Management
Development’, Harvard Business Review, 81(6): 130.
Lockyer, J. (2003) ‘Multi-source feedback in the assessment of physician
competencies’, Journal of Continuing Education in the Health
Professions, 23(1): 4-12.
King, J. (2003) ‘360° appraisal’, British Medical Journal, 324(7352).
Pangaro, L. (2003) ‘Assessment and outcomes in medical education’,
Military Medicine, 168(9): 21.
Pfau, B. and Kay, I. (2002) ‘Does 360-Degree Feedback Negatively Affect
Company Performance?’, H.R Magazine, 47(6).
Reichheld, F. and Detrick, C. (2003) ‘Loyalty: A Prescription for Cutting
Costs’, Marketing Management, 12(5): 24-26.
Rodgers, K.G. (2002) ‘360-degree feedback: Possibilities for assessment of
the ACGME core competencies for emergency medicine residents’,
Academic Emergency Medicine, 9(11): 1300-1305.
Storr, F. (2000) ‘360-Degree Appraisal’, People Management, 6(10).
Vinson, M. (1996), ‘The Pros and Cons of 360-Degree Feedback: Making it
Work’, Training & Development, 50(4).
Wimer, S. (2002) ‘The Dark Side of 360-Degree Feedback’, Training &
Development, 56(9): 37-42.