Evaluation of the Centre for Excellence in Developing

Document Sample
Evaluation of the Centre for Excellence in Developing Powered By Docstoc
					    Evaluation of the
 Centre for Excellence in
Developing Professionalism

       March 2010

Evaluation and Service Development Unit,
Educational Development Division,
Centre for Lifelong Learning,
University of Liverpool
128 Mount Pleasant,
Liverpool, L69 3GW


This report has been written with the support of many people. Special thanks must go to Dr
Helen O’Sullivan (Director of the Centre for Excellence in Developing Professionalism)
without whose support and cooperation the evaluation would not have been possible.
Thanks also go to Louise Jaeger (Centre Administrator), whose help in compiling evidence
for the evaluation with respect to outputs was invaluable. More generally, all of the staff of
the Centre who were interviewed have been open and forthcoming in their comments and

                               Information about the author

Dr Mark O’Brien is a researcher in the Evaluation and Service Development Unit based in the
Centre for Lifelong Learning at the University of Liverpool.

                     Executive summary

The Centre for Excellence in Developing Professionalism (CEDP) has
substantially met the original objectives set at its inception.

The Centre has significantly deepened the definition and understanding of
‘professionalism’ for medical education.

New objectives have emerged over the course of the lifetime of the CEDP in
response to new challenges and opportunities, principally that of the
development of learning technology for professional dialogue and mobility.

Collaboration and professional networks have been crucial to the success of
the CEDP.

Through academic publishing, conference presentation and involvement in
key professional bodies, the staff at the CEDP have, between them,
established a national and international reputation for expertise and
excellence in their area.

Research at the CEDP has produced new insight of great value to medical

The research insight produced at the Centre has been harnessed in the form
of practical innovations and new models of practice for curriculum design and
content and for student assessment and support.

Clear benefits have been produced for: the medical professional community;
the medical education research community; medical undergraduates; careers
advisers; medical curriculum managers; and for academics in non-medical
subject area through transferable outcomes.

The support for, and development of, a specialist medical careers advisory
service has been a particularly important and potentially consequential

With effective dissemination the achievements of the CEDP are likely to have
lasting influence and to provide and enduring legacy for the work of its staff.



Acknowledgements                                      iii

Information about the author                          iii

Executive summary                                     v

Introduction                                          1

The methodology                                       1

Findings                                              3

       Part One: Statistical information              3

       Part Two: Evaluative reflection                9

References                                            31

Appendix 1: CEDP research outputs                     33

Appendix 2: Average CEDP staffing                     41


The Centre for Excellence in Developing Professionalism (CEDP) is one of 74 Centres of
Excellence in Teaching and Learning (CETL) that have been supported by the Higher
Education Funding Council for England (HEFCE). Located within the Liverpool School of
Medical Education at the University of Liverpool, it commenced its existence in April 2005.
The work of the Centre has broken down very broadly into: a wide ranging research
programme; and a ‘direct-benefit’ careers advisory service for medical undergraduates. It is
now in its final academic output stage of activity. As the national CETL programme draws to
a close, HEFCE require evaluations of each Centre that was established under the scheme. A
common template has been produced for all local evaluations ahead of a ‘synthesis’ stage
through which these will be merged to produce one national evaluation. This evaluation
report conforms to the methodology and reporting structure set out by that template.


The evaluation template provided by HEFCE comprises two parts: a ‘statistical information’
part (Part One); and an ‘evaluative reflection’ part (Part Two). Set questions are given for
each part with the qualification that if a particular question is not relevant to the CETL in
question then it need not be addressed.

The objective information required for the completion of Part One of the template was, for
the purposes of this evaluation of the CEDP, provided by the Centre Administrator. This has
allowed for each of the questions of that part to be answered straightforwardly.

The information used to respond to the questions that comprise Part Two of the evaluation
has taken the form primarily of testimonial data from a round of interviews with CEDP staff.
These were conducted between December 2009 and January 2010. Each interview took
between 40 and 50 minutes. The questions from this part of the evaluation template were
used to provide a common semi-structured framework for each interview. The guidance
notes for the evaluation states the following:

     These reflective questions have been designed to provoke respondents to step
     back from day-to-day operational activity and take a reflective, longer term and
     evaluative view of the CETL. (HEFCE 2009)

In this spirit, interviewees were encouraged to take a long view of their work at the Centre
and to consider how their outputs related its wider aspirations. More specifically, the
interviewees were asked: to describe their specific roles and projects; then to reflect upon
what the CEDP had achieved generally; and finally to discuss how their work had
contributed to those achievements.

The interviews were semi-transcribed as researcher notes, again working with the thematic
structure provided by the HEFCE template. The mode of analysis chosen was ‘part thematic’
and ‘part-narrative’. This description is informed by the distinction made by Chase (2005:
663) between traditional thematic analysis – in which many interview recordings or
transcriptions are cross-themed to produce findings from a sample of informants within a
defined setting – and comparative narrative analysis – in which the testimony of the
individual informant is analysed in its own terms without reference to data from other
interviews before being subject to comparative analysis. For this evaluation, interview data
was themed from each interview separately and analysed to provide ‘professional-
narratives’ that were then subject to comparison across all interviews to provide some data-

The thematic findings that were produced from the analysis of interview testimony were
supplemented by the evidence made available to the evaluator. Principally this took the
form of a list and verbal account of the academic outputs of the CEDP over its lifetime that
was provided by the Administrator. It also took the form of information contained in
previous evaluations that had been conducted for the CEDP fellowship scheme and careers
advisory service, as well as some insights that were provided by interim evaluation reports.


Part One: Statistical information

a. Name of CETL

       Centre for Excellence in Developing Professionalism

b. Name of Institutions(s)

       Part of Liverpool School of Medical Education of the University of Liverpool

c. Contact name

       Dr Helen O’Sullivan

d. Name of person submitting the self-evaluation

       Dr Mark O’Brien

e. Start date of CETL

       1st April 2005

f. Anticipated end date of CETL

       Still under discussion

g. Lead Director and dates associated with CETL

       Dr Helen O’Sullivan: in post from March 2006

h. Total amount of award: capital and running

       £2 million (capital)

       £2.5 million (running)

i. Briefly describe what the capital was used for

       Refurbishment of offices and rooms of two floors of a block (Cedar House) at the
       University of Liverpool

j. What will these facilities be used for in academic year 2010-11?

       Teaching and office space

k. Average number over lifetime of the CETL of persons employed at any one time

Directorship and management: average fte. staffing: 0.65

Administration: average fte. staffing: 0.75

Developmental: average fte. staffing: 0.6

Student support: average fte. staffing: 0.7

Clinical research: average fte. staffing: 0.8

Academic research: average fte. staffing: 3.5

Total average fte. staffing: 7

(See calculations in Appendix 2)

l. What are staff employed by the CETL towards the end of its HEFCE funded life going to do
when this funding ends?

       One full-time clinical research fellow will return to practice within the NHS.
       One 0.5 fte. senior clinical specialist will return to practice within the NHS.
       One honorary 0.2 fte clinical research fellow will return to practice within the NHS.
       One funded PhD student will be funded beyond the life time of the CEDP by the
       Mersey Deanery.
       Two full-time research fellows will return to University roles.
       The 0.8 fte. Director’s contract continues until March 2011: future uncertain.
       One 0.8 fte. careers advisor’s contract continues until March 2011: future uncertain.
       One 0.2 fte careers advisor will continue working within the University’s careers and
       employability service.
       One full-time administrator: future uncertain.
       One full-time learning technologist: future uncertain.
       Three full-time research fellows: future uncertain.

m. Number of ‘spin out’ projects funded

CEDP fellowship projects

                Project title                      Award date      Status      Amount
   1. Valuing differences: using the Myers        January 2007   To be        £1,295
       Briggs Type Indicator (MBTI) to                           written up
       promote better learning, personal
       and professional development for
       undergraduate medical students in
       problem based learning (PBL) groups
   2. Medical students’ notions of knowing        January 2007   To be        £2,686.81
       in a problem based curriculum                             written up
   3. Bursary to attend conference -              January 2007   N/A          £856
       ‘International dimension to
       professionalism research’ in Cape
   4. Students’ as examiners; are they            January 2007   Ongoing      £2956
       effective and what are the benefits
   5. Developing ‘diversity awareness’            January 2007   Ongoing      £2,950
       workshops for medical students
   6. Doctor and patient in ancient Greece:       May 2007       May 2009     £2,340
       the historical origins of ethical
       medicine and medical
       professionalism in the Hippocratic
   7. Promoting a professional response to        May 2007       June 2009    £3,000
       medical students with psychological
       health problems
   8. The challenges of supporting                May 2007       Ongoing      £3,010
       international students in developing
       their learning
   9. Writing and reflecting: exploring the       Sep 2007       May 2009     £2,975
       use of wikis and online peer
       assessment tools to promote the
       personal and professional
       development of undergraduate
       medical students
   10. Deaf awareness: piloting a short           Jan 2008       July 2007    £1,472
       course for medical students
   11. Investigating professionalism in           May 2008       Ongoing      £18,250
       medicine in team working
   12. Bursary to attend AMEE conference,         May 2008       N/A          £960
       Prague, Sept 2008
   13. Women in positions of leadership           May 2008       Aug 2009     £16,684
       within the health care setting

   14. Developing medical insight using             Sep 2008   Sep 2009     £1,166
       visual arts: in collaboration with the
       Tate Liverpool
   15. Evaluating service-user delivered            Sep 2008   Sep 2009     £970
       Learning Disability Awareness
       Training for medical students
   16. Audio vignettes, the hidden                  Sep 2008   Ongoing      £3,500
       curriculum and the development of
   17. Get Involved 4 the Future: digital           Jan 2009   Sep 2009     £1,170
       storytelling and developing
   18. Identification of the value system of        Jan 2009   Ongoing      £9,795
       successful dental practitioners
   19. Developing professional practice in          Jan 2009   Ongoing      £11,200
       medical and management education:
       a comparative study
   20. Travel bursary – ‘Developing                 May 2009   N/A          £1,244
       professionalism: comparing the
       lessons learned in Liverpool and Cape
   21. Developing the reflective                    May 2009   Ongoing      £3,000
       professional: medical students’ use of
       resources and patterns of learning
   22. Learning and agency in the teaching          May 2009   Ongoing      £3,000
       of professionalism to medical
   23. Professionalism attitude survey in           May 2009   Ongoing      £3,778
       archaeology: testing the Liverpool
       Professional in archaeology as a non-
       cognate discipline

n. Number of peer reviewed outputs published that relate to CETL work

       Nineteen peer reviewed publications (more papers submitted, review pending)
       One invited review
       Six peer reviewed invited contributions: keynote presentations and workshops
       Nineteen peer reviewed oral presentations
       Twenty-five peer reviewed poster presentations

    (see Appendix 1 for full list of academic outputs)

o. Number of events held to develop or disseminate work beyond the CETL in the last three


This took place over 27-28 November 2006. It was an academic style of launch with 50
invited guests. Keynote speakers were: Professor Valerie Wass (Head of the Keele University
Medical School); Professor Richard Griffiths (Liverpool School of Medical Education);
Professor Louise Arnold (Associate Dean and Director, Office of Medical Education and
Research, School of Medicine University of Missouri-Kansas City); Professor Ronald Harden
(Centre for Medical Education, the University of Dundee); John Mclaghlan (Editor in Chief of
Medical Education and Professor of Medical Education the University of Durham). This event
successfully communicated the existence of the CEDP to an audience of influential academic
figures in the field of medical education. Many of these contacts were to be of enduring
value for the work of the Centre.

Symposium I

This took place on 6 February 2008. Keynote speakers were: Professor David Stern
(University of Michigan Medical School); and Dr Vikram Jha (Leeds Institute of Medical
Education, University of Leeds). Sixty two participants attended the event. At this event
CEDP researchers presented interim findings and work-in-progress reports of research
projects. In the Director’s view, important lessons were learned for the organisational
aspects of symposium management that were important to the success of subsequent

Symposium II

This took place on 24 March 2010. Over 75 participants took part. Keynote speakers were:
Dr Walther van Mook (Maastricht University Medical Centre); Professor Sean Hilton (St.
George’s Medical School); Dr Helen O’Sullivan (Director of the CEDP); Dr Kevin Eva, Editor of
Medical Education and Associate professor at the Department of Clinical Epidemiology and
Biostatistics, McMaster University). This highly successful symposium showcased the key
outputs of the CEDP and demonstrated the Centre’s high level international linkages and


These dissemination events were organised in partnership with the Kings Fund and the
Royal College of Physicians. They took the team to fifteen medical schools around the UK
and were important in providing a vehicle for dissemination of the work of the Centre and in
raising its profile in the world of medical education.

p. If any appropriately experienced member of your CETL team wishes to participate in a
peer review scheme of CETL self-evaluations, please provide their names and contact
details, especially email.

None forthcoming

Part Two: Evaluative reflection

Question 1: Please reflect on how effective your CETL has been in contributing to the
objectives set out for the CETL initiative when it started.

The substantial outputs and significant outcomes the CEDP has produced, as well as the
assessments of key professionals and those of the team themselves, strongly suggest that is
that its objectives have been substantially met. These objectives were met through five key
drivers of: research programmes; the establishment of a medical careers advisory service;
innovations in learning technology support; a fellowship scheme; and influencing change in
the undergraduate medical curriculum.

                               advisory                    Fellowship
                                service                     scheme

                           Research                            Curiculum
                         programmes       Objectives          enhancement

Crucial to the success and reputation of the staff of the CEDP has been their style of working
with professional associations, agencies and key individuals. Collaboration, networking and
the sharing of good practice have been key characteristics of the CEDP. Important
collaborative links have been with: the Mersey Deanery; the Centre for Career Management
Skills at the University of Reading; the Centre for Medical Education at the University of
Lancaster; the Medical Careers Advisory Network for the Association of Graduate Careers
Service Advisers; the University Hospital Aintree; the Northern Medical Schools Personal and
Professional Development Consortium; and many other institutions. Individual researchers and
clinical fellows have also brought their personal professional networks into the CEDP to
great effect for the full range of projects.

Within the Liverpool School of Medical Education (LSME), CEDP staff have been central to a
processes of curriculum review and enhancement. Areas of development such as guidance
to students for problem based learning and the introduction of more structured learning
objectives are now being incorporated. Significant curriculum innovation has also resulted
from the design and introduction of a special study module on career choice that
encourages the student to reflect upon the factors that influence their decisions about
speciality options.

A major innovation has been the introduction of a specialist medical careers adviser. This
service for students has been evaluated through in-depth interviews with informed key
medical and non-professionals and by student survey (O’Brien 2010). The findings revealed
very high professional assessments of the value of the service to the School and similarly
high satisfaction ratings by medical undergraduates. The recommendation of that
evaluation was that this service be retained after the lifetime of the CEDP. At the time of the
publication of this present report, no decision had been made public about this. The
evaluation also found that the model of a specialist careers service was likely to be
transferable to other academic disciplines that are linked to specific professions. One
indicator of the value of this service is the way in which the expertise of the post-holder has
been used at a national strategic level by, for instance the Quality Assurance Agency for
Higher Education for their 2010 good quality and standards in careers guidance for higher
education (QAA 2010).

Supplementing and supporting its developmental work, the CEDP also established an active
research programme, winning it a reputation as a centre of expertise in medical
professionalism. Some specific research projects are given here:

       Leadership. Leadership has been the focus of one significant research project. This
       project has looked particularly at leadership in the context of team-based processes
       and has borrowed its research model and conceptual framework from the ‘crew
       resource management’ approaches of the aviation industry. This model considers
       the human processes involved in high risk and high stress scenarios. Using facilities
       at the Merseyside and Cheshire Simulation Centre at Aintree , the research has
       particularly focused on ‘situational awareness’ and, linked with this, the ability of
       medical students to challenge decisions made by senior colleagues.

       Communication. One research project has considered the factors that contribute to
       excellent communication in the clinical setting. This takes into account many aspects
       of the medic’s communication skills including: those linked with inter-professional
       dialogue; communications with patients of different ages; communications with
       patients whose first language is not English; and the communication and handling of
       sensitive information.

       Professional reflection. The ability of medical teaching staff to support student
       learning through greater depth of understanding has been enhanced through
       research into the nature of reflection for learning as well as reflective professional
       practice. Based upon the California Critical Thinking Disposition Index and using
       Kember et al.’s (1999) Reflective Index Questionnaire, this project revealed
       statistically significant correlations between the student’s ‘critical thinking level’ and
       their ‘reflective level’. This insight is now informing the teaching of critical thinking
       within the MBChB curriculum. One special study module in critical thinking has been
       introduced into the curriculum offer.

       Employability. The employability of medical students is being enhanced greatly
       through the work of the medical careers adviser. This has proved to be of crucial
       importance to medical undergraduates in helping them to think and plan for their
       speciality choices as they approach Foundation Doctor status. Also of importance in
       this regard have been the strong and positive working relationships with the Mersey
       Deanery, NHS and the professional networks associated with these. These have
       resulted directly from the work of the careers adviser based within the CEDP, as well
       as from the recruitment of clinical research fellows who have retained their
       professional links and practice. A further example of how the work of the CEDP is
       contributing to the employability of medical graduates is that of the integration of
       peer assessment within the curriculum for the purpose of enhancing
       undergraduates’ understandings of medical professionalism.

       Graduate entrants. One research project that supports medical educators’
       engagement with leaner diversity is looking at the professional qualities of graduate
       entry undergraduate students. The focus of the project is upon the professional
       attributes such students bring to their study and preparation for a career in medicine
       from their previous occupational and professional experience.

Through these and other strands of work, the CEDP has contributed powerfully to
understandings of, and defining criteria for, professionalism within the medical
undergraduate curriculum. In each case work is ongoing to embed the Centre’s findings and
insights into the undergraduate curriculum, as well as disseminating these through papers
presented at the Association for the Study of Medical Education (ASME) and the Association
for Medical Education in Europe (AMEE) conferences, and through journal publication.

Question 2: Please set out the aims and objectives specific to your CETL at the start; and for
each one reflect how well these have been achieved.

The aims of the CEDP were set out in the original bid and cover the four themes of:
improving the teaching and learning of professionalism; developing the assessment of
professionalism; promoting professionalism through career guidance; and providing
mechanisms for evidencing excellence in teaching, learning and assessment (University of
Liverpool CETL bid to the Higher Education Funding Council 2004:2). These were translated
into the five objectives summarised here:

   1. to develop graduates who can demonstrate improved levels of professionalism;

   2. to develop a sector-wide definition of professionalism and tools to assess it that are
      robust and transferable;

   3. to enshrine within the undergraduate curriculum a system for personal
      development, career planning and reflection of professionalism that allows a
      seamless progression to postgraduate practice;

   4. to recognise and reward excellence in medical education;

   5. to be recognised nationally and internationally as a leading Centre for research and
      publication in professionalism in medical education.

Each of these will be considered for evidence of having been met.

1. Professional graduates

   The CEDP has made a significant contribution in this area. This has taken the form of,
   amongst other things, the following:

   Careers guidance

   The types of careers guidance offered to medical undergraduates have been strongly
   linked to the interest in professionalism emerging from the modernisation agenda for
   medicine in the UK (Modernising Medical Careers Working Group 2005). Of specific
   relevance here is the stress that has been placed upon the need for medical
   undergraduates to make high quality speciality choices in their 5th year of study, ahead
   of their Foundation placements.

   Improving the research base

   Several research projects are deepening the research base supporting the development
   of professionalism for medical education. Their foci include: understandings of
   professionalism amongst 4th Year undergraduates; team work and leadership in high
   pressure situations; and communications skills for clinical practice.


   Work that relates to teaching has comprised three strands: promoting reflective
   practice; peer assessment; and the use of technology. The promotion of reflective
   practice and the incorporation of peer assessment have already been highlighted in this
   report. With respect to new applications of technology linked to professionalism, one
   project has developed and trialled a Web 2 social networking platform that supports
   student feedback appraisal, professional skills development and career management.
   This platform allows the undergraduate to access on-line content, reflective

   asynchronous discussion boards and video clips for the purposes of student feedback,
   professional perspectives and general information.

2. Definition of professionalism

   The development of the definition of professionalism has been a priority for the Centre
   from the beginning. Making professionalism an explicit part of the medical curriculum
   has already significantly raised its profile. The intellectual conceptualisation of
   professionalism has also been deepened and strengthened by the range of research
   projects, developing its discourse qualitatively in areas where little previous work had
   been done. Furthermore, it has been facilitated by an active programme of almost
   monthly research seminars that have featured guest academics presenting relevant
   work and findings. These initiatives have also improved understanding of the
   requirements for professionalism in a range of learning and clinical settings.

3. Curriculum reform

   The location of the Centre within the LSME has meant that its staff have been well
   positioned to influence changes to the undergraduate curriculum. The Director sits on
   the School’s senior management team, providing direct involvement in strategic
   discussions relating to undergraduate teaching and learning.

   The introduction of special study modules that relate to career management and
   professional reflection has already been referred to in this report.

   In 2009 the researchers within the team began a curriculum review and this has led to
   recommendations for curriculum change for the problem-based learning pedagogical
   approaches used within the School. In turn, this has resulted in the incorporation of new
   levels of guidance and support for students, updated and re-focused structure in
   learning objectives, the promotion of the skills required for effective career-long self-
   development and enhancements in the use of learning portfolios. A recommendation
   for a comprehensive e-learning strategy was also contained in this report (Graham

4. Rewarding excellence

   A current project within the CEDP is that of developing a ‘professionalism’ award for
   students who can demonstrate professionalism in their conduct and practice.

   The values and personal skills associated with reflective practice and professional self-
   knowledge are also being promoted and developed. Examples are the introductions of:
   student feedback appraisal in the 1st and 2nd Years for the purposes of improving
   portfolios; reflective portfolios in the 3rd Year; undergraduate peer-assessment of
   professionalism; and the special study module in career management for 5 th Year

5. Publishing and recognition

   Moving into its final output phase, CEDP staff have already published and presented
   widely in well respected peer reviewed academic fora and journals. This has been
   quantified in Part I of this report. The quality of this work is evidenced by the highly
   reputable character of the journals that have published these papers. (A detailed listing
   of publications is provided in Appendix 1).

   The national and international standing of staff at the CEDP is evident from the extent of
   the networks they are active within, the invitations to publish and present that they
   receive, the links they have with professional bodies and the external positions they
   hold. For example, the Centre has been a key partner in collaborations with the General
   Medical Council (GMC), the Royal College of Physicians (RCP), the Kings Fund (KF), and
   the NHS. Other aforementioned important links include: Medical Careers Advisory
   Network for the Association of Graduate Careers Service Advisers; the Centre for Career
   Management Skills at the University of Reading; the Centre for Medical Education at the
   University of Lancaster; the University Hospital Aintree; and the Northern Medical Schools
   Personal and Professional Development Consortium.

   The international aspect of the reach of the work of the CEDP is also evident in the
   nineteen papers and posters that have been presented at the annual AMEE conference.

Question 3: Please add any objectives that emerged as the CETL developed, and reflect on

One key area of development across the work of the CEDP, which was not anticipated in the
original bid for the University of Liverpool CETL, was that of learning technology to support
professionalism. The potential of Web 2 social networking software for areas of work linked
to CEDP objectives is now being realised through projects developing e-portfolios, student
feedback appraisal for teaching and on-line undergraduate professional dialogue.

Naturally, many of the directions of research could not have been foreseen in their detail by
the authors of the original bid. Research projects looking at emotional intelligence in the
clinical setting, women in positions of leadership in medicine and the role of affective states
and personal value-linked inclinations such as altruism, cynicism and empathy, are all
examples of work responding to research intelligence needs identified by the research

Similarly, the CEDP Fellowships scheme has led to original directions of thinking and
discovery, which respond to current agendas within the NHS and may lead to further new
directions for research. Some notable examples of such fellowships, that illustrate the
creative and original thinking that has characterised the CEDP, are given here:

       evaluating service-user delivered Learning Disability Awareness Training for medical
       promoting a professional approach and response to the needs of medical students
       with psychological health problems;
       supporting international students in developing their learning;
       developing non-stigmatising pathways to support students experiencing mental
       health problems;
       developing medical insight using visual arts in collaboration with the Tate Liverpool;
       identifying the value systems of successful dental practitioners;
       exploring medical students’ notions of ‘knowing’ in a problem-based curriculum;
       assessing the effectiveness of students as examiners;
       investigating professionalism in medical team working;
       developing professional practice in medical and management education;
       exploring the use of wikis and online peer assessment tools to promote the personal
       and professional development of undergraduate medical students;
       researching the potential of audio vignettes for teaching and learning in the
       operating theatre;
       researching digital storytelling to develop professionalism;
       piloting short ‘deafness awareness’ courses for medical students;
       developing ‘diversity awareness’ workshops for medical undergraduates;

       researching the experience of women in positions of leadership within the health
       care setting.

Final challenges for the CEDP team are those of sustaining the work that has been done,
harnessing the insights that have been gained and embedding changes to the medical
undergraduate curriculum and improvements in teaching for medical professionalism. This
embedding work is currently central to the CEDP activity, and is taking the form of research
dissemination, seeking support for the continuation of research, and feeding into strategic
discussions around the recommendations of the Creating Tomorrow’s Doctors (Graham
2009) report on the review of the MBChB programme.

Question 4: Irrespective of your answers to questions 2 and 3 above, please reflect on, and
draw out the achievements and benefits of the CETL.

This questions calls for a reflection on the achievements and benefits of the CEDP over its
lifetime. This is broken down here into six areas: 1. benefits to the medical professional
community; 2. research outcomes; 3. student benefits; 4. career enhancement; 5.
curriculum benefits; 6. and transferable outcomes.

   1. Benefits to the medical professional community

       One major beneficial outcome of the work of the CEDP has been to substantially
       develop the concept of ‘professionalism’ within medical education and, by
       extension, for medical practice. Interview testimony revealed a view that research
       on professionalism that predated the work of the CEDP, whilst valuable, had been
       predominantly quantitative in nature and designed to support human resource
       management techniques of professional assessment. The research findings and
       developmental work of the CEDP, on this view, was distinctive in having significantly
       moved what we might call the ‘professionalism discourse’ onto more qualitative
       ground. With this shift, the meanings of professionalism can be explored in their full
       depth and complexity. This also facilitates the kinds of reflective practice demanded
       by the current professionalism agenda in British medicine.

   2. Research outcomes

       The broad range of research conducted by CEDP staff has already been highlighted in
       this report by reference to specific projects. This rich and multi-focused range of
       projects has contributed to the aims of the CEDP in three ways: engaging students

   and medical education staff in professionalism related research; generating useable
   data and producing insight that can be harnessed for curriculum design; and raising
   the profile of various aspects of professionalism for medical education through
   conference presentation and publication.

3. Students benefits

   The careers element within the CEDP was established as a careers advisory service.
   This service has been evaluated and the evidence presented in that evaluation
   strongly indicated that the aim of providing direct benefits to medical
   undergraduates is being substantially met (O’Brien 2010). This evaluation reported
   that 4th and 5th Year undergraduates in particular appreciated this service for the
   guidance and support they had been given with regard to speciality choices, C.V.
   design, and intercalation. The service is becoming increasingly embedded within the
   LSME with a steady, year-on-year increase in contact with students seeking

   A number of research projects have produced and are currently producing findings
   that will, if used to improve curriculum design and student support, be of benefit to
   students. One fellowship that has produced insight into ways in which support for
   undergraduates with psychological health problems can be improved, will contribute
   directly to student welfare work. Fellowships that have focussed upon types of
   disability or vulnerability on the part of patients have brought important lessons for
   diversity awareness amongst medical undergraduates and Foundation doctors. The
   wide range of technological aids for student support, teaching and assessment has
   also been explored for their applicability for medical education through the
   Fellowships scheme. Two examples of this have been: one project exploring the
   potential of wikis for professional dialogue; and the use of ‘audio-vignettes’ for
   feedback from clinical training sessions.

4. Career enhancement

   There are several senses in which the CEDP has enhanced careers. A number of
   members of staff, for instance, who have conducted research within the Centre,
   have done so in the form of PhDs. Naturally, this means that these pieces of research
   are not only valuable in terms of the insight they produce, but also for the qualified
   status they provide for individual researchers and the areas of medical education or
   professional development that may move into later in their careers. All members of
   staff who were interviewed, without exception, felt that their careers had been
   significantly enhanced by working at the CEDP, and by the professional profiles they

   had achieved as a result. Similarly, staff in various schools across the University, who
   had received fellowship awards have benefited in having been able to develop areas
   of their professional expertise and interest.

5. Curriculum benefits

   Those professionals responsible for the content and structure of the undergraduate
   curriculum will benefit from the work of the CEDP. The different strands of research
   it has supported have enabled new understandings of the meaning of
   professionalism in the medical context to be harnessed for the curriculum. This is
   true with respect to: the uses of technology already referred to; diversity awareness
   work; the teaching of aspects of professionalism; and the harmonising of the
   undergraduate curriculum with professionalism and the current modernisation
   agenda by the introduction of relevant special study modules and appropriate
   guidance for career awareness and decision making.

6. Transferable outcomes

   There are two ways in which the outcomes of the work of the CEDP have
   transferable potential at the general level. Firstly, the developments to the concept
   of professionalism that have been, and are still being achieved, are of direct
   relevance to other academic disciplines that have direct links to professions.
   Examples may be law, business and management studies, social work, teaching,
   accountancy etc. This list could obviously be greatly extended. The key aspects of
   professionalism that have been the foci of the CEDP’s research such as team
   working, situational awareness, communication in the professional environment,
   career management, student understandings of professionalism etc. could all be
   directly applied to these non-medical professions.

   One other sense, in which transferability has been a hallmark of the work of the
   CEDP, is that of the models of practice that have been designed and implemented.
   Principally, this refers to the model of the careers advisory service that it supported.
   The fact that this service was a dedicated medical undergraduate service, based
   within the main building of the LSME, meant that the advice it offered was
   appropriately specialised. Moreover, it was also able to provide advice and guidance
   that was consistently relevant to a fast changing careers agenda, as well as being
   able to promote the kinds of skills required of the qualifying and newly qualified
   medic in the early stages of their career. This kind of specialist careers support may
   well be of great value to non-medical as well as medical academic subject areas.

Question 5: Have there been any disappointments in how the CETL has developed or what it has
achieved? What are they? Why did they happen?

The overriding sense amongst staff at the CEDP is one of achievement and having met the
original aims and objectives laid out at its inception. Nonetheless, in interviews there were
some examples given of disappointments at strands of work that had not developed as well
as originally hoped. Here, a distinction will be made between disappointments at the
general level, and those that are more specific to projects or individual strands of work. In
each case, drawing upon the opinions and views of the interviewed staff, some speculation
will be given as to why such disappointments exist.

Generally, there was a view across many of the interviews conducted, that in the early
stages of its development, that the Centre had struggled to establish itself within the LSME.
Although expressed quite subjectively, these opinions are nonetheless significant in that
they were given by a number of interviewees and so cannot be seen as isolated instances of
personal frustration. This theme is discussed more fully in the response to Question 6

Other examples of disappointments were more specific. One example of an area that, with
hindsight, might have been approached differently and to more effect, was that of student
involvement in research. Although this situation improved over the Centre’s lifetime, and
particularly with the appointment of officers from the Liverpool Medical Students’ Society
as research fellows, this was seen by some interviewees as something that might have
informed the design of its work from an earlier stage. Finally, a specific disappointment was
expressed in relation to the ‘mobile technologies’ strand of work. The aim of this initiative
had been to develop and pilot mobile platforms for undergraduate professional dialogue
and portfolios. The constraints of existing University student support systems however has
resulted in it not moving substantially beyond the early planning stage. This example merely
illustrates the difficulties of achieving innovations within an institutional setting where
human and technological systems are already embedded. It does however, highlight the
extent of the achievement represented by those innovations that were introduced as a
result of the Centre’s work.

Question 6: Please reflect on the difficult and easier aspects of getting the CETL going and of
getting your messages across.

The problems outlined above are linked with structural and cultural barriers that were
encountered by the CEDP team in establishing the Centre within the LSME. This does not
mean that the story is only one of barriers and frustrations. Indeed, as this report has
highlighted the overall story is one of a sense of success and achievement. Moreover, the
difficulties faced by the CEDP staff were largely overcome with the support of colleagues

within the School itself. Although the response to this question will focus primarily upon the
difficulties faced then, the comments will also draw out the ways in which problems were
solved through active collaboration with colleagues within the LSME.

It is important to appreciate that the original bid was initiated, written and submitted from
within the LSME by a team led by its then Head. Nonetheless, according to the testimony of
many of the original staff of the CEDP, the relationship between it and the School in the first
year, was not one of active engagement. The Head of the School and originator of the
successful bid for a CETL based at the University of Liverpool, moved on to a new post
outside of the City and was not replaced for two years. The early retirement of the
individual originally designated to the lead professional role within the Centre was also a
setback. In this era of the Centre’s development little in the way of active structural
connection with the LSME at the senior administrative or strategic level existed. Referring to
this era, several of the evaluation research interviewees described having felt themselves to
be outsiders within the School.

The early phase of the Centre’s relationship with the School was characterised by what one
interviewee described as ‘institutional inertia’ rather than active resistance; one in which
CEDP staff found themselves having to compete for institutional support and attention in an
environment of multiple and competing agendas. The University of Liverpool, a member of
the Russell Group of research-intensive universities in the UK, was in this period
commencing period of a major change in the tempo of its institutional research agenda. The
upcoming Research Assessment Exercise that was scheduled for final submissions by
December 2008, was already exerting significant pressures upon the research output work
of all academic departments and the then Faculty of Medicine (now the Faculty of Health
and Life Sciences) was no exception to this. In this environment, a Centre devoted to
research for teaching and learning, which was not scheduled to reach its full output stage
until 2010, perhaps inevitably faced obstacles to making the kind of swift impact its staff
would have liked to have achieved.

Other, more internal factors to the CEDP were also mentioned in the evaluation research
interviews, and are worth considering here. One is that most of the members of research
staff employed were either relatively new to research, were early career researchers or had
come from clinical practice. No established academics with existing and recognised track
records in relevant areas of medical education research were appointed. The number of
PhDs supported by the CEDP has already been cited as a distinctive and crucial strength in
relation to the career development of individuals, and is indeed important to its longer term
legacy. Still, it is likely that this strategic design decision affected the ability of the team to
shape the medical education agenda within the School in its early phases of work. One more
factor, that was mentioned in a number of interviews, was that the ‘badging’ of the Centre
was not clear in the first two years of it existence. In this period the Centre was known and
profiled by the generic title of the Centre for Excellence in Teaching and Learning; as ‘the

CETL’. Eventually, 74 CETLs were supported by HEFCE around the UK, each with a specific
focus of specialism and expertise. The Liverpool CETL only became the CEDP in 2007, nearly
two years after its inception. On this view, it was only then that the Centre gained a clear
institutional identity that signalled its specialist focus on professionalism for medical
education to a wider public. Again with hindsight, this may have been a small contributing
factor in the early difficulties faced by the team in establishing the Centre within the

Key turning points contributed to improving the internal and external profiles of the Centre.
In November of 2006 the CEDP was launched. The first day of the launch was a social and
celebratory event. The second day, however, took the form of an academic symposium with
invitees carefully selected for their positions within the LSME and the academic-professional
world of medical education. This launch was described as having been a great success
measured by the significance of the figures that it attracted, and the ways in which it
established the Centre as promising new directions for the professionalism discourse within
medicine; of putting the Centre ‘on the map’. Another turning point came in the Spring of
2007, when the newly appointed team had come into post and were beginning to cohere
with a clear sense of identity and purpose.

As already stated, early difficulties encountered by the Centre were also mitigated and
largely overcome by developments and changes within the LSME. The arrival of a new Head
of School, after a two year period in which this post had been vacant, particularly heralded a
significant improvement in the Centre’s strategic position. The new incumbent had strong
links with the Mersey Deanery which would prove to be important for the careers strand of
the Centre’s work. Also, shortly after this appointment, the Director of the Centre was
brought onto the senior management team of the LSME. These developments brought a
level of structural connection between the Centre and the School that greatly enhanced its
impact. CEDP representation on the Full Review Group of the highly consequential review of
the MBChB Programme, for instance, was one indicator of this improved influence within
the School.

Question 7: Has your CETL adopted/used/been based around any specific theories, e.g. of
change, or of student learning? If so, what, how have these underpinned your work, have
they been useful?

No specific theories of change or of student learning have underpinned the activities and
work of the CEDP.

Nonetheless, in response to this question, evaluation research interviewees described
characteristics to the work of the Centre that together amounted to a strategic approach to
achieving change, albeit in a pragmatic, untheorised manner.

The involvement of staff of the CEDP as key partners with influential professional bodies
such as the General Medical Council, the Royal College of Physicians and the Kings Fund and
within various professional networks has already been mentioned in the response to
Question 2 of this report. These partnerships were described as having been important to
the CEDP in achieving influence and in having impact upon the theory and practice of
medical education. More specifically ‘road shows’ organised with these key medical
institutions have taken CEDP staff to fifteen schools of medical education around the UK.
These events have provided a powerful vehicle for the national dissemination of the
Centre’s work.

The position of the Head of the CEDP on the senior management team of the School, as well
as on other committees within the Faculty of Medicine (now the Faculty of Health and Life
Sciences), has provided some internal strategic leverage to the CEDP. One important
example of this, referred to in the response to Question 6, has been the role of CEDP staff in
the MBChB curriculum review process.

At a far more general level, interviewees described a culture within the CEDP whereby team
discussion and reflection on the changing environment in which it had to operate, was
normal and routine. This aspect of the Centre was said to engender flexibility and
responsiveness as the team navigated their way through the challenges they faced and the
opportunities they were presented with.

Question 8: Reflecting on the last five years what other important messages are there that
you want to convey about your CETL – its successes, difficulties, impact etc?

A range of general thoughts and specific comments were forthcoming in response to this
question. They are listed here.

       In the institutional context of a research intensive Russell Group university, the task
       of establishing excellence in teaching and learning is necessarily complicated by the
       research oriented agendas that prevail. Where HEFCE seeks to develop strategies for
       teaching and learning in the future, proper cognisance needs to be taken of this
       reality. This could mean the development of new pedagogical styles and
       methodologies that link teaching strategies with research in the forms of ‘research-
       led teaching’ or ‘research for student learning’ approaches.

       Networks are crucial to impact. The institutional, professional and personal networks
       through which CEDP staff have worked have been invaluable. Networking, however,
       requires skills that may not come naturally to the individual and resources that may
       not be ready to hand unless explicitly budgeted for. Networking then, perhaps
       should be given particular prominence as a criterion for successful bids for similar
       future initiatives.

       The full impact of a five year initiative, working within an institutionally ‘crowded’
       university environment, is likely to only achieve full impact in the final stages of its
       work. This is especially the case where, as with the CEDP, a substantial investment
       has been made into research. The embedding of outputs (recommendations for
       curriculum reform, theoretical insight into specialist support for students and so on)
       is therefore crucial to final success. Moreover, networks established during the
       lifetime of the initiative can remain valuable after it has ended. This makes the types
       of ‘exit strategies’ employed important for the legacy of the work carried out. This
       may be an area that would repay greater strategic focus by HEFCE.

       Although some links with a small number of CETLs around the UK were established,
       the Centre operated largely outside of the national network. The main focus of its
       work was within the LSME and the network of relevant professional medical bodies.
       With hindsight, a central strategy of identifying common organisational interests or
       areas for potentially fruitful collaboration, could have opened up routes through
       which findings, insights and outputs might have been shared across the national
       CETL network.

       The PhDs that have been supported by the CEDP have already been cited as an
       important outcome, both in terms of research insight and of transferable skills. As
       the CEDP comes to an end, however, the individuals concerned have, in number of
       cases, not yet finished their theses and are yet to publish their findings. In these
       circumstances there is a case to be made for an extension of support by HEFCE to
       enable these individuals to successfully complete their work for submission.

       The incorporation of findings, insight and recommendations for student support,
       improving teaching with respect to professionalism and enhancing the curriculum, all
       testify to the value of the work of the CEDP. With the end of the CEDP, only two
       short-term, externally funded medical education research posts will exist within the
       School. There may be a case then for sustaining a level of research in the longer term
       for the purpose of ongoing curriculum improvement.

Question 9: Reflecting on the last five years what important messages are there that you
want to convey about the experience of being part of a wider ‘movement’/experience of
other CETLs?

As reported in the responses to Question 8 of this report, the CEDP did not through the
national CETL network. Some efforts were made in the very earliest phase of the life of the
CEDP by the incoming Director to identify other CETLs that might have had some common
ground with it, but this search revealed little apparent potential for inter-CETL collaboration.

Evaluation research interviewees conveyed no sense at all of having felt a part of a ‘wider
movement’ of other CETLs.

Question 10: Please reflect on work emerging from your CETL that has been ‘transferable’,
i.e. useable beyond the home audience for which it was originally developed.

Examples of transferable outputs of the CEDP have already been given to illustrate themes
in this report. The theoretical insights provided by these pieces of work have either been or
are now being disseminated through the normal academic routes of conference
presentation and peer-reviewed publication. For each example, however, there is a case for
developing dissemination approaches that take the lessons learnt and key messages out to
practitioner and applied professional audiences.

These transferable outputs are repeated here with some additional examples.

       The focus of the Centre’s work in ‘professionalism’ means that the greater part of its
       outputs could find application in many, and arguably all, professions in the UK and
       beyond. Examples include:

           o one research project focussing on developing understanding of direct
             communication in the professional environment;

           o links that are being developed between a ‘professional communications’
             research project (referred to above), and the School of Psychology through
             work using ‘attachment theory’ in the professional setting;

           o one research project focussing on understanding ‘situational awareness’ in
             pressurised and stressful professional environments and, linked with this,
             developing the skills required to question or, when needed, challenge
             decisions made by more senior colleagues;

           o one research project and one conference organised in collaboration with the
             Mersey Deanery, focussing on understanding ‘leadership’ in the professional

           o one research project that is based upon a collaboration with the Open
             University, focussing on the application of Web 2 technology for inter-
             professional communication and exchange, and on developing associated
             guidelines and advice;

           o the model of curriculum evaluation developed (including student feedback
             appraisal) for the LSME that could be easily adapted for other health related
             areas of applied study such as dentistry, nursing etc.;

           o one research project that has developed the model of peer appraisal for
             undergraduate student assessment; a model that could be transferred to any
             academic subject area or profession.

The model of a dedicated careers advisory service that combines generic careers guidance
expertise with specialist knowledge of medical careers has been highly appreciated by
medical students (especially 4th and 5th Year undergraduates) (O’Brien 2010). This is clearly a
model that could be applied to many other academic schools and departments, particularly
those that are vocationally linked to a profession.

The transferable potential of the medical careers advisory service lies also in the
contribution that it can make to graduate employability agendas across the higher
education sector. Locally, at the University of Liverpool, all academic departments and
schools work towards the model of the ‘Liverpool Graduate’ that has employability as one of
its defining characteristics.

Question 11: How will the work and achievements of your CETL continue after HEFCE
funding ends?

The embedding of practical and strategic innovations for medical teaching has been a
concern of CEDP staff from the outset. The current, final phase of the Centre includes a
significant level of embedding activity. Work related to changes to the curriculum for
instance, is becoming embedded through the MBChB review and its recommendations. The
review, which was conducted over the 2008/9 and 2009/10 academic years, contains
recommendations that derive directly from the work of the CEDP.

Several interviewees emphasised the point that one of the key achievements of the CEDP
has been to make professionalism an explicit part of the curriculum within the LSME.
Whereas professionalism was always an implicit element, permeating the curriculum on
many levels, it was not highlighted for direct teaching per se. The work of the CEDP has now
put professionalism to the fore. This is now borne out by the following recommendation
contained in the curriculum review.

      Recommendation 6: The work of the Centre for Excellence for Teaching and
      Learning (CETL) should be incorporated into the Medical School to continue the
      development of professionalism and leadership. (Graham 2009:5)

The learning technology strand of the Centre’s work has also been similarly embedded in
the strategic directions of the LSME and the curriculum development work within it. A note
of frustration with the pace of movement in this area was cited in response to Question 5 of
this report. Despite this, the inclusion of e-learning in the curriculum review as a priority for
the School, testifies to a significant shift towards more online and virtual approaches to
learning that are current across modern teaching agendas within medicine and across the
higher education sector.

      E-learning strategy
      Recommendation 7: A comprehensive e-learning strategy should be developed to
      underpin the delivery of the programme. The strategy will include the
      development of a website with clear links to support students and those

      delivering the programme, both within and outside the University. (Graham

Indeed, the innovations made possible by such e-learning platforms may have further
applications as the School moves towards the development of learning portfolios for all
medical students, in anticipation of the Foundation stage in which every medical graduate
has to carry a portfolio. Although subject to the delay described in the response to Question
5, one strand of the work of the Centre has been an exploration of the potential and
applicability of mobile technologies for learning and for e-portfolios. This would all relate to
Recommendation 10 of the curriculum review.

      Learning Portfolio
      Recommendation 10: The use of a Learning Portfolio should be developed for all
      students. (Graham 2009:5)

Significant work has been done by staff within the Centre in the area of student feedback
appraisal and peer assessment of professional behaviour for medical undergraduates. These
approaches, and the creative thinking that supports them, have had an impact beyond the
immediate research project and have indeed had resonance more widely across the LSME.
Again this is reflected in one of the recommendations of the curriculum review.

      Assessment and feedback
      Recommendation 9: There should be an ongoing review of assessment which will
      enable it to drive and support appropriate, sustained learning. At the same time,
      processes will be put in place to enhance feedback to students. (Graham 2009:5)

With respect to the career management element to the Centre’s activities, recommendation
for the post of dedicated careers adviser to be continued within the LSME has been made by
the evaluation of this role that was completed in February 2010.

      Recommendation 1. The post of Careers Adviser within the CEDP should be seen
      as an indispensible service and retained and supported by the Liverpool School of
      Medical Education and the University of Liverpool. (O’Brien 2010:23)

At the time of publication of this present report the continuation of the post was still
uncertain. Nonetheless, the careers management approaches developed are already
incorporated into the teaching and learning culture within the LSME in the form of special
study modules and other curriculum enhancements.

Beyond the work connected to the curriculum review, there are other means by which the
CEDP’s achievements will be sustained in the longer term. One way in which this will happen
is through the acquired expertise that staff take with them as they move into new positions
of employment. In the cases of the full-time clinical fellow and the clinical specialist, this will
entail a return to practice within the NHS, thereby creating new links between the LSME and
the NHS and other professional medical networks. In the cases of two research fellows this
will mean moving into new positions with the LSME, so keeping research knowledge and
insights within the School.

The completion of doctorates by CEDP staff can also be seen as a means by which its
achievements will become incorporated for the purposes of teaching and learning. Each of
the members of staff undertaking PhD study declared their intention to pursue academic
careers within medical education and related fields.

Question 12: Do you think there are any emerging aspects of your CETL activity that will
have greater importance in the future?

All of the staff at the CEDP who were interviewed for this evaluation conveyed strong
senses, both explicitly and implicitly, in having made a real difference in their respective
sub-fields of professionalism related research and practice, and a belief that their work
would have longer term impact beyond their specific projects. Some areas for which this
was especially true are listed here.

       The Centre has gone a long way towards establishing the link, both in concept and in
       forms of working practice, between professionalism and career management. This is
       certainly the case with student awareness (especially with 4th and 5th Year
       undergraduates) and also with educational staff within the LSME. The need for high
       quality, well informed decisions regarding speciality, along with well grounded
       preparation for the post-qualification F1 and F2 stages, is now more firmly
       established than before the Centre’s existence.

       The reflective approaches required of the modern medical undergraduate, as well as
       for the Foundation doctor, have been hugely facilitated and enhanced by of the
       existence of the CEDP. The approaches embodied in the student feedback appraisal
       for example and the 3rd Year reflective journal that have been developed, are likely
       to be highly beneficial as they become increasingly embedded.

       The meanings of professionalism within medicine have been explored and developed
       for assessment in many more of its aspects and complexities. This work has been
       ground-breaking and, by virtue of its quality and profile, is likely to provide
       foundations for others to follow, so progressing the qualitative development of the
       concept of professionalism that the Centre has pioneered.

       The networks established through the CEDP have proved extremely valuable for the
       LSME, as well as for its own work. The Centre’s staff have for instance contributed to
       the strong and positive links between the School and the Mersey Deanery. Being
       highly valued, these are likely to be sustained in the longer term.

       The involvement of students in research is something that fits very well with the
       delivery of medical education to undergraduates as well as with research in that
       field. The possibilities that it offers to undergraduates to achieve publications will
       also greatly enhance the C.V.s of students who take up the opportunity. Again, this
       will enable these undergraduates to make effective decisions with respect to their
       own careers by virtue of the specialist knowledge they obtain in this way. It is also

       likely to enable some undergraduates to develop research skills that open up the
       new horizons for them as academic medical researchers.

Question 13: Any other comments

One final characteristic of the CEDP, something that is not captured in the responses to the
preceding questions asked by the HEFCE evaluation template, is the obvious enthusiasm
that has surrounded it and that has permeated its work. Previous evaluations of the
Centre’s work have provided a picture of very high levels of professional regard and
appreciation of what it has contributed and achieved on the part of key professionals who
are or have been in a position to provide informed judgements. The evaluation of the
Fellowships scheme (O’Brien 2009) provided academics in a wide range of subject areas
with the opportunity to develop creative projects that would otherwise have been unlikely
to receive support. This was greatly appreciated by the CEDP Fellows themselves. During the
evaluation of the Careers Advisory Service senior academic and administrative figures
repeatedly expressed the high esteem in which they held the Centre and its work. (O’Brien

More than this however, an air of pride in the work being done was conveyed very clearly by
the staff of the Centre themselves. In ways that are sometimes quite intangible, difficult to
measure and only apparent through person to person inquiry, the level of positivity felt by a
team, or individuals within it, will nonetheless be communicated to an evaluator. In the case
of this evaluation the sense of achievement was abundantly evident. One aspect of this
sense of achievement, or perhaps more accurately, one reason for it, was the collegial ethos
of the team that was described in all of the evaluation research interviews, and that had
been established from the outset. The story here was one of sharing ideas, joint efforts
towards presentation and publication, and general atmosphere of intellectual cooperation.

Although it cannot be proven in the manner of a science, the view across the team was that
this mutualism had been important for the prolific output achieved and imaginative work
done. It is this work of course that has won for the CEDP team an enviable reputation for
excellence in their areas of specialism, and for making cutting edge and highly relevant
contributions to current medical agendas. It is work that has also brought with it an
international profile that is more than evidenced by the nature of the team’s publishing
collaborations, conference invitations and presentations and keynote guest speakers at the
symposia they have hosted.

As already stated, it is difficult to connect a team ethos with performance in some causal
manner. In each of the interviews conducted however, a belief in professionalism was
apparent. This ‘belief’ was about more than seeing professionalism as one more social
object to be studied in an entirely dispassionate way. It was communicated rather as

something that informed the working life of the Centre itself. Perhaps we can conjecture,
that where an authentic commitment to professionalism exists, and where this is also the
focus of the work being done, there also are we likely to find high quality outputs and
genuine insight. In the evaluator’s opinion, based upon the evidence presented in this
report, this has been true of the CEDP.


Chase, S.E. (2005), ‘Narrative inquiry: multiple lenses, approaches, voices’. In Denzin,
N.K. & Lincoln, Y.S. (Ed) The SAGE handbook of qualitative research, 3rd Edition, Sage

Graham, D. (2009), Creating Tomorrow’s Doctors, University of Liverpool.

Kember et al., 1999 D. Kember, A. Jones, A. Loke, J. McKay, K. Sinclair and H. Tse et al.,
Determining the level of reflective thinking from students’ written journals using a
coding scheme based on the work of Mezirow, International Journal of Lifelong
Education 18 (1999), pp. 18–30.

Modernising Medical Careers Working Group (2005), Career Management: An approach
for medical schools, deaneries, royal colleges and trusts, NHS.

O’Brien, M. (2009), The CETL Fellowship Scheme, Evaluation and Service Development
Unit, Centre for Lifelong Learning, University of Liverpool.

O’Brien, M. (2010), The Liverpool School of Medical Education Careers Advisory Service:
Evaluation findings, Evaluation and Service Development Unit, Centre for Lifelong
Learning, University of Liverpool.

University of Liverpool CETL bid to the Higher Education Funding Council (2004).

QAA (2010) Section 8: Career education, information, advice and guidance.

                           Appendix 1: CEDP research outputs*

Peer Reviewed Articles

      Bowhay AR. An investigation into how the European Working Time Directive has
      affected anaesthetic training. BMC Medical Education 2008, 8:41.

      Bowhay AR, Watmough SD. An evaluation of the performance in the UK Royal
      College of Anaesthetists primary examination by UK medical school and gender. BMC
      Medical Education 2009, 9:38.

      Cherry MG, Brown JM, Neal T, Shaw NJ. What features of educational interventions
      lead to competence in aseptic insertion and maintenance of central venous
      catheters in acute care? Medical Teacher (in press).

      Fowell, S., Fewtrell, R. & McLaughlin, P. (2008) Estimating the Minimum Number of
      Judges Required for Test-centred Standard Setting on Written Assessments. Do
      Discussion and Iteration have an Influence? Advances in Health Sciences Education,
      13, 11-24.

      Garner, J. McKendree, J. O’Sullivan, H. Taylor, D. Undergraduate medical student
      attitudes to the peer assessment of professional behaviours in two medical schools.
      Submitted to Education for Primary Care (August 2009).

      Garner, J. O’Sullivan, H. Facebook and the professional behaviours of undergraduate
      medical students. Submitted to Clinical Teacher (August 2009).

      Finn, G. Sawdon, M. Garner, J. ‘You, like, plan how professional you’re going to be’:
      Pre-clinical medical students’ views on professionalism and the context in which it is
      applicable. Submitted to Medical Education (September 2009).

      Fletcher I; Leadbetter P; Curran A; O'Sullivan H. "A pilot study assessing emotional
      intelligence training and communication skills with 3rd year medical students"
      Patient, Education & Counseling 76 (2009) 376-379.

      van Mook Walther N.K.A., Willem S. de Grave, Scheltus J. van Luijk, Helen O'Sullivan,
      Valerie Wass, Lambert W. Schuwirth and Cees P.M. van der Vleuten (2010) “Training and
      learning professionalism in the medical school curriculum: Current considerations”
      European Journal of Internal Medicine (in press).

      van Mook, W., S. van Luijk, H. O’Sullivan, V. Wass, J. Harm Zwaveling, L. Schuwirth, C. van
      der Vleuten (2009). “The concepts of professionalism and professional behaviour:
      Conflicts in both definition and learning outcomes” European Journal of Internal
      Medicine, Volume 20, Issue 4, Pages 85-89.

      van Mook, Walter .N.K.R., Willem S de Grave, Valarie Wass, Helen O’Sullivan, Jan Harm
      Zwaveling, Lambert W Schuwirth and Cees P. M. Van der Vleuten. (2010).
      “Professionalism: Evolution of the concept” European Journal of Internal Medicine (in

      van Mook,W., S. van Luijk, H. O’Sullivan, V. Wass, L. Schuwirth, C. van der Vleuten
      (2009). “General considerations regarding assessment of professional behaviour”
      European Journal of Internal Medicine, Volume 20, Issue 4, Pages 90-95.

      van Mook, W., S. van Luijk, W. de Grave, H. O’Sullivan, V. Wass, L. Schuwirth, C. van
      der Vleuten (2009) “Teaching and learning professional behavior in practice”
      European Journal of Internal Medicine, Volume 20, Issue 5, Pages e105-e111.

      Watmough, S, O’Sullivan H & Taylor D (2009) Graduates from a traditional medical
      curriculum evaluate the effectiveness of their undergraduate curriculum through
      interviews, BMC Medical Education IN PRESS.

      Watmough S, Waddelove C & Jaeger L (2009) First year medical students’
      perceptions of a career in medicine – how can these inform careers support?
      Constructing the Future V: Career Guidance for Changing Contexts. IN PRESS.

      Watmough, S & Waddelove, C. (2009) Association of Graduate Careers Advisory
      Services (AGCAS), www.agcas.org.uk/articles/218-Liverpool-hosts-balloon-debate-
      for-medical-students (accessed 29th May 2009).

      Watmough, S., Ryland I., Taylor, D. (2007) Using questionnaires to determine
      whether medical graduates career choice is determined by undergraduate or
      postgraduate experiences. Medical Teacher, 29 : 830-832.

      Watmough, S. Graduates from a traditional medical curriculum evaluate the
      effectiveness of their medical curriculum through interviews. BMC Medical
      Education 2009, 9:64.

      Brown JM, Watmough S, Cherry MG, Fewtrell R, Graham D, O’Sullivan HM. How
      well are graduates prepared for practice when measured against the latest
      recommendations of the General Medical Council? British Journal of Hospital
      Medicine (in press).

Invited Reviews

      O’Sullivan, H.M, W.N.K.A. van Mook and V Wass (2009) “Integrating Professionalism”
      The Association for Medical Education in Europe Guide.

Other Articles (i.e. non peer-reviewed)

       Garner, J. O’Sullivan, H. Taylor, D. (2009) ‘Medical students – in healthy
       competition?’. The Academy Subject Centre for Medicine, Dentistry and Veterinary
       Medicine Newsletter 01, no. 18, Summer 2009 pp. 5. ISSN 1740-8768.

Invited contributions, keynote speaking and workshops

       Bowhay A. Invited Discussant on Paediatric Anaesthetic Problems. AAGBI
       ASM2009. Liverpool, September 2009.

       Garner, J. O’Sullivan, H. Taylor, D. Constructive Criticism or Popularity Contest: What
       Students Think of Peer Assessing Professionalism. SPARC (Salford Postgraduate
       Annual Research Conference) 09/05/08. Faraday House, University of Salford.

       Garner, J. O’Sullivan, H. Taylor, D. Student Engagement and Peer Appraisal. Staff-
       Student Collaborations: Partnerships within CETLs. 17/06/08. Robbins Conference
       Centre, University of Plymouth.

       Invited to deliver preconference workshop at European Association of
       Communication in Healthcare (EACH). Jantien van Berkel, Claartje ter Hoeven, Joona
       Pauliina Koponen, Giulia Lamiani, Thorsten Langer, Peter Leadbetter, Anne-Kathrin
       Steger, Yuefang Zhou “Learning from EACH other – handling practical questions in
       research”. September 6-9, 2010. Verona, Italy.

       O’Sullivan, H. “Medical professionalism: Can it be taught? Must it be assessed?”
       Innovate. Managing Education Change. University of Southampton School of Medicine.
       February 12th 2009.

       O’Sullivan, H. “The Liverpool Professional” ASME (Association for the Study of Medical
       Education) Conference - Thursday 5th June 2008 - Royal Institute of British Architects, 66
       Portland Place, London, “UK Medical Students & Professional Behaviour”.

       Waddelove, C ‘’ A Case Study from Liverpool – Developing Careers Management
       Skills with undergraduate medical students’’ Centre for Excellence in Preparing for
       Academic Practice and the MEDEV Subject Centre Conference Wednesday 24th
       February 2010 Lincoln EPA Science Centre, Museum Road, Oxford, OX1 3PX.

       Waddelove, C , McNamara F, Bennett C, S Pook, Graduate entry medicine and
       progression of medical students into the foundation and beyond. AGCAS Biennial
       Conference September 2007 University of Birmingham.

Peer reviewed oral presentations

      Cherry MG, Brown JM, Neal T, Shaw NJ. What features of educational interventions
      lead to competence in aseptic insertion and maintenance of central venous
      catheters in acute care? Proceedings of the Annual Scientific Conference of the
      Association for Medical Education in Europe (AMEE) 2009 August 29- September 2;
      Malaga, Spain.

      Fowell SL, Fewtrell R, Mclaughlin PJ. How low can you go? Estimating the minimum
      number of judges required for reliable standard setting of medical-school based
      assessments by application of generalizability theory. Association Study of Medical
      Education (ASME Annual Scientific meeting Newcastle) 11th – 13th July 2005.

      Collier D, Wilkie L, Fewtrell R, Fowell SL, Owens PM, McLaughlin PJ. What standard
      setting method provides a fair and defensible pass mark in clinical skills OSCE within
      the resources available. Association for Medical Education in Europe (AMEE
      Amsterdam) 2005.

      Fewtrell R, O’Sullivan HM, Pilot Study of Undergraduate Medical Students’ Attitudes
      towards Portfolios. Association Study of Medical Education (ASME Annual Scientific
      meeting Keele) 11th – 13th July 2007.

      Fewtrell R, O’Sullivan HM, How does the reflective level of students in the first two
      years of a PBL medical curriculum compare to other health care students?
      Association for Medical Education in Europe (AMEE Malaga) 29th August – 2nd
      September 2009.

      Garner, J. O’Sullivan, H. Taylor, D. Constructive criticism or popularity contest: what
      students think of peer assessing professionalism. AMEE Annual Scientific Meeting.
      01/09/08. Prague Conference Centre, Czech Republic.

      Leadbetter, P; O’Sullivan, H; & Fletcher, I. Annual International Association of
      Medical Education (AMEE) conference. The communication skills of medical students:
      simulated versus real patients. August 29 to September 2, 2009. Malaga, Spain.

      Leadbetter, P; O’Sullivan, H; & Fletcher, I. European Association of Communication
      in Healthcare (EACH); The Communication skills & attachment styles of medical
      students. Julius and Regine Steffens Junior Research Workshop, European
      Association of Communication in Healthcare (EACH). August 24-28, 2009. Vna,

      Leadbetter, P; O’Sullivan, H; & Fletcher, I. Annual scientific meeting for Association
      for the Study of Medical Education (ASME). The communication skills of medical
      students: video analysis, OSCE scores and attachment styles. July 15-17, 2009.
      Edinburgh, Scotland.

Leadbetter, P; O’Sullivan, H; & Fletcher, I. Faculty of Medicine E-Learning Support
Unit session of the Learning Technology User Group (LTUG). The use of video
technology for research. February 25, 2009. The University of Liverpool.

Leadbetter, P; O’Sullivan, H. Annual International Association of Medical Education
(AMEE) conference. An evaluation of a piloted programme of development on the
Emotional Intelligence, well being and communication skills of medical students.
September 1-3, 2008. Prague, Czech Republic.

O’Sullivan, H (presented); Leadbetter, P. Pedagogical research in Higher Education.
A pilot study investigating the Emotional Intelligence & well being of medical
students. June, 16-17. Hope University, Liverpool.

Leadbetter, P; O’Sullivan, H. Centre for Excellence in Developing Professionalism
Symposium- Developing professionalism in Liverpool: Progress and dissemination.
"An evaluation of a piloted intervention on the Emotional Intelligence and
psychological well being of 3rd year medical students" February 6, 2008. The
University of Liverpool.

Fletcher, I (presented); Leadbetter, P; O’Sullivan, H. International Conference on
Communication in Heathcare."An investigation of a piloted programme of
development on the communication skills and emotional intelligence of medical
students" European Association of Communication in Healthcare (EACH). September
2- 5, 2008. Oslo, Norway.

PBL graduates assess their undergraduate medical education 6 years after graduation
through interviews. (Watmough S, Taylor DCM, O’Sullivan HM). AMEE 2008. (Association
for Medical Education in Europe) 30 August to 3 September, Prague, Czech Republic.

“Can Emotional Intelligence be taught? An example from medicine” H.M. O’Sullivan and
P. Leadbetter. 2nd International Pedagogical Research in Higher Education, 16th and
17th June, Liverpool Hope University, Liverpool.

“Developing Strategies for Assessing Professionalism” A Garden & H O’Sullivan, ASME
Golden Jubilee Annual Scientific Meeting 2007 Medicine's Role in Future Healthcare
Provision, 11-13 July 2007: The Chancellor's Building, Keele University, Staffordshire.

Watmough S, Taylor D, O’Sullivan H. Using questionnaires to compare the perceived
competencies of graduates from a traditional and PBL curriculum 6 years after
graduation. Proceedings of the AMEE Annual Scientific Meeting (Malaga) 30th
August – 2nd September 2009.

Watmough S, Taylor D ASME Medical graduates assess the quality of their
undergraduate education 6 years after graduation: a qualitative study Proceedings
of the ASME Annual Scientific Meeting (Keele) 11 – 13th July 2007.

Peer reviewed poster presentations

      Bowhay A. How has the European Working Time Directive (EWTD) affected anaesthetic
      training and quality of life? The trainee’s view. Poster presentation. AMEE 2006. Genoa,
      Italy. September 2006.

      Bowhay A. The Future of Anaesthetic Training for Trainees and Consultants. GAT Annual
      Scientific Meeting 2008. Liverpool, England. July 2008.

      Bowhay A. ACE (Altruism, Cynicism and Empathy). Poster presentation. ASME Annual
      Scientific Meeting 2008. Leicester. September 2008.

      Bowhay A, Watmough S. Gender and country of primary medical qualification does
      affect performance in postgraduate examinations. Poster presentation. AMEE 2009.
      Malaga, Spain. September 2009.

      Fewtrell R, Fowell SL, Chamberlain JM, Griffiths RD, McLaughlin PJ. Investigation of
      reliability of undergraduate final year record of in-training assessment. Association
      Study of Medical Education (ASME Annual Scientific meeting) 11th – 13th July 2005.
      Bond VJ, Owens PM, Fewtrell R. Can the format of a study guide impact on student
      performance? Association for Medical Education in Europe (AMEE) 2005.

      PH Dangerfield, C Waddelove. Introduction of careers and personal development into an
      undergraduate medical school ASME (Association for the Study of Medical Education)
      Conference - Thursday 18th January 2007 - Royal Institute of British Architects, 66
      Portland Place, London, “A Careers Service for Doctors in Training.

      Dimmock PW, Bodger K, Fowell SL, Fewtrell R, Taylor DCM. LOCAS – a new objective
      clinical exam for undergraduate medical students. Association for Medical Education
      in Europe (AMEE Genoa) 14th – 18th September 2006.

      Fowell SL, Fewtrell R, Taylor DCM. Does time of leaving multiple choice-type
      assessments correlate with performance? Association for Medical Education in
      Europe (AMEE Genoa) 2006.

      PH Dangerfield, R Fewtrell, C Waddelove, DCM Taylor, An Online Personal
      Development Planning (PDP) Programme for Undergraduate Medical Students.
      Association Study of Medical Education (ASME Annual Scientific meeting Keele) 11 th
      – 13th July 2007.

      Fowell SL, Fewtrell R, Dangerfield PH, Taylor DCM, Can judges conceptualise the
      borderline student? Exploring the relationship between question ratings and
      question difficulty for the Ebel standard setting method. Association Study of
      Medical Education (ASME Annual Scientific meeting Keele) 11th – 13th July 2007.

Fewtrell R, O’Sullivan HM, Student opinions of that makes a reflective portfolio work
within Liverpool’s medical curricula (AMEE Prague) 30th August – 3rd September

Fewtrell R, O’Sullivan HM, What effect does one year at a PBL medical school have
on students’ critical thinking and reflective skills? (ASME Annual Scientific meeting
Edinburgh) 15th – 17th July 2009.

Dangerfield PH, Duvall P, Fewtrell R, Ralph M & Robinson DL, The Development of
an interface for Personal Development Planning (AMEE Malaga) 29th August – 2nd
September 2009.

Garner, J. O’Sullivan, H. Taylor, D. Constructive criticism or popularity contest: what
students think of peer assessing professionalism. 05/06/08. ASME Meeting – Medical
Students and Professional Behaviour. Royal Institute of British Architects, London.

Garner, J. O’Sullivan, H. Taylor, D. Constructive criticism or popularity contest: what
students think of peer assessing professionalism. 19/09/08. PRISM (Postgraduate
Researchers in Science Medicine). The Graduate School, University of Liverpool.

Garner, J. Williams, E. O’Sullivan, H. Peer, tutor and self review in Problem Based
Learning (PBL). 16/07/09. ASME Annual Scientific Meeting. Royal College of
Surgeons, Edinburgh.

Moneypenny MJ, O’Sullivan H, Guha A. Investigating the beliefs and attitudes of
medical students in relation to professionalism. Proceedings of the Annual Scientific
Conference of the Association for Medical Education in Europe (AMEE) 2009 August
29- September 2; Malaga, Spain.

“Developing professionalism: a multidisciplinary perspective”. (De Condappa O,
O’Sullivan HM, Senior A, Murphy M). AMEE 2008. (Association for Medical Education in
Europe) 30 August to 3 September, Prague, Czech Republic.

“Using social networking technology in developing professionalism” (Robinson D,
O’Sullivan HM). AMEE 2008. (Association for Medical Education in Europe) 30 August to
3 September, Prague, Czech Republic.

“Multisource feedback appraisal: working towards a measure of professionalism” G S
Vince, K Grant, H O’Sullivan, A Garden ASME (Association for the Study of Medical
Education) Conference - Thursday 5th June 2008 - Royal Institute of British Architects, 66
Portland Place, London, “UK Medical Students & Professional Behaviour”.

“Improving the student learning experience using Web 2.0 technologies,” Robinson
D, O’Sullivan H, Proceedings of the Annual Scientific Conference of the Association
for Medical Education in Europe (AMEE) 2009 August 29- September 2; Malaga,

       “Using Social Networking Technology in Developing Professionalism,” Robinson D,
       O’Sullivan, H., Annual scientific meeting for Association for the Study of Medical
       Education (ASME), July 15-17, 2009. Edinburgh, Scotland.

       Waddelove, C Introducing a careers education, information, advice and guidance
       service    for undergraduate medical students. Association Study of Medical
       Education (ASME Annual Scientific meeting Keele) 11th – 13th July 2007.

       Watmough S, Taylor D, O’Sullivan H, Using interviews to gain an insight into the
       factors which drive the career choices of junior doctors Proceedings of the ASME
       Annual Scientific Meeting (Leicester) 10 – 12th September 2008.

       Watmough S, Taylor D, O’Sullivan H, The impact of a Clinical Skills Resource Centre in
       an undergraduate medical curriculum. Proceedings of the ASME Annual Scientific
       Meeting (Edinburgh) 15th – 17th July 2009.

*Names of CEDP staff shown in bold font

               Appendix 2: Average CEDP staffing (April 2005 – March 2010)

Directorship and management

                   Post            Commencement                    Fte months
       0.8 fte. Director           March 2006         48 calendar months
                                                      38.4 fte. months
                                                      38.4 fte. months
‘CEDP months’ since inception (April 2005): 59 months
Directorship and management: average fte. staffing: 0.65


                  Post            Commencement             Fte months
       One full-time              July 2006    44 calendar months
       administrator (two post-                44 fte. months
                                                        44 fte. months
‘CEDP months’ since inception (April 2005): 59 months
Administration: average fte. staffing: 0.75


                  Post              Commencement                  Fte months
       One full-time learning       March 2007        36 calendar months
       technologist                                   36 fte. months
                                                      36 fte. months
‘CEDP months’ since inception (April 2005): 59 months
Developmental: average fte. staffing: 0.6

Student support

                   Post              Commencement                  Fte months
       0.8 fte. careers adviser      April 2006       47 calendar months
                                                      37.6 fte. months
      One 0.2 fte careers            March 2008       24
      advisor                                         4.8 fte. months
                                                      42.4 fte. months
‘CEDP months’ since inception (April 2005): 59 months
Student support: average fte. staffing: 0.7

Clinical research
                    Post                  Commencement                       Fte months
        One full-time clinical           June 2008               21 calendar months
        research fellow                                          21 fte. months
        One 0.5 fte. senior              April 2006              48 calendar months
        clinical specialist                                      23 fte. months
        One honorary 0.2 fte.            June 2009               9 calendar months
        clinical research fellow                                 1.8 fte. months
                                                                 45.8 fte. months
‘CEDP months’ since inception (April 2005): 59 months
Clinical research: average fte. staffing: 0.8 (rounded up from 7.7)

Academic research

                   Post                  Commencement             Fte months
        One full-time research           March 2007   36 calendar months
        fellow                                        36 fte. months
        One full-time research           June 2007    33 calendar months
        fellow                                        33 fte. months
        One full-time research           March 2007   36 calendar months
        fellow                                        36 fte. months
        One full-time research           April 2006   47 calendar months
        fellow                                        47 fte. months
        One full-time research           April 2007   35 calendar months
        fellow                                        35 fte. months
        One full-time research           Aug 2006-7   12 calendar months
        assistant                                     12 fte. months
        One 0.5 fte. student             July 2008    20 calendar months
        research fellow (two                          10 fte. months
                                                                209 fte. months
‘CEDP months’ since inception (April 2005): 59 months
Academic research: average fte. staffing: 3.5

Total staffing

    CEDP commencement                   CEDP calendar months             Total fte. months (all posts)
         April 2005                              59                         7 (rounded down from 7.13)