Course Leader and Departmental/Subject Perspectives on External Examining Martin Oliver*, Jan Smith & Norman Jackson *London Knowledge Lab, Institute of Education email@example.com Introduction This report describes work undertaken in order to gain perspectives on external examining and how this might be supported, looking in particular at the process from the perspectives of course leaders. This study is one of a series being undertaken for the HE Academy (initially, the LTSN Generic Centre) that are exploring how the work of external examiners can be understood and supported. Methodology The commissioning brief for this study was to undertake a series of interviews with course leaders at two universities – one pre-92, one post-92. These interviews were intended to bring in a range of disciplinary perspectives. However, given the timing of the project (falling during June and July, the main exam period) access to suitable individuals was problematic. In addition, the level of resourcing for the project (8 days in total) meant that the initial target of 20 interviews was impractical. These difficulties were ameliorated by Norman Jackson undertaking part of the research work. The final study involved one focus group, providing an illustration of practice at one institution, followed by a series of interviews at the second, which were intended to build upon the findings and questions identified in the focus group. The initial focus group took place at the post-92 institution and involved six programme tutors, an Associate Head of Department and a senior member of the University Quality and Standards Office. The group involved representatives of Life Sciences, Engineering & Information Systems, Electronic Communications and Electrical Engineering, Aerospace, Automotive and Design Engineering and Computer Science. The schedule of questions for the focus group is included in Appendix 1. Notes were taken, were checked with participants and were analysed by identifying themes that emerged from the discussion. The second phase of work involved seven interviews at the pre-92 institution. These included representatives of Anthropology, Human Communication Science, Electronic and Electrical Engineering, Education and Dentistry. The interviews were semi-structured, using a schedule derived from Appendix 1, refined in light of the focus group discussions (Appendix 2). All interviews were fully transcribed. A constant comparative method of analysis was used to identify themes from the data. Findings: focus group The role of the external examiner Four roles were discussed by the group: 1. A benchmarking agent who enables departmental standards to be compared with standards and practices in institutions that offer comparable (not identical) programmes. 2. An auditor of procedures and processes who judges the rigour of standards setting processes, looking for compliance and offering constructive challenge to improve practice. (The importance of transparency in responding to such challenges was noted.) 3. A „badly paid consultant‟, who: a. Confirms that teaching and assessment practices are broadly in line with what is happening elsewhere; b. Provides support and enhances confidence by highlighting departmental strengths. „Gives us confidence that we are doing the right thing‟; c. Helps to identify and think through particular problems; and d. May contribute new resources. 4. An aid to departmental decision-making, providing the perspective of an objective outsider in order to advise, arbitrate and help overcome resistance. These four roles map easily onto the three roles identified by Smith & Oliver (2004) in their parallel study (assessing outcomes, assessing processes and external consultant, which here covers both 3 and 4). Within the focus group, the idea of a „critical friend‟ seemed to underlie the conception of the external examiner, suggesting that roles 3 and 4 are the most valuable. There was a recognition that externals were often (though not always) from departments that were in competition and an acknowledgement that what was seen in the department would be taken and shared elsewhere; however, this did not seem to inhibit the role of critical friend. Thus implicit in this account is the belief that external examiners are agents for the diffusion of practice around the system. Reflecting this, programme tutors felt that informal opportunities for discussion with external examiners were what „really mattered‟ during the visits. Such discussions often took place during informal pre-meetings or over lunch. Qualities sought in external examiners Course leaders sought several qualities in external examiners: Subject specialism (without which all that can be judged are regulations and procedures). Empathy with the institutional context (students and programmes). Understanding and empathy of the pedagogic stance adopted. A non-competitor, where possible. The ability to listen and suggest, based on their experience, rather than a propensity to lead with personal opinions. Recruitment was typically led by the Associate Head of Department, after canvassing the views of key individuals such as programmed tutors. They would then invite the selected individual to take up the role and, if they agreed, would put them forward through faculty approval mechanisms. If necessary, online searches of selected departmental websites might be used to identify suitable individuals. (This was seen as a poor alternative due to the lack of personal knowledge.) Another way of identifying potential external examiners was through their prior involvement in programme validation. Experience of being a programme tutor was felt to provide a good foundation for becoming an external examiner, but the members of the focus group generally felt that these roles were in conflict because both were busy at the same time of year. The induction of external examiners Two day-long institution-wide induction events for newly appointed external examiners were provided each year, for which expenses were paid. Each consists of a general introduction to institutional processes in the morning, followed by a meeting with course teams and departmental staff in the afternoon. The format of the departmental section of the day varies, but typical topics include: Departmental outline Overview of programmes and how they are run Student numbers and backgrounds Scheduling of assessment, exam boards Specific EE tasks - how the role will be operationalised Admin procedures – key admin staff/contact details Previous EE reports Informal sharing of experiences – typical problems and how they were resolved In addition, programme tutors are responsible for ensuring that the external examiners get the information they need to fulfil their role, including programme/module documents, assessment timetable, exam papers for moderation, coursework tasks, sample coursework, exam scripts and projects. In some programmes the programme tutor also arranges visits to look at student presentations/posters during the year. Key issues associated with external examining The focus group identified two key issues: Finding good external examiners The exclusion of programme tutors and others who are already over-committed during June/July. More generally the way the role is currently configured – would we do it differently if we were inventing it again? Suggestions were also made of ways in which the Higher Education Academy might be able to support external examiners: Adapting the Edexcel model to prepare people for the role as an external examiner Opportunities for staff development, e.g. in assessment practice Databases and brokerage services to help programme teams find external examiners Covering staff expenses to travel to other departments to develop their knowledge of assessment outside their own environment Some sort of mentorship and coaching scheme Findings: interviews The changing role of external examiners As with the focus groups, the three roles described in Smith & Oliver (1994) – examining processes, examining outputs and external consulting – were also consistently present here. He will look at the exam marks at the end of the year and he will make sure that the marking is fair and so on, and he will do this by looking at some assignments and some project reports. […] He gives us feedback to assure that the process of examining is fair. […] The previous external examiner [….] was very experienced, sometimes he will come up with comments that we ought to be teaching something else, comments on the technical content of the modules. Some external examiners do this. (Interview 2) There was also a shared perception that the role of external examiners has changed, losing its emphasis on assessing outcomes and gaining an emphasis on judging processes. The role of the external examiners changed some years ago […] External Examiners are now expected to be moderators, they are not expected to vet every single paper and the way that their role operates is that primarily and centrally it‟s to do with benchmarking our examination procedures against national standards if you like, so cross-checking them against people who have experience not just of their own institution but of several other institutions. (Interview 1) This situation was only seen as getting yet more extreme. As I understand it […] the scope for exam boards and indeed external examiners to have any influence on the degree awarding process is going to change radically, it‟s going to become a much more automated process and I think at that stage I just don‟t know what the function of the exam board will be other than to look at cases of illness or something that has possibly affected the student Some concern and regret was expressed over this development. I did think that it was a real shame that we didn‟t have each script being read by three people but for practical reasons that‟s no longer how it works. […] Well I think it is a loss but I don‟t think it really is a major loss. (Interview 1) It was also suggested that many external examiners also find it hard to give up this role. I think some of our external examiners have found difficult the transition from the old mode of externalling to the new mode of moderating and in particular, because they find that difficult, what they tend to do is when they‟re sent benchmark scripts they can‟t resist remarking them. Now they re-mark them and come up with something that‟s very very similar, so they‟re not querying say a mid 2.1, but they‟re then puzzled if that altered mark isn‟t incorporated into the overall marksheet that‟s reported. Now we can‟t incorporate those tinkering at the edges because if you were to do that you‟d have to do it for every script. (Interview 1) Such moderation may include some sampling of scripts, typically illustrating grade boundaries and including cases that caused internal examiners concern. Linked to the changing role has been the growing emphasis on auditing processes, leading to a far greater consideration of possible sources of variability in marking. In relation to the role of assessing processes, one participant introduced and swiftly dismissed subject benchmarks. You would have to make comment about benchmarking, to say about how this course compares to other courses in comparable UK universities and basically we bullshit. If you understand what is the course, what it wants to achieve, then benchmarking is not that important. […] As an external you have to understand the ethos of the course then benchmarking is a trivial issue I think. (Anonymous) The selection of external examiners In contrast to the experiences of the focus group participants, the interviewees (not their managers) and their course teams proposed suitable external examiners and seek to establish consensus on who to approach. Several important qualities were identified when choosing external examiners. Amongst these was a supportive and positive style. In terms of personality you want somebody who‟s reasonably relaxed, you don‟t want a complete anal retentive because there‟s a degree of pragmatism and you know we‟ve got enough inside the department anyway. You want somebody who‟s going to help you progress and not get in the way all the time. (Interview 4) Seniority was not always a priority: The people whom we‟ve had nominated as externals range from very senior people who are very experienced in the sense that they‟ve been around for ever and have probably done it for lots of units, through to actually much younger people who are just actually felt to be alive to many different areas of the subject, so a safe pair of hands can really embrace a very wide range of different kinds of people. (Interview 1) Indeed, given the lack of reward for external examining, it was sometimes felt to be better to avoid senior people. It‟s also quite useful to not go for someone who‟s completely at the top of their career, and rather go for somebody who is building up their CV because they‟re more keen to do it. (Interview 3) This was a potential problem, however, since less well-established individuals were not well enough known within the discipline for the course team to be aware of them and thus approach them. Other qualities that were identified included an interest in education, experience as a course leader (preferably on a similar programme), experience of attending exam boards and expertise in the subject (or a closely related area). Appointing someone known to the team was also seen as useful because this can ensure that there is a productive working relationship between the examiner and the course team. Research profile was linked to appointments but was not necessarily the reason for them, although some participants felt that this would be important where projects formed part of the course. Instead, research networks tended to be the way in which suitable individuals were identified. Academics do not know other academics at other universities because of their educational activities, they would know them because of their research activities or because of past history. You wouldn‟t know someone because he teaches something; we go to conference to talk about research and to network. (Interview 2) This was also important because it was suggested that the ability to compare standards depending upon an individuals‟ engagement with disciplinary networks. He needs to be very much part of the community because of the background processes is that you‟ve got this co-normalisation. (Interview 4) However, although an individual‟s research profile might not be crucial when seeking appointments, it was suggested that the status of their department could be a criterion. You would either go to departments that are better or comparable. […] You don‟t have someone from an ex-poly. Very rarely. You have to argue a strong case and we had a case where we had to look for someone, it was not something the university would want. (Interview 2) One explanation offered for this was that this was seen as a useful proxy when seeking to recruit good examiners. It‟s correlated, not causal. You don‟t go, oh, that‟s 5*, we can only ask 5*, but the requirements I mentioned before would naturally mean that that department‟s working at quite a high level […]; that‟s an indicator of the pool of the best that you‟d be able to touch onto. (Interview 4) Where external examiners were non-academic, they were expected to be „high level practitioners‟ with considerable experience and responsibility. Such individuals were seen as particularly valuable for vocationally-oriented courses where “this material doesn‟t exist in books” and thus the course team comes to rely on the input of non-academic external examiners. Comments within the first three interviews raised a question about whether suggestions from the course team were ever rejected by the institution. This seemed to be a concern, rather than an experience, of most participants – in other words, they policed their own selection process according to centrally-specified guidelines rather than the institution taking a firm stance on the matter. Importantly, it seems that opinions about potential external examiners are shared amongst disciplinary networks and that these influence choices. I suppose the reason that I use the term a safe pair of hands is that just occasionally a name will come up and everybody groans and says no, I‟ve heard this person in seminars or whatever and I just think that they have a very narrow point of view or a very erratic response or for whatever reason they may be considered to be colleagues of great heuristic value but not a safe pair of hands. (Interview 1) Opinion differed on whether or not recruitment of external examiners was a problem. This might suggest that there are disciplinary differences at work. Where recruitment was a problem, it was suggested that one cause was the time pressure of this additional responsibility at an already busy time of year. There was suggestion that the processes of appointment was not always as open as it should be. How this guy was recruited, I‟ll tell you when you switch the machine off. (Anonymised) It was felt that, at least for broad undergraduate programmes, no individual would have the required breadth of expertise to examine on their own – as a consequence, teams of examiners were common. Inducting and training external examiners Several approaches were identified to the induction of external examiners, including: The initial informal, personal approach and discussions before formal appointment. Information on the role, particularly addressing its changing nature. Information provided to students, for context. Invitations to course meetings throughout the year. (It was noted that it was not common for externals to attend these.) Meetings at various levels – with institutional quality assurance officers, with the course leader and with module co-ordinators. There was strong opposition to the idea of training courses for external examiners. Developing confidence within the department A recurrent theme was that externals examiners‟ comments can bolster the confidence of the department, either through praise or by providing a feeling of validation and legitimacy. The whole process is helpful in that it gives you a degree of legitimacy to feel that your standards are acknowledged and somehow ranked with respect to standards elsewhere. [… We have developed] more new and different dimensions to the discipline and there‟s always a risk when you do that that people are going to feel that somehow this is not traditional and so it‟s always very rewarding when the external examiner really commends a particular part of the programme as being not just innovative but new and valuable approach to a discipline. (Interview 1) They can also boost confidence within a department, for example by praising individual course units so that individuals‟ efforts are appreciated. Developing practice within a department Several examples were identified where the external examiner introduced new forms of practice into the department. If a problem has arisen, they‟ll say, well yes, this is how I‟ve seen somebody do that, you don‟t have to do it that way, but it‟ll put ideas into the pot. […] It‟s how the community normalises itself, by sharing ideas and that‟s one of the most important things. (Interview 4) Specific examples included: Improving the clarity of papers or rubrics through changes to formatting. Introducing Multiple Choice Questions. Requiring model answers to be developed. Developing structured marking schemes. Advising on and approving changes to the course, including some input on matters of syllabus. There was inconsistency – even within single interviews – over the power of the external examiner to hold course teams to account over proposed changes. Some participants explained how the formal process of report forces them to attend to all the changes demanded; some, however, made it clear that although they welcomed advice (both informally as suggestions and formally through the submitted report), they felt entitled to reject it so long as they made their reasons for doing so clear. It was felt that such recommendations could be useful as leverage to advance changes that were being resisted by individuals in the course team. It was also suggested that the length of time between the exam board and the formal report arriving meant delays in making changes to the programme; electronic pro-formas were suggested as one way of speeding up this process. Problems with external examiners It was reportedly rare for there to be problems with external examiners, with some participants unable to identify any throughout their entire career. [Interviewer: Have you ever had any problems with external examiners?] I haven‟t, no, not in my experience. I‟m not aware of a department that has. (Interview 2) My experience has been that it has always been constructive; I don‟t think we‟ve ever had a case where it‟s not been constructive. (Interview 4) Several participants did manage to identify an incident that had caused some difficulty. These included: Externals who “disappear without telling you” (Interview 3). External examiners for cross-departmental programmes seeking to lower marks when an able cohort all performed „above average‟ because they were in the upper quartiles of each separate department‟s students. In each case, the participant described how these problems had been resolved. Such resolution sometimes involved a „gentlemen‟s agreement‟ to recognise differences of opinion. One respondent made it clear that they would not invite an external examiner back – whether or not they were at the end of their normal three-year term – if their work was sloppy or unprofessional. Motivation to become an external examiner Several reasons for becoming an external examiner were identified: Altruism, including a sense of duty to one‟s discipline. I know of several other people […] who are if you like senior statesmen who see it as a responsibility to the discipline and who manage to act as external examiners at least for a couple of institutions maybe not more than two and who do it in a meticulous and always useful way. (Interview 1) Networking It‟s also an opportunity for a little political maneuvering, so we‟ll pick somebody that we want to engender good relations with […] I think that‟s important too, part of the community building exercise. (Interview 4) It was made clear that external examiners are not motivated by the fees involved, which are very low given the amount of work involved. It was suggested that this situation would be helped if there was greater recognition for the role. The only thing that could be useful is if visiting examiners [….] have a higher profile somehow […] so that people really want to do it and one of the ways to attract people is (a) to give them more money and also to give them some sort of accolade, so maybe the ILTHE could give them some little tick. (Interview 3) References Smith, H. & Oliver, M. (2004) An Analysis of Personal Accounts of External Examining: Final Report for the Working Group. Unpublished project report, UCL. Appendix 1: Example enquiry themes to be developed by the researchers As course leaders: 1) What qualities, experiences and capacities do you feel makes a good external examiner? 2) How do you see and experience the external examining process? 3) How do you find and persuade people to become your external examiner? 4) What do you do to help your external examiners fulfill their role? Professional role 5) Do you see external examining as an important and valued part of an academic career profile? 6) What are the factors that stop people from becoming external examiners? 7) What sorts of professional development should be undertaken by people wishing to become external examiners in contemporary higher education? Improving what we do 8) What do you see are the key issues associated with external examining? 9) How can we improve the external support we give to external examiners? eg support given by the LTSN subject centres and professional bodies Imagining a different future 10) Are we doing the right things? Are there alternative ways of configuring the function of external examining to achieve the goals we want it to achieve? Appendix 2: Interview schedule 1. What do participants think the EE's role is? 2. How do they recruit to that role? (What are the qualities for which they approach people?) 3. How are EEs inducted into their course/department/institution? 4. What do EEs seem to find difficult (and how this can be addressed)? 5. Have they ever have problems with EEs, and how were these resolved? 6. What do EEs do that is particularly helpful (and whether this is the same as what's on the "job description")? In addition, the issue of approving selected External Examiners was raised during the series of interviews and a question about this was added.
Pages to are hidden for
"Course programme leader perspectives on external examining"Please download to view full document