Course programme leader perspectives on external examining by fionan

VIEWS: 10 PAGES: 11

									Course Leader and Departmental/Subject Perspectives on External Examining

Martin Oliver*, Jan Smith & Norman Jackson

*London Knowledge Lab, Institute of Education
m.oliver@ioe.ac.uk

Introduction

This report describes work undertaken in order to gain perspectives on external examining and
how this might be supported, looking in particular at the process from the perspectives of course
leaders. This study is one of a series being undertaken for the HE Academy (initially, the LTSN
Generic Centre) that are exploring how the work of external examiners can be understood and
supported.

Methodology

The commissioning brief for this study was to undertake a series of interviews with course leaders
at two universities – one pre-92, one post-92. These interviews were intended to bring in a range
of disciplinary perspectives. However, given the timing of the project (falling during June and July,
the main exam period) access to suitable individuals was problematic. In addition, the level of
resourcing for the project (8 days in total) meant that the initial target of 20 interviews was
impractical. These difficulties were ameliorated by Norman Jackson undertaking part of the
research work.

The final study involved one focus group, providing an illustration of practice at one institution,
followed by a series of interviews at the second, which were intended to build upon the findings
and questions identified in the focus group.

The initial focus group took place at the post-92 institution and involved six programme tutors, an
Associate Head of Department and a senior member of the University Quality and Standards
Office. The group involved representatives of Life Sciences, Engineering & Information Systems,
Electronic Communications and Electrical Engineering, Aerospace, Automotive and Design
Engineering and Computer Science. The schedule of questions for the focus group is included in
Appendix 1. Notes were taken, were checked with participants and were analysed by identifying
themes that emerged from the discussion.

The second phase of work involved seven interviews at the pre-92 institution. These included
representatives of Anthropology, Human Communication Science, Electronic and Electrical
Engineering, Education and Dentistry. The interviews were semi-structured, using a schedule
derived from Appendix 1, refined in light of the focus group discussions (Appendix 2). All
interviews were fully transcribed. A constant comparative method of analysis was used to identify
themes from the data.

Findings: focus group

The role of the external examiner

Four roles were discussed by the group:

1.    A benchmarking agent who enables departmental standards to be compared with
      standards and practices in institutions that offer comparable (not identical) programmes.
2.       An auditor of procedures and processes who judges the rigour of standards setting
         processes, looking for compliance and offering constructive challenge to improve practice.
         (The importance of transparency in responding to such challenges was noted.)

3.       A „badly paid consultant‟, who:

              a. Confirms that teaching and assessment practices are broadly in line with what is
                 happening elsewhere;

              b. Provides support and enhances confidence by highlighting departmental
                 strengths. „Gives us confidence that we are doing the right thing‟;

              c.   Helps to identify and think through particular problems; and

              d. May contribute new resources.

4.       An aid to departmental decision-making, providing the perspective of an objective outsider
         in order to advise, arbitrate and help overcome resistance.

These four roles map easily onto the three roles identified by Smith & Oliver (2004) in their
parallel study (assessing outcomes, assessing processes and external consultant, which here
covers both 3 and 4).

Within the focus group, the idea of a „critical friend‟ seemed to underlie the conception of the
external examiner, suggesting that roles 3 and 4 are the most valuable. There was a recognition
that externals were often (though not always) from departments that were in competition and an
acknowledgement that what was seen in the department would be taken and shared elsewhere;
however, this did not seem to inhibit the role of critical friend. Thus implicit in this account is the
belief that external examiners are agents for the diffusion of practice around the system.

Reflecting this, programme tutors felt that informal opportunities for discussion with external
examiners were what „really mattered‟ during the visits. Such discussions often took place during
informal pre-meetings or over lunch.

Qualities sought in external examiners

Course leaders sought several qualities in external examiners:

         Subject specialism (without which all that can be judged are regulations and procedures).

         Empathy with the institutional context (students and programmes).

         Understanding and empathy of the pedagogic stance adopted.

         A non-competitor, where possible.

         The ability to listen and suggest, based on their experience, rather than a propensity to
          lead with personal opinions.

Recruitment was typically led by the Associate Head of Department, after canvassing the views of
key individuals such as programmed tutors. They would then invite the selected individual to take
up the role and, if they agreed, would put them forward through faculty approval mechanisms. If
necessary, online searches of selected departmental websites might be used to identify suitable
individuals. (This was seen as a poor alternative due to the lack of personal knowledge.) Another
way of identifying potential external examiners was through their prior involvement in programme
validation.
Experience of being a programme tutor was felt to provide a good foundation for becoming an
external examiner, but the members of the focus group generally felt that these roles were in
conflict because both were busy at the same time of year.

The induction of external examiners

Two day-long institution-wide induction events for newly appointed external examiners were
provided each year, for which expenses were paid. Each consists of a general introduction to
institutional processes in the morning, followed by a meeting with course teams and departmental
staff in the afternoon. The format of the departmental section of the day varies, but typical topics
include:

       Departmental outline

       Overview of programmes and how they are run

       Student numbers and backgrounds

       Scheduling of assessment, exam boards

       Specific EE tasks - how the role will be operationalised

       Admin procedures – key admin staff/contact details

       Previous EE reports

       Informal sharing of experiences – typical problems and how they were resolved

In addition, programme tutors are responsible for ensuring that the external examiners get the
information they need to fulfil their role, including programme/module documents, assessment
timetable, exam papers for moderation, coursework tasks, sample coursework, exam scripts and
projects. In some programmes the programme tutor also arranges visits to look at student
presentations/posters during the year.

Key issues associated with external examining

The focus group identified two key issues:

       Finding good external examiners

       The exclusion of programme tutors and others who are already over-committed during
        June/July.

       More generally the way the role is currently configured – would we do it differently if we
        were inventing it again?

Suggestions were also made of ways in which the Higher Education Academy might be able to
support external examiners:

       Adapting the Edexcel model to prepare people for the role as an external examiner

       Opportunities for staff development, e.g. in assessment practice

       Databases and brokerage services to help programme teams find external examiners
       Covering staff expenses to travel to other departments to develop their knowledge of
        assessment outside their own environment

       Some sort of mentorship and coaching scheme

Findings: interviews

The changing role of external examiners

As with the focus groups, the three roles described in Smith & Oliver (1994) – examining
processes, examining outputs and external consulting – were also consistently present here.

     He will look at the exam marks at the end of the year and he will make sure that the marking
     is fair and so on, and he will do this by looking at some assignments and some project
     reports. […] He gives us feedback to assure that the process of examining is fair. […] The
     previous external examiner [….] was very experienced, sometimes he will come up with
     comments that we ought to be teaching something else, comments on the technical content
     of the modules. Some external examiners do this. (Interview 2)

There was also a shared perception that the role of external examiners has changed, losing its
emphasis on assessing outcomes and gaining an emphasis on judging processes.

     The role of the external examiners changed some years ago […] External Examiners are
     now expected to be moderators, they are not expected to vet every single paper and the way
     that their role operates is that primarily and centrally it‟s to do with benchmarking our
     examination procedures against national standards if you like, so cross-checking them
     against people who have experience not just of their own institution but of several other
     institutions. (Interview 1)

This situation was only seen as getting yet more extreme.

     As I understand it […] the scope for exam boards and indeed external examiners to have
     any influence on the degree awarding process is going to change radically, it‟s going to
     become a much more automated process and I think at that stage I just don‟t know what the
     function of the exam board will be other than to look at cases of illness or something that has
     possibly affected the student

Some concern and regret was expressed over this development.

     I did think that it was a real shame that we didn‟t have each script being read by three people
     but for practical reasons that‟s no longer how it works. […] Well I think it is a loss but I don‟t
     think it really is a major loss. (Interview 1)

It was also suggested that many external examiners also find it hard to give up this role.

     I think some of our external examiners have found difficult the transition from the old mode of
     externalling to the new mode of moderating and in particular, because they find that difficult,
     what they tend to do is when they‟re sent benchmark scripts they can‟t resist remarking
     them. Now they re-mark them and come up with something that‟s very very similar, so
     they‟re not querying say a mid 2.1, but they‟re then puzzled if that altered mark isn‟t
     incorporated into the overall marksheet that‟s reported. Now we can‟t incorporate those
     tinkering at the edges because if you were to do that you‟d have to do it for every script.
     (Interview 1)

Such moderation may include some sampling of scripts, typically illustrating grade boundaries
and including cases that caused internal examiners concern.
Linked to the changing role has been the growing emphasis on auditing processes, leading to a
far greater consideration of possible sources of variability in marking.

In relation to the role of assessing processes, one participant introduced and swiftly dismissed
subject benchmarks.

     You would have to make comment about benchmarking, to say about how this course
     compares to other courses in comparable UK universities and basically we bullshit. If you
     understand what is the course, what it wants to achieve, then benchmarking is not that
     important. […] As an external you have to understand the ethos of the course then
     benchmarking is a trivial issue I think. (Anonymous)

The selection of external examiners

In contrast to the experiences of the focus group participants, the interviewees (not their
managers) and their course teams proposed suitable external examiners and seek to establish
consensus on who to approach.

Several important qualities were identified when choosing external examiners. Amongst these
was a supportive and positive style.

     In terms of personality you want somebody who‟s reasonably relaxed, you don‟t want a
     complete anal retentive because there‟s a degree of pragmatism and you know we‟ve got
     enough inside the department anyway. You want somebody who‟s going to help you
     progress and not get in the way all the time. (Interview 4)

Seniority was not always a priority:

     The people whom we‟ve had nominated as externals range from very senior people who are
     very experienced in the sense that they‟ve been around for ever and have probably done it
     for lots of units, through to actually much younger people who are just actually felt to be alive
     to many different areas of the subject, so a safe pair of hands can really embrace a very
     wide range of different kinds of people. (Interview 1)

Indeed, given the lack of reward for external examining, it was sometimes felt to be better to
avoid senior people.

     It‟s also quite useful to not go for someone who‟s completely at the top of their career, and
     rather go for somebody who is building up their CV because they‟re more keen to do it.
     (Interview 3)

This was a potential problem, however, since less well-established individuals were not well
enough known within the discipline for the course team to be aware of them and thus approach
them.

Other qualities that were identified included an interest in education, experience as a course
leader (preferably on a similar programme), experience of attending exam boards and expertise
in the subject (or a closely related area). Appointing someone known to the team was also seen
as useful because this can ensure that there is a productive working relationship between the
examiner and the course team.

Research profile was linked to appointments but was not necessarily the reason for them,
although some participants felt that this would be important where projects formed part of the
course. Instead, research networks tended to be the way in which suitable individuals were
identified.
    Academics do not know other academics at other universities because of their educational
    activities, they would know them because of their research activities or because of past
    history. You wouldn‟t know someone because he teaches something; we go to conference to
    talk about research and to network. (Interview 2)

This was also important because it was suggested that the ability to compare standards
depending upon an individuals‟ engagement with disciplinary networks.

    He needs to be very much part of the community because of the background processes is
    that you‟ve got this co-normalisation. (Interview 4)

However, although an individual‟s research profile might not be crucial when seeking
appointments, it was suggested that the status of their department could be a criterion.

    You would either go to departments that are better or comparable. […] You don‟t have
    someone from an ex-poly. Very rarely. You have to argue a strong case and we had a case
    where we had to look for someone, it was not something the university would want.
    (Interview 2)

One explanation offered for this was that this was seen as a useful proxy when seeking to recruit
good examiners.

    It‟s correlated, not causal. You don‟t go, oh, that‟s 5*, we can only ask 5*, but the
    requirements I mentioned before would naturally mean that that department‟s working at
    quite a high level […]; that‟s an indicator of the pool of the best that you‟d be able to touch
    onto. (Interview 4)

Where external examiners were non-academic, they were expected to be „high level practitioners‟
with considerable experience and responsibility. Such individuals were seen as particularly
valuable for vocationally-oriented courses where “this material doesn‟t exist in books” and thus
the course team comes to rely on the input of non-academic external examiners.

Comments within the first three interviews raised a question about whether suggestions from the
course team were ever rejected by the institution. This seemed to be a concern, rather than an
experience, of most participants – in other words, they policed their own selection process
according to centrally-specified guidelines rather than the institution taking a firm stance on the
matter.

Importantly, it seems that opinions about potential external examiners are shared amongst
disciplinary networks and that these influence choices.

    I suppose the reason that I use the term a safe pair of hands is that just occasionally a name
    will come up and everybody groans and says no, I‟ve heard this person in seminars or
    whatever and I just think that they have a very narrow point of view or a very erratic
    response or for whatever reason they may be considered to be colleagues of great heuristic
    value but not a safe pair of hands. (Interview 1)

Opinion differed on whether or not recruitment of external examiners was a problem. This might
suggest that there are disciplinary differences at work. Where recruitment was a problem, it was
suggested that one cause was the time pressure of this additional responsibility at an already
busy time of year. There was suggestion that the processes of appointment was not always as
open as it should be.

    How this guy was recruited, I‟ll tell you when you switch the machine off. (Anonymised)
It was felt that, at least for broad undergraduate programmes, no individual would have the
required breadth of expertise to examine on their own – as a consequence, teams of examiners
were common.

Inducting and training external examiners

Several approaches were identified to the induction of external examiners, including:

       The initial informal, personal approach and discussions before formal appointment.

       Information on the role, particularly addressing its changing nature.

       Information provided to students, for context.

       Invitations to course meetings throughout the year. (It was noted that it was not common
        for externals to attend these.)

       Meetings at various levels – with institutional quality assurance officers, with the course
        leader and with module co-ordinators.

There was strong opposition to the idea of training courses for external examiners.

Developing confidence within the department

A recurrent theme was that externals examiners‟ comments can bolster the confidence of the
department, either through praise or by providing a feeling of validation and legitimacy.

    The whole process is helpful in that it gives you a degree of legitimacy to feel that your
    standards are acknowledged and somehow ranked with respect to standards elsewhere. […
    We have developed] more new and different dimensions to the discipline and there‟s always
    a risk when you do that that people are going to feel that somehow this is not traditional and
    so it‟s always very rewarding when the external examiner really commends a particular part
    of the programme as being not just innovative but new and valuable approach to a discipline.
    (Interview 1)

They can also boost confidence within a department, for example by praising individual course
units so that individuals‟ efforts are appreciated.

Developing practice within a department

Several examples were identified where the external examiner introduced new forms of practice
into the department.

    If a problem has arisen, they‟ll say, well yes, this is how I‟ve seen somebody do that, you
    don‟t have to do it that way, but it‟ll put ideas into the pot. […] It‟s how the community
    normalises itself, by sharing ideas and that‟s one of the most important things. (Interview 4)

Specific examples included:

       Improving the clarity of papers or rubrics through changes to formatting.

       Introducing Multiple Choice Questions.

       Requiring model answers to be developed.
       Developing structured marking schemes.

       Advising on and approving changes to the course, including some input on matters of
        syllabus.

There was inconsistency – even within single interviews – over the power of the external
examiner to hold course teams to account over proposed changes. Some participants explained
how the formal process of report forces them to attend to all the changes demanded; some,
however, made it clear that although they welcomed advice (both informally as suggestions and
formally through the submitted report), they felt entitled to reject it so long as they made their
reasons for doing so clear. It was felt that such recommendations could be useful as leverage to
advance changes that were being resisted by individuals in the course team.

It was also suggested that the length of time between the exam board and the formal report
arriving meant delays in making changes to the programme; electronic pro-formas were
suggested as one way of speeding up this process.

Problems with external examiners

It was reportedly rare for there to be problems with external examiners, with some participants
unable to identify any throughout their entire career.

    [Interviewer: Have you ever had any problems with external examiners?] I haven‟t, no, not in
    my experience. I‟m not aware of a department that has. (Interview 2)

    My experience has been that it has always been constructive; I don‟t think we‟ve ever had a
    case where it‟s not been constructive. (Interview 4)

Several participants did manage to identify an incident that had caused some difficulty. These
included:

       Externals who “disappear without telling you” (Interview 3).

       External examiners for cross-departmental programmes seeking to lower marks when an
        able cohort all performed „above average‟ because they were in the upper quartiles of
        each separate department‟s students.

In each case, the participant described how these problems had been resolved. Such resolution
sometimes involved a „gentlemen‟s agreement‟ to recognise differences of opinion. One
respondent made it clear that they would not invite an external examiner back – whether or not
they were at the end of their normal three-year term – if their work was sloppy or unprofessional.

Motivation to become an external examiner

Several reasons for becoming an external examiner were identified:

       Altruism, including a sense of duty to one‟s discipline.

    I know of several other people […] who are if you like senior statesmen who see it as a
    responsibility to the discipline and who manage to act as external examiners at least for a
    couple of institutions maybe not more than two and who do it in a meticulous and always
    useful way. (Interview 1)

       Networking
     It‟s also an opportunity for a little political maneuvering, so we‟ll pick somebody that we want
     to engender good relations with […] I think that‟s important too, part of the community
     building exercise. (Interview 4)

It was made clear that external examiners are not motivated by the fees involved, which are very
low given the amount of work involved.

It was suggested that this situation would be helped if there was greater recognition for the role.

     The only thing that could be useful is if visiting examiners [….] have a higher profile
     somehow […] so that people really want to do it and one of the ways to attract people is (a)
     to give them more money and also to give them some sort of accolade, so maybe the ILTHE
     could give them some little tick. (Interview 3)
References

Smith, H. & Oliver, M. (2004) An Analysis of Personal Accounts of External Examining: Final
Report for the Working Group. Unpublished project report, UCL.
Appendix 1: Example enquiry themes to be developed by the researchers

As course leaders:

1) What qualities, experiences and capacities do you feel makes a good external examiner?

2) How do you see and experience the external examining process?

3) How do you find and persuade people to become your external examiner?

4) What do you do to help your external examiners fulfill their role?

Professional role

5) Do you see external examining as an important and valued part of an academic career profile?

6) What are the factors that stop people from becoming external examiners?

7) What sorts of professional development should be undertaken by people wishing to become
external examiners in contemporary higher education?

Improving what we do

8) What do you see are the key issues associated with external examining?

9) How can we improve the external support we give to external examiners? eg support given by
the LTSN subject centres and professional bodies

Imagining a different future

10) Are we doing the right things? Are there alternative ways of configuring the function of
external examining to achieve the goals we want it to achieve?

Appendix 2: Interview schedule

1. What do participants think the EE's role is?

2. How do they recruit to that role? (What are the qualities for which they approach people?)

3. How are EEs inducted into their course/department/institution?

4. What do EEs seem to find difficult (and how this can be addressed)?

5. Have they ever have problems with EEs, and how were these resolved?

6. What do EEs do that is particularly helpful (and whether this is the same as what's on the "job
description")?

In addition, the issue of approving selected External Examiners was raised during the series of
interviews and a question about this was added.

								
To top