Summary of a review of literature on the impact

Document Sample
Summary of a review of literature on the impact Powered By Docstoc
					Summary of a review of literature on the impact of working context
and support on the postgraduate research student learning


This summary describes the findings from the first systematic review of existing research on
the learning experience of doctoral students in the UK.

It presents information on what we know about students views of their doctorate, and
specifically how their working context and the support they (do or don’t) receive impacts on
their progress and their wish to follow an academic career or not.

It also looks in detail at how those who experience a viva voce feel about their examination,
and whether their viva made them keen to do more research – or to leave academia.

It found clear evidence of the need for more and better research on how postgraduate
researchers themselves experience doing a doctorate, and the need for a stronger student
voice to be added as a key stakeholder alongside government, the Research and Funding
Councils, and deans and supervisors in universities.

Who is this summary for?
This summary will be of interest to: senior staff involved in the development of policy related
to doctoral research degree programmes, e.g. PVCs, Heads of graduate schools, faculties
and departments; staff working to support doctoral researchers, for example supervisors,
skills development trainers and co-ordinators, and careers advisors.

Why was the review undertaken?
UK Doctoral education is currently undergoing review and redefinition. Since research and
knowledge production is increasingly seen as important to sustaining international economic
competitiveness, there have been moves to redefine the doctorate as a training for future
researchers, rather than as primarily about the production of new knowledge or individual
education. This approach is also being promoted across the EU: the Lisbon Accord (2000) is
 For the full report see D. Leonard, J. Metcalfe, R. Becker and J. Evans (2006) Review of literature
on the doctoral experience for the Higher Education Academy
promoting the establishment of a European Research Area and aims to see Europe as 'the
most dynamic and competitive knowledge economy in the world by 2010’. It recognised the
importance of the researcher and resulted in political objectives specifically related to the
role of the researchers and the need to improve their conditions and career opportunities.

These changes have followed swiftly upon measures aimed at raising general standards in
doctoral education in the UK, including a revised Quality Assurance Agency (QAA) code of
practice for research degree programmes and inclusion in institutional audits. Changes to
the traditional PhD have also been instigated by universities and include new forms of the
doctorate, measures to increase numbers (particularly of international students), and new
modes of delivery. Supporting bodies and organisations such as UKCGE, the UK GRAD
Programme, and local graduate schools have emerged.

An adequate evidence-base to inform both practice and the development of new
policy is needed, and this review set out to identify the state of the research base on
research degree programmes, and to draw from it those elements that influence the
researcher’s learning experience.

This review is one of five literature reviews commissioned by the Higher Education Academy
into various aspects of the student learning experience. The full text for all reviews can be
accessed at For further information on and links to
relevant initiatives including European and UK policy related to research degree

What does the review cover?
The review used a systematic approach modelled on the Institute of Education’s EPPI-
Centre methodology (, the basis of which is described later in this
summary. Literature was initially collected from a range of countries offering a ‘UK-style’
doctorate, but the volume of literature required the review to be narrowed down to UK
studies. The review produced

   •   a map of the empirical literature addressing the experiences of doctoral researchers
       in the UK
   •   an annotated bibliography on Endnote software and a searchable online database,
       which includes 1135 references, many from outside the UK, with full details of the
       search methods employed
   •   an in-depth review of a specific question relating to UK doctoral researcher
       experiences of the viva examination
   •   a commentary on the systematic review methodology used, with reflections on future

Map of the current literature
The review considered literature produced from 1985 onwards. Following the Winfield report
of 1987, the Social Science Research Council commenced a research programme looking at
aspects of research degrees and focussing on supervision, writing skills, QA, training in
quantitative methods, completion rates and times and labour market demands in the social
sciences. This programme, including two studies looking beyond the social sciences, made
up the majority of pre-1995 work in this area. These were based on fieldwork conducted
before the main pressures on universities to improve supervision, to speed up time and rates
of completion of doctorates, and to provide taught classes in methodology began to have
effect. Since 1995, relatively little research on the UK doctoral experience has been funded.
What research has been done has focussed on the professional doctorate, mostly
addressing how to supervise and assess knowledge production, on supervision in general
and more recently, on the viva.

120 out of 415 references concerning doctoral education in the UK were identified as
relevant to the review topic, i.e. relating to postgraduate researcher experience. (The
selection criteria for these studies are given in ‘systematic review methodology’ below). In
the main, these studies were conducted by:

      •   interested academics working unfunded or with small sums from institutions, mostly
          focussed on social sciences and education
      •   funders looking at their own students. These tend to be website reports not recorded
          on the BEI2
      •   the UK Council for Graduate Education, based on postal questionnaires and usually
          from the Deans’ and faculties’ perspectives
      •   doctoral researchers: there are a few unpublished theses and related publications on
          the doctorate
      •   individuals writing accounts of their own PhD, some in response to having failed or
          being referred
      •   activist bodies concerned to draw attention to and change the situation of students.

There is no comprehensive evidence available on the doctoral experience from the students’
perspective as a basis to build policy and practice: in other words the field is mainly ‘gaps’.
As regards the specific concern of this review - on-course learning experience and support -
there is little work on the UK, certainly as compared to the development of this field in N

We have no benchmark from which to judge the effects of recent changes, such as the
recent emphasis on transferable skills. There is much general talk of inter-disciplinarity and
internationalisation in relation to the doctorate, but next to no research on whether and how
disciplinary and national academic identities may be changing. We do not know the effects
on all students’ experiences of there being now more international and more part-time
students enrolled, or of supervisors each having responsibilities for more research students.
We need also to know more about life after the doctorate, and the difference that
undertaking doctoral research has made to peoples’ lives.

One of the difficulties of using the process of a systematic review was that a majority of the
studies were not based on any discernible theoretical framework, and the majority presented
mainly qualitative data. Literature concerned with the experiences of doctoral research
students in the UK tends to focus on:

      •   the PhD, rather than on other forms of doctorate
      •   the views of HEI managers, supervisors or policy makers rather than those of
          doctoral candidates themselves
      •   the situation in the social sciences and, especially, education
      •   working/studying context in terms of institutional provision (just over half of studies)
          although there is no systematic comparison across institutions or areas of the UK
      •   outcomes e.g. completion rates (over a third of studies)
      •   pedagogy (a third of studies), the majority of these on supervision
      •   peer support (a third of studies)

    British Education Index
   •   the viva and other forms of assessment (a quarter of studies)
   •   international research students (a high proportion of more recent studies).

Research is lacking on:

   •   the mode of study and funding sources
   •   comparative analyses of gender, age, ‘race’, ethnicity, social class or disability
   •   Russell Group universities, although the nature of the host university is often not
   •   experience of research training
   •   student support, with the exception of academic literacy support
   •   causes of drop out.

Several general themes can be drawn from the literature:

   •   for potential science PhD candidates, the working environment and nature of training
       provided is as important, if not more so, than the research project itself. This
       contrasts with the arts, humanities and social sciences where the topic, supervisor
       and location are key
   •   major differences in working environment, e.g. allocated space, and incidental
       support, e.g. from close contact with colleagues other than a supervisor, are
       observed across the disciplines and appear to affect outcomes
   •   the nature of supervision is one of the main factors influencing the researcher’s
       experience of a doctorate, and several studies show a need for supervisors to be
       more aware of the way in which this relationship is developing, to assess
       expectations and discuss problems
   •   subject discipline is a major factor in submission times and rates
   •   we know little of what happens to doctoral researchers in the years following their
       doctorate, and almost nothing about destinations of international students

A main outcome of this review has been the identification of the general paucity of the
existing literature on the working context and support for postgraduate research students in
the UK. This suggests that doctoral researchers are not seen as ‘customers’ to be attracted
and consulted – not even to the same extent as undergraduates. With increasing
international competition for doctoral researchers, especially with Australia and North
America, and the UK’s lead role in moves to a more structured PhD within the EU, it is
important that we evaluate the changes underway. We need to know better what we are
moving from, in order to retain past strengths and evaluate new initiatives.

The following are recommendations to the sector based on outcomes from the review:

   •   the sector should consider ways to maintain an updated literature map on doctoral
       education for use by researchers and policymakers. The annotated database of
       literature produced by this review should be used as the basis for this.
   •   commissioned research needs to recognise the diversity of UK research students
       and modes of study, such that initiatives aimed specific groups of researchers, e.g.
       based on funding source, do not have unforeseen effects on other doctoral
      •   Funding Councils should recognise the importance of research into the doctorate
          degree in all disciplines and consider making research on the doctoral experience in
          all fields eligible for submission to the RAE

An online searchable database of the 415 UK references described in this review is available
at A fully-annotated Endnote database with all 1135 references
related to doctoral degree programmes can be found at the Academy website: in the Directory of References. Note that in the larger
Endnote file, the methodology of studies included has not been appraised and the reviewers
make no comment on the findings described.

Impact of the viva voce
The literature describing the impact on UK researchers of examination of doctoral degrees
by viva was reviewed in greater depth using an extension of the systematic methodology
employed for the overarching literature map. This topic was identified as a major focus for
existing literature and is of particular interest given that several recent initiatives aim to
improve methods of assessment, of which the viva is one part. It is hoped that this analysis
will feed into the emerging debate on ‘What is a PhD?’.

Using the categorisation protocol developed for the systematic review, 17 of the final 120 UK
references relating to the student experience were identified as specifically addressing the
impact of the experience of the viva examination on doctoral researchers in the UK. (See
‘systematic review methodology’ below for details of selection criteria.) In common with the
general map of literature on UK experiences, the literature concerned with the impact of the
viva lacks direct evidence from researchers themselves. Few studies were assessed as high
‘quality’ for systematic review, due mainly to a lack of detail in describing the methodology.

Some themes, however, have emerged from the analysis:

      •   there is a lack of clarity on the part of examiners, supervisors and candidates about
          the purposes of the viva
      •   the viva is perceived by both supervisors and candidates as an unpredictable
          process and difficult to prepare for
      •   ‘mock’ or practice vivas are rare, but useful
      •   mismatches between the research paradigms of researcher and examiner can lead
          to disagreement about the quality of a thesis
      •   the attitude and personal conduct of examiners is a key factor in whether the viva is
          perceived as a positive experience, even among successful candidates. Experience
          of the viva can have knock-on effects for an individual’s research career

It is clear that more could be done to improve students’ experience of the viva and to ensure
it encourages them to do further research and to publish. The in-depth review has endorsed
the importance of precepts 22-24 from the QAA Code of Practice for Research Degree
Programmes3, and set out in the report a list of considerations for HEIs relating to the viva.

          •   The Higher Education Academy, or other funding bodies, should consider
              extending the in-depth review of the viva to other aspects of doctoral education

Systematic review methodology
This review is not a review in the commonly understood sense, i.e. an essay informed by the
literature. Rather, it is a selection of a sub-set of the published literature using tightly-defined
inclusion criteria and based on clearly defined questions. The review was undertaken
alongside the EPPI-Centre at the Institute of Education (ESRC National Centre for Research
Methods) which specialises in this approach. Staff involved in the process underwent three
days’ training, and specialist software was used alongside a systematic protocol for literature
collection, inclusion and categorisation. Note that this is not an EPPI-Centre review, or a
full systemic review, as time constraints did not allow for the whole range of procedures to
be carried out, such as the inclusion of a peer review.

Collecting and analysing the data
A database of relevant literature was constructed on an Endnote file as the basis for the
review. The initial search included several countries offering a ‘British-style’ doctorate, i.e.
South Africa, Australia, New Zealand and the US as well as the UK. Readily accessible grey
literature4 and unpublished material was included. Entries were systematically annotated
with procedures and details, e.g. to indicate their possible approach i.e. a ‘think piece’ and
whether or not they subsequently passed the inclusion criteria.

Much more material was collected than had been expected, even though time limitations
meant that not all available sources could be searched. A decision was made to focus on UK
material specifically relating to researchers’ experiences. At the end of the review period, the
file contained 1135 references, the majority from the UK and Australia (although this does
not accurately reflect the balance of all available material). 415 full text UK references were
screened for inclusion using the protocol developed below.

Inclusion was based on criteria drawn up during the first stage of the process, which
included clarifying the review question and associated definitions, planning search strategies
and making internal quality assurance checks to ensure consistency of approach. The given
title was refined to ask the broad question: What are the effects of different elements of
the learning context on doctoral researchers’ experiences and outcomes? Of the 415
UK references, 120 (12 of these unpublished) were found to meet the selection criteria.

Post 1985 English language studies were included if they:

      •   referred to doctoral level postgraduate research studies (including the M.Res) in the
      •   were about experiences as students or in the short period following studies
      •   focussed on working/ studying/ researching/ domestic and/or financial context
      •   provided some kind of empirical evidence or data or a secondary analysis of data

Selected references were analysed using ‘EPPI-Reviewer’ software to categorise studies
according to 32 selected questions looking at features such as the type of approach adopted,
    grey literature includes material such as web, conference and business reports
the outcomes forming the focus of the study, etc. This produced a broad overview of the
literature and identified gaps.

The question What is the impact on research students of the process of examination of
a doctoral thesis by viva voce? was used to screen the 120 UK references meeting the
original inclusion criteria. 17 publications or reports representing 14 studies were identified as
relevant. These were analysed using a condensed method, reflecting the EPPI data
extraction process, to capture in structured summaries both results, and information on
strengths of methods used in the studies.

Reflections on the use of the methodology
Employing this method ensures that a systematic and rigorous approach is taken to
assessing the state of the literature on a given topic, and offers a high level of transparency
for users. EPPI-Reviewer is a flexible tool, allowing input from different team members to be
compared. A major advantage of using this approach is that it is possible for others to widen
or continue pieces of work because procedures and sources are recorded and explanations
for decisions given. The methodology is as much, or perhaps better, suited to providing
‘maps’ than to answering questions, i.e. to exposing the extent and emphasis of research,
gaps in knowledge, where information on a topic is held etc.

Based on experience of conducting this review, the reviewers recommend that:

   •   the HE Academy should consider the EPPI-Centre methodology and particularly the
       EPPI-Reviewer tool as a future mechanism for literature reviews
   •   academics and researchers generally should consider the method, and it could lead
       in many cases to improvements in the way empirical work is described
   •   HEIs and official bodies should ensure that theses and publications (including their
       own publications) are recorded on BEI and made available electronically
   •   researchers should specify fully the details of their sampling frame and of the
       individuals in their studies and justify their methodological approach.