DFEE Normal Template by pX372jh2


									             Developing evidence-informed policy and practice in education
                                     Judy Sebba
          Standards and Effectiveness Unit, Dept. for Education & Employment

Paper presented at the British Educational Research Association Conference, University
                     of Sussex, at Brighton, 2-5 September, 1999.


At last year’s BERA conference two reviews of educational research had been published in the
previous month and much debate about the validity and implications of their findings ensued.
This short paper attempts to provide an update on progress and issues in one specific but
major area of the findings from the Hillage et al (1998) review, that of the need to improve the
relationship between research evidence, policy and practice. This relationship underlies many
of the conclusions in the review and much of the debate about educational research that has
gone before it. Discussion of the quality and impact of educational research is covered more
fully in a number of sources including Furlong (1998); Gray (1998); Hargreaves (1994;
1996;1998); Mortimore and Mortimore (1999) and Rudduck and McIntyre (1998).

Criticisms have been made of the quality of existing research and Hargreaves’s (1996) often
cited comparison to medicine suggests that educational research is of relatively poor quality.
Yet, in the press over a year ago (Guardian 24 June 1998), the editor of the BMJ suggested
that only 5% of scientific papers in medicine come up to scratch and that research evidence
has little or no impact on the patient sitting in front of the doctor. As many of you will have
experienced personally, it is far from apparent when seeking advice on medical treatment that
reliable and consistent evidence is available and used by most doctors. While debate about
which profession has the best evidence may be unhelpful, we are learning a great deal from
attempts by those involved in health care intervention about improving access to research.

There are two distinct but overlapping issues discussed in a recent paper in the British Journal
of Educational Studies by Phil Davies (1999). The first is making better use of existing
evidence which includes ensuring access, determining its relevance and judging its quality. The
second is establishing better quality evidence in the future. The first should influence the
second more extensively than at present, by ensuring that researchers and funders have
access to all current and previous research findings and the problems and issues - both
substantive and methodological - that they pose, prior to undertaking further research. Other
important factors such as the adequate supply and training of high quality educational
researchers and more investment in longer term research and larger comparative studies are

It should also be acknowledged that not all research can, or should have an immediate impact
on policy and practice. In a helpful chapter reviewing the impact of research on policy and
practice in key areas of education, Mortimore and Mortimore (1999) note that the school
effectiveness research took a long time to impact on policy. There are likely to be many
reasons for this but some areas of research will require replications, extensions and long term
outcomes before it is appropriate for changes in policy and practice to be made. The focus on
informing policy and practice should not be interpreted as suggesting that the role of longer
term exploratory research is undesirable or unnecessary. It is only through this exploratory
research that fundamental concepts are challenged and new connections made between
underlying theories and frameworks. The development of better databases and systematic
reviews in education should assist rather than impede this process.

The rest of this paper is concerned with improving the use made of current evidence. Primarily
‘users’ of research are being defined here as other researchers, teachers, LEA staff and policy
makers at national, regional and local levels. However, as with the Cochrane Collaboration in
health care which has inspired this work, other users such as parents, governors and pupils
could become involved. Some of the areas of research prioritised in medicine for systematic
reviews have been initiated by patients seeking better quality information on diagnosis and

The Cochrane Collaboration

The Cochrane Collaboration in health care intervention prepares, maintains and disseminates
systematic reviews of the effects of health care interventions. There are 15 Cochrane Centres
and 50 review groups each with members from many countries. Each group is responsible for
undertaking, maintaining and updating a review in their area of interest supported by the
centres. Review groups are mainly focused on an area of health care e.g. pregnancy and
childbirth, some focus on methodological issues e.g. qualitative methods group and others
span disciplines e.g. developmental, psychosocial and learning problems group. Each group
has 4-5 editors and a much larger number of reviewers from different countries representing
the range of perspectives on that topic. The co-ordinating editor typically provides 20% of their
time to the group’s work.

The unit for evidence-informed policy and practice in education

 We are currently commissioning a team to set up a unit to develop a collaboration in education
similar to Cochrane. The unit will have two main functions:

1.     Setting up a database of recently published, less recent but seminal and ongoing
research which will be accessible at a variety of levels e.g:

   comprehensive list of references on a topic which have not been subjected to quality
    checks, which another researcher or Phd student requires;
   full systematic review of evidence in an area where each study has been subjected to
    quality criteria by the review group which another researcher, funder or practitioner with
    dedicated time may require;
   half a page of key points arising from the systematic review that might be used as a basis
    for discussion in a school staff meeting.

These are merely possible examples to provide a flavour of what is needed. The team
commissioned will develop their own suggestions in response to perceived needs.

2.   Registering research review groups who provide a proposal for their work including
membership, scope, time commitment and support from their institutions.

The proposed review groups will need to demonstrate that they represent international
interests, research, practitioner and policy inputs, and are committed to generating relevant,
reliable and up-to-date evidence which minimises bias. The DfEE will be providing some
funding to the groups but the unit staff will be expected to assist the group by hand-searching
of journals and grey literature, electronic searching and development opportunities such as
discussions with other review groups.

NfER have been conducting a small pilot on undertaking a research synthesis along similar
lines to that used in Cochrane and are providing feedback on this which will help inform future

The unit team will need to work closely with the co-ordinating centre being set up by ESRC to
cover evidence-informed policy developments across the whole of the social sciences. It will
also need to collaborate with colleagues in the United States who are setting up an
international collaboration focusing on randomly controlled trials in social, criminological and
educational interventions. These developments are seen as broader but complementary to the
setting up of a unit specifically to focus on education initially and with a view to expanding to
further and higher education and employment later. We are working together on ensuring that
the systems used are compatible and accessible to one another.

Challenges ahead

1. Developing quality criteria

The Cochrane Collaboration has developed a set of criteria for judging studies which focuses
mainly, although not exclusively on randomly controlled trials. As Davies (1999) notes some
kinds of research questions (perhaps those likely to begin with ‘what’ or ‘whether’) can be best
addressed through larger quantitative studies which may involve randomly controlled trials.
Other types of research questions (the ‘how’ ones) which focus on the processes in education
or those requiring rich descriptions of the consequences of interventions on pupils, parents or
teachers, such as studies of the consequences of school exclusion, require more qualitative
research. While we would wish to encourage more large-scale quantitative studies and
randomly controlled trials, high quality research in education that is predominantly qualitative
will need to be included in the reviews and common criteria for judging them need to be
established. Robson (1993) has argued for the strengths in combining quantitative and
qualitative methods in order to complement one another and it is hoped that the research
review groups will demonstrate the benefits of bringing together studies using differing

The need to develop criteria that are commonly accepted by researchers, funders and
practitioners will be an important first task for the unit. However, there are useful bases for this
work such as the hierarchy of evidence provided by Harlen (1997). Furthermore, the unit will be
well supported in this task by the National Educational Research Forum (membership to be
announced later this month), which will provide a steer on the criteria as part of its work on
developing an overall framework for research. Existing Cochrane review groups, in particular
those focusing on qualitative methodology, will also be a useful source.

2. Establishing support from researchers

In order to become effective, the database and synthesis groups will need the cooperation and
support of most educational researchers. Some are understandably suspicious that these
developments are based on the Government’s desire for greater control or power over
educational research. After many years in which little notice was taken of evidence in policy-
making (Hargreaves, 1994) and the Department of Education was known by some as a
‘knowledge - free zone’, it is unsurprising that some researchers are finding it hard to enter into
a dialogue with policy-makers about research. The role of the Government in this initiative is to
lead and establish national coherence in the approach, not to assume control of the research
agenda or the overall research programme. Establishing the database and research synthesis
groups require researchers to share information and collaborate which they may have
reservations about doing in the context of competition for research funding.

The commitment to funding longer term projects, longitudinal studies and dedicated research
centres poses another challenge for researchers. During a 3 or 5 year programme of research
the researchers will need to share interim findings with the user communities. Decisions cannot
always wait for a project to be completed, yet historically researchers have been reticent to

reveal interim findings. Those of us who have conducted research over many years may find if
we revisited, let alone reanalysed our earlier work, that our interpretations or conclusions may
be different. If research is to build on earlier findings it is inevitable that our conclusions in any
given area change over time whether within one longer study or over many studies in the same
area. The proposed changes require more open, ongoing dialogue and greater trust between
researchers and between researchers and funders.

3. Establishing support from policy-makers

We have a considerable challenge in the Department to create a culture of expectation that
policy-makers will ask for, consider and use research findings in their work. This needs to
happen both in formulating policy and in evaluating policy implementation. Recent comments
from ministers in speeches and the commitment to developing the unit to promote better use of
evidence suggests that they would rather have the evidence than be protected from it even
when it conflicts with policy or is inconclusive.

There have been notable recent examples of policy being informed by research, although the
link has not always been made explicit. For example, the report by Maurice Galton, John Gray
and Jean Rudduck (1999) on transition and transfer which has informed guidance on the
Standards Fund, the implementation of a common transfer form and QCA’s work on bridging
units that pupils begin in year 6 and complete in the new school in year 7. Carol McGuinness’s
(1999) review of thinking skills has informed the National Curriculum review and the model
schemes of work. Both these studies have identified important areas for future research. The
University of Manchester study (Farrell, Balshaw and Polat, 1999) of the career structure,
training opportunities and effective use of classroom assistants reported at this conference and
being published later this month is informing guidance, developments in training and
establishing greater consistency in career structures.

Research commissioned specifically to evaluate current policies include evaluations of the
literacy and numeracy strategies, Beacon schools, EAZs and provision for pupils from ethnic
minorities all of which are providing ongoing formative feedback for policy development. The
target should be to make explicit in every policy or guidance document, the research evidence
that informed the content.

4. Improving the quality and availability of evidence

Evidence-informed policy or practice does not mean picking up research findings and
automatically translating them into policy or practice. Hence, our choice of the term evidence-
informed rather than evidence-based policy and practice. As David Hargreaves (1998) and
others have suggested teachers do not pick up research findings and implement them directly
into the classroom. Presented with evidence, some teachers will mediate or process the
findings according to the specific context (pupils, resources, staffing) in which they are
teaching. Similarly, policy-makers translate evidence in the context of the policy process taking
into account resource and political implications.

In some areas the research evidence is inconclusive. For example, research on pupil grouping
does not provide clear overall messages about how to group pupils to maximise outcomes.
Ideological positions on setting and mixed-ability teaching have created a false dichotomy
which is unhelpful. Researchers need to help others clarify the combination of approaches to
pupil grouping which is most effective in each context. Research on the benefits of school-
community links comes mainly from studies that are characterised by small-scale, localised
projects evaluated by those funding or providing them, in other words those with a strong
vested interest.

In other areas the research evidence is unavailable either because it does not exist or
occasionally because it has not been published or only appears in a unknown source. The cost
effectiveness of educational interventions is an example of an area in which until very recently
there was no research evidence at all and those working in the area are grappling with the
need to adapt a methodology developed in economics to education. The DfEE is
commissioning a dedicated research centre in the economics of education which we hope will
stimulate further work in this important field.

Some research findings are impractical to implement. For example, research over many years
has established the benefits of immersion schemes for learning foreign modern languages in
the primary years, but the shortage of specialist language teachers in secondary schools
makes it impractical to attempt to provide this opportunity in every primary school.

The research synthesis groups in developing and updating their reviews will make a significant
contribution to tackling these issues. The reviews will identify key findings and any truly
inconclusive areas. They will assist in identifying priorities for extensions and replications of
existing work as well as for new areas. Crucially, they will ensure better access to existing
findings and provide assurances of minimising bias for those who need the information.

5. Development opportunities in the skills of using and doing research

Practitioners and policy-makers may get access to research and not feel able to make sense of
it. Researchers employed in government departments help policy-makers use research by
interpreting findings, relating them to connected policies, judging findings in terms of
significance and helping draw out practical implications. Some teachers who have been
involved in action research, undertaken higher degrees or more recently have had TTA teacher
research grants or have been part of one of the school-based research consortia have had the
opportunity to develop their research skills. The Government’s green paper Teachers meeting
the challenge of change (DfEE, 1998) provides the basis for exploring more opportunities for
staff development for teachers including those involving research. Policy-makers may need
similar opportunities. Researchers themselves need to look critically at the way they
communicate and decide whether they should work collaboratively with others to communicate
more accessibly.

Recently we have been involved with the Teacher Training Agency in setting up a teacher
research panel of teachers with substantial research experience who can assist the
government agencies and others to identify appropriate priorities, provide a user perspective on
research applications, take part in research project steering groups and provide feedback on
how research is influencing practice. This will provide an important development opportunity for
the teachers involved but could, through the dialogue generated, provide an opportunity for
researchers, funders and policy-makers to develop a better overview of what is required. This
involvement of users is also a feature of the ESRC Teaching and Learning Initiative and the
Education Panel for the next Research Assessment Exercise.

There are a few isolated examples of pupils being actively involved in research. Fielding and
Sims-Fielding (1998) describe an initiative at Sharnbrook Upper School in which students have
been actively involved in the school improvement process as researchers. With some support
on research skills, they have undertaken surveys, interviews and other types of data collection
in order to investigate an issue identified by the student body. They then present their findings
to staff and students with recommendations of changes needed. One recent example of their
impact on school improvement was in the area of personal, social and health education in
which as a result of the data collected they redesigned the syllabus and this was adopted by
the school.

An invitation to contribute

This paper has outlined briefly some of the current national developments that will contribute to
improvements in evidence-informed policy and practice. If they are successful, further
challenges will emerge. For example, the ultimate success criteria for the database and review
groups might be to find pupils accessing the best evidence on classroom practice in order to
challenge their teachers. The same information that gives rise to challenge should provide
greater confidence, but not complacency, that decision-making at every level can be done in
the knowledge of the best possible evidence. This requires an ever-expanding range of
research syntheses to be undertaken and updated and everyone to be committed to using the

If you would like to contribute to this process there are a number of ways in which you can do
so. These are some suggestions made at the meeting held on evidence-informed policy and
practice in July 1999 at the School of Public Policy:

Identify a group of key researchers (international) who are working in your area of interest;

Volunteer to be a co-ordinating editor, editor or reviewer in your specific area of expertise;

Offer to assist in searches for reviews by hand or electronically;

Offer to read drafts or copy edit and comment on accessibility;

Organise study leave for practitioners with whom you work to enable them to be involved;

Develop and participate in training programmes on research skills;

Comment on the forthcoming consultation from HEFCE on the criteria for the Research
Assessment Exercise to ensure reviews are adequately valued;

Explore cross-disciplinary developments - social work, criminology and others are involved in
similar initiatives;

Participate in advisory committees;

Offer moral and intellectual support to your colleagues who invest time in these developments.


Davies, P. (1999) What is evidence-based education? British Journal of Educational Studies 47

DfEE (1998) Teachers Meeting the Challenge of Change London: DfEE

Farrell, P., Balshaw, M. and Politz, F. (1999) The Management, Role and Training of Learning
Support Assistants (LSAs). Research Report 161 London: DfEE

Fielding, M. and Sims-Fielding, E. (1998) Evaluation of the students as researchers initiative:
Interim Report, Sharnbrook Upper School, Bedford

Furlong, J. (1998) Educational research: meeting the challenge. An inaugural lecture,
University of Bristol, 30 April 1998

Galton, M., Gray, J. and Rudduck, J. (1999) The Impact of School Transitions and Transfers on
Pupil Progress and Attainment Research Report 131 London: DfEE

Gray, J. (1998) The contribution of educational research to the cause ofschool improvement.
Professorial Lecture, Institute of Education, University of London, 29 April 1998

Harlen, W. (1997) Educational research and educational reform In: S.Hegarty (ed.) The Role of
Research in Mature Education Systems Slough: NfER

Hargreaves, D. (1994) The Mosaic of Learning: Schools and Teachers for the Next Century
London: DEMOS

Hargreaves, D. (1996) Teaching as a research-based profession: possibilities and prospects.
The Teacher Training Agency Annual Lecture 1996

Hargreaves, D. (1998) Creating Professionalism: the Role of Teachers in the Knowledge
Society London: DEMOS

Hillage, J., Pearson, R., Anderson, A. and Tamkin, P. (1998) Excellence in Research on
Schools London: DfEE

McGuinness, C. (1999) From Thinking Skills to Thinking Classrooms: a review and evaluation
of approaches for developing pupils’ thinking skills Research Report 115 London: DfEE

Mortimore, P. and Mortimore, J. (1999) Does educational research influence policy or practice?
In: I. Abbott (ed.) The Future of Education Research London: Falmer

Robson, C. (1993) Real World Research Oxford: Blackwell

Rudduck, J. and McIntyre, D. (eds.) (1998) Challenges for Educational Research London: Paul


To top