The need for an evidence based quality initiative

Document Sample
The need for an evidence based quality initiative Powered By Docstoc
					                                                DISCUSSION PAPER NUMBER 1
                                                Public discussion on the meaning of
                                                „evidence based‟ in higher education

      Towards an evidence-based approach to quality
           enhancement – a modest proposal
                         Clare Morris, University of Gloucester

The Quality Enhancement Network (QENet) and Institutional Research Network (IRNet) are
promoting discussion about an evidence-based approach in higher education. This paper by
Clare Morris is the first contribution to the discussion. It suggests how we might develop an
infrastructure to support an evidence-based approach to the enhancement of teaching and
learning. You can find out more about the discussion and join the QENet JISC maillist at:

LTSN is very grateful to Clare for her contribution to the public debate. If you would like to
contribute a perspective on this theme please send your contribution to

Norman Jackson – LTSN Generic Centre discussion co-ordinator

About the author
The author is Head of Quality Assurance at the University of Gloucestershire. An academic
statistician, she has previously held teaching posts and quality-related responsibilities at
Cardiff and Warwick Business Schools, the University of Hertfordshire and Bristol Polytechnic.
She chairs the Royal Statistical Society‟s Quality Improvement Committee, has consulted
widely in the field of industrial quality improvement, and is the author of a number of successful
texts in quantitative methods.


The current debate in relation to Quality Enhancement and its links with Quality Assurance,
taking place both here on the LTSN website and in other fora, has appealed on a number of
occasions to the idea of evidence-based practice. This paper is attempting to open up
discussion about how, as a community of practitioners engaged in the enhancement of
teaching and learning, we might work towards the goal of the improved use of evidence to
inform practice and policy.

What do we mean by evidence?

Evidence is factual knowledge or data that lends support to or casts doubt on a hypothesis. It
is information on which we base our beliefs and ideas of how the world works. Like everything
else to do with learning, it is associated with an ongoing process requiring the accumulation of
information. Each discipline has its own set of methodologies for the gathering and testing of
evidence, and for modifying the underlying concepts of the discipline as a result of this
process. In gathering evidence of the impacts of what we do on teaching and on students‟
learning we commonly draw on the methodologies used in the disciplinary fields of social
science, psychology and education.

The meaning of ‘evidence-based practice’
Jethro Newton‟s contribution to the debate on enhancement examines, from a research
perspective, how academics respond to quality-related policies. He highlights the importance
of this type of research to inform the formulation and implementation of institutional policy. In
the past both individual HEIs and national bodies have tended to develop policy and
procedures in something of an evidential vacuum – a fact of which I, no doubt in common with
many others working in the field, have been only too conscious.

Such considerations provide one important element in the development of a fully evidence-
based approach to the assurance and enhancement of teaching and learning. However, if we
revisit the definition of the term evidence-based as it is used in medicine, we find that there is a
second strand to the approach. Consider the following definition (Sackett et al 1996):

‘Evidence based medicine is the conscientious, explicit, and judicious use of current best
evidence in making decisions about the care of individual patients. The practice of evidence
based medicine means integrating individual clinical expertise with the best available external
clinical evidence from systematic research.’

If we substitute certain medically-oriented phrases in this definition with the corresponding
education-related terms; we obtain something along the following lines:

‘Evidence based teaching is the conscientious, explicit, and judicious use of current best
evidence in making decisions about the learning and learning experience offered to
students. The practice of evidence based teaching means integrating individual academic
and pedagogic expertise with the best available external evidence from systematic research.’

An important aspect of our individual expertise as teachers is the ability to engage in reflective
evaluation of our own practice in the light of evidence from systematic and other forms of
research. Thus we see that, while the systematic research base is indispensable, so too is the
professional input from individual practitioners. Figure 1 attempts to represent the process.

The accumulation of evidence by many practitioners is an important element of this evidential
framework. In the medical situation, the position might be something like this: one GP reporting
that a patient on medication A suffers palpitations may be no more than an anomaly; ten GPs
doing so may well provoke a research enquiry to investigate the phenomenon, the published
results of which then become part of the research base for informing practice. Having
developed a knowledge-base there is a need for the products of systematic research to be
synthesised and communicated in ways and at times that are useful and meaningful to
practitioners. It is not enough to provide this knowledge. If we are to progress to an evidence
based culture then we need to develop the capacities of individuals and institutions to use the
knowledge through reflective processes. Reflection and action are integral to an evidence
informed approach.

Figure 1 Simple representation of the basis for an evidence-informed approach
to teaching and learning.
    From policy to reality: enhancing quality is a messy business.

                                               Teaching / helping
                                               students to learn
            Action – changes
            to teaching

  Individual reflections on teaching                          Personal evidence of the results
  and students learning combining                             of teaching on students learning:
  knowledge gained through personal                           accumulation of      suggestive
  experience with knowledge gained                            evidence. This may include
  through systematic research leads                           personal action research and
  to new personal knowledge for                               feedback from students and
  improved teaching.                                          colleagues.

             Knowledge gained through systematic research. This
             knowledge will be derived from several sources including:
             educational research, pedagogic studies in the discipline by
             many practitioners, institution-based research.

Building the infrastructure to support an evidence-based approach

The point being made is that an accumulation of evidence from a number of individual
practitioners can have an impact which the separate items of evidence could not achieve. In
the learning and teaching arena, this has not traditionally taken place; as Liz Beaty pointed out
in a presentation given at the Institute for Learning and Teaching‟s Annual Conference in 2000
„reflection [on professional practice] is usually a private thing‟ and there may be „anxiety about
what counts as evidence‟, so that practitioners have tended not to share their experience with

However, we are now in an excellent position to alter this state of affairs. The LTSN, the
ILTHE, and in future the emergent Higher Education Academy, provide an appropriate forum
and collaborative data gathering capacity for the sharing of individual reflections and
experience, without demanding the amount of effort and „academic respectability‟ required of a
full-scale research publication. Without the interaction between the two elements identified
above – research informed by pedagogic experience and reflection, pedagogic practice with a
strong theoretical basis – development will be unbalanced. Once the „missing link‟ is provided,
practice and theory can advance hand in hand.

It might be argued that the LTSN and the ILTHE already provide a forum for this type of
activity. But there is still an element of formality about the current processes by which such
ideas may be disseminated – workshop presentations, papers which require research
underpinning and referencing, and so on. Of course, these more formal mechanisms are
essential to the development of the area as a discipline in its own right, but there is a place too
within the framework outlined above for brief, less formal contributions. It is this aspect which
the present proposal is designed to address.

A possible way forward

This proposal builds on good progress already made by LTSN and ILTHE to encourage the
sharing of knowledge about teaching and its impact on students‟ learning. It is proposed that
we would address the issue of systematising knowledge for the improvement of teaching and
learning through the creation of evidential databases by developing a mechanism to facilitate

their construction. In outline, it would operate as follows (following the medical analogy I am
using the term „intervention‟ to refer to the full range of innovations, activities etc which might
be involved):

1. The elaboration of a conceptual framework, which sets out the dimension of the
problem, called „what is evidence-based policy and practice in respect of teaching and
learning?‟ This paper makes a start.

2. A means of sharing knowledge about the impacts of different approaches to teaching on
students learning in different contexts. The website (which could be hosted by the HE
Academy) would provide a means whereby individual academics could briefly describe
successful (and less successful) innovations/interventions in the learning process. This could
be done via a template, so that information is obtained in a fairly standardised form, covering:

       the context of the intervention (subject area, level and background of students, size of
        student group). This aspect is particularly important, since these covariates may be
        crucial in determining the effectiveness or otherwise of the intervention;
       the impetus for the intervention            –   perceived   problem/student     evaluation
        comments/peer observation/etc;
       the nature of the intervention;
       details of evaluation of the intervention, and conclusions as to its effectiveness.

3. The data gathered via the website would in itself offer a valuable resource, acting as an
„ideas bank‟ for teachers, and saving precious time by pointing out possible pitfalls.

4. The full value would become apparent once a sufficient number of „cases‟ have been
gathered to permit some form of classification or taxonomy to be initiated. At this stage the
second element of the evidence-based approach would begin to be apparent, when a „critical
mass‟ of cases of one particular type might suggest a fruitful line for more formal research.
Insight may also be obtained into the type of covariate which may have a significant impact on
the success or otherwise of a particular enhancement initiative (for example, what aspects of
student demographics appear to be important).

This will not, of course, be an ab initio process; there is already a large literature evaluating,
very often on a case-study basis, the success of various types of intervention, which could
form a basis for the „clustering‟ of examples. Some examples might be:

       modifications to assessment practice
       effective use of student feedback in developing teaching methods
       discipline-specific approaches to skills development, – for example, group-working
        skills, leadership,

5. If a sufficient volume of „cases‟ is obtained, a searchable database could be constructed for
reference purposes. At this stage an evaluation or synthesis might be attempted to bring
together the essential learning points. These would be communicated to disciplinary
communities and the people who support and enhance teaching in institutions.

Additional benefits

The primary incentive for advancing this idea is my observation of a need to provide the
individual practitioner with a mechanism for contributing to the evidence base for policy and
practice, and for suggesting fruitful lines of theoretical research. However, this is not the sole
consideration and there are a number of other potential advantages to such an approach.

(a) Supporting the growth of ‘Scholarship of Learning and Teaching’ as a separate field
of academic activity

Recently there has been an explosion of interest in what has become known as SoLT,
prompted in part by the arrival on the academic scene of the ILTHE and the LTSN. This is a
welcome development, placing pedagogic research in a secure context and helping to raise
the profile (and, one hopes, the status) of those whose primary focus of scholarly interest lies
in the field of learning and teaching.

However, it does present some problems. Not surprisingly, perhaps, most work in the area
tends to be rooted in the traditions of social science research, and to adopt the methodologies
and paradigms of those traditions. This is in many ways appropriate, and serves to place the
developing discipline firmly within an established framework. But there is a danger that it may
create barriers between published work in SoLT and those academics whose background lies
in other disciplines, particularly the „hard‟ sciences and mathematics. To such colleagues, the
terminology and methods are unfamiliar, the conclusions sometimes obscure, and the whole
intellectual landscape somewhat alien. Published work may thus appear to have little to offer
to their subject. Conversations with colleagues in my own discipline of mathematics/statistics
suggest that this is indeed the case for many.

This point has been well made by Entwistle (in Gibbs (ed), 1994): „Unfortunately, many of the
innovations in teaching are not only advocated by social scientists, but the evidence of
effectiveness has come solely from the same area. Not surprisingly, staff from other disciplines
are unimpressed. We need good exemplars from different disciplines.‟

And Mary Taylor Huber, in an invited paper posted on the LTSN Generic Centre website in
October 2002, observes:

        „Biologists, historians, and psychologists may all agree that they want to foster “deep
        understanding” in their college classrooms, but what they mean by “deep
        understanding” is different…so too is the way they are likely to go about the
        scholarship of teaching and learning itself.‟

Thus opportunities need to be provided for practitioners to contribute to the quality
enhancement debate within the framework of their own disciplinary traditions. The proposed
project could offer a route towards achieving this.

(b) The time factor

Even where there is a willingness to engage with this unfamiliar territory, few busy academics
will have the time to develop a high level of expertise therein, particularly if they are also
research-active in their own discipline. At a recent SoLT-related seminar which I attended, a
leading expert observed that, just as one spends three to five years of „research
apprenticeship‟ before beginning to make a serious contribution to one‟s chosen field, so the
academic from another discipline should expect to serve a similar apprenticeship before
having something worthwhile to say about learning and teaching. This makes depressing
listening for the hard-pressed but enthusiastic teacher of history, mathematics or sports
science, possibly with several years of successful teaching and academic development to her
credit, and already trying to balance the demands of disciplinary research, assessment,
student support, and the dozen other activities which make up the life of an HE teacher in

We therefore risk allowing a two-tier system to develop, whereby we have on the one hand the
„professional teachers‟, expert in the business of learning and teaching, and on the other a
large number of experienced staff whose commitment to quality enhancement may go
unrecognised, and whose contributions to the advancement of the subject may be unreported,
because they are insufficiently grounded in theory.

(c) The ‘tricks of the trade’ approach

At the opposite extreme from the highly focussed, social science-based investigations cited
above, there is also a tendency for work to appear which generalises from the particular,
lacking any element of methodological rigour or repeatability. Such work sometimes bears the

name „case-study‟ or „action research‟ without meeting the real requirements of these
established and well-defined research methods. Again, Entwistle has encapsulated the
problem neatly: „there is often a problem in the published reports describing innovations in
teaching. They are often presented in unqualified and over-enthusiastic terms, without making
clear how dependent success will be on the context and the individuals concerned.‟ (in Gibbs
(ed), 1994).

A similar situation has arisen in the area of industrial quality improvement, where enthusiasts
for a particular methodology or approach (often associated with the name of an individual
quality „guru‟) have tended to „oversell‟ its advantages. Others eagerly adopting the same idea
then fail to obtain correspondingly impressive results, since no-one has bothered to analyse
fully the nature of the intervention, the reasons for its success, and the possible covariates
which may have a confounding effect. Disillusionment frequently follows, and may lead to the
conclusion that „these quality initiatives are just a waste of time‟.

All the factors cited above suggest that there is a need for a means of providing:

       a „halfway house‟ between full-scale SoLT-type projects and anecdotal evidence;

       a mechanism for input by academics who, while interested and engaged in the
        development of their professional practice, have neither time nor inclination to become
        involved in formal research in this area;

       the second element of the evidence-based approach – the accumulation of evidence
        via input from „individual academic and pedagogic expertise‟.

Potential difficulties

Clearly the main hurdle to be overcome in setting up a project as outlined above is one of
resources – the setting up and maintenance of a web-based mechanism for the collection of
cases, and in due course the summary and analysis of submitted examples, would require that
the scheme be hosted by an organisation such as the HE Academy.

Publicity could also be addressed via those organisations – obviously if colleagues do not
know of the existence of such a project, they will not contribute.

The template for collection of the details of cases will need to be carefully piloted to ensure
that it is reasonably transparent and gathers all the necessary data.

Finally, while the general intention is to be inclusive, some kind of „quality threshold‟ will need
to be operated in order to maintain the credibility of the project and to ensure that outcomes
are both useful and sound.

An evolving situation

There is probably no „good‟ time at which to be considering a project of this kind, since higher
education is never static; but we are certainly facing a particularly dynamic environment in the
wake of the White Paper published in January 2003. The tone of the White Paper with its
emphasis on professional development makes it even more important that mechanisms should
be available by which individual academics can demonstrate commitment to the teaching
element of their work, without necessarily having to sacrifice the time and intellectual energy
which goes into research and other scholarly activities in their disciplinary field. A project of the
kind described above could go some way towards meeting the requirement that all staff should
be „engaged in continuing professional development to maintain, develop and update their
skills‟ („The Future of Higher Education‟, paragraph 4.14). Indeed, if we can develop a culture
that believes in the approach, engagement with the process of creating and using an evidential
database would be one sort of evidence of professionalism in teaching.

The White Paper also advanced the notion of „Centres of Excellence in Teaching‟ (paragraph
4.28), an idea about which some concerns have been raised, not least at the recent LTSN
Conference (reported at One of the dangers in
such a development could be the exclusion of inputs from those academics not located in
„excellent‟ departments; a mechanism which helps ensure the recognition of developments
made outside the parameters of recognised „excellence‟ might be timely. More positively
Centres of Excellence in Teaching could provide a major source of knowledge for such an
evidential database. They could also be natural laboratories for the testing of hypotheses
about teaching and learning and create research data to enrich the sort of databases
described in this paper (Working Paper 4 Developing the concept of Centres of Excellence in
Teaching). Clearly the creation of a HE Academy would identify a natural home for such an
evidential database.


This paper presents a necessarily brief outline of the proposed project. I am grateful to the
LTSN for offering me a forum in which to air the idea, and would welcome comments and
suggestions at this stage. Please send comments to the author at


Beaty, L. (2000) Private Reflections and Public Accountability: the role of a professional
association [online]. Available from:

DfES (2003) White Paper ‘The Future of Higher Education’

Gibbs, G. (ed) (1994) Improving student learning through assessment and evaluation. Oxford
Centre for Staff Development.

HEFCE (2003) Final Report of the Teaching Quality Enhancement Committee.

Huber, M. T. (2002) Disciplines and the Development of a Scholarship of Teaching and
Learning in the United States of America [online]. Available from:

Jackson, N. (2003) Working Paper 3: Developing the concept of an Academy for the
Advancement of Learning and Teaching [online]. Available from:

Jackson, N. (2003) Working Paper 4: Developing the concept of Centres for Excellence for
Teaching & Learning. Available from:

Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B. and Richardson W. S.
(1996) Evidence based medicine: what it is and what it isn‟t. British Medical Journal. 312, 71-2.


Shared By:
Description: The need for an evidence based quality initiative