University of the West of England
SCONUL/LIRG Project on
Final Project Report
Table of Contents
Executive Summary 3
Project Background 4
The Project at UWE 5
Data collection methods 7
Key results 10
Appendix 1a. Faculty of Health and Social Care. Ethics sub-committee
Appendix 1b. Consent form
Appendix 1c. Participant information sheet
Appendix 2. Form for capturing informal comments
Appendix 3. Student questionnaire
Appendix 4. Interview questions for teaching staff
Appendix 5. Interview/observation schedule for researchers
Appendix 6. Results of teaching staff interviews
Appendix 7. Impact of EIS on research(ers) – outcomes of interviews, observed
task and questionnaire
The University of the West of England (UWE) Library was one of twelve UK
university libraries which participated in Phase II of the SCONUL/LIRG Impact
Measurement Initiative, running from 2004 to 2005. As part of this project we set out
to measure the impact of our Electronic Information Services (EIS) on students,
teaching staff, and researchers. The reason we chose this aspect of our library service
to evaluate was because, in common with all UK academic libraries, the UWE
Library Service spends increasing amounts on the purchase of electronic information
services each year, (even to maintain status quo) and our aim was to discover whether
this outlay is justified and effective.
Being a national initiative our involvement gave us opportunities to meet with and
share experiences with project teams from other participating libraries and in
particular from Bournemouth University (BU) whose choice of a similar theme gave
us further scope for collaboration.
One of the main objects of the Initiative was to develop tools that could be used by
academic libraries to measure the impact of all aspects of their library service on
teaching, learning and research. Consequently this report provides a reflection on the
processes of impact evaluation and on the effectiveness of the methodologies used.
We also outline some of the other outcomes of our participation in the Initiative.
The report sets out our research methods and the results obtained in full, and
draws on the results to show that while our spending on EIS is justified, especially in
the context of the move towards e-learning, the effectiveness of this outlay could be
Four recommendations are given for maximizing the impact of our EIS as follows:
i) raise awareness of EIS through promotion and training;
ii) work with academics to help them identify relevant resources
iii) train students how to evaluate information
iv) develop new partnerships for information literacy training for students
A further recommendation is that measurement of the impact of the library’s services
should be an ongoing process that will feed into the library’s strategic planning.
1. Project Background
The SCONUL (Society of College, National and University Libraries)/ LIRG (Library
and Information Research Group) Impact Measurement Initiative is a national
research project looking at the impact of higher education libraries on learning,
teaching, and research, with the aim of developing a toolkit of impact research
methodologies for HE libraries (Payne and Conyers, 2005).
It began as a one-year pilot project, running from July 2003 – July 2004, with ten
higher education libraries taking part, each identifying an impact theme and a topic
that they wished to explore in depth. A second phase of the initiative then ran from
July 2004 with a new cohort of twelve HE libraries including UWE Library.
To prepare for the study, in July 2004, project teams from these libraries met for a
two-day Impact Implementation seminar, during which they were guided through the
project planning process by the facilitators David Streatfield and Sharon Markless. It
was made clear that “impact” usually means impact on people, and in this context we
were measuring impact by assessing changes in behaviour, competence, and levels of
knowledge and attitudes. Qualitative research methods were recommended to elicit
this kind of information. The Measuring Impact Initiative is based on action research
principles and uses an established impact measurement methodology as the common
model for all the participants (Markless and Streatfield, 2005). Working to this model
participants were encouraged to set measurable objectives for their chosen impact
theme, identify success criteria, specify the evidence required to assess whether those
criteria have been met, and to work out what impact measures they would use. By the
end of the seminar, each institution had developed its own action plan relating to its
A second implementation seminar in November 2004 brought the teams back together
to exchange information and receive feedback on their research instruments.
A final meeting in which project teams from the participating universities came
together again to share their experiences was held on December 5th 2005
Throughout the project support was also offered through an email discussion list, LIS-
For the University of the West of England the Impact Initiative provided an
opportunity to contribute to and benefit from a collaborative research effort, not only
as one of the participating institutions, but also, more closely, with the University of
Bournemouth whose chosen theme was also on the topic of EIS. The advantages of
collaboration were, as we saw it, sharing experiences and ideas with other
participating institutions as well as additional opportunities for triangulation of results
from multiple data sources.
2. The Project at UWE
2.1 Project team
Our project team consisted of Jackie Chelin (Deputy Librarian), Dianne Nelson
(Faculty Librarian and project manager), Jane Redman and Pauline Shaw (Assistant
Librarians). Sarah Dawson (Assistant Librarian) was a member of the team from July
to December 2004. Also involved, at various junctures, were Faculty Librarians,
Assistant Librarians, IT Services and the Students’ Union.
2.2 Project topic
The aspect of the library service we were keen to evaluate was the impact of our
Electronic Information Services (EIS) on students, teaching staff, and researchers,
since, in common with all UK academic libraries, the UWE Library Service spends
increasing amounts on the purchase of electronic information services each year,
(even to maintain status quo) and our aim was to discover whether this outlay is
2.3 Project plan
A project plan was drawn up setting out the theme of our study, the aims and
objectives, success criteria, the evidence required, how we would collect it and the
implementation plan which outlined timescales taking into account the activities of
the academic year (Table 1).
Aim: To discover whether the increasing amounts spent on electronic information
services (EIS) each year are effective and justified
EIS, for the purposes of this study, are defined as: “Electronic information services
for which the library pays a subscription fee, e.g. Web of Knowledge, Science Direct,
Table 1. Project plan
Objectives Success criteria Methodologies Timescale
1. To measure Increased use of EIS, a) Observed task with 2nd and December
the impact of generally final year students 2004
EIS on students’ b) Structured interviews (or
learning survey) of 1st and 3rd year
Increased remote use a) Structured interviews (or January-
of EIS, e.g. from survey) of 1st and 3rd year March 2005
home, placement, students
workplace b) Use of existing statistical data
Use of EIS for a broad a) Structured interviews (or December
range of purposes, survey) of first and third year 2004
e.g. social, self- students
Increased number of a) Study of the “Source of Ongoing
references to EIS on reference” section on ILL forms
interlibrary loan to compare over a number of
2. To measure Increased number of a) Scrutinise differences in January-
the impact of links to EIS from module information/links from March 2005
EIS on staff virtual learning the centrally supported VLE this
teaching environments year, compared with same
modules in previous year
Integration of EIS into a) Semi-structured interviews January-
the curriculum March 2005
Increased promotion a) Semi-structured interviews January-
of EIS through faculty March 2005
Increased inclusion of a) Sampling of module November-
Information Retrieval documentation April 2005
skills in module
3. To measure Awareness/use of EIS a) Mix of structured December
the impact of within researchers’ questionnaire(s), observation of 2004
EIS on subject areas a specific task and training
Improved Mix of structured December
effectiveness of desk- questionnaire(s), observation of 2004
based research when a specific task and training
1, 2, & 3. Impact Informal comments will be
of EIS on all collected by library staff
groups throughout the year for all
relevant criteria and followed up
by critical incident interviews,
2.4 Publicising the project
Papers were tabled at the University’s Teaching Learning and Assessment Committee
and at the Library Advisory Group to alert colleagues to the project.
3. Data collection methods
We decided to use a variety of methods to enable triangulation which acts to
strengthen a study by overcoming some of the problems of research, such as bias, or
small sample numbers, and thus gives the results a greater validity. Another strength
of our research was the fact that two different sets of investigators at two separate
institutions were able to collaborate and compare methodologies and findings.
At Bournemouth University their impact study team decided to survey students and
staff of the Institute of Health and Community Studies (IHCS), as these users require
access to current, evidence-based information and many are located at a distance, and
while at BU they used the same methodologies for all groups, at UWE we selected
different methodologies for each user group, e.g. a questionnaire for students, semi-
structured interviews for teaching staff, and interview/participant observation for
researchers. Other data collection methods used were: capturing informal comments,
analysis of documentary evidence, and analysis of usage data.
Before we could proceed ethical approval had to be obtained from the local research
ethics committee for all the proposed data collection strategies (Appendices 1a, b, and
c). This was an essential requirement of the Health and Social Care Faculty.
3.1 Capturing informal comments
It was intended that informal comments on our electronic information services (EIS)
would be collected by library staff throughout the year for all relevant criteria and
followed up by critical incident interviews, where appropriate.
Collection points could be anywhere where we interface with our users - issue desks,
enquiry desks, offices etc. Forms on which to note users’ comments were circulated to
those points (Appendix 2). If anyone seemed to have more they could contribute on
the subject there was a space on the form for jotting down their contact details so they
could be contacted to take part in a survey sometime in the future - either by interview
or questionnaire. This method was subsequently discarded as being unusable.
3.2 Document analysis
Bibliographies in student projects.
Although this method of data collection was investigated it was discarded as a reliable
method because it may not always be obvious whether a cited journal article was
retrieved in print or electronically.
Module documentation. Work was begun on assessing whether the Learning
Objectives within module specifications had changed to include more mention of
information retrieval skills in recent years. However, on finding out that this section
of the module specification was not necessarily free text, but taken from a pick list,
then this method was also discarded.
Interlibrary Loans forms
Interlibrary Loans forms were analysed with the objective of demonstrating an
increased number of references to EIS on interlibrary loan forms over the past 5 years
to provide evidence of an overall increase in the use of EIS. The “Source of
reference” section on ILL forms was compared across a number of years.
3.3 Questionnaire surveys
The choice of an online survey to measure the impact of electronic information
services (EIS) on students’ learning was based on its relative low cost, potentially fast
response rate and its simplicity to complete in a short time.
The online survey (Appendix 3) was aimed at students only. It had an introductory
page which stated what the survey was about, where more information about the
project could be obtained, but more importantly it explained what EIS were in this
context, as students were unlikely to be familiar with this term. The survey was a
structured survey and consisted of 16 questions in three sections. Section A had
seven specific factual response questions, section B nine behavioural questions
focusing on the use of EIS and requiring a specific response or scaled specific
response, and section C allowed for any optional open response about EIS.
The survey questions were chosen by consensus for their perceived ability to measure
impact and against our stated success criteria. It was piloted among library staff of all
levels prior to the student launch and the design and question wording was
The survey was accessible via the UWE Student Intranet, Library Services and the
UWE Students Union homepages. It was launched and relaunched over a period of
two months in total during the second semester of the academic year from mid-April,
when it was likely that all students would have had a need to access electronic
resources for their academic work. The student sample was self-selecting. The
respondents were anonymous, but to comply with UWE ethics guidelines they had to
consent to the information they provided being used by us in order to successfully
submit their completed questionnaire.
The online survey was created as a web form using web editing software, ie
Macromedia Dreamweaver MX, with the response data imported into a Microsoft
Access database in real time. The collected data was exported into MS Excel for
manipulation and analysis.
Comparison with Bournemouth University’s online survey
The BU & UWE student questionnaires were very similar in structure and length,
even though they were devised independently. They also received a similar number of
responses (BU - 251 and UWE – 387), although it is worth noting that the BU survey
was sampling a smaller population. BU also offered an incentive in the form of a prize
draw and UWE did not.
The questionnaires were remarkably similar in appearance, even though BU used
commercially available software (The Bristol Survey Tool) and UWE used an in-
house solution. The main difference appeared to be that the results of the BU survey
were easier to present and analyse than the UWE survey results, as the Bristol
software had various options to cross-tabulate, export and filter results, whereas the
in-house software required more labour-intensive analysis.
3.4 Semi-structured interviews
Semi-structured interviews can elicit valuable data by enabling a wider exploration of
the issues than a questionnaire could. This method was used with the teaching staff.
The sample was drawn from five separate Faculties through a number of different
approaches. Initially the project was publicised at a meeting of the university-wide
UWEonline Support Group which was set up to support the development of e-
learning through the University’s VLE. It was felt that members of this group would
be particularly interested in participating in the project. Group members were invited
to volunteer to be interviewed. Also, selected academic staff were contacted by email,
and invited to participate. In some cases, a direct approach was made to academic
staff known to have a good rapport with the library and who would therefore be
willing and co-operative participants. In all cases the participants could be described
as being self-selecting.
A reasonable number of willing volunteers came forward, and 9 were interviewed.
Each interview, which lasted about an hour, was conducted by a Faculty Librarian,
with a note-taker also present. At the beginning of the interview the study and its aims
and objectives were outlined, and assurances given about the confidentiality of the
The interview questions were designed to be open questions. They followed a similar
format in each case (Appendix 4), focusing on the following key areas:
• Academics own use of EIS
• Use of EIS in teaching
• Perceived impact
Discussions around each topic were encouraged to elicit as much information as
The data was analysed according to the method proposed by Miles and Huberman
(1994) of reducing data through thematic grouping, and then collating interview data
into pre-identified themes.
3.5 Questionnaires/Participant observation
A different data collection method was used with researchers which involved
participant observation, as well as a questionnaire and interview. This followed the
format of : a brief questionnaire to collect biographical data; observation as the
researcher performed a set of specified information retrieval tasks; followed by
training on the basis of the observed behaviour. A short follow-up interview was
carried out about a month later (Appendix 5).
The same sampling procedure was used as for the teaching staff, resulting in a sample
of 13 participants from six different Faculties. The same procedure was used as with
the semi-structured interviews, with each interview/observation lasting about an hour,
conducted by a Faculty Librarian, with a note-taker also present.
The subject was asked to carry out three tasks using EIS while being observed. In
participant observation, the researcher takes an active role, in this case responding to
the subject’s comments. At the same time the scribe acted as a non-participant
observer, and recorded the actions and comments of the subject.
4. Key results
As the project progressed it inevitably became modified. Timescales changed and
some of the data collection methods were found to be more successful than others, or
had to be abandoned altogether. The methods which yielded the most useful results
were the online questionnaires, semi-structured interviews and participant
4.1 Analysis of Interlibrary Loan forms
A total of 12,141 interlibrary loan (ILL) forms covering a four week period around
Easter over five consecutive academic years, 2000-05, were examined. The forms
were collected from two campus libraries (Frenchay and Glenside) which service
eight different faculties.
The statistics gathered indicate that a total of 5,158 people (42.5%) had included some
information in the “source of reference” section on the form. This figure showed a
marked decrease over the five-year period i.e. users have tended to quote sources less
and less often, thus going directly against our initial success criteria. However, within
these figures, another useful statistic emerged. Where a source was quoted, users have
continued to name subscription databases (EIS) consistently throughout the five-year
period studied. Indeed, at Glenside library, of the people who quoted a source almost
75% did in fact quote a subscription database and at Frenchay this figure is over 50%.
Table 2. ILL form analysis
Statistics from Frenchay campus library:
Year % quoted a source % quoted EIS
2004 26 55.2
2003 38 56
2002 43 58.6
2001 36 47.5
2000 48 65
Statistics from Glenside campus library:
Year % quoted a source % quoted EIS
2004 37.5 83.7
2003 55.1 76.7
2002 58 70
2001 65.3 80.1
2000 45 57.8
From these statistics, we can conclude that our EIS are being used, although we still
cannot confirm whether or not there has been an increase in their usage.
4.2 Online questionnaire
Of the 387 students who responded to the online questionnaire 217 (56%) positively
stated that they were users of EIS. These 217 responses were analysed further.
The respondents were evenly spread across the 1st (24%), 2nd (30%) and 3rd years
(26%), but with fewer 4th year and postgraduate students (9% and 11% respectively).
All faculties were represented except Art Media and Design.
Students’ awareness and use of EIS
• 89% had access to the internet at home and 96% claimed to have good to very
good IT skills.
• Just over half (53%) had received library training on using EIS, nevertheless
98% claimed to have good to very good information finding skills.
• A wide range of EIS is used by the students - a total of 30 databases were
• In response to the question “How do you find out about useful EIS?” (where
more than one response could be given); 92 (22%) went by tutor
recommendation, 85 (21%) by module documentation, 74 (18%) through
UWEonline, 67 (16%) through the Library web pages, 41 (10%) through a
Faculty Librarian; and 28 (7%) through Library publicity (see Diagram 1).
This represented a ratio of faculty sources to library sources of 65:35.
Diagram 1. How respondents found out about EIS
Library publicity documentation
Increased use of EIS
• In response to the question “How has your use of EIS changed over the last
year?” 24% said it had greatly increased, 48% said it had increased, 26% said
it was the same, and 2% said it was reduced. Therefore there was an overall
increase in use by 72% of the sample.
Increased remote use of EIS
With no base line for comparison it is hard to say whether the following figures
represent an increase in remote use except within the context of the overall general
increase in use of EIS.
• Asked “Where do you mainly access the EIS you use?” 63% of the students
indicated that they accessed them remotely, in most cases from home. The other 37%
accessed them on campus.
Diagram 2. Where respondents mainly accessed EIS
Use of EIS for a broad range of purposes e.g. social, self-development
• It was felt that EIS could be judged to be truly embedded in students’ learning
when they became aware of their wider applications, other than for
coursework. From a multiple choice pick list the respondents selected the
following applications for which they most frequently used EIS: 194 (35%) to
find reading independently for themselves or their coursework; 149 (27%) to
find reading recommended for coursework; 71 (31%) to find primary data e.g.
statistics, financial data; 41 (8%) for reference e.g. maps, dictionaries etc.; 32
(6%) to find career or job related information; 27 (5%) for newspapers; and 20
(4%) for hobby or interest related information (Diagram 3 and 4).
Diagram 3. The type of information sought using EIS
Own coursework 194
Recommended reading 149
Primary data 71
Diagram 4. The type of information sought using EIS as a percentage of total
number of responses
Reference Recommended reading
4.3 Semi-structured interviews
These were carried out to evaluate the impact of EIS on staff teaching. The full table
of results from which the following key points have been summarised is presented in
Nine members of teaching staff from five different Faculties were interviewed.
Academics own awareness and use of EIS
• All nine had used library databases to a greater or lesser extent.
• Between them they identified 20 EIS which they regularly use.
• Eight of the nine used both the library and the library web-pages as a guide to
what is available and for information about and training on EIS. Indeed the
library web site was mentioned favourably a number of times.
• However, overall awareness of what EIS the library provides could have been
• There was a tendency in some cases to stick to the resources they already
knew about, rather than to spend time exploring new resources.
• The availability of full text was appreciated, especially in the social
• There were as many references made to freely available websites as to library
EIS e.g. government websites, professional organisations’ websites, blogs,
discussion boards etc. One respondent claimed to use Google rather than EIS.
Increased promotion of EIS
• All the interviewees claimed to promote EIS, in particular the resources which
they themselves are familiar with and confident in using.
Integration of EIS into the curriculum
• Seven of the academics claimed they had integrated EIS into the curriculum,
although the degree to which this occurred varied widely.
• The various ways in which EIS was used in teaching included:
- academics recommending specific web sites to students
- pointers to relevant EIS in assignment briefs
- the requirement for EIS to be used when completing assignments
- the inclusion of references EIS in module handbooks.
The feeling was that EIS is not yet integrated into the curriculum as much as perhaps
it could be. Two academics expressed their intention to integrate EIS further in the
future, and both felt this would come about as a result of greater use of the VLE.
Increased number of links to EIS from VLEs
• Seven of the academics were using the VLE to either link to EIS, information
about EIS, or articles obtainable through library EIS – or if they weren’t they
were intending to.
• It is hard to know whether this represents an increased number of links to EIS
from VLEs since we do not have a base line. The idea of quantifying this by
selecting modules that have run in consecutive years, and comparing the
number of links to EIS within them, was abandoned following discussion with
technical staff managing the VLE and its archived modules. However the use
of a VLE is a relatively new way to deliver the curriculum (2 years), and its
use has been increasing steadily.
• It was generally felt that use of a VLE would help to integrate EIS into the
curriculum and focus the staff on a new way to deliver the curriculum.
• Also, the VLE is driving a preference for the full-text resources that enable
linking at the article level.
• One interviewee noted that, by using the VLE to link to other e-resources that
are freely available over the web, there is a danger that the library-provided
EIS could be by-passed altogether.
Other factors affecting impact of EIS on staff teaching
According to our premise, the impact of our library EIS on teaching can be judged by
the integration of EIS into the curriculum, increased promotion of EIS by the teaching
staff and an increase in the number of links to EIS from VLEs. However, from the
themes which emerged from the interviews it is clear that these success criteria are
influenced by factors other than just the simple provision of EIS per se, such as:
• attitudes to EIS, whether positive or negative
• training and support received
Teaching staff attitudes to EIS
Whether the academics perceive EIS as having positive benefits for their students, or
as having negative aspects will affect their readiness to use them in their teaching. On
balance there was a greater appreciation of the positive benefits such as:
• EIS enables students to retrieve large amounts of up-to-date information
quickly and easily;
• being able to give better student support by providing them with links to
sources through EIS;
• student dissertations now contain a wider range of references;
• the students prefer electronic resources;
• EIS help to deliver resources which look more professional (“rather than
scruffy old photocopies”), resulting in better course evaluation.
There was some anecdotal evidence to suggest that there has been some improvement
in the standard of the student projects which could be attributed to EIS. However the
feeling was expressed that “good students will use EIS effectively and the weaker
ones will not take the opportunity”.
On the negative side :
• at least six of the respondents expressed their concern that the students were
unable to adequately evaluate the large amount of information available to
them, or to assess its quality. As one respondent put it “If we give them wide
access, we also have to give them the skills to be discriminatory and know
how to use information.”
• There was some concern about increased plagiarism “because this is easier in
an online environment”, but most of the respondents did not seem unduly
concerned and there appeared to be no evidence to substantiate it.
• There was also some concern expressed about a tendency to spoon-feed the
students by simply giving them links to reading materials from the VLE,
rather than letting them search for and retrieve them themselves.
Training and support received by teaching staff
• The EIS which teaching staff promote are the ones they themselves are aware
of and are confident in using. This is dependent to a large extent on the
training and information they have received.
• Teaching staff need to learn how to use VLEs effectively before they can
exploit them fully.
4.4 Questionnaires/Participant observation
Observation, training and follow-up questionnaires were used to evaluate the impact
of EIS on research(ers). A full table of results is presented in Appendix 7.
Thirteen researchers from six faculties were surveyed.
Awareness/use of EIS within researchers subject area
• All subjects were aware of the importance of EIS for their research. This was
illustrated by comments such as: “EIS is essential”; “Can’t imagine how I
managed without them”
Improved effectiveness of desk-based research when using EIS
Effectiveness of carrying out research using EIS was judged according to the
researcher’s ability to navigate, knowing what EIS to use, knowing what is available,
their search strategies, and knowledge of alerting services.
• Across the sample there was a wide range of ability. There was some evidence
to suggest that increased effectiveness correlated positively with increasing
years of research experience.
• There was also evidence that the impact of EIS on research(ers) could be
further increased by more research focused training.
Other factors affecting impact of EIS on research(ers)
• Positive factors included: multidisciplinary access to information provided by
the wide range of EIS available; the high quality of the resources; and the
support provided by Library staff and by the Library web pages.
• Negative factors included: access problems due to password requirements and
slowness of network; lack of knowledge about what EIS the library subscribes
to; and the information overload due to providing access to so many resources.
4.5 Impact of EIS as evidenced by quantitative data
Although the use of statistical information has its limitation, particularly with regard
to impact studies, it can be used to help to corroborate, or challenge, some of the
conclusions being drawn from the qualitative survey methods in use.
The surveys we carried out generated a list of databases which the subjects said they
used regularly, and for which the usage statistics for 2000/01 and 2003/04 could be
compared in many cases (see Table 3). Certain titles mentioned were taken out of the
list, either because they were not available in 2000/01, or were only available on CD-
ROM, which was not so readily accessible as the Web.
For all titles where data was available the increase in use has been emphatic.
Naturally, an increase in student numbers may account for some of this, although
certainly not by more than a few percentage points. The statistics appear to
corroborate our evidence of increasing use of EIS obtained from the student survey
and to confirm that use of EIS is increasing in leaps and bounds. The reasons for this
are expanded in the outcomes of the interviews with staff, i.e. that they themselves are
using more sources in their research, and are recommending them to students (indeed,
expecting students to be using them), and the fact that EIS are being promoted in a
variety of ways that helps to reach a wide cross section of users, including through the
library’s user education sessions and web pages..
Table 3. Database usage statistics
Databases 2000-01 2003-04 % increase
AMED 747 9688 1297
Business Source Premier (2001/02) 26464 84831 321
CINAHL 8410 40983 487
Compendex/El Village 442 1279 289
Education Indexes 975 3619 371
Emerald 5159 6130 119
IBSS 1341 2385 178
LEXIS-NEXIS 9662 31990 331
Medline 9023 31790 352
Mintel 809 5943 735
PsycInfo 809 7793 963
Science Direct 14225 15927 112
SportDiscus 414 1860 449
Zetoc 315 1443 458
5.1 To what extent do the outcomes confirm the aims/objectives and success
The success criteria for all the user groups appear to have been satisfied, thus, in
theory, proving the case for the impact of EIS at UWE. However these results need to
be viewed in context.
We need to be aware that certain aspects of the methodologies such as the sampling
and interview techniques could bias the results. For instance, an online questionnaire
is likely to attract IT literate respondents. Also, enthusiastic users of EIS or academics
who were very pro-library would be the ones most likely to volunteer for interviews.
If we accept the definition of impact as something which results in a change of some
kind, then in order to judge whether and to what extent a change has taken place we
need to have some baseline data for comparison but because this was the first study of
its type carried out by our library this was not possible. What we have achieved is a
snapshot in time, but one which should prove useful if we intend to carry out further
impact evaluations in the future.
We also need to ask ourselves if the success criteria chosen really were indicators of
impact, and furthermore, could we strictly speaking say we were evaluating the
impact of EIS, or were we in fact measuring the impact of training, the introduction of
the VLE, or the increasingly widespread access to the internet from home? The truth
is likely to be a complex combination of factors, as the interviews indicate. Perhaps
closer consideration of the design of the research instruments could identify ways of
tackling these uncertainties.
Interviews with teaching staff clearly revealed that the impact of library EIS on
teaching was influenced by more than just the provision of EIS per se; their
perception of the benefits for their students and the barriers that exist to their own use
of EIS would also affect the extent of the impact.
However, as methods of delivery change through the e-learning culture attendant
upon the introduction of UWE’s VLE, so the impact of EIS is likely to increase. The
student survey would suggest that this is already happening and it is likely to continue
as more teaching staff become familiar with the technology and learn how to create
links to the resources available.
5.2 Outcomes of participating in the LIRG/SCONUL Impact Implementation
Had it not been for this initiative I doubt if we would have contemplated such an
ambitious project, in terms of the depth and breadth of our investigations and the
amount of staff time needed to devote to it. However, as a result of our participation
there have been a number of valuable outcomes.
In the process of carrying out interviews, the interaction with our users has led to a
deeper understanding of the academic processes involved in teaching, learning and
research; it also gave us an opportunity to promote aspects of the library service; and
to raise the profile of the library to some extent.
There have also been staff development opportunities. Our project team was able to
gain expertise in project management and research methods.
Participation in this initiative has given us the opportunity to evaluate research
methodologies and to develop the confidence to employ them in the future. It will be
useful to re-use the instruments in two years' time after the introduction of our open
url resolver and library portal software. Furthermore, we now have some useful
baseline data with which to compare future evaluations.
The data collected has provided us with some valuable information which can feed
into our planning for the future. In particular it has highlighted ways in which we can
increase the impact of the library’s EIS, by targeting training where it will be most
effective, by working closely with teaching staff to embed EIS into the e-learning
environment, and by actively endeavouring to remove some of the barriers to impact.
An important outcome was the opportunity to collaborate with the project team from
Bournemouth University who were also evaluating the impact of aspects of their EIS.
We were able to compare methodologies and results and this led to a joint paper being
presented at the 6th Northumbria International Conference on Performance
Measurement in Libraries and Information Services, Durham, August 2005, (Beard et
Another paper, based solely on our project at UWE, was presented at the Libraries
Without Walls 6 (LWW6) conference which took place in Lesvos, Greece in
September 2005 (Nelson et al. 2006). Dianne Nelson was subsequently invited to give
a presentation about the project to the MSc. Librarianship and Information
Management students at Loughborough University and at the Evaluating Academic
Libraries Conference in Wolverhampton on March 2nd 2006.
Are we having an impact?
The project has clearly demonstrated the value of EIS at UWE with its potential to
have an enormous impact on teaching and learning. However, what has also emerged
is that the extent, of this impact is dependent on a number of contextual factors.
Whether the amount of money the library spends on EIS is value for money depends
on whether we can maximize their impact by addressing some of the contextual
issues. This study has highlighted some of the ways this can be achieved.
1. Raise awareness of EIS through targeted promotion and training
Two findings of this study :
i) that only 54% of students identified themselves as users of EIS;
ii) and that the academic staff, and especially researchers, with years of
experience tended to use the resources that they were used to, and not
discover what new and possibly more useful resources could be available;
suggest that there is both a need for the library to make even greater efforts to
promote EIS to students in general, and also that we should be targeting established
members of staff for the promotion of new EIS resources. It was also felt that this
group would welcome one-to-one research-focused training aimed at their specific
2. Work with academics to help them identify relevant resources
In common with JISC’s JUSTEIS project (JISC usage surveys : trends in electronic
information services) (Urquhart, 2003), this study shows that academic staff are the
main influence on students’ decisions to use particular EIS (or not), more so than
library staff. Academic staff are therefore key to the impact of EIS on students’
learning. Whereas in the past the focus of library training has been on the student
population, this study highlights the importance of providing information literacy
training for academics, to enable them to identify, utilize and promote the relevant
electronic resources for their teaching.
This area of staff development and information literacy has been investigated by JISC
in their i-skills work. The documents relating to this work can be found at
3. Train students how to evaluate information
Furthermore, given the amount of information now available to students via EIS and
the internet it seems clear that, as well as the information skills needed to find and
retrieve information, students should be given the skills needed to evaluate the e-
resources they are using and the information thus obtained. Without these skills there
is a danger that students will bypass the EIS we pay for in favour of Google.
4. Develop new partnerships for information literacy skills training for
Within the e-learning environment, in which we all now operate, a more holistic
approach is needed for the delivery of information literacy skills training for students.
This will require new partnerships between academic staff, library staff, e-learning
coordinators, and the Centre for Research, Innovation and Graduate Studies (CRIGS).
More than ever, there is a need for a teamwork approach, with librarians working with
academic staff to support use of EIS for learning. This would ideally lead to the
convergence of library and teaching roles within the framework of e-learning support.
VLEs are essential tools in the delivery of e-learning. It is therefore important that
librarians, as well as teaching staff, learn how to use them effectively and to
understand the part they now play in teaching and learning. Only then will they be
able to offer advice on how the library’s EIS can be used effectively within this
5. Future evaluation
A final recommendation is that measurement of the impact of the library’s services
should be an ongoing process that will continuously feed into the library’s strategic
Beard, J. et al. (2006) Measuring the Impact of Electronic Resources : Developing
simple tools. Performance Measurement and Metrics: The International Journal for
Library and Information Services (In press)
Markless, S. and Streatfield, D. (2005) Facilitating the Impact Implementation
Programme. Library and Information Research, 29 (91), 10-15
Miles, M.B. & Huberman, A.M. (1994) Qualitative data analysis: an expanded
sourcebook. Sage, London
Nelson, D.G. et al. (2006) We know we are making a difference but can we prove it?
Impact measurement in a higher education library. In : Libraries Without Walls 6 :
Evaluating the Distributed Delivery of Library Services. Facet Publishing, London (In
Payne, P and Conyers, A (2005) Measuring the impact of higher education libraries:
the LIRG/SCONUL Impact Implementation Initiative. Library and Information
Research, 29 ( 91), 3-9
Urquhart, C., Thomas, R., Lonsdale, R., Spink, S., Yeoman, A., Fenton, R.,
Armstrong, C. (2003) Uptake and use of electronic information services: trends in UK
higher education from the JUSTEIS project. Program: electronic library and
information system, 37 (3), 168 - 180