Docstoc

nelson

Document Sample
nelson Powered By Docstoc
					We know we are making a
difference but can we prove it?
Impact measurement in a higher
education library


         Dianne Nelson

    University of the West of
     England (UWE), UK
               LWW6-05

                                  1
Introduction
• Project to measure the impact of EIS at UWE
• Aim: to determine whether the increasing amounts
  we spend on EIS each year are effective and
  justified

• EIS were defined as:
  the electronic information services for which the
  library pays a subscription fee e.g. Web of
  Knowledge, Science Direct etc.
                                                      2
Context
          The University of the
          West of England
          (UWE)
          • 26,000 students
          • 1000 academic staff
          • 7 campuses
          • 10 Faculties

          • Virtual Learning
          Environment (VLE) -
          Blackboard
                                  3
What is impact?

     “any effect of a service (or other ‘event’)
     on an individual or group” (Brophy 2002)
    something has an impact when it results
     in a change of some sort
    generally agreed that impact of library
     services is difficult to measure


                                                    4
Impact!




          5
LIRG/SCONUL Impact
Implementation Initiative

   LIRG     Library and Information
             Research Group

   SCONUL   Society of College, National
             and University Libraries


                                            6
The LIRG/SCONUL Impact
Implementation Initiative model
for measuring impact
   Action research principles
   Impact measurement methodology
     Specify measurable objectives

     Determine the success criteria

     Identify the evidence needed and how to
      collect it
                                                7
Project plan
                      Are we getting value for money?

       Objectives: to measure the impact of EIS on students’ learning,
                      staff teaching and research

 Success criteria for evaluating                Ways of measuring
 impact of EIS
      Students
     Increased use of EIS
     Increased remote use of EIS
     Use of EIS for a broad range of purposes                Questionnaire
     Teaching staff
     Increased use of EIS
                                                             Observation
     Increased promotion of EIS
     Increased number of links from VLEs
     Integration of EIS into the curriculum
                                                             Interview
     Researchers
     Increased use of EIS
     Awareness/use of EIS                                    Usage statistics
     Improved effectiveness of desk-based
                                                                                8
     research
Methodologies

   Online questionnaire
   Semi-structured interview
   Interview/participant observation
   Usage statistics




                                        9
key results:
Impact on student’s learning
From 387 responses – 217 said they used EIS
Of these 217 students:
   89% had access to the internet at home

   96% claimed to have good to very good IT skills

   Students found out about useful EIS from faculty
    sources (teaching staff, module documentation, and
    the VLE) in the ratio of 65:35 compared to Library
    sources (librarian, library web pages and library
    publicity)
                                                         10
key results:
Impact on student’s learning
Increased use of EIS
   72% said their use of EIS had increased
    over the last year

Increased remote use of EIS
   63% mainly accessed EIS remotely, in
    most cases from home


                                              11
Student use of EIS
                                    1   1. own reading for
                                            coursework
                                    2   2. recommended reading
                   5%   4%                  for coursework
              6%
        8%                    35%   3   3. primary data – statistics,
                                            financial data
             14%
                        28%         4   4. reference – maps,
                                            dictionaries etc.
                                    5   5. career/job related
                                            information
                                    6   6. Newspapers

                                    7   7. hobby or interest related
findi

                                                                  12
Key results:
Impact on staff teaching
Academics’ own awareness and use of
 EIS

   All had used some library databases.
   Most used the library and the library web-
    pages to find out what is available.
   Awareness of what EIS the library provides
    could have been better.
                                                 13
Key results:
Impact on staff teaching
Of the nine members of teaching staff
  interviewed
   All promoted EIS, in particular the ones they were
    familiar with
   Seven claimed to have integrated EIS into the
    curriculum
   The ways in which EIS was used in teaching included:
     pointers to relevant EIS in assignment briefs
     requirement for EIS to be used when completing
      assignments
     inclusion of references to EIS in module handbooks
                                                           14
Key results:
Impact on staff teaching
Increased number of links to EIS from VLEs
   Seven were using the VLE to either link to EIS,
    information about EIS, or articles obtainable through
    library EIS – or were intending to
   It was felt that using a VLE would help integrate EIS into
    the curriculum.

Comments
“The VLE has made it easier to recommend the electronic
  information services”
“VLEs have created a culture of using EIS”
                                                                 15
Key results:
Impact on staff teaching
Factors that affect the impact of EIS on teaching
Positive
• Students can retrieve large amounts of up-to-date information
  quickly and easily
• Improved student support by using EIS to provide links to learning
  resources
• Student dissertations now contain a wider range of references
• Many students prefer electronic resources
Negative
• Students are unable to evaluate the large amount of information
  available to them
• Ease of plagiarism
• Spoon-feeding
                                                                       16
Key results:
Impact on researchers
Awareness/use of EIS within researchers’
 subject area
   All participants were aware of the importance of EIS
    for their research, as illustrated by comments such
    as:
“Very important – main source of information for my
   research”
“Vital – essential to the research process”
“Absolutely invaluable”
“Absolutely central. I cannot imagine what it would be like
   without them”
                                                              17
Key results:
Impact on researchers
Improved effectiveness of desk-based
  research when using EIS
 There was a wide range of ability, and some
  evidence that increased effectiveness
  correlated with increasing years of research
  experience.
 Evidence showed that the impact of EIS on
  researchers could be further increased by
  more research focused training and individual
  face-to-face training.                          18
Barriers to impact of EIS

    Not knowing what’s available
    Lack of time
    Habit
    Passwords/Access
    Lack of confidence with IT


                                    19
Usage data
                     2000/01 2003/04 %increase
AMED                 747     9688    1197%
CINAHL               8410    40983 387%
Compendex/EI Village 442     1279    189%
Education databases 975      3619    271%
Emerald              5159    6130    19%
IBSS                 1341    2385    78%
LEXIS-NEXIS          9662    31990 231%
Medline              9023    31790 252%
Mintel               809     5943    635%
PsycINFO             809     7793    863%
Science Direct       14225 15927 12%
SportDiscus          414     1860    349%
Zetoc                315     1443    358%
                                                 20
    Summary
   Increased use of EIS
   Change in learning behaviour – high level of remote use
   Students’ use of EIS is only in partly influenced by librarians’
    teaching and recommendations. Academic staff are key
   Teaching staff promote the EIS they are familiar with
   Teaching staff awareness of what EIS are available was varied,
    and in many cases limited and/or inaccurate
   Widespread enthusiasm for EIS, especially among researchers
   Signs are that academic staff are beginning to integrate EIS into
    the curriculum
   The VLE will be the main driver for this integration
   Effectiveness of researchers’ use of EIS correlated with the length
    of their research experience
   There are a number of barriers that are likely to reduce the impact
    of EIS                                                              21
Discussion:
To what extent did we succeed in
measuring the impact of EIS?
   Methodological shortcomings
       Self selecting participants
       Small sample numbers
       Interviewer bias

   Lack of base-line measurements
   Choice of success criteria
   Unable to separate out impact of EIS from impact
    of other factors, e.g.
       Impact of VLE
       Internet access from home
                                                       22
Conclusions 1.
Are we getting value for money from our
 library’s EIS?
    There are indications that EIS are having an
     impact on learning, teaching and research at the
     University of the West of England.
    Therefore the amount that we spend each year on
     EIS is justified
    However it could be more effective and this study
     has highlighted ways in which the impact of EIS
     could be increased to give us a greater return on
     our investment                                      23
Conclusions 2.
    Ways of increasing our return on investment
     (ROI)

   Review and develop promotion and training
   Recognise the centrality of academic staff as key to
    influencing student behaviour
   Librarian/academic partnerships
   VLE training for teaching and library staff
   Identify barriers and do what we can to remove
    them
                                                           24
Conclusions 3.
Outcomes of participating in the LIRG/SCONUL
 Impact Implementation Initiative

   Enhanced relationships with users
       Deeper understanding of academic processes by
        library staff
       Opportunity for promotion of specific aspects of
        library service
       Raising profile of library in general

   Staff development opportunities
   Useful baseline data for future evaluations            25
Future Direction

   Continue to monitor EIS usage
   Continue to use qualitative methods to
    survey the impact of our library
   Define a strategy for incorporating
    findings of evaluation processes into the
    library’s strategic planning services


                                                26
      Thank you

Questions/Comments?

Dianne.Nelson@uwe.ac.uk

                          27

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:0
posted:6/4/2013
language:English
pages:27
langkunxg langkunxg http://
About